Top
Best
New

Posted by never_inline 7 days ago

Things you can do with a debugger but not with print debugging(mahesh-hegde.github.io)
254 points | 214 comments
jonhohle 3 days ago|
I don’t mind using a debugger, but one of the advantages to printed debugging is that it is universal. I work on a project with code in five languages (6 if you count the build system, 7 if you add shell scripts). Two of them I know the debugger very well. One of them I might be able to get by. One of the others I might have use the debugger once and the last I’ve never touched.

printf debugging works in all of them, even the build system and shell scripts.

A debugger can be great, no question about it, even for remote debugging. In my experience, I’ve seen fewer people effectively debug unfamiliar systems quickly.

jbverschoor 2 days ago|
That’s just bc most tooling is utter crap, except for Java+eclipse
swaits 4 days ago||
Author missed one of the best features: easy access to hardware breakpoints. Breaking on a memory read or write, either a raw address or via a symbol, is one of the most time saving debugging tools I know.
jesse__ 4 days ago||
Oh my god, same. This literally catches bugs with a smoking gun in their hand in a way that's completely impossible with printf. I'd upvote this 100 times if I could.
sherincall 4 days ago||
> completely impossible with printf

Not printf exactly, but I've found bugs with a combination of mprotect, userfaultfd and backtrace_symbols when I couldn't use HW breakpoints.

Basically, mark a set of pages as non-writable so that any writes trigger a pagefault, then register yourself as a pagefault handler for those and see who is doing the write, apply the write and move on. You can do this with LD_PRELOAD without even recompiling the debugee.

antonchekhov 3 days ago|||
Back in the early '90s, I was a big fan of the C/C++ debugging library "electric fence" (written by Bruce Perens) - it was a malloc() implementation that used mmap() to set the pages of the returned buffer such that any writes in that region (and even read accesses!) caused a segfault to happen, and the program halts, so you can examine the stack. It was a godsend.
jesse__ 3 days ago|||
Oh yeah, I've done that trick before, it's quite handy. Sometimes catches bugs that would be tricky to find with watchpoints. I wrote an allocator that allocates an extra page either before or after the users allocation, and aligns the user allocation such that it is as close as possible (while respecting user-requested alignment) to butting up against that extra page (marked PROT_NONE). Can catch underflows or overflows with this method, but not both simultaneously. Very handy from time to time.
coderatlarge 4 days ago|||
windbg used to offer scripting capabilities that teams could use to trigger validation of any number of internal data structures essentially at every breakpoint or watchpoint trigger. it was a tremendous way to detect subtle state corruption. and sharing scripts across teams was also a way to share knowledge of a complex binary that was often not encoded in asserts or other aspects of the codebase.
bpye 4 days ago||
This still exists? You can also use JavaScript to script/extend and there is a native code API too.

Note: I do work at MSFT, I have used these capabilities but I’m not on the debugger team.

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

https://www.timdbg.com/posts/whats-the-target-model/

https://github.com/microsoft/WinDbg-Samples/tree/master

coderatlarge 4 days ago||
thanks for the pointers glad to hear it’s all still there

i haven’t seen this type of capability used in too many companies tbh and it seems like a lot of opportunity to improve stability and debugging speed and even code exploration/learning (did i break something ?)

delta_p_delta_x 3 days ago||
WinDbg also has time-travel/record-replay debugging.
coderatlarge 3 days ago||
that sounds like a super-powerful feature especially for multithreaded debugging though i’ve never used anything like it.
AndyKelley 3 days ago|||
Combined with time travel it's mind-blowing.

Stumble upon some corrupted memory? Just put a watch point on it, and run the program backwards. Two weeks just turned into two minutes.

mwcz 3 days ago||
What time traveling debugger(s) do you recommend? I'm a regular GDB user and keep meaning to try rr.
AndyKelley 3 days ago|||
Since you're already a GDB user, rr will probably feel comfortably familiar.

https://rr-project.org/ already has enough example usage to get you going on the home page right there.

adgjlsfhk1 3 days ago|||
rr is excellent. the only problem is it doesn't work for Windows
delta_p_delta_x 3 days ago||
It's a little odd to use GDB on Windows. Have you considered either the Visual Studio debugger, or WinDbg?

WinDbg has time travel debugging, and is arguably 'more excellent': https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...

praptak 4 days ago|||
From the same toolbox: expression watch. Set a watch on the invariant being violated (say "bufpos < buflen") and get a breakpoint the moment it changes.
roca 4 days ago|||
Especially with combined with reverse-execution in rr or UndoDB!
yakshaving_jgt 4 days ago||
Is there somewhere where this approach is described in more detail?
jesse__ 4 days ago|||
Very roughly, hardware watchpoints are memory addresses you ask the processor to issue an "event" for when they're read from, written to, or executed. This event is processed by the kernel, and passed through to the debugger, which breaks execution of the program on the instruction that issued the read/write/exec.

A concrete use case for this is catching memory corruption. If your program corrupts a known piece of memory, just set a hardware watchpoint on that memory address and BOOM, the debugger breaks execution on exactly the line that's responsible for the corruption. It's a fucking godsend sometimes.

lock1 4 days ago|||
Search for "watchpoint debugging". Usually in most garbage-collected environments, it just observes & breaks on symbols though, not raw addresses.

Vscode (or an editor with ADP support) supports unconditional breakpoints, watchpoints, and logpoints (observe & log values to the debug console).

SAI_Peregrinus 3 days ago||
What watchpoints observe depends on the device & environment. E.g. for ARM Cortex-M MCUs, the (optional but very common) Data Watchpoint and Trace Unit (DWT) provides 1 or 4 watchpoints by using hardware comparators. They operate on raw addresses, since they work at the machine code level. They issue a debug interrupt when the comparator matches the address & operation (read/write), which the debug probe (J-Link, ST-Link, or other such probe) detects & forwards to the debugger (GDB, LLDB, etc.). Hardware watchpoints are extremely useful for debugging memory corruption in embedded systems, since they bypass the code running on the device under test entirely.
makeitdouble 4 days ago||
While a debugger is of high value, having access to a REPL also covers the major use cases.

In particular, REPL tools will work on remote session, on pre-production servers etc. _if_ the code base is organized in a somewhat modular way, it can be more pleasant than a debugger at times.

Makes me wonder if the state of debugging improved in PHP land. It was mostly unusable for batch process debugging, or when the server memory wasn't infinite, which is kinda the case most of the time for us mere mortals.

never_inline 4 days ago|
I am the author of the posted flamebait. I agree.

I use IPython / JShell REPLs often when the code is not finished and I have to call a random function without entrypoint.

In fact its possible to jump to the graphical debugger from the Python REPL when running locally. PyCharm has this feature natively. In VSCode you can use a simple workaround like this: https://mahesh-hegde.github.io/posts/vscode-ipython-debuggin...

smlavine 4 days ago||
It's not a silver bullet, but Visual Studio is leaps and bounds ahead of gdb et. al. for debugging C/C++ code. "Attach to process" and being able to just click a window is so easy when debugging a large Windows app.
jesse__ 4 days ago||
lol, agree to disagree here. While the interface to gdb is annoying, there are many gui frontend alternatives.

VS, on the other hand, gets worse with every release. It is intolerably slow and buggy at this point. It used to be a fantastic piece of software, and is now a fantastic pile of shit.

cenamus 4 days ago||
Any recommendations on gdb frontends? Have tried with emacs, but I just really enjoy the point and click stuff, emacs keybinds don't work for me there.
westurner 3 days ago|||
From https://news.ycombinator.com/item?id=35710350 :

> ... py-list, py-up and py-down, py-bt, py-print, and py-locals GDB commands

> [ DDD, pyclewn (vim), trepan3k, Voltron, gdbghidra ]

gdbghidra: https://github.com/Comsecuris/gdbghidra

radare2: https://github.com/radareorg/radare2

voltron: https://github.com/snare/voltron

And from https://news.ycombinator.com/item?id=41943521 :

> pyvmidbg (libVMI + GDB)

But that's archived.

There's a QEMU gdbstub GDB interface.

To print registers with GDB:

  info reg
  info all-registers
  i r a
spacechild1 4 days ago||||
IMO the GDB frontend in QtCreator is quite nice.
jesse__ 3 days ago|||
GitHub.com/nakst/gf
dahart 3 days ago|||
It really depends. The VS debugger is excellent, and it is usually easier I agree there. But gdb is more easily scriptable and more powerful in certain situations, so I reach for gdb sometimes for the harder bugs even though I use VS more often. Of course, they don’t really compete with each other that much, since VS is mainly Windows and gdb is mainly Linux. Of course I know you can do either debugger on either OS, but in practice the debugger choice is primarily OS based.
o11c 3 days ago||
Under X11 that can be done with:

  gdb -p `xprop _NET_WM_PID | sed 's/.*= //'`
I think under Wayland it might not be possible since Wayland gratuitously breaks all nice things.
jasonjmcghee 4 days ago||
Something I haven't seen discussed here that is another type of debugging that can be very useful is historical / offline debugging.

Kind of a hybrid of logging and standard debugging. "everything" is logged and you can go spelunk.

For example:

https://rr-project.org/

smj-edison 4 days ago||
I've loved working with rr! Unfortunately the most recent project I've been contributing to breaks it (honestly it might just be Ubuntu, as it works on my arch install, but doesn't work when deployed where I need to test it).
cluckindan 4 days ago||
Print debugging is historical / offline debugging, just ad-hoc instead of systemic.

The ”debug” package on npm is something in between, as it requires inserting debug statements but they are hidden from output unless an envvar like DEBUG=scope.subscope.*,otherscope is used.

squirrellous 4 days ago||
IME console-based debuggers work great for single-threaded code without a lot of console output. They don't work that well otherwise. GUI-based debuggers can probably fix both of those issues. I just haven't really tried them as much.

pdb is great for python, though.

MrDarcy 4 days ago|
I frequently use the go debugger to debug concurrent go routines. I haven’t found it any different than single threaded debugging.

I simply use conditional break points to break when whatever go routine happens to be working on the struct I care about.

Is there more to the issue?

squirrellous 4 days ago||
Thinking back, the issue I had with multi-threaded code was two-fold:

- Things like "continue", "step" are no longer a faithful reproduction of what the program does in real time, so it's more difficult to understand the program's behavior. Some timing-related bugs simplify disappear under a debugger.

- There's usually some background thread that's logging things to console, which reduces to problem 2 in my comment.

I haven't used Go that much. I imagine since goroutines are such a cornerstone of the language, the go debugger must have some nifty features to support multi-(green)-threaded debugging?

wheybags 3 days ago||
In my experience, conditional breakpoints are so unreliable, that I just dont bother trying to use them. When I need one I add code like

    if (condition) 
        print("");
Then add a breakpoint on the print. Calling print ensures the line with the breakpoint won't be optimised out (rarely matters as I'm normally debugging in... debug mode, but it's just a reflex at this point, and I need to put something in there)
ziml77 3 days ago||
I love debuggers but yeah conditional breakpoints are pretty bad. I tend to need to conditionally break when the same code is being called repeatedly with different inputs where only one of them causes the problem. The conditional breakpoint slows things down so much that I'd probably turn to dust before it ever actually halted execution.

And I work in C#, so I have no clue why there's no option to inject the condition and a call to Debugger.Break() and then have the JIT recompile the function live. It would actually make conditional breakpoints usable!

fluoridation 3 days ago||
In my experience, they're not unreliable, but they do massively slow down the application being debugged. I often do what you suggest, except with __debugbreak() (which compiles to int3).
inglor_cz 4 days ago||
I don't really get the hate that debuggers sometimes get from old hands. "Who needs screwdrivers if we always used knives?" - You can still use your knife, but screwdriver is a useful tool.

It seems to me that this is one of the many phenomena where people want to judge and belittle their peers over something completely trivial.

Personally, I get the appeal of printing out debugging information, especially if some bug is rare and happens in unpredictable times (such as when you are sleeping). But the amount of info you get this way is necessarily lower than what can be gleaned from a debugger.

sfpotter 4 days ago||
It isn't either/or. Good programmers know how to use both and know how to choose the appropriate tool for the job.
z_open 4 days ago|
printing is never the appropriate tool. You can make your debugger print something when that line of code is reached anyway and automatically continue if you want. So what's the point of pritntf? It's just less information and features.
ahartmetz 4 days ago|||
Let me enumerate. Printf survives debugger restarts, shows up in git diff, usually messes less with the timing, can be exchanged with coworkers or deployed to users and be toggled with logging rules, has the full power of the programming language, the output is easier to run "diff" on to compare runs, works in CI containers, has no problems with mixed language environments...

As far as I'm concerned, breakpoints and backtraces, especially of crashes, are the superpower of debuggers. Where they are not immediately applicable, I usually don't bother.

horsawlarway 3 days ago||||
Nah.

Print debugging optimizes for timing detail and fidelity.

Debuggers optimize for data detail, at the expense of timing detail and fidelity.

In my opinion - timing & order of events is actually more meaningful.

I often don't need the extra data detail, and the debugger is going to change timing in meaningful ways.

Both are skills - both have their place. Know when logs are useful, know when debuggers are useful. Don't get stuck in the fallacy of "The one true way!" - that's the only thing I can promise is bullshit.

If you've never debugged a problem that goes away when a debugger is attached - you're woefully unprepared to have this conversation (It's much rarer, but I can even talk about times where adding a single print line was enough to change timing and make the issue harder to reproduce).

At the very least - with prints you get a tangible record of the order of operations during a reproduction. It doesn't go away, I don't have to remember how many times I've hit continue, I don't have to worry about whether pausing at this breakpoint is changing the world (and it is, because other things keep chugging - the network doesn't give a fuck that you're paused at a break point).

dahart 3 days ago||||
I’m a huge advocate for using debuggers, but saying never print is too dogmatic, and sometimes incorrect. There are plenty of environments where a debugger is not available or very difficult to setup - GPU shaders historically, embedded environments on small/custom hardware, experimental languages, etc.. Printing is both very easy, and often good enough. You should probably reach for a debugger if you keep adding prints and recompiling or if you don’t fix your bug in a couple of minutes. But aside from that, print debugging is useful and has its place, even on occasions when a good debugger is available. Never say never.
unwind 3 days ago||
I haven't touched GPU programming in ... uh ... decades, but is print debugging readily available for shaders? That was surprising, but glad to hear it! :)
dahart 2 days ago||
Good point, it’s getting better, but often print statement debugging is not available in shaders either, and you have to resort to the visual form of print debugging: outputting a tinted color. Crude, but often enough it’s plenty effective. Personally, I mentally put shader tint debugging in the same category as CPU print debugging.
jesse__ 3 days ago||||
I'm firmly in the "use a debugger" camp, but printf is sometimes indispensable when attempting to debug race conditions. By confirming timing invariants hold with printf, you can usually narrow in fairly quickly on the problem. Doing the same in a debugger is much more of a hassle. It's not impossible, certainly, but it's way more of a pain in the ass.
oreally 3 days ago|||
printing should definitely be the tool of last resort.

No one can argue how many keystrokes and brain cycles it saves using a debugger vs going through the task printing every variable.

old_bayes 4 days ago|
It may sound obvious to folks who already use a debugger, but in my experience a decent chunk of people don't use them because they just don't know about them.

Spread the good word!

makeitdouble 4 days ago||
Depending on the language or setup debuggers can be really crappy. I think people here would just flee away and go find a better fitting stack, but for more pragmatic workers they'll just learn to debug with the other tools (REPL, structured logging, APMs etc.)
adgjlsfhk1 3 days ago||
if your repl is good enough and the bug isn't a crash, repl based debugging can be excellent (especially if the language has good introspection capabilities)
giveita 4 days ago|||
I had a think about where I first learned to use a debugger. The combo of M$ making it easy for .NET and VB6 and working professionally and learning from others was key. Surprised it is less popular. Tests have made it less necessary perhaps BUT debugging a unit test is a killer move. You quickly get to the breakpoint and can tweak the scenario.
NitpickLawyer 4 days ago|||
> I had a think about where I first learned to use a debugger

Is this not taught anymore? I started on borland C (the blue one, dos interface) and debugging was in the curriculum, 25+ years ago. Then moving to visual studio felt natural with the same concepts, even the same shortcuts mostly.

giveita 4 days ago||
Nothing useful I do in my job was taught by another person in a classroom.
cluckindan 4 days ago||
Clearly, you have been in the wrong classroom.
marssaxman 3 days ago|||
A good number of us were never in any classroom, when it comes to the skills we use in our careers.
giveita 4 days ago|||
Or the wrong jobs!
robertfw 3 days ago|||
my development workflow with python often boils down to repeatedly running `pytest --ff --pdb -x`...

- run my tests, running any previous failures first - drop into the debugger when a test fails (or of course, hits a breakpoint) - end the test run after the first failure

use debugger to figure out what went wrong, fix it, wash, rinse, repeat

nchmy 4 days ago|||
yeah, tons dont know they exist. But there's also a lot of people - new and veteran - who are just allergic to them, for various reasons.

Setting up a debugger is the very first thing i do when i start working with a new language, and always use it to explore the code on new projects.

DrSiemer 4 days ago|||
These days I'll just dump all relevant code into an LLM and have it explained to me instantly.

Being able to ask questions about the parts that are unclear (or just plain wrong) is so much easier than trying to cram the entire thing into my brain RAM.

In my experience it actually helps me learn faster too, since I rarely get stumped on random gotcha's anymore.

nchmy 3 days ago||
Nah, i use debuggers all the time despite also using LLM agent all the time. They are great, but limited and often just go into a perpetual loop of not being able to debug something. I toss in some breakpoints and figure out exactly what is going on.
giveita 4 days ago|||
With VSCode it's often a 10 minute job to set up. We are spoiled! Back in the VS days using a Microsoft stack it was just there. Click to add breakpoint then F5.
nchmy 3 days ago|||
when i first started programming, with php, it took me literally weeks to figure out how to get xdebug set up. (pro tip: just use DDEV - it makes it all simple).

Not a second of that was a waste because xdebug (like all debuggers) is just that amazing.

gonzo41 4 days ago||
This also applies to testing. So much legacy code out there that's untested.
More comments...