Posted by never_inline 7 days ago
Anecdotally, dynamic expressions are impossibly slow in the cases I’ve tried them.
As the author mentions, there are also a number of cases where debuggers don’t work. Personally, I’m going to reach for the tool that always works vs. sometimes works.
This is only logical if you're limited to one tool. Would you never buy a power tool because sometimes the power goes out and a hand tool is your only choice?
1. stop the program
2. edit it to add the new log
3. rebuild the program
4. run it
5. get the program to the same state to trigger the log
And then you see the result of what you printed, figure out you need something else as well, and repeat. Instead you can just trigger a breakpoint and inspect the entire program's state.
* You're using to doing things the Windows way, where debug and release are completely different profiles rather than orthogonal flags. Or,
* You're compiling using Clang, which gives up and limits debuggability to simple backtraces when optimization is on. Or,
* Your project is hopelessly bloated and linking with debuginfo is slow, but you haven't figured out external debuginfo.
With GCC, `-g -O2` is a reasonable thing to use for all builds and provides a pretty good debugging experience with only minor competence required to find equivalences for any <optimized out>s.
There is an inherent tradeoff between interaction and reproducibility. I think the whole conversation of debugger vs print debugging is dumb. Just do whatever makes you the most productive. Often times it is immediately obvious which makes more sense.
For really hairy bugs in programs that can't be stopped (kernel/drivers/realtime, etc) logging works.
And when it doesn't, like when you can't do I/O or switching of any kind, log non-blocking to a buffer that is dumped elsewhere.
also, related. It is harder than it should be to debug the linux kernel. Just getting a symboled stack trace is ridiculously hard.
I was saying that for practical purposes, debugging the kernel is needlessly hard.
As one example, say you are running the current ubuntu. A kernel panic will give you a bunch of hex addresses. How do you get a symbol'd backtrace of the kernel panic?
(if you can't get that easily, running kgdb won't let you do symbolic debugging either)
Leave us be. We know what we’re doing.
- everyday bugs, just put a breakpoint
- rare cases: add logging
By definition a rare case probably will rarely show up in my dev environment if it shows up at all, so the only way to find them is to add logging and look at the logs next time someone reports that same bug after the logging was added.
Something tells me your debugger is really hard to use, because otherwise why would you voluntarily choose to add and remove logging instead of just activating the debugger?
Like the bugs "that disappear in a debug build but happen in the production build all the time".
Lucky you lol
What I've found is that as you chew through surface level issues, at one point all that's left is messy and tricky bugs.
Still have a vivid memory of moving a JS frontend to TS and just overnight losing all the "oh shucks" frontend bugs, being left with race conditions and friends.
Not to say you can't do print debugging with that (tracing is fancy print debugging!), but I've found that a project that has a lot of easy-to-debug issues tends to be at a certain level of maturity and as times goes on you start ripping your hair out way more
I'm having the most fun I've had in ages. It's like being Sherlock Holmes, and construction worker all at once.
Print statements, debuggers, memory analyzers, power meters, tracers, tcpump - everything has a place, and the problem space helps dictate what and when.
The only language where I've found a debugger particularly useful for race condition debugging is go, where it's a lot easier to synthetically trigger race conditions in my experience.
- You can add new traces, or modify/disable existing ones at runtime without having to recompile and rerun your program.
- Once you've fixed the bug, you don't have to cleanup all the prints that you left around the codebase.
I know that there is a good reason for debugging with prints: The debugging experience of many languages suck. In that case I always use prints. But if I'm lucky to use a language with good debugging tooling (e.g Java/Kotlin + IntelliJ IDEA), there is zero chance to ever print for debugging.
Print debugging in frontend JS/TS is literally just writing the statement "debugger;" and saving the file. JS, unlike supposedly better designed languages, is designed to support hot reloading so often times just saving the file will launch me into the debugger at the line of code in question.
I used to write C++, and setting up print statements, while easier than using LLDB, is still harder than that.
I still use print debugging, but only when the debugger fails me. It's still easier to write a series of console.log()s than to set up logging breakpoints. If only there was an equivalent to "debugger;" that supported log and continue.
no it's not lol. hmr is an outrageous hack of the language. however, the fact JS can accommodate such shenanigans is really what you mean.
sorry I don't mean to be a pedantic ass. i just think it's fascinating how languages that are "poorly" designed can end up being so damn useful in the future. i think that says something about design.
If I find myself using a debugger it’s usually one two things: - freshly written low level assembly code that isn’t working - basic userspace app crash (in C) where whipping out gdb is faster than adding prints and recompiling.
Even never needed a debugger for complex kernel drivers — just prints.
Perhaps the debugging experience in different languages and IDEs is the elephant in the room, and we are all just talking past eachother.
If the customer has their own deployment of the app (on their own server or computer), then all you have to go with, when they report a problem, are logs. Of course, you also have to have a way to obtain those logs. In such cases, it's way better for the developers to also never use debugger, because they are then forced to ensure during development that logs do contain sufficient information to pinpoint a problem.
Using a debugger also already means that you can reproduce the problem yourself, which is already half of the solution :)
I just debug release mode instead, where print debug is usually nicer than a debugger without symbols. I could fix the situation other ways, but a non-reversible debugger doesn't justify the effort for me.
When using Xcode the debugger is right there and so it is in qt creator. I’ve tried making it work in vim many times and just gave up at some point.
The environment definitely is the main selector.
But yeah, user space debugging with a software debugger (gdb, etc…) is certainly useful for some things.
But what to do if you have a race condition in a database stored procedure? Or in a GUI rendering code? Even web applications can experience race conditions in spite of being "single-threaded", thanks to fetches and other asynchronous operations. I never heard of somebody using ICE in these cases, nor can I imagine how it could be used - please enlighten me if I'm missing something...
> You're changing the conditions that prevent accurate measurement without modification.
Yes, but if the race condition is course-enough, like it often is in above cases, adding print/logging may not change the timings enough to hide the race.
And the print will 100% change the timing.
Interesting. I usually find those harder to debug with a debugger. Debuggers change the timing when stepping through, making the bug disappear. Do you have a cool trick for that? (Or a mundane trick, I'm not picky.)
I am in camp where 1% on the easy side of the curve can be efficiently fixed by print statements.
No shade, this was my perspective until recently as well, but I disagree now.
The tipping point for me was the realisation that if I'm printing code out for debugging, I must be executing that code, and if I'm executing that code anyway, it's faster for me to click a debug point in an IDE than it is to type out a print statement.
Not only that, but the thing that I forgot to include in my log line doesn't require adding it in and re-spinning, I can just look it up when the debug point is hit.
I don't know why it took me so long to change the habit but one day it miraculously happened overnight.
Interesting. I always viewed the interface to a debugger as its greatest flaw—who wants to grapple with an interface reimplementing the internals of a language half as well when you can simply type, save, commit, and reproduce?
I'm using IntelliJ for a Java project that takes a very long time to rebuild, re-spin and re-test. For E2E tests a 10-minute turn-around time would be blazingly fast.
But because of the tooling, once I've re-spun I can connect a debugger to the JVM and click a line in IntelliJ to set a breakpoint. Combined, that takes 5 seconds.
If I need to make small changes at that point I can usually rebuild it out exactly in the debugger to see how it executes, all while paused at that spot.
i do, because it's much faster than typing, saving, and rebuilding, etc.
If your code can be unit tested, you can twist and turn it in many ways, if it's not an integration issue.
No. You’re wrong.
I’ll give you an example a plain vanilla ass bug that I dealt with today.
Teammate was trying to use portaudio with ALDA on one of cloud Linux machines for CI tests. Portaudio was failing to initialize with an error that it failed to find the host api.
Why did it fail? Where did it look? What actual operation failed? Who the fuck knows! With a debugger this would take approximately 30 seconds to understand exactly why it failed. Without a debugger you need to spend a whole bunch of time figuring out how a random third party library works to figure out where the fuck to even put a printf.
Printf debugging is great if it’s within systems you already know inside and out. If you deal with code that isn’t yours then debugger is more then an order of magnitude faster and more efficient.
It’s super weird how proud people are to not use tools that would save them hundreds of hours per year. Really really weird.
On the development system, the program would only crash, under a heavy load, on the order of hours (like over 12 hours, sometimes over 24 hours). On the production system, on the order of minutes (usually less than a hour). But never immediately. The program itself was a single process, no threads what-so-ever. Core dumps were useless as they were inconsistent (the crash was never in the same place twice).
I do think that valgrind (had I known about it at the time) would have found it ... maybe. It might have caught the memory corruption, but not the actual root cause of the memory corruption. The root cause was a signal handler (so my "non-threaded code" was technically, "threaded code") calling non-async-safe functions, such as malloc() (not directly, but in code called by the signal handler). Tough lesson I haven't forgotten.
It is not the only tool in the bag. But literally the first question anyone should ask when dealing with any bug is “would attaching a debugger be helpful?”. Literally everyone who doesn’t use a debugger is less effective at their jobs than if they frequently used a debugger.
Wait until the memory is corrupted and causes a crash. Set a hardware breakpoint on the corrupted memory location and run backward until the memory location was written in the signal handler. Problem solved.
Memory corruption bugs in single-threaded code are a solved problem.
I use logs and printf. But printf is a tool of last resort, not first. Debugging consideration #1 is “attach debugger”.
I think the root issue is that most people on HN are Linux bash jockeys and Linux doesn’t have a good debugger. GDB/LLDB CLI are poop. Hopefully RadDebugger is good someday. RadDbg and Superluminal would go a long long way to improving the poor Linux dev environment.
There are a few other instances where the interaction offers notable benefits—bugs in the compiler, debugging assembly, access to registers, a half-completed runtime or standard library that occludes access to state so that you might print it. If you have the misfortune of working with C or C++, you have the benefit of breaking on memory access—but I tend to file this in the "half-completed runtime" category. There are also a few "heisenbugs" that may actually prevent the bug from occurring by using print itself; but I've only run into this I think twice. This is also possible with the debugger, but I've only run into that once. The only way out of that mess is careful reasoning, and i recommend printing the code out and using a pen.
I also strongly suspect that preference for print debugging vs interactive debuggers comes down to internal conception of the runtime and aesthetic preference. I abhor debuggers—especially thosr in IDEs. I think they tend to reimplement the runtime of a language a second time, except with more bugs and a less intuitive interface. But I have the wherewithal to realize that this is ultimately a preference.
So many problems can be solved with these.
And then there's some more specialized tooling depending on what you're doing that can be a huge help.
For SQL, the query planner and index hit/miss / full table scan.
And things like valgrind or similar for cache hit/miss.
Proper observability (spans/ traces) for APIs...
Knowing that the tools exist and how to use them can be the difference between software and great software.
Though system design / architecture is very important as well.
I mean, don't get me wrong, I do agree engineers should at least be aware of the existence of debuggers & profilers and what problems they can solve. It's just that not all the stuff you've said belongs in the "must know" category.
I don't think you'll need valgrind or query planning in web frontend tasks. Knowing them won't hurt though.
All of these are useful skills in your toolkit that give you a way of reasoning about programs. Sure you can plop console.logs everywhere to figure out control/program flow but when you have a much more powerful tool specifically built for this purpose, wouldn't you, as an engineer, attempt to optimize your troubleshooting process?
But I think promoting profilers is much more important than debuggers. Far too many people I know are too eager to jump on "optimization" just because some API is too slow without profiling it first.
For JS in the browser, there's a often chain of transformations - TypeScript, Babel, template compilation, a bundler, a minifier - and each of these makes the browser debugger work worse -- and it's not that great to begin with, even on plain JS.
Add that to the fact that console.log actually prints objects in a structured form that you can click through and can call functions on them from the console, and you start to see why console.log() is the default choice.
I work on maintaining a 3D rendering engine written completely in Typescript, along with using a custom, stripped down version of three.js that I rely on for primitives; and no amount of console.logging will help when you're trying to figure out exactly what's going wrong in a large rendering pipeline.
I do use console.logs heavily in my work, but the debugger and profiler are instrumental in providing seamless devex.
> TypeScript, Babel, template compilation, a bundler, a minifier
During development you have access to source maps, devtools will bind breakpoints, show original typescript code and remap call stacks across bundlers. All modern browsers support mapped debugging, also wrt profiling it can also be symbol mapped to the original sources which makes minified builds diagnosable if you ship proper source maps, which during development you ideally should.
-=-
edit: additional info;
I would also like to say console.log and debugging/profiling are not in a competition. both are useful in different contexts.
for example I will always console.log a response from an API because I like having a nice nested representation that I can click through, I'll console.log objects, classes and everything to explore them in an easier way. this is also great for devex.
I'll use the debugger when I want to pause execution at an intermediate step; for example see the result of my renderer before the postprocessing step kicks in, stop it and inspect shader code before its executed. it's pretty useful.
As mentioned originally; these are TOOLS in your toolkit, you don't have to do a either/or between them.
If you don't already know which tool to use / how to diagnose the problem, you'll instead of banging your head against the wall, you'll think - "how do i figure out this thing - what is the right tool for this job"? and then you'll probably find it, and use it, because people are awesome and build incredibly useful free / open source software.
"try stuff until it works" is so common, and the experience needed to understand how to go about solving the problem is within reach.
Like especially with llms, "what's the right tool to use to solve problem x i'm having? this is what's going on. i'm on linux/macos, using python" or w/e
1. Code where the granularity of state change is smaller than a function call. Sometimes you actually have to step through things one instruction at a time, and I'm lucky enough to have such problems to solve. You can't debug your assembly with printf(), basically[1a].
2. State changes that can't be easily isolated. Sometimes you want to log when something change but can't for the life of you figure out when it's changing. Debuggers have watchpoints.
But... that's really it. If I'm not hitting one of those I'm not reaching for the debugger. Logging is just faster, because you type it in right at the code you're already reading.
[1a] Though there's a caveat: sometimes you need to write assembly and don't even have anything like a printk. Bootstrap code for a new device is a blast. You just try stuff like writing one byte to a UART address or setting one GPIO pin as the first instructions and hope it works, then use that one bit of output to pull the rest up.
But I spend far more time reading and thinking than I do typing. Input mechanics just aren't the limiting factor here.
Secondly, in your example, no need to label the names. This is almost always understood by context. So, pretty manageable. e.g. in JS: log(`${longvarname}, ${secondvarname}`)
Some other things I'd add:
Some debuggers allow you to add actions. For example logging at the breakpoint is great if I can't modify the source, plus there's nothing to revert afterward. This just scratches the surface. Some debuggers allow you to see entire GPU workloads, view textures etc.
Debuggers are extremely useful for exploring and helping edit code. I can't be the only person that sprinkles breakpoints during development which helps me visualise code flow and quickly jump between source locations.
They're not just for debugging.