Introduction
David J. Agans starts every chapter in his book “Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Hardware and Software Problems” with a quote from a different Sherlock Holmes book. This gives you the feeling of being a detective trying to solve a crime, only you don’t wear a funny hat, and the crime to solve is the bug you’re trying to diagnose. It seemed to me like a fun book to read (plus it’s only 192 pages, how hard could it be?), so I delved in, hoping to find the rare mix of fun and knowledge.
“The most important point of this book is reminding: debugging is full of methodology!” I read this line in a review of this book on Goodreads, and it got me interested to read it. No one teaches you debugging, and not enough books are written on the subject. This book isn’t perfect of course, but as the latter comment said, it’s a good reminder that debugging requires a framework as well. This book provides you with bold lines that are close but not strictly, a methodology for debugging.
A little bit about the author and his style. David J. Agans is an electronics engineer, so you’ll find his book full of hardware references. He claims that this book is for everyone (and it’s true), but if I had to present this book to someone, it would be a hardware or an embedded software engineer.
He was born in 1954, and the book’s first edition was written in 2002, so expect some outdated examples and tools. His style is easy, concise, short, and to-the-point, he doesn’t yap too much about the rules, he narrates use cases which he witnessed in his experience to explain each of the rules (he calls them war stories), the chapters are around 13-20 pages long with a summary page at the end of each chapter which has mnemonics from the war stories he tells (great for remembering the rules). He starts the book by listing all the rules, then goes on to explain each rule per chapter, then finishes off by combining all the rules into 1 war story, and gives you some exercises to practice detecting rule violations.
Rule #1: Understand the System
We all copy and paste code (hell, some of us can just let AI write their code altogether), but it’s really important to understand the code that you decide to add into your codebase, you should be aware of the differences between your system and the code you’re copying, you should make sure that this particular piece of code aligns with all the input and output that will interact with it. The author starts by stating the obvious (which might not be THAT obvious) and stressing the “Read the Manual” mantra. He explains that this might be the logical thing to do when facing some kind of error or bug, but he advises reading the manual, aka code documentation, before starting out.
He suggests tackling this in a specific order:
Read the documentation thoroughly
Know the fundamentals of that technical field (probably a given)
Understand the top-level view of your system
Know your debugging tools
Look it up
With these logical steps, you’ll start debugging from a strong place. First, you’ll read the documentation (or at least the part related to the change/bug), then you need to know any frameworks/libraries/tools in use to know what’s reasonable and what wouldn’t make sense. You should know what goes across all the APIs and communication interfaces in your system. After that, you need some kind of overview of your system. When there are parts of the system that are “black boxes,” meaning that you don’t know what’s inside them, knowing how they’re supposed to interact with other parts allows you to at least locate the problem as being inside the box or outside the box. Then you’ll need to know your debugging tools and when to use each of them. Do you need a memory profiler? Or will a thread dump be enough? How will you simulate the environment? A debugger can help expose a logic flaw, but you’ll need profilers for timing or multithread problems. At last, you’ll need to look it up; maybe a good old Google search or an AI agent will help, maybe the tech lead at the next desk will help.
It’s worth sitting with the “black box” point a little longer. In a microservices architecture, almost everything is a black box to someone. The auth service, the payment gateway, the message broker — you didn’t write them, and you can’t step through them. But if you know what they’re supposed to return given a particular input, you can test the boundaries. Is the failure happening before the call or after? Is the response malformed, or is the problem how you’re interpreting it? That question alone can save you hours of looking in the wrong place. And you’ll probably find the answer to that question in an API documentation or an architectural diagram of the system.
The debugging tools point is also underrated. Many engineers default to print statements for everything. Those work, but knowing when to reach for a thread dump, when to attach a remote debugger, when to pull a heap dump — that’s a skill that compounds over time. Agans frames knowing your tools as part of preparation, not as a reaction to a specific bug. That reframing matters.
Bottom line is: Collect as many details about the system and about your bug as you can before you start your debugging process.
Rule #2: Make It Fail
An important thing to do when starting your debugging journey is to reproduce the bug/issue. This chapter is dedicated to that, at first Agans explains the 3 reasons why we need to make it fail:
So you can look at it
So you can focus on the cause.
So you can tell if you’ve fixed it.
You need to first observe what’s happening, log everything, check your memory/time profiles, etc. Then you need to identify the cause exactly, what conditions caused the bug. Lastly, you’ll use those conditions and those logs again to verify that you actually fixed it.
An important note about simulations is that you need to carefully simulate the environment and conditions in which the failure occurred, but never simulate the failure mechanism itself. Recreate the system but within limits. It’s also useful to automate or amplify your testing, automation can make a problem happen when it’s intermittent, and amplification can make a subtle problem more obvious.
The amplification technique is one worth highlighting. If a bug only shows up under high concurrency, crank up the thread count. If it’s a memory issue, reduce the heap. If it’s timing-sensitive, add an artificial delay around the suspect code. You’re not changing what the system does — you’re making the symptom louder so it’s easier to hear.
The simulation point trips people up more than you’d expect. The temptation when you can’t reproduce a bug locally is to reconstruct what you think happened — to simulate the failure itself. Agans is firm: don’t. If you simulate the failure mechanism, you’re testing your hypothesis, not the system. You may convince yourself you’ve reproduced it when you’ve actually built a slightly different bug. Reproduce the environment, let the failure happen naturally.
An often cumbersome type of bugs are intermittent bugs. For those, you can break down the conditions around the failure and the factors that affect it. If you can control those conditions, you can control your intermittence better. So firstly, you need to list down those factors that affect your system (and specifically the part that’s failing), then you simply have to vary these conditions and wait for your system to fail at a particular setting.
If you’ve tried everything and it’s still intermittent, you can focus on the 3 reasons why we make it fail and achieve them another way. First, you want to look at it, so keep a debug log, capture and record everything, and have a log of a “good run” and a “bad run”. Second, you want to focus on the cause, you may start to see patterns in your actions that seem to be associated with the failure. Third, you want to know that you fixed it. If you use statistical testing, the more samples you run, the better off you are. But it’s far better to find a sequence of events that always goes with the failure—even if the sequence itself is intermittent, when it happens, you get 100 percent failure. Then, when you think you’ve fixed the problem, run tests until the sequence occurs; if the sequence occurs but the failure doesn’t, you’ve fixed the bug.
The last lesson from this chapter is:
“Never throw away a debugging tool.”
Sometimes a test tool can be reused in other debugging situations. So it will be useful to maintain and upgrade a test tool, keep it in your source control system, and don’t code it like a throwaway tool. Some companies even sell their debugging tools alongside (or instead of, yikes) their actual product. So keep your debugging tools around, they might prove handy in the future.
Rule #3: Quit Thinking and Look
When hunting for the root cause of bugs, guessing can be dangerous. If you have pre-biased thoughts about the issue, you might end up fixing something that isn’t the bug, worse yet, you might waste time and money on something that you simply imagined.
Instead, the author encourages us to stop thinking (i.e., guessing or analysing) and look (i.e., use the debugging tools to investigate further). “Looking” is often hard, it requires you to set up the tools, look into log files, or set up a complicated environment.
Many problems are easily misinterpreted. If you turn the light switch and the light isn’t on, it could be a bad light bulb or a broken bulb filament, or you could have just flipped the wrong switch. And each time you look into the system to see the failure, you learn more about what’s failing. This will help you decide where to look even deeper to get more detail.
The light switch analogy is deceptively simple, but it captures something important: symptoms and causes are not the same thing. A NullPointerException is a symptom. The missing null check is a cause. A 500 error is a symptom. The uncaught exception in three services upstream is a cause. Training yourself to stay at the symptom level a little longer — to really understand what you’re seeing — before jumping to causes is what separates systematic debugging from random poking.
How do you know when to stop searching and start thinking of a solution? The author answers this simply: “Keep looking until the failure you can see has a limited number of possible causes to examine.”
It’s crucial when talking about debugging to talk about the importance of instrumentation. You have to put instrumentation into or onto the system. Into the system is best; during design, build in tools that will help you see what’s going on inside.
Building on Agans’ point, in practice, this means adding logging (properly), making sure you capture what’s important, and removing redundant information. Adding metrics, values like request count, memory usage, active users, and latency percentiles. Introducing distributed tracing, it’s important to track a request across multiple services and properly visualise the traces. Profiling instrumentation is another important aspect to consider, measuring resources of your system to find CPU hotspots, memory leaks, and GC pressure. Security instrumentation is also crucial, you have to monitor security events like SQL injection attempts and authentication failures.
The reason Agans stresses building instrumentation during design rather than adding it after is that retrofitting observability into a system is painful. You’re working around existing code paths, you’re worried about side effects, and you’re often doing it under pressure because something is already broken. If your system has good logging, metrics, and tracing from the start, you arrive at the debugging session with data already in hand. The failure often explains itself.
Ever heard of a Heisenberg bug? It’s a type of bug that changes behaviour or disappears when you try to observe it or debug it. Named after the Heisenberg uncertainty principle, which states that certain pairs of physical properties can’t both be known with certainty at the same time. This is a testament that any instrumentation can affect the system to varying degrees. Since this is unavoidable, make sure to “Make It Fail” because minor changes while debugging can hide the bug completely.
Heisenbugs are particularly nasty in multithreaded systems. A race condition that only manifests under real load disappears the moment you add synchronization for the sake of debugging. A deadlock that happens in production vanishes when you attach a debugger because the timing changes. The lesson isn’t to avoid instrumentation — it’s to keep it as lightweight as possible and to rely on it in combination with Rule #2: if your instrumentation changes the behaviour, you need a way to reproduce the failure without it.
Lastly, the author recommends guessing (or “thinking”) only to focus the search. There are 2 cases in which you can allow your mind to wander in imagination of what might have caused the error. Case #1: In a more likely cause of and an easy fix to a problem, you can try that at first, just to eliminate a trivial case. Case #2: is to narrow down your search, make sure you dig in deeper before you devise a fix. If you understand your system well, your guesses will get you closer and closer to the actual cause of the bug.
That was a quick summary (and an introduction if you want to read it) for the first 5 chapters of this book. If you’d like to know a bit more details and read the author’s original words, you can find a more detailed quoted version I curated on my Notion here.
I hope you enjoyed this as much as I did writing it. Next post will cover Rules #4, #5, and #6. If you’ve read the book, I’d love to hear which rule you found most counterintuitive, and of course, I’m open to discussions and criticism =)
