Sometimes the Best Way to Find a Bug Is to Move It
A few weeks ago, the check engine light came on in my 2007 SUV. I plugged in an OBD scanner and got P003F: “Intake Camshaft Profile Control Stuck On, Bank 2.” Sounded menacing enough…
Then there’s me. I know enough about cars to be adventurous. With my dad’s sage advice rattling around in my head, I can change parts, follow instructions and generally understand what some systems are trying to do, but I’m not a mechanic. So I did what most of us do when a warning light appears… went straight to the mechanic. Nah. I pulled out my phone and looked it up. (There’s another whole post on where we are with that process.)
Back on the case: the code pointed toward the cam profile system, but that didn’t mean there was one obvious part to replace. According to the sources I was finding, it could be a solenoid, wiring, a connector, oil flow or something farther into the mechanical side of the engine. The more I read, the more familiar the whole thing started to feel. I had an error message and a collection of possible causes, but the error wasn’t actually telling me which one was responsible.
Basically, I had a bug.
What finally made the diagnosis interesting was discovering that the car has two identical solenoids controlling the same system across two groups of cylinders. That meant I had something much more useful than a list of possible problems. I had a way to test one.
I did what any problem solver would do and swapped the solenoids, cleared the code and drove the car. Sure enough, the check engine light came back, which normally isn’t something to celebrate. This time it was! The code had changed from Bank 2 to Bank 1.
The problem moved with the solenoid.
At that point I wasn’t replacing a part because a forum post said it was probably bad. I had taken a component from one side of the system, moved it to the other side and watched the fault follow it. The wiring didn’t move. The connector didn’t move. The oil passage didn’t move. The solenoid did… and so did the error. I ordered a new one. At that point, the hard part was over.
Somewhere during all of this it occurred to me that I spend a surprising amount of my working life doing essentially the same thing, just without an engine cover in the way.
Finding where it went wrong
When something breaks in a website or application, the visible error tends to get most of the attention. Maybe a component doesn’t render, an API request fails, a Lambda function throws an error or the browser reports that something is undefined. That’s where you first see the problem, so naturally that’s where you start looking.
But that’s not necessarily where anything went wrong.
A front-end problem can begin several steps before the browser finally complains about it. The API may have returned something I wasn’t expecting. The API may be fine, but I transformed the response incorrectly. The error is often just the point where the system finally admits it has a problem.
That’s where the car analogy started to stick with me. P003F told me Bank 2 had a problem. It didn’t tell me that Bank 2 was the problem. Once I moved the suspect component somewhere else, the distinction became obvious.
We can do the same thing with code, and in many ways we have it easier because software lets us move things around without getting our hands dirty.
Say I click a button and the number that appears on screen is wrong. There are a few places that could have happened: maybe I read the wrong value from the form, maybe the calculation is wrong, or maybe the function that puts the result on the page isn’t doing what I think it is. Rather than staring at all of it, I can give that last function a value I already know is correct. If it displays $125.00 when I hand it 125, then I haven’t fixed the bug… but I’ve probably eliminated the display code as the source of it.
So I move backward. What value came out of the calculation? If that’s wrong, what values went into it? I might hard-code one of those values, bypass a function for a moment or drop a console.log() between two steps to see exactly where the expected number changes. Each little test changes one thing and asks the system the same question the car test did: does the problem stay here… or does it move?
That is a much different mindset than changing code until something starts working again.
console.log() as a diagnostic tool
This is probably why I’ve never quite accepted the idea that console.log() is somehow an unsophisticated way to debug. Used badly, sure. A dozen logs that say HERE, HERE 2, and WHY IS THIS NOT WORKING aren’t exactly observability.
I’ve certainly written them anyway.
Used deliberately, though, logging is really just a way of putting gauges, or breakpoints, along the route. When something goes wrong somewhere in a chain of JavaScript functions, I want to know what the data looked like along the way. I don’t necessarily need to read every line between point A and point B. I need to find the last place where everything was still what I expected it to be.
So I might put a log before a function and another after it. If the value is right going in and wrong coming out… well, we’ve narrowed things down considerably. If there are eight meaningful stages between the user’s click and the bad result on screen, I might start somewhere around stage four rather than stage one.
If everything looks right there, half of the system becomes considerably less interesting. If it’s already wrong, the other half does. Then I split the remaining half again. It’s essentially binary search applied to debugging, although I’d never really thought about it in those terms while doing it. Half the code on, half off. Bypass this section. Feed that section known-good data. Find the point where good becomes bad.
Eventually there isn’t much room left for the bug to hide. Squeeze that little bugger…
The important part is resisting the urge to change too much along the way. We’ve all had the debugging session where you modify three functions, update a package, clear a cache, restart something and suddenly everything works. Great… except now you don’t actually know why.
The car test worked precisely because I didn’t do that. I changed one thing. I moved the solenoid. When the fault moved with it, the car had answered the question for me.
No lack of tooling
Of course, software gives us another advantage. We have an absurd amount of tooling designed to catch problems before we ever get to this stage. Linters, types, tests and IDE warnings are all standing along the road waving little flags before the check engine light even comes on. We’re a long way from hunting through 1,698 lines of JavaScript for the missing semicolon.
console.log() isn’t the Holy Grail here. It’s just one particularly simple diagnostic tool, and one that happens to make the parallel with the car especially obvious.
Taking things apart
Somewhere in the middle of all this, I realized I’d probably been primed to think about the problem this way because I’ve spent an unreasonable amount of time watching Mat Armstrong rebuild damaged cars on YouTube. He buys cars that most sensible people would look at and immediately decide should remain someone else’s problem, then starts taking them apart.
A McLaren or Ferrari can look catastrophically damaged when it arrives, but the interesting part is figuring out what is actually damaged. Some things obviously need replacing. Others look terrible but are fine. Something that appears fine turns out not to be. One repair reveals the next problem, and eventually what began as a wreck becomes a collection of smaller, understandable problems.
I find it captivating to watch them chase those problems down, probably because it scratches the same problem-solving itch that development does.
I rarely understand the whole problem at the beginning. I have a symptom and a rough mental model of the system, then I start taking it apart, figuratively at least, until I understand enough of it to make the next decision.
That might be the real parallel here. I didn’t understand the entire cam profile system when that check engine light came on, and I didn’t need to. I just needed to understand enough to figure out what to test next.
The same is true when I’m staring at a piece of code that isn’t doing what I expected. I don’t necessarily need to understand the entire system before I can start narrowing things down. I need a hypothesis, a way to test it and enough patience not to change three other things while I’m waiting for the answer.
Sometimes that means better tooling. Sometimes it means a breakpoint. Sometimes it means turning half the code off.
And sometimes it’s just console.log().
Either way, the goal is the same… give the problem fewer and fewer places to hide.
