This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
🦄 I debated writing this for a long time, but I finally talked myself into really writing again after a hiatus, and there's no better way than story time. So here's one of the most challenging bugs—or really, the series of them—I've run into in the enterprise world. Grab some popcorn and Skittles, because this one takes a while.
Better yet, cue up Jerry's actual theme song—Jerry Was a Race Car Driver by Primus, because of course it is—and let the best bass player on the planet score the whole mess while you read.
And yes, it's the Summer Bug Smash and my entire cast is dressed for Christmas. Stay with me.
Meet Jerry 🪦
If you work with software any length of time, you already know the particular nightmares that come with legacy applications. This one is no different. It started life as a rewrite of some antiquated, bash-flavored system back when Java 8 was the coolest kid at the table. Let's call him Jerry.
Jerry is a well-rounded app—or he was, before he let himself go. He came up on a then-modern Java stack and served exactly one purpose: get data from upstream into the database, correctly and on time.
He was good at his one job. Then his one job got split into parts, and the sum of those parts did not add up to a whole—Jerry just expanded along the midline with no particular purpose or direction in life.
You can imagine how it goes: a few retirements, a couple of half-finished rewrites, several well-meaning somebodies who swore they'd whip him into shape and left him half-done every time. Take your eyes off him at Christmas and he's the weird uncle who shouldn't have been left alone with the punch. That's about when Jerry and I met, more than three years ago.
The Infestation Begins 🪰
Jerry did his best to keep up with everything we kept piling on him, but communication was never his strong suit—a patch here, an upgrade there, enough to keep the lights on and the punch bowl full. Then performance testing showed up to the party, started creating records at a pace Jerry had never been asked to imagine, and he began falling over in ways that didn't look related to each other at all.
The first one looked easy, the way the easy ones always do. Jerry reached DB2 through a shared JDBC library with no effective connection timeout, HikariCP handed him a connection pool you could count on your fingers and toes, and the new workload was shoving tens of thousands of queries through it. The pool ran dry, everything backed up behind it, and Jerry offered us DB2 connection pool not available as if that explained a damn thing.
So we did what you do when the pool swears it's empty. We tuned the database, grew the pool, adjusted the connection-acquisition timeouts, and fixed the nasty little mismatch where a request or JVM-level operation could time out up top while the JDBC call underneath it kept merrily working—leaving zombie threads clutching the database resources everyone else was standing in line for. None of that was wrong. Production still runs faster today because of it. It just wasn't the fix.
Here's how you know you're in real trouble: swap in a different set of test data, layer on whatever secondary fix we'd just shipped, and the whole thing would look solved—cured, even. Then it would wander back a few weeks later like nothing had happened, we'd dig in, find nothing but more timeouts and deadlocks, tune the symptoms until Jerry decided to behave, and call it a night. Testing was blocked for weeks at a stretch. My team, the neighboring teams, our principal engineers, a few DBAs from the DB2 team, the on-prem crew—everybody got pulled into the group therapy debugging sessions, and everybody had a reasonable theory, because Jerry had thoughtfully supplied enough separate problems for every theory to be right and not one of them to be the answer.
A Hundred Million Lines Later 🪵
So we did the only thing left when all the smart people run out of theories: we turned on the logging. All of it. HikariCP leak detection, JDBC trace logging, Hikari thread logging—every connection, every timeout, every thread Jerry so much as thought about abandoning, piped straight out for debugging.
Then I forgot to turn the trace logs back off.
Overnight, Jerry wrote somewhere in the neighborhood of a hundred million lines, and I woke up to a message addressed to every org owner on the platform, asking—in the polite, political version of the question—what in the world was going on over here? I read between the lines. The lines were not subtle. Whoops. My bad...
In my defense, the crime came with a reward attached. Buried in that absurd trace mountain was something useful: Copilot followed a deep stack trace down to a line close enough to the real problem that we finally knew where to start digging. I couldn't tell you today exactly what it flagged—I've slept since then—but it pointed, and for the first time in weeks the pointing was in the right direction.
The light came on during yet another round of group debugging, months after that first exception waved its little red flag. When the pieces clicked into place I said some words I won't reproduce here, took an immediate walk, and decided that if Jerry had been a real person, I'd have shoved him. Hard.
The Finite Set That Never Ended ♾️
Instead of the real business terms, let's call each allocation boundary a logical scope. Jerry leaned on a legacy database table as a kind of identifier lock shared across several systems: one row per scope, and a specific, finite set of identifiers each scope was allowed to hand out.
The idea was reasonable enough. Jerry would grab a candidate number, run some validations, and lock the row for that scope so nothing else could snatch the same number out from under him. But a number could get used in the gap between Jerry picking it and Jerry acquiring the lock, so he ran one last check before he'd call it safe.
If the number was taken, he bumped it by one and checked the next in line, still holding the row lock. Taken again? Next one. And again, and again, all the way to the end of the allowed set—at which point he looped back to the beginning and started the whole march over.
Indefinitely.
I'm sure there was a good reason for this design once. Though neither Jerry nor I could tell you what it was. The real thing carries more baggage—because Jerry—but the part that mattered looked something like this:
int candidate = selectCandidate(scope);
validateRequest();
lockAllocatorRow(scope);
while (identifierExists(scope, candidate)) {
candidate = nextIdentifier(candidate, allowedRange);
}
return candidate;
And yes, the infinite loop is intentional. The assumption baked into it is that you should never run out of numbers, and if you have run out, the bug is in your data—because rewriting the loop won't conjure a free number, it'll just change how loudly Jerry complains about not having one.
Rewriting the loop wouldn't have made an identifier available. It would have stopped Jerry from taking the whole connection pool hostage while he went looking, though.
The real root cause? Performance testing had created tens of thousands of records in an environment where the old ones were never cleaned up, until it had quietly occupied every last identifier in at least one of those finite sets. So Jerry checked every number, hit the end, wrapped to the top, and kept going—one query after another, holding the row lock and the JDBC connection he needed to run the search the entire time.
Meanwhile the callers stacked above JDBC would time out—but the JDBC work underneath kept right on running, because we hadn't fixed those timeout boundaries yet. Those zombie threads ate the resources while fresh requests waited on HikariCP, and every connection that did return was immediately claimed by the next request. The few calls already inside the allocator kept searching exhausted scopes for identifiers that did not exist. They were playing leapfrog over the last few live connections, all of them looking for an open identifier to plug into a slot that wasn't there anymore.
And he'd have kept it up until somebody cleaned the data or killed the app.
The Monster Was Housekeeping 🪤
Here's the part that still makes me laugh. The loop wasn't wrong, exactly. It was written to assume the data would always leave at least one identifier free—and in production, it always did. Production never ran its ranges dry, so nobody ever had to picture what Jerry would do if one of them hit zero.
Our test environment, on the other hand, never got cleaned up. Not late, not now and then—never. Ordinary testing trickled in data slowly enough that the missing cleanup could hide behind everything else, and then performance testing showed up, cranked the record count to a pace nobody had planned for, and filled the last open slot Jerry had left to give.
The pool exhaustion, the zombie work, the timeouts, the deadlocks—all real. The pool changes, the timeout fixes, the database tuning—all genuinely useful. Every one of them changed the symptoms and bought Jerry a little more time, which is a big part of why it took us so long to spot the data sitting underneath the whole mess.
Once we finally found the loop, we added more logging—the right logging this time—around the allocator, cleaned out the stale data, and ran the same testing again.
It was magical.
Everything worked. The deadlocks and timeouts vanished. HikariCP came back healthier than it had been before any of this started, thanks to all the tuning we'd done chasing ghosts; performance testing passed, and every other kind of testing that had been stuck behind Jerry passed right along with it. Production was safe because it had never exhausted its ranges, and it still pocketed the benefit of all the tuning. The test environment was safe because I'd finally spent a weekend playing janitor and cleaning it out by hand.
So yes, one of the most expensive bugs I have ever chased turned out to be a simple chore nobody remembered assigning.
Four Failures in a Trench Coat 🧥
Strip away Jerry's personality and it was four fairly ordinary problems standing on each other's shoulders:
- the test environment had no data cleanup, even though the allocator counted on identifiers eventually freeing back up
- the allocator had no exhausted state, because "every number is gone" was filed under can't happen
- the timeout killed the caller but not the JDBC work underneath it, so zombie threads kept holding the resources
- the logs told us all about the pool, the timeouts, and the deadlocks without ever once mentioning that the identifier range was full
None of the pool sizing, database tuning, or timeout work was wasted—it fixed real problems, made production faster, and stopped abandoned work from loitering under callers that had already walked away. It just couldn't manufacture an identifier out of thin air.
The backlog carries two items now: automate cleanup in the test environment, and bound the search to a single trip through the allowed range, so the next person to hit this gets a clean identifier range exhausted instead of weeks spent improving everything around the actual bug.
Jerry's Still at the Punch Bowl 🪅
I'd love to tell you we fixed Jerry. But we didn't.
We cleaned the data, kept every performance and timeout improvement we'd made along the way, and left an inline warning on the loop that says, more or less: this is intentionally infinite because you should never run out of numbers—and if you did run out, go fix the data, because changing the loop won't make a number appear. There's a matching pile of notes in Confluence too, because nothing says permanently solved like a page future me has to remember to go search for.
If I could make that comment flash, I would.
Automated cleanup and real exhaustion detection are still sitting in the backlog, so for now Jerry's still at the punch bowl and I'm still wandering by with a mop every so often. He's a few years out from his own retirement party—though I've started hanging the banners early, which means I secretly updated the app banner in Confluence—and I'll keep him upright until it's actually time to say goodbye.
Here's the thing about the loudest problem in the room: it's almost never the one actually biting you. Jerry screamed pool exhausted and deadlock and timeout for weeks, and every one of those was true, and not one of them was the actual root cause. The bug was a finite set quietly counting down to zero in the one environment nobody had ever thought to clean, riding an assumption that held in exactly one place—right up until performance testing walked in and drank the punch bowl dry.
🛡️ Fewer Lines Than That One Log File
This post was written by me, with ChatGPT and Claude rubber-ducking in the corner—catching my tangents and mercifully producing slightly fewer lines than the trace logs did. Jerry, the hundred million lines, and the choice words are all mine.
🪦 Approved by Jerry, who does not know he did anything wrong.
I still blame the punch.
