The correlation notice fired. Three sites, same anomaly type, inside the time window. The orchestrator caught it and logged it, live, against the deployed service. Clean, first try.
Then I asked myself a question I almost didn't bother asking, because the thing had just worked: why did it work?
The answer wasn't "because the logic is correct." It was "because Cloud Run happened to route both requests to the same running instance." Well, shit.
My orchestrator was holding its list of recent risk events in a plain Python list, in process memory. Worked in local testing because there's only one process. Worked live because Cloud Run, under light traffic, often reuses the same instance instead of spinning up a second one. Neither one's a guarantee. The moment traffic patterns shifted and two readings landed on two different instances, the second instance wouldn't have a clue the first one existed. A correlation that should fire would just silently not.
A bug that passes its own demo is the hardest kind to catch. There's no error to chase. There's just a checkmark.
What I was building
VES Fleet is a network of independent site-agents (Bori, Choba, Etche, three real survey sites in the Niger Delta). Each one reads an underground electrical survey, send current into the ground, measure how it flows back, a real physical signal of what's down there, and calibrates its own contamination-risk threshold from its own site's real history. Not a number copied from anywhere else. An orchestrator watches for the same risk signature showing up at more than one site inside a time window.
It's my submission to the Fortified Enterprise Fleet track of Google's All Things Agentic Hackathon. Architectural discipline is 30% of the score there. Proving it actually runs on Google Cloud is a separate 30%. So a bug that only looked fixed was never going to survive someone actually reading the state-management story.
Checking the thing that already worked
Once I understood the actual failure mode, I started checking every other early success the same way, not just the correlation notice.
The site list itself came from the original spec: Bori, Ogbogoro, Onitsha. I nearly built straight from it. A quick check against my own prior project's data-provenance notes turned up that Ogbogoro's source paper has been unreachable for months, dead DNS, not a typo, and Onitsha only ever had a rough estimated range from a search summary, used to fake a synthetic sample, never a real survey. Two of the three original sites had no real underground data to calibrate against. For a project whose entire pitch is "calibrated to each site's real observed normal," that would have been a demo that looked identical and meant nothing. I swapped to Bori, Choba, Etche. All three fully real, surveyed, independently published.
The correlation bug got the real fix, not a patch. Recent-event state moved out of process memory into Firestore, the same store already holding each site's isolated survey history, so I wasn't introducing a second, separate way of sharing state for this one problem. Every instance now reads and writes the same shared record instead of trusting that traffic happened to land in one place. I verified it the only way that proves anything: fresh station IDs, submitted live, twice, on two separate runs, watching a genuinely new correlation notice appear both times against the shared store, not against whatever instance happened to still be warm.
The actual throughline
"It worked" and "it's correct" are not the same claim. A live demo will happily let you conflate them. It proves your happy path executed once. It doesn't prove the mechanism holds under a condition you didn't happen to hit. The fix is boring: stop after each success and ask what specifically made it succeed, before deciding it's done.
Where it landed
VES Fleet is deployed on Cloud Run, backed by Firestore and Pub/Sub. None of those three were the default choice. A site-agent publishes an event when it escalates instead of the orchestrator polling Firestore for new rows on a timer, because "the fleet watches for patterns across sites" should be a real subscriber reacting to a real message. Not a cron job that's fast enough most of the time.
Gemini, via Google's Agent Development Kit, does exactly one job: drafting a human-readable summary of a decision the deterministic code already made. It never re-derives the risk numbers themselves. A number that decides whether a site gets escalated shouldn't be re-derived by a model on every call. Same reasoning that made the correlation bug worth fixing properly instead of shipping it.
Worth being precise here instead of rounding up: three of those four things are load-bearing in a way I can point to directly. Pull Firestore or Pub/Sub or Cloud Run out and something upstream breaks, the bug above is the actual proof. Gemini's real too, it does real work every time a case escalates, but it's not structurally necessary the same way. Cut it and every flag, gate, and correlation still fires exactly the same. A human just writes the summary by hand instead of getting one drafted.
Fifteen tests pass, including the correlation suite, now running against the shared Firestore store instead of the in-process list that got lucky. The correlation notice fires because the state is actually shared, not because of how Cloud Run felt like routing traffic that day.
I don't know what else in this build passed for a reason I haven't checked yet. That's not a comfortable note to end on. It's the honest one.
I wrote this piece for the purposes of entering Google's All Things Agentic Hackathon.
