The dashboard was green and the product was down.
Not all the way down. Worse: down in the two places I wasn't looking, while every check I had wrote green to the wall. This is the postmortem of the day I pointed an agent swarm at my own SaaS and it came back with 38 findings, 32 of which survived human verification, including two live outages nobody had noticed and a security hole sitting on every scheduled job in production.
Some context so the rest makes sense. I run a small social-posting SaaS solo, with an AI-agent workforce doing most of the labor. The engine learns a customer's voice once, watches their industry overnight, drafts platform-aware posts, a human approves in a tap, and it publishes to Facebook and Instagram today, with more platforms rolling out. Customer #1 is a high-end audio dealer whose feed the engine writes every night. Real money, real posts, real permalinks.
Which is exactly why I audited it.
Why audit a product that looks fine
When you build with agents at speed, drift accumulates in the gaps between what you shipped and what you believe you shipped. My monitoring measured the things I had thought to measure, which is a very different set from the things that break. So I ran a sweep: an agent swarm over the codebase and the production state, eight review lenses, each lens a separate agent with its own brief and no knowledge of the others' conclusions. Security. Correctness. Tenancy. The publish path. Cron hygiene. And so on.
The swarm raised 38 findings. I verified every one by hand against production before believing any of them. 32 confirmed, 6 killed as noise. Hold onto that ratio, because it's the honest part: an audit that confirms everything it raises is a flattery machine, and an unverified finding is a hypothesis, not a bug.
Here are the three that mattered.
Outage one: the 3.26-second OAuth race
The finding read like a support ticket from the future: no customer could connect a new social channel. Every attempt died in a way that looked exactly like a popup blocker. The OAuth window opens, the customer authorizes on the platform's consent screen, the window closes, and then nothing. No new channel in the list. No error. Try again, same thing.
The mechanism was ugly in the way only race conditions are ugly. The connect flow mints a one-time claim so the OAuth callback knows which customer initiated the dance. A background job was consuming that claim 3.26 seconds after creation, while the customer was still reading the consent screen. By the time the callback arrived, the claim was gone, and the bind failed silently. No exception, no error-level log line, no failed job anywhere. Just a window that closed and a channel list that didn't change.
No human authorizes an OAuth grant in under 3.26 seconds. Which means the failure rate was 100 percent, deterministically, and the whole thing was invisible because every individual component reported success.
Think about the cost of misdiagnosing this one. It presents as a client-side problem. Popup blocker, third-party cookies, wrong browser. You could burn a week of support cycles telling customers to try Chrome while the actual bug sits in a background job that is doing precisely what it was told to do, at precisely the wrong moment.
The fix: the claim is now consumed exactly once, at the callback, atomically. And there is a counter where there used to be faith: connects started versus channels bound. When those two numbers diverge, I get an email before a customer notices anything.
Outage two: the silent draft famine
The second outage was quieter, which I did not think was possible.
The nightly research stage reads a curated set of industry publications so the composer has real, current material to draft from. The swarm found that those fetches were being blocked at the network level from our cloud egress. Not erroring loudly. Blocked. So the research stage returned nothing, the composer had nothing worth drafting, and drafts stopped landing for a billable account.
And here is the part that earned this finding its own section: the scheduler kept reporting "cadence satisfied" the entire time. The cadence check was counting the wrong thing. It measured whether the pipeline ran, not whether posts landed. The pipeline ran beautifully. It ran on schedule, completed every stage, and produced zero output, and every status surface stayed green while landed sat at zero.
That is my working definition of an invisible outage now: every component succeeds and the system produces nothing.
The fix had two parts. The mechanical part was rerouting the feed fetches so they actually resolve from our infrastructure. The important part was changing what the check believes. The only number that means the pipeline works is landed: a post that a human approved and a platform confirmed live. Not drafted, not scheduled, not accepted-by-the-API. Landed. The cadence check now alarms when landed is zero for any paying account, because a green dashboard over an empty feed is not a monitoring system, it's a lullaby.
The security hole: 19 cron routes and one spoofable header
The swarm's security lens found the one that made me put the coffee down.
The product runs on scheduled jobs. Research, compose, publish, reconcile, engagement, the lot: 19 cron routes in total. Every one of them authenticated the caller by checking a user-agent string, the one the hosting platform's cron runner sends, something to the effect of "vercel-cron". A user-agent is not authentication. A user-agent is a suggestion. Anyone on the internet can send that header from curl in one line.
There was supposed to be a real check behind it, a shared secret the routes would verify. The environment variable for it, "CRON_SECRET", was unset in production. So the code path that did the actual verifying never engaged, and the spoofable header was the entire perimeter.
Practical exposure: anyone who guessed or read the route names could have triggered publishes on customer accounts, burned my LLM budget, or fired arbitrary pipeline stages at will. The logs show nobody did. That is luck, not security, and the whole point of the fix is never needing to check the logs with my heart rate up.
The fix took an afternoon: set the secret, verify it on all 19 routes, reject the header-only path entirely, and add a test that asserts no cron route ever answers 200 without it. The lesson took longer to absorb: the swarm found this because one lens existed solely to ask "who can call this, really?" about every route. I had asked that question when I wrote the first cron route, months earlier, and never asked it again as the other 18 accumulated.
What the swarm got wrong
Six findings died in verification, and the shapes are instructive. Duplicate-looking code that was intentional. A "missing index" on a table with a few hundred rows where a scan is fine for years. Pattern-matched bug shapes that dissolved on contact with production reality.
So the operating rule I run now: treat swarm output like a static analyzer with opinions. It is tireless, it reads everything, it holds eight perspectives at once, and it does not know your system's history. The human is the confirmation gate, and 38 raised to 32 confirmed is a ratio I trust. If it had been 38 for 38, I would have suspected the swarm of telling me what I wanted to hear.
Three lessons I paid for
- Alarm on absence. Both outages were absence failures: the absence of a channel bind, the absence of a landed post. Monitoring that only fires on errors misses the failure mode where everything succeeds and nothing happens, and in my experience that mode is more common and far more expensive than the loud crash. Count the thing that should exist, and alarm when the count is zero.- Verify before assert. A 201 from a publish API is a receipt for the request, not evidence of the post. The engine now reconciles against the platform's actual state before believing its own database, and the same discipline applies one level up: a green status page is an assertion your checks are making, and assertions want verification.- Landed greater than zero is the only metric. Every intermediate number lies eventually. Drafts generated, jobs completed, requests accepted: all of them stayed healthy through a total output failure. The metric that survives is the one measured at the far end of the pipe, in someone else's system, where your bugs can't reach it.## The cost, and the verdict The sweep ran in a day. The fixes landed over the following days as a stack of small pull requests, each scoped to one confirmed finding. Cheap, against the alternative: the OAuth race alone would have gated every new customer at the exact moment I tried to onboard them, live, with the failure cosplaying as their browser's fault. I build this product with agents, so there is a certain symmetry in agents being the ones to catch what I missed. But the symmetry is not the point. The point is that fresh eyes at scale are now cheap, and the only reason not to point them at your own production system is that you would rather not know. The dashboard is still green. The difference is that now I have reasons to believe it.
