I run two RTX 3090s in the house, one local inference lane answering on :8080 and a second on :8081, with an agent that routes between them so a bad night costs me zero API calls. That setup died on the morning of 2026-10-20 and every status layer above it stayed green. The first signal was a greeting I sent at 07:10, answered by nothing.
systemctl said active. The health endpoint said loaded=true. The router answered with the model identity it was configured to serve. All three statements were about the request that carried them, none was about whether any work happened. A stack that survives on those three checks is a stack that can be asleep and still look alive, and that's why an autonomous rig needs a guard that never asks the thing it's supposed to protect.
What actually happened
The journal had it. The primary lane stopped and got masked at 00:44, the second at 02:23, and a rendering job logged the deliberate GPU reclamation plus the assumption that my agent would quietly fall back to a cloud model. My config had both fallback entries pointed at those same two lanes, so there was no route left to save the request. The greeting at 07:15 landed on a dead chain, the first reply completed at 07:19:47 on the lane I'd just repaired.
A fallback is a pointer to a lane, and the lane is the thing that went down. Fallback config that points at the failure is decoration, it gets exercised by the outage it can't survive.
The guard
The fix that held was one Python file plus a systemd service and a timer, checking the backend health endpoint and the router's model identity every 60 seconds, two failed observations before it starts a repair, a 240 second cooldown so a cold model load can't loop. It restores from saved copies of the production units. No model, no gateway, no cloud call anywhere in the loop, it runs on the box with the electrons that were already there.
I tested it against the exact failure class. I stopped the primary services and replaced the unit with a persistent /dev/null mask, then watched. The watchdog restored the saved unit, started the services, both lanes printed RECOVERED, and a real request aimed at the deliberately dead primary came back through the fallback lane at :8081, which proved the text chain had no cloud route to leak into. The production units finished with NRestarts=0. A repaired lane that's quietly crash-looping wears the same green badge, so I read that counter after every repair.
That test passed, so the guard was worth keeping. It was also incomplete, and the gap cost me the rest of that day.
The failure the guard didn't have
At 11:07 three systemd drop-in files appeared, one per unit, each of them replacing ExecStart with /usr/bin/sleep 86400. A unit running sleep is a unit that starts, holds its state as active forever, and does no work at all. systemctl stayed green, no socket bound, the health endpoint went 404.
The watchdog was right about the endpoint and wrong about what to touch. It restored the masked base unit, and the base unit still had the drop-in sitting on top of it, so every restart brought back the sleep. Same failure, every tick, and the timer kept reporting healthy lanes once the counts reset. From the 11:07 symptom to the repair starting at 12:51 is 104 minutes of a rig I thought was running.
The repair archived all three sleep drop-ins, restored the ExecStart values that run the engine and the router, and widened the watchdog so it archives only the exact known sleep-86400 and Restart=no override contents, which keeps genuine tuning like a RestartSec line intact. I ran that against an isolated test drop-in directory before pointing it at the live units, one override got archived, the tuning survived untouched, both failure counts went back to zero, both lanes printed STRATA_WORKING, and the request I'd sent at 12:46 came back with three completions at 12:52:27, 12:52:49 and 12:53:01 plus the tool calls around them.
The rules that came out of it
A shutdown path needs a gate it reads before it acts. Two separate jobs on that box were reclaiming GPUs on schedules nobody checked against each other, so I put an explicit maintenance marker at the entry of each one, absent marker means return without running a stop command, and I regression-checked the absent-marker branch with a direct call, because a review never runs the branch it reads. The watchdog gets the same marker, a file that says suspend recovery while a maintenance window is open.
Guards get audited for what they can see. Mine could see masks, it couldn't see overrides, and the difference between those two states is one listing of a drop-in directory. Every guard you write needs a second pass over its own blindness, the first pass only proves the guard reacts to the failure you already knew about.
Boundaries stay in the record. A 200 from a health endpoint proves the endpoint answered, it doesn't prove a long inference finishes, and a watchdog can't repair a box that's powered off or a host it isn't running on. Later that evening I moved both lanes to 524288 tokens with yarn rope scaling and restarted only the router that owns the second engine once its slots were idle, both slots reported n_ctx=524288 and a tool-call probe echoed tokens=524288, which verifies the limits loaded and says nothing about a 524K prompt surviving.
What I'd want in my own stack
A health check that asks for a probe with a real answer in it, model identity plus one completion, before a lane counts as up. A recovery loop that reads overrides and drop-ins on top of the base unit. A shutdown that requires an explicit marker. And a log line for each repair with the state that was archived, so the second failure in a day gets found by comparison, and a day full of silent repairs reads back as a list of states I can diff.
So, what does your stack say while it's doing nothing? The question worth answering is which layer in your chain would notice first, and how long you'd wait before a human did. Drop the layer you'd most like to catch your rig asleep and I'll tell you what I'd check before I believed it.
