Originally published at aideazz.xyz — cross-posted here with canonical link.
A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. June 23, 2026.
Two silent failures in one LangGraph pipeline — a stripped state key and a pause that never resumed.
What it looked like from outside
VibeJobHunter's LangGraph pipeline reported clean runs and produced nothing usable. Right-fit postings were scored and routed and then never reached the Telegram card or the HubSpot deal. Separately, postings were failing the iron-clad fit gate for a reason that did not match the posting being read. Neither fault raised an exception, logged a warning, or changed an exit code. The pipeline looked healthy in every place a person would think to look.
What was actually happening
Two independent faults in the same state machine, both presenting as absence rather than as error. First, LangGraph strips any key not declared in the pipeline's TypedDict state, and the location field the gate depends on was being passed but never declared — so it was dropped between nodes and every posting was judged against a value that had silently become empty. Second, the human-approval interrupt sat before the submit node, which was correct while the bot auto-applied and an irreversible send needed a human to authorise it. The bot had since changed to LEAD mode, where the submit node no longer applies to anything — it surfaces the job for Elena to apply herself. The same interrupt now paused every qualifying posting immediately before the only step that would have surfaced it, and no thread ever resumed, because in that mode nothing was waiting to approve.
The fix
Declare every key the graph carries in the state schema, and comment the ones whose absence is invisible at runtime so the field cannot be removed by someone reading the file cold. Make the interrupt conditional on the mode that needs it — interrupt_before is applied only when AUTO_APPLY_ENABLED is true, so the pause exists only on the path where a human decision blocks an irreversible action, and LEAD mode runs straight through submit to notify.
How I know it worked
Both faults were found and fixed on 23 June 2026 and both fixes are in production today. Commit 20e5710, "declare location in JobState TypedDict — LangGraph stripped it, breaking iron-clad", and commit 4806a7e, "disable submit_node interrupt in LEAD mode — THE reason jobs never surfaced". The pipeline that carries them was added on 26 April 2026 and still runs on langgraph 1.0.6 with langgraph-checkpoint-sqlite 3.0.3 — seven nodes, gate to score to route, branching to submit, outreach or discard, all converging on notify, with an AsyncSqliteSaver checkpointer and one thread per posting keyed on the job id.
The rule this earned
In a stateful graph, failure is silence. A key you did not declare is dropped and a thread you paused is not finished, and neither one raises. Treat the state schema as an interface contract, and treat every interrupt as something that must be proven to resume in every mode the system can run in — a guard that is correct in one mode becomes a trap in the mode where the step it guards no longer does the dangerous thing.
The named concepts behind it
Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.
Silent failure
The system did something reasonable, and told nobody.
The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.
A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.
The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:
- Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
- Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.
This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.
No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.
