The symptom
We run a small autonomous agent that closes out its own day: it pulls every task it ran, hands the results to an LLM, and asks it to extract lessons — what worked, what didn't, what needs a human. One of those lessons is supposed to flag action_required: the field a task sets when it's stuck waiting on something only a person can do (confirm an email address, approve a listing, whatever).
For several days running, the nightly summary kept missing a blocker that was sitting in plain sight. Five demo deployments were stalled on one specific owner action. Every single task that hit that blocker wrote it into action_required. And every night, the LLM-generated lessons talked about task volume, about revenue estimates, about strategy mix — and said nothing about the blocker at all. Not "low priority." Not "deprioritized." Just absent, like it had never been logged.
The wrong theories
First guess: prompt problem. Maybe the lesson-extraction prompt wasn't asking the model to look for blockers explicitly enough. We tightened the prompt, added an explicit instruction to surface action_required when present. No change.
Second guess: the model was just deprioritizing it — LLMs summarizing long inputs are known to drop details that seem secondary. We reordered the prompt to put escalation instructions first, added a one-line example of what a good blocker-flag looks like. Still nothing.
Third guess, and the one that wasted the most time: maybe the extraction function itself had a bug — some off-by-one in how it parsed the model's response before writing lessons to the database. We reread _extract_lessons() line by line. It was fine. It was correctly reading whatever the model gave it. The model just was never being given the field in the first place.
That reframing — stop looking at what happens after the LLM call, start looking at what goes into it — is what actually cracked it.
The actual root cause
The function that builds the nightly context, _build_day_summary(), takes each task's result object, serializes it, and truncates it before folding it into the day's summary:
summary_line = json.dumps(result.as_dict())[:400]
That looks harmless. It's a guard against one verbose task result blowing up the token budget for the whole day's summary. Four hundred characters felt generous.
The problem was as_dict()'s key order:
def as_dict(self):
return {
"earned_usd": self.earned_usd,
"spent_usd": self.spent_usd,
"notes": self.notes,
"action_required": self.action_required,
"potential_revenue": self.potential_revenue,
"artifacts": self.artifacts,
"failed": self.failed,
}
notes comes before action_required. notes is free-text and often runs long — a task explaining what it did, what it found, what it's thinking about trying next. On any task where notes ran past roughly 300-350 characters, the JSON serialization pushed action_required past the 400-character cutoff. json.dumps[:400] doesn't care that it's mid-object; it just chops the string. The field wasn't malformed, wasn't null, wasn't skipped by any explicit logic anywhere. It simply didn't exist yet at the byte offset where the string got cut off.
No exception. No warning log. No test failure. The truncated JSON was still syntactically broken in a way that would fail to json.loads() if anyone tried — but nobody did, because the string was only ever going into a prompt as text, never parsed back out. A field being silently amputated by a length limit produces no error signal anywhere in the stack. It just produces an LLM that was never shown the thing it needed to reason about.
And it's a genuinely nasty bug to catch by inspection, because it's data-dependent. Short notes field, the bug doesn't fire. Verbose notes field, it does. Whether you see it depends entirely on how chatty that day's tasks happened to be.
The fix
The fix stopped truncating raw JSON text and started truncating after deciding what mattered:
def _task_summary_fields(result_json: str) -> dict:
try:
parsed = json.loads(result_json)
except (json.JSONDecodeError, TypeError):
return {"raw": result_json[:400]}
return {
"notes": str(parsed.get("notes", ""))[:300],
"action_required": parsed.get("action_required", ""),
}
action_required is pulled out and given its own untruncated slot instead of competing with notes for space inside a single character budget. We also added a rollup — pending_action_required_count — computed once per day and handed to the LLM as an explicit number, so the model doesn't have to infer backlog size from scattered prose across a dozen task summaries. Malformed JSON still falls back to a raw truncated string, so nothing regresses for genuinely broken task output.
The lesson
Character-count truncation on serialized structured data is a silent information-loss bug by construction: it has no failure mode that looks like a failure. It doesn't throw, doesn't return an error, doesn't even produce invalid output most of the time — it just quietly decides, based on how verbose an unrelated field was that day, whether your most important field exists.
The test suite for this code passed the whole time. It asserted that lessons got extracted, that the LLM call didn't blow up, that malformed results were handled — all real, all necessary, none of it enough. Nobody had written a test that asserted which specific fields survive summarization when other fields are long, because that's not the kind of thing you think to test until you've watched it fail in production for two days straight.
That's exactly the class of bug a CI pipeline is built to catch and a code reviewer is built to miss — it requires running the function against a synthetic input engineered to be adversarial (a long notes, a short action_required), not reading the function and reasoning about what it does. A five-line golden test — feed the summarizer a task result with a 500-character notes field and a non-empty action_required, assert the output still contains it — would have failed on the very first commit that introduced the character slice, months before it ever reached production. The fix isn't "write more tests" in the abstract. It's: for any code that trims, truncates, or samples structured data before it reaches a consumer that can't ask follow-up questions, the pipeline should have a check for exactly which fields are guaranteed to survive — and it should run on every push, not get discovered by a human noticing a pattern across several days of missing alerts.
