Zero errors, missing records
We asked the agent to build a one-off importer that reads an exported spreadsheet, maps each row into our internal record, and writes it to the database. It returned a clean summary: 'import complete, 0 errors, done.' We ran it, the job exited 0, and CI-style validation reported success.
A day later a user could not find the records from one section of the file. Tracing the source, about a third of rows never reached the database. The parser had encountered rows whose column names or nested shape differed from the single example it was given, skipped them inside a for loop, and moved on. The loop counted nothing, logged nothing, and exited clean.
What actually happened
The agent modeled the task from one representative row. It wrote a happy-path mapping: read column A, read column B, build the record, insert. When a row did not match — a renamed header, an optional field present in one section, a blank cell where the example had a value — the code took the defensive path of skipping rather than raising. Skip feels safe: it keeps the job running and avoids an ugly exception on a single bad row.
Nothing in the report reported how many rows were read versus how many were written. A loop that processes an array and ignores unmatched items naturally exits 0, because skipping is not an error to the program. The success signal the agent wrote itself — no exception, exit code 0, 'complete' in the summary — measures whether the code ran, not whether all input survived.
The same shape appears outside batch imports. A generated client reads only the fields it was taught and drops the unknown keys the server now sends. A migration script skips a table whose schema it did not model. A normalization step filters out values that do not match its assumed type and counts the survivors rather than the source. Each time the program runs to completion, and each time the missing slice is invisible in the final status.
Why the agent chooses silence over failure
One example defines the schema. Handed a sample row, the agent treats that shape as the whole input. Divergent rows do not appear in the prompt, so handling them never enters the plan; skipping is the only path the code knows.
Defensive code reads as good engineering. 'Never crash on one bad row' is a reasonable instinct, so the agent wraps parsing to keep going. It stops at tolerance and never adds the accounting that turns tolerance into visibility.
Success is checked on exceptions, not on totals. The natural assertion is 'no error thrown.' A dropped row raises nothing, so a clean run reads as a full run.
Checks that make a dropped row impossible to hide
Reconcile input count against output count. Count source rows and count accepted records; refuse success when the two diverge, or write every skipped row to an explicit rejection file with the reason. A job that processes 1,200 input rows and writes 800 must not exit 0 without naming the missing 400.
Parse in strict mode by default. Reject unknown headers, missing required columns, and mismatched types loudly instead of dropping them; relax per section only where the source genuinely varies, and log the exception.
Surface unknown keys instead of discarding them. When a parser or client meets a field it was not taught, collect it and report it rather than silently dropping it. Unknown input is a signal the model is incomplete, not clutter to throw away.
Assert on a known sample's row. Take a row from each distinct section of the source and confirm it lands with the expected values. Testing only the representative example proves the happy path; the skipped sections stay untested.
Return a structured result, not 'done.' The summary should carry read, accepted, rejected, and error counts so the human sees the full picture before trusting the exit code.
The contract version of the same silence
API clients carry this trap hard. A client generated from a single observed response reads the fields it saw and ignores the rest; when the documented schema grows a field, the client drops it without a warning, and the consumer never learns it arrived. Prose in a README cannot account for every key, so unknown fields need a documented handling rule. We keep that loop local in Powerduck: validate the response against the documented schema, surface unmatched keys, and report them instead of letting the client silently discard what it was not taught.
What I check before accepting an importer or parser as done
- Did the run report read count, accepted count, and rejected count — not only 'no errors'?
- Does a mismatched row fail loudly, or does the loop skip it by default?
- Are unknown headers, missing required fields, and unexpected types surfaced rather than discarded?
- Was at least one row from each distinct source section verified to land correctly?
An exit code of 0 proves the program ran to the end. It says nothing about whether every input survived. A clean summary that omits the totals is a success report for the rows it kept, not the rows it dropped.
