The Harness Is All You Need

Verification-first architecture for language-model repair pipelines.

A method for putting a language model inside a maintenance system without letting it decide anything.

Abstract

A large class of engineering maintenance work has the same shape. A small machine-readable rule, which locates or transforms a value inside an input the system does not control, silently stops producing correct output because the input changed. The rule is short, the failures are many, and a competent engineer can repair any single case in minutes once the evidence is in front of them. The cost is not difficulty. It is volume and triage.

This is an attractive target for a language model and a dangerous one. The model can propose a repair from evidence, but a wrong repair is worse than no repair, because it produces plausible output that no alarm fires on. The naive pipeline, which collects failures, shows them to a model and applies the answers, fails in three independent ways at once, and every one of those failures is invisible in aggregate metrics.

This document describes a harness that makes the approach work. It is derived from a production deployment that maintains roughly 1,400 such rules against inputs from as many independent external sources, covering about 1.6 million monitored items, and it has been checked against two further domains. The vocabulary is deliberately abstract so that the method can be mapped onto any problem with the same shape.

The central claim is measurable and, in that deployment, measured. The quality of such a system is governed by the oracle and by the evidence given to the generator, not by the capability of the generator. Tripling model size produced no usable improvement. Constraining the prompt and narrowing the evidence roughly doubled precision.

Contents

1. When this method applies
2. Terminology
3. Five invariants
4. Reference architecture
5. Stage A. Attribution
6. Stage B. Compression
7. Stage C. Generation
8. Stage D. The oracle
9. Stage E. The human gate
10. Designing the outcome taxonomy
11. Evaluation protocol
12. Diagnosing failure: model or evidence?
13. Cost model
14. Anti-patterns
15. Design checklist
16. Instantiating the method in a new domain

1. When this method applies

The method fits a problem when six conditions hold together. If any of them fails, a different design is appropriate, and section 16 says which.

The first condition is that the rule is small and structured. It must be short enough to generate in full and, more importantly, short enough to verify exhaustively rather than by sampling.

The second is that correctness is machine-checkable against evidence you already hold. This is the load-bearing condition. If you cannot decide, mechanically, whether a proposed repair is correct using evidence already in your possession, you do not have this problem. You have a research problem. Do not build a harness; build the oracle first, then reconsider whether the harness is still needed.

The third is that failures are numerous and repetitive. Below some volume a human repairs them faster than you can build the machinery, and the machinery will be obsolete before it pays for itself.

The fourth is that a wrong repair is silent: it produces output, just wrong output. This is what separates the problem from code generation. A generated function that is wrong usually throws, fails a test, or refuses to compile. A wrong rule returns a value. The pipeline stays green and the data quietly rots. Every design decision in this document follows from taking that seriously.

The fifth is that the evidence for a repair is local, meaning one or two examples suffice. If a repair requires reasoning across the whole system, the generator has the wrong shape and no harness will fix that.

The sixth is that a human can adjudicate a proposal in under a minute given the right display. The gate is deliberate, but it must not become the bottleneck, and whether it does is a property of your interface rather than of your reviewers.

Field note. In our deployment the oracle predated the model by a year and could be replayed over historical evidence at will. That single fact is why the project was tractable at all: every design question could be settled by re-running the oracle over inputs already on disk, and no question required a new data collection.

2. Terminology

The vocabulary below is used consistently throughout. It is worth fixing before the design discussion begins, because most disagreements about systems of this kind turn out to be disagreements about which component is being described.

A rule is the small structured object being repaired. A source is the external system whose inputs the rule is applied to, and which changes without warning or coordination. An instance is one failing case, meaning a rule together with the evidence of its failure, and evidence is the stored input on which the rule produced wrong output or none at all.

The generator is the model, in its single narrow role: evidence in, candidate rule out. The oracle is deterministic code that decides whether a candidate is acceptable, and a verdict is the oracle's typed judgement on one candidate. A gate is a particular kind of oracle check, one that can only demote a verdict and never promote it. A trial is empirical evaluation of a candidate against a held-out population of real instances. The adjudicator is the human who approves or rejects, and is the only writer to production. The harness is everything except the generator.

The asymmetry in that list is deliberate and should be stated explicitly. The generator has one job and no authority. Everything that decides anything is either deterministic code or a person.

Field note. We introduced the word "gate" only after the second production incident, when it became clear that the team was arguing about whether a check should be able to rescue a candidate. Naming the demote-only property made the answer obvious and closed the argument permanently.

3. Five invariants

State these in your design document and test them. They are not style preferences. Each one was learned by violating it.

I1. Deterministic before generative

Every filtering, grouping, classification and verification step is ordinary code. The generator is invoked only on instances that deterministic code could not dispose of, and only for the one judgement that genuinely requires reading unstructured evidence.

The consequence is that the generator sees a tiny fraction of the input. This is what makes the economics work, and it is also what makes the results interpretable: when quality moves, you know which stage moved it.

Field note. In one production run, 435,702 failing cases entered the pipeline and 137 pieces of evidence reached the model. That is roughly one in three thousand. Everything else was disposed of by rules that cost nothing to run.

I2. The oracle runs the production code path

Verify a candidate by executing the same code that will consume it in production, with the same parser, the same normalisation and the same coercion. Never a reimplementation, never an approximation, and never "the model says it matches".

The consequence is that a candidate which passes verification passes because production would pass. A reimplemented oracle drifts from production silently, and its drift is indistinguishable from a model failure, which contaminates every subsequent measurement.

Field note. Our generation loop needed a fast in-process check for speed, so a second, lighter implementation was unavoidable. Rather than assume the two agreed, we measured: 3.2% of candidates were decided differently by the two parsers. Small enough to tolerate, and large enough that assuming zero would have been a mistake we could not have detected later.

I3. Every claim is downgraded to a measurement

Do not let a qualitative judgement survive into the pipeline. "The candidate looks right" becomes a verdict from the oracle. The verdict becomes a hit rate over a held-out population. The hit rate becomes a comparison against what production currently achieves on the same evidence.

Field note. Three candidates in one review batch carried an identical verdict. Under trial, the first was a genuine repair, moving from zero successes in fifty to fifty in fifty. The second was pure churn: thirty-three successes in thirty-three both before and after, with byte-identical output. The third had no testable population at all, so no evidence existed in either direction. Three correct decisions, three different actions, one verdict. Without the trial layer all three would have been treated alike.

I4. Precision over recall, deliberately

In a human-gated system the scarcest resource is the adjudicator's trust. A queue at high precision gets reviewed. A queue at low precision gets ignored, and once ignored it is dead regardless of what it contains.

The consequence is that most tuning should make the generator answer less often. Design the contract so that "no answer" is an explicitly correct response, and say so in the prompt rather than hoping it is inferred.

Field note. Across our whole sequence of prompt changes, the count of genuinely good candidates barely moved, staying between eight and eleven on a fixed evaluation set, while precision rose from 40% to 91%. The entire gain was the model learning to decline. Nothing we did made it better at finding answers; everything we did made it better at not inventing them.

I5. Decisions are durable

An approval is never silently overwritten by a later run. A rejection is never resurrected: the same candidate must not reappear as though it were new, and a genuinely different candidate for the same slot should arrive annotated with what was already rejected there.

Field note. Before we added rejection memory, a monthly cycle re-proposed the same rejected candidates every month. Reviewers learned within two cycles that the queue contained things they had already dismissed, and review throughput fell accordingly. The fix was a few lines of persistence and it recovered more reviewer time than any model change we ever made.

4. Reference architecture

flowchart TD
    IN["Failing instances"] --> A
    A["A. ATTRIBUTION<br/>metadata only, no evidence decoded"] --> Aq{"Can repairing the<br/>rule fix this?"}
    Aq -->|no| ROUTE["Routed to the owning<br/>discipline, with the reason"]
    Aq -->|yes| B
    B["B. COMPRESSION<br/>group by structural equivalence"] --> Bq{"Is this evidence<br/>a valid basis<br/>for a repair?"}
    Bq -->|no| EXCL["Excluded, reason recorded"]
    Bq -->|yes| C
    C["C. GENERATION<br/>focused evidence to candidate"] --> CC["critique loop<br/>(mechanical, one round)"]
    CC --> D
    D["D. ORACLE<br/>production code path + gates"] --> Dq{"verdict"}
    Dq -->|not acceptable| FILE["Filed with a typed verdict"]
    Dq -->|acceptable| Q["Review queue"]
    Q --> E["E. ADJUDICATION<br/>evidence displayed, trial on demand"]
    E -->|approve| PROD["Production rule"]
    E -->|reject| MEM["Durable rejection"]

Figure 1. The five stages, and the single path into production.

Two properties of this structure matter more than the individual stages. The first is that the funnel is monotone in cost: each stage costs more per instance than the one before it, so each stage must remove work rather than add it. If a stage does not reduce the population by roughly an order of magnitude, it is not earning its place and should be merged into its neighbour. The second is that exactly one arrow reaches production, and a person is standing on it. Everything else writes to a queue, a file, or a log.

5. Stage A. Attribution

The purpose of this stage is to partition failures by which discipline can fix them, using metadata alone, with no evidence decoded and no model invoked. It is the highest-value stage relative to its cost, and it is almost always skipped by teams who begin from "we have a lot of failures, let us use a language model".

Four rules govern its design.

Buckets must be mutually exclusive and exhaustive. Compute them with a single first-match-wins rule chain rather than with independent conditions. Independent conditions allow one failure to count in two buckets and allow some failures to count in none, and both errors make every downstream number wrong. Assert that the buckets sum to the total, and fail loudly when they do not.

Order the chain by ownership rather than by frequency. The question the chain answers is "who fixes this?", so the most specific owner should be tested first.

Each bucket names a team, not a symptom. "Rejected" is not a bucket. "Access control, owned by the anti-abuse team" is a bucket, because it tells you where the work goes.

A bucket with no owner is a bug in the taxonomy. Add an explicit catch-all and watch it.

Field note. Across one estate, 78% of failures attributed to something other than the rule: access denials, entities that had been deleted at the source, and infrastructure faults. In a single run, 29,852 failures traced to one unreachable gateway, which was a five-minute configuration fix. Had those cases reached the model, it would have produced confident and useless repairs for a problem no repair could touch. Separately, when we replaced four independent bucket conditions with one exclusive chain, 4,307 previously uncounted failures appeared in the catch-all. They had been invisible for months, not because anyone hid them, but because no condition matched them and nothing asserted that the buckets summed to the total.

One blind spot must be documented rather than solved here. A failure can look healthy at the metadata level and be semantically dead, for example an input that announces success and contains only a notice that the entity no longer exists. Metadata cannot see that. Stage B must.

6. Stage B. Compression

The purpose of this stage is to collapse N failing instances into K distinct modes, where K is the number of genuinely different repairs required. Four rules govern it.

Group by structure, never by content. Two instances belong together when the shape of the evidence matches: the vocabulary of identifiers, the schema, the template. Content differs within one mode by definition, so keying on content makes every instance its own mode and the stage accomplishes nothing.

Hard-split before you cluster softly. Partition first on cheap categorical facts, then apply similarity only within each partition. This prevents a similarity threshold from merging categories that are qualitatively different but superficially close.

Take one representative per mode, and carry a sibling. The representative is what the generator sees. The sibling, a second instance of the same mode, is what makes the resulting candidate testable for generality rather than memorisation. This rule is not optional. A candidate validated on exactly one example is indistinguishable from a candidate that encodes that example, and requiring it to hold on a second instance is the cheapest generality test available.

Classify the representative's validity, and exclude with a recorded reason. Not every piece of evidence is a legitimate basis for repair. Evidence drawn from the wrong kind of entity produces a rule that is correct for that evidence and wrong forever afterwards.

Field note. One run compressed 26,481 failing instances to 114 representatives, in 88 seconds of ordinary computation. Most sources had exactly one failure mode; a minority had two or three genuinely distinct variants, and that minority is precisely the population a naive one-instance-per-source deduplication would have mis-repaired without ever revealing that it had. In the same run, 424 of 561 candidate groups were excluded because their evidence was the wrong kind: listings rather than single entities, notices rather than content. Our highest-ranked source by raw failure count was one of them. It had thousands of failures, and no repair existed, because 99.9% of its evidence did not contain the value at all. Ranking by raw failure count points at the loudest source. Ranking by attributable, valid failures points at the fixable one.

7. Stage C. Generation

The purpose of this stage is narrow by construction: given focused evidence and the current rule, propose a candidate or decline.

7.1 Evidence selection is the highest-leverage variable

Do not hand the model the whole input. Anchor on the strongest signal for the value you want, expand to a bounded context around that anchor, and attach only the structured metadata that could plausibly carry the answer.

This is not primarily a cost optimisation. The excluded region is where the wrong answers come from: repeated blocks, adjacent entities, alternate renderings of the same value. Removing that region removes a class of error rather than trading it for another.

The corresponding hazard is real and must be handled explicitly, because a focused view that omits the answer is worse than an unfocused one. Fall back to the full input whenever the anchor is not found with confidence, and measure how often the answer lies outside the focused region.

Field note. Our focused view is roughly 5% of the input. When we finally measured the hazard rather than assuming it away, we found that for one of the two fields we repair, the correct value lay outside the focused region on about one instance in ten. On those instances the model mostly declined, which cost recall and protected precision. That measurement converted a suspicion into a stated bound, and it also identified the single highest-value improvement available to us, which is widening the anchor for that field rather than changing anything about the model.

7.2 The contract

The prompt is a contract rather than a request, and it must state five things.

It must state the task as repair, not discovery. "This rule stopped working, here is the evidence, what should it be" outperforms "find the value", because the framing keeps the model anchored to the semantic slot the original rule meant rather than to any plausible-looking value in the input.

It must state the current rule and the observed failure, which are context the model cannot infer.

It must state hard constraints as prohibitions with reasons: uniqueness, generality across instances, and forbidden constructs such as positional indexes, instance identifiers and state-dependent conditions. Give the reason for each. A constraint with a stated failure mode is followed more reliably than a bare rule.

It must state that declining is a correct answer, explicitly, with an example of when. This single clause carries a large share of the precision gain described in I4.

It must specify a machine-parseable output shape, with a place for the model to record what it saw and why. That rationale is not for the pipeline. It is for the adjudicator.

Field note. Two constraints in our contract exist because of specific incidents. The prohibition on positional constructs was added after a candidate that counted siblings returned an unrelated value the moment an optional element appeared. The prohibition on state-dependent conditions was added after a candidate that keyed on a status marker worked perfectly while the entity stayed in that state and returned nothing, silently, the moment it changed. Neither would have been caught by a check on the evidence in front of us, because on that evidence both were correct.

7.3 Determinism

Fix the sampling temperature at zero and pin the seed. Without this you cannot attribute a metric change to a design change, and every comparison is contaminated by sampling noise.

Field note. Before pinning, two runs of an identical configuration disagreed on 4 of 46 slots. That is enough to move a headline number by two points, which is the same magnitude as several of the design changes we were trying to evaluate. Residual nondeterminism from batched inference remained even after pinning; we measured it once, recorded it, and stopped worrying about it.

7.4 The critique loop, exactly one round

After the first answer, run the cheap mechanical checks. If any fail, return them to the model together with the evidence of what its candidate actually did, not a bare "invalid" but a statement of the form "it matched these three things, whose contents are X, Y and Z". Allow exactly one revision.

The reason for exactly one is that iterating against the same evidence teaches the model to satisfy that evidence, which is overfitting with extra steps.

Field note. In our measurements the second round mostly produces withdrawal to a null answer rather than a better candidate. Under I4 withdrawal is a win, so the round pays for itself without ever needing to produce a repair. When we later gave a much larger model the same feedback on the same cases, it did not do better, which is the subject of section 12.

7.5 A note on reasoning budgets

If your generator supports an explicit reasoning phase, be aware of a specific failure mode. When reasoning and answer draw from one output budget, the reasoning expands to consume the entire budget and the model returns nothing.

The fix is a separate hard limit on the reasoning phase, leaving the answer budget intact. The general lesson outlives this one setting: verify that "more of a good thing" is monotone before assuming it, particularly where two behaviours draw from a shared pool.

Field note. We enabled reasoning with a generous budget and got 19 empty answers out of 23. Assuming the budget was too small, we raised it, and the failure got worse rather than better, because the reasoning simply expanded to fill whatever it was given. Bounding the reasoning phase separately at a few hundred tokens produced the best configuration we have measured, better than no reasoning at all, and it also stopped the model proposing state-dependent constructs without being told to.

8. Stage D. The oracle

The oracle is the component that makes the system trustworthy, and it deserves more design attention than the prompt.

8.1 Layers

Run the checks in increasing order of cost and stop at the first that rejects.

Begin with syntax, which asks whether the candidate parses or compiles at all. Then uniqueness, which asks whether it resolves to exactly one thing. Then value, which asks whether the production consumer accepts what the candidate yields. Then generality, which asks whether it also holds on the sibling instance and on other evidence from the same source. Then form, which asks whether it violates a stated constraint such as an embedded instance identifier or a positional index, and which is nearly free because it is a property of the candidate's text rather than of its execution.

Two further checks are more expensive and are described separately below, because they are the ones most systems omit.

8.2 Gates demote, never promote

A gate can only lower a verdict, never raise one. This asymmetry is what lets you add gates over time without re-validating everything that came before: a new gate can only make the queue more conservative, so its introduction can never invalidate a past approval.

8.3 The two gates people forget

The regression gate requires something most pipelines do not keep, namely evidence of success. Store, for each source, the most recent input on which the rule worked, together with the value production derived from it. Without that store, every candidate is evaluated only where the incumbent fails, which is the single place where the incumbent is guaranteed to lose, and the system will ship candidates that repair the broken minority while breaking the working majority.

The null-hypothesis gate asks whether the incumbent already works on the failing evidence. If it does, this rule is not the cause of the failure, something else is, and the candidate should be demoted with a pointer to the real question.

Field note. The success store was the single highest-value addition we made after the first production run. It is inexpensive, being one bounded, compressed input per monitored item with a two-week lifetime, and it immediately caught thirteen candidates in one run that would otherwise have shipped. It also supplies the population for the trial in stage E, so one storage decision paid for two features. The null-hypothesis gate, added at the same time, reclassified a large fraction of a subsequent run's queue: those cases were failing on the other field, and every minute spent reviewing them as repairs was wasted.

8.4 Merging, not overwriting

If a source can have several modes, several candidates will arrive for the same slot. Do not key your queue by source alone.

The correct behaviour is to verify everything first and then merge per slot. Identical candidates merge with their impact summed. Genuinely different candidates either compose, when the consumer supports ordered alternatives, in which case compose them and verify the composition as a unit, or are published as one winner with the alternatives attached and visible to the adjudicator.

A corollary applies as soon as more than one entry point feeds the queue. The queue key must include the entry point, or one stream's run will silently retire the other stream's work.

Field note. We keyed the queue by source and slot. One source had four modes and therefore four candidate repairs, which overwrote one another in arrival order. The survivor came from the smallest mode, representing 18 failing cases, while three larger repairs covering 252 cases vanished without any record that they had existed. After the fix, the same source published a composed candidate whose honest impact was 189 cases, and which the trial confirmed at fifty successes in fifty. Separately, when we later added a second entry point, its first run retired the first entry point's pending work, because both wrote the same keys. The review queue emptied overnight. Both incidents were schema decisions made in week one and paid for in week three.

9. Stage E. The human gate

The adjudicator is part of the system, and the display is part of the design. The card should show evidence rather than conclusions.

It should show the current rule and the candidate side by side, with what each yields on the same evidence. It should show the counts behind the verdict, meaning how many things matched and how many sibling instances the candidate held on. It should show the impact, summed honestly across merged modes rather than reported from whichever mode happened to survive. It should show the reason string for any demotion, phrased so a reviewer can argue with it. It should link to the raw evidence and state that evidence's classification. Finally, it should show the model's self-reported confidence in visually de-emphasised form, because it is the only unverified number present.

Make the candidate editable before approval. A large share of near-miss candidates need one qualifier added. An editable field converts them from rejections into repairs in seconds, and it is the cheapest recall you will ever buy.

Approval must be minimal and targeted. Write exactly one field, to the location the consumer actually reads, and record on the approval record the exact path written.

Field note. We shipped a version in which approval wrote to a location the consumer never read for one whole class of sources. The review showed green, the record said approved, and production behaviour did not change at all. Nothing failed, nothing alerted, and the only reason we found it was that a later audit compared what had been approved against what production was doing. Since then, every approval records its own write path, and the audit is a one-line query.

9.1 The trial

The strongest instrument available to the adjudicator runs both the incumbent and the candidate over a sample of real instances drawn from both populations, meaning instances that currently fail and instances that currently succeed, and reports per-instance results.

Compute a verdict from that comparison and persist it, so it survives the session. Five outcomes suffice: gain, where the candidate wins on failing evidence and loses nothing; no gain, where behaviour is identical to the incumbent and the change is therefore churn; loss; regression; and unverifiable, where no testable population exists.

The trial is the only stage that samples instances nobody selected, which is exactly why it catches what every earlier stage missed.

Field note. Our trial samples fifty failing and fifty working instances per candidate. It is also the mechanism that rescues a class of candidate the oracle demotes: when the regression gate reports that a candidate returns a different value on working inputs rather than no value, that is sometimes precisely the intent, because the incumbent had been reading an adjacent value all along. The only approved repair of that shape in our system was demoted by the oracle, rescued by the trial, and confirmed correct in production.

10. Designing the outcome taxonomy

Verdicts are the interface between the harness and the human, so design them as a typed enumeration with an action attached to each. Four rules apply.

Every verdict names an action. If two verdicts imply the same action, merge them. If one verdict implies "it depends", split it until it does not.

Distinguish "produces nothing" from "produces something different". These look alike in a diff and are opposites in meaning. The first is a break. The second is frequently the entire point of the repair.

Include a verdict for "not a repair at all". Some instances are telling you that the entity is gone, the reference is stale, or the access is broken. A taxonomy without that bucket forces them into a repair verdict, where they are reviewed forever and never resolved.

A verdict says the rule is valid, not that the action is wise. A candidate can be perfectly correct and still derive a value from evidence you should not be processing at all.

Field note. Two candidates in our queue were marked acceptable by every check and were correct in the strict sense. One faithfully returned a sentence stating that the entity no longer existed; the other faithfully returned a notice that the caller was not authorised to see it. Both were flawless rules applied to evidence we should never have collected. The display now carries the evidence classification next to the verdict, because no verdict can express this on its own. Our regression verdict still conflates the two meanings named above and separates them only in its reason string; in hindsight they should have been two verdicts, and that is the change we would make first.

11. Evaluation protocol

Most reported numbers for systems of this kind are not comparable across time, because the checks change as the system improves. Fix that before anything else.

11.1 Re-score history with today's oracle

Keep every run's raw generator output. Score all runs, always, with the current oracle. A run from month one and a run from month three then differ only in what changed by design.

This has a consequence people find uncomfortable and should not avoid. A stricter oracle makes past runs look worse, and it makes the current run look worse than last month's. That is the metric working correctly. A benchmark that only ever improves is measuring your optimism.

Field note. Adding the regression gate dropped a headline precision figure from 51% to 49% on comparable populations, because the gate rejected candidates the previous run had happily called acceptable. We published the drop with the explanation attached. The alternative, quoting the older number because it was more flattering, would have meant reporting a quality improvement that consisted entirely of a weaker test.

11.2 Define precision to exclude declining

Precision is acceptable candidates divided by candidates proposed. An instance the generator declined is not counted against it, because declining on evidence with no valid answer is correct behaviour, and penalising it would optimise directly against I4.

Report the decline rate separately. It is a real quantity, since it bounds recall, but it is not an error rate and must not be folded into one.

11.3 Two populations, reported separately

The laboratory population is a fixed, hand-picked set of twenty to fifty instances, used to attribute the effect of one change at a time. It is not representative and must never be quoted as system performance. Its job is causal attribution, and for that job a small fixed set is exactly right.

The field population is unselected production traffic. This is the honest number, and it will be lower.

Field note. Our laboratory figure is 91%, on twenty-three instances chosen because a repair was known to exist in each. Our field figures are 76% where the evidence is healthy and 49% where the evidence is already failing. All three are true, all three are quoted in different documents, and each one answers a different question. Quoting the laboratory number as system performance would have been the single most misleading thing we could have done, and it would have been entirely defensible from the raw data.

11.4 Stratify by difficulty, and say why the strata differ

Do not average across populations of different difficulty. Say what makes them differ, in terms of the evidence rather than in terms of the outcome.

Field note. Our two entry points differ structurally. One draws evidence where the rule already fails, which is hard, because a substantial share of that evidence has no valid answer at all: the entity is gone, or the value was never present. The other draws evidence where the system is healthy, which is easier, because the value is present and a reference value exists for comparison. The gap between 49% and 76% is almost entirely that difference, and averaging the two would produce a number describing no workload we actually run.

11.5 One change at a time, with the question written down

For each experiment record five things: the question it tests, the single change made, the population it ran on, the result, and the decision taken. A sequence of such records is the most useful artifact your project will produce. It stops the team re-running a dead end six months later, and it lets a newcomer see that the gains came from constraints rather than from scale.

12. Diagnosing failure: model or evidence?

When precision plateaus, the instinct is to reach for a larger generator. Test that instinct, because it is cheap to test and it is usually wrong.

12.1 First, categorise every failure mechanically

Take a full run's rejected candidates and bucket them by why the oracle rejected them, then compute the agreement rate on the accepted ones.

Field note. Across 442 rejections in one run, just under a third, 31%, resolved to nothing at all: a guessed chain of identifiers matching nothing in the evidence. A further 37% resolved to several things, meaning the candidate was not scoped to the intended entity and would silently read a neighbouring one. Both are generator failures, and both are aggravated, because the model had already been told in its revision round exactly what its candidate matched. A further 13% violated a form constraint stated in the prompt, typically a positional index or an embedded identifier. Another 9% resolved to exactly one thing whose content was not a value at all but a label, an identifier or an unrelated attribute. A residual 2% did not parse.

The remaining 3% were ours: a defect in our own numeric coercion, which read a comma-decimal amount as an integer a hundred times larger for sources using European conventions. No model change would have fixed those, and without the categorisation we would have attributed them to the model and tuned the prompt against our own bug.

Two conclusions followed immediately. First, 68% of the failures were candidates the generator's own in-loop check had already flagged, and the model returned them regardless. Those never reach a human, so queue quality is unaffected, but they establish that the ceiling is bounded by the model's compliance rather than by its ability to find answers. Second, where candidates were accepted, they reproduced the value production itself derived from the same evidence in 98.5% of cases, which established that the evidence was sound and the failures were not a data-quality problem wearing a disguise.

12.2 Then run the escalation experiment

There is a cheap and decisive test of the "bigger model" hypothesis. Take only the candidates the oracle rejected. Send each to a larger or differently-trained generator, with the full context: the same evidence, the first generator's answer, and the oracle's explanation of exactly what that answer did. Score with the same oracle, and report four outcomes: repaired, withdrawn, still failing, and damaged, where damaged means slots that were correct before and are wrong now.

Field note. We ran this with a model roughly three times the size of our production one, trained specifically on code. On healthy evidence, 129 failing slots were sent; 28 of them, or 22%, were repaired; 4 were withdrawn; 97 remained wrong; and 24 slots that had been working were damaged. On failing evidence, 34 slots were sent and none at all were repaired: 14 were withdrawn, 20 remained wrong, and 5 working slots were damaged. The larger model made the same two mistakes as the smaller one, resolving to nothing and resolving to several things, while looking at an explicit list of what its predecessor's candidate had matched. Twelve minutes of computation and two dollars settled a question the team had been arguing about for weeks.

The inference is that the limiting factor was the evidence window rather than the parameter count. The information needed to disambiguate lay outside the focused region we were sending. That is a retrieval and context problem, and it is fixed by widening and structuring the evidence, not by scaling the generator.

12.3 Escalate per-slot, never per-instance

If you do escalate, apply the second generator only to the specific slot that failed, and merge its answer back into the existing record.

Field note. Letting the second model rewrite whole instances made the system worse, moving it from 50% to 46% on the same population, entirely through collateral damage to slots that had been correct and were not in question. Restricting it to the failing slot and merging moved the same population from 75% to 80%. The difference between those two outcomes is not the model. It is the scope it was allowed to touch.

13. Cost model

At realistic volumes, cost is dominated by fixed setup rather than by inference, and the intuition that inference is the expensive part leads directly to the wrong optimisations.

Attribution and compression are effectively free, running in seconds to minutes with no model involved. Generator provisioning is the dominant fixed cost. Inference itself costs cents and a few seconds per instance. The oracle is free. Adjudication is the real cost, and it is measured in human minutes rather than in currency.

Five consequences follow. Batch aggressively, because two runs of a hundred instances cost far more than one run of two hundred. Parallelise by sharding the input rather than by scaling the machine, since several modest workers on several shards beat one large worker. Keep the environment warm across iterations during development, where the setup cost would otherwise be paid on every experiment. Make teardown independent of your session, because a crashed terminal must never leave a meter running: track resources by an external identifier, provide a one-command teardown, and, because local state files lie, verify against the provider's own interface that nothing is still running. Finally, optimise the adjudicator's minute rather than the inference cent, because at our volumes review time exceeds compute cost by orders of magnitude, and every display improvement that removes a click is worth more than a model change that removes a fraction of a cent.

Field note. On a typical run of ours, environment setup and model loading consumed 21 of 46 billed minutes. The inference for the whole run cost less than a dollar. Separately, a state file recording a single rented machine led us to believe a teardown had succeeded when three machines were still running, one of them expensive; only an audit against the provider's own listing revealed it. Both lessons are cheap to learn from someone else's writing and expensive to learn from an invoice.

14. Anti-patterns

Each of the following was either done by us or seriously proposed, and each carries a specific cost.

Going straight to the model, skipping attribution and compression, is the fastest thing to build and the most expensive to own. The model spends its budget on problems no repair can fix, and because nothing upstream partitions the input, the results cannot be attributed to any cause.

One instance per source looks like obvious deduplication and silently hides every source with more than one failure mode. You repair the smallest mode and declare the source fixed.

Reimplementing the consumer inside the oracle is faster to write and produces an oracle that drifts from production. The drift then masquerades as model error, and you tune the prompt to chase a bug in your own verifier.

Verifying only on failing evidence is tempting because that is the evidence you collected. It guarantees the incumbent loses every comparison, and you will ship regressions with confidence.

Auto-applying high-confidence candidates removes the bottleneck by removing the only component that can see semantic wrongness. Because wrong repairs are silent, you will not notice.

Trusting self-reported confidence is tempting because the number is right there in the output. It is uncalibrated, and it correlates with fluency rather than with correctness.

Iterating the critique loop until it passes looks like convergence and is overfitting to the single instance inside the loop.

Keying the queue by source alone is the simplest schema and causes multi-mode repairs to overwrite one another, and parallel entry points to destroy each other's work.

Reaching for a bigger model at the plateau is the culturally default move, and in our measurement it repaired 22% of one population and none of the other while damaging slots that already worked.

Reporting a single precision number makes for a cleaner narrative and averages populations of different difficulty, producing a figure that describes no real workload.

Normalising values into a fixed vocabulary before storing them produces tidier data and destroys the evidence. A substring match once mapped a negative statement onto its positive counterpart, inverting the meaning; because only the mapped value was retained, the error was undetectable after the fact. Store what the source said, and normalise on read.

15. Design checklist

The three lists below are ordered as they arise. Each item is a condition to be satisfied rather than a task to be ticked, and a design that cannot satisfy one should record why in the same place it records the rest of its decisions.

Before the first model call.

1. The oracle exists, runs the production code path, and can be replayed over historical evidence.
2. Attribution buckets are mutually exclusive, exhaustive, named after their owning discipline, and asserted to sum to the total.
3. Compression groups by structure and carries a sibling instance for every mode.
4. Evidence validity is classified, and invalid evidence is excluded with a recorded reason.
5. A store of successful evidence exists, holding the value production derived from each stored input.
6. The queue key includes the source, the slot and the entry point.
7. Verdicts are typed, each carries an action, and gates can only demote.
8. The generator's contract states its constraints, the reason for each, and that declining is a correct answer.
9. Sampling is deterministic and seeded.
10. The critique loop is bounded at one round.
11. Approval writes a single field, to the location the consumer reads, and records that path on the approval.
12. Teardown of paid resources works from a cold start and is verified against the provider rather than against local state.

Before quoting a number.

1. Every run has been re-scored with the current oracle.
2. Laboratory and field populations are reported separately and labelled as such.
3. Populations of different difficulty are reported separately, with the structural reason for the difference stated.
4. The decline rate is reported alongside precision rather than folded into it.
5. The experiment log records, for each change, the question it tested and the decision it produced.

When precision plateaus.

1. Rejections have been categorised mechanically by oracle reason before any model change is considered.
2. The agreement rate on accepted candidates has been computed, to rule out an evidence problem.
3. Escalation has been tested on rejected slots only, scored with the same oracle.
4. Damage to previously-correct slots has been counted, not only repairs.

16. Instantiating the method in a new domain

Map your problem onto the terminology of section 2, then work through the checklist. Three domains illustrate how little the method changes.

Rules that locate a value inside a semi-structured input. The rule is a path or pattern; the source is an external system that emits inputs on its own schedule; the evidence is a stored input on which location failed. The oracle is the production parser together with the value coercion that follows it. Uniqueness means the candidate resolves to exactly one element, and generality means it holds on a second input of the same shape. The regression gate replays stored successful inputs and requires their known values back. The null-hypothesis gate asks whether the current rule already succeeds on the failing evidence. The trial runs both rules over fifty failing and fifty working inputs, and the adjudicator sees the value each one derives, side by side.

Field mappings in schema or configuration migration. The rule is a mapping or a configuration key; the source is an upstream system; the evidence is a rejected record. The oracle is the real ingestion validator rather than a copy of its rules. Uniqueness means the candidate maps to exactly one destination field, and generality means it holds on a second record of the same shape. The regression gate replays previously-valid records and requires that they still validate. The null-hypothesis gate asks whether the current mapping already accepts the record. The trial runs both mappings over a held-out batch, and the adjudicator sees which records each accepts and rejects.

Assertions and wait conditions in flaky-test repair. The rule is an assertion or a wait condition; the source is a test suite; the evidence is a failing run's trace and timing. The oracle is the real runner executing the real suite. Uniqueness means the condition resolves to one target, and generality means the behaviour holds across N repeated runs rather than one lucky one. The regression gate requires that previously-passing tests still pass. The null-hypothesis gate asks whether the test is failing for a reason unrelated to timing. The trial runs both versions fifty times, and the adjudicator sees the pass rate and the timing distribution of each.

Four situations call for something other than this method.

When no mechanical oracle exists and condition C2 fails, build the oracle. If the oracle turns out to be the hard part, then the model was never your bottleneck and a harness will not help you.

When instances are few and individually valuable and condition C3 fails, use an interactive agent with tool access on individual cases. It costs an order of magnitude more per case and reads context a batch pipeline cannot, which makes it right for a handful of hard sources and wrong for thousands.

When repairs require global reasoning and condition C5 fails, restructure the problem until the unit of repair is local, or accept that what you have is design work rather than maintenance work and staff it accordingly.

When wrong repairs are loud and condition C4 fails, you may be able to auto-apply with a rollback and skip the human gate entirely. Verify first that the failure really is loud in production rather than merely loud in a test environment, because the two are routinely confused and only one of them protects you.

Field note. We run the first domain in bulk and reserve an interactive agent for a handful of sources whose inputs carry their values inside embedded state rather than in the structure itself. The agent costs roughly forty times more per case and is the only tool we have that reads those inputs correctly. The boundary between the two is volume, not difficulty, and it is worth stating in the design document so that nobody has to rediscover it under deadline.

Summary

The generator is the least important component of the system. What determines whether such a system produces trustworthy repairs is seven things: attribution, which refuses to work on failures no repair can fix, and which accounted for 78% of ours; compression, which yields one representative per genuine mode with a sibling for generality; evidence selection, a small and well-chosen window, because the excluded region is where the errors live; the oracle, running the production code path through layered checks and gates that only demote; a store of success, without which regression cannot be detected and will therefore be shipped; typed verdicts and durable decisions, so that reviewer effort accumulates rather than resetting; and honest evaluation, meaning re-scored history, separated populations, and a metric that is permitted to get worse.

Build those seven and a small general-purpose model is sufficient. Skip them and no model is.

Derived from a production deployment maintaining roughly 1,400 rules over inputs from as many independent external sources, covering approximately 1.6 million monitored items. All quantitative claims are measurements from that deployment. They are offered as evidence for the design rules, not as benchmarks to reproduce.