Agentic Software Will Have More Test Code Than Product Code

In several products we are building, Codex App Server is no longer just a developer tool. The agent loop is part of the product.

It can inspect state, choose tools, mutate persistent data, request approval, recover from failures, and reach the same valid outcome through different paths.

As we increased that autonomy, I started noticing a strange imbalance.

The product code was growing, of course. But the amount of code and machinery around verification was growing in a different way.

Not just unit tests or E2E.

We started accumulating evals, fuzzers, deterministic verifiers, independent oracles, capability definitions, traces, replay, policies, audit evidence, and release gates.

At first I called all of this "test code." That stopped feeling accurate fairly quickly.

The better name is verification infrastructure.

I do not literally mean that every agentic application will end up with more test LOC than product LOC. The more interesting pattern is that a relatively small amount of implementation can now expose a surprisingly large amount of behavior.

That changes what we have to verify.

A small agent runtime can create a large behavioral surface

Traditional CRUD software usually has a fairly visible path from user action to system effect.

A click maps to an endpoint. The endpoint validates something, updates state, and returns a result. The system can be complex, but there is usually a fairly direct relationship between implementation paths and behavior.

Agentic software breaks that relationship.

A user gives an intent. The agent may inspect state, choose one tool, observe the result, choose another tool, retry after a failure, pause for approval, or reach the same outcome through a different sequence altogether.

Codex App Server is a good example of the shift. It exposes the Codex harness as a long-lived bidirectional JSON-RPC process. A request can produce multiple events, tools can execute during the run, and the process can pause while waiting for approval.

Once that runtime is embedded in a product, it is no longer useful to think of it as "an LLM call."

I wrote about that earlier in The Agent Runtime Got Heavy: Rethinking Sandboxes with Codex App Server.

The mental model I use now is roughly:

Behavioral Surface
≈ Intents × States × Tools × Permissions × Failures × Recovery Paths

Not as a real formula. Just as a reminder that the number of things an agent can do grows much faster than the amount of orchestration code required to expose those choices.

This is the part I think is easy to underestimate.

A read-only copilot is one thing. An agent that edits persistent state, calls external systems, changes permissions, or performs irreversible actions is another. The code difference between those products may not look huge. The verification burden does.

That is why source LOC feels less useful to me as a proxy for complexity than it used to.

Conventional tests still matter, but they no longer describe the whole contract

I do not think the test pyramid is obsolete.

If authentication breaks, I still want a deterministic test to fail. API contracts still need integration tests. Ordinary UI paths still need E2E.

What changed for us is that those tests stopped answering all of the important questions.

Anthropic's agent-eval guidance describes the same basic problem: multi-turn agents use tools, modify environment state, and adapt to intermediate results. That makes the execution trace and the resulting environment state part of what needs to be evaluated.

One concrete consequence is that a test case is not always a single run anymore.

In some of our agent E2E work, a single Green result did not tell us enough. We needed repeated trials and had to distinguish product failures from environment failures and inconclusive runs.

That starts to feel closer to reliability engineering than ordinary regression testing.

I wrote more about that in The Hard Part of Agent Evals Is Deciding What Counts as Green.

Coverage also starts to mean something different.

We cannot enumerate every prompt, every workspace state, every tool ordering, and every recovery path. So instead of trying to cover the entire behavior space directly, we end up encoding the risks we care about:

• the agent must stay inside the permitted workspace;
• destructive operations require approval;
• cancellation must stop durable side effects;
• a claimed artifact must actually exist;
• recovery must not corrupt persistent state.

At some point the eval suite stops looking like a collection of examples and starts looking like a machine-readable description of the product boundary.

I am still not sure where the right abstraction is here.

Some of these belong in tests. Some belong in policy. Some are better represented as deterministic verifiers. Some only make sense as production evidence.

But that ambiguity is part of the point: the verification layer is becoming substantial enough that it needs its own architecture.

Where facts exist, I prefer facts

A natural response to agent uncertainty is another LLM judge.

Sometimes that is the right answer.

If I need to evaluate whether a generated report is useful, whether an explanation is clear, or whether an artifact is semantically good, there may not be a deterministic oracle.

But many agent claims are not subjective.

If the agent says it created a file, inspect the filesystem.

If it says it updated persistent state correctly, inspect that state.

If it says it stayed inside a workspace boundary, inspect what it actually touched.

This is the pattern I increasingly prefer:

Probabilistic Agent
        ↓
Observed Execution
        ↓
Facts / Trace
        ↓
Deterministic Verification
        ↓
Policy Decision
        ↓
Evidence

This is also why fuzzing, evals, traces, and oracles keep showing up together in the same systems.

They are doing different jobs.

• Fuzzing explores what can happen.
• Eval asks whether the agent can succeed.
• Trace records what actually happened.
• Oracle decides whether the result is correct.
• Policy decides whether the behavior was allowed.
• Evidence makes that decision reproducible.

I ran into the same issue from another direction while working on fuzzing: Your Fuzzer Is Only as Smart as Its Oracle.

Generating more cases is cheap. Knowing whether they reveal a real failure is the hard part.

There is also a distinction I did not appreciate enough at first: the final state and the trajectory are not always equivalent.

If an agent can safely produce the same artifact through ten different tool sequences, I usually do not want to constrain it to one path.

But if it reads sensitive data, sends it externally, and only then asks for approval, the final state does not rescue the trajectory.

So the rule I use for now is simple:

Constrain the path when the path itself has safety or policy meaning. Otherwise, prefer the outcome as the oracle.

Production traces also start to matter here. Once they are structured and durable, failures can be replayed and converted into regression or eval cases.

That creates a loop from production behavior back into verification.

I wrote about the observability side of this in Installing OpenTelemetry Won't Close Your AI Agent's Feedback Loop.

The other half of the story is economics

So far, the argument is mostly about demand.

More autonomous behavior creates more things we need to verify.

But there is another change happening at the same time: the cost of building verification is falling.

AI can generate not only product code, but also fixtures, fuzz harnesses, eval scenarios, graders, mocks, replay tooling, fault injection, and CI checks.

OpenAI's Harness engineering is a useful public example. In that experiment, Codex generated application code together with tests, CI, documentation, observability, internal tooling, and evaluation harnesses.

The part I find most interesting is not the amount of generated code.

It is that once implementation throughput increased, human attention became the scarce resource.

That changes the economics.

Historically, many teams knew they could benefit from another simulator, another failure-replay tool, another invariant suite, or better fuzzing, but the engineering cost was hard to justify.

Now some of that infrastructure is much cheaper to produce.

So two things are happening at once:

Agent autonomy increases verification demand.

AI-assisted engineering lowers the cost of verification supply.

That combination is probably more important than either trend alone.

It also means we can easily overshoot.

Verification infrastructure has its own failure modes: stale tests, noisy evals, conflicting policies, expensive traces, slow suites, and huge amounts of low-value generated coverage.

So "more verification" is not the goal.

What I actually want is more trusted behavioral coverage for less human attention.

A good invariant can be worth more than hundreds of generated tests. A real production trace can be worth more than many synthetic scenarios. An independent oracle can be more useful than another probabilistic judge.

Verification is becoming part of the product

Verification-heavy software is not new.

SQLite is an extreme public example. For SQLite 3.42.0, the project reports roughly 155.8 KSLOC of library code and a test corpus roughly 590 times larger, including multiple independent harnesses, fuzzing, out-of-memory tests, I/O failures, crash and power-loss recovery, boundary tests, and mutation testing.

I do not think ordinary agent SaaS will look like SQLite.

The useful point is narrower: when the state space is large and incorrect behavior is expensive, verification can become much larger than the implementation it protects.

Agents may bring some of that pressure into much more ordinary products.

A UI can still expose one simple instruction:

"Update these records from this document."

Behind it, the system may interpret data, inspect state, choose tools, call external systems, mutate records, recover from failures, and decide when approval is required.

The user-facing surface remains simple. The behavioral surface does not.

That is why I no longer think "test code" is the right description.

What grows around an agentic product is a mixture of tests, evals, graders, oracles, fuzzers, policies, traces, replay, audit data, and evidence.

That is closer to verification infrastructure.

For most of my career, I thought of tests as support code around the product.

That model feels incomplete once the product itself can decide how to act.

An agentic product is also defined by answers to questions like:

• What may it do?
• What must it never do?
• What counts as success?
• How reliable must success be?
• When is approval required?
• What evidence must exist before we trust the result?

I do not know whether most agentic applications will literally contain more test LOC than product LOC.

That is not really the prediction.

The prediction is that implementation will continue to compress relative to the behavior it enables, while executable evidence expands around that behavior.

We did not just add an agent to the application.

We had to build a verification system around it that increasingly feels like part of the product itself.

AI is making behavior cheap. Verification is becoming the product boundary.