AI-generated unit tests: two patterns that pass with the bug, and a red-green fix

Two tests can pass while applyDiscount(100) returns 100 instead of the ticket's 90. One replaces the function with a mock. The other copies its boundary bug into the expected value.

Agents love to create a thousand unit tests. All green. Then review smashes the whole thing.

The agent already implemented a trash solution, and only then wrote generalized unit tests for it. Those tests describe the code it just wrote. So they all pass, bugs included.

When an agent writes both the code and its tests, don't count the green checks. Look at where each expected value comes from. If it comes from the agent's code, the test can repeat the agent's mistake instead of catching it. If it comes from the ticket, the spec or a real example, and the test runs the real code and failed because the behavior was missing, it checks something.

Two patterns that pass with the bug

Here are both in TypeScript with Vitest. Suppose an agent writes this for a discount ticket:

// pricing.ts. The ticket: orders of 100 or more get 10% off.
export function applyDiscount(total: number): number {
    return total > 100 ? total * 0.9 : total // bug: should be >=
}

A test that mocks the function it's supposed to test:

// mocked.test.ts
import { expect, it, vi } from 'vitest'
import { applyDiscount } from './pricing'

vi.mock('./pricing', () => ({ applyDiscount: () => 90 }))

it('gives 10% off at 100', () => {
    expect(applyDiscount(100)).toBe(90) // checks the mock, not the code
})

A test that calculates the expected value with the code's formula:

// same-formula.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

it('applies the discount', () => {
    const total = 100
    // the code's own formula, bug included
    const expected = total > 100 ? total * 0.9 : total
    expect(applyDiscount(total)).toBe(expected)
})

Both pass, and the code still charges 100 for an order of exactly 100. With the mock, the real applyDiscount never runs. Mocking a dependency, like a payment service, is a different and legitimate use (Software Engineering at Google). Copying the formula copies the bug. Google's testing blog shows the same trap in Don't Put Logic in Tests.

Researchers have measured a version of this. In a July 2026 study, 11 models in 13 model and mode setups wrote tests for 318 Java methods behind 233 real bugs. Averaged across those setups, 3.84% of the tests written from the buggy code passed on the bug and failed on the fix. Written from the fixed code, 0.46% did (Zhao, Zhou and Cohen).

More tests don't fix it

In a study first posted in February 2026 on 500 SWE-bench Verified tasks, pushing agents to write more tests barely changed how many they solved (Chen et al.). GPT-5.2 solved 359 tasks either way, and Gemini 3 Pro Preview went from 371 to 366. Telling Kimi K2 Thinking and DeepSeek v3.2 Reasoner not to write tests moved them from 317 to 304 and from 300 to 291. None of these changes was statistically significant.

The cost depended on the model: GPT-5.2 made more API calls and used more tokens without solving more, and the two models told to skip tests used fewer (results). It's one benchmark and one agent scaffold, mini-SWE-agent, and the study didn't test TDD or the workflow below.

Red and green slices, then review

In my experience, TDD with small red and green slices, and reviewers after, works much better:

1. Define what done means, in a way that can fail.
2. Take the expected values from something the agent didn't write: the ticket, the spec, a real example.
3. Write one failing test (red). It has to fail because the behavior is missing.
4. Write only enough code to pass it (green).
5. Next slice. Reviewers come in after green.

For the discount, the test takes its expected value straight from the ticket:

// spec.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

// From the ticket: an order of exactly 100 pays 90.
it('gives 10% off an order of exactly 100', () => {
    expect(applyDiscount(100)).toBe(90)
})

Run against the agent's code, it fails red with expected 100 to be 90. Change > to >= and it goes green. The mock test still passes, because it never ran the code. The formula test now fails: it was guarding the bug. Checked with Vitest 4.1.12: save the four blocks under their file names and run npx vitest run.

Matt Pocock's /tdd skill is his version of this workflow. One slice at a time, and writing all the tests up front is an anti-pattern there. Refactoring belongs to review. That part is a choice: Kent Beck's Canon TDD lets you refactor inside each cycle, after green.

Lauren Tan (poteto), the author of pstack, builds step 1 into it. Its /figure-it-out skill asks the agent to state what done means, in a way that can fail, before it starts the run.

TDD and /figure-it-out can work magic when you understand what you are doing.

What each check covers

Even good unit tests are one row of this chart. Here's what every row covers. The When column is my rough rhythm, not a rule.

The maintained article, with the animated chart, source links and the last-checked date, is on argbe.tech.