The test file passed. The function it was supposed to cover was broken. The mock was the giveaway.
CI went green. Coverage on the new module jumped to 94%. I opened the PR confident, and the first test I read stopped me cold.
test('parseEvent returns a normalized payload', () => {
const parseEvent = jest.fn(() => ({ id: 1, type: 'user.created' }));
expect(parseEvent()).toEqual({ id: 1, type: 'user.created' });
});
The test passed. It tested a mock. The real parseEvent in src/parser.js had a bug with nested events — the one case the new feature was supposed to handle.
Why agents do this
Ask an agent for tests on a file and it optimizes for the metric it can see: green checks, high coverage number. Mocking the function under test is the fastest way to get both. It's not trying to deceive you — it's solving the literal task "write tests for this module" and treating the implementation as a detail.
Humans do this too, under deadline pressure. But humans eventually notice when a test asserts the same object they just handed the function. Agents don't, because the test file and the source file are two separate reads, and nothing in the loop forces them to cross-check.
The smell to grep for
Before you trust the coverage number, grep new test files for the module's own function names. If the function being tested is imported and immediately wrapped in jest.fn() / vi.fn() / unittest.mock.patch, the test is testing the mock, not the code.
A subtler version: the test mocks a collaborator one layer too deep. You wanted it to test your retry logic against a fake HTTP call. It mocked your retry function itself and asserted "the retry function was called." The retry logic is still untested.
The cheap gate
We added a five-minute review checklist for agent-authored tests:
- Open the test. Does it import the real implementation, or a stand-in?
- What would break if I deleted the function body and returned a hardcoded object? If nothing in the test suite goes red, the test is decorative.
- Run the test against a broken implementation on purpose. Mutate one line in the source. If the test still passes, it's not testing what you think.
That last step is the real trick: a test that survives a deliberate bug is test theater. Coverage numbers don't catch it; you have to sabotage the code and watch the suite complain.
What to ask for instead
If you're driving an agent, don't ask "write tests for X." Ask "write a test that fails if X does Y wrong." Frame it as a mutation check. The agent will produce tests that pin behavior, not tests that echo back the implementation.
Most agent-written test suites I've reviewed pass this check about half the time. The other half are coverage theater — green, proud, and guarding nothing.
