Every sprint adds tests. Almost no sprint deletes one.
That one-way flow has a predictable end. The suite gets slower. Flaky tests
(tests that fail randomly) pile up. Engineers stop trusting red builds.
Eventually someone suggests rewriting everything, and the cycle restarts.
There is a cheaper fix. Audit the suite with five questions.
On the last suite I inherited, they retired a third of the tests.
Nothing those tests "guarded" ever broke.
Question 1: When did this test last fail for a real reason?
Your CI history (the record of server test runs) already knows.
A test that has been green for a year has two possible explanations.
Either the code it guards never changed, or the test cannot detect change.
Both are worth knowing. Only one deserves compute on every commit.
Practical rule: pull the last 90 days of runs. Tag every test that failed
only for environment reasons, or never failed at all. Those are audit candidates.
Question 2: What user risk does it guard?
Write the risk in one sentence, naming a user.
"It tests the profile page" is not a risk.
"A user loses saved work when the session expires" is a risk.
If nobody on the team can produce that sentence, the test is guarding
an implementation detail, not a user. Details change on purpose all the time.
Tests guarding them fail on purpose all the time. That is where flakiness lives.
Question 3: Would anyone notice if it vanished?
Run the experiment instead of debating it:
# in a branch: skip the suspect test, run everything else
npx playwright test --grep-invert "@museum-candidate"
If coverage of the named risk survives through other tests, and no gap
appears in the risk list from Question 2, the test was furniture.
Delete it in the branch. Keep the branch open a week. Merge with confidence.
Question 4: Does it check results or steps?
A test that clicks through checkout and asserts the button was clickable
is a tour, not a test.
// a tour: asserts the step happened
await page.getByRole('button', { name: 'Pay' }).click();
await expect(page).toHaveURL(/confirmation/);
// a test: asserts the outcome is real
await expect(page.getByTestId('invoice-total')).toHaveText('$34.20');
await expect(page.getByTestId('invoice-number')).not.toBeEmpty();
The URL can change while the invoice ships blank. Assert on the thing
the user came for.
Question 5: Can it fail at all?
Flip the condition and run it.
// original
await expect(status).toBe('paid');
// flipped: this MUST fail. If it passes, the test is dead.
await expect(status).not.toBe('paid');
A test that passes both ways asserts nothing. AI-written tests fail this
check more than any other kind, because generators optimize for green.
Sixty seconds per suspicious test. The flip never lies.
What you get back
Five questions, one afternoon, on the oldest third of your suite.
The suite you keep is faster, and every test in it can answer
"what breaks if I go red?" That is what makes a red build mean something.
And when an AI agent starts writing tests into your suite, this audit
is the contract you hold its work against. A generated test enters
only if it survives the same five questions.
Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.
