A flaky test is one that passes and fails on the same code with no change in between. It's the most corrosive thing in a test suite — not because any single failure is scary, but because flakiness teaches your team the worst possible habit: ignoring red.
Once "just re-run it" becomes the reflex, your suite stops being a safety net and becomes a slot machine. Here's a practical field guide to the usual culprits and how to actually kill them — built from years of chasing these in real automation suites.
Flakiness is never truly random
It feels random, but it almost always traces to one of a handful of non-determinisms. The job is to find which one.
1. Timing / async (the #1 cause in UI and integration tests)
The test checks for something before the app is ready. It passes when the machine is fast, fails when CI is loaded.
-
Fix: never sleep for a fixed time. Wait for a condition — an element state, a network response, a value. Modern frameworks (Playwright, Cypress) auto-wait on assertions; lean on that instead of
sleep(2). -
Smell to grep for:
sleep,setTimeout, hard-coded waits.
2. Test order dependence / shared state
Test B passes only because Test A ran first and left data behind. Run them in a different order (or in parallel) and B falls over.
-
Fix: each test sets up and tears down its own state. No reliance on execution order. Tools like
pytest-randomlyoriDFlakiessurface these by shuffling order.
3. Shared external resources
A real database, a shared test account, a live third-party API. Two runs collide, or the API has a bad day.
-
Fix: isolate data per test (unique keys), and stub the network at the boundary (
cy.intercept, WireMock, VCR). Test your logic, not someone's uptime.
4. Time, timezones, randomness, locale
Tests that use now(), random seeds, or locale-dependent formatting fail at midnight, on a CI box in another timezone, or on the one day the random value hits an edge.
-
Fix: inject the clock and the seed. Freeze time (
freezegun,sinon), pin the seed, pin the locale.
5. Resource leaks and ordering in the app itself
Sometimes the test is honest and the code is flaky — an unawaited promise, a race in the app, a dictionary iteration order assumption. The test just exposed it.
- Fix: this one's a gift. The flaky test found a real concurrency bug. Fix the app, not the test.
A triage process that works
- Quarantine, don't ignore. Move the flaky test to a separate suite that doesn't block the pipeline — but tracks it. The goal is to stop the "ignore red" habit, not to hide the problem.
-
Reproduce deterministically. Run it 50 times locally (
pytest --count=50, or loop it). If it never fails locally but fails in CI, the difference is the clue — load, parallelism, environment. - Bisect the cause. Run it in isolation vs. with the suite (order dependence?). Disable the network (external dependency?). Freeze the clock (time?).
- Fix the root cause, then de-quarantine. And add the reason to your commit message, so the next person learns the pattern.
Measure it, don't vibe it
Track a flaky rate per test (failures ÷ runs over the last N runs). It turns "the suite feels flaky" into a number you can drive down, and it tells you which tests to fix first. A test that's failed 8% of the last 200 runs is a priority; one that failed once six weeks ago is noise.
The mindset
Every flaky test is a small lie your suite tells. The fix isn't more retries — retries just make the lie quieter. The fix is removing the non-determinism, one category at a time, until green means green again.
What's the flakiest test you've ever had to hunt down — and what turned out to be the cause? I collect these war stories.
I'm Krishnasai Rollamadugu — a test automation architect who writes about test reliability and trust in AI-driven testing.
