Most teams don't fix flakiness. They add a retry, watch the suite go green, and move on — and six months later nobody trusts a red build, because half of them are noise and telling which half is a coin flip.
The reason it never converges isn't a lack of effort. It's that "flaky" names five completely different problems with completely different fixes, and teams debug whichever one they thought of first.
Here's an order that terminates.
Step 0: measure, don't remember
You cannot triage from impressions. Before anything else, get one number per test: failure rate over the last 50 runs on unchanged code.
If you don't have that, the cheapest way to get it is a nightly job that runs the suite against main five times and appends results to a CSV. A week gets you 35 data points per test, which is enough to rank.
Two things fall out immediately and both are worth the week:
- Usually fewer than 10% of tests produce more than half the flaky failures. Fix those five tests and the suite feels transformed. Without the data you'd have spent that time on whichever test annoyed someone loudly.
- Some "flaky" tests aren't flaky at all — they fail 100% of the time on one device or one OS version and pass everywhere else. That's not flakiness, it's an unfiled bug wearing a disguise, and it's the highest-value thing in the list.
Rank by failure_rate × runs_per_day. Work top-down. Stop reading Slack for candidates.
Step 1: is it the test, or is it the app?
Before assuming a test bug, take the top offender and run it 20 times in isolation on one device.
- Fails at a similar rate in isolation → it's the test, or a genuine app bug. Continue to step 2.
- Passes 20/20 in isolation, fails in the suite → it's interference from the tests around it. Skip to step 3.
This one question splits the work in half, takes twenty minutes, and is skipped almost universally.
Step 2: the sync problem (most flakiness lives here)
Nearly all single-test flakiness is a race between your test and the app's rendering, and nearly all of it comes from the same three patterns.
Sleeping instead of waiting. sleep(2) is a bet that the app is ready in under two seconds on every device, every network, every CI load. It passes on your M-series laptop and fails on a mid-range Android under thermal load. Every fixed sleep is a flaky test with a delay fuse.
Waiting for presence, acting on interactivity. The element exists in the tree well before it's ready. A button that's rendered but still animating in eats the tap and reports success — the tap landed, it just didn't do anything, and you fail three assertions later on a screen that never changed. Wait for enabled and stable, not merely present.
Waiting for the wrong thing. The classic: waiting for a spinner to appear, when on a fast response it appears and vanishes before the poll interval catches it, and your wait times out on a screen that already loaded successfully.
// Flaky: passes when the spinner is slow, times out when the app is fast
wait.until(visibilityOfElementLocated(By.id("loading_spinner")));
wait.until(invisibilityOfElementLocated(By.id("loading_spinner")));
// Stable: wait for the post-condition, not the transition
wait.until(elementToBeClickable(By.id("results_list")));
The general rule: assert on the state you want, never on the transition into it. Transitions are racy by definition; end states are not.
One more that only shows up on real hardware — animations. A screen that has finished loading but is still sliding in will hand your tap to whatever pixel is under the finger mid-flight. Disabling system animations on the device (appium:disableWindowAnimation, or the developer options scale settings) removes an entire class of failure and speeds the suite up as a bonus.
Step 3: state leakage (the one that scales with your suite)
If the test passes alone and fails in the pack, something outside it is changing what it sees.
Work through the shared surfaces in this order — roughly most to least common:
- Device state. Cached logins, granted permissions, dismissed onboarding, a notification still on screen, keyboard left open, app left backgrounded. Test 40 inherits whatever test 39 left behind.
- Backend state. The account whose cart another test just emptied; the record another test renamed; the feature flag someone flipped.
- Shared credentials. One login used by parallel shards, where a new session invalidates the old one and some unrelated test gets bounced to the login screen.
- Time and clock. A test that passes except around midnight, or on the last day of a month, or in a CI runner on UTC when the assertion assumed local time.
- Data accumulation. Test 1 creates an item, so test 12's "list should show 3 items" assertion has been quietly counting up all week.
The fix is always the same shape and it's rarely "clean up afterwards": make each test create what it asserts on. Teardown-based cleanup fails exactly when the test fails — which is when the leakage matters most — so the next test inherits the mess from the run that already went wrong, and one real failure cascades into six fake ones.
Step 4: environment (rule out before rewriting anything)
If steps 2 and 3 found nothing, stop editing the test and look at where it ran.
- Device health. Low battery throttles CPU. Full storage makes installs and screenshots fail in creative ways. Thermal throttling after an hour of continuous runs slows everything enough to blow timeouts that were fine at run 1.
- Network. A staging API with a p99 of eight seconds will produce flakiness no test-side fix can remove. Log the request timings; if the p99 moves with the failures, the fix is in the backend, not the suite.
- Rate limits. Parallel shards multiply your request rate. A 429 usually surfaces in the app as a generic error screen, so it reads as an app bug.
- OS-level interruptions. An OS update banner, a "storage almost full" alert, a carrier message landing on top of your flow. Rarer than people assume, but genuinely random when it happens — which makes it the most confusing category to debug from a stack trace alone.
This is also where inconsistent devices show up as inconsistent tests. If two runs of the same test get two differently-configured handsets — different locale, different OS patch level, different leftover state — the test is being blamed for the environment's variance. Running against a device that's dedicated and stable between runs doesn't fix a badly written test, but it does remove the variable, which is what lets you conclude anything at all from step 1.
Step 5: it's a real bug
Some flaky tests are correct. They're catching a genuine race in the app — a request that sometimes resolves after the screen is gone, a cache that's occasionally stale, a listener registered twice.
The tell: the failure is not at a findElement, the screenshot shows a plausible screen, and the app behaves wrongly rather than the test looking in the wrong place. Deleting or retrying this test is deleting a bug report that reproduces.
What to capture so triage is possible at all
Most of the above is impossible from a stack trace. On every failure, capture:
- A screenshot at the moment of failure — separates "wrong screen" from "right screen, bad selector" instantly.
- The element tree at that moment — tells you what locators actually existed, which is what you need to write the fix.
- Video of the run — the only way to catch a dialog that appeared and disappeared, or a tap that landed mid-animation.
- Device logs and network requests — turns "the app showed an error" into "the API returned 429."
- Which device, OS version, and build — so the "fails only on one device" pattern can surface at all.
Without these you re-run locally to reproduce, which for genuinely flaky tests is the single biggest time sink in the process — you're trying to reproduce something that happens 1 in 8 times, and every attempt costs a full suite setup.
On retries
Retries are a measurement tool, not a fix. Configure them so that a test that passes on retry is still reported distinctly from one that passed first time — most runners support this, and it's the difference between a retry policy and a cover-up.
Then treat the retry-pass count as your flakiness metric, and hold it to a budget. A test that needs retries is on a clock, not on a list.
The quarantine rule
Quarantine works only with an expiry date. Without one it's a graveyard: tests nobody runs, nobody deletes, and nobody trusts.
A policy that survives contact with a real team:
- A test over your flakiness budget gets quarantined — still runs, doesn't block the build.
- It gets an owner and a two-week deadline, both written down.
- At two weeks it is either fixed or deleted.
Deleting is a legitimate outcome. A test nobody trusts and nobody fixes has negative value: it costs runtime, it costs triage attention, and it trains everyone to ignore red.
Run your suite on dedicated devices with video, logs and the element tree on every failure →
Originally published at robotactions.com/blog/flaky-mobile-tests-triage.
