If you've ever recorded a UI test, shipped one button change to production, and watched 40 tests explode in CI — you know exactly why "record and replay" has a bad reputation.
I've spent the past year building a browser testing tool, and I want to talk about the unglamorous engineering that decides whether a recorded test survives a redesign — or dies on first contact. Not the AI magic. The four problems underneath it.
1. A single locator is a single point of failure
Most recorders store exactly one locator per element — an XPath or a CSS chain — and freeze it at record time. That's the root of almost every "my tests broke" story:
- XPath like
//div[2]/main/section[3]/button[1]breaks when someone adds one<div>to the layout. - CSS chains like
.v-btn.theme--dark > .v-btn__contentbreak on any styling refactor. - Even
data-testid— the community's favorite answer — isn't bulletproof. Third-party components don't have it, and a cleanup sprint that renames IDs silently kills dozens of tests.
The fix: don't store one answer, store a ranked list of candidates.
{
"action": "click",
"target": "Add to cart button",
"candidates": [
{ "strategy": "test-id", "value": "add-to-cart" },
{ "strategy": "role-text", "value": "button 'Add to cart'" },
{ "strategy": "text", "value": "Add to cart" },
{ "strategy": "css", "value": ".btn-primary.cart-action" },
{ "strategy": "xpath", "value": "//button[contains(., 'Add to cart')]" }
]
}
At replay time, try the strongest match first and fall through on a miss. Then — and this matters more than people think — record which candidate matched. If your test passed via the XPath fallback, the page has changed in a way that deserves human review, even though the run is green.
Ranking rules that survived contact with real apps:
-
Semantic attributes first (
data-testid,aria-label,name) — most stable, but often missing. - Role + accessible name — survives styling changes.
- Text content — human-meaningful; great for buttons and links, useless for inputs, fragile under copy edits.
- CSS chains and index-based XPath last — useful signals, never a primary answer.
No single strategy wins. The goal is that a test degrades gracefully instead of snapping.
2. Synthetic events are not real input
element.click() from JavaScript is not a click.
It invokes the event handlers, but it skips the browser's native input pipeline: no focus management, no :active state, no scroll-into-view, different behavior with native controls like <select>, date pickers, and file inputs. Tests pass on synthetic events and fail for real users — or the reverse.
The sturdier path is driving input at the browser level. The Chrome DevTools Protocol dispatches events through the same pipeline a real mouse and keyboard use:
await cdp.send('Input.dispatchMouseEvent', {
type: 'mousePressed', x, y, button: 'left', clickCount: 1,
});
await cdp.send('Input.dispatchMouseEvent', {
type: 'mouseReleased', x, y, button: 'left', clickCount: 1,
});
For typing, Input.insertText behaves much closer to a human than setting .value and firing an input event.
The trade-off: browser-level input is stricter — and that's the point. If a cookie banner covers your button, a CDP click fails, correctly, because a human couldn't click it either. Synthetic events would have "passed" while hiding a real bug. The price is that you must handle overlays deliberately instead of pretending they don't exist.
3. Don't fail at the first miss — and don't hide the miss
When the primary path can't locate an element, naive tools do one of two dumb things: fail instantly (flaky suite), or silently fall back (the test drifts away from what it was testing). Both destroy trust in the suite.
The distinction that matters is why the element wasn't found:
- Not there yet — hydration, lazy loading, an animation still running. Wait and retry with a deadline.
- There, but covered — modal, toast, cookie banner. This is a real finding; surface it, don't wait it out forever.
- There, but different — the element moved or re-rendered. Try the next candidate, and log that you did.
So the failure ladder looks like:
CDP locate (primary candidates, ranked)
→ retry within deadline # "not there yet"
→ DOM-level fallback # CDP hit-test missed
→ next candidate in the list # element changed
→ fail, with full evidence # never silently
Every fallback gets logged and shown in the run report. A green run that used three fallbacks is not the same as a green run that didn't — and your team should see the difference.
4. The expensive part isn't running tests. It's triaging failures.
Do the math on flaky tests: one ambiguous failure costs a QA engineer 15–30 minutes of "is this a real bug or is it the test?" Twenty failures a day is a person-day of triage, most of it wasted on non-bugs.
So the most valuable feature of a testing tool isn't execution speed — it's the quality of evidence attached to a failure:
- screenshot at the exact failing step
- the step list up to the failure, with which locator each step matched
- the candidate list the failed step tried
- console errors and page context (URL, viewport, browser version)
We use AI to summarize the likely cause, but the recorded steps stay the source of truth. That's deliberate: an AI that silently "fixes" tests is just moving the drift somewhere you can't see it.
What record-and-replay is not for
Honest limits, because trust beats hype:
- It won't replace unit tests or API tests. UI tests are for user-visible behavior.
- It's weakest for canvas-heavy visualizations and deeply random data flows.
- Best fit: critical-path regression — checkout, onboarding, admin workflows. The longer and more business-critical the flow, the more replay stability pays off.
That's the core of what I've learned building CueCast — a no-code tool built around these ideas: multi-candidate matching, browser-level input, and evidence-first failures. The techniques above work in Playwright or Selenium too; steal them either way.
How do you handle locator brittleness? data-testid everywhere? Playwright's getByRole? Visual AI matching? Curious what's actually holding up at your scale.
