We Scored 17 Real Playwright Test Suites: The State of Playwright Quality in 2026

Every managed QA vendor, including us, claims to write "good" Playwright tests. Almost none of that claim is measurable. So we built a scorer that doesn't take anyone's word for it — an open-source, deterministic, AI-free tool called playwright-score — and pointed it at 17 real, public Playwright suites we don't own, didn't write, and didn't curate for a good outcome.

The result: 1,214 spec files, 5,943 tests, 168,902 lines of test code, and 5,467 individual rule violations, scored on 2026-08-19 against each project's live main/master branch. Every number in this post is reproducible from a single script against the public repos linked throughout — no surveys, no self-reported data, nothing from our own customers.

The headline numbers

• 17 public repos scored
• 5,467 rule violations found
• 30.1% of all locators are raw CSS/XPath
• 12 / 17 repos have a test with zero assertions

The corpus

We didn't pick repos to make the tool — or the industry — look good or bad. The corpus mixes heavily-engineered platforms (Supabase, Grafana, Mattermost, n8n) with smaller, less mature projects found by searching for real @playwright/test usage. Every repo is scored on the exact subdirectory that holds its Playwright suite, via a fresh, shallow, sparse clone — nothing is vendored or cached.

Run on 2026-08-19 with playwright-score sqs-v1, standard profile, threshold 80. Reproduce it yourself with scripts/validate-corpus.sh in the GitHub repo.

What's actually wrong with real Playwright suites

Grades hide the interesting part. We also kept every individual finding — 5,467 of them — and ranked which specific anti-patterns show up most often across the corpus. This is the part that isn't on our own tool's landing page.

Two patterns dominate everything else. Locator anti-patterns account for 4,067 of the 5,467 total findings — 74% of every violation in the corpus — and no-raw-locators alone is 60% of all findings, present in 16 of the 17 repos. This is not a niche mistake. It's the default way most teams still write Playwright selectors, three years after Playwright's own docs started recommending role-based locators over raw CSS.

Zoomed out across every locator call in the corpus — not just findings, every single getByRole/getByTestId/getByText vs. page.locator()/frame.locator() call, 17,118 of them — 69.9% are role-based and 30.1% are raw selectors. Two repos in the corpus, livecodes and openplayerjs, use zero role-based locators: every single locator in both suites is a raw CSS or XPath selector. Both fail the default threshold.

A B-grade suite can hide more findings than an A

cal.com's suite triggers 1,215 individual findings across only 53 files — roughly 23 per file, the highest density in the corpus — and still scores an 85 (B), passing the default threshold. Mattermost, in contrast, has 284 files and 1,157 tests but only 1,476 findings total — about 5 per file — and scores a 90 (A).

The reason isn't that the scorer is lenient on cal.com. It's that playwright-score caps repeated occurrences of the same rule in the same file at 3 for penalty math (every occurrence still shows up in the full findings list — the cap only affects the score itself), so one anti-pattern repeated 40 times in one file doesn't auto-fail a suite the way one anti-pattern spread across 40 different files does. cal.com's actual drag on its score is its locator ratio: 613 role-based locators against 875 raw ones — 41.2% native, well under the corpus average of 69.9%. High finding count and a passing grade aren't a contradiction; they're measuring different things.

Tests that assert nothing

playwright/expect-expect — a test block that runs, exits without an error, and never calls expect() — appeared 102 times across 12 of the 17 repos (71%). These aren't failing tests. They're passing tests that prove nothing, the exact failure mode we've written about before: a green suite that doesn't mean the product works. A scorer that only checked pass/fail would never catch a single one of these 102 cases — they all pass.

Methodology

Every score in this post comes from @qaguardian/playwright-score, scoring version sqs-v1, standard profile, threshold 80 — the same package anyone can npm install -D today. The scorer is fully deterministic and AI-free: it wraps eslint-plugin-playwright for community best practices, then adds a versioned 0–100 score, a locator-ratio metric, and assertion-delegation tracing through local imports. No LLM is in the scoring path.

Each repo was scored via a fresh, shallow, sparse git clone of the exact subdirectory holding its Playwright suite — nothing from any of these repos is vendored into ours. The rule-frequency breakdown in this post comes from the raw JSON findings output of that same run, aggregated by rule across all 17 reports. Full methodology, weights, and the frozen scoring formula are documented in METHODOLOGY.md. Scores will drift slightly over time as these projects' own suites change — that's expected, and the whole point of scoring live code instead of a frozen snapshot.

What to do with this if you maintain a Playwright suite

Run npx -p @qaguardian/playwright-score playwright-score ./tests --format text against your own suite before you assume it's fine. Given that 74% of every finding in this corpus was a locator anti-pattern, that's the highest-value place to look first — a raw page.locator('.btn-primary') breaks the moment a class name changes; getByRole('button', { name: 'Submit' }) survives a restyle. If your suite is CI-gated, the tool exits non-zero below your threshold, so this is a five-minute add to an existing pipeline, not a rewrite.

If you'd rather have engineers own that quality bar for you — the same bar we hold AI-generated Playwright to internally — book a demo and we'll score your actual suite live, on the call.