As a QA director at a Japanese IT company, I spent the last months building E2E test suites with an unusual pair programmer: Claude Code, Anthropic's AI coding agent. Not for a toy project — for production QA pipelines.
Here's an honest breakdown of what AI is genuinely good at in test automation, where it fails, and the workflow that ended up sticking.
TL;DR
- Claude Code is excellent at generating Playwright test skeletons from plain-language flow descriptions — roughly 70% of the boilerplate work disappears.
- It is terrible (without guardrails) at selector strategy and flaky-wait hygiene. You need conventions it must follow.
- The sweet spot: AI writes, human stabilizes. My rule: every AI-generated test gets run 5× locally and once in CI before it counts as "done".
The setup
Nothing exotic:
your-app-e2e/
├── playwright.config.ts # 3 projects: desktop / mobile / foldable
├── tests/
│ ├── pages/ # Page Object Model
│ │ ├── ContactPage.ts
│ │ └── DashboardPage.ts
│ └── e2e/
│ ├── contact-form.spec.ts
│ └── dashboard.spec.ts
└── ci/github-actions-e2e.yml
The trick is in what you tell Claude Code before asking for tests. I keep a CLAUDE.md in the repo with three hard rules:
# Test conventions (non-negotiable)
1. Selectors: getByRole / getByLabel / [name=] attributes only.
NEVER CSS class chains.
2. No waitForTimeout. Playwright auto-waiting + expect() polling only.
3. Every test must pass in isolation AND in parallel (no shared state).
With these three lines, the quality of generated tests jumps dramatically. Without them, you get page.locator('.css-1kxonj9 > div:nth-child(3)') and waitForTimeout(3000) — the stuff flaky suites are made of.
What AI-generated tests look like (real example)
I asked Claude Code: "Write tests for the contact form: success, server error, pending state. Never hit the real API."
It produced a Page Object with network stubbing:
/** Stub the contact API so tests never send real notifications. */
async mockContactApi(status: number, delayMs = 0): Promise<void> {
await this.page.route("**/api/contact", async (route) => {
if (delayMs > 0) {
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
await route.fulfill({
status,
contentType: "application/json",
body:
status === 200
? JSON.stringify({ ok: true })
: JSON.stringify({ error: "Submission failed." }),
});
});
}
And a pending-state test I honestly wouldn't have bothered writing by hand:
test("disables the submit button while the request is in flight", async () => {
// 1.5s of artificial latency makes the pending state observable.
await contact.mockContactApi(200, 1500);
await contact.fillValidForm();
await contact.submit();
await expect(contact.submitButton).toBeDisabled();
await expect(contact.successHeading).toBeVisible({ timeout: 5000 });
});
That last test caught a real bug class I see constantly in manual QA: double submission. The button only disabled after a React re-render — two fast clicks under slow network = two POST requests. On a contact form that's an annoyance; on a checkout it's a double charge.
Where the AI fails (and how I patched it)
1. It trusts the DOM too much. Generated selectors matched what exists today, including auto-generated class names. Fix: the conventions file above, plus review of every selector during stabilization.
2. It doesn't know what's worth testing. Ask for "tests for the dashboard" and you get 15 shallow assertions about text being visible. The test plan still has to come from a human who understands risk: payment flows, auth edge cases, state transitions. I write the plan; AI writes the code.
3. It can't feel flakiness. An AI-generated test that passes once isn't done. My stabilization loop: run 5× locally (--repeat-each=5), once in CI, review traces for silent retries. About 1 in 10 generated tests needs a rework at this stage.
An underrated trick: test the viewports nobody tests
My config includes a project most suites never have:
{
// Foldable half-open posture — where responsive bugs hide.
name: "foldable-half-open",
use: { viewport: { width: 717, height: 1172 }, isMobile: true, hasTouch: true },
}
717px sits between Tailwind's sm and md breakpoints. I own a foldable (Motorola Razr), and the number of layout bugs living exactly in that gap is remarkable. Costs one config block, catches a bug class your users on premium Android devices hit daily.
Results after a month
- Test authoring time: roughly −60%. The plan takes as long as before; the code doesn't.
- Coverage went up, because tedious-to-write tests (pending states, error paths, viewport edge cases) stopped being skipped.
- Maintenance stayed manageable — but only because of the selector conventions. I've seen AI-generated suites without conventions, and they rot in weeks.
The honest summary: Claude Code didn't replace QA judgment. It replaced QA typing.
I'm a QA director in Japan (ex-SHIFT, Japan's largest QA firm) and I build Playwright + Claude Code test suites as a freelance service — async, text-first, from JST. If your team wants a suite like this, you can find me on Fiverr or see more at s-nagaya.vercel.app. Questions about the workflow are welcome in the comments — happy to share the conventions file.
