Five Versions of an AI Testing Tool: What I Tried, What It Cost, Where It Broke

I wanted automated health checks for an e-commerce storefront. Standard stuff: search works, product page renders, cart accepts items, checkout doesn't explode.

But I had one requirement that made it non-standard: tests should be written in plain natural language. Not code, not a DSL, not record-and-replay. A text file that says "search for a drill, open the first result, add it to the cart, verify the cart shows 1 item." That's the whole test.

Why insist on this? Two reasons. The person writing the scenario shouldn't need to maintain Playwright selectors. And selectors are where E2E testing goes to die — every UI tweak breaks them, and maintenance quietly becomes more expensive than the bugs the tests catch.

I went through five versions before this project ended. The path was not a straight line — at one point I went backwards on purpose. Here's the whole thing, dead ends included.

v0: LLM Drives the Browser

The obvious answer in the current landscape: put an LLM in the execution loop. Libraries like browser-use wrap Playwright and let a model drive the browser directly from a natural language instruction.

I set it up in Docker on a dev server. It works — genuinely. The model reads the instruction, looks at the page, clicks the right things.

Then I looked at the bill. A few runs cost me actual dollars, and the API rate limits kicked in almost immediately. And I wanted these tests on a cron, many times a day, forever. On top of the cost: latency per step was seconds, and the same test could pass or fail depending on how the model interpreted the page that day.

The realization that killed this approach: an LLM interpreting your test on every execution is renting intelligence for a task you only need intelligence for once. The scenario doesn't change between runs. The page (usually) doesn't change between runs. Why pay a model to re-figure out the same clicks every hour?

Conclusion: the LLM belongs at generation time. Generate a normal Playwright test once, run it for free forever. Every version after this is a different answer to one question: how does the model learn what's on the page?

v1: Bookmarklet — the Human Points at Things

First answer: the human tells it. A bookmarklet on the target page let me click elements with the mouse; the references got collected alongside my plain-text steps and sent to a small Flask backend, where the model assembled a Playwright test. Around it: a /run endpoint, pytest, cron scheduling, screenshot reports, a little web UI with a code editor for fixups.

It worked, and generation was cheap (~600 tokens — the human had already done the element-finding). But it had a structural flaw: the page was loaded in a way that didn't execute JavaScript properly, so anything rendered client-side was invisible. And clicking through every element of every scenario by hand is exactly the kind of manual labor I was trying to remove.

v2: The Accessibility Tree — the Browser Points at Things

Then I found the actual answer sitting inside the browser the whole time: the accessibility tree. It's the page as assistive technology sees it — roles, names, states, structure — with all the div soup gone. Playwright exposes it as a snapshot after full JS rendering.

The numbers made the decision for me: a real product page is ~2MB of HTML — tens of thousands of tokens of noise. The accessibility snapshot of the same page is ~2KB of JSON, 500–1000 tokens, and it contains exactly the things a test interacts with: buttons, links, inputs, their labels.

So v2: you provide a URL and a plain-text scenario, the server loads the page headless, snapshots the tree, and a small cheap model (Haiku) writes the test from scenario + tree. No clicking, no bookmarklet, JS rendering solved. Cost per generated test: a fraction of a cent.

v2.5: Step-by-Step Generation in a Live Browser

Generating a whole multi-step test from one snapshot has an obvious hole: the page changes as the test progresses. The tree of the search page tells you nothing about the cart page.

So the generation loop became interactive. A persistent browser session over CDP, and an agent (Claude Code CLI as the orchestrator) working step by step: snapshot the current page → generate code for one step → execute it in the live browser → snapshot the new state → generate the next step. Like a human writing a test with the browser open next to the editor — except each step is verified against reality the moment it's written. At the end, the steps get glued into one test file, pytest confirms it passes, and it's saved.

This is also where self-healing fell out almost for free. Each test is a folder with three files:

tests/add-to-cart/
├── scenario.txt   # URL on line 1, then plain-language steps
├── tree.json      # accessibility tree snapshot at generation time
└── test.py        # generated Playwright code

That tree.json snapshot is a change detector. Before a run, grab the current tree, diff it against the stored one. No diff → the page hasn't changed → run the existing test as-is, $0. Diff → the UI moved → regenerate against the new tree, save the new snapshot. The model only gets paid at the exact moments its work is needed: first generation and healing. Execution lived in GitHub Actions on a schedule; a separate Dockerized report service collected screenshots and pytest output.

v4: Going Backwards on Purpose

Here's the zigzag. The accessibility tree has a weakness: sometimes the element you need simply isn't in it — custom widgets, canvas, badly-built markup. The v2 fallback was "paste selectors into the test by hand," which is the old maintenance problem sneaking back in through the window.

So v4 returned to the human pointing at things — but properly this time: a Chrome extension. You write the scenario in a single textarea, click elements on the live page, and the extension inserts each element's full XPath inline into your text: Click the login button [/html/body/div[2]/form/button]. State survives page reloads via chrome.storage; submit sends text + XPaths to the server, the model writes the test.

Human intent in plain language, exact element identity captured mechanically, LLM as the translator between them. Precise where v2 was fuzzy — and manual where v2 was automatic. Neither version dominated the other; they traded the same problem back and forth.

The Economics (Why I Kept Going)

What kept me pushing through versions was the market gap. AI-testing SaaS — Momentic, Mabl, testRigor and friends — charges roughly $300–2000+/month for this exact promise: tests in plain language, self-healing on UI changes. Enterprise players run to tens of thousands per year, and your tests live in their proprietary format.

My version: $0 per execution, pennies per UI change, output is standard Playwright code that I own, running on a free CI tier. The gap isn't magic on their side — it's architecture. They keep a model (or a platform) in the loop; I paid for intelligence only twice per test lifetime.

Where It Landed

While deciding whether to develop this further, I found that Playwright itself now ships built-in test agents (planner / generator / healer, since v1.56) covering the same generate-once, heal-on-change cycle natively — driven, naturally, by the accessibility tree. A vendor-maintained implementation of the same idea beats a personal framework on every axis that matters: docs, ecosystem, someone else fixing the bugs.

So I stopped. Kept the glue that's specific to my setup — scheduling, reporting, screenshots. The scenario files survived; the framework didn't.

No regrets about the detour. Five versions bought me something reading can't: I know why the accessibility tree is the right input, why the LLM belongs at generation time, why healing must be diff-triggered — because every alternative personally cost me money, time, or both. That understanding transfers. The code didn't need to.