Which browser MCP is fastest. Playwright MCP vs Chrome DevTools MCP vs Agent Browser. Token economics — MCP-driven workflows burning roughly 4x the tokens of a CLI-skill pattern for the same coverage. Selector stability. Tool-call counts. Whether the agent survives bot detection.
Every one of those benchmarks measures the same half of the loop: can the agent drive the browser.
That half is close to solved. The half that decides whether any of this is worth shipping is the verdict -> did the agent look at the resulting page and get the call right. Almost nobody is benchmarking that, and the numbers we do have are ugly.
The verdict is the weak link, and there's evidence
Three data points.
Judges disagree with humans about a third of the time.
BrowserArena found GPT-4o, used as a judge over agent traces, matched human evaluation roughly 68% of the time. Not the agent's ability to act. Its ability to assess what happened.The ground truth itself is frequently broken.
OpenAI audited SWE-bench Verified and found 59.4% of the audited problems had flawed tests. Models were being graded against wrong answers. OpenAI stopped reporting the score.Models game graders when graders are gameable.
METR reported o3 and Claude 3.7 Sonnet reward-hacking in over 30% of evaluation runs, monkey-patching graders, manipulating scores, rather than solving the task.
Put those together and the shape of the problem is clear. We built agents that can reach any pixel on the page and then asked those same agents to decide whether the page was correct. Actuation improved fast because it's mechanical and easy to measure. Judgment didn't, because it isn't.
The benchmark environments hide this. WebArena and its descendants run clean apps — no cookie dialogs, no interstitials, no network jitter. Real-world evidence-gathering is the messy part, and the clean harness systematically overstates how robust the whole loop is.
I've been on both sides of this. Before O2, DevAssure was a no-code test automation platform built for QA engineers. Thirty-plus customers. It worked, and the thing that eventually broke wasn't test creation But, it was that nobody could tell a real pass from a hollow one at a glance.
In April 2025 we pivoted, and the rule we rebuilt around was one line: the author cannot be the examiner.
we had the same problem in 2016 with a human writing the code and the tests in the same sprint written by the same human, and the fix wasn't better tooling, it was separation.
Four constraints that actually separate author from examiner
1. The judging context must not contain the diff.
If the same context window holds both the change and the question "did this work," you've built a self-grader. It will rationalise. Judge against intent that was written down before the change existed, not against the change.
2. Assertions come from the requirement, not the rendered DOM.
This is the one that silently kills suites. Point an agent at a page and ask it what to assert, and it will assert whatever it finds. Broken state included. A green run then just proves the page still looks the way it looked when the agent last read it. That isn't a test. It's a snapshot with opinions.
3. Failure evidence has to survive without the agent.
Trace, video, network log, deterministic replay. If the only artifact of a failure is the agent's own prose summary, you have a witness and no record. A developer at 11pm cannot debug a paragraph.
4. Run a deliberate-break drill.
This one takes an afternoon and tells you more than any benchmark. Break something on purpose in a build — remove a validation, invert a boolean, drop a required field. Run your agent suite against it.
If it goes green, your coverage is just a showcase. If it goes red, read the failure message and ask whether it points at the actual break or somewhere three steps downstream.
What I'd tell you if you're building this yourself
Stop optimizing the tool-call count. It's the cheapest part of the system to fix and the least likely to be what hurts you in production.
Instead ask,
when my agent says pass, what is that claim actually grounded in?
If the answer is "the agent looked at the page and felt okay about it," you don't have a test suite. You have a very expensive smoke check with excellent vocabulary.
The trust gap in AI-generated code doesn't close by generating more tests. It closes by making verification independent of generation. That's an architecture decision, and most teams are making it by accident right now.
I'm building O2 at DevAssure — a PR-native testing agent that reads the diff and verifies against stated intent, in CI, in a real browser. The separation above is the whole design.
If you've run the deliberate-break drill on your own agent setup, I want to hear what happened. Especially if it went green.
