Agentic Test Creation: From Plain-Language Requirements to End-to-End Test Cases

Throughout my career, I've watched sprints stall in the same place. Everything was going fine: the features were merged, the requirements were clear. But then we hit a roadblock: waiting on test cases. Because the QA team was hand-converting Jira stories into steps and expected results. 

Most test automation strategy conversations skip right past this authoring bottleneck. I've done the conversion work myself. It's not hard work, but it's necessary, slow, and repetitive. Worst of all, it's disconnected from the actual skills that made anyone want to work in quality engineering in the first place.

In my previous article, I drew a line between two architectures that both get sold as "AI test generation." One is simply a large language model behind a prompt template. The other is an actual agent that reads your requirements, your attachments, and your existing test libraries before it writes anything. 

Agentic test creation means an AI agent reads your requirements, attachments, and existing test library, then drafts structured test cases traced to each acceptance criterion, with a human review gate before anything enters the suite.

That article included a 6-step example of what an agentic pipeline does with a promo code user story. The result: 7 cases reused, 6 generated, 1 of which was rejected at review.

This time I want to expand that example into a full walkthrough, drawn from my own experience. The example isn't special. These are the same plain-language requirements your team probably already writes. But the process, and outcome, are interesting.

Starting with a Plain-Language Requirement

When I say plain-language requirements, I mean the artifacts an agile team already produces. User stories. Acceptance criteria, whether they're bullet points or structured Gherkin scenarios. The Jira ticket itself, with its comments and revision history. Even the wireframe or screenshot somebody attached during refinement. None of it is specifically written for an AI. But the good news is that none of it needs to be.

That's the premise behind AI test case generation from user stories: the story is the spec, and its criteria are the test conditions. It's also what shift left testing looks like in practice. If the requirement itself is the test input, test design starts the moment the story is written.

The Agent Loop vs an LLM Call

I covered the agent loop in detail last time, so I'll just give an overview here. 

A typical and single LLM call is a stateless function. You call, you get an answer. You move on to the next call. 

But an agent is a loop: it reasons about a goal, calls tools to gather information, observes the results, and revises its plan before producing output. 

That's the reasoning-and-acting loop the ReAct paper formalized (Anthropic's Building Effective AI Agents is my recommended primer). Applied to test creation, the agent loop parses the story and its criteria, pulls the linked test cases, reads the attachments, and builds a coverage map—all before generating any steps.

The Agentic Test Case Walkthrough

Here is our requirement:

As a returning customer, I want to apply a promo code at checkout so that my discount is reflected in the order total.

The acceptance criteria attached to the story (let's call it PROMO-214):

1. A valid promo code reduces the order total and shows a discount line item.

2. An invalid or expired code shows an inline error and leaves the total unchanged.

3. A promo code and a gift card can be applied to the same order; the promo discount is applied first.

4. Percentage discounts round half up to the nearest cent at the order level, not per item.

5. Removing a code restores the original total.

Plus there's an attachment: checkout-mockup.png, showing the promo field, the discount line, and the gift card entry point. That's our entire input package. Here's what it looks like:

What the agent surfaces before generating

Before writing anything, the agent first checks the test cases linked to the checkout area. And it finds 40! It maps the story's scenarios against them and proposes 7 for reuse: valid code, invalid code, expired code, empty field, case sensitivity, code removal, and re-application after removal. 

In a generic tool, those 7 would come back as duplicates. Here they arrive as reuse suggestions with their existing case IDs, so their regression history stays attached.

The output

Our agent now generates 6 new cases against actual gaps. Here's one:

Let's look at what the agent did with acceptance criterion 3.

The criterion states an ordering rule: promo first, then gift card. The generated steps verify the amounts in that order, with the expected results as the actual arithmetic. That arithmetic came from reading the criteria together: 20% off $79.99 is $15.998, which becomes $16.00 because criterion 4 rounds at the order level. Rounded per item, the same discount would be $15.99. 

The rate, the ordering rule, and the rounding rule all interact.

A second generated case, TC-1208, targets the expired-code path in a checkout that already has a saved payment method on file. It renders like this:

The remaining 4 cases cover rounding at the half-cent boundary, a gift card that exceeds the discounted total, code removal after a gift card is applied, and discount persistence across a session timeout. 

None of these are uncommon ... but they are very typical of the type of cases we might skip when running out of time in a sprint.

Just as telling is what the agent didn't generate. Nothing in the batch covers 2 browser tabs applying codes to the same cart, and nothing checks whether the inline error is announced to a screen reader ... because no criterion mentions either. The agent's coverage tracks what's written down—and when it strays past that, the review gate catches it.

Deciding what should have been written down is still your job.

One objection you might raise: these are structured test cases, not automated end-to-end scripts. 

That's true. But the structure is the point. A case with discrete steps and explicit expected results is an automatable artifact. Whether a person executes TC-1207, an automation engineer scripts it, or an execution agent picks it up downstream, the hard part is already done: what to verify, in what order, and with what data. 

Prose test ideas can't make that handoff. But steps with expected results can.

Your Output Is Only as Good as Your Input

Next, let's take a look at the limits of this approach. Like most solutions, agentic test cases have some best practices that should be followed.

First, vague acceptance criteria produce vague steps. If criterion 4 hadn't specified order-level rounding, the agent would have had to infer a rounding rule, and a reviewer would have needed to confirm it.

What does a testable criterion look like?

So what does a testable criterion look like? Compare "discounts should work with gift cards" to criterion 3 above: a promo code and a gift card can be applied to the same order, and the promo discount is applied first. 

The first version tells the agent a feature exists. The second gives it an ordering rule it can verify with arithmetic. A few habits close this gap: give ordering rules explicitly, specify rounding and limits, name the expected error behavior, and attach your mockups. 

None of this is new, right? Testers have been pushing for the best practices for years. The agent just makes it more important.

Second, a thin or messy test library weakens the reuse step. An agent can only propose reusing cases it can find!

And finally, treat the review gate as mandatory. In the original run of this example, our reviewer edited 2 cases and rejected 1. In my previous article I recommended tracking the reviewer rejection rate; I'll refine that here: track 2 numbers separately, the rejection rate and the edit rate. 

Rejection rates going up usually means the context feeding the agent is broken. Maybe it's a thin test library, or stories that aren't linked to their cases, or even criteria that are silent on a whole scenario. 

Edit counts going up usually means your acceptance criteria are vague.

Rejected Test Cases

Let's pause for a moment to look back at our rejected case from earlier. We didn't dig much into that. 

Agentic failures sometimes don't look like failures. In this example, our reviewer rejected a session-timeout case. It had clean steps, specific expected results, and was formatted exactly like the other 5. But the problem was that we gave no acceptance criterion on what happens to a discount when a session expires. The agent inferred a behavior and then tested its own inference. 

That's the pattern you should use to train reviewers: the most dangerous output isn't the sloppy case, it's the really good looking case ... that just happens to verify a requirement that no one wrote.

The good news is that the rejection itself was useful. It made its way back to the product owner as a requirements question, which is precisely the kind of gap that previously wouldn't surface until production.

Implementing Agentic Test Creation

Finally, let's look at options for implementing agentic test creation. As is often the case, you can build or you can buy.

Buy. Vendors have started building true agentic solutions. For example, Agentic Test Creation in Tricentis qTest is one implementation of this pattern. It runs the loop inside the test management platform itself, so the reuse suggestions and the review gate land where the test cases already live.

Build. You can also build this yourself. The building blocks are all there: the ReAct loop, an LLM that calls your tools, and community-maintained open-source connectors like the MCP Atlassian server that lets an agent read Jira stories directly. A motivated platform team can assemble this pipeline themselves.

The benefit to buying? The plumbing. Existing test libraries, traceability links, and review workflows are already wired together.

This can matter more than you might think. Reuse detection is only as good as the agent's view of what exists, and a platform that already holds your test library already has that view. Same story for the review gate: it's a workflow with roles, permissions, and an audit trail, which is exactly the kind of thing that's boring to build and easy to underestimate.

The benefit to building, of course, is flexibility. You pick the model, you own the prompts, you can encode your team's house style for test cases, and you can wire in internal systems no vendor will support. 

But be sure you budget honestly. Writing an agent that drafts steps from a story might be a weekend prototype. But the review workflow, the permissions, and keeping the prompts current as your requirements evolve are real work. That you now own.

How to Evaluate Agentic Test Creation Tools

Whichever route you take, I'd ask these 3 questions:

1. Does the solution read everything attached to the requirement, including images?

2. Does the solution propose reuse before it generates?

3. Does every machine-written case pass through a review gate before entering the project?

A test automation strategy is ultimately a decision about where humans create the most value. When structured test cases can be drafted from the requirements your team already writes, the hours that went into transcription can instead move to the work only humans can do: exploratory testing, risk analysis, and deciding what should be tested in the first place.

Revisiting Your Test Automation Strategy

My readers may recall my personal mission statement, which I feel can apply to any IT professional:

"Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else." — J. Vester

Your team already writes user stories and acceptance criteria. Those artifacts are enough to drive end-to-end test creation, provided the system reading them knows what already exists. Transcribing them into test steps by hand never really added much value. But a context-aware agent drafted from them just might.

Have a really great day!