The owner of a home-services franchise did the diligent thing: every Friday, she pulled two random AI conversation transcripts and read them herself. Forty-eight conversations reviewed across six months. She found one typo and one awkward greeting, fixed both, and told me she felt good about her AI oversight.
Then I asked for the conversation count. 3,412 handled conversations in six months. She had reviewed 1.4% of them — and her two-per-week sample was random in the worst sense: it sampled every conversation equally, when the problems live almost entirely in a small, identifiable slice of them.
Here's what she missed, and it took me twenty minutes to find once I had export access:
- 6 conversations where the AI quoted a price from the old rate card (a January pricing update never made it into the knowledge base — every quote was $85 low, and two customers had already paid the wrong invoice).
- 9 conversations where the customer mentioned a competitor, the AI ignored the mention entirely, and the conversation just... ended. No capture, no callback flag. That's nine warm leads deleted without a trace.
- A single conversation where a customer pasted in their credit card number and the AI echoed it back in full, unmasked, in the confirmation message. She found that one four months after it happened.
None of these were findable by reading two random Friday transcripts. All of them were findable with a sampling plan that costs about the same amount of her time per week.
Why uniform sampling is dead weight
Reading transcripts at random is the QA equivalent of testing a smoke detector by smelling the air. The math is unforgiving: if a failure mode affects 2% of conversations, a weekly sample of two gives you roughly a 9% chance of catching it in a year. You will run your AI for years and never see your most persistent defect.
Meanwhile the failures concentrate. Across the deployments I audit, a small number of conversation types carry a wildly disproportionate share of the risk:
| Conversation type | Share of volume | Share of serious defects |
|---|---|---|
| Contains a price, quote, or discount | ~11% | ~40% |
| Mentions a competitor | ~4% | ~15% |
| Contains payment or identity data | ~3% | ~20% |
| Ends angry or goes silent mid-thread | ~7% | ~15% |
| Escalated to a human | ~6% | mixed — but where the handoff fails, it fails here |
| Everything else | ~69% | ~10% |
The conclusion is not "read everything else." It's "read the first five categories almost entirely, and sample the rest only for drift."
The tiered sampling plan
Tier the review by risk, not by randomness. This is the plan I install on every deployment:
Tier 1 — read 100%: any conversation containing a number that could become money (price, quote, discount, warranty term), any mention of a competitor, any message containing data patterns (card numbers, partial SSNs, addresses that look like homes when the business is commercial). This is where the franchise owner's $85 pricing error, the nine dead competitor leads, and the card-number echo all lived. At typical volumes this is 15–25% of conversations — very readable if the AI pre-flags them.
Tier 2 — read 100% of the first 40, then 20% forever: escalated conversations, and every conversation where the customer expressed frustration. Handoffs are where AI deployments fail most quietly, because the failure is an interface problem: the human receives a transcript with no summary, or worse, a summary the AI wrote that's wrong. The first 40 conversations teach you your handoff failure modes; after that, a 20% sample catches regressions.
Tier 3 — sample 5–10% weekly: everything else. This tier exists to detect drift, not defects — the day the AI's tone shifts because someone updated the system prompt, or a knowledge base edit subtly broke a flow. You don't need depth here; you need a tripwire.
The pre-flagging is the part humans can't do and the AI does trivially. A single nightly pass with a classifier prompt — "does this conversation contain a number that looks like money, a competitor name, data patterns, or an abrupt negative ending?" — turns a random 2-a-week review into a targeted one. The franchise owner now reviews about 30 conversations a week, all high-risk, in about the same time she used to spend on two random ones.
The five flags the classifier must catch
If you build the flagging pass, don't let it get vague. These are the five flags that earned their keep across audits:
- MONEY — the conversation contains a price, quote, discount, refund amount, or warranty term. Highest-yield flag in every deployment.
- COMPETITOR — a named competitor appears anywhere. The correct behavior on flag is a captured lead with a callback reason, and almost no AI does this unprompted.
- PII-DATA — card numbers, government IDs, credentials. The failure is usually echoing the data back, which turns a customer mistake into a stored liability.
- NEGATIVE-END — the conversation ends angry, or the customer goes silent after an AI message that should have invited a reply. Silence after a question is often a defection, not a resolution.
- HANDOFF — escalated to a human. Reviewed for summary quality: does the human get a two-line brief, or a 4,000-word dump?
The prompt that runs the pass
This is the classifier prompt, trimmed to the part that matters — paste your transcripts in batches of twenty against it:
For each conversation below, output a JSON line:
{"id": "...", "flags": [...], "note": "..."}
Flags (any combination):
MONEY - contains any price, quote, discount, refund, or
warranty term; put the exact figure in note
COMPETITOR - a competitor is named; put the name and the
customer's stated reason in note
PII-DATA - card number, government ID, or credential appears;
state whether the AI echoed it back in a reply
NEGATIVE-END - last customer message is angry, OR the customer
went silent after an AI question
HANDOFF - conversation was escalated; grade the human-facing
summary GOOD if a new human could act on it in
under 60 seconds, BAD otherwise
Be conservative: if unsure, flag it. Flagging extra is free;
missing MONEY is not.
The "conservative" instruction is doing real work. In one deployment's first month, the classifier over-flagged by about 2x, which sounds bad until you account for what it caught in the noise: three conversations where the AI had offered a discount nobody authorized, in exactly the over-talkative way the flag exists to catch.
The Monday review table
The review output goes in one table, same five rows every week:
Week of: ____
Tier 1 conversations flagged: ____ of ____ total
MONEY issues (wrong/old price): ____
COMPETITOR leads captured: ____ → called back: ____
PII echoes (AI repeated data): ____
Tier 2 handoff reviews: ____
Summaries graded BAD: ____ → common failure: ____
Tier 3 drift sample read: ____ → tone/flow issues: ____
The single number to watch over time is MONEY issues per 100 Tier-1 conversations. It should start high (pricing and rate-card errors are the most common first-month finding) and drive toward zero as you fix the knowledge base. If it spikes, something changed — a rate card edit, a new service tier, a promo that ended. The spike is your change-management signal, arriving weeks before a customer complains about it.
At the franchise, six weeks of this produced: the January rate card finally fixed (the $85 error class went to zero), a competitor-capture rule added to the system prompt (nine recovered leads in the first month, one a $4,200 job), an echo-suppression patch on PII, and a handoff summary template graded GOOD on 94% of escalations, up from 40%.
What to do with a finding
Two failure types, two different fixes — and mixing them up is the most common workflow error I see:
- A defect (wrong price, echoed card number, ignored competitor): fix the system, then write a regression test. The fix goes into the knowledge base, the rate card sync, the system prompt — wherever the defect originates. The regression test is one synthetic conversation that must produce the correct behavior before any future prompt or document change ships.
- A drift (tone shifted, a flow broke after an edit): find the change, revert or correct it, and log it in the change journal. Drift findings are cheap to fix and expensive to ignore, because drift compounds.
And one rule with no exceptions: every PII echo found gets checked for where it was stored, not just said. An echo in a message is bad; the same number sitting in a conversation log on a vendor's servers for 400 days is the actual incident.
The 12-line card
[ ] Tiered sampling, not uniform: 100% / 20% / 5-10% by risk
[ ] Tier 1 = money, competitor, PII — read every one
[ ] Nightly classifier pass pre-flags; humans review, not hunt
[ ] Classifier errs toward over-flagging
[ ] MONEY per 100 Tier-1 conversations tracked weekly
[ ] COMPETITOR mentions become captured leads with callbacks
[ ] PII echoes checked for storage, not just the message
[ ] Handoff summaries graded GOOD/BAD against a 60-second rule
[ ] Defects get a system fix + one regression test each
[ ] Drift gets a change-journal entry, not a prompt tweak
[ ] Review happens Monday, findings logged same day
[ ] One owner; total review time under 2 hours/week
If you install only one piece of this, install the nightly flagging pass. It converts QA from a ritual that proves nothing into a filter that makes two hours of weekly review cover more risk than the full-time equivalent of random reading.
The full classifier prompt pack, the tier rules, the Monday table and the regression-test template are in the kit I run on deployments: Agent Audit Pack
