"Handled without a human" is the most flattering and least honest number in an agent report. Here is the difference, a transcript-sampling method, and how to report both numbers without lying.
An operator opened a monthly report that said 82% of conversations were "handled without a human." The same week, a 1-star review appeared: the bot had told a customer that next-day installation was available in a zip code the company does not serve. Both facts were true at once. The dashboard number answered the question did a human join the chat. The review answered the question did the customer get what they came for. Only the second one predicts whether the tool keeps its contract.
Why containment flatters itself
Containment means a conversation ended without escalating to a human. That is all it means. It measures the agent's willingness to keep talking, not whether anything correct happened.
Three conversations can all be contained, and they are not the same outcome: one was genuinely resolved; one ended because the customer gave up after two useless replies; one ended with the customer confidently misinformed. The dashboard counts all three as wins, because from the system's point of view nothing required a human — and that is exactly the problem. The third conversation cost more than a human would have.
Containment is also easy to inflate without anyone technically lying: make the "talk to a person" option harder to find, let the bot re-ask its clarifying question one extra time, nudge customers toward a form instead of an agent. Every one of those raises the containment rate and lowers the quality of the containment. A number that rises when you make the product worse is not a metric you can manage by.
Contained versus resolved — the definition that matters
| Term | Definition | What it measures | What it hides |
|---|---|---|---|
| Contained | conversation ended without human escalation | the agent's scope and deflection | correctness, satisfaction, damage done |
| Resolved | the customer's actual question was answered correctly, and the transcript supports it | the outcome | nothing — this is the number that matters |
Resolution cannot be read off a counter. It needs judgment applied to real transcripts, which means it needs a sample. That is the whole method: a small, honest, repeatable sample beats a large, automatic, flattering number.
How to sample transcripts and measure it
Once a week, pull 30 transcripts at random — or all of them if you had fewer. Exclude obvious spam and test messages. Read each one and tag exactly one outcome:
- Resolved — the question was answered correctly, and the transcript shows the customer got it (a confirmation, a booking, a coherent next message).
- Contained, unresolved — no human joined, but the question was not actually answered or was answered wrongly.
- Escalated — handed to a human. This is not a failure; it is the escalation path doing its job.
- Abandoned — the customer stopped responding mid-thread.
Write the tags down every week, in the same file, with the same definitions. A week looks like this:
| Transcript | They asked | What happened | Tag |
|---|---|---|---|
| #104 | Saturday hours | correct answer from the facts file, customer booked | Resolved |
| #107 | price for drain cleaning | agent answered with a retired price, customer went quiet | Contained, unresolved |
| #112 | warranty coverage | offered a callback, owner followed up | Escalated |
| #118 | permit requirements | two vague replies, customer stopped replying | Abandoned |
And the scoring block:
WEEK [x] RESOLUTION SAMPLE — n = 30 random transcripts
Resolved ........................ [ ] /30
Contained, unresolved ........... [ ] /30
Escalated ....................... [ ] /30
Abandoned ....................... [ ] /30
Resolution rate = Resolved / 30 = [ ]%
Containment rate = (Resolved + Contained-unresolved + Abandoned) / 30 = [ ]%
Two rules keep the numbers honest. First, resolution rate is resolved divided by all sampled — no excluding the messy ones. Second, definitions stay fixed. If you redefine "resolved" mid-quarter, every week-over-week comparison you have built becomes decoration.
Three leading indicators that predict trouble before the reviews do
You will not sample every transcript, so you need cheap signals that tell you when to look. These three are visible in any transcript file and they degrade before customers reach a review site.
- Abandonment after the agent's answer. The customer reads a reply and leaves without acknowledging it. Silent exits after answers usually mean the answer did not answer. Track the count weekly; a rise is the earliest warning you get.
- The same question phrased two or three ways in one conversation. When a customer re-asks, they are telling you the first answer did not land. Count conversations containing a near-duplicate question; these transcripts are your highest-value reading, because each one points at a specific facts-file gap.
- "Speak to a human" appearing earlier in conversations. One occurrence per fifty conversations is a working escalation path; the phrase showing up in the first two turns, and climbing week over week, means something in scope or tone recently broke. Find what changed that week.
All three come from text you already have. None of them requires a dashboard you do not.
Reporting the pair honestly to an owner
The report line has both numbers and the sample size, in one sentence:
MONTHLY REPORT — [month]
Contained: 78% of ~900 conversations
Resolution: 24 of 30 random transcripts judged resolved (80%)
Unresolved: 6 — listed below with the fix made for each
The unresolved transcripts, with one line each on the fix, are the most valuable part of the report. They convert the metric into next week's facts-file edits. What you never do is report containment alone: the moment an owner sees one number, the organization optimizes that number, and containment is the one that can be gamed by making help harder to reach.
Four ways teams fool themselves
- Counting abandoned conversations as contained. The customer left. Nothing was handled. This one accounting choice can swing the headline number by double digits.
- Sampling only the good transcripts. Pulling the conversations the platform flags as successful is not a sample; it is an advertisement. Random means random.
- Self-grading without written definitions. If "resolved" means whatever the reader feels that day, the trend line is fiction. Write the four tag definitions down and keep them fixed.
- Reporting monthly averages that bury one bad week. A terrible week after a pricing change averages into invisibility by month's end. Report the four weekly numbers, not just their sum.
[!WARNING]
If your containment rate is high and you cannot produce a resolution sample, assume the gap is full of unresolved conversations, not successful ones. Containment is the number that rises when nobody checks quality. Resolution is the number that rises when quality is real. Only one of them is produced by reading.
What to do today
- [ ] Pull 30 random transcripts from last week and tag them with the four outcomes above.
- [ ] Write your four tag definitions in one paragraph at the top of the tracking file, so next week's you grades the same way.
- [ ] Set a recurring weekly 30-minute block for the sample — it does not survive as an intention.
- [ ] Report both numbers to the owner this month, with the unresolved transcripts and their fixes attached.
Containment tells you what your agent avoided. Resolution tells you what your customers got. Only one of those numbers should ever be allowed near a decision, and it is the one that requires reading.
This guide lives at AgentFast, where you'll also find a free audit tool that checks your own agent's instructions against these failure modes. Posted here with a canonical URL pointing at the original.
