The metric caused the experience
Most chatbot dashboards lead with deflection rate: the share of conversations that ended without a human.
Read that definition again from the customer's side. Every handover counts as a failure. Every refusal to hand over counts as a win. A customer who gave up and closed the window scores identically to a customer who was helped.
A system optimised against that number learns to keep people circling through restated questions and suggested articles. This is not a subtle misalignment. It is the precise mechanism behind the experience people are describing when they say they hate chatbots, and it was designed in on purpose, by whoever chose the metric.
Three other things that were probably true
| What went wrong | What it looked like to the customer | Fixable by |
|---|---|---|
| Scoped to answer everything | Confident wrong answers on edge cases | Defining what it may answer |
| No access to your systems | Paraphrasing the FAQ page | A read path into orders, stock, billing, calendar |
| Nobody read the transcripts | The same failure every week for a year | One person, one hour, weekly |
The second is the one that quietly determines everything. A support automation connected to nothing can only restate published content, so it competes with your own search box and loses, because the customer read that page before opening the chat.
The questions that generate contacts are specific and personal. Where is my order. Is this still available. Why was I charged this. Can I move my appointment. Answering any of them needs a read path into a real system, and building that path is most of the actual work.
A proposal that skips it is quoting for a wrapper around a knowledge base. The demo will look excellent, because demos ask general questions.
What a working version does differently
It answers only what it can verify, and it says where the answer came from.
In a brokerage deployment we ran, the automation answered what the listing itself could answer: floor, area, price, what is included, whether the property is still available. Everything else went to a named agent with the conversation attached. Response time went from 6 hours to 8 minutes, and the reason it worked is that the automation never guessed.
Anything binding goes to a person by design. Pricing outside the published rate, terms, commitments. That boundary is the subject of our rules for keeping a person in the loop, and it is not a limitation we apologise for: an automation that agrees something on your behalf at 2am is a liability rather than a feature.
Handover is the whole product
Three things go wrong here and all three are cheap to fix.
The escape hatch is hidden, so the customer has to guess a magic phrase to reach a person. Say in the first message that a human is available.
The handover arrives as a bare alert, so the agent opens by asking what the customer has already explained twice. Carry the transcript across, or the automation has cost time rather than saved it.
The queue behind the handover is not staffed for what the bot escalates, so a fast refusal becomes a long silence. That is a capacity decision, and it has to be made before launch rather than discovered in week two.
Route on the first sign of frustration, not the third. The cost of an unnecessary handover is a few minutes of an agent's time. The cost of a refused one is the customer.
When we say do not build it
More often in this category than in any other, and usually for one of three reasons.
The contacts are mostly things a bot cannot verify. The volume is too low for anyone to maintain it. Or the real problem is that human response is slow, and support automation would be a decoration over that.
The third case deserves naming because it is common. If enquiries wait four hours because nobody is there at seven in the evening, a bot that says something friendly and unhelpful at seven in the evening has not fixed the wait, it has automated it. The money is better spent on routing, on coverage, or on removing the reason people are contacting you at all.
That is the same test as the fifth readiness condition: if the bottleneck is not here, making this part faster changes nothing anyone can bank.
What to ask the next vendor
Which systems will it read from, and what will it do when that read fails.
What is it measured on, and if the answer is deflection rate, what happens to the number when it hands over correctly.
How does a customer reach a person, in how many messages, and who is waiting when they arrive.
And ask to see a transcript from a real deployment on a bad day. The order-processing split between rules and model is what a defensible answer to the first question looks like: deterministic questions answered by rules, unstructured input read by a model, and the two never swapped.
