Once an auto-reply setup has been running for a month, the question stops being "does it reply" and becomes "how do I know what it actually sent." Teams that skip this step find out the same way: a customer screenshots a reply that contradicts the price list, and nobody can say when it went out or which rule produced it.
A confidence score does not answer that question. A review queue does.
Why scores are the wrong instrument
Model-side confidence measures how plausible a candidate answer looked to the model. What you need to audit is something different: whether the message that reached the customer matched the store's current rules. A reply can be fluent, confident, and wrong about stock at the same time. Weekly audit of real output catches that; a score on the generation side does not.
What to capture, per sent message
Four fields, stored where you can query them later:
| Field | Why |
|---|---|
| Inbound text and channel | Lets you reproduce the trigger that fired |
| Rule or knowledge entry matched | The audit trail's most important column |
| Final text actually sent | Not the draft — the sent version, including human edits |
| Operator and timestamp | Who released it, and whether it was inside the active window |
If your tool only keeps the generated text, the audit is already broken: human edits are where most corrections happen, and they are the part you most need to review.
The weekly pass, in order
Thirty minutes a week, fixed sequence:
- Pull the money categories first. Everything touching price, refund, compensation, or shipping time. Read all of them, no sampling. This is the set where a single bad message costs more than the whole week of everything else.
- Sample the rest. Ten to twenty messages across channels and times of day. Random, not cherry-picked.
- Flag three patterns. Answers that contradicted a current rule; answers that were technically right but read as evasive; triggers that fired on messages they should not have. Each pattern gets one line in a fix list.
- Close the loop. Every flagged pattern becomes either a rule edit, a knowledge entry, or a deliberate decision to leave it. "Leave it" is allowed — but it has to be written down, or the same flag comes back next week.
The rewrite log is the real output
The most valuable artifact of the weekly audit is not the report. It is the list of drafts the operator rewrote before sending. Each rewrite marks a gap between what the system would have said and what the store actually means. Collect those for a month and you have a prioritized list of exactly which rules and knowledge entries to write — no guessing about what to automate next.
One rule of thumb: if the same rewrite shows up three weeks running, the fix is not a better prompt. It is a missing rule or a stale knowledge entry.
What good looks like after a quarter
Three trends, not three targets: the count of money-category messages needing a manual correction trends toward zero; the rewrite list shrinks as gaps get closed; and the share of weekly messages that require no human touch rises — but not to one hundred percent. A setup where nothing ever needs review is usually one where the money categories were quietly left in the auto-send path.
