At 03:07, the page says that answers over the logistics support archive have started citing the same incident repeatedly. The on-call opens one trace and sees three PDF tickets about a delayed container, all admitted as separate evidence even though two are near copies created by an email export. The least complex useful response is a similarity gate with three outcomes: accept a clearly new ticket, suppress a clear duplicate, and send the uncertain middle to review. Do not turn one similarity threshold into an automatic delete button.
TL;DR: normalize and embed each ticket with the same pipeline, search only within an operationally meaningful candidate set, and treat similarity as a ranking signal rather than truth. Keep both sides of every decision, the model and preprocessing version, the score, the chosen boundary, and the reviewer outcome. Page on user-visible retrieval damage or sustained decision drift, not on a lone high score.
This matters before generation. Retrieval-augmented generation combines retrieved external material with generation; if duplicated tickets occupy several retrieval slots, the evidence presented downstream is narrower than the result count implies. The pager should therefore fire on the consequence: repeated evidence, missing distinct incidents, or a review queue that can no longer meet its service objective.
Quiet is not healthy.
What should have fired before the answer repeated itself?
The earlier signal is a change in the gate's decisions, segmented by the dimensions that can alter ticket language: carrier, route, document template, language, and ingestion version. A global average can stay calm while one newly deployed PDF template strips spaces or repeats headers, moving an entire cohort across the review boundary. Dashboards are weak evidence here. I want the page to name the affected cohort, the first bad deployment or corpus window, and a few decision identifiers that can be replayed without exposing ticket text.
A useful event is compact but reconstructable. Record a content hash for exact-copy handling, stable identifiers for the candidate pair, the embedding and normalizer versions, similarity, gate outcome, and timestamp. Keep the raw PDFs under the archive's existing access and retention controls; telemetry does not need their contents. Exact hashes catch byte-identical exports cheaply, while vector similarity handles paraphrases and formatting changes. Those are different mechanisms and should have separate counters.
Track the distribution, not merely the mean. Counts entering new, review, and duplicate, plus reviewer reversals for each outcome, expose a threshold that has drifted. A rise in duplicate decisions may be legitimate after a bulk import. The same rise paired with reviewers repeatedly splitting those pairs is evidence of bad suppression.
How should Go dedupe near-duplicate support tickets for review?
Cosine similarity for nonzero vectors is the dot product divided by the product of their magnitudes. Higher scores mean the representations point in a more similar direction; they do not prove that two tickets describe the same operational event. Two reports can share every phrase about a delayed shipment yet refer to different containers, days, or customers. Conversely, an update may add a resolution while still belonging to the original incident.
No score proves identity.
The gate below uses explicit lower and upper boundaries. It assumes the vectors were produced by one pinned embedding pipeline and rejects dimension mismatches and zero vectors instead of manufacturing a score. The policy compares a new ticket only with candidates selected by metadata such as tenant and a bounded event window; comparing every shipment with the entire archive creates plausible but operationally irrelevant neighbors.
package dedupe
import (
"errors"
"math"
)
type Decision string
const (
New Decision = "new"
Review Decision = "review"
Duplicate Decision = "duplicate"
)
type Policy struct {
ReviewAt float64
DuplicateAt float64
}
func Cosine(a, b []float64) (float64, error) {
if len(a) == 0 || len(a) != len(b) {
return 0, errors.New("vectors must have equal, nonzero dimensions")
}
var dot, aa, bb float64
for i := range a {
dot += a[i] * b[i]
aa += a[i] * a[i]
bb += b[i] * b[i]
}
if aa == 0 || bb == 0 {
return 0, errors.New("cosine is undefined for a zero vector")
}
return dot / (math.Sqrt(aa) * math.Sqrt(bb)), nil
}
func (p Policy) Decide(score float64) (Decision, error) {
if p.ReviewAt < -1 || p.DuplicateAt > 1 || p.ReviewAt >= p.DuplicateAt {
return "", errors.New("invalid policy boundaries")
}
switch {
case score >= p.DuplicateAt:
return Duplicate, nil
case score >= p.ReviewAt:
return Review, nil
default:
return New, nil
}
}
There are deliberately no universal numbers in that policy. A copied subject line, boilerplate carrier disclaimer, or OCR artifact can change the score distribution, as can any change to chunking or embeddings. Boundaries belong in versioned configuration and must come from labeled pairs that resemble current traffic. The upper boundary should reflect the cost of hiding a distinct logistics incident; the lower one should reflect the cost and capacity of human review.
This approach has limits. It is not suitable as the sole decision mechanism when regulations or contractual controls require a human to preserve every report, when the archive is too small to produce representative labeled pairs, or when exact business identifiers already determine identity without ambiguity. In those cases, deterministic matching or mandatory review is the better boundary. The trade-off is plain: a wider review band protects recall but spends analyst time and delays indexing, while a narrow band lowers latency by accepting more classification risk.
Do not compare isolated chunks and then discard the document relationship. A PDF header can be nearly identical across thousands of tickets. Build the decision from ticket-level evidence, preserve identifiers such as shipment or case references as structured features, and give reviewers both complete documents with the matching passages highlighted. A vector match proposes a case; an auditable workflow decides it.
Calibrate with incidents, not convenient examples
Start with labeled pairs sampled from the same path production will use. Include exact exports, paraphrased reports, follow-up messages, shared templates, OCR damage, and hard negatives: similar wording attached to distinct shipment identifiers. Split calibration and evaluation by time or incident group so revisions of one ticket do not leak into both sides and make the gate look cleaner than it is.
Then sweep candidate boundaries and count operational errors. A false duplicate can remove the only evidence of a separate lost shipment. A false new decision consumes index space and may crowd retrieval results. A review decision costs analyst attention and adds latency. Those costs are asymmetric, so a single aggregate accuracy number cannot choose the policy.
Use a replay set when the normalizer, PDF extraction, chunk construction, or embedding version changes. Run the proposed pipeline beside the current one, compare transitions among all three outcomes, and inspect the pairs that cross a boundary. Deploy the decision version with the indexed vectors; mixing new queries with old vectors produces a test you did not design. If either pipeline cannot reproduce its own stored decision from the retained inputs and version metadata, rollback and replay are guesses.
The review interface needs two distinct actions: confirm the same incident, or keep both and record why. That reason can become a future evaluation slice, for example same_template_different_container. Reviewers also need an escape hatch for linked but nonduplicate tickets, because a follow-up report may be valuable retrieval evidence even when it belongs to the same incident family.
Keep both.
Instrument the action path
After the noisy page, add instrumentation where a decision changes the corpus, not only where similarity is computed. The trace should connect PDF ingestion, exact-hash lookup, candidate selection, vector scoring, gate outcome, reviewer action, index mutation, and the retrieval result that later feeds an answer. One correlation identifier across those stages lets the on-call move backward from a repeated citation to the exact gate decision.
The service should fail closed around irreversible actions. A vector timeout can route a ticket to retry or review; it should not silently label the ticket new or duplicate. Invalid vectors and mismatched versions are errors. Queue age deserves an alert when it threatens the promised review latency, while individual review-band matches are normal work and should not wake anyone.
A compact operational record might look like this:
type GateEvent struct {
DecisionID string
IncomingTicketID string
CandidateID string
ContentHashMatch bool
NormalizerVersion string
EmbeddingVersion string
Similarity float64
Outcome Decision
ReviewedOutcome Decision
}
Keep metric labels bounded. Ticket IDs belong in trace events or protected logs, not metric dimensions. Metrics can safely aggregate by policy version, outcome, source format, and a controlled cohort vocabulary. The page then links to sampled decision IDs, while the chart remains usable under load.
The decisive checks are end to end: the proportion of retrieved results occupied by one duplicate family, distinct incident coverage in a labeled query set, reviewer reversal rates, and review age. Generation quality cannot repair missing evidence. If deduplication suppresses a distinct ticket before retrieval, the answerer never sees it.
The threshold can create its own incident
Move the upper boundary downward and apparent duplicate removal rises, but so does the chance of hiding separate events that happen to use the same logistics template. Move it upward and reviewers or retrieval results absorb more copies. Widen the middle band and decision quality may improve while latency grows. There is no free setting.
For rollout, shadow the new policy, inspect boundary crossings, then canary it on a defined cohort with a rollback condition tied to reviewer reversals and retrieval coverage. Preserve suppressed documents rather than deleting them, and make the index mutation idempotent so replaying a decision does not create another copy. The runbook should say which decision version to disable, how to drain pending reviews, and how to restore suppressed ticket references.
The false-positive cost deserves the final word. An aggressive threshold can make a quiet dashboard by erasing the disagreement that should have reached a human, and in logistics that disagreement may be the difference between two similar complaints and two separate shipments needing action. A useful page fires when the system is losing distinct evidence or cannot process uncertainty in time. Everything else is a graph to inspect after breakfast.
Pages need consequences.
Further reading
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
