My search engine scored 65% on its own docs site

A couple of weeks ago I launched chops-search with a blog post claiming 92% recall@1. That number is real and it still stands: 24 labelled queries, my blog corpus, methodology in the post. Then I wired the engine into its own docs site, added a CI gate to block deploys on eval failures, and ran the eval honestly for the first time.

65%.

This is part one of three on what happened next. It ends at a plateau, on purpose: recall@1 stuck in the mid-70s on a fixture that kept growing, every ranking knob swept flat or regressive, and a hypothesis about rank fusion waiting to be tested. Part two is what the test did to the hypothesis.

The claim this part earns: every change that moved recall changed what evidence existed for the ranker to see. Every knob that merely reweighted existing evidence went flat or backfired. I built four ranking levers during this arc, measured all four, and shipped all four inert. That sentence would have read as an indictment to me a month ago. It is the most useful thing this corpus taught me. (Part two complicates it, with receipts. Evidence changes still win, but reweighting knobs eventually earn their values, one inside part two's arc and one at its very close, and the rule for telling those cases apart from noise is part two's story.)

The scoreboard

The query set changed twice during the arc (23 cases grown to 46, plus a gate/known-failures split), so no single percentage pair is an honest before/after. Every checkpoint with its own denominator:

Read the passing column, not the percentage column, and read the last two rows together, because the number goes down on a better engine. What happened there is the floor: rows three through five were measured at a hand-raised min_cos 0.34, and the last row put it back to its derived default. Some green under that raised floor was measurement debt, passes that only existed at a configuration nobody ships, and the honest configuration collected it. The full audit of that debt is part two. The rule it taught fits in one line here: a gate that measures a configuration nobody runs is not a gate, it is a scoreboard.

Every number was measured at dims = 128. The rows pin to commits: 047ebd8 (baseline), 588e4e3 (diagnosis pass), 7e3e280 (the BM25F row on its own fixtures, and the grown-fixture row against the fixture from 2205069), fc1e235 (descriptions), and 42593bf (honest configuration; the fusion sweep at that commit reproduces this row as its self-check cell). Rows one, two, and six are one git checkout from reproducible; rows three through five additionally pass --min-cos 0.34, the floor they were measured under, and rows four and five point eval at the fixture from 2205069, since the grown fixture landed after the engine work in commit order.

Why a docs site is the hostile corpus

The setup: 19 pages, Diátaxis structure, 132 chunks, 1,592 keyword terms at the BM25F commit (1,607 by the descriptions commit and 1,629 at the plateau; the count moved with docs edits during the arc). CI builds the CLI from workspace HEAD, builds the index, and gates deploys on the eval. Dogfooding with teeth.

A docs site about one tool is topically homogeneous, and homogeneity hurts both halves of a hybrid engine at once:

Keyword side: vocabulary collapse. Words like "search", "index", "query", "build" appear on every one of the 19 pages. Their IDF is near zero. BM25 has nothing to discriminate with, because the vocabulary that describes the tool is the vocabulary of every page about the tool. The extreme case: "chops" has document frequency 19 of 19. On this corpus, the project's own name is a stopword.

Semantic side: elevated noise floor. Every page is "about chops-search", so the cosine between a query and the wrong pages is uniformly higher than on a diverse corpus. Off-topic queries were clearing the relevance floor with confident nonsense.

The engine was calibrated on my blog: mixed topics, distinctive vocabulary per post, exactly the corpus where both halves work as designed. The docs corpus is the opposite in every dimension that matters. Not an excuse for 65%, but the variable the whole post turns on.

Diagnose before touching any knob

The first move was explicitly not tuning. The ladder, in order:

1. Read the per-kind recall table. The red row tells you which subsystem is failing. Exact and navigational failing points at the keyword side. Paraphrase failing points at the semantic side. Negative controls failing means the relevance floor is too permissive for this corpus.

2. Compare recall@3 to recall@1. If @3 is high while @1 is low, the right answers are on the podium and you have a tie-breaking problem. If @3 is also low, retrieval itself is failing, and no threshold sweep will save you. Those are fixed in completely different places.

3. Audit every failure as a possible labeling question first. The docs-specific trap: many queries have two legitimate answers, a flag documented in both a how-to and the CLI reference. expect is a list and the check is contains, so both URLs belong in it. I had a handful of cases where the engine returned the reference page and I had labelled the guide page. The engine was right. My labels were wrong.

4. Only then touch the engine.

One eval UX gotcha that fed a false sense of failure: the URL column in per-case rows shows the top-1 result, not the expectation. On PASS rows they coincide, so you never notice. On failures it shows what beat you. Several of my "failures" were the engine returning a page that legitimately answered.

The audit also surfaced two real bugs. Hyphenated identifiers like data-chops-open were split into parts, and with "chops" a corpus-wide stopword, compound-identifier queries were ranked on noise. Fix: emit the whole token alongside its parts. And stopword-only queries were entering rank fusion with junk keyword lists that still vote. Fix: a keyword confidence floor (kw_floor) that suppresses the list when matched IDF mass falls below a threshold fraction of potential mass, with the trailing word exempt from the denominator when it produced prefix expansions, so mid-typing stays live while completed junk gets gated.

I remembered that pass as a seventeen-point jump. The committed history says otherwise, and the correction is the most on-theme finding in this post: at any held-constant configuration, the engine side of the diagnosis pass was net minus one case. It won exactly the case it targeted, data-chops-open flipping top3 to PASS in the per-case diff, the compound-token fix visibly doing its job, and it paid two collateral cases. Everything else I had attributed to the pass lived in fixture repair (the label widening is worth one case in every configuration), in content fixes that had not been committed yet, and in a relevance floor that would not exist for another two days. The scoreboard above is the committed version, and the diagnosis ladder below is still the right procedure; what it produced first was better labels and better docs, not a better ranker, which is precisely this post's thesis wearing its own clothes.

The fixture split. I'll be straight about the motive: I wanted CI green, and a set of diagnosed engine issues stood between me and green. Quarantining failures to pass a gate is the classic way to lie to yourself, so the split only works with discipline attached. fixtures/queries.toml is the contract the engine meets today, gating CI. fixtures/known-failures.toml holds annotated cases reproducing open issues, each with a diagnosis and candidate fix in a comment, runnable manually, not gating. The rule is written into the file header: when a fix lands, promote its case back in the same PR, and never "fix" a case by blessing whatever the current noise winner is in expect. A gate you tuned to green on a small set is a gate that lies. This one has a paper trail instead. (Part two shows the paper trail doing something I did not expect: predicting its own future, twice.)

The work that moved the number

Three changes account for most of the movement in this part, and all three share a property: they put evidence in front of the ranker that it previously could not see.

Field weights were doing the wrong thing, confidently

The keyword engine had supported field weights from early on. A title mention counted like two body mentions, a tag like four, and the builder implemented that literally: it multiplied the term frequency before writing the posting.

That is wrong in a way that is easy to miss, because the direction is right. BM25 saturates term frequency against k1 (1.2 by default). The saturation curve is what stops a document that says "caching" forty times from scoring forty times higher than one that says it once. Pre-multiplying tf pushes a term straight past the interesting part of that curve: a single tag occurrence entered scoring as tf 4, already deep into saturation, so the distinction between "tagged with this" and "tagged with this and about it" disappeared. Worse, the inflated tf also inflated the document's computed length, so weighting one field quietly penalized every other term on the page.

BM25F is the standard answer and it inverts the order of operations. Each field carries its own term frequency and its own average length. Each field's tf is normalized by that field's length, the normalized values combine under the weights, and saturation applies once to the result:

ntf_f  = tf_f / (1 - b + b * dl_f / avg_f)
tfw    = w_title * ntf_title + w_tag * ntf_tag + w_desc * ntf_desc + ntf_body
score += idf * tfw * (k1 + 1) / (tfw + k1)

The practical difference: a term in a five-word title is now scored against the average title, not against two thousand words of body. That is the entire reason a query like "artifacts" can pick the page titled Artifacts out of five pages that use the word in prose. By the next checkpoint, navigational recall measured 100%, and it stayed there through the plateau.

Two implementation notes that turned into rules. The posting record widened twice on the way here: per-field term frequencies took it from four bytes (doc, tf) to eight, and the description field below takes it to ten. Each widening forces a format version bump, and the committed history reads v1 to v3 to v4; the mid-session v2 never landed, a small sibling of the phantom 44. The bump is not for tidiness, but because a reader one version behind would parse the newer file into plausible garbage rather than failing, since every field is a u16 and nothing about the byte stream announces its own shape. And the weights moved from build time to query time, persisted in index.bin, so they can be swept against an existing index with eval --w-title instead of a rebuild per value, and the browser scores with whatever the site configured. That introduced a new way to be wrong: any caller that builds a ScoreOpts::default() instead of starting from engine.score_opts() silently discards the index's own weights. That is now a test.

The description experiment

Nineteen pages, every one with a hand-written description in its front matter, none of them indexed. The descriptions read exactly like the questions a searcher asks: "rebuilds the search index whenever content changes", "why chops-search degrades to keyword-only". Three of four failing paraphrase cases had their answer sitting in an unindexed field. The obvious move was to index it, and the obvious place was the semantic side, since descriptions are natural language rather than rare identifiers.

The obvious place was wrong. Five configurations, same 46 cases, all at --min-cos 0.34:

Every row is measured on the 46-case fixture. The keyword rows are --w-desc 0, 1, and 2 at fc1e235 (descriptions reach the keyword engine as a BM25F field there, which is this table's "keyword side"); the two placement rows are one-commit experiment branches atop fc1e235, pinned at bea4a0b (chunk 0) and 6e93675 (own chunk). The "tf counted twice" run reproduced the "keyword field only" per-case lines byte for byte. One case separates this table's baseline (34) from scoreboard row four (35), and the commits explain it: row four is 7e3e280, before the description field existed, while this baseline is fc1e235 with the field present but disarmed, and the format and chunking changes between the two commits carry the case.

Every configuration that put description text into an embedding lost exact cases: 13 of 14 with descriptions on the keyword side alone, 11 or 12 in every cell with an embedding placement. And the recall@1 column says something sharper than "keyword won": pure embedding placement lands below doing nothing (33 cases against the baseline's 34), and stacking an embedding placement on top of the keyword field only gives back cases the keyword field had won (35 and 36 against keyword-only's 37). The embedding contribution is negative in every cell it appears in.

The mechanism is short synthetic chunks. An embedding is the mean of its token vectors, so a twenty-word chunk is dominated by those twenty words in a way a six-hundred-word chunk never is. Title plus tags is fine there: eight words that name the page produce a vector pointed at the page. A description is summary prose, and across a documentation corpus every summary is written in the same register. "The headers file for Cloudflare and Netlify, the CSP directives, and the three gotchas." "Fetch, lock, and verify the model2vec model." Those sentences are more similar to each other than the pages they describe are. Giving every document a compact vector pointed at its own summary clusters all nineteen documents together and hands topical pages a strong best-chunk score on queries they merely share vocabulary with. The keyword engine has the opposite property: it cares which specific rare words appeared, and a description contributes a handful of high-idf terms the prose never used.

The finding, stated the way it is worth remembering: the field that reads most like a search query is the field the vector engine handles worst. Summary prose is low-variance across a corpus. Low variance is poison for a similarity measure and harmless for an inverted index.

Descriptions now live as their own BM25F field at w_desc = 1.0. Worth exactly three cases: 34 to 37 on the 46-case set, re-measured, six and a half points. And 1 versus 2 is byte-identical per case, re-measured too, which is the saturation math saying the signal is presence, not weight. The two-line experiment that established weight-insensitivity is also what avoided building a heavier mechanism first and justifying it afterward.

The knobs that could not reach

Now the other half of the ledger. Four ranking levers built, measured, and shipped inert or at defaults in this part of the arc, each with a recorded reason.

chunk_penalty: structurally inert on this corpus. Three remaining failures were won by the same page, a getting-started tutorial that touches install, model, build, and serve, and therefore has a chunk near almost any query. That is precisely the bias this knob exists to correct: max-pooling over n chunks is a biased estimator, and the correction subtracts coeff * sqrt(2 ln n). Sweeping 0.02 and 0.04 produced byte-identical per-case runs. At 0.06 exactly one case moved: "which files land in the deployed site" slid from the caching how-to to the configuration reference, two pages that both answer it, and part two records the fixture widening to say so. The arithmetic explains both halves: what reorders two documents is the difference between their penalties, and for a twelve-chunk document against a five-chunk one that difference is sqrt(2 ln 12) - sqrt(2 ln 5) = 0.44, which at coeff = 0.02 is 0.009 of cosine against rank-gaps of 0.02 to 0.05 on this corpus, and at 0.06 is 0.026, just inside the band, where it found the one pair sitting at the bottom of it. Below roughly 0.05 the knob cannot reach. The reason is corpus shape: 132 chunks over 19 documents is seven per document, and a correction that scales with the log of a count needs the count to vary. It was designed against a blog corpus where a 23-chunk essay competed with a one-page note. Here every page is a screen of prose, so at the default and twice the default it is structurally inert, and the first thing it reaches when pushed is a label, not a ranking. Not mistuned, inert here. Different diagnoses: a mistuned knob wants a sweep, an inert one wants to be left at default and understood. And the same arithmetic that convicts the range names the escape: the differential scales linearly with the coefficient, so at roughly six times the default it clears the top of the observed rank-gap band. "Inert" is a claim about a range, not a knob, and part two's closing records what happened when a later sweep went where this one did not.

min_gap / strong_cos: built for a real problem, shipped disarmed. An earlier experiment at higher dimensionality broke both negative controls, and the failure taught me the shape of the problem: on a homogeneous corpus, the signature of noise is flatness, not low absolute cosine. An off-topic query produces a flat pack where every doc scores 0.15 to 0.35 with no standout. A real paraphrase produces one document clearly above the field. So I built a corroboration gate on the relative statistic top_median_gap, suppressing an uncorroborated semantic list only when the field is flat and nothing is strongly relevant in absolute terms. In this part of the arc it ships with both knobs disarmed (min_gap 0, strong_cos off, an asymmetry documented on the fields because it is exactly what someone "cleans up" later). What ships armed is the diagnostics: --explain prints gap and top, so that if this gate is ever needed, calibrating it is a lookup rather than a sweep. It turned out to be needed sooner than expected. Part two.

rrf_alpha: built, first sweep regressive, ships at 0. Confidence-weighted rank fusion cost points at every armed value on the first pass. Its story is not over either, and the way it comes back is the best argument in the whole arc for keeping measured dead ends written down. Part two.

w_tag: flat above zero. Every one of the 19 pages carries hand-curated tags under [taxonomies], and the field is live: at 42593bf, w_tag 0 costs one case, "shrink vectors to save bandwidth", a page that is found by its tags when the weight is armed and by nothing else when it is not. 2, 4, and 8 are byte-identical per case. Presence is the signal and weight is not, the same shape the description field showed, and the same reason a sweep across the armed range reads flat. The default of 4 is inherited; the sweep says any value above zero would do, and the config comment now says that instead of implying the number was earned.

Four levers shipped inert. Not waste: each has a recorded reason, and three of them ruled out a hypothesis about one stubborn failure that I would otherwise have re-tried in a month. A measured dead end you wrote down is cheaper than the same dead end explored twice.

Two methodology lessons, learned the hard way

Summary-level A/B testing hid a cancellation. The description experiment initially ran two changes in one commit: text into synthetic chunk 0, and terms into the keyword body. The summary read 77%, then 75%, then 77% again, which looked like a change that did nothing. It was two changes pulling in opposite directions, each worth one to two cases, netting to zero, and I read "descriptions do not help this corpus" off the summary line twice. What exposed it was diffing per-case lines instead:

chops-search eval | grep -E '^(PASS|top3|FAIL)' > /tmp/a.txt
# change exactly one thing, rebuild
chops-search eval | grep -E '^(PASS|top3|FAIL)' > /tmp/b.txt
diff /tmp/a.txt /tmp/b.txt

A one-line diff is a real result. An empty diff is a real result. An unchanged summary is not, because it cannot distinguish "nothing happened" from "two things happened and cancelled".

Growing the fixture lowered the score, which was the point. The set went from 23 cases to 46, chosen for coverage rather than for passing: every indexed page became the primary expectation of at least one case, which closed three pages nothing had been asking about. Recall@1 read 83% before and 76% after (19/23 to 35/46, same engine, same floor, both re-measured), on the same engine facing a fixture that finally asked about everything. The denominator changed and the honest response is to say so, which is what the scoreboard at the top of this post is for. The growth also flushed out fixture bugs wearing ranking-bug clothes: one case handed a distinctive word to a wrong-but-plausible page (fix: reword the case), another asked a question whose only answer lives in a section landing page the indexer skips by design (fix: delete the case, or improve the docs for their own sake). The distinction matters. Editing content so a test passes is only legitimate when the content is better afterward. Otherwise it is fitting the corpus to the fixture, which is the same failure as fitting thresholds to the fixture, wearing a different hat.

The plateau

Where part one ends: 46 cases, honest configuration, 34 passing. 74% recall@1, 87% recall@3, both re-measured at 42593bf with no flags (34 passing, 40 within the top 3).

That thirteen-point gap between @1 and @3 is the tell. The right page is almost always retrieved. It is just not first. Which means the remaining problem lives wherever "first" gets decided, and in a hybrid engine that place is rank fusion. So I had a hypothesis, and it was attractive: RRF's conventional k = 60 is calibrated for TREC-scale runs fusing thousand-deep lists. On a 19-document corpus the reciprocal-rank curve is nearly flat (rank 1 versus rank 2 is 1/61 versus 1/62, a gap of 0.0003), so fusion degenerates toward "best average rank wins" and a decisive first place in one engine cannot survive mediocrity in the other. That is exactly the @1-fails, @3-passes signature. Nobody calibrates k. It looked like the constant nobody had checked.

I built a sweep to test it: a full grid over rrf_k and rrf_alpha, every cell running all 46 cases, with the k=60 / alpha=0 cell required to reproduce the baseline exactly as a self-check on the harness.

What the grid did to the hypothesis, what the per-kind table said instead, a regression test that turned out to be passing by a margin of eight millionths, and the fixture file that predicted its own future: part two.

The engine, the eval harness, and both fixture files with their header rules are in the repo: github.com/gitbadger-clan/chops-search.