A memory search came back with a chunk titled "Getting Started with React" above the fold. Nothing in the query was about React. No error was logged, no warning, no fallback line. The relevance floor that exists precisely to drop results like that had looked at the score and passed it.
I spent the first pass assuming the ranking was bad: bad keys, bad chunk, cross-encoder having a dumb day. Wrong. The cross-encoder never ran. Under memory pressure the model failed to load, the rerank step returned early, and the raw cosine scores from the vector stage were still sitting in the score field. The floor did its job perfectly against a number it was never calibrated for.
Cosine spread 0.10 across the set; the cross-encoder spread 0.997
Here is the part that made it obvious once I dumped a real score breakdown. Two chunks from the same result set, same query, both scored by both stages:
[
{"text": "reranker had four early-return paths...", "cosine": 0.8434, "rerank": "fired", "reranker_logit": 0.9998},
{"text": "Mine three months of incident notes...", "cosine": 0.7463, "rerank": "fired", "reranker_logit": 0.0025}
]
Those two are 0.098 apart on cosine and 0.997 apart on the cross-encoder. Across the whole set, cosine ran 0.73 to 0.84. Every candidate in a retrieval set is semantically nearby, so cosine is a narrow, high, nearly useless band. The cross-encoder is the thing that spreads them out. A floor tuned on that spread is a real filter. The same floor applied to cosine passes literally everything, including a React tutorial.
Four early returns, and only the cosine gate leaves a safe number behind
The rerank step had four ways to return without scoring. One of them, the skip when the top raw cosine is already very high, is safe: the results are good enough that ordering by cosine is fine and the floor is not the load-bearing part. The other three leave stale numbers in the score field with no marker at all: fewer than two candidates, reranking disabled by config, and the model returning an error. Model-load failure under memory pressure lands in that third bucket. rerank_model in memory.toml defaults to a roughly 1 GB checkpoint. Containers get squeezed. Loads fail. The code handled it "gracefully."
That word is the trap. The Neural Base's fallback guide is right about most of this: define failure precisely, catch framework-specific exceptions rather than just TimeoutError, handle 429s that the SDK swallows. What it treats as the goal is falling back to the original ranked list instead of returning garbage. That is a ranking remedy, and my ranking was fine. Order was preserved. The gate downstream was reading a number from a different distribution and had no way to know. Tianpan's post on thresholds as couplings gets closer: if score > 0.4 asserts the distribution of score is stable. The standard advice is "fall back gracefully to the previous ranking." That advice does not cover the case where a threshold downstream is still enforcing the arithmetic of the stage that just disappeared.
Any gate that reads a score nobody stamped with a producer
The class: an optional, expensive scorer that can fail to load, a cheaper scorer whose output is left in place when it does, and a cutoff calibrated on the expensive one that keeps enforcing. Failure is silent because nothing threw and the ranked list looks plausible. It shows up in LangChain and LlamaIndex reranker chains that degrade to embedding similarity, in pgvector and Elasticsearch hybrid search with a hard score cutoff, and in any sentence-transformers CrossEncoder loaded lazily in a memory-capped container.
Every score a threshold reads must carry the identity of the scorer that produced it, and the threshold must refuse to compare across identities. Not "log the fallback." Stamp the row, and make the gate branch on the stamp. Our breakdown already carried "rerank": "fired" versus "rerank": "tail" with a null logit for anything unscored, so the fix was to make the three unsafe returns stamp themselves and make the floor read the stamp instead of the number.
Five minutes: take the model away and watch the gate not care
Against your own stack, not mine:
import os
from your_app.retrieval import search # your real production entry point
QUERIES = [...] # 20 real ones, plus 5 you know should return nothing
FLOOR = 0.65 # your cutoff, whatever it actually is
def run(tag):
scores = [h.score for q in QUERIES for h in search(q)]
passed = sum(1 for s in scores if s >= FLOOR)
print(tag, "n=", len(scores), "passed=", passed,
"min=", round(min(scores), 3), "max=", round(max(scores), 3))
run("baseline")
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_HOME"] = "/tmp/definitely-empty"
run("model-gone") # or: docker run --memory=512m, same effect
Passing looks like a raised exception, or passed=0, or an identical range. Failing looks like this: no exception, a pass count in the same neighborhood, and a range that quietly moved. Mine went from min=0.002 max=0.999 to min=0.73 max=0.84. Same floor, same code path, different universe. If you log scores, the SQL version is one line: SELECT scorer, count(*), min(score), max(score) FROM search_log GROUP BY scorer; and if there is no scorer column to group by, you already have the answer.
A cutoff is not a property of a pipeline. It is a property of one scorer inside it. If your code can produce that number two different ways, write down which one it was, and make the comparison fail loudly when it does not know.
Source: Your reranker fell back to cosine and the floor kept firing by Chad Priest, from Building Vodou in Public.
