Entry 17 ended on a problem I'd made for myself: I'd been judging rankings with nothing to check them against. This entry fixes that, at least a little. Twelve questions, each with one article that should answer it, plus two controls where the collection has no answer. I labeled each right article from the filenames and the descriptions in the repo index, not by rereading the articles, so the labels are my best read, not verified ground truth.
The script asks each question against all 741 chunks, finds the right article's best-scoring chunk, and ranks it two ways: among all chunks (what an uncapped query returns) and among distinct articles (what Entry 17's cap of one per document returns). It also counts how many of the top five chunks come from the right article. That last column matters more than it sounds. A cap can only move the right article up, because it removes chunks that sat above it, so document-level hit rates will always flatter the cap. What the cap costs is extra chunks from the right article, and only the chunk count shows that.
python3 eval_retrieval.py
# doc_rank chunk_rank exp_in_top5 top1
1 1 1 5 retry-logic-and-tiered-alerting-github-actions.md
2 1 1 4 secrets-management-multi-cloud-pipelines.md
3 1 1 4 terraform-managed-openshift-state.md
4 1 1 5 database-migrations-zero-downtime-deploys.md
5 1 1 2 prompt-versioning-ci-openshift.md
6 1 1 3 swapping-llm-providers-without-rewriting-stack.md
7 5 8 0 ai-tooling-openshift-evaluation-framework.md
8 1 1 4 pitw-01-zero-downtime-deployments.md
9 1 1 5 ai-in-the-stack-02-rag-runbooks.md
10 1 1 3 ai-in-the-stack-06-n8n-workflows.md
11 1 1 3 what-are-containers.md
12 3 7 0 kubernetes-at-scale.md
Ten of twelve questions put the right article first, both ways. After four entries of staring at rankings I couldn't judge, that was the first real evidence the index mostly works.
The two misses both lost to broader articles. "How can an AI agent safely read from my Kubernetes cluster?" went to the AI tooling evaluation framework, with the MCP architecture article fifth among articles and eighth among chunks. "Should I run managed or self-hosted Kubernetes?" went to the Kubernetes-at-scale piece, with the managed-vs-self-hosted article third and seventh. So I printed the chunk that won each one. For the agent question, the winner (distance 246.5) comes from the framework's prerequisites section and says that any AI tool touching a cluster operates within or around OpenShift's security constraints and RBAC. That's on topic. It reads more like a second defensible answer than a retrieval error, and my label may simply have been too narrow. For the managed question, the winner (235.4) is a section of the Kubernetes explainer on the managed platforms you'll encounter, which says nobody runs Kubernetes completely from scratch in production. It matches the words "managed Kubernetes" but describes platforms rather than weighing managed against self-hosted, so that one looks like a real miss. In both cases I've read only the winning chunk, not the articles.
I'm keeping the original labels and the 10-of-12 figure. Relabeling the agent question now would raise the score based on what I saw, not on what I measured.
One worry going in was that the series' own entries about MCP, n8n and RAG, embedded in the same index, would outrank the real articles. At the top spot, none did, zero of twelve. The check only looks at first place, so they may still be crowding slots two through five. I haven't looked.
Now the cap, with the numbers to judge it:
| Uncapped (by chunk) | Capped at 1 per article | |
|---|---|---|
| Right article at #1 | 10 of 12 | 10 of 12 |
| Right article in top 3 | 10 of 12 | 11 of 12 |
| Right article in top 5 | 10 of 12 | 12 of 12 |
| Chunks from the right article in the top 5 (average) | 3.17 | 1.00 |
The capped 1.00 follows from how the cap works: all twelve right articles land in the top five articles, and each can appear only once. Neither miss reached first place even with the cap, so it did nothing for the top result. What it did was pull the right article into a short list for the two questions that missed, which is where the top-3 and top-5 gains come from.
The cost shows in the last row. For the ten questions that worked, the uncapped top five held 3.8 chunks from the right article on average. Under the cap that drops to one. If what you want is a list of sources, the cap pulled the labeled article into view for both misses, though only one of them looks like a clear miss. If what you want is passages to hand a model, it would have discarded roughly three of every four chunks from the right article on the questions that were already working.
The controls behaved the way Entries 15 and 17 would predict. "How do I check pod status with oc?" returned Entry 08 at 225.6, and "What's the best pizza topping?" returned Entry 06, the entry that contains that exact question, at 448.4. Both nearest matches are write-ups that quote the query. And 225.6 looks like a very strong match for a question the collection doesn't answer, so a distance cutoff alone wouldn't have flagged it as "not in here."
Twelve questions is a small set. Each one moves a percentage by eight points, this was a single run, and the question wording and the labels are mine. Treat the 83% as a rough reading, not a benchmark.
This answers the question Entry 17 left open, narrowly: the cap is a trade, not a fix. It earns its place when the right article only needs to show up in a short list, and it costs a lot when the consumer needs depth. I haven't tested the middle, a cap of two or three per article, and with this eval set I now can. Before I do, I need to settle whether a question can have more than one acceptable article, and I want that decided by reading the articles, not by checking which labels would raise the score.
