A RAG chatbot demo is easy to get right on day one. You load three documents, ask about refunds, the bot quotes the refund policy with a [1] next to it, and everyone is happy. The trouble starts weeks later, after the documents have been edited, someone has tried a better embedding model, and users have started asking questions your documents never covered. Nothing crashes. The answers just get a little more wrong every week.
I built a small RAG proof project in ASP.NET Core (.NET 10, Microsoft.Extensions.AI 10.11, the official Qdrant.Client 1.19) to find out where that rot comes from. It has 21 tests: 18 unit and API tests, plus 3 that run against a real Qdrant 1.19.2 binary instead of a fake. This post isn't a build-it-from-scratch tutorial. It's the four failures I now check for in any RAG codebase, with the check that catches each one.
If you'd rather start from a finished RAG API than wire this yourself, the ASP.NET Core RAG Kit is a ready ASP.NET Core + Qdrant starter (more on it at the end), and Tech Skill Builder has complete, tested .NET projects with full source code.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.
The 30-second version of the pipeline
So we're talking about the same thing:
-
Ingest: split the document into chunks, embed every chunk with an
IEmbeddingGenerator<string, Embedding<float>>, upsert the vectors into a Qdrant collection with the chunk text, title, document id and chunk index as payload. -
Ask: embed the question, query Qdrant for the closest chunks, number them
[1]..[n]in a prompt, call anIChatClient, and map the[n]markers in the answer back to real sources.
Every failure below lives in one of those steps, and none of them throws an exception unless you make it.
Rot 1: The old version of a document is still answering questions
Say your refund policy used to be five chunks long. Legal rewrites it and it's now three. You re-ingest it. If your point ids are random GUIDs, Qdrant now holds eight chunks for that document: three new ones and five old ones. If they're sequential ids, chunks 4 and 5 of the old version are still there. Either way, a question can be answered from a paragraph that no longer exists in the real policy, with a perfectly valid-looking citation.
The fix has two parts. First, give every chunk a deterministic id derived from the document id and chunk index, so a re-ingest overwrites the same points instead of adding new ones:
// Same document + same chunk index => same Qdrant point id, every time.
public static Guid IdFor(string documentId, int chunkIndex)
{
// SHA-256 of $"{documentId}#{chunkIndex}", first 16 bytes as a Guid
// full implementation in the complete project
}
Second, after the upsert, delete whatever is left over from the longer version. Index document_id (keyword) and chunk_index (integer) as payload fields so this filter is cheap:
var stale = MatchKeyword("document_id", documentId)
& Range("chunk_index", new Qdrant.Client.Grpc.Range { Gte = newChunkCount });
await client.DeleteAsync(collection, stale, wait: true, cancellationToken: ct);
The order matters: upsert first, delete second. If you delete the whole document and then re-insert it, there's a window where the document isn't searchable at all.
The check: ingest a long version, re-ingest a shorter one, and assert the point count in Qdrant dropped. My integration test ingests three documents into a real Qdrant, re-ingests a one-sentence refund policy, and asserts the collection's point count drops to 3 (one chunk per document). A unit test does the same on an in-memory store, so it also runs when Qdrant isn't available.
Rot 2: Someone switched the embedding model
A Qdrant collection's vector size is fixed when you create it. Your fake or local model gives 384-dimensional vectors, someone moves to OpenAI's text-embedding-3-small (1536) or an Ollama model like nomic-embed-text (768), and now the vectors you're writing don't fit the collection you already have. Depending on how your code is written, you'll get a confusing error from deep inside the client, or worse, someone "fixes" it by recreating the collection with half the documents missing.
The fix: check the size before the first write and fail with a message a human can act on.
if (await client.CollectionExistsAsync(collection, ct))
{
var info = await client.GetCollectionInfoAsync(collection, ct);
var size = info.Config.Params.VectorsConfig.Params.Size;
if (size != (ulong)dimensions)
throw new InvalidOperationException(
$"Collection '{collection}' stores {size}-d vectors but the embedder produced {dimensions}-d. ...");
}
// else: create it with Distance.Cosine and the two payload indexes
// (locking and the create branch: full EnsureCollectionAsync in the complete project)
Even better, don't let two models share a collection at all. I name them after the model: docs-openai-1536, docs-ollama-768. Then switching models is a config change plus a re-ingest, not a migration.
The check: one test points a 128-d embedder at a store that already holds 384-d vectors and expects the exception. Another creates a 384-d collection in a real Qdrant and expects a request for 1536-d to be rejected with that message.
Rot 3: The model gets called when nothing relevant was found
Users will ask things your documents don't cover. "Which Kubernetes version do you run?" against an HR handbook. If your code always calls the LLM with "the top 5 chunks", it'll send five barely related chunks and the model will try to be helpful. That's where most made-up answers come from, and you pay for every one of them.
The fix: use a score threshold on the search, and if nothing clears it, return a fixed "not found" answer without calling the model.
var queryVector = await embedder.GenerateVectorAsync(question, cancellationToken: ct);
var hits = await store.SearchAsync(queryVector, topK, minScore, ct);
if (hits.Count == 0)
return new AskResult(NotFoundAnswer, [], Grounded: false, SourcesConsidered: 0);
// build the numbered prompt, call the chat client, extract citations
// full AskAsync in the complete project
The threshold itself (scoreThreshold on Qdrant's Query API) has to be tuned per embedding model. Scores from different models aren't comparable, so a value that works for one is meaningless for another.
The check: wrap the chat client in a small DelegatingChatClient that counts calls, ask an unrelated question, and assert the count is zero. "It returned I don't know" isn't enough. You want proof the model was never asked.
Rot 4: Citations that point to nothing
You ask the model to cite sources as [1], [2]. Most of the time it does. Sometimes it writes [7] when you only sent two sources. If your UI turns every [n] into a link, the user sees a confident citation to a source that was never in the prompt.
The fix: never render the model's citation numbers directly. Parse them, keep only the ones that map to a source you actually sent, and build the citation list from your own data:
var numbers = Marker().Matches(answer) // matches [1] and [1, 2]
.SelectMany(m => m.Groups[1].Value.Split(','))
.Select(n => int.Parse(n.Trim(), CultureInfo.InvariantCulture))
.Where(n => n >= 1 && n <= sources.Count) // drop invented numbers
.Distinct()
.Order();
// mapping each number to document id, title, chunk index, score and snippet:
// full Citations.Extract in the complete project
I also mark an answer as grounded only when at least one valid citation survives. That one boolean is very useful for a UI ("this answer isn't backed by your documents") and for monitoring.
The check: a test that feeds an answer containing [1] and [7] with two sources and asserts that only [1] comes back.
A pre-release checklist
Here's what I'd ask about any RAG chatbot before it goes to real users:
- [ ] Are point ids deterministic per document and chunk?
- [ ] Does re-ingesting a shorter document remove its leftover chunks, and is that tested?
- [ ] Does the code refuse to write vectors of the wrong size to an existing collection?
- [ ] Does each embedding model get its own collection?
- [ ] Is there a score threshold, and does "nothing found" skip the LLM call?
- [ ] Are citation numbers validated against the sources actually sent?
- [ ] Is there at least one test against a real Qdrant, not just a fake?
The last one matters more than it looks. My real-Qdrant tests each run in a throwaway collection that's deleted afterwards, and if Qdrant isn't reachable they're reported as skipped with a message saying how to start it, instead of failing. CI without Qdrant stays green and honest, and a developer with Qdrant running gets the full check.
What I didn't verify
The tests use a deterministic hashing embedder and an extractive fake chat client, so no API key is needed. The hashing embedder matches words, not meaning, so its scores tell you nothing about real retrieval quality. The OpenAI and Ollama paths compile but never ran in my tests, Qdrant was a single local node without an API key, and the proof project has no streaming endpoint.
Skip the wiring: the ASP.NET Core RAG Kit
The checks above are the parts I'd add to any RAG codebase. If you want the rest already built, the ASP.NET Core RAG Kit is a ready ASP.NET Core RAG starter with Qdrant for a one-time $20. It's a .NET 9 Web API with full C# source: an ingestion pipeline that embeds your documents into Qdrant, vector search for context, a chat endpoint that builds context-aware prompts and streams answers from OpenAI, Swagger UI for every endpoint, Docker setup for a local Qdrant, and a step-by-step README. You start with the API and streaming already working and spend your time on your documents and prompts.
And if you learn best by reading and running complete code, Tech Skill Builder gives you complete, working, tested .NET projects with full source code and step-by-step PDF guides, with new ones added regularly. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder
Which of these four have you already run into, and which one did your users notice before your tests did?
