A RAG prototype takes an afternoon. Chunk some documents, embed them, stuff the top matches into a prompt, ship a chat box. It works. You demo it, everyone is impressed, the project gets funded.
Then you point the same code at the real drive and it starts lying with total confidence.
I build AI voice agents and intake systems for a living, which means I build knowledge bases whether the client calls them that or not. An agent that answers a real caller is a knowledge base with a phone attached, and it fails in exactly the same places. My honest position now is that the model is the least interesting part of the build. Every project I have seen stall has stalled on four things, none of them the model.
The demo works because someone picked the documents
That is the whole trick, and nobody says it out loud.
A prototype corpus is curated. Someone chose the good documents, the current versions, from one team, in one format, then asked questions they could already verify. Every one of those conditions breaks at once when you swing the pipeline at production content.
The real corpus has the 2023 policy and the 2026 policy side by side with nothing marking which one is live. It has a slide deck someone made for one client that reads like an official company position. It has the same procedure written three times by three departments who quietly disagree. It has scanned PDFs, spreadsheets where the meaning lives in the layout, and a folder called Final Final.
None of that is a model problem. Retrieval faithfully returns what it finds. If what it finds contradicts itself, the answer comes back confidently wrong in a voice indistinguishable from the answers that were right.
That last part is the actual danger. A search engine that returns a bad document still lets a human notice it is bad. An assistant that summarizes one hides the evidence inside fluent prose. You have taken the one review step that was working and buried it.
Problem 1: your documents disagree and code cannot settle it
The first real deliverable on these projects is usually not software. It is a list: for each topic, which document governs, who owns it, and what is deliberately excluded from the index. Drafts, superseded versions, personal folders, client-specific one-offs, all out.
This cannot be done by the vendor. It needs a person from the business who is allowed to make a call, and it is the single highest-leverage day in the project. Nice side effect: teams routinely discover that two departments have been operating on different rules for a year, which is a finding worth more than the chatbot.
Engineers hate this step because there is no library for it. Do it anyway. Every hour skipped here comes back as a retrieval bug that better chunking will not fix.
Problem 2: half the answer was never written down
The failure I see most is not a wrong answer. It is a system that cannot answer the questions people actually ask, because those answers live in someone's head, in a Slack thread, or in how tickets get resolved rather than in any document at all.
So collect the questions before you build anything. Pull them from the support inbox, the internal help channel, the calls the team already takes, then check how many have a document behind them. The gap is your content backlog, and it is real work that happens during the build rather than after it.
This is the same intake mapping I do before building a voice agent. On Fortell AI, the voice and SMS intake system we build for Community Action Agencies answering in over 100 languages, the work starts with eligibility rules and program policy, not with speech. When the policy itself is ambiguous, no amount of model quality rescues you.
Problem 3: permissions belong at retrieval, not in the prompt
This is the requirement that quietly separates a toy from a production system, and it has to be settled early because it constrains everything else.
If the assistant can read HR files, contracts and salary bands, then who is asking has to change what the retrieval step is allowed to return. Two tempting shortcuts, both wrong:
- Retrieve everything, then filter the results after the fact. The restricted content already entered the context. Whatever the model wrote is downstream of material this user should never have touched.
- Tell the model in the system prompt to keep certain things secret. Instructing a model is not an access control. It is a suggestion with good intentions.
The permission check goes at retrieval time, as a filter on the query itself, so restricted passages never enter the context in the first place. Doing that properly means the index carries access rules from the source system, group membership stays in sync as people change roles, and revocation is honored quickly rather than at the next rebuild.
Two consequences fall out of this:
Answers become user-specific. The same question can legitimately return different results for two employees, which means your cache key is not the question. It is the question plus identity. I have watched a naive response cache turn a correct permission model into a leak.
And your logs now contain who asked what about which document. Useful for improving the system, and also sensitive material that needs a retention policy before launch rather than after the first awkward conversation.
There is a legitimate shortcut here: index only genuinely company-wide content for v1, ship in weeks, add permission-aware sources once the value is proven. That is a good phasing decision. It is a terrible accident.
Problem 4: the index is a pipeline, not an upload
An assistant that was accurate at launch and wrong three months later is worse than no assistant, because people stopped double-checking it in between.
Content changes constantly, so plan for incremental sync from the systems of record, not a manual re-upload. Make deletion propagate, so a removed document stops being quotable. Show effective dates in answers, because a correct passage from a superseded policy is still a wrong answer. And give the thing an owner, because a knowledge base without one decays exactly like the shared drive it was supposed to fix.
A few chunking notes that matter more than the embedding model you argue about: split by meaning rather than character count, keep headings attached to the text underneath them, preserve tables as tables, and carry metadata such as title, owner, effective date and department alongside every passage so answers can be filtered and cited. Scanned documents need real OCR that someone checked. Spreadsheets are usually the worst case, because the meaning is encoded in position and formatting that flattening destroys. Query the underlying data directly instead of pretending the file is a document.
Three habits that make it trustworthy
Cite everything and let people click through. Every answer carries its sources, linked and openable. This is not a nicety. It is the mechanism that lets staff catch errors, and the difference between a tool people verify and one they quietly abandon after being burned twice.
Teach it to refuse. The default behavior of a fluent model is to produce something. What you want, when the retrieved passages do not contain the answer, is a clean admission and a route to a human. Saying that in a prompt is not enough. It has to be tested, because refusal is where systems regress silently after a prompt tweak nobody thought was risky.
Write the eval set before you write the interface. Collect real questions, write the correct answers with the source that proves them, rerun the set after every change to retrieval, chunking, prompts or models. A hundred real questions beat a thousand invented ones, and you have to include the ambiguous ones and the ones with no answer, because those decide whether anyone trusts it. This is unremarkable engineering discipline applied to a probabilistic component. Without it, every change you make is a vibe.
The last one is a placement decision
Adoption is mostly about where you put it. A separate portal people have to remember to visit competes with asking a colleague, and the colleague wins. Put it where the question actually arises: the chat platform, the helpdesk where agents draft replies, the CRM next to the record.
And log every question it could not answer. That log is the most valuable output of the first quarter, because it tells you precisely which documentation to write next.
None of this is about the model. It is content ownership, an access-control decision, a sync pipeline and a test suite. Which is to say it is normal engineering, and the part everyone wants to skip.
I wrote the longer client-facing version of this, including what drives the cost and the questions to ask before commissioning one, over on the Null Studio blog.
