AI Chatbot Giving Wrong Answers? Why It Can't Find What's in Your Documents

When an AI chatbot gives wrong answers about your own documents, the AI model is usually not the problem. The model only reads the few passages a search step hands it, and when that search hands it the wrong ones, a perfectly good model answers badly.

You have probably seen both versions of this. The bot says "I couldn't find that information" when the answer is sitting on page 4 of your PDF. Or it answers confidently with something your documents never said. Same root cause, different symptom.

I build these pipelines in my day job: pulling text out of documents, splitting it, indexing it, and running sets of test questions that catch wrong answers before users do. When an answer comes out wrong, the cause is usually in one of those earlier steps, not in the model.

In this post I'll explain why it happens in plain English, the five places it usually breaks, how to check each one, and how to prove a fix worked. It applies to chatbots built on your own documents (often called RAG, short for retrieval-augmented generation) and to ChatGPT custom GPTs with uploaded files.

How a document chatbot answers a question

A chatbot that answers from your documents does not read all of them every time someone asks something. That would be slow and expensive. Instead, each document is prepared once, and every question goes through the same short pipeline:

1. Extraction. The text is pulled out of the PDF, page by page.
2. Chunking. The text is cut into short passages of a few paragraphs each.
3. Embeddings. Each passage is turned into a list of numbers that captures its meaning, so passages can be compared with questions.
4. Search. The question is converted the same way, and the closest passages are found.
5. Context. The best few passages are pasted into the prompt next to the question.
6. Answer. The model writes a reply from what it was given, ideally citing the document and page.

Stages 1 to 5 decide what the model sees. The answer can only use what reaches it.

The model at the end can only be as good as the passages it receives. If the right passage never reaches it, it either says it found nothing, which is honest, or fills the gap with something that sounds right, which is worse.

A worked example: the refund question

Here is the failure in miniature, using a sample refund policy. The document, scores and results are illustrative, but the pattern is a very common one.

A customer asks: "When can I request a refund?" The rules are on page 4. The chatbot replies: "I couldn't find that information in your documents."

Problem 1: the chunk with the answer never says "refund"

The heading "5. Refunds" sits at the very bottom of page 3, and the rules start at the top of page 4. The document was split page by page, so the heading landed in one chunk and the rules in another:

Chunk A has the word the customer typed but no answer. Chunk B has the answer but never says refund.

Chunk B holds the answer, but it talks about "money back" and a "return window". The word the customer typed lives in a different chunk.

Problem 2: search finds the right chunk, then throws it away

Search does find chunk B, but it scores 0.76 for similarity, just under a cut-off of 0.80 that was set to filter out weak matches. Only three results are kept anyway:

The real answer scores 0.76 and falls just under the 0.80 cut-off, so the model never sees it.

So the model receives a passage about shipping and one about cancellations, neither of which mentions refunds, and correctly says it found nothing. Every part of the system did what it was told. Two settings, chosen once and never tested, did the damage.

The five places it breaks, in the order to check them

Do not guess which stage is broken, and do not start by swapping the AI model. For each question that fails, look at what every stage produced. The root cause becomes something you can see, not something you argue about.

1. Extraction: is the text actually there?

Open the extracted text for the page that holds the answer. Scanned PDFs are pictures of pages, with no text at all unless they go through OCR (text recognition) first. Tables often come out as a jumble of numbers, and two-column layouts can be read straight across both columns, mixing sentences. If the text is wrong here, nothing later can fix it.

2. Chunking: does each heading stay with its text?

Check the chunk that should contain the answer. Splitting by page or by a fixed number of characters cuts sections in half, exactly as in the refund example. Split along headings instead, keep each heading attached to the text under it, let neighbouring chunks overlap a little, and store the document name and page number with every chunk so answers can cite them.

3. Embeddings: are documents and questions measured the same way?

Documents and questions must be converted with the same embedding model. If the model was changed, or a PDF was replaced, every chunk has to be converted again. A half-updated index produces search results that look random.

4. Search settings: is a filter hiding good matches?

Look at how many passages are kept and whether there is a minimum score. Both are usually guessed when a chatbot is built; set them from test questions instead. When customers use different words from your documents ("money back" versus "refund"), meaning-based search alone can miss. Combining it with plain keyword search, known as hybrid search, catches those cases.

5. What reaches the model: is the found text sent, and is "not covered" allowed?

Check the final prompt. Sometimes the right passages are found and then cut to fit a length limit. And if the instructions never say "answer only from these passages, and say so when they do not cover the question", the model fills gaps with confident guesses.

Why it makes up answers instead of saying it doesn't know

A made-up answer is the same failure as "I couldn't find that", with a different ending. When search returns passages that are near the topic but do not contain the answer, the model tries to be helpful and stitches together something plausible.

Two changes stop most of it: an instruction to answer only from the passages provided, and a citation (document and page) on every answer. A citation makes a wrong answer easy to spot, for you and for your customers.

Prove the fix with the same questions, before and after

Before changing anything, agree on a set of test questions and the right answer for each. Include three kinds: direct questions, reworded ones (how customers actually talk), and questions your documents cannot answer, where the right reply is "that isn't covered". Run them before the fix and after. The fix only counts if this table improves:

Before: not found, and a made-up answer. After: the right answer with its page, and an honest 'not covered'.

In this sample, 3 of 8 answers were right before the fix and 8 of 8 after, each with the page cited. Look at the Canada question: before the fix the bot invented an answer, and after it says the documents do not cover it. For a business, that is the most important row in the table.

Keep the questions. Every time you add documents or change a setting, run them again.

If you use ChatGPT custom GPTs or file uploads

The same mechanics apply. A custom GPT searches its knowledge files for relevant passages instead of reading them end to end, so the same things go wrong:

• Scanned PDFs may have no readable text. Run OCR first, or upload a text version.
• Long files with vague headings are hard to search. Split them into smaller files with clear, descriptive headings.
• In the instructions, tell it to search the knowledge files before answering, to name the section it used, and to say when the files do not cover a question.
• Test it with the same before-and-after questions.

If a custom GPT still misses answers after that, it has reached the limits of a tool you cannot tune. That is usually the point where a small chatbot of your own, where you control the chunking, the search settings and the prompt, starts paying for itself.

The principle: find the stage that breaks, fix only that

Wrong answers from a document chatbot feel like an AI problem, so the instinct is to try a bigger model. Usually the fix is earlier and cheaper: text that was never extracted, a heading cut off from its section, or a search filter set by guesswork. Trace one failing question through every stage, fix the stage that breaks, and prove it with the same questions before and after.

It is the same rule that got two of my fixes merged into a 10 million user app: trace the data, not the symptom. More guides like this one are in AI chatbots and AI features.

If you would rather hand it over, I trace chatbots exactly this way. Send me the PDFs that cause trouble and a few questions that fail today, and you get the same before-and-after table for your own documents, with a short note on the root cause.

Common questions

Why does my chatbot say it can't find information that is in my PDF?
Because the passage with the answer never reached the AI model. Usually the document was split so the answer sits in a chunk without the words people search for, or a search setting (a minimum score, or too few results kept) dropped it. Check the extracted text and the search results for that question before changing anything else.

Why does my AI chatbot make up answers?
When search returns passages that are close to the topic but do not contain the answer, the model fills the gap with something plausible. Tell it to answer only from the passages it is given and to say when they do not cover the question, and require a document and page citation on every answer.

Why is my custom GPT not using its knowledge files?
Custom GPTs search their knowledge files instead of reading them end to end, so scanned PDFs, very long files and vague headings all make the right passage hard to find. Use text-based PDFs, split long files into smaller ones with clear headings, and tell the GPT in its instructions to search the knowledge files before answering.

Will a better AI model fix wrong answers?
Rarely. A stronger model still only sees the passages it is given. If the problem is extraction, chunking or search settings, a new model gives the same wrong answers in nicer words.

Can this be fixed without rebuilding the chatbot?
Usually, yes. Most fixes are a change to how documents are split, a re-index, adjusted search settings and a clearer prompt. A fixed set of test questions tells you when it is done.

This post first appeared on tusharbhendarkar.com. I build and fix AI chatbots and RAG systems for small businesses. If something here is broken for you, email me at tusharbhendarkar44@gmail.com.