I Almost Shipped a RAG Assistant That Lied About APIs That Don't Exist

I wrote this on X a few weeks ago:

I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn

I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win.

Ask my RAG assistant to verify an Interswitch webhook signature, and it didn't say "not in my knowledge base." It wrote a full authentication flow — real-looking endpoint, real-looking headers — and cited a source URL. The URL wasn't in my corpus. It wasn't anywhere. The model invented a citation for content it also invented, with zero hedging.

I'm building StacksNG for the Africa Deep Tech Challenge 2026 — an offline coding assistant scoped to the African fintech stack: Paystack, Flutterwave, Monnify, Termii. Before I submitted, I ran a 20-prompt adversarial batch against my own pipeline. Category A (in-corpus baseline) and D (phrasing brittleness) came back clean. Category B — five prompts asking about payment providers I deliberately never scraped into the corpus, Kuda, PalmPay, Interswitch, Paga, OPay — did not.

Three of five ignored a system prompt that already said, in plain language, "if the context doesn't contain enough information, say so."

That's the failure mode that zeroes out half the score in a hackathon where accuracy is 50% of the total.

My first theory was wrong, and I could prove it

My instinct was: this is a retrieval-confidence problem. Set a similarity threshold, refuse to answer below it, done.

I checked the actual numbers before writing that fix.

The worst hallucination had higher retrieval similarity than the cleanest correct decline. There's no threshold that lets the good case through and blocks the bad one — they're on the wrong side of each other. A confidence cutoff would have been a fix that felt right and did nothing.

What was actually happening

The chunks my retrieval pulled back for "Interswitch Quickteller" were real — Monnify's quickstart, Paystack's accept-payments guide. Genuinely similar topic: authentication, checkout, webhooks. Not out-of-domain confusion. Same-domain brand substitution. The model wasn't confused about the topic. It never checked whether the retrieved text actually named the provider I asked about, versus a different provider talking about something similar.

That's a sneakier bug than "doesn't know when it doesn't know." It's "knows something adjacent and doesn't notice the adjacency."

The fix was one paragraph, not an architecture change

I didn't retrain anything. I didn't touch retrieval. I added one rule to the system prompt:

Before answering, check whether the specific provider named in the question is actually named in the context excerpts. Retrieval is similarity-based and will sometimes hand you excerpts from a different provider just because the topic is similar — that is not the same as the named provider being covered. If it isn't named, say so. Do not substitute another provider's instructions under the asked-about provider's name, and do not invent a source URL.

Re-ran the five failing prompts plus two controls. Kuda, PalmPay, Interswitch, Paga, OPay: five for five now correctly decline. A cross-provider prompt that used to answer itself into a contradiction — "you should not fall back to X," followed immediately by a full explanation of how to fall back to X — now declines cleanly. The in-corpus control prompt is untouched, still correct.

One honest regression: my out-of-domain control prompt got slightly more hedge-y. It used to say flatly "not in my knowledge base." Now it draws an unprompted analogy to a similar provider before getting there — still no fabricated specifics, just wordier than it needs to be. Three fabrications became zero at the cost of one prompt getting less clean. I'll take that trade. I wrote the regression down instead of pretending the fix was perfect.

That was one run, though. I shipped it as the result.

An independent check found the fix wasn't a fix, it was a coin flip

I asked Antigravity (no access to my corpus authorship, my prompts, or this article) to reproduce the submission cold: fresh clone, download the model, run the official profiler, and re-test the five adversarial prompts above. Not once. Three times each, fifteen trials total.

Ten of fifteen came back clean. Not five of five. Two-thirds.

And it wasn't random noise spread evenly across providers. It split cleanly in two. Interswitch, Paga, OPay: nine for nine, 100% reliable. Kuda and PalmPay: one of three, zero of three. PalmPay fabricated a x-palmpay-signature header and a full HMAC handler on every single run. Kuda got silently rerouted to Paystack's live charge endpoint, with an invented bank code stated as fact, in two of three.

Two things were true and I'd only checked one of them. First: the chat call runs at temperature=0.2 with no fixed seed, so the same prompt doesn't reliably produce the same answer. My original "five for five" was one draw from a distribution, not a property of the fix. Second: the failure wasn't random across providers. It was concentrated exactly where retrieval is most ambiguous. Kuda and PalmPay's webhook-verification content is topically near-identical to Paystack's and Monnify's, same HMAC-SHA512 shape, same header pattern. Their retrieved chunks sit in the tightest, most confusable similarity band I measured (cosine 0.654–0.676, five chunks within 0.022 of each other). The instruction I wrote asks the model to notice when a retrieved chunk doesn't actually name the asked-about provider. It's least able to notice that exactly when the retrieved chunk is close enough to look plausible.

A soft instruction was never going to close that gap reliably, because the thing it's fighting, retrieval similarity between near-duplicate topics, doesn't go away just because I asked nicely.

The actual fix: stop asking, start checking

The corpus only covers four providers. That's a small, enumerable set. Which means the question "is this provider actually in scope" doesn't need an LLM's judgment at all. I added a deterministic gate ahead of retrieval: a list of ~25 known African fintech and banking brand names that are not in the corpus, matched by word boundary against the incoming question. Name one of them without also naming an in-corpus provider, and the question gets declined before retrieval or generation ever runs. No temperature, no seed, no chance to fabricate — just a string match.

Antigravity again, same fifteen trials, same corpus, freshly re-cloned: fifteen for fifteen, every response returned near-instantly with no LLM call at all. Regression-checked clean too. An in-corpus question still runs the full retrieval-and-generation pipeline untouched, and a genuine comparison question ("how does Kuda compare to Paystack for webhook handling?") correctly falls through to the softer instruction instead of getting blanket-refused, since that's a legitimate question the gate isn't built to answer.

It doesn't generalize. A provider I didn't think to enumerate still depends on the same soft instruction that measured 100% for three providers and 0-33% for two. That's a real limitation, not a solved problem, and it's written down as one in the repo instead of implied away.

Why this is the argument for RAG over fine-tuning, not just the reasoning

Paystack's docs are public. Flutterwave's are public. Termii's are public. A frontier lab has access to all of it, the same way it has access to Igbo and Yoruba text scattered across the public web. Access isn't the same as behavior. I made the same point on X about language before I made it about payments APIs:

Frontier labs having access to a dataset and frontier labs training on it are different things. Igbo or Yoruba text that exists publicly still gets drowned out by the sheer weight of English in pretraining. The model isn't ignorant of your language, it's just statistically indifferent to it. Representation in the data pile is not the same as representation in the model's behavior.

Paystack's docs are a rounding error in a pretraining corpus next to Stripe's. That's not a knowledge gap I can fix by asking nicer. It's why the corpus exists — RAG re-injects the drowned-out content at query time instead of hoping it survived pretraining.

I'd already decided to stay RAG-only instead of spending GPU credits on fine-tuning — the corpus is documentation, not instruction pairs, and citations matter more than they'd survive baked into weights. The hallucination bug is the evidence, not just the reasoning. I found a real correctness bug in an afternoon, and even the fix that turned out to be incomplete was a paragraph of English and, later, a list of strings — not a training run. If this had been in the weights, finding out my first attempt only worked two-thirds of the time would have meant retraining, not rereading a diff.

The number that matters

Before any of this, I had a submission that fabricates working-looking code with fake citations for three out of five providers outside its training data — in a payments assistant, where "looks right but isn't" is worse than "doesn't know."

I almost didn't re-run my control prompt after the first fix. I'd have shipped the fabrication count as zero and missed the one place it got worse instead of better. And I almost stopped there: one clean run, five for five, box checked.

All twenty of those original prompts were mine, though, written by the person who also wrote the fix, which is exactly the setup that lets a bug hide. So I went looking for a test I didn't write, twice. First, one I couldn't have written: a developer on X and a stranger on Reddit, unconnected to each other, both stuck on the same real thing: making a Paystack webhook handler idempotent. Neither had my corpus in mind. Neither had my prompts. I ran it anyway. Top retrieved chunk was Monnify, same shape of mismatch that caused the original bug. It stayed on Paystack, gave the real fix, cited only what it actually retrieved.

Second, Antigravity re-running my own adversarial set (the one I had already tested) fifteen times instead of once. That's the run that found the fix was two-thirds reliable, not fully. The fifteen-trial number is less flattering than the five-for-five I almost shipped, and it's the one that's actually true. The deterministic gate that replaced the soft instruction got checked the same way: not "does it look right," but "does it hold up when someone with no stake in the answer runs it enough times to catch the unlucky draw."

The bug wasn't boring. My first fix of it wasn't finished.

StacksNG is an entry in the Africa Deep Tech Challenge 2026. Code, corpus scrapers, and the full stress-test transcript are in the repo.