A self-consistency check can't catch a number that's internally coherent but simply wrong β so I designed a different kind of verifier
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
My automated trading system generates an AI-written report for every ticker, every day. There's no way I can read every single one and check every number by hand.
Those reports are dense with concrete figures β prices, financial numbers. If a number is wrong, the reasoning built on top of it is wrong too, so I'd had some kind of check on report numbers in place for a while.
Then I had reason to look at that check again. I wasn't sure "there's a check running" actually meant "the thing I care about is being checked."
What the existing check actually did
The check I already had looked at whether numbers agreed with each other inside a single report. If a figure appeared early in the report and again later, it verified the two mentions matched β a self-consistency check.
That catches a report contradicting itself. It does nothing if the report got a number wrong from the very start and then just repeated that same wrong number consistently throughout.
Say a number from the source data gets converted to a different unit and a digit gets dropped in the process. If the report keeps citing that wrong number the same way everywhere, the document is internally coherent from end to end. To a self-consistency checker, that's a clean pass.
So I designed a different kind of verifier
What I actually needed was something that compared report sentences directly against the source data β not internal consistency, but agreement with ground truth outside the document.
Numbers in these reports fell into two rough categories: price and technical-indicator figures pulled from candle data, and financial figures like revenue or market cap. Because the source data behind each is shaped differently, I split the verifier into two separate axes β a price axis and an amount axis.
That split mattered more than I expected. Candle data (the ground truth for the price axis) keeps accumulating every trading day, so it can always be checked retroactively. The ground truth for the amount axis, on the other hand, only exists as the financial data block that was actually fed into that day's report β once a day passes, there's no ground truth left to retroactively check against. That asymmetry ended up shaping how I later had to measure the scale of the problem.
The first version cried wolf
The first version of the price-axis verifier fired constantly the moment I built it. Most tickers tripped it, but on inspection, none of it was real β it was all coincidence.
The problem was volume: candle data gives you hundreds of candidate reference values. With a loose tolerance window, almost any number in a report will happen to land close to one of those candidates by chance alone.
I tightened it with three layers: only exact matches count, the comparison window is limited to recent candles, and a flag only counts as real when the same error pattern shows up in a cluster rather than in isolation. Net effect: lower recall, much higher precision.
That trade-off was deliberate. In an unattended system, a verifier that keeps crying wolf eventually gets ignored altogether. I'd rather miss the occasional isolated error than train myself to tune out the alerts.
What showed up the moment it was wired in
Once both axes were wired into the actual production pipeline, I measured the first real baseline against production data. The price axis came back nearly clean, matching what I'd already observed informally.
The amount axis was the opposite story. More than half the tickers in a single day's reports had a financial-figure error. Looking at one concrete case, a source number had been converted to a different unit and the trailing digits got dropped along the way β inflating the reported value well past the true figure. Because the report cited that wrong number consistently throughout, the old self-consistency check had been letting it through this whole time.
Wiring it in also surfaced two bugs I hadn't anticipated. The caller was making a wrong assumption about the shape of the data structure the verifier returned, and this only broke once real production data ran through it. One version of the bug would have silently logged genuine detections as "tool errors." The other would have silently crashed the step that writes results out, wiping an entire night's output. Neither was visible from reading the source β only running it for real exposed them.
Looking back 45 days
A single day's result right after wiring wasn't enough to tell whether this was a fluke or the norm. So I re-ran the same check retroactively across the past 45 days of reports.
The price axis was easy to check retroactively, since its ground truth persists. Re-running it across the past several weeks of trading days turned up almost nothing β consistent with what production had already shown.
The amount axis was harder. Since the ground-truth financial data block only survives for a single day, I couldn't do a direct retroactive comparison. Instead I used "a physically impossible amount" β a figure that exceeds the entire domestic stock market's total capitalization β as a proxy signal for an error.
The result: this class of error showed up every single day across all 45 days, without exception. The daily rate varied, but there wasn't one clean day in the whole window.
That number has to be read as a floor, not a full count. The proxy can only catch errors that inflate a value, so on the one day I could do a direct ground-truth comparison, the proxy caught only a fraction of what direct comparison found. The real error rate was almost certainly much higher.
This retrospective sweep also surfaced another gap in the verifier itself β a specific unit-scaling error pattern it hadn't been built to recognize at all. Building the verifier turned out to be an ongoing process of finding the verifier's own blind spots.
Why the old check was structurally blind to this class
To sum it up: the old self-consistency check only ever asked "does the report contradict itself?" It never went back and looked at the source data again.
The error class I found here came from mistranscribing a unit or a digit count while copying a number from source into the report. As long as the report then repeated that wrong number consistently, nothing inside the document ever contradicted itself.
"There's a check running" and "this specific error is being checked for" turned out to be two different sentences. Until I built the new verifier and actually ran it, I had no idea this error had been happening every day.
Generalizing it
If you're verifying output from an AI-generated report or any LLM pipeline, a few things from this are worth carrying over.
Self-consistency checks and ground-truth checks measure different things. A document not contradicting itself and a document being factually correct are separate claims. Having only one of the two means you're only half-verified.
Building the verifier isn't the finish line. Keep asking what error shape it's structurally incapable of seeing. Even while building this one, I found another blind spot in it along the way.
Decide the false-positive/false-negative trade-off on purpose. In an unattended system especially, a verifier that cries wolf too often does more damage than one that occasionally stays quiet. Pick a side deliberately and write down why.
Nothing is safe until it's wired into real data. The two interface bugs here were invisible on code review and only surfaced once real production data flowed through the pipeline.
When ground truth only partially exists, a proxy metric is fine β as long as you label it a floor. Knowing the proxy likely undercounts keeps you from mistaking that number for reassurance.
Every time I bolt on a new verifier now, "what is this thing still missing" goes straight onto the next checklist.
