This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
My AI agent was misreading financial data 1 in 5 times. No crashes. No exceptions. Just quiet, dangerous wrong numbers.
Some bugs scream. Null pointer exceptions. Segmentation faults. Crash logs that light up your screen like a Christmas tree.
This one whispered.
The Setup
I'm building an AI agent that controls Android phones using Gemma 4. Natural language commands get parsed into structured JSON, translated into ADB instructions, and executed on-device. No cloud. No API keys. Just a phone that does what you tell it.
By Day 16, I'd hit a milestone: the agent completed its first multi-app workflow. "Copy my bank balance and send it to Mom on WhatsApp."
Three apps. Banking → WhatsApp. Seven steps. Success.
I celebrated. Shipped the code. Published the build log. Went to sleep feeling like an engineer.
Then I tested it again the next morning. And again. And again.
The Whisper
Out of 50 test runs, the balance was wrong 10 times. Twenty percent.
The agent wasn't crashing. No exceptions were thrown. The logs showed success. But "₦15,000" was quietly becoming "₦15.000" or "₦15,00" or "₦15000." A currency symbol mistaken for a digit. A comma parsed as a decimal. A zero dropped from the end.
The wrong number was being sent to Mom. And I had no idea for two weeks because nothing was failing loudly enough to make me look.
The Root Cause
I traced the problem to a single bad assumption: I was treating OCR output as ground truth.
Banking apps scored F on my accessibility audit. Zero UI labels. The agent had to rely entirely on optical character recognition to read anything on screen. And banking apps are a hostile environment for OCR:
- Condensed fonts designed for dense data, not readability
- Grey text on slightly-darker-grey backgrounds
- Currency symbols (₦) that OCR models frequently confuse with digits
- Commas in large numbers that OCR sometimes parses as decimal points
One screenshot. One OCR pass. One number stored in memory. No verification. No double-check. No sanity check. I had built a pipeline that trusted a single, unreliable data source with financial information.
The Fix: Defense in Depth
I didn't try to make the OCR better. Tesseract and ML Kit are what they are. Instead, I built a three-layer verification system that treats every financial number as unverified until proven otherwise.
Layer 1: Format Validation
Before a number is accepted, it must match a valid currency pattern. The regex checks for currency symbols (₦, $, £, €) followed by properly formatted digits. Anything that doesn't match gets rejected instantly.
python
pattern = r'[₦$£€]?\s?\d{1,3}(?:,\d{3})*(?:\.\d{2})?'
matches = re.findall(pattern, extracted_text)
if not matches:
return False, "", "No currency values found"
This alone catches about half the errors garbled text, partial reads, and non-numeric data that OCR sometimes returns.
Layer 2: Double-Read Confirmation
The agent captures two separate screenshots and runs OCR on both independently. If the numbers match, the value is accepted with high confidence. If they differ, a third screenshot is captured and the best-of-three wins.
# Read twice
text1 = extract_text(screenshot1)
text2 = extract_text(screenshot2)
num1 = extract_number(text1)
num2 = extract_number(text2)
if num1 == num2:
return True, num1, "High confidence"
# Tiebreaker
text3 = extract_text(screenshot3)
num3 = extract_number(text3)
if num1 == num3:
return True, num1, "Medium confidence (2/3)"
elif num2 == num3:
return True, num2, "Medium confidence (2/3)"
This catches transient OCR errors—pixel-level variations between screenshots that produce different readings. The cost is 4-6 extra seconds per financial read. The benefit is catching errors that would otherwise silently corrupt data.
Layer 3: Range Validation
Even a correctly formatted, double-confirmed number can be nonsensical. A balance of "₦0" might mean an empty account—or it might mean the OCR failed completely and returned nothing. A balance of "₦999,999,999,999" is almost certainly an error.
if 1 <= value <= 1_000_000_000:
return True
else:
return False # Suspicious. Re-read.
This catches the edge cases—numbers that pass format and double-read checks but are clearly wrong in context.
The Celebration
After implementing all three layers, I re-ran the 50-test battery. The error rate dropped from 20% to 6%. For the specific "₦15,000" test case, 20 out of 20 reads were correct with double-read confirmation.
The agent now succeeds 19 times out of 20 on multi-app financial tasks. The one failure is a timeout (banking app loads slowly on an old device), not a misread. And critically, when verification fails, the agent now knows it failed. It reports low confidence and aborts the task instead of silently sending wrong data.
What This Taught Me
The bug wasn't in the OCR engine. It was in my architecture. I had built a system that trusted a single, unreliable data source with financial information. That wasn't a Tesseract problem. That was a design problem.
The fix wasn't a better model. It was a better process. Three cheap, fast checks that together create confidence where no single check could. Defense in depth applied to data integrity.
This changed how I think about every feature I build. I now ask: "If this component silently returns wrong data, will I know? If not, where's the verification?"
The Code
The full verification system is open source on GitHub:
👉 github.com/Dexter2344/phone-agent
vision.py contains the verify_financial_data() function. agent.py calls it before storing any financial value in task memory.
The Build Log
I've been documenting this project publicly for 17 days on Dev.to. Every breakthrough, every dead end, every "why is this broken" debugging session at 1 AM.
This bug was Day 17. The one that almost shipped. The one that whispered.
It's not the loudest bug I've ever caught. But it's the one that taught me the most.
This is my entry for the Dev.to Smash Stories contest. The Gemma 4 model powers the agent's command parsing and reasoning. The verification layer is my own work.
