I spent the last several months building out a financial data API
(XFINLAB) that pulls from about 20 different government/exchange sources
— SEC EDGAR, FINRA, CFTC, FDIC, USDA, CBOE, EIA, openFDA, CPSC, a couple
of crypto exchanges. Along the way I made one architectural rule that
ended up shaping almost every collector module: a field is either a
real value with a source, or it's null with an explicit reason. Never
an estimate presented as if it were real.
This sounds obvious written down. In practice it's a real design tax.
The tempting shortcut
Say you're building a collector for USDA agricultural commodity prices.
The USDA Quick Stats API doesn't always have this year's data yet for a
given commodity — sometimes the most recent real observation is from 8
months ago. The tempting move is to interpolate: draw a line between the
last two known points, guess where "now" would sit on it, ship a number.
It looks like better data. It demos better. And it's fabricated.
What we did instead
Every collector in this codebase follows the same shape:
def get_context_for_ticker(ticker: str) -> Optional[dict]:
if ticker not in _TICKER_TO_SOURCE_MAPPING:
return None # no linkage, not a guess
...
if fetch_failed:
return _load_persisted_fallback(ticker) # last REAL value we saw, honestly dated
return real_fresh_value
No collector has a code path that produces a plausible-looking number
that isn't traceable to an actual source and an actual fetch timestamp.
When a value truly isn't available, the API returns an explicit
"unavailable" with a reason (not configured / rate-limited / genuinely
no match for this ticker) — not a zero that looks like a real zero, and
not last year's number silently relabeled as current.
Making it checkable, not just claimed
Every collector self-registers into a small SQLite-backed registry at
import time:
register_source("usda_agriculture", "USDA Agricultural Commodity Prices", "agriculture")
with record_run_start / record_run_success / record_run_error
wrapping the actual fetch calls. That registry backs a public,
unauthenticated status page — https://www.xfinlab.com/trust.html — so
"this data source is honestly reporting, not guessing" isn't just a
claim in the docs, it's something anyone can go check live, updated in
real time, including which sources are currently down.
Why this matters more in 2026 than it used to
Q1 2026 reportedly saw $2.3B in trading losses tied to AI-generated
misstated earnings figures, and FINRA's 2026 Annual Oversight Report
devoted its first-ever dedicated section to AI/hallucination risk for
broker-dealers. 62% of enterprise AI users now cite hallucinations as
their top deployment barrier. None of that is about financial data APIs
specifically, but it's the same underlying failure mode: a plausible
number that isn't real, presented with no way to tell the difference.
The fix isn't really an AI problem to solve with a better model — it's
an interface design problem. Make the "I don't know" path a first-class,
explicit response, not a bug to route around.
If you want to see it live
The API's free tier is instant, no card — 20+ endpoints, Python/JS SDKs,
and an MCP server if you're wiring this into an agent:
https://www.xfinlab.com/intelligence-api.html
Curious how other API builders here handle this same tradeoff —
especially anyone working with genuinely gappy upstream sources. Do you
interpolate, forward-fill, or null-and-explain?
