A few weeks ago I was demoing a small FastAPI service that called an LLM to summarize text. Worked great in every test I ran. Then, live, the provider API took almost 9 seconds to respond and my endpoint just... hung. No error, no timeout, just silence while I sat there making small talk to fill the gap.
That was the moment it clicked for me: when you put an AI API behind your own API, you've inherited two sets of failure modes — yours and theirs. Rate limits, timeouts, malformed JSON in the response, the model deciding to add a chatty preamble before the JSON you asked for. None of that is hypothetical. It's Tuesday.
Here's how I've learned to structure a FastAPI backend so that when (not if) the AI call misbehaves, it fails in a way you control.
Don't let the router touch the AI call directly
The most common pattern I see (and the one I used to write) is catching exceptions right inside the endpoint function. It works until you have more than one endpoint calling the model, and suddenly your error handling is copy-pasted five times with five slightly different bugs.
Instead, the router's only job is to receive the request, validate it, and hand it off:
@router.post("/summarize")
async def summarize(payload: SummarizeRequest, service: SummaryService = Depends()):
result = await service.run(payload)
return result
All the "what could go wrong" logic lives one layer down, in the service.
Give the AI provider's failures their own exception types
Generic except Exception blocks tell you nothing about why something failed, which makes writing a sane HTTP response back to your client impossible. I define a small hierarchy instead:
class AIProviderError(Exception):
"""Base error for anything the model provider throws at us."""
class AIProviderTimeout(AIProviderError):
pass
class AIProviderRateLimited(AIProviderError):
pass
class AIResponseMalformed(AIProviderError):
"""The provider responded, but not with what we asked for."""
The service layer catches the provider SDK's raw exceptions and re-raises them as one of these. A dedicated exception handler at the app level then maps each one to the right HTTP status: 504 for a timeout, 429 for rate limits, 502 when the model's output doesn't parse. Your client gets something actionable instead of a raw 500 and a stack trace.
Never trust the model's output — validate it like a client request
This is the one people skip. We're all trained to validate incoming requests with Pydantic, but the response coming back from the AI is just as untrusted. Models occasionally return almost-JSON, or valid JSON with a field renamed, or an extra sentence wrapped around it.
class SummaryResult(BaseModel):
summary: str
key_points: list[str]
def parse_model_output(raw: str) -> SummaryResult:
try:
data = json.loads(raw)
return SummaryResult.model_validate(data)
except (json.JSONDecodeError, ValidationError) as e:
raise AIResponseMalformed(str(e)) from e
If it doesn't fit the schema, it doesn't get past this line — full stop. That one function has saved me from shipping garbage data downstream more times than I'd like to admit.
The pattern, not the code
None of this is complicated once you see it laid out, but I didn't see it laid out anywhere when I needed it — I pieced it together across a few projects, mostly the hard way, after enough live demos went sideways. Routers that stay thin, a service layer that owns the provider calls, a typed exception hierarchy, and Pydantic validating both directions instead of just one.
I ended up packaging exactly this structure — routers, services, schemas, and the error-handling layer between your FastAPI app and an AI model — into a small starter template so I'd stop rebuilding it from scratch every time. If you want the full working reference instead of piecing it together yourself, you can grab it here. Worth noting: it's a structural pattern, not a full production app — no JWT auth or database included, on purpose, so it stays easy to read and adapt to whatever you're building.
I'm also currently open to backend/full-stack roles (Python, FastAPI, Flask, AI integrations) and freelance work — feel free to check out more of my code on GitHub.
