A single model gives you a single point of failure: when it's confidently wrong, you get no signal that it's wrong. A cheap, surprisingly effective guard is to ask the same question to a few independent models and use their agreement as a confidence signal.
The idea: fan the question out concurrently, then rank the answers by how much they agree with each other. When the models converge, you can trust the answer. When they diverge, you flag it for review instead of shipping a guess.
import asyncio
from difflib import SequenceMatcher
from pydantic_ai import Agent
MODELS = ['anthropic:claude-sonnet-4-6', 'openai:gpt-4.1']
async def answer(model: str, q: str) -> str:
return (await Agent(model).run(q)).output
async def ask(q: str):
answers = await asyncio.gather(*(answer(m, q) for m in MODELS))
# agreement = mean pairwise similarity of each answer to the others
def agree(a):
others = [b for b in answers if b is not a]
return sum(SequenceMatcher(None, a, b).ratio() for b in others) / len(others)
best = max(answers, key=agree)
return {'answer': best, 'agreement': round(agree(best), 2)}
If agreement is high, the models independently reached the same place — a strong signal. If it's low, you've caught a disagreement before it reached a user, and you can route to a human, ask a tie-breaker model, or return "uncertain".
You can make the similarity check smarter (embeddings instead of difflib), add a local model as a third vote, or weight by each model's self-reported confidence. But even this 20-line version turns "hope the model is right" into a measurable number you can gate on. Agreement isn't proof of truth — but disagreement is a reliable smoke alarm, and that's most of the value.
I package patterns like this as small open-source pydantic-ai + FastAPI templates — the repos are on GitHub, and complete, ready-to-run versions are on Gumroad. Feedback and issues welcome.
