You shipped your LLM-powered feature. It worked great in testing. Then users started reporting hallucinations, inconsistent outputs, and responses that completely ignored their instructions. Sound familiar?
I've been there. Three times in the last year alone. The problem isn't that LLMs are unreliable — it's that most of us are flying blind once our AI features hit production. We don't have the observability, evaluation, or guardrail infrastructure that we'd never skip for a traditional backend service.
Let me walk through how I stopped guessing and started actually debugging my LLM applications.
The Root Cause: You Can't Debug What You Can't See
With a traditional API, debugging is straightforward. You check logs, look at status codes, trace the request through your system. With LLM applications, the failure mode is completely different.
Your API returned a 200. The response was valid JSON. The model even sounded confident. But the answer was wrong, or it leaked context from another user's session, or it ignored a critical instruction in your system prompt.
The root causes usually fall into three buckets:
- Prompt drift — your prompts work for your test cases but fail on real-world input patterns you didn't anticipate
- Context window mismanagement — you're stuffing too much (or too little) context, and the model loses track of what matters
- Missing guardrails — there's no validation layer between the model's output and your user
The fix isn't just "write better prompts." It's building proper infrastructure around your LLM calls.
Step 1: Add Tracing to Every LLM Interaction
Before you can fix anything, you need visibility. Every call to an LLM should be traced — the full prompt, the response, latency, token counts, and any metadata about the user's session.
Here's the pattern I use with OpenAI's Python client, wrapping calls with trace context:
import time
import json
import logging
logger = logging.getLogger("llm_tracing")
def traced_completion(client, messages, model="gpt-4", **kwargs):
trace_id = f"trace_{int(time.time() * 1000)}"
start = time.perf_counter()
# Log the full request for later analysis
logger.info(json.dumps({
"trace_id": trace_id,
"type": "llm_request",
"model": model,
"messages": messages,
"params": kwargs
}))
response = client.chat.completions.create(
model=model,
messages=messages,
**kwargs
)
duration = time.perf_counter() - start
result = response.choices[0].message.content
# Log the response alongside the request trace
logger.info(json.dumps({
"trace_id": trace_id,
"type": "llm_response",
"content": result,
"duration_ms": round(duration * 1000),
"tokens_used": response.usage.total_tokens
}))
return result, trace_id
This is the bare minimum. In production, you want this data flowing into something queryable — not just log files.
Open-source platforms like FutureAGI provide end-to-end tracing, evaluation, and guardrail infrastructure specifically for this problem. It's Apache 2.0 licensed and self-hostable, which matters if you're dealing with sensitive data. But even if you roll your own, the principle is the same: trace everything.
Step 2: Build Evaluation Pipelines, Not Just Unit Tests
Here's where most teams get stuck. You can't unit test an LLM the way you test a function. The output is non-deterministic. So what do you do?
You build eval pipelines. The idea is simple: maintain a dataset of input-output pairs that represent what "good" looks like, and continuously run your prompts against them.
import json
def run_eval(client, eval_dataset_path, prompt_template):
with open(eval_dataset_path) as f:
dataset = json.load(f)
results = []
for case in dataset:
messages = [
{"role": "system", "content": prompt_template},
{"role": "user", "content": case["input"]}
]
response, trace_id = traced_completion(client, messages)
# Score using your criteria — could be exact match,
# semantic similarity, or even another LLM as judge
score = evaluate_response(
response,
case["expected_output"],
criteria=case.get("criteria", "accuracy")
)
results.append({
"input": case["input"],
"expected": case["expected_output"],
"actual": response,
"score": score,
"trace_id": trace_id # link back to the full trace
})
passing = sum(1 for r in results if r["score"] >= 0.8)
print(f"Eval results: {passing}/{len(results)} passing")
return results
The key insight: your eval dataset should grow over time. Every production failure you catch? Add it to the dataset. Every edge case a user reports? That's a new eval case. After a few months, you'll have a regression suite that actually reflects how your app is used.
Step 3: Add Output Guardrails
Tracing tells you what happened. Evals tell you if things are getting worse. But guardrails prevent bad outputs from reaching users in the first place.
I keep my guardrails simple and composable:
class GuardrailPipeline:
def __init__(self):
self.checks = []
def add_check(self, name, check_fn):
self.checks.append((name, check_fn))
def validate(self, response, context=None):
failures = []
for name, check_fn in self.checks:
passed, reason = check_fn(response, context)
if not passed:
failures.append({"check": name, "reason": reason})
return len(failures) == 0, failures
# Example checks
def no_pii_leak(response, context):
"""Catch common PII patterns in the output"""
import re
patterns = [
r'\b\d{3}-\d{2}-\d{4}\b', # SSN
r'\b\d{16}\b', # credit card (simplified)
]
for pattern in patterns:
if re.search(pattern, response):
return False, f"PII pattern detected: {pattern}"
return True, None
def response_not_empty(response, context):
if not response or not response.strip():
return False, "Empty response"
return True, None
# Wire it up
pipeline = GuardrailPipeline()
pipeline.add_check("no_pii", no_pii_leak)
pipeline.add_check("not_empty", response_not_empty)
Run this between your LLM call and your user-facing response. When a check fails, you can retry with a modified prompt, return a fallback response, or flag it for human review.
Step 4: Close the Feedback Loop
The most important step is one that most teams skip: feeding production observations back into your eval datasets and prompt iterations.
Here's the workflow that actually works:
- Trace every LLM call in production
- Flag low-quality responses (via guardrails, user feedback, or automated scoring)
- Add flagged cases to your eval dataset
- Iterate on prompts using the eval pipeline
- Deploy and repeat
This isn't a one-time setup. It's a continuous loop, and it's the difference between an LLM feature that degrades over time and one that gets better.
Prevention: What I Do on Every New LLM Project Now
After getting burned enough times, here's my checklist before any LLM feature goes to production:
- Tracing from day one — not after the first incident
- At least 50 eval cases before launch, covering happy paths and known edge cases
- Output guardrails for PII, format validation, and content policy
- A fallback strategy — what happens when the model fails? Users should see a graceful degradation, not a hallucination
- Cost monitoring — runaway token usage can wreck your budget overnight
If you want a batteries-included solution rather than building all this from scratch, check out FutureAGI. It bundles tracing, evals, simulations, datasets management, a gateway, and guardrails into a single self-hostable platform under an Apache 2.0 license. It's the kind of thing I wish existed when I first started shipping LLM features.
But whether you use a platform or build your own, the principle is the same: treat your LLM like any other critical system dependency. Observe it, test it, and put safety nets around it. Your users — and your on-call rotation — will thank you.
