The most common agentic system failure I encounter in production is not a bad prompt. It is not a context overflow. It is a tool that changed without its registration changing.
I have seen this cause weeks of debugging in systems that were working fine until they weren't — and because the failure is silent, teams often spend time looking at the model first.
What Tool Schema Drift Looks Like
Most agentic frameworks let you register tools with a name, a description, and a JSON schema for parameters. The model reads the description to decide when to call the tool. It reads the schema to know what parameters to send.
# Initial registration — looks fine
tools = [
{
"name": "search_entities",
"description": "Search the entity registry by name. Returns a list of matching entities.",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search term"}
},
"required": ["query"]
}
}
]
Six months later, the underlying function has changed. The team added a required entity_type filter after learning that unfiltered searches returned too many results. They updated the implementation. They forgot to update the registration.
# Current implementation — mismatched with registration
def search_entities(query: str, entity_type: str) -> list[dict]:
# entity_type is now required — "company", "person", "location"
return registry.search(query=query, type=entity_type)
Now the model calls search_entities with {"query": "Samsung"}. The function raises a TypeError because entity_type is missing. Or worse — if you have a default fallback somewhere upstream — it silently runs with the wrong parameters and returns results the model was not expecting.
The model sees a response. It generates text from whatever came back. No exception fires. The output just drifts from what it should have been.
Why This Is Hard to Catch
The failure mode is subtle for two reasons.
First, evals usually test output quality, not the call-response cycle. If your eval checks whether the final answer is correct, a tool that silently misbehaves can still produce plausible-looking output — especially if the task has any ambiguity in what "correct" means.
Second, the description is what the model acts on, not the schema. A schema mismatch causes a runtime error. A description mismatch causes behavioral drift — the model calls the tool in situations where it should not, or does not call it when it should, and there is no error signal.
Both failure modes happen in production. Neither shows up in type checking or unit tests.
Three Checks That Catch Drift Before Production
1. Validate the response shape, not just the call shape
Most frameworks validate that the model produced a well-formed tool call (correct JSON, required fields present). Fewer validate that the tool's response matched what the model was told to expect.
from pydantic import BaseModel, ValidationError
from typing import Any
class EntitySearchResult(BaseModel):
entity_id: str
name: str
entity_type: str
confidence: float
def call_tool_with_validation(tool_name: str, args: dict) -> Any:
raw_response = dispatch_tool(tool_name, args)
# Validate response matches what the model expects
if tool_name == "search_entities":
try:
validated = [EntitySearchResult(**item) for item in raw_response]
return validated
except ValidationError as e:
# Response shape changed — surface this immediately
raise ToolResponseSchemaError(
f"Tool '{tool_name}' returned unexpected shape. "
f"Registration is out of sync with implementation.\n{e}"
)
return raw_response
This makes the mismatch loud. A ToolResponseSchemaError is unmistakable. A silently wrong answer is not.
2. Version the tool description alongside the implementation
Treat the description as part of the implementation contract, not as documentation.
The practical pattern: keep tool schemas in a registry file under version control, and make updating the registry a required step when any tool function signature or return shape changes. Code review that accepts a function change without a registry update is accepting a potential drift.
# tools/registry/v2.py — versioned alongside implementation
TOOL_REGISTRY = {
"search_entities": {
"version": "2.0.0",
"description": (
"Search the entity registry by name and type. "
"Returns matching entities with confidence scores. "
"entity_type must be one of: company, person, location."
),
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"entity_type": {
"type": "string",
"enum": ["company", "person", "location"]
}
},
"required": ["query", "entity_type"]
},
"response_schema": {
"type": "array",
"items": {
"type": "object",
"required": ["entity_id", "name", "entity_type", "confidence"]
}
}
}
}
If the interface changes in a backward-incompatible way — new required parameter, different return shape — give it a new name (search_entities_v2) rather than silently updating the existing entry. Agents that depended on the old interface continue to work until they are explicitly migrated.
3. Add canary evals that cover the full call-response cycle
A single eval prompt per tool is enough to catch schema drift in CI.
# tests/evals/test_tool_contracts.py
import pytest
def test_search_entities_contract():
"""
Canary eval: triggers the tool and validates the full cycle.
If search_entities changes its interface, this breaks before production does.
"""
agent_response = run_agent(
prompt="Find all company entities named Samsung in the registry.",
tools=TOOL_REGISTRY,
)
# Check the tool was called
tool_calls = extract_tool_calls(agent_response)
search_calls = [c for c in tool_calls if c["name"] == "search_entities"]
assert len(search_calls) > 0, "Agent did not call search_entities"
# Check the call used the current schema
for call in search_calls:
assert "entity_type" in call["args"], (
"Agent called search_entities without entity_type — "
"tool description may be out of sync with schema"
)
# Check the response matched expected shape
assert agent_response.tool_errors == [], (
f"Tool returned unexpected shape: {agent_response.tool_errors}"
)
This eval does not test whether the final answer is good. It tests whether the tool call cycle completed without a schema mismatch. That is exactly what breaks first when a tool drifts.
The Root Cause
The reason this happens repeatedly is that tool descriptions live outside the usual code review discipline. They are strings in a config or a dict literal. Nobody has a linter that flags "the function signature changed but the description did not."
The fix is not complicated. It is a process discipline problem more than a technical one. The registration is the contract between the model and the implementation. Versioning it like a contract — with change control, backward-compatibility rules, and automated validation — closes the gap that produces silent drift.
The agent is not wrong when it mishandles an unexpected tool response. It was not told the tool changed. If you treat the registration as the contract, you start maintaining it like one.
I work on entity resolution and agentic infrastructure at er-api.hannune.ai. If you have seen this failure mode in your own systems, curious what the trigger was.
