README examples go stale the moment you merge a breaking change. Users copy-paste, hit an error, and open an issue that is really a documentation bug. This case study shows a small detector that uses a free model to catch that drift before users do.
The problem: docs decay silently
A library's README is its front door. When it lies, every new user pays the tax. The typical failure looks like this: a function was renamed in v2.0, the README still shows the old name, and the issue tracker fills with "your example doesn't work."
Manual review does not scale. You can read your own README a hundred times and still miss what changed. You know what you meant to write, not what you wrote.
Goal
The detector had four requirements:
- Extract code examples from a README.
- Compare them against the actual API in the codebase.
- Flag mismatches with a reason, not just a warning.
- Run on free tiers, because this is a maintenance tool, not a product.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option; the current free allowance is 10 million tokens at the time of writing. A drift check that runs on every release fits comfortably inside that budget.
Why a model instead of static analysis
Static analysis can catch renamed functions if you have a symbol table. It cannot catch semantic drift: a parameter that changed meaning, a return type that changed shape, an import path that moved. A model can read both sides and judge whether they still agree.
The trade-off is nondeterminism. The model might miss a drift or flag a false positive. That is acceptable for a maintenance tool, because the output is a report for a human, not an automated gate.
Implementation
The detector has three stages: extract, compare, report.
Stage 1: extract code blocks
import re
def extract_code_blocks(readme_path):
with open(readme_path, encoding="utf-8") as f:
content = f.read()
blocks = re.findall(r"```
(?:python)?\n(.*?)
```", content, re.DOTALL)
return [b.strip() for b in blocks if b.strip()]
Deliberately simple. Fenced blocks with a python tag are the target; other languages are ignored for now.
Stage 2: extract API calls
def extract_api_calls(code_block):
calls = re.findall(r"\b(\w+)\.(\w+)\s*\(", code_block)
return [f"{obj}.{method}" for obj, method in calls]
A crude heuristic, but enough for a first pass. It finds client.post(...), parser.add(...), and friends. It misses dynamic dispatch, which is fine — the model gets the full block anyway.
Stage 3: let the model judge
import json
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["MONKEYCODE_API_KEY"],
base_url=os.environ["MONKEYCODE_API_BASE"],
)
def check_block(code_block, api_summary):
prompt = f"""
You are reviewing a README example against the real API of a library.
README example:
{code_block}
Actual API (signatures only):
{api_summary}
For each API call in the README, decide:
- MATCH: the call exists and the signature is compatible.
- DRIFT: the call is renamed, removed, or has a breaking signature change.
- UNKNOWN: you cannot tell from the information given.
Return JSON:
{{"calls": [{{"call": "...", "status": "MATCH|DRIFT|UNKNOWN", "reason": "..."}}]}}
"""
resp = client.chat.completions.create(
model=os.environ["MONKEYCODE_MODEL"],
messages=[{"role": "user", "content": prompt}],
)
return json.loads(resp.choices[0].message.content)
The prompt asks for three states, not two. UNKNOWN matters because it prevents the model from guessing when the API summary is incomplete.
Stage 4: generate the report
def generate_report(readme_path, api_summary):
report = []
for i, block in enumerate(extract_code_blocks(readme_path), 1):
result = check_block(block, api_summary)
drifted = [c for c in result["calls"] if c["status"] == "DRIFT"]
if drifted:
report.append({"block": i, "drifted": drifted})
return report
The report lists only drifted blocks. A clean README produces an empty report, which is the best kind.
Example run
Here is the output on a sample library where create_client() was renamed to Client() in v2.0:
[
{
"block": 3,
"drifted": [
{
"call": "lib.create_client",
"status": "DRIFT",
"reason": "create_client was removed in v2.0; use Client() constructor instead."
}
]
}
]
That is exactly the kind of comment a user would file as an issue. Catching it before release turns a support ticket into a one-line fix.
Limitations
- The extractor is regex-based. It misses examples that build calls dynamically. A proper AST parser would help, but adds complexity.
-
The model can hallucinate. A
DRIFTverdict on a healthy call is possible. The report is for a human, so this is tolerable, but do not wire it to an automated deploy gate. -
API summaries must be generated. In the demo,
inspect.signature()dumps the public surface. For large libraries, that output can exceed the model's context window; chunking is required. - Not for private code. Sending proprietary API signatures to an external model may violate policy. Run a local model instead; the prompt stays the same.
Lessons learned
-
Three states beat two. Adding
UNKNOWNreduces false positives, because the model stops guessing when the API summary is incomplete. - The report is the product. The model's raw output is noisy; the filtered report is what a maintainer actually reads.
- Free tiers are enough. A per-release drift check on a small library is well within the 10-million-token free allowance. The cost of maintenance tooling should be near zero, and here it is.
Try it
The full script is about 80 lines. If you maintain a Python library, point it at your README and your public API, and see what it finds. MonkeyCode's free model access and free server option are a reasonable place to run it.
