A Docs Agent on a Token Budget: What the Model Drafts, What the Human Owns

Code fails loudly. Documentation fails silently. A CLI change merges, the build is green, the tests pass, and the README still documents the old flag. Nobody notices until a developer burns forty minutes debugging against a lie.

A recurring theme in this week's DEV discussions: as AI generates more of the pipeline, the human role shifts to review — and the review process itself is the least tested part. Documentation is the sharpest version of that problem. The model drafts. The human reviews. But nobody defines what the review must catch.

This article is a documentation-generation workflow built on that gap. It defines what the model may draft, what a human must own, and how to verify the boundary with a deterministic script.

The problem: docs have no compiler

A hallucinated flag in code costs a CI run. A hallucinated flag in docs costs a developer 40 minutes of debugging against a lie.

LLMs are fluent. They will happily document a --retry flag that does not exist, or describe a default that was removed three releases ago. The output is confident, formatted, and wrong in ways that are hard to spot because the prose is plausible.

The fix is not better prompting. The fix is a boundary: a machine-checkable contract between what the model writes and what the human owns.

The contract: drafts vs. ownership

Before generating anything, split the documentation into two layers.

What the model may draft:

• First-pass explanations of code that exists
• Examples that mirror the source
• Changelog summaries from merged commits
• API reference skeletons
• Migration notes from a diff

What the human must own:

• Flag semantics and defaults
• Deprecation policy
• Security notes
• Compatibility promises
• Any sentence that commits the team to a behavior

The table is the contract. The interesting row is the last one: some artifacts should not be generated at all.

The artifact: a flag-diff validator

Here is a reproducible check that catches the most common documentation hallucination: flags that do not exist in the source.

Step 1: extract real flags

# extract_flags.py
import ast
from pathlib import Path

def extract_flags(path: Path) -> set[str]:
    tree = ast.parse(path.read_text())
    flags = set()
    for node in ast.walk(tree):
        if not isinstance(node, ast.Call):
            continue
        fn = node.func
        if not (isinstance(fn, ast.Attribute) and fn.attr == "add_argument"):
            continue
        for arg in node.args:
            if isinstance(arg, ast.Constant) and isinstance(arg.value, str) and arg.value.startswith("-"):
                flags.add(arg.value)
    return flags

Step 2: validate the draft

# validate_docs.py
import re
import sys
from pathlib import Path
from extract_flags import extract_flags

source_flags = extract_flags(Path(sys.argv[1]))
doc_text = sys.stdin.read()
doc_flags = set(re.findall(r"`(--?[a-zA-Z0-9-]+)`", doc_text))

invented = doc_flags - source_flags
missing = source_flags - doc_flags

if invented:
    print(f"FAIL: docs mention flags not in source: {sorted(invented)}")
if missing:
    print(f"FAIL: source flags missing from docs: {sorted(missing)}")
if invented or missing:
    sys.exit(1)

print(f"OK: {len(doc_flags & source_flags)} flags verified")

Step 3: constrain the model

You are a documentation writer for a CLI.
- Only describe flags that appear in the source file provided.
- Never invent a flag, default, or behavior.
- If a behavior is unclear, omit it.
- Output Markdown.

The pipeline is then:

# The agent call goes here; it produces draft.md
python validate_docs.py cli.py < draft.md

If the model invents a flag, the command exits 1. The build fails. The hallucination never reaches the README.

Why this fits a token budget

The validation step costs zero tokens. It is a deterministic diff, not a model call.

The generation step is bounded by source size. A 2,000-token CLI source, ten iterations of drafting and fixing, lands around 20,000 tokens. That is a rounding error on a 10M-token free allowance.

The pipeline above does not care which model generates the draft. It also runs comfortably on a free stack: MonkeyCode is an open-source agent project that offers free model access and a free server option (the operator reports a 10M-token free allowance). Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters for a scheduling reason. A docs agent earns its keep when it runs per merge or nightly. A paid server turns that into a line item; a free server turns it into a cron job.

Limitations: what the validator cannot see

The flag diff catches invented flags. It does not catch invented semantics.

A model can write "retries three times by default" when the code retries only with --retries. The flag exists, so the validator passes. The sentence is still a lie. That is why the human owns semantics, defaults, and side effects — the machine can only verify countable facts.

Other limits:

• Free allowances change. Verify current terms before wiring this into CI. A free tier is not an SLA.
• The extractor above handles argparse, not Typer or click. Adapt the AST walk for your framework.
• The regex assumes flags appear in backticks. Enforce that in the prompt.

Who should not use this workflow

• Teams under compliance sign-off requirements. The human owns the docs anyway, so the pipeline adds ceremony without shifting liability.
• Projects with a dedicated docs engineer and healthy docs. This workflow is for the majority of projects where docs are an afterthought.
• Marketing-facing documentation. A model draft reads like a model draft; tone is a human artifact.

The review checklist for the human

When the validator passes, the human still owns four checks:

1. Defaults — does every default in the doc match the code?
2. Side effects — does the doc say what each flag changes, not just what it is called?
3. Breaking changes — is the changelog classification correct? The model cannot know your compatibility policy.
4. Omissions — the prompt told the model to omit unclear behavior. Treat omissions as questions, not failures.

The division of labor

The model drafts. The validator checks the countable facts. The human owns everything else.

That division is the whole workflow. It fails the build on hallucinated flags, and it keeps the human in the only role that matters: ownership.

If you want to try it on your own CLI, the stack is open source — the free model access and free server option make the cost of a failed experiment zero.