Your Generated Docs Passed Lint. Did Any Example Actually Run?

Most AI-assisted documentation pipelines share one blind spot: the prose is linted, the links are checked, and the code blocks are never executed. A generated document can pass markdownlint, a link checker, and a spelling pass while shipping examples that crash on the first keystroke. The fix is not a better prompt; it is a mutation gate that treats every example as a test.

The current DEV conversation about AI turning every developer into a reviewer applies equally to documentation. The model became the writer, and nobody built a test for the writer. Structural checks measure formatting, not truth, so a doc review that stops at lint is reviewing the wrapper instead of the content. This article describes a reproducible pipeline where a free model drafts the prose and a small script executes every fenced block; a human then signs off only on the claims that automation cannot verify.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The drafting step in this workflow runs on MonkeyCode's free model access and its free server option, which keeps the marginal cost of a documentation pass at zero. The expensive part is the human review, and that is where the value lives.

Why lint is not evidence

Markdown linting verifies that headings are ordered, fences are closed, and line lengths are sane; it cannot know whether a requests.get call still matches the current library signature. Link checkers confirm that URLs resolve, yet they say nothing about whether the surrounding sentence is accurate. Spell checkers catch typos, not semantic drift. Each tool validates structure, and structure is the easiest part of documentation to get right.

The pipeline

1. Draft with a free model. Generate the first pass of prose, parameter tables, and examples using MonkeyCode's free model access on the free server; keep the session scoped to one page of docs so the model cannot drift across topics.
2. Extract every fenced code block. The script below pulls each block with its language tag and source line number.
3. Execute each block. Run it in a clean subprocess with a timeout so a hanging example fails fast.
4. Mutate each block. Change one assignment and re-run; if the mutated example still exits zero, the example asserts nothing.
5. Review claims by hand. The human owns version pins, security statements, and compatibility claims.
6. Publish with a gate report. Attach the pass/fail summary to the pull request so reviewers see what was actually executed.

The gate script

Save this as doc_gate.py and run it against any Markdown file:

#!/usr/bin/env python3
"""Execute every fenced code block in a Markdown file, then mutate each
block to prove the example actually asserts something."""

import re
import subprocess
import sys
import tempfile
from pathlib import Path

FENCE = re.compile(r"```

(\w+)?\n(.*?)

```", re.DOTALL)


def extract_blocks(markdown: str) -> list[tuple[str, str, int]]:
    blocks = []
    for match in FENCE.finditer(markdown):
        lang = match.group(1) or "text"
        code = match.group(2)
        line = markdown[: match.start()].count("\n") + 1
        blocks.append((lang, code, line))
    return blocks


def run_block(code: str, timeout: float = 10.0) -> tuple[int, str]:
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
        fh.write(code)
        path = fh.name
    try:
        proc = subprocess.run(
            [sys.executable, path],
            capture_output=True,
            text=True,
            timeout=timeout,
        )
        return proc.returncode, proc.stderr[-500:]
    finally:
        Path(path).unlink(missing_ok=True)


def mutate(code: str) -> str | None:
    match = re.search(r"^(\s*)([A-Za-z_]\w*)\s*=", code, re.MULTILINE)
    if not match:
        return None
    indent, name = match.groups()
    return code.replace(f"{indent}{name} =", f"{indent}{name}_mutated =", 1)


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: doc_gate.py README.md")
        return 2
    blocks = extract_blocks(Path(sys.argv[1]).read_text())
    failures = 0
    for lang, code, line in blocks:
        if lang not in {"python", "py"}:
            continue
        base_rc, _ = run_block(code)
        if base_rc != 0:
            print(f"[FAIL] line {line}: example does not run")
            failures += 1
            continue
        mutated = mutate(code)
        if mutated is None:
            print(f"[WEAK] line {line}: no assignment to mutate; example asserts nothing")
            failures += 1
            continue
        mutated_rc, _ = run_block(mutated)
        if mutated_rc == 0:
            print(f"[WEAK] line {line}: mutation still passes; example asserts nothing")
            failures += 1
        else:
            print(f"[OK]   line {line}: example runs and mutation is caught")
    total = sum(1 for lang, _, _ in blocks if lang in {"python", "py"})
    print(f"gate {'failed' if failures else 'passed'}: {failures} of {total} Python examples need attention")
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())

The script requires Python 3.10 or newer for the str | None annotation. A typical report on a generated README looks like this:

$ python doc_gate.py README.md
[FAIL] line 42: example does not run
[WEAK] line 67: mutation still passes; example asserts nothing
[OK]   line 89: example runs and mutation is caught
gate failed: 2 of 3 Python examples need attention

Reading the three outcomes

Every block produces one of three outcomes, and each outcome points to a different fix. [FAIL] means the example does not run in a clean environment, which is the most common failure mode for generated docs that reference uninstalled packages or renamed APIs. [WEAK] means the mutation still passes, which tells you the example is decorative: it computes something and never checks the result. [OK] means the example runs and the mutation is caught, which is the only outcome that adds real evidence to the review.

The gate measures the harness as much as the model, because a doc that extracts cleanly and fails on execution tells you more about your environment than about the prompt. When a generated document looks perfect, you are really evaluating the extraction and execution harness, not the model's writing. That distinction matters when you decide whether to spend another prompt iteration or fix the environment instead.

What the model may draft and what you own

Two rows deserve emphasis. Version pins should be banned from the model's draft entirely, because a free-tier model has no reliable memory of the current release line; the human pins versions from the actual environment. Security claims should be flagged, not trusted, because a plausible sentence about encryption is worse than no sentence when an auditor reads it.

Limitations

This gate is a smoke test, not a proof, and it should not be sold as more. A mutation that renames the first assignment catches only examples that depend on that assignment, so it will miss a block that runs successfully while computing the wrong result. The script also assumes Python, which means shell, SQL, and JavaScript examples need language-specific runners before they can join the gate. Free-tier models can produce plausible but outdated API usage, and the execution gate cannot detect that an example matches documentation that no longer exists; the human must compare against the current primary source.

Who should not use this

Teams documenting proprietary algorithms with examples that cannot run outside a licensed environment will find the execution step useless. Documentation with heavy external dependencies, such as examples that require a live database or a cloud account, needs a containerized fixture before this gate makes sense. If your team already uses formal verification or property-based tests for examples, this script is a step backward in rigor; treat it as a floor, not a ceiling.

The cheapest way to test this idea is to point the script at your oldest generated document and count how many examples survive mutation. If the first hour surfaces even one [WEAK] block, that single discovery justifies the gate. Start with the script above, add one mutation strategy per week, and let the failures tell you where the model's drafting actually needs supervision.