Score Before You Read: A Review Budget for AI-Generated Documentation

Human review is now the most expensive stage of AI-assisted documentation, yet most teams still read generated sections in the order the generator printed them. This article presents a four-signal scoring model that ranks every section of a Markdown doc by risk, plus a small Python script that turns the ranking into a review queue with an enforced budget. The workflow assumes drafts are nearly free to produce; the scarce resource is the reviewer's attention, and it should be spent where failures actually surface.

The DEV discussion "AI promoted every developer to reviewer" describes the pressure accurately: generation volume grows, but review capacity stays flat because it is human time. Earlier articles in this series showed that generated doc examples can fail a real execution gate, and that a run-before-merge check catches those failures mechanically. What those articles did not answer is where the human should spend attention after the machine check passes, and that is the question the budget model addresses.

Why ranking beats equal review

An equal-time review assumes every paragraph carries the same risk, which is rarely true for API-style documentation. A quickstart with a runnable block, four strong claims, and an environment variable can mislead a new user for hours, while a deep reference page reproduced from type signatures may need only a mechanical check. Scoring makes that asymmetry visible before a human opens the diff, so the team can agree on what deserves a named reviewer and what does not.

The four signals

Each section receives a score from zero to eight across four signals. Execution measures whether the section contains a runnable fenced code block; claims counts strong verbs such as return, throw, require, and guarantee; exposure reflects how early the section appears in the reading path; coupling counts external URLs, environment variables, and exported names that can drift silently.

The thresholds are deliberately coarse because the goal is priority, not precision. A section that scores zero to two qualifies as model-draft, meaning automated checks are sufficient and no human is required. A score of three to five requires one named reviewer, while anything above five becomes human-owned, so a human must author or rewrite the section rather than merely approve it.

The review budget script

The script below parses a Markdown file into heading-anchored sections, computes the four signals for each, and prints a ranked report. Its exit code fails the pipeline when the number of sections demanding human review exceeds the team's agreed budget, so the discussion about scope happens in the pull request rather than after merge.

#!/usr/bin/env python3
'''review_budget.py -- rank generated doc sections by review risk.'''

import re
import sys
from pathlib import Path

CLAIM_VERBS = [
    'return', 'throw', 'raise', 'guarantee', 'require',
    'ensure', 'must', 'always', 'never', 'support', 'accept',
]
EXTERNAL_TOKENS = ['https://', 'export ', 'env[', 'getenv', 'API_KEY']


def split_sections(text: str):
    '''Split a Markdown file into heading-anchored sections.'''
    sections, current = [], {'title': '(preamble)', 'level': 1, 'lines': []}
    for line in text.splitlines():
        if re.match(r'^#{1,6} ', line):
            if current['lines']:
                sections.append(current)
            current = {
                'title': line.lstrip('# ').strip(),
                'level': len(line) - len(line.lstrip('#')),
                'lines': [],
            }
        else:
            current['lines'].append(line)
    if current['lines']:
        sections.append(current)
    return sections


def score_section(sec):
    '''Return (total, verdict, execution, claims, exposure, coupling).'''
    text = sec['title'] + '\n' + '\n'.join(sec['lines'])
    runnable = re.search(r'^```

(python|bash|sh|js|ts)\b', text, re.M)
    execution = 2 if runnable else (1 if text.count('

```') >= 2 else 0)

    claims = sum(1 for stem in CLAIM_VERBS
                 if re.search(rf'\b{stem}\w*', text, re.I))
    claim_density = min(2, claims // 2)

    lowercase = text.lower()
    if 'quickstart' in lowercase or sec['level'] == 1:
        exposure = 2
    elif sec['level'] <= 3:
        exposure = 1
    else:
        exposure = 0

    coupling = min(2, sum(1 for token in EXTERNAL_TOKENS if token in text))
    total = execution + claim_density + exposure + coupling
    verdict = 'model-draft' if total <= 2 else (
        'human-review' if total <= 5 else 'human-own')
    return total, verdict, execution, claim_density, exposure, coupling


def main(path: Path, budget: int) -> int:
    sections = split_sections(path.read_text())
    rows = [score_section(sec) for sec in sections]
    required = sum(1 for row in rows if row[1] != 'model-draft')
    print(f'# Review budget for {path.name} (limit {budget})')
    for sec, (total, verdict, *parts) in zip(sections, rows):
        print('- {:02d} {:12s} {} {}'.format(total, verdict, sec['title'], parts))
    print(f'Human reviews required: {required} of {len(sections)} sections')
    return 0 if required <= budget else 1


if __name__ == '__main__':
    path, budget = sys.argv[1], int(sys.argv[2])
    sys.exit(main(Path(path), budget))

The workflow in practice

The budget model is useful only when drafting is cheap enough to regenerate on every pull request.

1. Draft everything with the cheapest generation tier you trust.
Generous drafting is the foundation of the model, because you cannot score a section that was never generated. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If your team already uses MonkeyCode, its free model access and free server option let you regenerate the whole doc set for each pull request without standing up extra infrastructure; any other free tier your team trusts works the same way.

2. Run the report in CI on changed files.
Add python review_budget.py README.md 2 to the workflow so every pull request prints its ranked queue before a human opens the diff. The non-zero exit code fails the build when the required review count exceeds the budget, which forces an explicit decision instead of silent approval of generated prose.

3. Apply the verdict policy.
A model-draft section passes when the existing example-execution gate is green, and no human is required. A human-review section needs one named reviewer who records Reviewed by @name in the pull request description. A human-own section needs Reviewed by @name (YYYY-MM-DD) inside the section itself, and the merge must not happen without it.

4. Re-score after every merge.
Generated docs drift faster than code because generators rewrite whole files. A section that scored model-draft in January accumulates claims, environment variables, and code blocks by March, and its score climbs above the human-own threshold. Re-running the script on each merge catches that drift before anyone trusts an outdated verdict.

A sample run

Consider a single file with two sections:

## Quickstart

Run this to check your credentials:

```python
import sample
print(sample.ping())
```

The client returns a session object, requires a token, ensures the connection is live, and never reuses an expired session.

## Reference: Client

The client accepts an API key and raises an error on failure. The key comes from os.getenv('API_KEY') and must be set before the client starts.

With a budget of one reviewer, the report prints:

# Review budget for quickstart.md (limit 1)
- 06 human-own    Quickstart [2, 2, 2, 0]
- 04 human-review Reference: Client [0, 1, 1, 2]
Human reviews required: 2 of 2 sections

The exit code is 1, so the team either raises the budget to two or moves the credentials paragraph out of the quickstart to reduce its claim density. Both outcomes are more informative than a silent read of every paragraph in print order.

Limitations and honest boundaries

The heuristic rewards precise sections and punishes vague ones, which means an untitled preamble with no heading, no code, and no claims can still land in model-draft territory because exposure alone is worth two points. That is exactly the kind of ambiguous prose a human should read, so the script must be paired with a mandatory read of any section titled (preamble). The model also assumes API-style documentation with consistent headings; marketing pages, design rationales, and narrative tutorials have different failure modes and different signals.

Any free generation tier may rate-limit batch regeneration during busy hours, so teams should schedule generation jobs instead of sending every commit through a free endpoint. The approach is wrong for legal documents, security runbooks, and any page where a mistaken claim carries liability, because a score can never excuse the missing human. Small projects with three documentation pages do not need a budget; ranking adds ceremony without saving time.

The conclusion

The budget model does not remove the human from documentation; it removes the human from the parts where a machine check is stronger. Score first, read second, and re-score on every merge, and the review queue will reflect risk instead of print order. If you adopt the script, tune the point tables against your own docs for a week before trusting the verdicts.