Last Thursday, a colleague approved a generated query that returned exactly the rows the ticket asked for. The query also scanned fourteen million rows to find them, and the production dashboard timed out at 9:14 a.m. Nothing locked and nothing corrupted, which is why the incident never reached the postmortem. The reviewer checked correctness and missed cost, and nobody on the team could say how often that happens.
Recent engineering discussions keep circling the same observation: AI coding tools promoted every developer to reviewer, but almost nobody tests the reviewer. The claim is easy to repeat and hard to verify, because review accuracy is rarely measured and almost never published. This article treats that gap as an engineering problem with a structured debate, a reproducible harness, and a decision rule.
The Free-Tier Question
MonkeyCode is an open-source AI coding assistant whose current offer includes free model access, a published free tier of ten million tokens, and a free server option for small projects. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The interesting question is not whether the offer is generous; it is whether free compute can make SQL review measurably safer. Two credible positions exist, and both deserve evidence before you pick a side.
Position A: Free Compute Makes the Reviewer Measurable
A blind calibration harness costs almost nothing when tokens and a sandbox server are free, so the measurement can run weekly instead of annually. One round with twenty queries at roughly two thousand tokens per query consumes about forty thousand tokens, which funds roughly two hundred and fifty rounds on the published ten-million-token tier. The arithmetic is simple, but the organizational effect is not, because teams can finally track whether review accuracy improves after training, after a checklist change, or after a new model starts generating SQL.
The free server matters for a different reason. It removes the standard excuse that there is nowhere to run the harness, and it keeps calibration out of the production cluster where it could cause real damage. A team that measures its reviewers will find blind spots, and finding blind spots is the precondition for fixing them.
Position B: Free Tiers Measure the Reviewer, Not the Risk
A sandbox database has no concurrency, no lock contention, and no fourteen-million-row tables, so a reviewer who performs well in calibration can still approve a query that destroys production. The harness in this article would flag the missing WHERE clause, but it cannot reproduce the lock chain that a real UPDATE would trigger under load.
Token grants also create a coverage illusion. Ten million tokens feels like a budget, but it says nothing about whether the team can interpret the results or whether the free server reflects production characteristics. Calibration decays as well; a reviewer who scores ninety percent in August can drop to sixty percent by November if the harness stops running. Free compute pays for the measurement, but it does not pay for the discipline to keep measuring.
The Decision Rule
The two positions are not contradictory once you separate the artifact from the environment. Free compute is excellent for measuring the reviewer and terrible for simulating production, and the decision rule follows directly. Adopt the free tier for weekly calibration and smoke gates, and never as the only control; keep a human gate for writes and DDL, and re-run calibration whenever the model or the schema changes.
| Situation | Use the free tier for | Keep a human gate for |
|---|---|---|
| Small team, Postgres, no dedicated DBA | Weekly blind calibration, smoke tests on generated SQL | All DDL and bulk UPDATE/DELETE |
| Regulated data (PII, PCI) | Calibration on a synthetic schema only | Any query that touches real data |
| High-write OLTP workload | Read-only query evaluation | Writes, locks, migrations |
| Nobody can read an EXPLAIN plan | Nothing; learn plans first | Everything, until the team can interpret cost |
The Harness: A Blind Calibration for SQL Reviewers
The harness below is a condensed version of the one I run for review calibration. It targets Postgres 14 or newer, works on the free server option or any local sandbox, and never executes a write outside a rolled-back transaction. This is not another regression job for generated SQL; it is a calibration harness for the human who approves it.
- Create the sandbox schema with
python calibrate_reviewer.py init. - Define cases with ground-truth verdicts; the file ships with five, and you should grow the set from real incidents.
- Run every case through
EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON), wrapping UPDATE and DELETE statements in a transaction that rolls back. - Emit
review_pack.jsonwith plans but no verdicts, so the reviewer cannot see the ground truth. - Ask the reviewer to classify each query as
correct,slow, ordangerousin a separate file. - Score the verdicts and print accuracy with
python calibrate_reviewer.py score review_verdicts.json.
#!/usr/bin/env python3
"""calibrate_reviewer.py — measures how accurately humans review generated SQL.
This is a reviewer calibration harness, not a model benchmark. It runs a
small set of SQL cases through EXPLAIN, hides the ground truth, and scores
the human who reviews them.
Usage:
python calibrate_reviewer.py init
python calibrate_reviewer.py run
python calibrate_reviewer.py score review_verdicts.json
"""
import json
import subprocess
import sys
from pathlib import Path
DSN = "postgresql://localhost:5432/review_sandbox"
SCHEMA = """
CREATE TABLE IF NOT EXISTS customers (
id INT PRIMARY KEY,
region TEXT NOT NULL,
signup_date DATE NOT NULL
);
CREATE TABLE IF NOT EXISTS orders (
id INT PRIMARY KEY,
customer_id INT NOT NULL REFERENCES customers(id),
total NUMERIC(12, 2) NOT NULL,
status TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL
);
"""
CASES = [
{
"id": "ok_indexed_filter",
"sql": "SELECT id FROM orders WHERE customer_id = 42 AND status = 'paid' "
"ORDER BY created_at DESC LIMIT 10;",
"verdict": "correct",
"note": "Sargable predicate on an indexed foreign key.",
},
{
"id": "missing_where",
"sql": "UPDATE orders SET status = 'archived';",
"verdict": "dangerous",
"note": "No WHERE clause; touches every row in the table.",
},
{
"id": "cartesian_join",
"sql": "SELECT c.id, o.total FROM customers c JOIN orders o "
"ON c.region = o.status;",
"verdict": "dangerous",
"note": "Join predicate compares unrelated columns.",
},
{
"id": "non_sargable",
"sql": "SELECT id FROM orders WHERE DATE(created_at) = CURRENT_DATE;",
"verdict": "slow",
"note": "Function on the column prevents index usage.",
},
{
"id": "implicit_cast",
"sql": "SELECT id FROM customers WHERE signup_date = '2026-08-26';",
"verdict": "correct",
"note": "Literal comparison is safe and index-friendly.",
},
]
def psql(sql: str) -> str:
result = subprocess.run(
["psql", DSN, "-At", "-c", sql],
capture_output=True,
text=True,
check=True,
)
return result.stdout.strip()
def explain(sql: str) -> dict:
if sql.lstrip().upper().startswith(("UPDATE", "DELETE")):
sql = f"BEGIN; EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) {sql}; ROLLBACK;"
else:
sql = f"EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) {sql};"
for line in psql(sql).splitlines():
if line.startswith("["):
return json.loads(line)[0]["Plan"]
raise RuntimeError("no plan found in EXPLAIN output")
def cmd_init() -> None:
psql(SCHEMA)
print("sandbox schema ready")
def cmd_run() -> None:
pack = []
for case in CASES:
plan = explain(case["sql"])
pack.append(
{
"id": case["id"],
"sql": case["sql"],
"plan": {
"node_type": plan.get("Node Type"),
"total_cost": plan.get("Total Cost"),
"estimated_rows": plan.get("Plan Rows"),
},
}
)
Path("review_pack.json").write_text(json.dumps(pack, indent=2))
print("review_pack.json written; verdicts are hidden until scoring")
def cmd_score(verdicts_path: str) -> None:
truth = {case["id"]: case["verdict"] for case in CASES}
human = json.loads(Path(verdicts_path).read_text())
rows = []
for case in CASES:
verdict = human.get(case["id"], "missing")
rows.append(
{
"id": case["id"],
"human": verdict,
"truth": truth[case["id"]],
"match": verdict == truth[case["id"]],
}
)
correct = sum(row["match"] for row in rows)
print(f"reviewer accuracy: {correct}/{len(rows)}")
for row in rows:
marker = "ok " if row["match"] else "MISS"
print(f"{marker} {row['id']}: human={row['human']}, truth={row['truth']}")
if __name__ == "__main__":
command = sys.argv[1]
if command == "init":
cmd_init()
elif command == "run":
cmd_run()
elif command == "score":
cmd_score(sys.argv[2])
else:
raise SystemExit("usage: calibrate_reviewer.py init|run|score <file>")
Reading the Report
Accuracy alone is not the whole story. A reviewer who marks everything dangerous will score well on a pack full of dangerous cases and uselessly on a pack full of correct ones, so track false positives separately from false negatives. The script prints per-case matches precisely so you can see which bug classes the reviewer misses; a MISS on non_sargable means the team needs more training on index usage, not more tokens. Re-run the same pack monthly to measure decay, then rotate in new cases from real incidents so the pack does not become a memorized exam.
When This Approach Fails
The harness has three honest limits. First, the sandbox does not simulate production: no concurrency, no lock contention, and tiny row counts, so a query that scores well here can still be catastrophic at scale. Second, reviewer accuracy is not reviewer vigilance; a reviewer can score well on known bug classes and miss a novel one. Third, published free tiers can change, so verify the current token allowance and server terms before building a workflow around them.
Do not use this approach if your team has no one who can author ground truth by reading an EXPLAIN plan, because the calibration is only as good as its labels. Do not use it if your production data cannot be mirrored synthetically, since the harness should never run against real customer data. And do not expect the free tier to replace load testing, staging, or a DBA review gate; it measures reviewers, not production.
If you want to see how your own review instincts hold up, fork the script, run it against the free server option, and classify the pack before you read the notes. The verdicts are usually humbling, and that is exactly the point: the reviewer is the last line of defense, and the last line deserves a measurement.
