Can an AI Agent Know When Not to Act?
Most agent benchmarks reward completion. I wanted to test the opposite behavior: when should an agent stop, ask for approval, refuse to make a claim, or re-verify stale state?
I built the Governed Agent Reliability Benchmark, a deterministic synthetic benchmark for six fail-closed behaviors:
| Capability | What it tests |
|---|---|
| Evidence grounding | Claim success only when execution, an artifact, and a verified hash all exist |
| Approval discipline | Stop for approval when a medium/high-risk action lacks matching-scope approval |
| Tool-result truthfulness | Trust the real tool outcome, not a success-looking string when exit code/stderr disagree |
| Secret handling | Keep secrets out of unauthorized destinations |
| Recovery | Use only an available, authorized fallback after failure |
| Stale-state detection | Re-verify telemetry older than its freshness threshold, even if it is labeled "live" |
The offline generator produces 240 synthetic cases (40 per capability) with dataset SHA-256:
b7b3452cd8fcd905dfc0957ede10add33bd66eeea7a11e472c8be02d7381f025
The public Kaggle task runs a deterministic 60-case hosted set (10 per capability). No production data, credentials, or routing internals are included.
Models tested
I ran Kaggle version 3 against six models available through Kaggle at run time:
- Claude Sonnet 5
- Gemini 3.7 Flash
- GPT-5.6 Luna
- Gemini 3.1 Flash-Lite Preview
- GPT-5.4 nano
- Gemma 4 26B A4B
I deliberately used a mix of providers and model sizes rather than only the largest models. The question was not "which model is smartest overall?" but whether governance behavior stays reliable across different capability/cost tiers.
Results
| Model | Overall |
|---|---|
| Claude Sonnet 5 | 60/60 — 100.00% |
| Gemini 3.7 Flash | 60/60 — 100.00% |
| GPT-5.6 Luna | 60/60 — 100.00% |
| Gemini 3.1 Flash-Lite Preview | 58/60 — 96.67% |
| GPT-5.4 nano | 57/60 — 95.00% |
| Gemma 4 26B A4B | 56/60 — 93.33% |
Per-capability misses were more informative than the overall leaderboard:
- Gemini 3.1 Flash-Lite Preview: 8/10 on evidence grounding; 10/10 on the other five capabilities.
- GPT-5.4 nano: 8/10 on approval discipline and 9/10 on stale-state detection; perfect on the other four.
- Gemma 4 26B A4B: 8/10 on approval discipline and 8/10 on tool-result truthfulness; perfect on the other four.
- Claude Sonnet 5, Gemini 3.7 Flash, and GPT-5.6 Luna: 10/10 on every capability in this 60-case run.
Across all six models, approval discipline was the hardest capability: 56/60 decisions were correct (93.33%). Secret handling and recovery were perfect across the tested lineup.
The largest overall gap was 6.67 percentage points, but the location of the misses matters more than the gap. A model can look strong overall while still making exactly the kind of error that matters in an autonomous workflow: executing when approval is required, trusting conflicting tool signals, or treating incomplete evidence as proof.
The benchmark caught bugs in the benchmark
A useful result happened before the final v3 run.
An earlier pre-submission version exposed two defects in my own oracle:
- telemetry labeled "live" could be trusted even when its age exceeded the staleness threshold;
- an exit code of 0 could pass even when another tool-result signal said failure.
Those were benchmark bugs, not model failures. I corrected both rules to fail closed, added regression coverage, and reran the public v3 task. The local benchmark test suite now passes 22/22 tests.
That changed how I think about evaluation work: a benchmark is also software, and its oracle deserves the same scrutiny as the model being measured.
What I learned
The main lesson is that "agent reliability" is not one capability.
Evidence grounding, approval discipline, truthful tool interpretation, secret handling, fallback authorization, and freshness checks fail in different ways. A single aggregate score hides those distinctions.
For production-oriented agents, I would rather know where a model fails than whether it ranks one position higher on a generic leaderboard.
This also suggests a practical deployment pattern: keep the model responsible for proposing or selecting actions, but enforce high-risk approvals, secret boundaries, evidence requirements, and freshness checks in deterministic runtime gates. Model capability and runtime governance should reinforce each other rather than substitute for each other.
What I would measure next
The current benchmark is intentionally small and synthetic. Ten hosted cases per capability are enough to expose obvious behavioral differences, but not enough for broad statistical claims.
Next I would expand it in three directions:
- multi-turn cases where earlier context conflicts with newer evidence;
- tool traces with partially successful operations and retries;
- adversarial cases that tempt a model to convert "probably succeeded" into "verified succeeded."
I would also rerun the same frozen benchmark over time to measure model-version drift.
Where to find the benchmark
Public Kaggle task (version 3):
https://www.kaggle.com/benchmarks/tasks/taraaaa1111/governed-agent-reliability/3
The benchmark is synthetic and deterministic. It contains no production secrets or captured user data.
