Three perfect scores weren't enough: testing AI outage decisions one fact at a time

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

My first website-outage benchmark gave three models perfect scores. That was useful feedback about the benchmark, but it did not help choose between the models.

The question comes from my WebSecOps focus: given the evidence and permissions in an incident report, what should happen next? Sometimes the answer is to act. Sometimes a missing observation or permission should change that answer.

The first version used five fictional incidents covering DNS, TLS, deployment rollback, backup recovery and an incomplete outage report. Each model selected an action and a supporting evidence statement, then gave a short explanation. Three shuffled answer orders produced 15 responses per model. All three scored 15/15.

The follow-up changed the design. It uses six matched pairs, with one observation changed inside each pair. The incident description and available choices stay identical. Both the keyed action and evidence statement change.

For example, the rollback pair holds the outage, database migration and operator permissions constant:

The other pairs concern DNSSEC diagnostics, cached versus origin errors, backup validation, DNS-change approval and queued-job durability. Each variant runs in a fresh conversation, so the model cannot see its partner. Matched variants use the same option positions for each of three shuffled orders.

That gives 36 responses per model across six authored pairs, not 36 independent incidents. The cases and deterministic scorer were frozen before the follow-up calls. The follow-up itself was informed by the pilot's ceiling; it is not an untouched holdout from the original study.

Models Tested

I kept the three models selected before the pilot: one available model from each of three providers, with no claim that they represent each provider's strongest offering.

• Gemini 3.7 Flash — google/gemini-3.7-flash
• Claude Haiku 4.5 — anthropic/claude-haiku-4-5@20251001
• GPT-5.4 mini — openai/gpt-5.4-mini-2026-03-17

All runs used Kaggle's platform defaults, without sampling overrides, on September 24, 2026. The original pilot, its saved task reruns and the follow-up are separate result sets; they are not pooled to make the sample look larger.

A response earns one point only if it follows the exact JSON schema and selects both keyed choices. The explanation is retained but is not automatically judged. Infrastructure errors would invalidate a run rather than become wrong answers. All 108 follow-up responses were retained, matched to the frozen prompts and rescored locally; the aggregates match Kaggle's task results.

Findings

The follow-up separated the models on this particular task:

The last column is deliberately demanding: one miss in any of a pair's six responses prevents that pair from passing. Haiku's three misses occur in three different pairs. They are not three instances of the same failure, and they are not three demonstrated unsafe actions.

First, recognizing the evidence did not always produce the keyed action. In the DNS case where both normal and validation-disabled queries return SERVFAIL, Haiku selected the correct statement that the test had not isolated DNSSEC validation. In order 47, it nevertheless chose to prioritize DS/DNSKEY inspection. The key calls for a broader resolution trace before prioritizing that defect. Its other two answer orders passed this variant.

Second, extra caution can disagree with an action key. In the cache case, a simultaneous cache-bypass request returns the expected page while the normal request serves a cached 503. The key prioritizes evicting the affected path and verifying recovery. Haiku recognized the successful bypass evidence, but in order 47 chose to inspect origin health first, arguing that one successful request does not establish universal health.

That is a useful limitation of my scoring design. The key expresses a particular immediate-action priority under the supplied conditions. Additional diagnostics can be defensible; this result does not establish that Haiku's caution is unsafe. A future version should have independent operators review such alternatives before deciding whether to accept multiple actions or award partial credit. I kept the original key for this run.

Third, a format failure can hide correct choices. In the durable-queue case, Haiku selected both correct IDs but added an unrequested reason2 field. The prompt required exactly action_id, evidence_id and reason, so the frozen scorer gave zero. It was the only invalid-format response across all 108 calls. Ignoring the extra field would recover that one point, but that would be a different scoring contract.

These distinctions change how I read the headline 91.7%: two action-key disagreements and one schema violation are more informative than a single undifferentiated failure rate.

The order variants also expose variability that a single prompt would miss. They do not prove that option position caused the differences: I did not repeat each exact prompt enough to separate position effects from sampling variability.

The two perfect follow-up scores have limits too. These are short multiple-choice exercises with explicit runbooks and some easy distractors. The models did not investigate a live outage, execute a change, handle new evidence over time or demonstrate recovery. The evidence choices measure recognition of appropriately scoped claims, not general confidence calibration. Six authored pairs cannot establish a general model ranking.

What I would measure next is narrower than “more questions”: expert agreement on the disputed actions, repeated runs of the same prompts, and a staged incident where the model must request missing evidence before proposing a change. Those would test gaps that this benchmark leaves open.

AI tools drafted the cases, implemented and executed the evaluation, analyzed the retained outputs and wrote this article. No independent expert validation or manual review by a human is claimed. All incidents are fictional; no customer data or real infrastructure changes are involved.

My Benchmark

Website outage decisions: pilot and paired cases on Kaggle

The benchmark displays the two task scores separately, without a pooled overall score. Read the task rows: Kaggle currently renders 0.00 in the model headers with its “No overall score” setting; those header values are not additional measured results. It includes the pilot task, version 3 and the paired task, version 1. Open Compare Outputs on the paired task to inspect each model's 36 conversations, including the three Haiku cases discussed above.

The paired notebook publishes the full corpus, key, scorer and run exports. The notebook output view can reflect a particular model run; use the registered task’s Compare Outputs view for all three models’ conversation traces. The Kaggle Benchmarks SDK supplies task registration and model execution; the case content and scoring logic were created for this entry. The work is public under Apache 2.0.

For reproduction: order seeds are 11, 29 and 47. The frozen paired corpus/scorer SHA-256 is 0def44fe0c0e9d483487ecaaa0b8a8ccba4a30c8127b02e11e3a91d1eab34295.