MY Edit or Abstain: AI Code-Repair Decision Benchmark

This is a submission for the Kaggle Benchmarking Challenge

For this challenge, I wanted to test a practical question that often comes up in real software debugging:

When a test fails, does the AI know whether the production code should actually be changed?

A failing test does not always mean there is a bug in the implementation. Sometimes the real problem is a missing fixture, an outdated test expectation, an environment mismatch, an unclear specification, or conflicting evidence.

So I created a benchmark called Edit or Abstain: AI Code-Repair Decision Benchmark.

Instead of asking models to generate a patch, I asked them to make one of three decisions:

• EDIT
• NO_EDIT
• NEED_MORE_EVIDENCE

The benchmark includes realistic Python debugging situations such as missing test files, stale tests, CI environment problems, conflicting behavior, boundary bugs, timezone assumptions, ambiguous regressions, and clear production defects.

What interested me most was not only whether a model could recognize a bug, but whether it could show restraint when changing the code was not justified.

For me, that is an important part of reliable AI-assisted software development: knowing when to fix, when to leave the code alone, and when to first gather more evidence.

I tested the benchmark across eight different models:

• Gemini 3.7 Flash
• Claude Haiku 4.5
• DeepSeek-R1
• Gemma 4 31B
• GLM-5
• GPT-5.6 Luna
• Qwen 3 Coder 480B
• Qwen 3 Next 80B Thinking

I deliberately chose models from different families rather than testing several closely related versions of the same system.

The group includes general-purpose models, reasoning-focused models, coding-oriented models, and open models. This gave me a broader view of how different systems approach the same software-repair decision.

I used the same benchmark tasks for every model so the comparison would remain fair. I was especially interested in whether stronger reasoning or coding specialization would help models distinguish between a real implementation bug, a bad test, an environment problem, or a situation where the evidence was simply not strong enough to justify a change.

The most interesting result was that the models were generally very good at the obvious cases, but the differences became clearer when the evidence was ambiguous.

Across the first nine tasks, six models completed every task correctly:

• Gemini 3.7 Flash — 9/9
• Claude Haiku 4.5 — 9/9
• Gemma 4 31B — 9/9
• GLM-5 — 9/9
• GPT-5.6 Luna — 9/9
• Qwen 3 Coder 480B — 9/9

Qwen 3 Next 80B Thinking scored 6/9, while DeepSeek-R1 scored 4/9.

Overall, the models made 64 correct decisions out of 72, which is about 88.9%.

The easiest tasks were the clear ones. When the implementation obviously contradicted the documentation or a test was clearly broken, most models handled the decision correctly.

The harder tasks were the ones that required restraint.

For example, some models struggled when:

• the documentation and tests pointed in different directions;
• a regression could have been caused by either production code or the environment;
• the evidence was incomplete;
• or the safest answer was NEED_MORE_EVIDENCE rather than immediately changing the code.

That surprised me because I expected the more difficult cases to be the actual coding bugs. Instead, the more challenging behavior was deciding when not to act.

For me, that is the main insight from this benchmark: a reliable coding assistant should not only be good at finding and fixing bugs. It should also be able to recognize uncertainty, avoid unnecessary edits, and know when more evidence is needed before making a change.

If I continue this benchmark, I would like to add more tasks involving flaky tests, dependency regressions, incomplete issue reports, conflicting changelogs, backward compatibility, and cases where several fixes look possible but only one is safe.

I would also like to compare these decision-only results with a second version where the models must generate the actual code patch after making the decision.

My Benchmark

You can explore the full benchmark on Kaggle here:

Edit or Abstain: AI Code-Repair Decision Benchmark

The benchmark contains the decision-making tasks used in this project and lets readers compare how different models performed on the same software-repair scenarios.

I designed it to focus on a simple but important question: should the model edit the code, leave it unchanged, or ask for more evidence?

The benchmark is public so the tasks, model comparisons, and results can be reviewed directly on Kaggle.