How do you know a failing test is a real regression when the model you ask for a verdict is just as nondeterministic as the flaky suite? I spent 48 hours answering that exact question with a free server, free model access, and a small CI job that labels failing tests. The labels were the flakiest part of the system, and the fix was not a better model.
Why I tried it
A theme in dev discussions this week is that AI promoted everyone to reviewer without anyone testing the reviewer. I wanted to run that test on my own pipeline before asking a model near pull requests. The goal was narrow: read failed test output, decide REAL or FLAKY, and decide whether to page a human.
The 48-hour setup
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I used MonkeyCode's free model access and free server option because I wanted to see if a zero-budget stack could handle triage without wearing me out. I did not benchmark latency or model quality, and I won't pretend I measured a clear winner. The job ran after every push and again every four hours from the free server, which means I saw both cold starts and warm paths.
The job packed each failure into JSON: test name, error text, stack trace, the last 20 lines of stdout, and the diff for the file under test. It then asked the model to explain before labeling.
Explain first, label second
The most useful prompt change was forcing an explanation before the label. Without it, the model returned confident guesses and no reasoning I could audit. With it, I could compare explanations across runs.
You are a triage assistant for a CI system.
Explain the likely cause of this failure in two sentences.
Then print LABEL: REAL, FLAKY, or UNKNOWN.
If the cause is unclear, choose UNKNOWN.
That prompt alone made me trust the tool more, not because accuracy improved, but because I could see what the model was looking at.
The aggregator
One label per run was useless because the same test output sometimes produced different labels. I changed the job to keep every raw response and aggregate at the end.
# pseudo-code: keep raw labels, decide only after 2+ runs
from collections import Counter
def aggregate(labels):
counts = Counter(labels)
label, count = counts.most_common(1)[0]
if count >= 2:
return label
if label == "UNKNOWN":
return "UNKNOWN"
return "NEEDS_RERUN"
This is pseudo-code, not a tuned system. The key change was refusing to act on a single label.
Decision table
| Labels seen | Verdict | Action |
|---|---|---|
| REAL, REAL | REAL | keep the build red and page a human |
| FLAKY, FLAKY | FLAKY | quarantine the test for 24 hours |
| REAL, FLAKY | NEEDS_RERUN | run once more, then page |
| UNKNOWN, anything | UNKNOWN | don't block, log it |
The table is the real artifact of these 48 hours. The model was not the source of truth; the agreement rules were.
What broke
The model contradicted itself on identical inputs. Same stack trace, two runs apart, and the verdict flip-flopped. A single-label pipeline would have paged someone for nothing.
Model confidence was not calibrated. It used phrases like "almost certainly" for guesses, and the guess was often wrong. I stopped reading confidence text and started counting labels.
Time-based error messages confused it badly. Expired tokens and clock comparisons looked like real regressions to the model, even when the surrounding diff was unrelated.
The free server's cold start also turned into a failure. My first aggregator treated a missing model response as a test failure, which polluted the very data the model was supposed to clean. I had to separate infra errors from test errors before any labeling made sense.
What held up
The explain-then-label format held up because it gave me a way to disagree with the model. Logging raw responses held up because I could reconstruct why a decision was made.
Keeping the model out of the merge path held up the most. The model suggested actions, but the decision table executed them, and a human was always in the loop for REAL verdicts.
Who should not use this approach
If you do not have historical labels for a few past failures, you cannot calibrate the agreement rules. You will be guessing about the cutoff.
If your test suite is slow, waiting for two labels means waiting for two full runs. That cost can be bigger than the time you save.
If compliance or auditability matters, you need to store every prompt and raw response. The final label alone will not survive a review.
If your team ignores UNKNOWN, the whole system becomes theater. The unknown bucket only works when someone actually reads it.
What I'd do differently
I would spend the first six hours building a small golden set of past failures with known verdicts. That would tell me the actual disagreement rate before the experiment started.
I would also separate infrastructure failures from test failures at the collection layer, not inside the prompt.
And I would record every raw response before the model saw the next run. The labels lie, the logs do not.
The honest takeaway
The free model and free server were not the weakest link. My measurement was. The model was flaky in predictable ways, but my aggregator let me turn that noise into a usable triage channel.
If you run a similar experiment, keep the raw logs and start with a small script like the one above. Write down what the model says before you decide whether it matters.
