Every CI failure lands in someone's lap, and on a small team that someone is usually you. I wanted to know whether a free model could handle the boring first pass: read the logs, guess the cause, and decide if a human should look. So I built a tiny triage bot, connected it to my project's failing jobs, and let it run for 48 hours. These are the field notes from that window, including the parts I would rather forget.
The Setup That Made It Cheap
The experiment was cheap by design, which shaped every decision I made, including the ones that broke. I ran the bot on MonkeyCode's free server option and sent its calls through the free model access, so the whole thing cost nothing except my attention. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The point here is not to sell you on a provider; it is to show what breaks when you trust a free pipeline with real failures.
Hour 9: The Naive Version Hits the Rate Limit
Monday morning started with the simplest possible design: every failing job became a model call, and every model call returned a paragraph of prose. It worked for about ninety minutes, and then calls started failing with rate-limit errors because the whole team pushed at once. The bigger problem was that the prose was useless for automation, because a human can act on a paragraph and a script cannot.
# Illustrative: adapt the request shape to your provider's client.
def naive_triage(logs: str) -> str:
prompt = f'Here is a CI failure. What is the root cause?\n\n{logs[:4000]}'
response = client.chat.completions.create(
model='<free-model-id>', # from your provider's docs
messages=[{'role': 'user', 'content': prompt}],
)
return response.choices[0].message.content
Three lessons landed in the first morning, and they all pointed in the same direction: stop treating the model like a free-text oracle. Raw logs are too long and too noisy, so the model wasted tokens on stack traces that did not matter. Every call is a latency bet, and a queue of twenty failures turns into a very long wait. The model also hallucinated a root cause for a flaky test with total confidence, which is worse than no answer at all.
Hour 14: The Pre-Filter Changes the Math
The fix was embarrassingly simple, and it came down to one rule: do not call the model unless the failure is actually ambiguous. I wrote a deterministic pre-filter that classifies each job by exit code, log pattern, and changed files, and only the leftovers reach the model. The decision table below is the part I would copy into any project, regardless of provider.
| Signal | Action |
|---|---|
ModuleNotFoundError in logs |
Rule: dependency issue, no model call |
Exit code 124 and timeout in the test name |
Rule: mark flaky, rerun once |
Changed files are all *.md
|
Rule: docs only, no model call |
Exit code 1 and panic: in logs |
Model call: high priority |
| Anything else | Model call: normal priority |
def should_call_model(failure: dict) -> bool:
if 'ModuleNotFoundError' in failure['logs']:
return False
if failure['exit_code'] == 124 and 'timeout' in failure['test_name']:
return False
if all(p.endswith('.md') for p in failure['changed_files']):
return False
return True
The effect was immediate and a little humbling, because most failures are the same boring problems repeating themselves; roughly two thirds of the failures never reached the model. Why spend a model call on a problem the logs already solved? The model only saw the interesting cases, which made its answers easier to trust and much cheaper to produce. I would repeat this step in any future experiment, and I would do it on day one instead of hour fourteen.
Hour 27: Structured Output, Then the Confidence Trap
The next change was to force the model into a JSON object with a schema, so the bot could act on the result instead of printing it. I added a summary, a confidence score, and a suggested action, and I validated every response with JSON Schema before anything else touched it. That part worked exactly as intended, and the trap was hiding in the confidence score I had added so casually.
import json
import jsonschema
SCHEMA = {
'type': 'object',
'required': ['summary', 'confidence', 'suggested_action'],
'properties': {
'summary': {'type': 'string', 'maxLength': 200},
'confidence': {'type': 'number', 'minimum': 0, 'maximum': 1},
'suggested_action': {'enum': ['fix', 'rerun', 'investigate', 'ignore']},
},
}
def parse_model_output(raw: str) -> dict | None:
try:
data = json.loads(raw)
jsonschema.validate(data, SCHEMA)
return data
except (json.JSONDecodeError, jsonschema.ValidationError):
return None
By hour thirty, I had collected a pile of confident wrong answers, and every one of them was beautifully formatted. The model said confidence: 0.95 on a diagnosis that was completely backwards, because it had never seen the test's setup code. The schema guaranteed the shape of the answer, not the truth of it, and I had confused those two things. The fix was to treat low confidence as a reason to escalate, but never to treat high confidence as a reason to skip the human.
Hour 38: The Kill-Switch and the Feedback Log
By the second evening I added two small safety rails that cost almost nothing to build. The first was a kill-switch file: if the file exists, the bot skips every model call and falls back to a plain 'needs human' message. The second was a JSONL event log that recorded every decision, every model response, and every human correction.
import pathlib
import json
import time
KILL_SWITCH = pathlib.Path('/tmp/stop_model_calls')
def model_enabled() -> bool:
return not KILL_SWITCH.exists()
def log_event(entry: dict) -> None:
with open('triage_events.jsonl', 'a') as f:
f.write(json.dumps({'ts': time.time(), **entry}) + '\n')
The kill-switch saved me once when the free server restarted and my bot came back broken; one touch /tmp/stop_model_calls stopped the bleeding instantly. The JSONL log became the most valuable artifact of the whole experiment, because it let me replay every wrong answer after the fact. I could see exactly which failure patterns fooled the model and which ones the pre-filter had already caught. That, more than anything, is the difference between a demo and a debugging session.
What I Would Repeat, and What I Would Never Do Again
- Repeat the pre-filter: deterministic rules are cheaper, faster, and more honest than a model call for the same recurring failure.
- Repeat the JSONL log: if you cannot replay a bad decision, you cannot fix it.
- Repeat the kill-switch: a free server will restart at the worst possible moment, and you want a one-command off switch.
- Never trust the confidence score: it is a number the model invented, not a measurement.
- Never send raw logs without truncation: the token budget goes to noise, and the answer gets worse.
Limitations and Who Should Not Do This
This approach is for small teams with tolerant failure modes, not for anyone whose CI gates a release. A free model will occasionally be wrong, and a wrong triage label is annoying, but a wrong release decision is an incident. If you need an audit trail for compliance, or if your logs contain secrets, do not send them to any external model endpoint without redaction. And if your team already has a reliable triage process, the bot is a toy; the pre-filter and the log are the parts worth keeping.
Would I run the same experiment again? Yes, but with the pre-filter from the start, the kill-switch from hour one, and more respect for confident wrong answers. The free model access and the free server option made the whole thing possible without a budget meeting, which is why I could afford to break it in public. If you try this yourself, start with the JSONL log; future you will want to know what actually happened.
