swarm-gym for goose: grading local models by running the code they write

Key takeaways

• swarm-gym is a Python harness (python -m harness ...) that drives goose local-edition's coding swarm, then grades what it built by RUNNING it, not by trusting the model.
• Two modes: benchmark (a frozen 3-spec suite for reproducible, variant-paired A/Bs) and exploratory (a brain invents work and vibes follow-ups, with idle-node monitoring and auto-tuning).
• The brain is pluggable — a Claude judge, a local Qwen, or the operator (you) via a plain req/resp file handshake with no API key.
• Verification is seven dimensions across four sources: a deterministic golden-check core, cluster/fleet health, an AI judge, and static diagnostics.
• It is a flywheel: measure, diagnose a fleet or drift problem, flip a swarm flag or tune a knob, then re-run the same frozen ruler.

A small local model is a confident liar. Ask a 27B-class coder running on your own hardware to build a CLI app and it will happily report "All tests pass, the feature works" — while the binary it wrote crashes on the first real command, or its own unit tests pass around a feature that was never implemented. If you grade a coding swarm of these models by reading its final message, you are grading its confidence, not its software.

swarm-gym is the answer. It is the self-driving test harness for goose local-edition — the fork of goose whose swarm runs a fleet of small local models as one coordinated coding team. This tutorial walks you through the whole thing: set it up, run both of its modes, read every output it produces, and use what you learn to tune the swarm. By the end you will be able to benchmark a model build, drive an open-ended session as the brain yourself, and read a verdict down to the individual failing check.

The one principle everything follows from: the harness never trusts the model's self-report. It re-runs the produced app against golden values committed before the run, and grades on what the software actually does. A green test suite the model wrote is not evidence; the built binary printing the right number is.

Setup

The harness lives under evals/swarm-gym/ in the goose local-edition repo and runs as a Python module. You need three things: the Python environment, a built swarm binary, and a brain.

1. Create the environment cd evals/swarm-gym, then python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt. Everything is invoked as python -m harness .
2. Build the binary under test the harness shells out to swarm.binary, which defaults to ../../target/debug/goose. Build it with cargo build -p goose-cli from the repo root. This is the thing being measured, so it must exist.
3. Pick a brain the component that invents tasks and (optionally) judges them. claude (default) uses the Anthropic SDK and reads ANTHROPIC_API_KEY from the environment; local points at an LM Studio endpoint and needs no key; operator lets you be the brain with no key and no SDK. More on all three below.
4. (Optional) MCP secrets if you want workers to use tools like context7 (library docs) or web-search, list them under swarm.mcp and set the matching secret env vars. Secrets always come from the environment or a local .env, never from a committed file.

Nothing about the harness config holds a secret. config.yaml names models and endpoints; ANTHROPIC_API_KEY, CONTEXT7_API_KEY, WEBSEARCH_BEARER and friends live in your shell or .env.

The core idea: grading by running

Every spec the harness runs carries deterministic golden checks — commands run against the finished app whose output must equal a specific, pre-decided value. The model's reasoning, its claimed test results, the tidiness of its code — none of it enters the score. Only this: you run the command, and you either get the golden value back or you don't.

Here are three real checks, verbatim from the frozen benchmark suite, one per archetype:

# crud / invtrack — an inventory store. Add two items, then the total value must be 25.
python3 -m invtrack --db bench.db item add A1 --name Bolt --qty 10 --price 2 && \
python3 -m invtrack --db bench.db item add A2 --name Nut  --qty 5  --price 1 && \
python3 -m invtrack --db bench.db report value | grep -qE '(^|[^0-9])25([^0-9]|$)'

# compute / rdcalc — a recursive-descent calculator. ^ is right-associative, so 2^3^2 = 512, not 64.
python3 -m rdcalc eval '2^3^2' | grep -qE '(^|[^0-9])512([^0-9]|$)'

# txn / txkvbench — a transactional KV store. A nested ROLLBACK must undo only the inner scope.
python3 -m txkvbench --db bench.db exec 'SET x 1; BEGIN; SET x 2; GET x; ROLLBACK; GET x; COUNT' \
  | tr '\n' ' ' | grep -qE '2.*1.*1'

That last one is the whole philosophy in a line. 2 1 1 is the only correct output: the inner GET x sees 2, the ROLLBACK discards it, the outer GET x sees 1 again, COUNT is 1. A model can write a beautiful transaction class with a passing unit test and still print 2 2 1 here because it merged the savepoint the wrong way. The harness catches that; the model's self-graded suite does not.

The two modes

The harness runs in two families, tagged on every ledger record so you always know which one produced a result.

Use benchmark mode to measure one model build against a fixed bar and compare variants. Use exploratory mode to discover where the swarm breaks, seed apps, exercise MCP tools, and auto-tune the fleet.

Tutorial part 1 — your first benchmark

Benchmark mode replays the frozen suite through graduated tiers, tagged with a --variant label. Start with a smoke run:

python -m harness bench --tier smoke --variant mlx      # 1 spec, 1 rep — "did I wire it up right?"
python -m harness bench --tier medium --variant mlx     # 3 specs × 5 reps = 15 runs, a real signal
python -m harness bench --tier medium --variant gguf    # same suite, same checks, the other build
python -m harness bench-report                           # head-to-head + writes runs/BENCHMARK.md
python -m harness bench-csv                              # export benchmark-runs.csv + benchmark-summary.csv

The tiers scale the same three specs from a sanity check to a paired A/B you can trust:

The runs interleave — spec 0, spec 1, spec 2, spec 0, … — so a campaign cut short still covers all three archetypes evenly. The suite is graded entirely by the deterministic golden checks, so benchmark mode needs no brain and no API key.

The frozen suite is frozen on purpose: editing a prompt or a golden check silently breaks every prior comparison paired against it. New coverage goes in a new suite. Note too that bench accepts --turns, --no-judge, and --tweak but ignores them — they only do something in exploratory mode.

Reading the benchmark output

bench-report regenerates runs/BENCHMARK.md from the ledger (so MLX and GGUF accumulate into one file) and the two CSVs give you the raw numbers:

benchmark-runs.csv — one row per run: variant, tier, spec, app_slug, session_id, wall_secs, functional, overall, checks, swarm, cluster, diagnostics, tasks_done, tasks_failed, exit_code. The functional column is the honest bottom line: 1 only if the checks dimension passed — i.e. the app actually did the golden thing.

benchmark-summary.csv — per variant×spec plus an ALL roll-up: pass percentages and wall-clock median/mean/p90/stdev/min/max.

Tutorial part 2 — an exploratory session

Exploratory mode is where a brain invents work and reacts like a real user. The simplest entry point is one session:

python -m harness once --archetype heavy-spec --turns 6      # a densely-specced new app + follow-ups
python -m harness once --archetype minimal-spec              # a terse one-liner; score the gap-filling
python -m harness once --archetype continue-existing         # extend a previously-green app, with regression checks
python -m harness loop --n 3                                 # cycle the three archetypes, seeds 1000, 1001, 1002
python -m harness report                                     # print the last 30 sessions from the ledger

once takes --archetype (default heavy-spec), --persona, --seed (default 1), --turns (falls back to the config's default_turns, 6), --no-judge, and --tweak. loop runs --n sessions (default 3), cycling the archetypes and bumping the seed each time.

The three exploratory archetypes

Each archetype tells the brain to invent a different kind of task, chosen to stress a different swarm weakness:

For continue-existing, the substrate is a real app kept on disk from an earlier green session (the ledger tracks which apps are still passing). From turn two onward the brain issues follow-ups drawn from a fixed vocabulary — feature, fix, tests, refactor, mcp-feature, change-direction — or sets done to end the session early.

What one session actually does

Whether benchmark or exploratory, a session moves through six steps on the same evolving app workspace (apps/<slug>/, kept on disk).

1. open the brain invents the opener: an app slug, language, prompt, visible and hidden requirements, and the deterministic checks it will be held to.
2. run the harness shells out: goose swarm run "" --output-format json in the app workspace, appending --mcp for any expected tools.
3. collect it joins three sources of truth: the swarm's JSON result, the structured .swarm/run-.jsonl event log, and every worker's full session trace (opened read-only from goose's session DB by its logged session_id), plus a file snapshot of the workspace.
4. verify the seven-dimension stack runs (next section). This is where the golden checks execute against the built app.
5. next_move for another turn, the brain reacts like a real user and amends the same app; repeat to the turn budget or until it says done.
6. tweak if a systemic fleet problem showed up, propose a scope-guarded knob change, apply it, re-run, and record the before/after.

Each turn is written to runs/<session_id>/turn-<n>.json, and the whole session to session.json + a report.html you can open in a browser.

The brains, and being the brain yourself

The brain is pluggable via SWARMGYM_PROVIDER (which overrides the config without editing it):

claude — the default. Uses the Anthropic SDK; the generator/vibe model defaults to Sonnet and the judge to Opus, because the verifier should be the strongest model in the loop.

local — an LM Studio endpoint running a local model such as Qwen 27B. It forces a large token budget because reasoning models spend tokens "thinking" before they answer.

operator — no key, no SDK: the operator driving the session (for me, Claude Code) is the brain, through a plain file handshake.

The operator handshake is the mode I lean on most — a capable brain steering the exploration at no API cost. It works through one directory:

req-<n>.json     # the harness writes a request, then blocks
resp-<n>.txt     # you write the answer here to unblock it
PENDING.md       # a running checklist of open requests

Each req-<n>.json carries { id, role, system, user, max_tokens, ts } — the full system and user prompts the brain would have received. You read it, produce the completion it asks for (which must contain exactly one JSON object of the requested shape), and write that to resp-<n>.txt. The harness polls every three seconds and returns your text verbatim; an empty file does not unblock it, and it gives up after thirty minutes. Kick one off with:

SWARMGYM_PROVIDER=operator python -m harness explore --n 3 --tweak

explore is the exploratory driver built for this: --n sessions (default 3), an optional --archetype (otherwise it cycles), and --tweak to let it auto-tune. It prints each session's verdict plus any monitor findings, and flags idle or starved nodes inline.

The verify stack — seven dimensions, four sources

No single signal is trusted. The verdict composes up to seven dimensions from four independent sources, and the run is only as healthy as the weakest dimension that matters. A run is fail if any high-severity finding or any fail dimension appears; partial if anything is partial or warned; otherwise pass.

checks is the floor — the model-free core that cannot be argued with. command_succeeds is how build, run, and tests are graded; file_exists/file_contains/file_matches assert the shape of the tree; tool_called confirms an MCP tool actually fired.

cluster is the dimension whose findings decide whether the tweaker runs. A run can pass every check and still reveal that two of three machines sat idle — a throughput bug worth fixing.

diagnostics catches the local-model tell: a task that reported "done" with zero tool calls (it narrated instead of acting), or a breakpoint() and a stubbed NotImplementedError left in the shipped source.

the judge reads the actual code for what a grep can't see — requirement coverage, correctness, subtle bugs — and is deliberately kept honest by the deterministic floor beneath it. Turn it off with --no-judge when you only want the model-free signal.

The reason all of this exists is the failure mode local models specialise in: passing tests around a broken feature. The deterministic golden check on the real advertised command is the one lens that mechanically refuses to be fooled, so it is the floor and everything else builds on top.

Tutorial part 3 — tuning the swarm from what you find

Exploratory mode does not just score the swarm; it can improve it. Two model-free components handle this.

The monitor watches how work was spread across the fleet each session. It flags a node as starved (got zero tasks), idle (under 5% of its fair share), or underused (under half its fair share), and marks the session unbalanced if any of those hold. Those findings appear inline in explore output and in the report.

The tweaker turns a fleet finding into an action, but only within a hard scope guard — it will only ever touch swarm knobs, never core:

Anything outside that list is rejected outright. When --tweak is on and a cluster finding appears, the tweaker proposes a change with a rationale, applies the ones it safely can (pool weights and enables, plus the two env knobs), re-runs the swarm, and records the before/after per-device distribution in the turn — a self-contained A/B you can read back later.

The ledger and every artifact

Every session lands in the ledger under ledger/: an append-only sessions.jsonl (the full record) and a queryable ledger.sqlite. A record carries the mode tag, the archetype and app slug, the turn count, the overall verdict, wall time, the per-dimension statuses, and — for exploratory runs — the idle-node report. python -m harness report prints the last thirty.

Per session, under runs/<session_id>/:

report.html — the turn-by-turn view: each turn's kind and overall badge, the prompt, every dimension badge, the diagnostics per-model table, a colored list of findings each with a fix hint, and a blue banner when a tweak ran.

session.json — the complete session including every turn.

turn-.json — the task, verdict, and run result for one turn.

And the benchmark aggregates under runs/: BENCHMARK.md (the head-to-head, regenerated from the ledger), benchmark-runs.csv, and benchmark-summary.csv.

The swarm flags the harness measures

The harness measures the effect of the swarm's own feature flags — the knobs on the goose local-edition side that change how the fleet builds. Flip one on, re-run the frozen benchmark, and read whether it moved the numbers. The load-bearing ones:

The flywheel

That is the whole point. A local-model swarm will always tell you it's done. swarm-gym is how you find out whether it actually is — and then, because every finding has a fix hint and every fix is a flag or a knob, how you close the gap and prove it on the same frozen ruler. Measure, diagnose, tune, re-run. It is that loop, run over and over, that moves the ceiling of what a fleet of small local models can actually ship.

python -m harness bench --tier medium --variant mlx     # measure
python -m harness bench-report                           # read the head-to-head
GOOSE_SWARM_GOALS=1 python -m harness bench --tier medium --variant mlx   # flip a flag, re-measure
python -m harness bench-report                           # did it move?

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.