Two days before my hackathon deadline, I re-read the judging criteria for
the track I was submitting to and found a sentence I'd apparently skipped
the first time through:
"...a measurable efficiency gain over single-agent baselines."
My project, RedCouncil, is an adversarial multi-agent system — five
specialized AI agents (Growth, Risk, Legal, TechDebt, Customer) debate a
business decision, cross-examine each other, and a Synthesizer produces a
severity-scored report. 68 tests passing, auth, storage, deployment, all
done. I had never once measured whether it actually beat a single model
call.
My first instinct was to bail. My second, better instinct was to ask: how
fast can I actually get real evidence for this, instead of assuming it's
either fine or hopeless?
Building the eval, fast
The fastest honest version of this isn't a new subsystem — it's reusing
everything you already have. Same scenarios, same model, one new prompt
that asks a single Qwen call to do what all five agents do combined, and a
purely deterministic scoring function — no LLM judge, just counting and set
math. Two numbers mattered most:
- Cross-agent conflicts surfaced. A single model call has no mechanism to represent disagreement between specialist viewpoints. This number is 0 for the baseline by construction, which makes it a free, structurally guaranteed data point.
- High-severity findings caught, compared head-to-head on the same 1–10 scale, same model, so the comparison isolates the architecture, not model size.
I ran it. Numbers came back. They looked good. I started building the
slide deck.
Then I actually read the results file instead of just the summary, and
found the first problem.
Bug 1: the metric that was empty by design
One of the fields I'd built, domains_missed_by_baseline, was empty in
every single scenario — which I initially read as "RedCouncil never misses
anything the baseline catches." It took a second look to realize the field
was always going to be empty, because of how I'd defined it: it only
checked domains present in RedCouncil's output but absent from the
baseline's. Since the baseline prompt forced one finding per domain every
time, that set difference was structurally guaranteed to be empty
regardless of how RedCouncil actually performed. Not a finding — an
artifact of how I'd defined the metric.
Worse, when I checked the reverse direction, the one that actually
mattered, I found RedCouncil was missing an entire domain — Growth — in 3
of 5 scenarios. My own multi-agent debate system was covering less ground
than the single dumb baseline it was supposed to beat.
Bug 2: the retry that gave up too early
Root-causing this with real trace data (not speculation) showed a
two-part failure: the coverage checker only flagged a domain as "missing"
if its findings scored above a severity threshold — so a legitimately
lower-severity Growth finding could slip through undetected. And when a
retry did get triggered, the acceptance logic only kept the retry if it
strictly improved on the previous attempt. If the model's retry attempt
failed to recover the missing domain, the system gave up and kept the
broken result — because "no better than before" and "still broken" looked
identical to the acceptance check.
Fix: lower the threshold so every domain gets checked regardless of
severity, and give the retry loop several bounded attempts instead of one,
tracking convergence properly instead of a single strict comparison.
I re-ran the comparison. Growth showed up in all 5 scenarios. I updated the
deck. I thought I was done.
Bug 3: zero, again
Then I actually opened the live product UI to record demo footage, and the
"conflicts surfaced" counter read 0 — on a scenario that, reading the
transcript, clearly had agents contradicting each other's core claims in
plain English.
Round one of debugging this found a genuine plumbing bug: one part of the
code wrote a result to a field called conflicts, another part read from a
field called conflict_count. Nobody had ever written to that second key.
Classic. Fixed it, redeployed, tested again.
Still zero.
Round two went deeper, and this is the part I actually learned something
from: the conflict detector was gating candidate disagreements through a
lexical-overlap check before it would even evaluate whether they opposed
each other. My Growth agent argues in terms of savings, cost, ROI. My
Customer agent argues in terms of churn, friction, retention. Zero
shared vocabulary — so the gate filtered the pair out before the actual
stance-comparison logic ever ran, even though the underlying disagreement
was completely real.
This is the kind of bug that's obvious once you see it and invisible until
you do: specialists with genuinely different mandates use genuinely
different words by design. A similarity gate built for a system where
everyone talks about the same thing is exactly the wrong tool for a system
whose entire premise is that the agents don't think alike.
The fix: stop inferring disagreement from vocabulary overlap. Trust the
signal the agents themselves already produce when they explicitly rebut
each other during cross-examination.
Bug 4: the one hiding inside the fix
Trusting each agent's own rebuttal signal introduced a new failure mode
almost immediately: rebuttals are often mutual. If Growth rebuts Risk and
Risk rebuts Growth in the same exchange, and each rebuttal becomes its own
conflict entry, you double-count every disagreement that goes both ways.
Caught this one before it shipped, with a simple fix: dedupe on the
unordered agent pair before counting, not the directional edge.
What actually changed by the end
Conflict detection went from a broken 0, to a plumbing fix that still
read 0, to a real but partial fix, to a detector that could finally see
disagreements that don't share vocabulary — at which point the true
conflict count roughly doubled from what the vocabulary-limited version had
found. Every one of those jumps looked like "the fix worked" right up until
the next layer surfaced.
What I'd actually tell someone building an eval harness
- A metric that's always the same value isn't a metric — check what it's structurally capable of showing before you trust it. Mine was empty-by-definition for an entire category of finding.
- If your system's whole premise is that components think differently, don't build detection logic that assumes they'll describe things similarly. That's not an edge case, it's the design working as intended running into a detector that assumes the opposite.
- Fixing your eval can be as much engineering work as fixing your product — arguably more, because a broken product fails loudly and a broken eval fails silently, as a confident, wrong number.
- Non-determinism compounds across every layer you don't pin down. Between run-to-run model variance and the layers of retry logic in the system itself, "the number" was never really one number — it was a distribution I was sampling from once and calling final. Worth deciding on purpose whether that's good enough for your use case, rather than discovering it by accident three reruns in.
RedCouncil runs on Qwen Cloud (Qwen-Max) end to end — all five debate
agents plus the Synthesizer. If you're building anything where multiple
model calls are supposed to disagree with each other on purpose, budget
real time for your evaluation code, not just your product code. Mine
needed just as much debugging.
