Building the AI Arms Race: A Reproducible Experiment to Test Whether LLMs Evolve Strategies When They Compete

Two LLMs. One hidden state. 1,000 rounds. Do they independently evolve increasingly effective strategies — or just chase noise?

I built an open-source, terminal-based research experiment that pits glm-5.3-flash against deepseek-v4-flash-0731 in a repeated strategic game called HIDDEN SIGNAL. The goal wasn't to prove an arms race exists — it was to build an experiment that could disprove it, and report whatever the data actually says.

This post walks through the design, the architecture, the hard-won lessons about real LLM endpoints, and the code that makes it all reproducible.

The Research Question

If two LLMs repeatedly compete against each other, do they independently evolve increasingly effective strategies?

That's a question for computational social science, not just ML engineering.

To answer it honestly you need:

1. A real game with asymmetric private information, strategic communication, and competing incentives.
2. Real models genuinely playing it — no hard-coded outputs.
3. Information isolation — each agent sees only what it legitimately could.
4. Controls — random opponents, heuristic opponents, model-vs-model baselines — so you can tell "adaptation" from "statistical noise".
5. Deterministic metrics — deception, leakage, exploitability computed from ground truth, never from an "LLM judge".
6. Falsifiability — the experiment must be able to report no evidence of strategic escalation.

Let's see how each of those is engineered.

The Game: HIDDEN SIGNAL

Each round, the engine (the sole authority) draws a hidden state S ∈ {A,B,C,D}
and gives each agent a two-state private set that always contains S:

def private_set_for(state: str, agent: str) -> tuple:
    """GLM sees {S-1, S}; DeepSeek sees {S, S+1} (mod 4)."""
    idx = STATE_ALPHABET.index(state)
    if agent == "A":
        return (STATE_ALPHABET[(idx - 1) % 4], state)
    return (state, STATE_ALPHABET[(idx + 1) % 4])

Agents act in alternating turns (randomized first mover, to remove any
structural advantage). On a turn they choose message, guess, or withhold.
The scoring creates the central strategic tension:

A truthful claim that excludes a state in the opponent's set narrows their
uncertainty from 2 states to 1 — a 1-bit leak that costs you −5 but hands the
opponent +135 if they exploit it. That's the tradeoff between revealing
and extracting information, and it's what makes the game genuinely
strategic rather than a coin flip.

The engine owns all state. Agents submit actions; the engine validates,
scores, and records them. They can never modify the hidden state, their
scores, or their opponent's private information.

The Information-Isolation Boundary

This is the most important safety property in the experiment. Each agent's
observation is constructed by get_observation(), which only ever receives
the agent's own private set plus the public transcript:

def get_observation(agent, rs, current_public_log, recent_public_rounds,
                    own_history_summary=None, opponent_history_summary=None,
                    strategy_note=None, game_number=None, total_games=None,
                    condition_feedback="none") -> dict:
    obs = {
        "agent": agent,
        "round_number": rs.round_number,
        "your_private_information": sorted(rs.private[agent]),   # ONLY own set
        "current_round_public_events": current_public_log,
        "recent_rounds_public": recent_public_rounds,
        ...
    }
    if condition_feedback in ("self", "self+opponent") and own_history_summary:
        obs["your_performance_history"] = own_history_summary
    if condition_feedback == "self+opponent" and opponent_history_summary:
        obs["opponent_observable_statistics"] = opponent_history_summary
    return obs

The opponent's private set, hidden reasoning, system prompt, and internal
beliefs are never included — enforced by construction, and verified by
tests that assert obs["your_private_information"] never equals the
opponent's set and that the hidden state never appears as a field.

After each round, the hidden state is revealed publicly (both agents learn
it) — but private sets are never revealed. That's a deliberate choice: it
gives agents real historical evidence without leaking the game's secret
structure.

Anti-Contamination: Don't Tell Them to "Adapt"

The single most important design decision: the in-game prompt never mentions
adaptation. Telling the agents to "develop an increasingly sophisticated
strategy" would contaminate the experiment by priming exactly the behavior
you're trying to measure.

Instead, there are two prompts:

1. In-game prompt (stable, every turn): "Maximize your cumulative reward. Do not assume your opponent is truthful. Do not assume they are deceptive. Use the evidence." No mention of adaptation.
2. Strategy-update prompt (every 50 games): "Identify patterns in your performance and formulate a concise strategy for the next block." This is the only place adaptation is explicitly requested.

Strategy memory is bounded — each update replaces the previous note — and every
version is archived to strategies/A_v01.txt … A_v20.txt so we can trace
strategy evolution over time.

The LLM Client: The Hardest Lesson

The openai SDK makes calling an OpenAI-compatible endpoint trivial. The
hard part was a subtle failure mode that silently broke the first smoke test:

Both models' default reasoning_effort is max, which burns the entire
output budget on hidden chain-of-thought and returns empty content.

I watched the models "play" by withholding every single turn. The raw records
showed valid JSON actions — all withhold. The models weren't broken; they
were returning empty strings because their hidden reasoning consumed all 700
tokens.

The fix was two-fold:

# arms/config.py
reasoning_effort: str = "low"   # default "max" burns the budget on hidden CoT
max_tokens: int = 4000          # generous: hidden reasoning consumes part of it

and a safety net in the client that treats empty content as a recoverable
failure with a constrained retry:

# arms/llm.py
if not content.strip():
    r.error = "empty content (reasoning consumed budget?)"

This is the kind of thing that only shows up when you run real models at
scale, and it's exactly why the smoke test exists. I also added per-call
token/latency tracking so the final run reports honest cost numbers:

costs A: {'calls': 24, 'failures': 0, 'total_tokens': 24414, ...}
costs B: {'calls': 31, 'failures': 5, 'total_tokens': 98960, ...}

Architecture

run.py / run_all.py
        │
        ▼
  Experiment Config ──▶ Replicate (×N) ──▶ GameEngine (authoritative state)
        │                                        │
        ▼                                        ▼
  get_observation(agent)  ◀── public transcript + own private set only
        │
   ┌────┴────┐
   ▼         ▼
 LLMAgent A  LLMAgent B        (glm-5.3-flash) (deepseek-v4-flash-0731)
   │         │
   └────┬────┘
        ▼
  structured action JSON ──▶ validated ──▶ engine applies + records
        │
        ▼
  raw/ (games.jsonl, messages, actions, errors)
        │
        ▼
  analyze.py ──▶ deterministic metrics → change points → findings.json
        │            └─▶ 7 publication figures
  report.py ──▶ report.md (generated from actual data, never invented)
  run_cross_eval.py ──▶ strategy × opponent matrix (frozen snapshots)

Module map

arms/
├── config.py        # game rules, scoring, conditions, LLM config
├── credentials.py   # API-key resolution (env → ~/.dsh/.credentials.yaml)
├── llm.py           # OpenAI-compatible client: retries, cost tracking, CoT stripping
├── game.py          # HIDDEN SIGNAL engine + observation isolation
├── prompts.py       # stable in-game prompt + strategy-update prompt
├── actions.py       # strict structured-output validation
├── agents.py        # LLMAgent + Random/Heuristic control agents
├── strategy.py      # bounded strategy memory + version archive
├── metrics.py       # ground-truth metrics (deception, leakage, extraction)
├── history.py       # per-agent performance summaries
├── analysis.py      # change-points, bootstrap CIs, Mann-Whitney, signature
├── runner.py        # replicate runner: games, checkpoints, probes, generalization
├── experiment.py    # experiment driver
├── dataset.py       # experiment directory schema + JSONL writers
├── figures.py       # 7 publication figures
└── ui.py            # rich terminal UI (live mode)

Deterministic Metrics

Every metric is a pure function of raw records plus ground truth. The
deception signal, for example, is not a semantic judgment — it's a
ground-truth heuristic: a claim is a lie when it contradicts the agent's
own private knowledge (e.g., claiming "true" about a state not in your own
set).

Information leakage is computed in bits from the actual finite state space —
the opponent's entropy before minus after your truthful exclusions:

def info_leakage_bits(round_record, agent):
    opp = "B" if agent == "A" else "A"
    opp_private = round_record[f"private_{opp}"]
    before = uncertainty_of_set(opp_private)     # log2(2) = 1 bit
    after = before
    for e in round_record["events"]:
        if e.get("agent") == agent and e.get("claim_reveal"):
            after = 0.0                          # narrowed to 1 state
    return before - after

No LLM judge. No hand-waving. Just entropy over the true state space.

Statistical Analysis & the Arms-Race Signature

The system detects candidate strategy shifts automatically via greedy binary
segmentation on reward series (change points), tests trends with Spearman
rank correlation, compares first vs second half with Mann–Whitney U, and
builds bootstrap confidence intervals.

The "arms-race signature" is an explicit detector for the alternating-response
pattern: A exploits → B declines → B adapts → B recovers → A adapts → A
recovers. It counts sign alternations in block-level advantage:

def arms_race_signature(series_a, series_b, block, min_effect=0.0):
    adv = np.array([
        series_a[i*block:(i+1)*block].mean()
        - series_b[i*block:(i+1)*block].mean()
        for i in range(n_blocks)
    ])
    signs = np.sign(adv)
    # ... count sign changes ignoring zeros ...
    detected = alternations >= 2 and len(runs) >= 3

Crucially, the detector can return false — and the report will then say
"no strong evidence of an alternating strategic arms race was detected", which
is a perfectly valid scientific outcome.

Controls That Make It Publishable

• Random opponent (RandomAgent): uniform random policy.
• Heuristic opponents (HeuristicAgent): fixed truthful / bluff / cautious policies.
• Model-pair controls: GLM vs GLM, DeepSeek vs DeepSeek, GLM vs DeepSeek under every condition (run_all.py).
• Exploitability probe: at each checkpoint, current strategy vs a random baseline; exploitability = perf vs real − perf vs baseline.
• Generalization test: final strategies re-evaluated against fresh instances, a random baseline, and a fixed truthful heuristic — answering "did they learn genuinely useful strategies, or just overfit to each other?"

What the Smoke Test Taught Me

A 10-game smoke test (55 real LLM calls, ~25 min) was enough to validate the
entire pipeline and produce genuine — if preliminary — signal. The corrected
numbers from that pilot:

• DeepSeek won 70% of games, ending at +790 cumulative reward vs GLM's −171.
• GLM did adapt its strategy: after block 1 it changed its note from "guess immediately" to "stop auto-guessing; withhold until you have a signal."
• But that adaptation made it worse in this tiny sample: its block-2 average reward dropped from +3.8 (20% wins) to −38 (0% wins).

The analysis correctly reported arms_race_detected: false for that short
run — exactly the falsifiability the design demands. Ten games is far too
little to claim a race; that's what the 1,000-game / multi-replicate runs are
for. What the pilot did prove: the pipeline works end-to-end, the metrics
are grounded in truth, and the experiment is willing to tell us "nothing
conclusive yet" instead of manufacturing a narrative.

Reproducibility

Every experiment directory records: model names, endpoint identifier (never
the key), game rules, scoring, prompt version, seed, temperature, sampling
parameters, game/replicate counts, timestamp, git commit, software version,
and condition. Game-state generation is fully reproducible from the seed
(verified by tests). Raw data is never overwritten — each run gets a fresh
directory.

Try It

pip install -r requirements.txt

# 10-game smoke test (~25 min)
python run.py --smoke

# 1000-game run
python run.py --games 1000 --checkpoint 50 --replicates 1 --condition self-history

# Full matrix
python run_all.py --replicates 3 --games 200 --checkpoint 50

# Analyze + report
python analyze.py experiments/ARMS-self-history-... --figures
python report.py experiments/ARMS-self-history-...

The full dataset, 7 publication figures, and a research report are generated automatically. The repo is MIT-licensed and designed to be forked — see the README's "Ideas for New Features" for extensions like richer games, embedding-based strategy analysis, minimax exploitability, and resumable long runs.

The point of the whole project: build the experiment so the data can surprise you. If the models don't arms-race, that's a finding. If they do, that's a finding. Either way, the numbers are real.

Code & more: https://www.dailybuild.xyz/project/236-ai-arms-race