Every test passed and the level was unwinnable

I shipped a wilderness survival game with eight scenarios. The hardest one — a five-star map behind the paywall — could not be finished. Not "was brutally hard." Could not be finished. Walking onto the exact tile the design calls the ending did nothing at all.

Every test was green. That is the part worth writing down.

The bug

Locations connect through exits, and some exits stay hidden until you find them. travel() handled that discovery like this:

if exit.hiddenUntilDiscovered, !state.discoveredLocationIds.contains(target.id) {
    state.discoveredLocationIds.insert(target.id)   // marked before the move
    state.log("log.hidden_passage_found", ...)
}

And arrive() decided first-visit rewards like this:

if !state.discoveredLocationIds.contains(location.id) {
    ...
    effects.apply(location.onFirstVisitEffects, to: &state, rng: &rng)
}

One set, written before the move and read after it. Any location reached through a hidden exit is already a member by the time arrival checks, so its entire onFirstVisitEffects block is skipped in silence. Getting lost and retreating did the same thing: the destination got marked, and the real arrival later never counted as a first.

Sixteen locations across all eight maps lost their first-visit payload that way — standing shelters, supply caches, morale, story flags. On one of them, the payload was the ending.

highland_mine's only win condition is reaching vent_shaft. vent_shaft's only entrance is a hidden exit. The win was unreachable through the single door that led to it, and the player starves in a dead end that the content files consider the finish line.

The fix is boring: let arrive() own discovery exclusively, and move the secret-passage log line to after arrival is confirmed.

Why nothing caught it

Run down what a content-driven game normally validates. Map connected from spawn. Every recipe's ingredients exist as items. Every scenario has a victory path. No dangling ids anywhere. All green — and all still green with the bug in place.

Every one of those asks is this configuration valid. The configuration was valid. The victory location existed, was reachable, and carried a victory effect. The effect simply never ran.

That's the shape: structure correct, execution never happens. You cannot find it by validating the graph. Something has to walk it.

The instrument: a bot that plays

So I wrote one. The important decision wasn't writing it, it was splitting it into instruments that don't contaminate each other.

1. Can you win when survival isn't the obstacle?

Every step, the harness tops up hydration, body heat, stamina, calories, and clears injury and infection. Everything from the content side stays: location tags, disabledActionIds, per-location resource exhaustion, flag gates, recipe materials.

The bot pathfinds ignoring flag gates, and when travel refuses the step, it degrades to "look for the key here" — trying every action that can set the flag it's missing, which it collects once by scanning effects:

private lazy var flagActions: [String: [String]] = {
    var map: [String: [String]] = [:]
    for action in content.actions.values {
        for eff in action.successEffects where eff.type == .flagSet {
            guard let flag = eff.flag else { continue }
            map[flag, default: []].append(action.id)
        }
    }
    return map
}()

The win condition itself is inferred from content rather than a hardcoded table, because a hardcoded table keeps passing after a scenario is redesigned and hides exactly the problem you wrote the test for.

If you can't win under those conditions, it is a materials or path problem, full stop. Currently 8/8.

2. Do the winning materials exist at all?

Pure static pass, no simulation: given disabledActionIds and the location tags actually present, compute the theoretical ceiling on every material a raft or a signal fire needs — direct gathering plus anything craftable. A ceiling of zero means unwinnable.

This exists because survival scenarios are designed by subtraction. The desert has no water source, the burnt forest has no standing wood, the mine has no vegetation — highland_mine disables eight gathering actions and burnt_range disables cutting firewood. Subtracting the interesting thing and accidentally subtracting a victory material is one edit apart.

3. Win rate under normal play — printed, never asserted

This is the one I'd defend hardest, and it asserts nothing.

Partway through, I gave the bot one piece of common sense: don't travel below 62 body heat. Reasonable. Humans do that.

frost_ridge went from 7/8 to 0/8.

On a five-star snow map body heat is almost never above 62, so the bot sat by the fire and froze in place. Nothing about the scenario changed. The number moved 88 points because I edited the player.

A metric with your agent's competence baked into it cannot be an assertion. It's a trend line and nothing more. The moment I let that number gate CI, every future improvement to the bot reads as a content regression and every content regression can be argued away as bot noise.

The check that needed no bot at all

Separately: every scenario, crossed with every row of its own weather table, must survive its first night when the player does everything right.

Push the clock to sunset. Then simulate the night hour by hour with the fire lit, fuelled, and rain-protected, and a shelter at the tier that difficulty has the right to demand:

// Difficulty should decide how much the player must invest,
// not whether a solution exists at all.
let required: ShelterLevel = scenario.difficultyStars >= 4 ? .leanTo : .basic

Night length comes from the scenario's own sunset and sunrise, not a hardcoded 12 hours — long nights and short nights differ by a factor of two, and hardcoding it means the hardest maps are tested on the easiest assumption.

Then assert body heat above zero, and condition above 30, because surviving with 4 health left is not a passing grade.

highland_mine failed this too, arithmetically. −11 °C at night with snow drains 26.8 body heat per hour. The best insulation the entire game can assemble — coat 2.5, natural cover 1, fire 9 — is 12.5. A twelve-hour night takes 136 points. You start the night with 76.

No balance pass gets you out of that. The fix was per-location microclimate: inside a mountain doesn't take the weather.

let indoors = node?.shieldsFromWeather ?? false
let ambient = baseTemp + (indoors ? 0 : weather.tempOffsetC) + (node?.tempOffsetC ?? 0)

Baselines relative, never absolute

The older, dumber probe — the one that only forages in place and answers "does the player die on day two" — compares each new scenario's worst seed against a known-good free scenario, with one day of slack:

XCTAssertGreaterThanOrEqual(
    worst, baseline - 1,
    "\(scenario): worst seed died on day \(worst), baseline \(baseline)"
)

Absolute survival days measure how dumb my bot is. The same dumb bot across different scenarios measures the scenario. Anchoring to a baseline instead of a constant is what makes an unreliable instrument produce a reliable comparison.

The determinism bug sitting underneath all of it

None of this reproduces without deterministic randomness. Same seed, same run, forever — otherwise a failing probe is a rumour. Each action draws from its own stream derived from one save seed, SplitMix64.

The obvious way to derive stream s is to offset the initial state by s. Do not.

SplitMix64 advances by a fixed gamma — 0x9E3779B97F4A7C15 — on every step. If your stream offset is any integer multiple of that gamma, then stream s's i-th output is exactly stream s−k's (i+k)-th output. Individually each stream still passes every randomness smell test you'd casually apply. Collectively they're one sequence walking in lockstep, and "independent stream per action" is a comment, not a fact.

Avalanche the stream index before it ever becomes state:

init(seed: UInt64, stream: Int = 0) {
    var mixed = seed ^ (UInt64(truncatingIfNeeded: stream) &* 0xD1B54A32D192ED03)
    mixed = (mixed ^ (mixed >> 33)) &* 0xFF51AFD7ED558CCD
    mixed = (mixed ^ (mixed >> 33)) &* 0xC4CEB9FE1A85EC53
    state = mixed ^ (mixed >> 33)
}

There's a second-order constraint that comes with this, and it caught me on the discovery fix above: the repair had to preserve RNG call order. Move a generator call earlier or later and every existing save and every seeded result shifts underneath you — the bug is fixed and the whole test suite goes red for unrelated reasons. Once determinism is load-bearing, "fix this without changing how many times you call the generator" becomes part of the diff review.

Where it landed

216 tests. Eight scenarios reachable once survival pressure is removed, no material gaps, and both the iOS and watchOS builds walked through the previously-broken vent shaft on a real simulator before I believed any of it.

The thing I actually changed my mind about: for a content-driven game, the valuable test is not the one that validates your data. It's the one that consumes your data the way a player does and then tells you what it couldn't reach.

The game is Wilds Survivor — on the App Store. Eight ways to be stranded, five difficulty tiers, fully offline, no ads, no accounts, no energy timers, and a watchOS app that runs the entire engine standalone with the phone left at home. Two scenarios are free, which is enough to see whether the decision loop is your kind of thing. Every number in this post came out of its repo.

Still short of a good answer on the win-rate problem: a bot competent enough that its win rate describes the scenario rather than the bot. For now I just refuse to assert on it. If you've solved that for a game with hidden information, I'd like to hear how.