Your evals assume the world holds still
Almost every agent benchmark in wide use is turn-based. The harness hands the model an observation, the model emits an action, the harness applies the action, and then — crucially — nothing happens until the model produces its next token. The environment is a pure function of the interaction history. Time is logical, not physical.
That is a deliberate and usually correct simplification. It isolates reasoning from scheduling noise, it makes runs reproducible, and it lets you compare two checkpoints without worrying that one got a slower GPU. It is also a bad model of deployment.
In production, the world keeps moving while your agent thinks. A price changes between the quote and the order. A lease expires. A human answers the email you were drafting a reply to. Another replica of your own agent takes the lock you were about to take. A recent preprint, "The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock," takes this mismatch as its starting point. The title names the problem precisely enough that the engineering consequences are worth spelling out on their own.
What wall-clock time actually breaks
Two distinct failures get conflated under "the agent is slow," and separating them is the first useful step.
Staleness. An observation is a claim about the world at some instant. Turn-based harnesses make it a claim about the world now, forever. Once you add a clock, every observation carries an implicit age, and the agent's context becomes a mix of fresh and expired beliefs with no visual difference between them. A log line, a search result, and a file listing taken ninety seconds apart look identical to a transformer. The model has no way to discount the stale ones unless you tell it.
Validity windows. Some actions are only meaningful within a time band. A session token, a signed upload URL, a pagination cursor, a reservation hold, a one-time code — all of these are affordances that exist at t and are gone at t + Δ. In a frozen environment, a failed tool call means the agent made a mistake. In a live environment, it might just mean the agent deliberated too long.
The concrete failure modes that show up once you stop freezing the world:
- Read-modify-write races. The agent reads state, spends seconds reasoning, then writes back a value derived from a snapshot that no longer exists.
- Duplicate side effects. A tool call times out. The agent retries. The first call had actually succeeded, and now the customer has two orders.
- Expired affordances. The plan is right, but step four uses a handle that expired during step three.
- Blocking serialization. The agent waits on a synchronous tool while an independent subtask could have progressed in parallel — the plan is fine, the schedule is not.
- Deadline overrun. The answer is correct and arrives after the SLA it was supposed to satisfy. Under a turn-based scorer, this is a perfect score.
What a live-clock evaluation requires
Mechanically, there are three ways to build one, and they trade off differently.
Real time. Run the harness against actual concurrent processes and score on wall-clock. Highest fidelity, lowest reproducibility: your numbers now depend on your serving stack, your network, and your machine's load.
Virtual-but-live time. Introduce a clock abstraction, let the environment advance on its own simulated ticks, and give the agent a monotonic clock it can read through the tool layer. This is the interesting design point. If the agent can query time and always gets a frozen value, you have reverted to turn-based evaluation with extra steps. The clock must be observable to the agent but not controllable by it, exactly like environment state. Observations must carry timestamps and the agent must be able to act on them.
Recorded timelines with replay. Log real interactions with their timestamps, then replay them against a controllable clock for scoring. Cheap, reproducible, and it lets you sweep the agent's own latency without re-running the world.
Two scoring decisions follow. First, a live-clock eval cannot report a single number. It should report a latency-accuracy curve: what fraction of tasks complete correctly within a budget, at several budgets. Second, because the interleaving of agent deliberation and environment change is nondeterministic, you need seeded scheduling and a distribution over seeds rather than one run. Some of those seeds should be adversarial — a change landing just after the agent reads.
Teaching, not just measuring
Training under a live clock changes the objective, not just the test. Latency becomes part of the action space: how long to deliberate, when to re-check before committing, when to act on a stale-but-good-enough belief.
Concretely, that means putting a clock reading into the observation, charging wall-clock time into the reward rather than treating it as free, and being careful about credit assignment. The outcome of an action depends on the state at the moment the action landed, not the state at episode start. Standard advantage estimation over a trajectory silently assumes those are the same, and under a live clock they are not.
What to do before the benchmark exists
You do not need a research harness to get most of the benefit. Start here:
- Timestamp every observation you put in context, and state freshness requirements in the tool description ("this cursor is valid for 60 seconds").
- Prefer compare-and-swap and idempotency keys over blind writes in your tool implementations, so retries are safe by construction.
- Make timeouts return a distinguishable unknown, not a generic failure, and teach the agent to re-check rather than assume failure.
- Log wall-clock per step. The histogram usually has a long tail that dominates the total, and it is often a small number of tools.
- Add one eval where the environment mutates on a timer independent of your agent, and measure whether the agent notices.
Open questions
Is live-clock performance a capability axis or a robustness axis? It plausibly needs to be reported separately, because a slower model with better staleness handling can beat a faster one on the same tasks. Reproducibility is unresolved: seeded nondeterminism may be the honest default, but the community has no convention for it yet. Transfer is unproven — latency distributions in production are heavy-tailed and machine-specific, so training on one clock may not survive another. And multi-agent settings turn the environment from a Markov decision process into a game, where the other players also own clocks.
