Your Free AI Lane Is Slow in the Queue, Not in the Model

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped.

Nothing in the model got slower. The jobs sat in a queue you never charted.

Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation.

This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff.

Start the clock at enqueue, not at first token

Your job record needs five timestamps. Anything less and you cannot separate waiting from working.

1. enqueue_ts — the job was accepted by your scheduler.
2. start_ts — the first request hit the serving path.
3. accept_ts — the artifact passed your acceptance check.
4. deadline_ts — the wall-clock deadline for that batch.
5. attempt — the retry index, so rework stays visible.

From those you get three derived numbers:

• wait = start_ts - enqueue_ts
• gen = accept_ts - start_ts
• wait_share = wait / (accept_ts - enqueue_ts)

Gate on wait_share. A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing.

The two-clock ledger in one script

The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database.

#!/usr/bin/env python3
# two_clock.py — lane verdict from job events (JSON Lines)
import json, sys

def load(path):
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)

def pct(values, q):
    if not values:
        return 0.0
    s = sorted(values)
    i = min(len(s) - 1, int(round(q * (len(s) - 1))))
    return s[i]

def main(path, wait_share_max=0.60, margin_frac=0.15):
    rows = list(load(path))
    waits, gen, share = [], [], []
    for r in rows:
        total = r["accept_ts"] - r["enqueue_ts"]
        if total <= 0:
            continue
        wait = r["start_ts"] - r["enqueue_ts"]
        waits.append(wait)
        gen.append(total - wait)
        share.append(wait / total)

    drain = max(r["accept_ts"] for r in rows)
    first = min(r["enqueue_ts"] for r in rows)
    deadline = max(r["deadline_ts"] for r in rows)
    slack = deadline - drain
    margin = (deadline - first) * margin_frac

    print(f"jobs={len(rows)} wait_p50={pct(waits,0.5):.0f}s wait_p95={pct(waits,0.95):.0f}s")
    print(f"gen_p50={pct(gen,0.5):.0f}s wait_share_p95={pct(share,0.95):.2f}")
    print(f"slack={slack:.0f}s margin_budget={margin:.0f}s")

    if pct(share, 0.95) > wait_share_max:
        print("VERDICT: spill — wait dominates, throughput is not your lever")
    elif slack < margin:
        print("VERDICT: spill — projected drain lands inside the safety margin")
    else:
        print("VERDICT: hold — the lane is inside budget")

if __name__ == "__main__":
    main(sys.argv[1])

Run it against a finished batch:

python3 two_clock.py lane-free.jsonl

Expected output shape (illustrative, not a measurement from your system):

jobs=412 wait_p50=38s wait_p95=611s gen_p50=22s wait_share_p95=0.81
slack=1740s margin_budget=1080s
VERDICT: spill — wait dominates, throughput is not your lever

Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second.

Why those two thresholds

wait_share_max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data.

margin_frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits.

Inject the wait locally before you trust the gate

Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell.

#!/usr/bin/env bash
# fake_lane.sh — synthetic congestion, zero production impact
set -euo pipefail
HOLD=${HOLD_SECONDS:-90}
DEADLINE=$(( $(date +%s) + ${WINDOW_SECONDS:-3600} ))
while read -r id; do
  ENQ=$(date +%s)
  sleep "$HOLD"
  echo "{\"job_id\":\"$id\",\"enqueue_ts\":$ENQ,\"start_ts\":$(date +%s),\"accept_ts\":$(date +%s),\"deadline_ts\":$DEADLINE}"
done

Generate job ids, then run the same 40 jobs twice:

seq 1 40 > ids.txt
HOLD_SECONDS=5  ./fake_lane.sh < ids.txt > lane-fast.jsonl
HOLD_SECONDS=90 ./fake_lane.sh < ids.txt > lane-slow.jsonl
python3 two_clock.py lane-fast.jsonl
python3 two_clock.py lane-slow.jsonl

The fast file should print hold. The slow file should print spill. If both print the same verdict, your gate is dead code and you should fix it before touching production routing.

Decision table: when free capacity is the wrong bet

Read the table top-down. The first row that trips decides. You do not average these signals into one score.

Where the free tier fits into the drill

You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload.

One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change.

Rollback and cleanup

Keep the escape hatch boring and tested.

1. Gate routing behind one flag, for example FREE_LANE_ENABLED=0.
2. Flip the flag and stop new admission into the free lane.
3. Let in-flight jobs finish; do not kill them mid-acceptance.
4. Keep the ledger files. They are the evidence for the next threshold change.
5. Re-run two_clock.py on the spill window and compare wait shares.

If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic.

Who should not use this

• Teams without timestamped job events. You cannot reconstruct two clocks from logs you never wrote.
• Interactive or latency-critical endpoints. Queue-shaped capacity belongs behind batch work.
• Work where a rework costs more than the tokens you saved.
• Anyone planning to hard-code today's free allowance into a capacity plan for next quarter.

If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.