Your first request timed out. You clicked retry. The second attempt sailed through. You just spent two round-trips, a couple thousand tokens, and eleven seconds to get one answer. The answer was free. The retry was not.
Free model access is a supply chain, not a gift. You pay for it with queueing, throttling, and the occasional timeout. The real cost is not the token balance. It is the hidden currency you spend on retries: time and repeated token burns.
Free tiers deserve credit for lowering the barrier. MonkeyCode, an open source project with a free model tier and a free server option, is one of those attempts. Disclosure: This article was prepared as part of MonkeyCode's product outreach. But wiring free capacity into production without thinking about retries will turn a generous quota into a lottery.
The math gets interesting quickly. If each attempt costs C tokens and succeeds with probability p, the expected attempts until success is 1/p when p stays constant. A 95% success rate gives about 1.05 attempts. A 70% success rate gives about 1.43 attempts. That is a 43% token tax. Time is worse: a 2-second success and a 30-second timeout blend into an average attempt time of 2p + 30(1-p). At p=0.7, that is 10.4 seconds per attempt, times 1.43 — almost fifteen seconds per success. At p=0.95, about 2.1 seconds. Free capacity can be seven times slower before counting a single queued request.
Why does success rate drop in the first place? Free tiers often run on shared infrastructure. Cold starts, noisy neighbors, and rate limiters all turn a simple prompt into a flaky call. The prompt didn't change. The environment did.
You don't need a benchmark suite. You need a calculator. The script below turns your observed success rate into an effective cost per success.
#!/usr/bin/env python3
# retry_math.py - effective cost per successful LLM call
def expected_attempts(success_rate: float, max_retries: int = 5) -> float:
attempts = 0.0
fail = 1.0
for _ in range(max_retries + 1):
attempts += 1.0
fail *= (1 - success_rate)
return attempts
def avg_attempt_time(avg_success_latency: float, timeout: float, success_rate: float) -> float:
return avg_success_latency * success_rate + timeout * (1 - success_rate)
def report(success_rate: float, tokens_per_call: int,
avg_success_latency: float = 2.0, timeout: float = 30.0,
max_retries: int = 5) -> None:
ea = expected_attempts(success_rate, max_retries)
eff_tokens = tokens_per_call * ea
eff_time = ea * avg_attempt_time(avg_success_latency, timeout, success_rate)
print(f"success_rate={success_rate:.2f} attempts={ea:.2f} tokens={eff_tokens:.1f} time={eff_time:.1f}s")
report(0.95, 1000) # healthy
report(0.70, 1000) # throttled
report(0.50, 1000) # broken
Run it with your own numbers. At 1000 tokens per call, a healthy 0.95 success rate yields around 1050 tokens and 2.1 seconds per success. At 0.70, the same call costs roughly 1430 tokens and 14.9 seconds. That is the retry tax in numbers.
But the script assumes independent attempts. Real retries need exponential backoff and jitter, or you will turn a transient error into a self-inflicted outage. Use a backoff policy: start at 1 second, double it, cap at 30 seconds, and add random jitter. That way the retry loop doesn't become the new load balancer.
The tax compounds when queues get involved. A failed call re-enqueues; the next attempt adds load while the first one is still timing out. That is how a free-tier hiccup becomes a retry storm. Teams end up watching a dashboard they didn't need because their retry loop was hammering an endpoint that was already down. The endpoint was not the problem. The retry loop was.
Here is a practical gate: measure your actual success rate over at least a hundred production-like calls with real prompt sizes. If the success rate is below 0.9, calculate the effective time per success. If that number exceeds your user-visible latency budget, free capacity is the wrong bet. Paid per-token APIs or a self-hosted model with a fixed cost can give you predictable latency and deterministic concurrency. That predictability is often worth the price.
This approach is not for everyone. It is wrong for workloads with a hard latency SLO, for batch jobs that must finish before a deadline, or for prompts where retrying produces inconsistent results. In those cases, free capacity is not free. It is a tax on your time to market.
MonkeyCode's free model access and free server option are a decent sandbox for measuring that trade-off. You can test a workflow, plot your retry curve, and decide whether the free tier fits. Just don't mistake the quota for your budget. Your budget is the number of attempts that actually succeed.
If you try the script, bring your own retry count. And if you want to explore the project, the MonkeyCode repository is a reasonable place to start.
