Your free model server is a shared parking lot. Other cars arrive all day. Their engines change your commute. Most latency posts pretend the road is empty. Shared free tiers never are. Here is a myth-busting FAQ. It comes with one reproducible probe. You can run it in an afternoon. No vendor, no hype.
Why bother? Because your client code makes assumptions. A solo run teaches you nothing about contention. Under a neighbor burst, assumptions fail. The failure shows up in production, not in tests. This probe makes contention visible early.
The probe
The probe sends the same prompt at rising concurrency. One call, then two, then four, then eight. It records latency for every request. Run it at different hours. Watch the spread, not just the average.
# shared_fate_probe.py
# Probe a shared free model endpoint without hammering it.
# Run one concurrency level at a time. Ask the endpoint owner first.
import asyncio
import json
import os
import time
import urllib.request
from typing import List
ENDPOINT = os.environ.get("FREE_MODEL_URL", "https://example.invalid/v1/chat/completions")
TOKEN = os.environ.get("FREE_MODEL_TOKEN", "")
PROMPT = "Reply with one word: ready."
def one_call() -> float:
payload = json.dumps({
"model": "sample-model",
"messages": [{"role": "user", "content": PROMPT}],
"temperature": 0,
"max_tokens": 4,
}).encode("utf-8")
req = urllib.request.Request(ENDPOINT, data=payload, headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {TOKEN}",
})
start = time.perf_counter()
with urllib.request.urlopen(req, timeout=10) as resp:
resp.read()
return (time.perf_counter() - start) * 1000
async def run_batch(concurrency: int) -> List[float]:
return await asyncio.gather(
*(asyncio.to_thread(one_call) for _ in range(concurrency))
)
async def main() -> None:
for level in (1, 2, 4, 8):
samples: List[float] = []
for _ in range(5):
samples.extend(await run_batch(level))
await asyncio.sleep(1) # Your sleeping spot in the lot.
samples.sort()
p50 = samples[len(samples) // 2]
p95 = samples[max(0, int(len(samples) * 0.95) - 1)]
print(
f"concurrency={level} n={len(samples)} "
f"p50={p50:6.0f}ms p95={p95:7.0f}ms max={max(samples):7.0f}ms"
)
if __name__ == "__main__":
asyncio.run(main())
Sample output, marked clearly as illustrative:
concurrency=1 n=5 p50= 812ms p95= 1043ms max= 1102ms
concurrency=2 n=10 p50= 901ms p95= 1298ms max= 1401ms
concurrency=4 n=20 p50= 1128ms p95= 2104ms max= 2234ms
concurrency=8 n=40 p50= 1906ms p95= 3841ms max= 4107ms
That output is not a benchmark of any product. Your numbers will differ. The shape matters more than the sizes.
How to read the output
Start at the concurrency=1 line. That is your solo reference. Compare p95 growth across levels.
- If p95 stays flat, the endpoint tolerates parallel calls.
- If p95 bends upward, you hit the shared edge.
- If max spikes without warning, a neighbor burst arrived.
- If p50 and p95 move together, the queue is growing.
Keep every run in a file. One screenshot teaches nothing. A week of runs reveals the pattern. Save the time window with each sample.
Now the three myths.
Myth 1: "My latency is my own"
Queueing theory says it plainly. Delay grows as arrivals speed up. A shared line includes other tenants. Their load becomes your tail.
Evidence in your own data: run the probe at two different hours. Keep the prompt and concurrency identical. Watch p95 move while nothing on your side changes. That is your neighbor fingerprint.
Corrected mental model: think of a shared buffer, not a private pipe. Your result is a distribution. The shape moves with outside demand. Label every sample with its time window. A snapshot is not a baseline.
Myth 2: "More concurrency means more throughput"
Parallel calls can increase speed. On a shared server, that stops somewhere. Extra requests add queue time after the knee. The probe table shows it. p50 rises, p95 rises, maximum rises. Throughput stops climbing. You are borrowing from your neighbors.
Corrected mental model: concurrency is a loan, not a budget. Test in small steps. Find the level where tail latency bends upward. Back down one level. Use that as your hard cap.
Myth 3: "A healthy endpoint stays healthy for my whole job"
Endpoint health has a half-life. Free tiers share capacity with many tenants. A quiet block can turn crowded in minutes. Your earlier success is not current load.
You will likely meet this pattern. Probe at 9am, and the tail looks flat. At 3pm, another tenant starts a batch job. p50 stays similar, p95 triples. This is an illustration, not a promise. The lesson still holds.
Corrected mental model: refresh measurements before important jobs. Do not reuse a week-old health label. A short smoke test beats a stale baseline. Measurement is rent, not property.
Where a free server honestly helps
This is where MonkeyCode's free model access and free server option fit. Free model access gives you an endpoint for the probe. The free server can host scheduled runs. You check contention without paying a VM bill. Keep the schedule light and polite.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Who should not use this method
The probe is a diagnostic tool, not a benchmark. Several people should skip it.
- Teams with contract SLAs and audit requirements.
- Production traffic measurement for customer-facing SLOs.
- Users probing an endpoint without the owner's permission.
- Engineers who need one clean number for a slide.
The last group should use a load-testing tool behind a private queue. Shared-fate probes are for learning. They are not for revenue tables.
Limitations
The probe measures one variable: latency spread. It cannot separate scheduling, model routing, or network paths. It never shows all tenants. It shows your luck at that exact moment. Treat outputs as relative signals. Run the probe across several days before drawing conclusions. Also note: p50 is a summary, not a promise. The tail carries the neighbor story.
The corrected mental model
- Latency is shared state.
- Concurrency is a loan.
- Health is a timed sample.
- Retest before you trust an old number.
A free model server is never empty. Every millisecond you spend is borrowed. Stop asking "is the server slow?" Ask "what did my neighbors just do?" Then measure your answer.
