Harbor evals on Vercel Sandbox: benchmarking without a big machine

You can now run Harbor evals on Vercel Sandbox, which means benchmarks like Terminal-Bench and SWE-bench no longer need a machine you personally own. Vercel announced it in their changelog on 17 September 2026.

The feature is one flag. What it changes is who gets to verify a model's claims instead of taking a vendor's word for them. That second part is why I think it is worth 1,000 words.

🔍 One flag, and your laptop stops being the ceiling

Harbor is the open-source harness behind Terminal-Bench, and its registry carries other benchmarks too: SWE-bench, tau3-bench, OSWorld. Until now, running one of those meant your own hardware set the pace. Add --env vercel to harbor run and each trial gets its own isolated Firecracker microVM.

The command from the changelog:

harbor run -d terminal-bench/terminal-bench-2-1 \
  --agent fx \
  --model vercel_ai_gateway/anthropic/claude-fable-5 \
  --env vercel \
  --n-concurrent 8

That --n-concurrent 8 is the whole pitch. Eight trials at once is trivial in a datacentre and painful on a 16GB laptop that is also running your editor.

🛡️ The firewall sits outside the VM, and that is the part worth copying

Two details in the announcement matter more than the parallelism, and they are design lessons even if you never type --env vercel:

1. A task's network policy is enforced at the sandbox firewall, outside the VM. The thing being sandboxed does not get to define its own sandbox. If the policy lived inside the microVM, a sufficiently capable agent could edit it.
2. Optional credential injection attaches secrets to matching outbound requests at that firewall, so the secrets never enter the sandbox at all.

Key takeaway: the agent gets the result of an authenticated request without ever holding the credential. That is the correct shape for running untrusted or semi-trusted code, and most homegrown eval setups I have seen get it wrong by mounting a .env file into the container.

If you are building anything that executes model-written code, steal that boundary. Enforcement belongs one layer above the thing you do not trust.

💰 Parallelism is a budget decision, not a capability

Here is where I will be blunt, because the changelog is not: Vercel published no pricing, no free-tier allowance, and no trial limits in this post. I am not going to guess at numbers. But the cost structure is obvious enough to reason about:

The token bill is the one that surprises people. A terminal-agent benchmark is not one prompt per task; it is a loop of tool calls, file reads and retries, and a hard task can run dozens of turns. If you want a sanity estimate before you commit, our AI agent cost calculator models exactly that multi-turn shape, and the LLM API price comparison will tell you what swapping providers does to the total.

Because swapping is the point. Vercel pairs this with AI Gateway, where one AI_GATEWAY_API_KEY reaches hundreds of models across providers. Benchmarking a second model is the same command with a different --model:

harbor run -d terminal-bench/terminal-bench-2-1 \
  --agent fx \
  --model vercel_ai_gateway/openai/gpt-5.6-luna \
  --env vercel \
  --n-concurrent 8

🌐 Why this lands differently from Colombo than from San Francisco

If you work from Sri Lanka, the constraint on evaluating AI models has never been curiosity. It has been the box.

• A 16GB laptop running a Docker-based eval harness at concurrency 8 will thrash, and you cannot do anything else while it runs.
• Long unattended runs and grid power are an uneasy pair. A cloud fleet does not care if your house does.
• Nobody here is expensing an H100 workstation to satisfy a hunch about which model is better at shell tasks.
• Benchmarks are batch work. Latency to a European or US region is irrelevant when the job takes hours.

So the asymmetry this closes is real: the person deciding which model to build on can now be the person actually running the benchmark, not the one reading someone else's leaderboard screenshot. For a two-person team in Colombo choosing between two providers for a client build, that is the difference between an opinion and a measurement.

I would still treat public leaderboards as a starting point rather than an answer. If you want the published numbers side by side before you spend anything, we keep an LLM benchmark comparison for that. Then go measure on your own task.

🧪 How I would run a first eval without torching the budget

1. Read the step-by-step guide first. Vercel says setup, configuration and troubleshooting live there. Do not improvise the config.
2. Confirm your Harbor version. Below 0.22.0, --env vercel does not exist.
3. Run a handful of tasks, not the suite. Prove the plumbing works before you pay for 200 trials that fail on a misconfigured key.
4. Start at low concurrency, read the bill, then raise --n-concurrent. Concurrency changes how fast you spend, not how much.
5. Only then run two models. The comparison is the deliverable; a single model's score in isolation tells you very little.

Warning: a benchmark score is evidence about the benchmark. Terminal-Bench measures terminal-agent competence. If your product is a Sinhala-language support bot, a high Terminal-Bench number is close to meaningless for you. Build a small eval set from your own real inputs and run that alongside.

🚀 What this means for you

If you are a student or a solo builder, the practical change is that a serious, reproducible eval run is now an expense line instead of a hardware requirement. You can decide to spend on it. Previously you often could not spend your way in at all.

If you run a small team, the change is procurement. "Which model should we use" stops being a taste argument and becomes a run you can attach to a proposal. Clients notice that.

And if you build agent infrastructure, copy the security shape even if you host elsewhere: policy at the firewall, credentials injected outside the VM, isolation per trial. That design survives an agent that tries to escape. A .env mounted into a container does not.

Bottom line: the barrier to checking an AI claim yourself just dropped from "own a workstation" to "have a card on file". Use that before you pick your next model.

Facts here come from Vercel's changelog post linked above. Where the source gives no number, I have not supplied one.