Most LLM code I see is written as if a human is staring at a spinner. Every request is synchronous, every job fans out into 50 parallel workers, and the bill comes in at full price.
But a lot of that work runs at 3 a.m. Nobody is waiting for it. And all three big vendors will cut the price in half for exactly that kind of work — you just have to tell them you're willing to wait.
Two words first. A token is roughly ¾ of a word; models bill per million of them, with input (what you send) cheaper than output (what the model writes). A Batch API is a separate endpoint where you upload many requests at once, walk away, and download the answers later.
The deal, as written by the vendors (checked 15 Oct 2026)
- OpenAI: "50% cost discount compared to synchronous APIs." Each batch completes within 24 hours, often sooner. Up to 50,000 requests or 200 MB per batch. Works for chat, responses, embeddings, moderations and images.
- Anthropic: "a 50% discount on all usage compared to standard API prices" — input, output, everything. Up to 100,000 requests or 256 MB; "most batches completing within 1 hour", hard expiry at 24 hours. Requests that expire are not billed.
- Google Gemini: "50% of the standard cost", target turnaround 24 hours, "in majority of cases, it is much quicker." Jobs expire after 48 hours.
Same lever, same size, everywhere: you give up the guarantee of an answer in seconds, and you get half the price. That's the whole trade.
What it looks like in dollars
A realistic background job: every night, summarize the day's 20,000 support tickets for an internal dashboard. Each ticket is about 1,500 tokens in and 200 tokens out. I'll use GPT-5.6, whose official price is $2 per million input tokens and $12 per million output.
TICKETS, IN_TOK, OUT_TOK, NIGHTS = 20_000, 1_500, 200, 30
PRICE_IN, PRICE_OUT = 2.00, 12.00 # GPT-5.6, $ per 1M tokens, official
def monthly(discount):
m_in = TICKETS * IN_TOK / 1e6 # 30M input tokens per night
m_out = TICKETS * OUT_TOK / 1e6 # 4M output tokens per night
night = m_in * PRICE_IN + m_out * PRICE_OUT
return night * (1 - discount) * NIGHTS
print(f"sync: ${monthly(0.0):,.0f}/month") # $3,240
print(f"batch: ${monthly(0.5):,.0f}/month") # $1,620
$3,240 versus $1,620 a month, for the same summaries landing on the same dashboard before anyone arrives at work. The only code change is where you send the requests and when you collect them.
Notice what parallelism does to this number: nothing. Fifty workers finish the job in twenty minutes instead of three hours, but they pay full price for every token. Concurrency buys speed; batch buys price. At 3 a.m. you don't need speed.
Which jobs can wait
A quick test: if the answer arrived tomorrow morning, would anyone notice? If not, it belongs in a batch. Typical candidates:
- Labeling — tagging tickets, classifying reviews, marking spam for a training set.
- Summarizing an archive — last year's call transcripts, a backlog of documents.
- Embeddings — turning texts into number vectors for search. Re-embedding a knowledge base is the textbook batch job (OpenAI batches embeddings directly).
- Nightly reports and evals — running your test prompts against a new model version.
One objection I'd raise myself: "24 hours is too unpredictable for a nightly job." In practice the vendors say most batches return far sooner, and you can always fall back to the synchronous API for whatever hasn't returned by your deadline. You pay full price only for the stragglers.
Where waiting loses
Batch is wrong whenever the next step depends on the previous answer.
- Anything interactive. A chat, an autocomplete, a support bot. The user is the spinner.
- Agent chains. An agent that plans, calls a tool, reads the result and decides again is a sequence, not a pile. Ten dependent steps at "usually under an hour" each is an afternoon; at the 24-hour worst case it's ten days. Each step is half price, the whole is useless.
- Small volumes. Fifty requests a day saves you cents and costs you a polling loop, a results parser and error handling for expired requests. Not worth the code.
The rule: batch the wide part of the job (thousands of independent items), keep the narrow part (dependent steps, humans waiting) synchronous.
What this doesn't solve
Batch doesn't make a bad prompt cheaper — a 1,500-token ticket you could trim to 600 is still waste at half price. Results can come back in any order, so you match them by your own ID. Anthropic notes that batch and prompt caching discounts stack, but cache hits inside a batch are "best-effort", so don't budget on them.
And an honest gap on our side. We run a gateway that resells these same models 10–25% below the vendors' official prices — GPT-5.6 at $1.70/$10.20 against $2/$12, which puts the job above at $2,754 a month. We have no batch endpoint and no embeddings. A gateway discount and a vendor batch discount are different levers: ours applies to synchronous, interactive traffic; batch's 50% applies only to work that can wait, and right now you get it only by going to the vendor directly. For the 3 a.m. job, vendor batch beats us.
Open your billing page and find the job that runs while everyone sleeps. Then ask it whether it really needs an answer in seconds.
