HotKV cut Claude token use by 70% in our Banking77 benchmark

A semantic cache promises to cut LLM spend by answering repeated and similar questions from memory instead of calling the model. We measured that end to end: the same 3,000 customer support requests, sent to each of four Claude models twice, once straight to the model and once with HotKV in front, with Anthropic's own prompt caching left on in both runs. Here is exactly what we ran and what it found, including where it falls short.

In short. HotKV cut model calls, input plus output tokens and API cost by about 70% on each of the four models. Most of that comes from questions that repeat exactly. The semantic layer adds roughly 12 points on top. At the default 0.81 similarity threshold, cached answers were judged to fit the exact question 5.8 to 10.7 percentage points less often than fresh answers. On the one model we swept (Sonnet 5.5), a threshold of 0.88 or higher showed no such gap.

Updated 12 October 2026: this article now reports our larger run on four current Claude models. It replaces an earlier 1,500-request run on GPT and Claude Sonnet 5 that measured about 60% fewer tokens, which we no longer present.

What we tested

The workload

We replayed customer support questions from the test split of Banking77 (PolyAI, CC BY 4.0), a public dataset of short banking questions labelled with 77 intents. It was introduced in Casanueva et al., 2020. The 3,000 requests were drawn with a fixed seed. Intent popularity follows a Zipf distribution with exponent 1.0, a synthetic skew in which a few intents come up far more often than the rest, and the question within an intent is picked uniformly, with repeats. All 77 intents appear, and 1,264 of the 3,000 requests are distinct questions (counted by replaying the seeded request generator). The bank and its support policy are fictional. We wrote a policy of about 2,200 tokens, sent as the system prompt on every request, that asks for two to four short sentences.

Models and settings

We ran claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5 and claude-haiku-5-5. Fable 5.1 and Opus 5.5 ran at their lowest effort setting, because their thinking cannot be switched off. Sonnet 5.5 ran with up-front thinking off, its lowest setting, and Haiku 5.5 with thinking disabled. Maximum output was 2,000 tokens for Fable and Opus and 400 for Sonnet and Haiku, and every answer finished normally rather than being cut off. Eight requests ran at a time in both modes, calls were not streamed, and each condition ran once.

The two runs

• Baseline: every request goes straight to the model, with Anthropic's prompt caching on (cache_control on the system prompt, default five-minute lifetime). This is the configured baseline used here, not a claim that every provider-side optimization was exhausted.
• With HotKV: HotKV sits in front of the same traffic and the same system prompt, with the cache starting empty.

What we counted

Every token count comes from the usage block of the API response: uncached input, cache reads and writes, and output (thinking included). Embedding tokens are separate. Costs use Anthropic's list prices of 11 October 2026, include the embedding calls for semantic lookups, and exclude HotKV hosting and operations. They are not a quote for current pricing.

How the cache works

For each question, HotKV tries two lookups before the model is called:

1. Exact match: a GET on a SHA-256 key built from the model, the system prompt version and the normalized question (lower-cased, whitespace collapsed). It catches a question asked in identical words.
2. Semantic match: an SCACHE.GET call with an embedding of the question (OpenAI text-embedding-3-small, cosine similarity). It looks for a similar question. A similarity score alone cannot guarantee the cached answer fits.

Only a miss on both reaches the model, and its answer is written back with SET and SCACHE.SET. The similarity threshold for a semantic hit was fixed at 0.81 before the run. We chose it on a separate Banking77 train split as the lowest value, in steps of 0.01, whose semantic hits pointed at the same intent at least 98% of the time (98.2% there). In this run, on the untouched test split, 96.7% to 96.8% of semantic hits matched the intent of the question asked: 30 or 31 wrong-intent hits per model out of 926 to 936.

Results

Calls, tokens and cost

Across the four models, calls fell from 12,000 to 3,588, tokens by 70.1% and cost by 69.6%. About 70% of requests were answered from the cache on every model: roughly 39% by exact match and 31% by semantic match.

Calls, tokens and dollars fall together because HotKV removes whole calls. Read the token figure with care: 90% to 93% of the counted tokens are cache reads of the same 2,200-token policy prompt, which Anthropic bills at a small fraction of the normal input price. Output tokens are only 6% to 9% of the tokens but 73% to 92% of the baseline cost.

Speed

The median response time across the whole run fell from 2.7 seconds to 127 ms on Fable 5.1, from 2.9 seconds to 126 ms on Opus 5.5, from 1.8 seconds to 128 ms on Sonnet 5.5 and from 1.4 seconds to 125 ms on Haiku 5.5, which is 11 to 23 times faster. A repeated question came back in a median of about 0.05 ms, measured by a Python client over loopback on the same machine as HotKV, so it is a local figure rather than a network one. A paraphrased question, served from the semantic cache, took about 130 ms, almost all of it the embedding API call.

The whole-run median sits inside that semantic-hit band because exact hits (about 39% of requests) are too few to carry the median on their own, while exact and semantic hits together (70%) pass the halfway mark. The slow end moves less: 30% of requests still call the model. The 99th percentile was about 6.4 seconds with HotKV against 6.6 seconds without it on Fable 5.1, and about the same on Haiku 5.5. It was lower on Opus 5.5 (4.7 seconds against about 6) and on Sonnet 5.5 (2.7 seconds against 3.3). Calls were not streamed, so these are times to the complete answer, not to the first token.

Warm-up

The cache starts empty, so early requests still reach the model. The share of requests served from the cache was about 52% in the first thousand requests and about 81% in the last thousand.

Where the saving comes from

Most of the saving is plain repetition. An exact-match cache alone would have needed at most 1,264 model calls out of 3,000, one for each distinct question, which is about 58% fewer. The semantic layer took that to about 70%, so it contributes roughly 12 points on this workload. It is also the part that carries the answer-quality risk, as the next section shows. This is derived from the request sequence and the logs, not a separate measured run.

Answer quality

A semantic hit means the same intent, not the same correct answer. To measure correctness we had a different Claude model judge every semantic hit. The judge saw the customer's question and two answers in random order: the cached answer HotKV served, and the fresh answer the same model gave to that exact question in the baseline run. It said whether each answer correctly answered that exact question. Opus 5.5 judged the answers of Fable 5.1, Sonnet 5.5 and Haiku 5.5, and Fable 5.1 judged Opus 5.5, so no model graded its own output.

At the default threshold

At 0.81, a cached answer fit the exact question 5.8 to 10.7 percentage points less often than a fresh answer from the same model. Semantic hits are about 31% of requests, so across all responses, and counting exact hits as neutral, that is roughly 2 to 3 points (1.8 to 3.3). The judges differ in strictness, so compare cached with fresh only within one model.

Raising the threshold

We reran Sonnet 5.5 at four more thresholds, each from an empty cache, with the same judge. The baseline cost was $5.51.

From 0.88 upward cached answers did not show a deficit against fresh ones, while still cutting cost by 64% at 0.88 and 61% at 0.90. The evidence is thin. At 0.90 the 95% interval for the difference runs from about 4 points worse to 7 points better, the 0.95 row has only 50 hits, and each row judges a different subset of hits. This is one model, one judge and one run per threshold, so it is not proof of equivalent quality or a universal safe threshold.

Near-identical questions

The real risk is questions that look almost identical but need different answers, such as depositing versus withdrawing. We had Opus 5.5 write 119 pairs, each a real training-split question plus a version with one decisive detail changed, and counted how many changed questions would clear the threshold against a cache holding only the original. These pairs are built to be hard, were not reviewed by a person, and show a worst case rather than how often it happens in real traffic.

Raising the threshold reduces the risk but does not remove it. Even at 0.90, 25% of the deliberately close pairs would match, and fewer true paraphrases match too.

What this means at scale

Extrapolating the measured cost per request to 1 million requests a day over a 30-day month, and assuming the same traffic pattern, gives these monthly figures:

• claude-fable-5-1: about $303,100 without HotKV, $91,800 with it, roughly $211,300 saved.
• claude-opus-5-5: about $146,000 without HotKV, $45,000 with it, roughly $101,000 saved.
• claude-sonnet-5-5: about $55,100 without HotKV, $16,600 with it, roughly $38,500 saved.
• claude-haiku-5-5: about $2,700 without HotKV, $800 with it, roughly $1,900 saved.

This is arithmetic on the measured cost per request, not another measured run. It excludes hosting, and the capacity needed for that volume was not measured. At the 3,000 requests we ran, the saving is only $0.19 for Haiku 5.5 and $21.12 for Fable 5.1. The four models ran with short answers, and Fable 5.1 and Opus 5.5 at their lowest effort, so longer or reasoning-heavy answers would make each avoided call worth more.

Limitations

This is not a general "HotKV saves you 70%" claim. It is what we measured on this workload, and the workload does the heavy lifting.

• Savings track repetition. With every intent equally likely instead of Zipf-skewed, cost fell 56% on Sonnet 5.5 and Haiku 5.5. With 3,000 all-different questions it fell 38%, entirely from semantic matches between different questions of the same intent, which Banking77 has plenty of. A workload with no repeated or similar questions saves nothing.
• Synthetic traffic. Draws are independent: no bursts, sessions or multi-turn context. One public dataset of short banking questions, a fictional policy written for this test, and a finite pool of about 3,080 test questions.
• A model judges the answers. No human reviewed them, there was no second judge, and every judge is a Claude model. Fresh answers failed the judge's "fits" test between 1% and 21% of the time, so it is a strict bar.
• Same intent is not same answer. The 0.81 threshold was calibrated on intent labels, not answer quality. The threshold sweep and the near-identical test cover one model and one embedding model.
• Small test rig. One run per condition, a single four-vCPU server with the client on the same machine, and a cache that was not warmed before the HotKV runs.
• Costs are API costs. They exclude HotKV hosting and operations. Validate answer quality on your own questions, and use exact caching or fresh answers where semantic reuse is unsuitable.

The per-model figures are also on the AI page.

Try it yourself

The exact-match cache in this test uses plain GET and SET, which every HotKV plan includes. The semantic cache (SCACHE.*) comes with the Performance plan and above (pricing). Start on HotKV Cloud and point your existing traffic at it, or talk to sales if you want to work through the numbers for your own workload first.

Originally published at hotkv.com.