Claude Haiku 5.5 Bills 5x for Prompts Over 100K Tokens. A Node Guard That Counts First

Book: AI That Ships
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

A prompt of 99,000 tokens and a prompt of 101,000 tokens look the same in your code. On Claude Haiku 5.5 the second one costs about five times as much. As of October 2026, Anthropic's pricing page lists Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, and $0.50 / $2.50 for prompts over that line. Output gets the higher rate too.

You moved your RAG endpoint to Haiku 5.5 when it launched on October 7 because the base rate is a tenth of Haiku 4.5's. Then someone raised topK from 12 to 20 in a one-line PR, or the docs team started writing longer pages, and a slice of your traffic drifted over the line. Nothing errors. The model has a 1M-token context window, so every request still succeeds. The bill is the first place it shows up.

The fix is a small TypeScript guard that counts tokens before every request and drops the lowest-ranked retrieved chunks until the prompt fits. It also logs which tier the API actually billed, and a Vitest test fails CI when your worst-case prompt crosses the line.

What the line costs, in dollars

Take one request with 1,000 output tokens, at the list prices:

99,000 prompt tokens: 99,000 × $0.10/M + 1,000 × $0.50/M = $0.0104
101,000 prompt tokens: 101,000 × $0.50/M + 1,000 × $2.50/M = $0.0530

Two percent more prompt, 5.1 times the cost. At an illustrative 100,000 requests a day, before caching or batch discounts, that is the difference between about $1,040 and $5,300.

Two details in the long context pricing section change how you build the guard. First, "a request's prompt length counts all of its input tokens, including cache reads and cache writes." Second, "a request over the threshold pays the higher prices even when part of its prompt is a cache hit." So a 90,000-token cached system prefix plus 15,000 fresh tokens is a 105,000-token prompt, and the cache read itself moves from $0.01 to $0.05 per million. Prompt caching does not get you under the line. Only a shorter prompt does.

The threshold is per request, and "up to 100,000" means a prompt of exactly 100,000 tokens stays in the base tier. The guard treats anything over 100_000 as the long tier.

Count before you send

Anthropic's TypeScript SDK exposes the token counting endpoint as client.messages.countTokens. It takes the same model, system, messages and tools you would send to the Messages API and returns { input_tokens }. Three facts from that page shape the code:

It is free, with its own rate limit (5,000 requests per minute on the Start tier) that does not count against message creation.
The count is an estimate: "the actual number of input tokens used when creating a message might differ by a small amount." So the guard aims for 97,000, not 100,000.
Haiku 5.5 uses the newer tokenizer, so the same text produces roughly 30% more tokens than on Haiku 4.5 (the Haiku 5.5 overview says the same). Counts you measured on the old model are wrong now. Recount.

Here is the start of src/guard.ts. It builds the request in one place, so the thing you count is the thing you send:

// src/guard.ts
import Anthropic from "@anthropic-ai/sdk";

export const MODEL = "claude-haiku-5-5";
export const TIER_LINE = 100_000; // over this = long tier
export const BUDGET = 97_000; // headroom: counts are estimates

export type Chunk = { id: string; score: number; text: string };

export type Params = {
  model: string;
  system: string;
  messages: Anthropic.MessageParam[];
};

export type Counter = (p: Params) => Promise<number>;

const SYSTEM = "Answer only from the documents. Cite doc ids.";

export function buildParams(q: string, chunks: Chunk[]): Params {
  const docs = chunks
    .map((c) => `<doc id="${c.id}">\n${c.text}\n</doc>`)
    .join("\n");
  return {
    model: MODEL,
    system: SYSTEM,
    messages: [
      { role: "user", content: `${docs}\n\nQuestion: ${q}` },
    ],
  };
}

export function apiCounter(client: Anthropic): Counter {
  return async (p) =>
    (await client.messages.countTokens(p)).input_tokens;
}

Counter is a plain function type on purpose. Production passes apiCounter(client); unit tests pass a fake, so the trimming logic runs without a network or a key. Params leaves out max_tokens, which belongs to the create call, not the count.

Trim the lowest-ranked chunks until it fits

Your retriever already ranks chunks, best first. When the full set is over budget, the guard keeps the longest prefix of that ranking that fits. Token count only grows as you add chunks, so a binary search finds that prefix in about log2(n) counts instead of one count per dropped chunk:

export type Fit = {
  chunks: Chunk[];
  tokens: number;
  dropped: number;
};

export class OverBudget extends Error {}

Fit records the trimmed chunk list, its token count and how many chunks were dropped, which is what the log line reports later. OverBudget covers the one case trimming can't fix. The search itself:

// ranked: best chunk first
export async function fitToBudget(
  count: Counter,
  q: string,
  ranked: Chunk[],
  budget = BUDGET,
): Promise<Fit> {
  const all = await count(buildParams(q, ranked));
  if (all <= budget) {
    return { chunks: ranked, tokens: all, dropped: 0 };
  }
  // invariant: lo chunks fit, hi chunks do not
  let lo = 0;
  let loTokens = await count(buildParams(q, []));
  if (loTokens > budget) {
    throw new OverBudget(`${loTokens} tokens with no docs`);
  }
  let hi = ranked.length;
  while (hi - lo > 1) {
    const mid = Math.floor((lo + hi) / 2);
    const t = await count(buildParams(q, ranked.slice(0, mid)));
    if (t <= budget) {
      lo = mid;
      loTokens = t;
    } else {
      hi = mid;
    }
  }
  return {
    chunks: ranked.slice(0, lo),
    tokens: loTokens,
    dropped: ranked.length - lo,
  };
}

On the happy path this is one count per request. Over budget with 40 chunks, it is two counts plus about six more for the search. Each count is a round trip, so you pay some latency for the check. If that matters, skip the count for prompts that are obviously small: Anthropic's models overview puts 1M tokens at roughly 2.5M Unicode characters on this tokenizer. That ratio is an average, not a bound, so only use it to skip counting prompts far below the line.

Two edge cases to handle in the caller. OverBudget means the system prompt plus question alone are too big. That is a bug upstream, and trimming chunks won't fix it. And if the top-ranked chunk is itself huge, lo can end at zero and you get an empty context. Check fit.chunks.length and decide whether to answer at all.

Log the tier the API actually billed

The count is an estimate. The response usage is what you pay for, so log both. The trap here is input_tokens. The prompt caching docs say it is only the tokens "after the last cache breakpoint", and the total is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. A dashboard that charts input_tokens alone can show 15,000 for a request billed at the long tier.

type PromptUsage = {
  input_tokens: number;
  cache_creation_input_tokens: number | null;
  cache_read_input_tokens: number | null;
};

// USD per million tokens, Haiku 5.5, cache rates ignored
const PRICE = {
  base: { input: 0.1, output: 0.5 },
  long: { input: 0.5, output: 2.5 },
} as const;

export function billedPrompt(u: PromptUsage): number {
  return (
    u.input_tokens +
    (u.cache_creation_input_tokens ?? 0) +
    (u.cache_read_input_tokens ?? 0)
  );
}

billedPrompt is the prompt length the tier rule counts, so the tier check uses it and never input_tokens alone. ask runs the guard, sends the trimmed request and prices the call from what came back:

export async function ask(
  client: Anthropic,
  q: string,
  ranked: Chunk[],
) {
  const fit = await fitToBudget(apiCounter(client), q, ranked);
  const res = await client.messages.create({
    ...buildParams(q, fit.chunks),
    max_tokens: 2048,
  });
  const prompt = billedPrompt(res.usage);
  const tier = prompt > TIER_LINE ? "long" : "base";
  const p = PRICE[tier];
  const usd =
    (prompt * p.input + res.usage.output_tokens * p.output) / 1e6;
  console.log(JSON.stringify({
    event: "llm_call",
    model: res.model,
    counted: fit.tokens,
    billed_prompt: prompt,
    tier,
    dropped: fit.dropped,
    est_usd: Number(usd.toFixed(5)),
  }));
  return res;
}

est_usd prices every prompt token at the uncached rate, so it overestimates when you read from the cache and slightly underestimates when you write to it (cache writes cost more than base input on the rate card). Use it to spot the tier jump on a chart and leave invoice reconciliation to your billing data. Alert on two things: any tier: "long" line, and a rising dropped count, which tells you retrieval is fetching more than the budget can hold.

Why the guard trims instead of switching models

The obvious alternative is to send long prompts to a different model. On price alone, it doesn't help. Haiku 5.5's long tier at $0.50 / $2.50 is still below Haiku 4.5 at $1 / $5 and Sonnet 5.5 at $2 / $10 on the same rate card. Even allowing for Haiku 4.5's older tokenizer, where the same text comes out about 23% shorter, a 101,000-token Haiku 5.5 prompt is about 78,000 tokens there: around $0.078 of input against $0.0505. Switch models when a request needs a stronger model, and keep the cheaper one when the only problem is length.

When trimming would hurt the answer, such as a summary that has to read the whole document, you have three options. Split the input into chunks under the line and summarize each, then merge the results (more requests, each at the base rate). Use the Batch API, which halves the long tier to $0.25 / $1.25 if the work can wait. Or accept the long tier on purpose and let the log line say so.

A Vitest test that fails the build

Runtime trimming catches drift per request. A CI check catches it in the pull request that causes it. Keep a fixture of the biggest prompt your config can produce: the longest question you accept and topK chunks at your maximum chunk size, taken from your real corpus. Then count it.

// test/budget.test.ts
import { readFileSync } from "node:fs";
import Anthropic from "@anthropic-ai/sdk";
import { expect, test } from "vitest";
import { BUDGET, buildParams, type Chunk } from "../src/guard";

type Fixture = { question: string; chunks: Chunk[] };
const path = new URL("./fixtures/worst-case.json", import.meta.url);
const worst: Fixture = JSON.parse(readFileSync(path, "utf8"));

const hasKey = Boolean(process.env.ANTHROPIC_API_KEY);

test("CI provides a key for the budget check", () => {
  if (process.env.CI) expect(hasKey).toBe(true);
});

test.runIf(hasKey)(
  "worst-case RAG prompt stays in the base tier",
  async () => {
    const client = new Anthropic();
    const { input_tokens } = await client.messages.countTokens(
      buildParams(worst.question, worst.chunks),
    );
    expect(input_tokens).toBeLessThanOrEqual(BUDGET);
  },
  20_000,
);

test.runIf skips the network test on a laptop without a key. The first test closes the gap that opens up: GitHub Actions always sets CI=true, so a pipeline with a missing secret fails instead of silently skipping the check. The count call is free, so running it on every PR costs only rate-limit headroom.

The trimming logic gets a plain unit test with a fake counter, about four characters per token, no network:

// test/guard.test.ts
import { expect, test } from "vitest";
import { fitToBudget, type Params } from "../src/guard";

const fake = async (p: Params) =>
  Math.ceil(JSON.stringify(p).length / 4);

test("drops lowest-ranked chunks until it fits", async () => {
  const ranked = Array.from({ length: 10 }, (_, i) => ({
    id: `c${i}`,
    score: 1 - i / 10,
    text: "x".repeat(4_000),
  }));
  const fit = await fitToBudget(fake, "q?", ranked, 5_000);
  expect(fit.tokens).toBeLessThanOrEqual(5_000);
  expect(fit.dropped).toBeGreaterThan(0);
  expect(fit.chunks).toEqual(ranked.slice(0, fit.chunks.length));
});

When the budget test goes red, the PR that raised topK or the chunk size has to say why the extra context is worth five times the price. Sometimes it is, and then a reviewer signs off on the cost before it ships.

If this was useful

A cost cliff like this one is exactly what AI That Ships is about: the guardrails, eval suites and cost controls that sit between a working LLM feature and one you can leave running on Node.js. The book does it all in TypeScript, from budget checks like this guard to the tests that keep them honest in CI.