A Field Guide to Beating Amazon Bedrock Throttling

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.

This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.

TL;DR

• Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).
• Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.
• Cap concurrency so you stop generating load the quota cannot absorb.
• Add a circuit breaker so a throttle storm backs off instead of hammering.
• Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.
• Use cross-region inference profiles to spread token demand across regions.
• Reserve provisioned throughput only when on-demand plus the above is not enough.

Step one: know the quota you are hitting

Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.

• Quotas for Amazon Bedrock — the overview, including that default quotas vary by account age, usage history, and region.
• Quotas for the bedrock-runtime endpoint — the runtime invoke limits you hit at inference time. View them in the Service Quotas console under Amazon Bedrock.
• Scaling and throughput best practices — how per-model TPM and RPM quotas interact.

Soft limits vs hard limits

This distinction decides your whole strategy.

• Soft limits (adjustable). In the Amazon Bedrock endpoints and quotas table these are marked Adjustable: Yes. You can raise them. Request an increase through Service Quotas; smaller increases often auto-approve, larger ones route to AWS Support for review.
• Hard limits (not adjustable). Marked Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.

The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.

One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.

Layer one: retry with backoff and jitter

The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:

• Exponential backoff. Each attempt waits longer: base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
• Full jitter. Randomize the delay inside that window. Without jitter, every throttled caller retries at the same instant and you get a synchronized stampede that throttles again. With jitter, retries spread out.

A compact pattern:

import random, time
from botocore.exceptions import ClientError

THROTTLE = {"ThrottlingException", "TooManyRequestsException"}

def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except ClientError as exc:
            code = exc.response.get("Error", {}).get("Code", "")
            if code not in THROTTLE or attempt == max_retries:
                raise
            delay = min(base * 2 ** attempt, cap)
            time.sleep(random.uniform(0, delay))  # full jitter

A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it (max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.

The same retry in a Step Functions workflow

If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.

"InvokeBedrock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Retry": [
    {
      "ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
      "IntervalSeconds": 2,
      "MaxAttempts": 15,
      "BackoffRate": 2.0,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    { "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
  ],
  "Next": "NextStep"
}

The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.

Layer two: cap concurrency

Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.

Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:

import asyncio

semaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls

async def bounded_call(fn):
    async with semaphore:
        return await fn()

Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:

resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
  event_source_arn = aws_sqs_queue.work.arn
  function_name    = aws_lambda_function.worker.arn
  batch_size       = 1
  scaling_config {
    maximum_concurrency = 5
  }
}

Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.

Layer three: a circuit breaker for throttle storms

Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.

The state machine:

• Closed (normal): calls flow, throttles are counted.
• Open (tripped): after N throttles in the window, pause all calls for a cooldown and halve the concurrency limit.
• Half-open (recovering): after the cooldown, resume at the reduced limit; if clean for a while, restore full concurrency; if it throttles again, reopen.

This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.

Layer four: handle timeouts, they are not throttling

Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.

Model timeout: shrink the work

A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.

async def call_shrinking(fn, items, *, max_retries=3):
    batch = list(items)
    for attempt in range(max_retries + 1):
        try:
            return await fn(batch)
        except Exception as exc:
            code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
            if code != "ModelTimeoutException" or attempt == max_retries:
                raise
            batch = batch[: max(1, len(batch) // 2)]  # halve and retry

Client timeout: tune the budget, then retry the transient ones

A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.

Two moves:

1. Set a realistic read_timeout on the boto3 client so a slow-but-valid response is not cut off.
2. Treat the transient connection timeouts as retryable, alongside throttling, rather than letting them bubble up as hard failures.

from botocore.config import Config

client = boto3.client(
    "bedrock-runtime",
    config=Config(read_timeout=300, retries={"max_attempts": 0}),
)

RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
# also retry ReadTimeoutError / ConnectTimeoutError by exception class name

Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.

Layer five: spread demand with cross-region inference profiles

Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.

It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.

Layer six: provisioned throughput, only when you need it

If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.

Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.

Tuning in practice

The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.

Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.

Two caveats that are easy to get wrong:

1. A "fallback model" can quietly resolve to the same model. If both the primary and fallback point at the same model id through shared config, the fallback functions as extra retries against one model, not as model diversity. Calling it a fallback overstates what it does until the config points at a genuinely different model.
2. Retry, backoff, and concurrency are not the same as reserved capacity. A defense built entirely from retries, backoff, concurrency caps, and on-demand cross-region inference profiles has no reserved-capacity floor under a true spike. That can be a deliberate on-demand posture, but name it as one, because on-demand plus retries still has a ceiling.

How to decide what to reach for

A short decision order when you see throttling:

1. Check the quota. Is the limit soft or hard? If soft and you are near it, file the increase now, in parallel with the rest.
2. Confirm retries have backoff and jitter. If not, fix that first. It is the cheapest win.
3. Cap concurrency to keep steady-state rate under the TPM limit, at both the app and the infrastructure layer.
4. Add a circuit breaker if throttle storms are recurring rather than occasional.
5. Separate timeouts from throttles. Shrink the batch for model timeouts, tune the read timeout and retry for client timeouts.
6. Spread across regions with a cross-region inference profile for on-demand traffic.
7. Buy provisioned throughput only for steady, predictable, high volume the above cannot cover.

Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.