SnapStart on containers: the bake-off against provisioned concurrency

On September 2, 2026 AWS extended Lambda SnapStart to functions packaged as container images, the format many organizations adopted because of a corporate deployment standard, or simply because they needed more than the 250 MB a .zip allows. The promise is honest: instead of pulling image layers and initializing the runtime for every new execution environment, Lambda snapshots the already-initialized environment when you publish the version and resumes from there. But trading initialization for a snapshot is not a neutral performance flag. It changes when you pay, what breaks, and where the deployment fails. This is the bake-off I would run before turning it on in production.

What actually got unlocked

The blocker was never the snapshot mechanism: Firecracker had been freezing memory and disk since the original Java SnapStart. The blocker was the lifecycle contract: a container image can carry any runtime, and Lambda needed a standard way to know when initialization was done and when the environment came back from the freeze.

That contract is now public, and it lives in the Runtime API. At the end of your initialization, if AWS_LAMBDA_INITIALIZATION_TYPE equals snap-start, you run your before-snapshot hooks and call GET /runtime/restore/next. The call blocks, the process literally parks there, until Lambda restores that environment from the snapshot and returns HTTP 200. Then you run the after-restore hooks and enter the normal invoke loop. Errors go to /runtime/init/error (which fails PublishVersion) or /runtime/restore/error (which fails the in-flight invocation and tears the environment down).

If you use the AWS base images for Java 11+, Python 3.12+ or .NET 8+, Lambda coordinates all of it and the experience is identical to .zip. If you bring your own base image, your own Runtime Interface Client, or the base images for provided.al2023, Node.js and Ruby, there are two paths: implement /restore/next, or declare LABEL com.amazonaws.lambda.feature.snapstart="Allow" in the Dockerfile. Without one of the two, publishing the version simply fails.

The SnapStart lifecycle for a container image

Initialization moves off the request path and onto the deployment path. The cost meters follow the snapshot, not the invocation.

🏗️ Build e publicação

• Dockerfile LABEL ...snapstart=Allow (ci)
• Amazon ECR imagem ≤ 10 GB descomprimida (storage)
• PublishVersion ApplyOn=PublishedVersions (ci)

❄️ Init único (tempo de deploy)

• Init runtime + pesos do modelo (compute)
• beforeCheckpoint GET /runtime/restore/next (compute)
• Snapshot Firecracker cifrado, cacheado, replicado (storage)

⚡ Invocação (tempo de request)

• Restore retoma do snapshot (compute)
• afterRestore re-semeia CSPRNG, reconecta (security)
• Handler resposta ao cliente (compute)

📊 Sinais de observabilidade

• INIT_REPORT Init Duration (data)
• REPORT Restore + Billed Restore (data)
• X-Ray subsegmento Restore (data)

💸 Medidores de cobrança

• Cache do snapshot US$0,0000015046/GB-s (mín. 3h) (external)
• Restauração US$0,0001398 por GB (external)

Flows

• dockerfile -> ecr: docker push (same Region)
• ecr -> publish: image optimization → Active
• publish -> init: Lambda initializes exactly once
• init -> hook: max(timeout, 130 s) ceiling
• hook -> snap: blocks until frozen
• snap -> restore: resume per new environment
• restore -> after: HTTP 200 on /restore/next
• after -> handler: enters the invoke loop
• init -> initreport: init cost, at deploy time
• restore -> report: cold start = Restore + Duration
• restore -> xray: replaces Initialization
• snap -> cache: per active published version
• restore -> restorecost: per restored environment

The physics: what the snapshot removes and what it does not

SnapStart removes three expensive things from the request path: downloading and optimizing image layers, loading the runtime, and your own initialization, imports, framework wiring, schema compilation, loading model weights into memory. All of it now happens once per published version, at deployment time.

What it does not remove is whatever is genuinely per-environment. Network connections opened during init have no guaranteed state after the resume; the AWS SDK usually reconnects on its own, the rest is your job in the after-restore hook. Entropy is the dangerous case: if you generated a UUID, a secret or a seed during init, that value goes into the snapshot and comes out identical in every restored environment. Lambda already helps on one sensitive point, with SnapStart active the runtime switches to container credentials (AWS_CONTAINER_CREDENTIALS_FULL_URI) instead of the access-key variables, precisely so credentials do not expire frozen inside the snapshot.

And there is a design inversion almost nobody catches on first reading: with SnapStart, /tmp is capped at 512 MB, against the 10,240 MB a normal function can configure. The classic inference pattern, pulling weights from S3 into /tmp during init, no longer fits. What is left is baking the weights into the image itself (up to 10 GB uncompressed) and loading them into memory, where the snapshot captures them already materialized. That is exactly the scenario AWS cites, but it requires rewriting how you load, not just flipping the flag.

Four ways to kill cold start on a large artifact

The math, using the published prices

The us-east-1, x86 numbers: snapshot cache at US$0.0000015046 per GB-second, restore at US$0.0001397998 per GB, provisioned concurrency at US$0.0000041667 per GB-s and on-demand duration at US$0.0000166667 per GB-s. For Java managed runtimes the documentation waives the SnapStart price; the math below applies to Python and .NET.

A 2 GB function with one published version kept active for a full month costs roughly US$7.80 in cache alone. A single provisioned concurrency unit at the same 2 GB costs roughly US$21.60 a month. Each 2 GB restore costs US$0.00028, meaning you would need approximately 77,000 restores per month for SnapStart's variable cost to match one provisioned unit. For the overwhelming majority of APIs, SnapStart wins by a wide margin.

The problem is not the unit price; it is the multiplication. The cache charge is per published version and continues for as long as the version exists, with a 3-hour billing minimum. A pipeline that publishes a version on every merge and never prunes leaves dozens of paid snapshots running behind it. Twenty forgotten 2 GB versions cost more than US$150 a month to serve exactly zero traffic. Add that Lambda periodically regenerates snapshots to apply runtime patches, and each re-run of your init is billed. A version retention policy stops being hygiene and becomes financial control.

How I would decide, by workload profile

Container + SnapStart

Pros

• Best latency-per-dollar for bursty traffic with a large artifact
• Keeps the corporate container deployment standard and the ECR pipeline intact
• Model weights loaded into memory enter the snapshot already materialized

Cons

• The 512 MB /tmp cap kills the download-from-S3-during-init pattern
• Requires a uniqueness audit: entropy, IDs and connections opened during init
• Incompatible with provisioned concurrency, no hedging possible

Verdict: The default for interactive APIs and inference with weights baked into the image.

Container + provisioned concurrency

Pros

• Double-digit millisecond startup, with no resume step at all
• Works with any runtime and with /tmp up to 10 GB
• No uniqueness requirements on the initialization code

Cons

• Roughly 2.8× the snapshot cache cost per equivalent GB-hour
• Sizing it wrong is expensive; spillover falls back to a full cold start

Verdict: Only when the SLO is strict enough that the resume step does not fit inside it.

.zip + SnapStart

Pros

• No image optimization step and no container Pending/Inactive state
• Lifecycle coordinated by the managed runtime, zero hook code

Cons

• 250 MB unzipped is not much for ML dependencies
• Moving from container to .zip requires a brand-new function, package type is immutable

Verdict: Still the simplest path when dependencies fit. Not somewhere to migrate to.

Always-on ECS Fargate

Pros

• No runtime, /tmp or state-uniqueness restrictions
• Persistent connections, warm local cache and long-running processes

Cons

• You pay for idle 24×7 and inherit scaling, patching and a load balancer
• Scaling out takes minutes, not milliseconds

Verdict: Justified by high sustained utilization, not by fear of cold starts.

What now breaks in the pipeline, not in the request

This is the part that interests me most as an architect: SnapStart moves initialization to deployment time, and it moves the failure modes along with it.

Init and the before-snapshot hooks share a combined timeout of max(function_timeout, 130 seconds). Blow past it and PublishVersion fails. An inference function that loads 4 GB of weights, and previously was merely slow on the first call, now breaks the release pipeline, which, honestly, is the right place to break, provided the pipeline treats it as a deployment error and not as a flake.

Second: SnapStart only exists on a published version or an alias pointing at one. $LATEST is never SnapStart. That means the development loop invoking $LATEST exercises a different code path than production; contract tests need to hit the alias. Third: package type is immutable. You do not convert a container function to .zip, you create another function, with another ARN, another alias, other event source mappings. Plan it as a migration, not as a toggle.

And there are the states that only show up under rare traffic. A container function left uninvoked for weeks has its optimized image reclaimed, returns to Pending and rejects the first invocation. For Java runtimes, the snapshot is deleted after 14 days without invocation and you get SnapStartNotReadyException. Both are caller-retryable errors, as long as the caller has retry with backoff and the operation is idempotent. Without that, your low-traffic function has an error built into the calendar.

How to measure this without fooling yourself

The first side effect of SnapStart is that your cold-start dashboards silently stop working. The Init Duration field disappears from REPORT, because initialization no longer happens at invocation; it moves to a separate record, INIT_REPORT, along with the duration of the before-snapshot hooks. Two new fields appear in REPORT: Restore Duration and Billed Restore Duration. They are not the same thing, the first includes work done outside the microVM, which the user waits for but you are not charged for; the second covers only runtime load and the after-restore hooks.

The formula that matters is simple and needs to land in your SLO: cold start = Restore Duration + Duration. If your alarm only looks at @duration, it will show an improvement that does not exist. In X-Ray the change is analogous: there is no more Initialization subsegment, there is Restore. Through the Telemetry API you receive platform.restoreStart, platform.restoreRuntimeDone (with status success, failure or timeout) and platform.restoreReport.

Two practical details. The AWS_LAMBDA_LOG_GROUP_NAME and AWS_LAMBDA_LOG_STREAM_NAME variables do not exist in a SnapStart function, logging libraries that depend on them break quietly. And the honest measure of impact is not function duration: it is API Gateway's IntegrationLatency or the function URL's UrlRequestLatency, at p99 and p99.9. That is the only place you see what the client actually felt.

The five mistakes I would expect in the first three months

• Turning on ApplyOn=PublishedVersions without auditing init: seeds, UUIDs and secrets generated before the snapshot become identical across every restored environment.
• Publishing a version per merge and never pruning, snapshot cache is billed per active version, with a 3-hour minimum, forever.
• Keeping the pattern of pulling model weights from S3 into /tmp during init, ignoring the 512 MB cap SnapStart imposes.
• Testing against $LATEST and concluding that 'nothing changed': $LATEST never uses SnapStart.
• Declaring victory based on @duration, without adding Restore Duration or looking at integration latency at the edge.

The real trade-off is not latency versus cost: It is determinism versus uniqueness. Provisioned concurrency keeps N independent environments, each with its own entropy, its own connections and its own lifecycle. SnapStart keeps one canonical initial state and clones it. Everything that was accidentally unique per environment becomes deliberately shared. In financial-grade systems that stops being a performance detail and becomes a correctness question: a cloned pseudorandom generator, a pre-computed idempotency token or a frozen connection pool produce bugs that never show up in load testing, they show up in reconciliation.

Reading it through the pillars

• security: The snapshot is encrypted, but any secret materialized during init gets cloned; fetch credentials in the handler and re-seed CSPRNGs on after-restore.
• reliability: SnapStartNotReadyException and the container Pending state require caller-side retry with backoff and idempotent operations.
• performance: Measure Restore Duration + Duration and validate against IntegrationLatency at the edge; the improvement is only real at p99.9.

Curator's note: I would turn this on, but in week three, not week one. The order I use is always the same: first instrument Restore Duration in a staging environment against the real alias, then audit init hunting for entropy, credentials and connections, it is always that audit that surfaces the uncomfortable finding. Only then do I touch production. I once saw a payments system whose init pre-computed an idempotency-key suffix 'to save time'; under provisioned concurrency it never collided, because each environment had its own. With a cloned snapshot it would have collided, and the incident would not have shown up in latency, it would have shown up as a duplicated transaction, two days later, at close of books. Performance you cannot reconcile is not performance, it is debt.

Recommendation

For interactive APIs and inference packaged as container images, on Java 11+, Python 3.12+ or .NET 8+ over the AWS base images, SnapStart is now the default choice: it delivers sub-second startup at roughly a third of the monthly cost of an equivalent provisioned concurrency unit, and the restore-cost breakeven sits so far out (~77,000 restores a month at 2 GB) that it rarely matters. Reserve provisioned concurrency for endpoints whose SLO cannot absorb the resume step, or that need /tmp above 512 MB. Do not migrate to .zip just to get SnapStart, package type is immutable and the benefit does not pay for the ARN swap. And before anything else, do two thoroughly unglamorous chores: a uniqueness audit of your init code, and an automated pruning policy for published versions. SnapStart is won or lost in those two.

Rating: 8.5/10

References

• AWS What's New: Lambda SnapStart for container image functions (Sep 2, 2026)
• AWS Lambda Developer Guide: Improving startup performance with Lambda SnapStart
• AWS Lambda Developer Guide: Implementing SnapStart hooks for container images
• AWS Lambda Developer Guide: Activating and managing Lambda SnapStart
• AWS Lambda Developer Guide: Monitoring for Lambda SnapStart
• AWS Lambda Developer Guide: Create a Lambda function using a container image
• AWS Lambda Pricing: SnapStart, provisioned concurrency and on-demand rates

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.