SambaNova Compiles Models Into a Rack

SambaNova's most important product is not its accelerator chip. It is a compiled model deployment that spans the chip, three memory tiers, a 16-RDU rack, and the software routing requests across them.

That system boundary separates SambaNova from a conventional accelerator card. A model configuration becomes a Processor Executable Format binary, its weights occupy a planned memory hierarchy, and SambaStack deploys the result as a model service rather than treating every request as an arbitrary program arriving at a general-purpose processor.

The thesis: SambaNova compiles models into a rack. The Reconfigurable Dataflow Unit, or RDU, is the execution engine, but the useful commercial unit is the complete inference system around it.

TL;DR

• The RDU maps a model's dataflow graph onto programmable compute, memory, and communication resources instead of repeatedly launching isolated kernels.
• SN40L combines distributed SRAM, HBM, and attached DDR; the memory system is part of model placement, not merely storage behind the processor.
• The mature deployment boundary is SambaRack SN40-16 plus SambaStack on-premises, or the same architecture exposed through SambaCloud.
• SambaNova's software compiles specific model, sequence-length, batch-size, and speculative-decoding configurations into deployable binaries and validates whether a deployment fits available RDU memory.
• SN50 extends the thesis toward larger scale and disaggregated inference, but its strongest public evidence remains preview benchmarks and vendor roadmap claims rather than broad production measurements.

Dataflow changes what the compiler owns

CUDA's programming model presents a GPU as a device executing kernels. Each kernel performs a bounded operation, while software schedules launches and moves tensors through the memory hierarchy. Fusion can reduce that traffic, but the execution model remains kernel-oriented.

SambaNova makes a different cut. Its current Dataflow Architecture description shows a grid of programmable compute units and SRAM-based programmable memory units connected as a streaming pipeline. While one operator computes, the system can fetch data for the next. Intermediate activations stay near the operators consuming them.

The underlying idea predates the current product naming. SambaNova's architecture material describes an RDU as a tiled array of compute and memory units joined by a programmable communication fabric. The SN40L paper presented at MICRO 2024 adds the current system details: dataflow cores contain compute units, memory units, address-generation units, and an on-chip reconfigurable network.

This does not mean the RDU executes without instructions or memory movement. It means the compiler can plan more of the communication path and keep a longer section of the model graph resident as a spatial pipeline.

flowchart TD
    A[Model checkpoint] --> B[Conversion and compilation]
    B --> C[PEF: compiled model configuration]
    A --> D[Model weights]
    C --> E[ModelProfile: supported runtime shapes]
    D --> F[Model: checkpoint and tokenizer]
    E --> G[Model and profile pair]
    F --> G
    G --> H[Optional ModelBundle]
    G --> I[ModelDeployment]
    H --> I
    I --> J[Memory-fit legalizer]
    J --> K[Inference router and runtime]
    K --> L[Attached DDR: model catalog and cache]
    L --> M[HBM: weight streaming]
    M --> N[Distributed SRAM: active data]
    N --> O[PCU and PMU dataflow pipeline]
    O --> P[OpenAI-compatible API response]

The diagram exposes the adoption trade. A familiar checkpoint enters the system, but it does not remain an arbitrary framework program at runtime. The current SambaStack deployment model defines a PEF as a compiled model binary for an RDU. A ModelProfile groups compatible PEFs and the runtime shapes they support. A Model identifies checkpoints and tokenizers, while a ModelDeployment creates replicas and a routable endpoint. ModelBundle is now optional: it groups several model/profile pairs when they need to be deployed, validated, or shared as one unit.

That compilation boundary is where SambaNova gains control over data movement. It is also where model support, graph coverage, compiler maturity, and artifact lifecycle become commercial constraints.

Three memory tiers make the rack the machine

The RDU's second architectural choice is capacity placement. The current SambaRack SN40-16 datasheet specifies 16 SN40L RDUs with three memory tiers:

The same rack contains host CPUs, NVMe storage, management components, and 400/200 GbE networking. SambaNova publishes 10 kW typical and 16.4 kW maximum inference power for the air-cooled rack. Those are rack specifications, not measured energy per token.

This hierarchy supports a different capacity strategy from placing every active model entirely in HBM. Large DDR capacity can hold more weights and configurations close to the RDUs. HBM provides the high-bandwidth path, while SRAM feeds the active graph. SambaNova's claim is that this enables fast switching among resident models and reduces repeated transfers from external storage.

The software reveals how tightly memory and deployment are coupled. SambaStack runs a legalizer before deployment to verify that the selected PEFs and checkpoints fit the node's memory constraints. A deployment can include multiple models, sequence-length profiles, batch sizes, and a target/draft pair for speculative decoding. The runtime then selects among compiled configurations according to the request.

The deployable object is therefore not "a model on an RDU." It is a memory-valid deployment distributed across a rack.

The commercial boundary has three forms

SambaNova now exposes the same architectural thesis through different operational boundaries.

The on-premises path is not a simple appliance with one network cable. SambaNova's current installation guide requires a front-end data network, a private inter-rack network, a management network, customer identity and DNS services, storage, load balancing, and a Kubernetes control plane. SambaStack installs through Helm and exposes deployed models through an OpenAI-compatible API.

At the other end, SambaCloud hides the compiled artifacts and rack operations. Its July 2026 release added prompt caching and continued expanding API compatibility, tool use, structured output, and model support. That surface is easier to adopt, but it offers the curated models and configurations SambaNova has prepared rather than arbitrary code execution.

This is the commercial read: the buyer chooses how much of the system boundary to own. Infrastructure operators can run a private RDU system, let SambaNova operate dedicated capacity in their facility, or consume the architecture as an API without seeing the compiler boundary.

SN40L is deployed; SN50 is the next claim

The generation boundary matters because SambaNova's public material now discusses SN40 and SN50 together.

SN40L is the fourth-generation RDU behind the documented SambaRack and the current on-premises software guides. Independent deployment evidence exists. The Argonne Leadership Computing Facility opened its Metis system to researchers in October 2025 with two 16-RDU SambaRacks. Its current user guide describes a larger six-rack SN40L cluster serving multiple models through OpenAI-compatible endpoints.

Earlier systems also operated alongside conventional supercomputers. Lawrence Livermore National Laboratory integrated SambaNova dataflow hardware into its computing environment for cognitive simulation research. That establishes real installation history, although the older system used an earlier RDU generation and does not prove current LLM-serving economics.

SN50 is SambaNova's fifth-generation inference processor. The company says a 16-chip SambaRack averages 20 kW, offers five times the compute and four times the network bandwidth of SN40, and can connect as many as 256 RDUs across racks. Its February 2026 announcement said customer shipments would begin in the second half of 2026.

The most interesting SN50 direction is not a larger standalone rack. It is disaggregated inference. SambaNova demonstrated NVIDIA GPUs handling compute-heavy prefill while RDUs handled memory-bound decode. A July 2026 demonstration used four H200 GPUs for prefill and one 16-RDU SN50 rack for decode on MiniMax M2.7.

The evidence has advanced beyond first-party performance charts, but it remains early. SambaNova's summary of a SemiAnalysis-run benchmark reports roughly 800 tokens per second at the highest-interactivity point and roughly 400 tokens per second as throughput approached the fastest B200 configuration tested on the same model. The available public summary does not provide the power, utilization, availability, or economic data required for a complete rack comparison.

That changes the competitive framing. SambaNova does not need to replace every GPU to become useful. It can become a specialized decode tier behind a serving layer that sends each phase to different hardware.

Where the architecture fits

Artificial Analysis currently reports SambaCloud output rates above 700 tokens per second for some gpt-oss-120b configurations and above 400 tokens per second for MiniMax M2.7. Those measurements establish that the public service can deliver fast single-request generation. They do not establish aggregate rack throughput, sustained concurrency, power efficiency, availability, or total service cost.

That distinction matters. The architecture is optimized around planned model execution, but a production service also contains queues, routers, caches, network paths, replicas, failures, and model-specific quality choices. Tokens per second from one endpoint are evidence about the service experience—not a complete system benchmark.

What's missing

SambaNova publishes enough material to understand the shape of the system. It does not publish enough comparable data to close the economic argument.

There is no current, public evidence set that places SN40 or SN50 beside competing systems with the same model, precision, context distribution, batch policy, concurrency, accuracy target, measured wall power, and availability window. The SN40L MICRO paper is detailed, but its headline comparisons focus on SambaNova's Composition of Experts workload rather than a neutral industry benchmark. The SN50 material is earlier still.

Compiler coverage is the other open boundary. SambaStack documents stable, preview, deprecated, and removed PEF and checkpoint versions. That is healthy operational machinery, but it also shows that model support is a maintained product matrix. A framework checkpoint being available does not mean every shape, feature, quantization, or serving optimization is immediately deployable on an RDU.

So what

SambaNova is best understood as a system for controlling model movement.

The RDU turns a model graph into a spatial dataflow pipeline. The memory hierarchy decides what remains resident and what streams into active execution. The compiler turns framework artifacts into hardware-specific configurations. SambaStack packages those configurations into a rack-level service, while SambaCloud hides the machinery behind an API.

That full boundary is the product. Comparing one RDU with one GPU misses it.

The unresolved question is whether SambaNova can keep compiler and artifact delivery ahead of changing model architectures while proving that phase-specific routing still wins after network, utilization, power, and operational costs are included.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.