d-Matrix presented its Raptor 3D-DRAM accelerator at Hot Chips 2026 with a design built around stacking DRAM and compute logic rather than relying on conventional external high-bandwidth memory. ServeTheHome’s live coverage describes the architecture as an attempt to bring very high memory bandwidth directly into a generative inference accelerator. The company’s reported figures are ambitious. Technical coverage describes a 32 GB 3D-DRAM design delivering more than 100 TB/s of internal bandwidth with very low energy per bit. Those are vendor claims presented around the Hot Chips disclosure, so they should be evaluated through independent benchmarks and production deployments before being treated as universal performance results. The architecture is still worth paying attention to because inference is increasingly constrained by memory movement, not only raw compute. Mr.PlanB’s guide to AI and GPU workloads on Kubernetes explains the surrounding infrastructure problem: accelerator utilization depends on scheduling, storage throughput, networking, drivers and data movement across the stack.
Inference has a different bottleneck from model training
Training receives much of the attention in AI infrastructure because it requires large clusters and enormous capital investment. Inference is different. Once a model is deployed, the system repeatedly moves model weights and working data through memory while generating output for users. Large language models can therefore become memory-bandwidth intensive. A processor may have enough arithmetic capability to calculate the next token but still wait for data to arrive from memory. HBM has become the standard answer in high-end accelerators because it provides much more bandwidth than conventional server memory. The disadvantage is cost, packaging complexity and power. d-Matrix is arguing that a different memory architecture can improve the economics of inference by placing DRAM much closer to the compute fabric. Whether Raptor can deliver that advantage at scale remains to be proven. The important architectural idea is that inference hardware does not have to copy the same memory design used by every training accelerator.
Specialized inference hardware changes cluster design
If inference accelerators become more specialized, infrastructure teams may end up operating more heterogeneous clusters. Training nodes, general GPU nodes and inference-optimized accelerators can have different driver stacks, scheduling requirements, networking profiles and failure behaviors. Kubernetes can provide a common orchestration layer, but it does not make every accelerator interchangeable. Device plugins, node labels, workload placement, model formats and runtime software still have to match the underlying hardware. Mr.PlanB’s comparison of managed Kubernetes and bare metal is relevant because accelerator-heavy workloads often force teams to decide how much hardware control they need. A managed environment can reduce operational burden, while bare metal can provide more direct control over devices, network topology and performance tuning. New accelerators increase the importance of that decision. A platform designed around one dominant GPU stack may need additional engineering before an alternative inference architecture can be introduced cleanly.
Memory innovation does not remove storage and networking requirements
Raptor is focused on memory bandwidth inside the accelerator. Model deployment still requires data to reach the node in the first place. Weights may come from object storage, shared filesystems or local caches. Requests enter through network services. Outputs may be logged, ranked or passed to other applications. A fast accelerator can still sit idle if the surrounding system cannot supply work reliably. This is why infrastructure architecture needs to separate the memory wall inside the accelerator from the data pipeline around it. Solving one bottleneck can simply expose the next one. Mr.PlanB’s guide to Kubernetes business workloads emphasizes storage, networking, security, observability and recovery alongside orchestration. Those considerations apply to AI inference even when the hardware is highly specialized. The practical test is end-to-end throughput per unit of cost, power and operational effort, not the peak bandwidth of one component.
Alternative accelerators create a portability question
The AI infrastructure market is increasingly interested in alternatives to dominant GPU platforms, but hardware choice is only one layer of portability. Applications depend on model runtimes, compiler stacks, kernels, libraries and observability tooling. If an accelerator requires substantial software changes, the migration cost can offset part of the hardware advantage. That does not mean specialized hardware is unattractive. It means buyers need to evaluate the full operating model. How easily can workloads move? Which model formats are supported? How are failures surfaced? What tools exist for monitoring utilization and memory pressure? How difficult is capacity expansion? These questions are familiar from every infrastructure transition. A benchmark can prove that a component is fast. Production readiness requires evidence that teams can deploy, operate and recover it repeatedly.
Raptor is evidence that the AI memory architecture is still unsettled
The most interesting part of d-Matrix’s Hot Chips presentation is that the industry has not converged on one permanent answer to the memory problem. HBM will remain important because it offers proven bandwidth and a large ecosystem. At the same time, 3D stacking, near-memory compute and specialized inference architectures are creating alternatives that may be better suited to specific workloads. For infrastructure teams, the safest assumption is that future AI clusters will become more heterogeneous. That makes orchestration, observability and workload portability more important, not less. Mr.PlanB’s broader Kubernetes hub is useful for understanding the control layer that sits above changing hardware choices. The hardware underneath may evolve quickly, but production teams still need repeatable deployment, scaling, networking, storage and recovery. d-Matrix Raptor should therefore be treated as a technical signal rather than a settled replacement for HBM-based accelerators. It shows that vendors are attacking AI inference from the memory side, and that the next competitive gains may come from changing how data is physically placed around compute.
Originally published on the Mr.PlanB blog.
