The race to speed up AI no longer rides on building more powerful GPUs alone. d-Matrix has detailed Raptor, its upcoming inference architecture, which will stack DRAM directly with logic to reach up to 100 TB/s of bandwidth with 32 GB of capacity, per the specs the company presented. The goal is to cut one of the biggest costs in today’s models: constantly moving huge amounts of data between memory and compute.
Raptor and 3D DRAM in 20 seconds
- d-Matrix proposes 32 GB of 3D DRAM and around 100 TB/s of bandwidth.
- The company measures transfer energy near 0.37 pJ/bit with its technology.
- The logic sits directly on the DRAM through 3D integration.
- The aim is to approach SRAM bandwidth with much higher capacity.
- Raptor is built specifically for AI model inference.
The technology didn’t come out of nowhere. d-Matrix introduced its 3DIMC concept in 2025 and later built Pavehawk, an experimental chip it says validated stacking DRAM within its power and performance targets. Raptor is the commercial product meant to bring the architecture to data centers.
The approach is interesting because it tackles a problem growing right alongside the models: accelerators have to store ever larger weights and hold KV (Key-Value) caches that swell with longer contexts and more concurrent users.
One of d-Matrix’s slides shows the issue with Kimi K3. Assuming 8-bit weights, the company estimates a model size of about 2,800 GB. For its KV cache example, a one-million-token context takes 14.6 GB per user; with 64 concurrent users, that’s around 935 GB.
The message is simple: memory is becoming one of the main physical limits in AI inference.
Between SRAM and HBM, a third option
Current architectures face a hard trade-off.
SRAM is extremely fast and can sit very close to the compute units. d-Matrix’s current Corsair accelerator uses an SRAM-based architecture precisely to keep latency low during inference.
The company puts the bandwidth of a couple of Corsair cards at around 300 TB/s, with latency near 1 nanosecond.
The problem is capacity.
A conventional 6T SRAM cell takes far more space than a DRAM cell, and its density isn’t enough to store today’s enormous models. d-Matrix puts the practical limit of its SRAM solution around 4 GB per card pair.
HBM (High Bandwidth Memory) solves part of that.
By stacking many DRAM chips and connecting them through interposers, manufacturers can offer hundreds of gigabytes next to GPUs and accelerators. NVIDIA, AMD, and nearly the whole AI acceleration industry rely on it.
But HBM has physical limits too.
The number of stacks around the processor is capped by the space in the package, and raising bandwidth means more and faster connections.
d-Matrix estimates that HBM4-based configurations will reach around 20 TB/s in next-generation systems, well below what SRAM can deliver.
Its answer is to change the geometry of the problem.
Instead of putting memory around the processor, Raptor places the logic directly on top of the DRAM.
100 TB/s without a traditional HBM interface
The 3DIMC (3D In-Memory Compute) architecture uses very short vertical connections between memory and compute.
d-Matrix explains that this removes much of the physical interface needed when data has to move between separate chips.
Its earlier documentation already pointed to roughly 20 TB/s per stack, about ten times the per-stack bandwidth it uses as a reference for HBM4, at 0.3 to 0.4 pJ/bit.
Raptor builds on that.
The presented configuration pairs a logic die made on TSMC’s N4 process with custom DRAM, joined by face-to-face links with roughly 36-micron spacing.
The company proposes a 32 GB configuration that can deliver around 100 TB/s of bandwidth.
The energy figure is just as notable: 0.37 picojoules per bit.
For comparison, d-Matrix references an HBM4 setup of about 192 GB and 18-20 TB/s, at several picojoules per bit.
Take the comparisons carefully. These are different architectures with different capacities, and the metrics come from the manufacturer. Raptor is also still in development, and no commercial platform has shipped to verify the figures in independent use.
d-Matrix does have experimental silicon. It says Pavehawk validated its 3D DRAM technology, with tests across voltages and temperatures recording about 0.4 pJ/bit even in the toughest scenarios studied.
The big saving comes from moving less data
The energy cost of memory is becoming a real problem for AI data centers.
It’s not enough to measure how much a matrix multiplication consumes. You also have to count the energy to fetch data from memory, move it to the processor, and send results back.
Greater physical distance between memory and compute generally raises those costs.
d-Matrix calculates that a hypothetical HBM architecture reaching 100 TB/s at 2.4 pJ/bit would need about 1.92 kW just to move data from memory.
The 3D architecture cuts that distance sharply.
DRAM banks can connect directly with tensor engines placed on top. Raptor packs 256 tensor engines per chiplet, per the technical specs.
The company also aims to physically line up the memory banks with those engines to cut internal movement further.
That’s why d-Matrix talks about memory-centric computing. The goal isn’t just faster DRAM but redesigning memory, compute, and interconnects together.
3D stacking brings new challenges too
Stacking logic and memory one on top of the other removes some limits and adds others.
A big one is temperature.
Compute circuits run much hotter than plain DRAM. If the processor sat underneath, that heat would have to pass through the memory layers before reaching the cooling system.
d-Matrix chose to put the logic on top, so it can touch a cold substrate.
Still, the DRAM has to run hotter than usual. The architecture is designed to hold a junction temperature up to about 105°C.
That means adjusting the memory refresh process.
Raptor’s DRAM refresh cycles run roughly every 4 milliseconds, about eight times more often, per the documentation. The company estimates this uses about 1.37% of the available bandwidth.
Manufacturing is another challenge.
A 3D architecture only makes economic sense if memory defects don’t force scrapping whole sets. d-Matrix adds spare banks and error correction to raise fault tolerance.
That includes a Reed-Solomon T=2 scheme with CRC, meant to correct errors within transferred blocks.
Raptor targets the inference phase
d-Matrix isn’t trying to beat GPUs across every workload.
Its architecture is tuned for low-latency generative inference, where the balance between compute and memory bandwidth differs from training.
Token generation can be heavily memory-bound. Model weights have to be read again and again as the KV cache for each user’s context grows.
Bigger models, longer contexts, and more concurrent users all push harder on the memory subsystem.
That’s why d-Matrix is also building heterogeneous architectures where GPUs and specialized accelerators handle different parts of the process. It suggests pairing Corsair with GPUs in disaggregated pipelines can meaningfully improve inference, though results depend on the model and configuration.
Raptor is the next phase.
The company is developing the accelerator with tech partners like Alchip, while Andes Technology supplies RISC-V AX46MPV cores for orchestration and vector compute. d-Matrix has confirmed Raptor will be its first commercial product based on 3DIMC.
The current challenge is how much of the experimental silicon’s performance carries into production.
Hitting 100 TB/s with 32 GB of memory at about 0.37 pJ/bit would be a rare mix of bandwidth and efficiency. But Raptor also has to prove sustained performance, reliability, manufacturing cost, chip yield, and large-scale production.
The industry is hunting for answers here. The next generation of AI infrastructure won’t just need more FLOPS; it will need to feed those FLOPS with more data without turning memory into the system’s energy and performance bottleneck.
Frequently Asked Questions
What is d-Matrix’s 3D DRAM memory?
It’s an architecture that stacks logic and DRAM physically to create very short vertical connections. d-Matrix calls the technology 3DIMC and plans to use it commercially in its Raptor accelerator.
What bandwidth does Raptor promise?
The configuration d-Matrix announced targets roughly 100 TB/s with 32 GB of 3D DRAM. These are manufacturer specs for an architecture still in development.
Will 3D DRAM replace HBM?
Not necessarily. HBM offers high capacity and a broad industrial ecosystem. d-Matrix positions its 3D DRAM as an alternative, especially suited to inference workloads where bandwidth and data-movement power are critical.
How much energy does Raptor’s memory use?
d-Matrix reports about 0.37 pJ/bit, with earlier Pavehawk tests near 0.4 pJ/bit. The commercial platform’s full performance will be confirmed once Raptor is available.
via: wccftech
