Open-source AI models have once again grown to sizes that seemed hard to manage just a few years ago: Kimi K3 reaches 2.8 trillion parameters and DeepSeek-V4-Pro hits 1.6 trillion. The rise of Mixture of Experts (MoE) architectures lets only a small part of the model activate to process each token, but it doesn’t eliminate a physical problem: all of those parameters still have to be stored and available. For d-Matrix, this new race is turning memory — not just compute capacity — into one of the main bottlenecks in inference.
The AI memory race in 20 seconds
- Kimi K3 reaches 2.8 trillion parameters, though it activates 104 billion per token.
- DeepSeek-V4-Pro uses 1.6 trillion in total and 49 billion active.
- MoE models cut the compute needed, but their weights still require memory.
- HBM delivers capacity and bandwidth, while SRAM prioritizes speed.
- d-Matrix is betting on stacking DRAM directly next to compute.
The paradox is that models may need to compute over relatively few parameters at any given moment while, at the same time, requiring an infrastructure capable of keeping a giant library of weights available for whenever they’re needed.
Kimi K3 shows this clearly. Moonshot AI specifies 2.8 trillion total parameters and 104 billion activated per token, around 3.7% of the model. Its architecture contains 896 experts and selects 16 for each token.
DeepSeek-V4-Pro follows a similar idea at a different scale: 1.6 trillion total parameters, of which 49 billion are activated while processing each token.
It’s precisely this gap between stored parameters and active parameters that’s changing how accelerators are designed.
MoE Models Save Compute, But They Don’t Make the Weights Disappear
Mixture of Experts makes it possible to build very large models without running all of their internal networks on every operation.
Instead of always using every parameter, a selection mechanism decides which experts should step in for each token. The result can substantially cut the math work compared with a dense model of equivalent size.
But the experts that don’t take part in a given token don’t disappear.
They’re still part of the model.
When another token needs one of them, the system has to be able to reach its parameters quickly. That forces it to keep enormous amounts of information in memory, or to design mechanisms capable of moving it in from slower layers without stalling inference.
This is where the so-called memory wall comes in.
An accelerator can have huge numbers of math units and still leave them partly idle because memory can’t deliver data fast enough.
In generative inference, the problem is especially visible during the decode phase, when the model generates the response’s tokens one after another.
Adding more compute capacity doesn’t necessarily fix that wait.
HBM, SRAM, and DRAM Solve Different Problems
The industry has several technologies for building an AI system’s memory hierarchy, but none of them combines huge capacity, extreme bandwidth, low latency, low power, and low cost all at once.
| Technology | Main advantage | Main limitation |
|---|---|---|
| SRAM | Very low latency and huge bandwidth | Limited capacity and high cost per bit |
| HBM | High bandwidth with substantial capacity | Cost, power, packaging, and capacity limited per accelerator |
| Conventional DRAM | High capacity at lower cost | Lower proximity and bandwidth relative to the accelerator |
| 3D DRAM next to compute | Shortens distance and can raise bandwidth | Greater manufacturing complexity, heat, and integration |
SRAM is extremely fast and can sit very close to the compute units. The problem is that it takes up a lot of silicon area.
Building hundreds of gigabytes or terabytes of SRAM around an accelerator would hardly be viable.
HBM has been the industry’s answer for AI GPUs. Several stacks of DRAM sit next to the processor and connect through very wide interfaces, delivering several terabytes per second.
But models are growing faster than the memory available on each accelerator.
The usual fix, then, is to spread the model across more GPUs.
And that introduces a new cost: communication between accelerators.
More GPUs Also Means Moving More Data
Spreading a gigantic model across dozens of cards delivers the capacity needed, but it forces those cards to constantly exchange information.
The internal network becomes part of inference’s critical path.
NVLink, InfiniBand, high-speed Ethernet, and other technologies try to ease that problem, but moving a piece of data to another card is still more expensive than finding it inside the same accelerator.
d-Matrix argues that increasing the amount of memory available near compute can cut down on those transfers.
The company has spent several years developing accelerators built specifically for inference, and it’s now working with DRAM stacked directly on top of the compute logic. Its upcoming Raptor architecture uses exactly this 3D approach to shrink the physical distance between data and the units that process it.
The idea has a simple consequence: if a card can locally hold a much bigger slice of the model, fewer hops between accelerators are needed.
That doesn’t mean networking stops being necessary. Models with several trillion parameters still require distributed systems.
But cutting every unnecessary transfer can improve latency, power consumption, and hardware utilization.
Kimi K3 Shows How Far the Problem Has Grown
Kimi K3’s size helps put the problem in perspective.
Moonshot AI unveiled a 2.8 trillion-parameter model in July, with 896 experts and 104 billion active parameters per token. It also has a context window of roughly one million tokens.
Its design uses MXFP4 quantization for the weights, which considerably cuts the memory needed compared with storing each parameter in FP16 or BF16.
Even so, deploying a model this size still requires substantial infrastructure.
Quantization reduces the number of bits used per parameter, but it doesn’t change a basic reality: 2.8 trillion parameters is a lot of data even when each one takes up just a few bits.
And the memory needed during inference isn’t limited to the weights, either.
You have to add activations, internal buffers, and especially the KV cache, whose footprint can grow as context length and the number of concurrent requests increase.
That explains why the next round of competition between accelerators probably can’t be summed up in FLOPS alone.
Throughput and Latency Can Pull in Opposite Directions
There’s another problem for those running AI services.
GPUs can process many requests together using batching. Grouping requests raises hardware utilization and usually improves the total number of tokens processed per second.
But waiting to fill batches and then processing them can raise the latency each individual user experiences.
Providers are therefore trying to balance two different metrics:
throughput, or how many tokens the whole infrastructure can produce, and latency, or how long each user waits to receive them.
d-Matrix builds a good part of its architecture specifically around low-latency inference with small batches.
The company argues that bringing memory closer to compute makes it possible to sustain a higher data flow without relying as much on large batches to use the processing units efficiently. That’s a claim from the manufacturer that will need to be checked against comparable commercial products once Raptor reaches the market.
The company has already shown early silicon for its 3D DRAM technology and is working with Alchip on its commercial integration.
Stacking Memory Brings Its Own Problems
Physically bringing DRAM and compute closer together looks like an obvious solution — until you try to manufacture it.
An accelerator’s logic generates quite a bit of heat.
DRAM, for its part, is sensitive to temperature because its cells need to hold onto the electrical charge that represents the data.
Stacking one on top of the other requires carefully managing cooling, memory refresh, and reliability.
The layers also have to be connected through an enormous number of microscopic links, and the whole assembly has to be manufacturable at economically viable yields.
Academic work on Raptor’s early silicon describes specific techniques for managing temperature, redundancy, error correction, and DRAM refresh.
That’s why the proposal is considerably more complex than simply “putting memory on top of the chip.”
The potential payoff comes precisely from accepting that manufacturing complexity in order to simplify another part of the system: moving the data.
The Race Is No Longer Just About Building a Faster GPU
For much of the generative AI boom, infrastructure performance has been explained through the number and type of GPUs available.
The way models are evolving is making that description less and less complete.
A facility can have enormous compute capacity and still be limited by memory, interconnect, or power.
The new MoE models make that situation more pronounced.
Kimi K3 activates only 104 billion of its 2.8 trillion parameters per token. DeepSeek-V4-Pro activates 49 billion of 1.6 trillion. It’s a very efficient way of avoiding unnecessary computation, but it forces a system to keep a far larger set of weights ready to be used.
The industry is responding from several directions.
HBM keeps increasing capacity and speed. High Bandwidth Flash is trying to create a new, high-capacity layer between HBM and SSDs. Interconnect makers are raising bandwidth between accelerators. Other designs use specialized SRAM.
d-Matrix is betting on changing memory’s physical layout.
None of these technologies necessarily has to completely replace the others. It’s more likely that future AI systems will end up using several layers of memory, each one handling the data that best fits its capacity, latency, and bandwidth characteristics.
The comeback of models with more than a trillion parameters shows why this issue is gaining importance.
MoE techniques have made those sizes usable again without having to run trillions of parameters to produce each token.
Now the problem has shifted.
It’s no longer just about having enough operations per second — it’s about getting the right parameter to the right place exactly when the accelerator needs it.
Frequently Asked Questions
How many parameters does Kimi K3 have?
Kimi K3 has 2.8 trillion total parameters, according to Moonshot AI. Its Mixture of Experts architecture activates roughly 104 billion of them to process each token.
How many parameters does DeepSeek-V4-Pro have?
DeepSeek specifies 1.6 trillion total parameters and 49 billion active parameters per token for DeepSeek-V4-Pro.
Why does an MoE model need so much memory if it only uses a few parameters?
Because every expert is part of the model and has to stay accessible even though only some of them activate for each token. MoE mainly cuts compute, but it doesn’t eliminate the need to store the weights.
What does stacking DRAM directly on top of an accelerator achieve?
It shortens the physical distance data has to travel and can massively increase the bandwidth available for compute. In exchange, it raises the complexity of packaging, cooling, and memory management.
source: d-matrix.ai
