Samsung is working on a profound evolution of HBM memory for AI accelerators. Their proposal culminates in zHBM, an architecture that aims to place memory vertically over the processor, reduce data travel distances, and eliminate elements of current 2.5D packaging. The manufacturer projects up to a 230% increase in bandwidth and around 70% less DRAM power consumption compared to a conventional HBM4E setup, though these are still objectives presented by Samsung and not yet commercial product specifications.
Key points of Samsung zHBM in 20 seconds
- Samsung proposes evolving from conventional HBM to a three-dimensional architecture called zHBM.
- The memory would be positioned vertically over the xPU, such as a GPU or AI accelerator.
- The company estimates up to 230% more bandwidth and 70% less DRAM power consumption.
- This development will follow several intermediate stages of customized and advanced HBM.
This proposal was explained by Samsung in a technical presentation about the evolution of the die base: the chip located at the bottom of HBM stacks. The main idea is for this component to progressively take on functions that currently reside in the processor, beyond just communication, control, and testing.
This change addresses an increasingly evident problem in AI systems: making GPUs faster is of little use if memory can’t supply data at the necessary speed.
Current HBM is beginning to face physical limits
High Bandwidth Memory (HBM) was developed specifically to solve bandwidth issues. Instead of placing conventional DRAM chips around the processor, multiple memory layers are stacked and connected via vertical TSVs (Through-Silicon Vias).
An HBM stack mainly consists of two elements. The core dies or C-die contain the DRAM cells where data is stored, while the base die or B-die beneath manages communications and various control and testing functions.
The stack then interfaces with the GPU, TPU, or other accelerators through a physical or PHY interface.
Samsung identifies this interface zone as a key obstacle to further performance increases. The PHY takes up significant space on the base die, and memory-to-processor communications have space and energy costs.
Historically, the evolution of HBM shows bandwidth being the primary focus. Samsung cites approximately 3 TB/s per stack with HBM4, around 4 TB/s with HBM4E, and over 6 TB/s projected for future HBM5. These last generations are part of ongoing technological development, so such figures are target architecture goals rather than finalized specifications.
However, increasing pin counts, connection numbers, or speeds indefinitely becomes more challenging.
AI exacerbates the issue because it’s no longer just about increasing model size; longer contexts and multi-user environments significantly boost the KV cache, the temporary memory used during inference to store token information from previous steps.
This results in dual pressure on: capacity and bandwidth.
Samsung aims to make HBM base more active and integrated
Samsung’s strategy unfolds in three major phases.
The first involves reducing the area occupied by traditional physical interfaces.
In what’s called custom HBM (cHBM), Samsung suggests replacing part of the conventional PHY with direct die-to-die (D2D) connections. Shortening these connections aims to reduce power consumption and free surface area for the accelerator.
Samsung estimates this reorganization could free roughly 5-10% of the xPU’s area, with potential performance improvements of 10-20%, depending on the design. These are Samsung’s estimates and not generalizable to all GPUs.
Proximity of components also introduces thermal challenges.
To address this, Samsung has developed its Heat Path Block (HPB) technology, designed to improve heat evacuation paths. According to data from the company, it could reduce maximum temperature in certain interface zones by over 35%.
The next step is even more intriguing: shifting some functions from the GPU directly into the HBM base die.
Possible functionalities include the memory controller. Additional integrated features could encompass repair mechanisms via SRAM, advanced telemetry, RAS (Reliability, Availability, and Serviceability) functions, internal testing, and controllers capable of connecting additional memory modules.
This would transform memory from a relatively passive component beside the processor into an active participant in system operation.
AHBM would bring processing closer to memory
The second phase extends this idea further.
Samsung is exploring the integration of Processing Elements (PEs) within the memory, creating what it calls Advanced HBM or AHBM.
The motivation stems from a major cost of modern computing: data movement consumes energy.
GPUs can perform mathematical operations extremely quickly but need to continuously transfer data from memory to compute units and back, consuming significant energy.
Reducing this movement by performing operations closer to data can greatly improve efficiency.
This doesn’t mean HBM will replace GPUs. Instead, specific functions could be moved into the memory environment when doing so reduces communication and power draw.
Samsung also considers using the base die to incorporate expansion interfaces capable of connecting with external memory. This is particularly useful for language model inference with large context windows, where KV cache can consume tens or hundreds of gigabytes.
This leads into the third stage of the proposal.
zHBM would place memory directly on top of the processor
Current AI accelerators typically use a 2.5D packaging approach.
The GPU and several HBM stacks are placed close together and connected via an interposer. This offers much higher bandwidth than conventional memory outside the package but still requires data to move laterally between components.
zHBM aims to change this geometry.
Samsung proposes stacking memory vertically over the xPU itself. The result is a 3D architecture where processing and memory are connected through extremely short vertical links.
This approach eliminates the need for a traditional 2.5D interposer, allowing input/output connections to be distributed over a larger surface area.
Samsung is exploring packaging technologies like Wafer-on-Wafer (WoW) and hybrid bonding techniques to achieve very high connection densities.
The company targets a communication energy consumption close to 0.5 picojoules per bit.
Compared to a conventional HBM4E setup, simulations and estimates for zHBM project a 230% higher DRAM bandwidth, approximately 70% lower memory power consumption, and potential for about 100 watts of additional thermal margin for other system parts.
Again, these are technological goals set by Samsung. zHBM still faces challenges related to manufacturing, thermal performance, costs, reliability, and large-scale production.
Memory and processor functionalities are starting to merge
Samsung’s approach reflects a broader industry shift.
For decades, it was relatively straightforward to distinguish between processor and memory: one performs calculations, the other stores information.
AI is blurring that line, as models require moving enormous amounts of data, and energy spent just transporting this data has become a significant portion of total consumption.
As a result, memory and accelerator designers are experimenting with compute-near-memory, processing-in-memory, custom memory, and 3D packaging.
Samsung has a particular advantage here, as it participates simultaneously in DRAM manufacturing, logic processes, and advanced packaging technologies.
zHBM aims to leverage these capabilities.
However, placing memory on top of a processor that can dissipate hundreds of watts isn’t trivial. Excess heat can impair DRAM data retention, increase refresh requirements, and complicate cooling strategies.
Manufacturing also needs sufficiently high yields: integrating multiple chips vertically means a defect in one can jeopardize a much more expensive assembly.
Therefore, Samsung isn’t proposing to jump directly from HBM4 to zHBM.
Instead, its path advances first through custom HBM, then adds new functions to the base die via AHBM, and finally culminates with the 3D integration proposed for zHBM.
The overall technological direction is clear: the next leap in AI accelerators won’t depend solely on building faster GPUs.
It also involves ensuring that data is located much closer to where processing occurs.
Frequently Asked Questions
What is Samsung zHBM?
zHBM is a memory architecture proposed by Samsung that vertically integrates HBM over an xPU, such as a GPU or AI accelerator. It aims to reduce communication distances, increase bandwidth, and lower power consumption.
How does HBM4 differ from zHBM?
HBM4 maintains a architecture where processor and memory stacks are connected via advanced packaging. zHBM aims for a 3D integration where memory is directly placed on top of the processor.
What performance can zHBM deliver?
Samsung projects up to 230% more DRAM bandwidth and around 70% lower power consumption compared to a conventional HBM4E setup. These are estimates and development goals, not final commercial specifications.
When will Samsung zHBM be available?
No specific commercial release date has been announced. Samsung positions zHBM as the third stage after custom HBM and Advanced HBM in its evolution roadmap.
via: wccftech
