The race to train ever larger language models is hitting a less visible problem than the size of parameters or data quality: hardware logistics. In practice, setting up large-scale machine learning infrastructure is no longer just about “buying more GPUs,” but about sourcing, integrating, and making them work together seamlessly—without turning the system into a compatibility puzzle.
This is where HetCCL comes in, a new collective communication library introduced by a team of researchers affiliated with Seoul National University and Samsung Research. Their approach targets a specific bottleneck: efficiently and transparently using heterogeneous clusters with GPUs from different manufacturers to train language models and other massive deep learning workloads.
The real problem: communication, not computation
In distributed training, much of the time isn’t spent on “calculating,” but on synchronization. Operations like all-reduce, all-gather, or reduce-scatter are essential for aggregating gradients and maintaining consistency across nodes. In homogeneous environments, the industry has well-tuned tools: NVIDIA promotes NCCL as the de facto standard for its GPUs, while AMD offers RCCL within its ROCm ecosystem.
Problems arise when the cluster stops being uniform. As organizations combine generations, models, and vendors to scale capacity, the communication layer becomes a barrier: backends are optimized for “their” hardware, and interoperability between vendors isn’t always cleanly supported in common workflows. Infrastructure teams are familiar with the consequences: higher costs, resource underutilization, and at worst, abandoning parts of the installed hardware because they “don’t fit” into the same training environment.
What HetCCL proposes
HetCCL positions itself as a collective communication library designed to operate between NVIDIA and AMD GPUs within the same cluster, leveraging RDMA for direct, fast transfers, without requiring driver modifications. Instead of “reinventing” communication from scratch, the approach integrates and utilizes existing optimized libraries (NCCL and RCCL) through mechanisms that coordinate collective operations in a heterogeneous setting.
The practical goal is ambitious but concrete: enable distributed workloads (e.g., via PyTorch) to use GPUs from different vendors without rewriting training code or redesigning the stack from scratch. In other words, turn heterogeneity from an engineering nightmare into a buying decision—adding capacity without complex technical overhead.
RDMA as a shortcut: bypassing CPU (when needed)
The key technical enabler is RDMA (Remote Direct Memory Access), a family of technologies that allow low-latency access to remote memory with minimal OS intervention. In GPU environments, this translates into something especially valuable: a NIC with RDMA capabilities can interact directly with GPU memory, avoiding unnecessary intermediate copies and reducing CPU load.
According to the authors, HetCCL establishes direct point-to-point communication leveraging RDMA and memory registration mechanisms, enabling the network to “see” GPU memory regions via specific APIs (CUDA/HIP) and common RDMA network stacks (such as IB Verbs). Practically, this fits with the prevalent high-performance network fabrics in AI at scale: InfiniBand and RoCE (RDMA over Converged Ethernet) in certain deployments.
Performance: approaching “native” speeds without blocking costs
In their evaluations, the team reports that HetCCL achieves performance comparable to NCCL and RCCL in homogeneous setups, and scales well in heterogeneous scenarios. The figures show high efficiencies compared to baseline runs, with averages near 90% and peaks approaching 97% in specific cases. They also observe near-linear scaling when increasing from 8 to 16 GPUs in their tests.
Another common concern in such integrations is “quality”: does changing the communication back-end affect training? The work indicates that the numerical differences in final loss are within small tolerances (relative errors below 7 × 10⁻³ in their comparisons), which is crucial for teams that can’t afford surprises in convergence caused by backend switches.
Why this matters for sysadmins and platform teams
For operational and development-focused teams, HetCCL offers operational advantages:
- Reduced dependency on a single vendor: if communication is no longer a bottleneck, purchasing strategies can prioritize availability, total cost, and schedule.
- Reusing inventory: GPUs that aren’t identical can coexist in the same cluster under a shared goal, rather than being secondary hardware.
- More realistic scaling: in real life, clusters grow in waves and budgets, not in perfect homogeneous purchases.
- Deployment efficiency: if no code changes are needed, the adoption barrier lowers—especially in mature organizational pipelines.
However, it’s important to highlight the flip side: this kind of approach depends heavily on RDMA infrastructure (networks, NICs, compatibility, tuning) and requires rigorous isolation and observability. In AI clusters, the network isn’t an accessory—it’s core to performance. Any layer touching communication must incorporate monitoring, testing, and realistic expectations.
The context: heterogeneity is no longer an exception
The push to train larger models with budgets that don’t grow proportionally drives hybrid infrastructures—more pragmatic than ideal. HetCCL aligns with this trend: assuming heterogeneity will be the norm rather than an anomaly, and that software (not just silicon) will determine how much capacity can be truly unlocked from a data center.
If this approach gains traction beyond academia, it could provide midsize companies and labs with a way to reduce friction in building powerful clusters—without falling into complete vendor lock-in. Most importantly, it tackles a common organizational challenge: “We have GPUs, but we can’t use them together.”
Frequently Asked Questions
What is a collective communication library (CCL), and why does it matter so much for training large language models?
A CCL accelerates operations like all-reduce or all-gather, which are key for synchronizing gradients and parameters across GPUs. If this communication is slow or inefficient, the cluster spends more time waiting than computing, dramatically increasing iteration costs.
Can I train an LLM with both AMD and NVIDIA GPUs in the same cluster without modifying PyTorch code?
HetCCL aims to enable exactly that—using different vendors’ GPUs without changing training code—by replacing the communication layer with a backend capable of operating in a heterogeneous environment. Actual adoption depends on integration with each stack and RDMA configuration.
What network requirements are typically needed for RDMA to make a difference in AI workloads?
Usually, a network with RDMA capabilities (InfiniBand or RoCE), compatible NICs, and proper tuning to minimize latency and packet loss. In AI deployments, network setup and observability are as critical as GPU hardware choices.
What benefits does direct GPU memory communication (GPUDirect RDMA) bring to distributed workloads?
It reduces intermediate copies and CPU intervention, decreasing latency and freeing system resources. In distributed training, this can boost effective cluster performance and improve scalability as node counts increase.
via: quantum zeitgeist and HetCCL: Accelerating LLM Training with Heterogeneous GPUs