A node with eight GPUs can run for weeks without issues and still be considered too unreliable for training that utilizes thousands of machines simultaneously. The apparent contradiction is that when a synchronous task depends on all participants remaining available, a small chance of individual failure multiplies, turning disruptions into a routine part of operations.
A recent infographic summarizes the problem with a striking figure: a daily failure rate of 1.5% per node would result in an 84.8% chance of at least one incident in a system of 1,000 GPUs. The calculation is correct under certain specific conditions, but other claims in the image require more context. The reliability of a cluster cannot be deduced solely from a formula or the published percentage per server.
Key Points on GPU Cluster Reliability in 20 Seconds
- Distributed training often synchronizes thousands of GPUs at each step.
- A single failed node can halt or slow down all others.
- With 1,000 GPUs spread across 125 nodes, a 1.5% daily failure rate per node yields an 84.9% chance of at least one incident.
- The calculation assumes independent failures and nodes with eight GPUs each.
- In practice, correlated errors from network, power, cooling, firmware, and software also occur.
- Effective reliability is measured by actual training time, not just individual server availability.
- Quick detection and isolation of a node are often more valuable than full diagnostics before acting.
- Checkpoints reduce lost work but can overload storage systems.
- Newer systems allow removing a damaged replica while the rest continue training.
- The figure of one failure every 30 minutes in 100,000 GPUs isn’t directly derived from the 1.5% example.
Technical literature confirms that disruptions become less exceptional as cluster size increases. Training Llama 3 on 16,384 H100 GPUs recorded 466 interruptions over 54 days, while ByteDance documented over a hundred failures spanning several weeks operating MegaScale with 12,288 GPUs. Distributed system studies also report incidents occurring every one or two days in configurations of around 1,000 accelerators.
How does 1.5% become an 84.9% probability?
The formula used in the image is:
P(at least one failure) = 1 − (1 − p)^N
where p is the daily failure probability per node, and N is the number of nodes participating in the task.
A system with 1,000 GPUs doesn’t necessarily mean 1,000 independent elements. If each server hosts eight GPUs, the infrastructure comprises 125 nodes. With a 1.5% daily failure rate per node, the probability that none fail is:
0.985^125 ≈ 15.1%
Therefore, the chance that at least one fails is approximately 84.9%. The infographic’s figure is accurate assuming this distribution of eight GPUs per node, a consistent failure rate, and independent failures.
This doesn’t imply that 84.9% of trainings will completely fail. It indicates that, on any given day, there’s that probability at least one of the 125 nodes experiences the included failure type. The final impact depends on training architecture, incident duration, and the ability to remove or replace the faulty machine.
Also, what counts as a failure varies: a GPU that stops responding, a non-recoverable ECC error, HBM issues, NVLink link loss, degraded network interface, a frozen process, or a server that an operator proactively removes after detecting anomalous metrics.
Therefore, excluded nodes don’t directly equate to failed hardware inventory. A machine may be sidelined due to confirmed failure, intermittent degradation, or behavior that risks slowing down thousands of accelerators in collective operation.
Work done on a production cluster with 504 Nvidia B200 GPUs illustrates this challenge. Researchers analyzed 751 Prometheus metrics and ten failures identified via XID codes, but no single metric consistently dominated all error types. They also observed that three of the 63 nodes accounted for over half of the exclusions, indicating that problems are not always evenly distributed.
Why a 1.5% failure rate doesn’t directly translate to a 30-minute failure interval
The infographic states that a 100,000-GPU cluster would experience a failure roughly every 30 minutes. The general idea is reasonable: at that scale, incidents are frequent. However, this figure isn’t simply derived by extrapolating the 1.5% daily failure rate.
With eight GPUs per node, 100,000 accelerators would require about 12,500 machines. If each has an independent daily failure probability of 1.5%, the expected number of failures per day would be 187.5, leading to an average interval of about 7.7 minutes, not 30.
A failure every 30 minutes would correspond to roughly 48 incidents daily across 12,500 nodes, with an individual node failure probability of around 0.38% per day. Alternatively, the figure could result from a narrower failure definition, different architecture, or operational data using different assumptions.
The reference cited for that estimate, arXiv:2605.09370, doesn’t analyze a 100,000-GPU cluster either. It studied 63 nodes with 504 B200 GPUs over 55 days of metrics and 73 days of logs, offering valuable insights into detection, checkpoints, node exclusion, and retries, but not directly supporting a 30-minute failure interval for a 100,000-GPU setup.
There is specific research on training of approximately 100,000 GPUs. Its authors report an effective efficiency of only 44% in a fully synchronous setup, due to the frequency of incidents and recovery times. The proposed FT-HSDP system reduces recovery pauses from ten to three minutes and boosts effective training time to 80%. This work doesn’t need to assume a failure every half hour to illustrate the scale of reliability challenges.
The key point isn’t the exact error interval, which varies with hardware and methodology. What’s important is that architectures designed to halt on any failure significantly reduce productive capacity when operating with tens of thousands of components.
Building reliability beyond hardware
Buying higher-quality servers helps but doesn’t solve the problem outright. In synchronous training, all participants exchange gradients or intermediate results. If one responds late, disappears, or loses bandwidth, others may be forced to wait.
Infrastructure needs mechanisms to detect which node causes issues, remove it, and reorganize the workload. Speed in decision-making often outweighs initial diagnostic precision.
Waiting several minutes to confirm whether the issue is memory, GPU, driver, or network link causes machines to be kept offline longer than necessary. Isolating first and analyzing later can restore capacity faster, especially if reserve groups or architectures allow operation with fewer participants.
This approach explains the use of pre-checks, activity signals, Nvidia DCGM metrics, NCCL logs, and periodic network tests. Systems seek both obvious errors and lagging nodes that still function but reduce overall group performance.
Partially degraded network links can be particularly harmful. Some libraries reroute traffic via alternative interfaces to prevent total failure, but the affected server may still slow down collective operations. Recent algorithms, like AllReduce variants, aim to maintain near-normal performance even when one node is bandwidth-limited.
Operationally, the goal shifts from keeping all machines healthy at all times to preventing a problematic unit from dragging the whole system down.
Checkpoints save training but at a cost
Traditional failure handling involves periodically saving the model state. When interrupted, the system reloads the latest checkpoint and resumes from there.
More frequent saves reduce lost work but require transferring large amounts of data from GPUs to server memory and storage. In large models, this can generate bursts that saturate network or filesystem bandwidth.
A study of 504 B200 GPUs analyzed 523 save/load operations. During recovery, read bandwidth reached only 21.5% of maximum, and write bandwidth was about 16%. Researchers observed queues at NFS, RPC, and transport layers, highlighting that fast storage alone doesn’t guarantee performance when hundreds access simultaneously.
Projects like DLRover aim to lessen this overhead by saving states in memory and asynchronously persisting data. They report that combining fault tolerance with quick recovery improved the goodput of training a GLM-65B model across thousands of GPUs from 69% to 95%. This metric, published by the project itself, directly measures the proportion of time spent producing useful new work.
A GPU might be busy waiting for communication, redoing lost computations, or loading checkpoints. Device utilization doesn’t equate to productivity.
From full restart to replica-based recovery
Recent training systems aim to break the dependency of all nodes on the entire task at all times.
FT-HSDP employs data-parallel replicas: when a GPU or server fails, only its replica is removed. Others keep progressing while the affected one resets, loads its state, and rejoins via an incorporation protocol.
This shifts fault tolerance units from the entire 100,000-GPU training to multiple smaller groups capable of enduring the temporary loss of one.
Asynchronous recovery also reduces the need for massive idle hardware reserves. Capacity is still needed for replacement but managed as part of the cluster rather than as a full duplicate of training.
Data from the 504-GPU cluster shows automatic retry chains succeeded 33.3% of the time versus 12.5% for manual retries, with a median of eleven minutes between attempts. While not always perfect, systematic responses can outperform ad hoc interventions.
What an AI infrastructure operator should measure
The published failure rate per GPU or server is just one piece. Effective monitoring must track how many disruptions occur per thousand GPUs per hour, detection times, work lost, and the minutes to restore previous performance levels.
It should also categorize incidents by source: GPU, HBM, NVLink, network interfaces, switches, storage, power, cooling, drivers, and software. Without this classification, a hardware improvement might be overshadowed by a bottleneck shifted elsewhere.
The claim that HBM memory surpassed GPU compute as the main source of incidents in a 32,000-accelerator training is plausible given the complexity of stacked memories but isn’t documented in the sources cited. Until the original report is available, treat this as an unverified assertion rather than industry-wide fact.
Reliability of a large cluster isn’t found in GPU specs alone. It results from observability, isolation, reserve capacity, rapid checkpoints, and a training model accepting machine failures.
In small setups, failure remains an incident. In thousands-of-GPU systems, it’s a normal operating condition. The difference between an expensive and a productive cluster lies in what happens during the minutes following a failure.
Frequently Asked Questions
Is the 84.8% failure probability for 1,000 GPUs accurate?
Yes, assuming 125 nodes with eight GPUs each, an independent daily failure rate of 1.5% per node, and the same probability across all nodes. The result is roughly 84.9%.
Does a node incident always stop training entirely?
Not necessarily. It happens in highly synchronized designs, but fault-tolerant systems with replicas can remove only the affected group while others keep operating.
Does a 100,000-GPU cluster fail every 30 minutes?
It can experience frequent incidents, but the interval depends on the failure rate and what defines a failure. The 30-minute figure isn’t directly derived from the 1.5% example.
Which metric best describes cluster reliability?
The goodput, representing the percentage of time spent generating new, useful work, is more indicative than the mere availability of individual GPUs.
