Subscribe
Infrastructure
6 min read

One Failure Every Three Hours: What Frontier Model Training Actually Does With Broken Hardware

Meta's Llama 3 paper is still the only frontier-lab disclosure of hardware failure inside a training run. Its numbers, one interruption every three hours at 16,384 GPUs, are the baseline the rest of the industry should be measured against.

Olivetti-style poster: one failure every three hours, a robot arm swapping a sparking component in a machine cabinet
Olivetti-style poster: one failure every three hours, a robot arm swapping a sparking component in a machine cabinet

AI compute is sold as inventory. Press releases count GPUs the way real-estate listings count square feet. But a training cluster is not storage. It is a machine that breaks, continuously, in the view of no one. The single most useful document about what actually happens inside a frontier training run is still Meta's disclosure from the Llama 3 pre-training campaign, and its numbers should reset any assumption that scale buys stability.

The only honest dataset

During a 54-day snapshot of the Llama 3 405B pre-training run on 16,384 Nvidia H100 80GB GPUs, the job was interrupted 466 times. Forty-seven interruptions were planned (firmware upgrades, configuration changes). The other 419 were surprises, an average of roughly one every three hours, around the clock, for eight weeks.

About 78% of the unexpected interruptions were attributed to confirmed or suspected hardware faults, per the Llama 3 paper. GPU problems were the largest single category at 58.7% of unexpected interruptions. Within the breakdown compiled from the paper by Glenn Lockwood and reported by DCD: faulty GPUs caused 148 interruptions (30.1%), and the HBM3 memory on those GPUs caused another 72 (17.2%). Software bugs in dependencies added 54 (12.9%), network switches and cables 35 (8.4%), GPU SRAM errors 19, GPU system-processor faults 17. Actual CPU failures: two, in 54 days, across thousands of hosts.

Two footnotes matter as much as the headline. First, the run still kept more than 90% effective training time, because virtually everything was handled by automation. Manual intervention was required only three times. Second, Meta is nearly alone in publishing any of this at all. No other frontier lab has released comparable interruption telemetry for its flagship runs, which makes every other cluster's reliability a vendor claim rather than a measurement.

The math that turns rare into routine

Failure rates are per-device. Training jobs are whole-fleet, synchronous, and additive. A GPU with a mean time between failures measured in years will, in a fleet of tens of thousands, fail every few hours, and one dead accelerator stalls a job spread across thousands of machines because every rank waits for every other rank in each step.

Meta quantified the scaling curve with two production research clusters (11 months, 150 million A100 GPU-hours, 4 million jobs): mean time to failure for an 8-GPU job was 47.7 days. For a 1,024-GPU job it was 7.9 hours. The paper's modeling goes further. To reach a 0.9 effective training time ratio on a 100,000-GPU run at observed cluster failure rates, checkpoint intervals and restart overhead would have to shrink to about two minutes.

Infographic: a decay curve falling as device counts grow along the axis

Newer hardware is not obviously safer. The failure anatomy shifts: in a Microsoft and Nvidia study of a production cluster, 91% of non-user GPU errors were uncontained memory or MMU faults, with NVLink interconnect errors propagating beyond a single chip in another 5%. And the unit of repair is getting bigger. An Nvidia paper on nonuniform tensor parallelism notes that GB200 NVL72 racks pair 72 GPUs into one NVLink domain with 36 CPUs, that it is often operationally simpler to discard a group of GPUs than one, and that the added cabling and HBM density give failures more ways to happen than in the H100-era systems Meta described.

Why the broken GPU is the cheapest part of the outage

The component costs a few tens of thousands of dollars. The event costs are downstream, and Crusoe's engineering team frames them in one formula: goodput equals one minus checkpoint cost, failure loss and queue wait, all multiplied by efficiency. In its worked example, replacing a three-hour queue wait with a 15-minute wait lifted goodput from 81% to 92%.

The pieces of that loss stack:

  • Rollback. Anything computed since the last durable checkpoint is thrown away. Google's TPU fleet efficiency paper defines goodput to exclude exactly this: work done between the last checkpoint and a failure does not count as productive.
  • Checkpointing itself. Saving terabyte-scale state takes cluster time, which is why PyTorch's TorchTitan team reported cutting a checkpoint stall from 841 milliseconds to 34 milliseconds, a step catalogued in Cloud Girl's roundup of recent reliability releases.
  • Detection. The dangerous machines are the sick-but-not-dead ones. A straggler GPU that runs a little slow, or silently corrupts data instead of crashing, drags the whole synchronous job without triggering an alarm. The Llama 3 table logged six silent-data-corruption interruptions (1.4%), and Nvidia built a system called Minerva specifically to hunt faulty-but-alive GPUs, while SemiAnalysis notes the paper's tracked diurnal 1-2% throughput swings as a data center warms through the day.
  • Replacement logistics. Hot spares, burn-in and automated remediation are what compress queue time. vCluster's operations guide notes that NVLink and InfiniBand faults often appear only under sustained collective workloads, which is why 24- to 48-hour burn-in gates before a node enters production have become standard practice.
Illustration: a gear slipping out of a gear train, a robotic arm placing a replacement, a rewind arrow sweeping back over a row of snapshot tiles

The mitigation is the product

Read across the disclosures and the same answer appears: nobody is trying to stop failures. They are trying to make failures cheap. Meta's 419 interruptions cost only three human touches. Microsoft says the networking in its Fairwater supercomputer (multipath reliable transport, multi-plane topology, static segment routing) exists to route around faults and degrade gracefully rather than restart jobs at 100,000-GPU scale. Its MAI-Thinking-1 report went further than any lab before it, making goodput the primary production KPI for the cluster and decomposing the gap against ideal training time into restart loss, recomputation since the last checkpoint, startup delays and placement perturbations. Elastic schemes like nonuniform tensor parallelism let the job resize around dead GPUs instead of dying with them. Together AI shipped automatic node repair. The pattern is detect, drain, repair, resume, run without an operator.

The disclosure gap is the buying problem

What is missing is independent ground truth for the rental market. Operator data quoted via Glenn Lockwood's reliability notes shows what customer runs look like behind the curtain: in Nvidia's account of a 6,000-GPU, four-month DGX Cloud training campaign, frequent failure root causes included undefined "UB timeouts" and "NaN in gradients," a symptom that is usually silent data corruption in a lab coat. Crusoe's GTC 2025 talk on its H200 cluster attributed 41.5% of fault cases to full GPU host replacements. Meanwhile SemiAnalysis's ClusterMAX ratings found seven distinct interruptions in a two-month test rental of a supposedly serious GPU cloud, from firmware faults to a power outage, and vendors' own goodput marketing (CoreWeave claims up to 96%) is measured by the vendor.

The practical lesson for anyone renting large clusters: the warranty term is not the spec. Ask for interruption rates, checkpoint overhead, queue time on node replacement, and what the goodput number counts. Meta's experience is the only full-scale public benchmark, and it says this: 90% effective training time at 16,000 GPUs, with one hardware event every three hours, counts as a good run. Anything priced or planned around fewer interruptions than that is a marketing figure, not a measurement.

Sources

InfrastructureAnalysis