Meta has published a detailed look at Prometheus, the AI training cluster it is building in New Albany, Ohio. In an Engineering at Meta post on October 19, the company calls Prometheus a "gigawatt-scale supercluster," says the first phase is complete, and says it is "working toward scaling it to comprise ~1 million GPUs." The first phase trained Meta's Muse foundation models, including Muse Wonder, which we covered in Meta Muse Wonder Explained.
The headline GPU count is not the most useful part of the post. The useful part is how plainly Meta describes what breaks at this scale, and what it costs to keep a training run alive.
What Prometheus is today
According to Meta's post, Prometheus "comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings." Those buildings include traditional data center halls, AI-optimized buildings and "rapid deployment structures," linked by fiber so they work "as a single machine." Meta says that two years ago New Albany "was a conventional data center site running web traffic." Meta also says it adds "leased capacity from strategic partners" on top of its own buildings to speed up deployment.
At full scale, Meta says, Prometheus "will comprise roughly 1 million distributed GPUs, all functioning in lockstep." The post does not give a date for full scale or an exact power figure beyond "gigawatt-scale."
Three Nvidia rack generations in one cluster
Prometheus runs two Nvidia platforms now, with a third "arriving soon." All three use Nvidia's 72-GPU NVLink rack design, which Meta calls its NVL72 rack architecture. The post lists these specs:
- Catalina (Nvidia GB200 NVL72). 72 Blackwell GPUs with 192 GB of HBM3e each, 1,440 PFLOPS at FP4, 130 TB/s peak NVLink bandwidth, and "~120 kW total system power across the two-rack pair." Meta used the two-rack "Ariel" variant so each GPU gets 800G of scale-out networking instead of 400G.
- Clemente (Nvidia GB300 NVL72). 72 Blackwell Ultra GPUs with 288 GB of HBM3e each, "over 1 exaFLOP of peak FP4 compute," 130 TB/s peak NVLink bandwidth, "~135 kW rack power," and a "5-10% performance improvement over the GB200 NVL72." The full 72-GPU domain fits in one rack.
- San Miguel (Nvidia Vera Rubin NVL72). 288 GB of HBM4 per GPU, 2.52 peak EFLOPS of FP4 training compute per rack, 216 TB/s peak NVLink bandwidth, "over 170 kW per rack," and a "10-20% improvement in performance per watt over the GB300 NVL72."
Meta says it will keep expanding Prometheus "with a mix of NVIDIA GB300 NVL72s and NVIDIA Vera Rubin NVL72s." Meta says Vera Rubin will "integrate directly into Prometheus's existing network fabric." For another Vera customer plan, on the CPU side, see Modal's plan to use Nvidia Vera CPUs.
The network: one cluster across 100 km
Each building has an Ethernet fabric that links "up to 20,736 GPUs." A second layer, which Meta calls the Backend Aggregation network, connects those building fabrics across distances "spanning 2 km to over 100 km." Meta says it is built on Arista 7800-series chassis with Broadcom Jericho3 silicon. Meta says running pretraining across that distance "required a dedicated optimization effort" with Meta Superintelligence Labs, "especially given speed-of-light constraints."
In total, Meta counts "millions of miles of fiber strand, tens of thousands of switches, and millions of optics."
The 72-GPU failure domain and the 13% spare tax
This is the part other cluster builders will read closely. In the older H100 generation, a tightly coupled NVLink group was eight GPUs. With NVL72 it is 72, which Meta says increases "the fault blast radius by 9X."
Meta's answer is to run each rack as a 64-GPU slice and keep the other eight GPUs as in-rack spares. When a GPU fails, a spare replaces it and the job restarts from its last checkpoint without waiting for a repair. For whole-rack faults, Meta holds "roughly 2% additional racks in reserve."
Meta gives the math for one example. A training run with 32,768 active GPUs needs 523 NVL72 racks, or 37,656 GPUs, "about 13% above the active footprint." Meta says "most of that margin (11%) falls out of the 64-of-72 slice geometry rather than being an absolute reliability tax."
Interruptions fell 99% in a year
Meta says that in the initial phase of Prometheus in mid-2025, "interruption rates reached a peak of nearly 1,000 events per 8K GPUs daily." It reports "a 99% reduction in interruptions" after a year of work. The failure categories it lists include:
- NVLink signal integrity. This was "initially the primary bottleneck for GB200 deployments," across "over 600,000 NVLink connections."
- Memory faults. HBM and LPDDR errors "became a major source of interruptions" as GB300 scaled. Meta says it worked with Nvidia on refined HBM firmware settings, memory scrubbing, better LPDDR error correction and predictive detection that retires degrading parts early, and that it is still deploying these changes across the fleet.
- NVLink switch faults. Meta says a single switch fault can compromise a whole 72-GPU domain.
For Muse Wonder training, Meta reports "more than 85% goodput for pre-training runs and more than 90% goodput" for reinforcement learning runs, with a mean time between interruptions of "~47k GPU hours." Later firmware fixes raised that to "more than 54K GPU hours." Meta's target is "two interruptions per 8K GPUs per day, which is ~100k GPU hours MTBI."
Meta also says its firmware tooling can push fixes "to more than 100k GPUs within a few hours," and that the P90 single-pod upgrade time fell from "more than seven hours" to two hours.
Power swings of tens of megawatts
Meta says synchronous distributed training "introduces a power delivery hurdle." When GPUs move between compute and communication phases in unison, they cause "sudden load fluctuations of 30-40% within the cluster," and that starting or pausing a large job "can shift loads of more than tens of megawatts in just seconds."
Meta handles this with software it calls Dynamo Dimmer, which lowers GPU power limits as the cluster nears its electrical ceiling instead of stopping jobs. Rack batteries, capacitor trays, site battery storage and coordination with utilities back it up. Meta says it pays "the full cost of energy" for the New Albany site.
Why it matters
Most public talk about AI data centers is about how many GPUs or gigawatts a company has. Meta's post shows the other side of the ledger. At this scale, a cluster's real output depends on goodput, spare capacity and failure rates as much as on chip count. The numbers here give buyers and rivals a concrete benchmark: 64 of every 72 GPUs doing work, about 13% overprovisioning, and a stated MTBI of more than 54K GPU hours against a target of ~100k.
It also shows how much of the work is shared with Nvidia. Meta describes "a co-engineering model with NVIDIA that goes beyond a traditional vendor dynamic," in which "teams are deeply embedded at our data centers" and run "joint triage sessions" on every system halt. In that split, Meta handles scheduling and infrastructure, and Nvidia handles firmware, signal diagnostics and memory handling. A rival chip vendor would need to match that kind of support, not only a spec sheet.
Next for Meta is Hyperion in Richland Parish, Louisiana, which the company says is being designed "from the ground up with insights from Prometheus in mind." For wider context on how fast these builds are moving, and where they stall, see SemiAnalysis Forecasts +50GW of Data Centers in 2027.
