Subscribe
Networking
6 min read

The Quiet Interconnect War: How Frontier AI Clusters Actually Talk

Network topology decides whether the run finishes. Late 2026 marks a shift where Ethernet overtook InfiniBand at major hyperscalers, while new standards like UALink challenge Nvidia's dominance in rack-scale connectivity.

Diagram illustrating network topology comparison between NVIDIA NVLink, Ethernet, and Google TPU interconnects for AI clusters
Diagram illustrating network topology comparison between NVIDIA NVLink, Ethernet, and Google TPU interconnects for AI clusters

By late 2026, Ethernet has surpassed InfiniBand as the dominant fabric for connecting AI racks, validated by large-scale deployments at Meta and xAI that reported no network bottlenecks. While Ethernet handles scale-out traffic, scale-up connectivity within racks remains contested by Nvidia’s NVLink and emerging open standards like UALink.

GPU counts get the headlines. Network topology decides whether the run finishes. Every frontier training cluster is two machines at once: a computer made of thousands of accelerators, and the cabling that makes them behave like one. That cabling is now contested ground, and late 2026 is the clearest moment yet to map it.

Left: one server rack with dense thick scale-up links between eight chip panels. Right: four separate racks joined by sparse long dashed scale-out links.

Two problems, not one fabric

Cluster networking splits into two jobs with different physics. Scale-up connects accelerators inside one rack so tightly they act as a single chip. Scale-out connects racks into a cluster using conventional datacenter switching.

On Nvidia's Blackwell platform, fifth-generation NVLink gives each GPU 1.8 TB/s of bidirectional interconnect, and an NVL72 rack binds 72 GPUs into one NVLink domain with 130 TB/s of aggregate bandwidth; Nvidia says a single domain can stretch to 576 GPUs (Nvidia Blackwell architecture, GB200 NVL72 page). The scale-out NICs running between those racks move 400 to 800 gigabits per second, so roughly 50 to 100 GB/s. A GPU talks to its rack neighbors about twenty to forty times faster than to any machine outside the rack. That gap is the reason both markets exist.

The scale-out contest: InfiniBand's long slide

Until 2024, the default answer for connecting racks was Nvidia's InfiniBand. When Dell'Oro Group began tracking AI back-end networks in late 2023, InfiniBand held more than 80 percent of back-end switch sales (Dell'Oro, July 2025).

The reversal was empirical, not theoretical. In late 2023 Meta built two identical 24,576-GPU H100 clusters side by side, one on Quantum-2 InfiniBand, the other on RoCEv2 Ethernet over Arista and OCP switches (DCD; Meta engineering blog, September 2025). Meta trained Llama 3 on the Ethernet cluster and reported no network bottlenecks (Meta, Building GenAI Infrastructure, March 2024). In 2024 xAI then wired its 100,000-H100 Colossus cluster with Nvidia's own Spectrum-X Ethernet instead of InfiniBand, a choice Nvidia framed as Ethernet catching up at scale, claiming 95 percent data throughput and no flow-collision latency degradation (Nvidia newsroom; The Register).

The market followed the deployments. Dell'Oro reported that Ethernet surpassed InfiniBand in AI back-end networking in 2025, and that over the full year Ethernet was already more than twice the size of InfiniBand, with most ports shipped at 800 Gbps (Dell'Oro, 4Q2025). Its projected port cadence: 1600 Gbps shipping in the second half of 2026, 3200 Gbps by 2030. Ethernet's advantage was never raw speed. It is vendor diversity and supply, the reason Amazon, Microsoft, Meta, Oracle and xAI standardized on it, as Dell'Oro's Sameh Boujelbene put it.

Ethernet did not stay vanilla. On June 11, 2025 the Ultra Ethernet Consortium released Specification 1.0, an AI-tuned stack built on standards-based Ethernet: the Ultra Ethernet Transport sprays one RDMA message's packets across every available path, tolerates out-of-order delivery, retransmits selectively, and defines dedicated AI profiles so fabrics no longer depend on lossless PFC tuning (UEC press release; UE Spec 1.0 PDF).

A geometric timeline band with four milestone motifs: a switch stack, an interlaced cable knot, a light-path diamond, and a rack ring.

The scale-up contest opens

Inside the rack, NVLink has had almost no competition, and that is where the next fight is. Three challengers arrived in eighteen months.

The UALink Consortium ratified UALink 200G 1.0 on April 8, 2025: an open, load/store memory-semantics interconnect for up to 1,024 accelerators in one pod, 200G per lane, up to 800 Gb/s per four-lane station (UALink 1.0 overview PDF; press release). On April 7, 2026 it published four further specifications adding in-network compute, a chiplet definition, manageability, and a split 200G data-link/physical-layer spec (UALink 2.0 press release PDF). The honest caveat: UALink silicon was still scarce in mid-2026, which is why independent performance comparisons against NVLink do not yet exist.

Nvidia's counter-move was to open the gate partly. NVLink Fusion, announced at Computex in May 2025, lets third-party CPUs and custom accelerators join an NVLink fabric instead of being stranded behind PCIe (Nvidia press release, May 18, 2025). Scale-up is now itself a standards war, and Dell'Oro expects 2026 to be defined by vendor announcements chasing both Ethernet and UALink in the scale-up segment (Dell'Oro 2026 predictions).

AMD is fighting on both layers at once. At Hot Chips 2026 it detailed the Helios rack: 72 MI455X GPUs whose scale-up fabric runs 1.8 TB/s per direction per GPU over 72 Ethernet-based Infinity Fabric links, an approach AMD calls UALoE, built on open Ethernet and ESUN standards and merchant switch silicon rather than a closed proprietary fabric, with up to three 800G Vulcano NICs per GPU for Ultra Ethernet scale-out (ServeTheHome). Production ramp is targeted for the second half of 2026 (TechPowerUp preview).

The heretic: one fabric, optical switches

Google never played either market. Its TPUs connect through ICI, a dedicated inter-chip interconnect wired into a 3D torus, and reconfigure that torus with optical circuit switches that physically reroute light paths to wrap rings and bypass failures (TPU deep dive; SemiAnalysis on TPUv7). The Ironwood generation carries 1.2 TB/s of bidirectional ICI bandwidth per chip across pods of 9,216 liquid-cooled chips (Google blog). It collapses the scale-up/scale-out distinction into one vendor-controlled stack. No one else can buy the recipe piecemeal.

What it means if you are buying

Three takeaways, none of them subtle. Ethernet parity at frontier scale is proven twice over, at Meta and at xAI, but Meta's own record says it comes from years of co-design and tuning, not from the spec sheet. The per-GPU bandwidth gap between racks and within racks is still one to two orders of magnitude, so the rack-scale domain, NVL72 today, Helios and UALink pods tomorrow, is the unit of compute that matters. And the standards bodies (UALink, UEC, ESUN) have spent 2025 and 2026 writing the paperwork for a post-monopoly scale-up market; whether shipping silicon appears before the next Blackwell generation doubles the lead again is the question to watch.

Sources

  • Nvidia: Blackwell architecture page (NVLink 5, 1.8 TB/s, 130 TB/s NVL72, 576-GPU domain)
  • Nvidia: GB200 NVL72 product page
  • Dell'Oro Group: "Ethernet is Winning the War Against InfiniBand" (July 15, 2025)
  • Dell'Oro Group: "Ethernet More Than Doubles Size of InfiniBand" (4Q2025 report)
  • Dell'Oro Group: Data Center Networking 2025-2026 predictions
  • Ultra Ethernet Consortium: Specification 1.0 press release and spec PDF (June 11, 2025)
  • UALink Consortium: 200G 1.0 Specification overview and press release (April 8, 2025); 2.0 press release (April 7, 2026)
  • Meta Engineering: "Building Meta's GenAI Infrastructure" (March 2024); "Meta's Infrastructure Evolution and the Advent of AI" (September 2025)
  • Datacenterdynamics: Meta's two 24k GPU clusters
  • Nvidia newsroom: Spectrum-X powers xAI Colossus; The Register: xAI picks Ethernet
  • Nvidia investor relations: NVLink Fusion announcement (Computex, May 18, 2025)
  • ServeTheHome: AMD Helios MI400 architecture at Hot Chips 2026; TechPowerUp: MI400/Helios preview
  • Google blog: Ironwood TPU; SemiAnalysis: TPUv7; henryhmko: TPU Deep Dive

Frequently asked questions

When did Ethernet surpass InfiniBand in AI back-end networking?

Dell'Oro Group reported that Ethernet surpassed InfiniBand in AI back-end networking in 2025, with Ethernet becoming more than twice the size of InfiniBand by the end of the year.

Did Meta or xAI use InfiniBand for their large clusters?

No; Meta built an identical H100 cluster on RoCEv2 Ethernet with no reported bottlenecks, and xAI wired its 100,000-H100 Colossus cluster with Nvidia's Spectrum-X Ethernet instead of InfiniBand.

NVLink is Nvidia’s proprietary high-bandwidth scale-up interconnect, whereas UALink is an open standard ratified by the UALink Consortium designed to provide load/store memory-semantics interconnect for up to 1,024 accelerators.

How does Google connect its TPUs compared to other providers?

Google never joined the InfiniBand or Ethernet markets for scale-out; instead, it uses ICI wired into a 3D torus with optical circuit switches that physically reroute light paths to bypass failures.

NetworkingInfrastructureAnalysis