NVIDIA has moved its Groq 3 LPX dedicated inference accelerator into full commercial production. Announced at Hot Chips 2026, the rack-scale accelerator system is designed as a purpose-built extension for NVIDIA's Vera Rubin NVL72 data center platform, targeting the compounding decode latency bottlenecks created by multi-step autonomous AI agents.
European neocloud provider Nebius Group N.V. has committed as the first cloud infrastructure customer to deploy the accelerators, integrating them into its Nebius Token Factory production inference platform.
Independent Benchmarks and Token Generation Speeds
In independent benchmark evaluations conducted by Artificial Analysis, the Groq 3 LPX rack system recorded an output speed of 3,400 tokens per second for a single stream running Google's open-weight Gemma 4 31B model with a 100,000-token prompt sequence.
This result represents a 4x throughput advantage over the closest alternative entry on the Artificial Analysis leaderboard, which recorded 882 tokens per second under comparable test conditions on Cerebras CS-3 wafer-scale systems.
The underlying silicon stems from technology licensed from Groq Inc. in late 2025 under a $20 billion transaction that also brought Groq founder Jonathan Ross and president Sunny Madra to NVIDIA. Groq's language processing units (LPUs) rely on a deterministic, dataflow architecture built around high-bandwidth on-die static random-access memory (SRAM) rather than conventional high-bandwidth memory (HBM) or GDDR DRAM stacks.
Heterogeneous Inference: Disaggregating Prefill and Decode
Production agentic workflows generate significant decode latency because multi-agent systems often require dozens of sequential reasoning steps, tool invocations, and verification calls where each generation step depends directly on the output of the previous step.
To optimize serving economics without compromising interactive responsiveness, NVIDIA is pairing its GPU compute with LPU dataflow silicon in a disaggregated architecture:
- Prefill Phase (Vera Rubin GPUs): Compute-heavy ingestion of long context prompts, system instructions, and codebase representations, as well as the creation of key-value (KV) caches, runs across Vera Rubin graphics processors equipped with high-capacity HBM4 memory.
- Decode Phase (Groq 3 LPUs): Memory-bandwidth-intensive token-by-token generation is offloaded to Groq 3 LP30 dataflow accelerators.
Each liquid-cooled NVIDIA MGX rack configuration integrates up to 256 LPUs, providing a pooled 128 GB of on-chip SRAM backed by 640 TB/s of scale-up interconnect bandwidth. Each individual LPU delivers roughly 500 MB of local SRAM, offering up to 150 TB/s of memory bandwidth per chip.
Architecture Trade-Offs and Model Capacity Limits
While the 3,400 tokens per second metric demonstrates SRAM bandwidth efficiency, the architecture introduces physical footprint and cluster scaling trade-offs.
Because on-die SRAM has substantially lower memory density than HBM or DRAM, storing model weights requires significant silicon area:
- Dense Models: Gemma 4 31B running at FP8 precision requires approximately 31 GB of memory, which fits across 64 LPUs within a single 256-LPU rack. This allows a single rack to host up to four parallel execution pipelines or replicate models for higher concurrency.
- Mixture of Experts (MoE): Scaling to large sparse architectures presents steeper hardware requirements. Serving a model like DeepSeek V3 (671 billion total parameters with 37 billion active parameters per token) requires housing the full parameter set across roughly 1,342 individual LPU chips, requiring more than five linked LPX rack systems.
Furthermore, hardware competition in ultra-fast inference continues to evolve. Cerebras recently unveiled its CS-4 system, which triples accelerator density per rack and doubles memory bandwidth, while partnering with AMD and AWS to develop competing GPU-plus-wafer-scale heterogeneous inference configurations.
Production Cloud Deployment
Nebius plans to offer Groq 3 LPX acceleration directly through its standard Token Factory API, maintaining compatibility with existing function calling, structured JSON output formats, and autoscaling tooling without requiring developers to adapt their software to proprietary hardware SDKs.
NVIDIA also highlighted at Hot Chips that SpaceX will deploy Vera CPUs alongside Rubin infrastructure to handle orchestration, simulation, and tool execution workloads across terrestrial data centers and orbital systems.
Sources
- SiliconANGLE: Nvidia’s Dedicated Inference Accelerator Groq 3 LPX Enters Full Production to Supercharge AI Agents
- The Register: What Nvidia's First Groq 3 LPU Benchmarks Do and Don't Tell Us About Its $20B Gamble
- Nebius: Nebius Token Factory Becomes First AI Cloud to Adopt NVIDIA Groq 3 LPX
- NVIDIA Newsroom: NVIDIA Groq 3 LPX Now in Full Production With World-Class Speed for Agentic AI
