Continuous LLM Performance Profiling in Production: Roofline Models, Model FLOPs Utilization, Model Bandwidth Utilization, and Hardware Bottleneck Diagnostics

Evaluating the runtime performance of large language model serving infrastructures requires looking beyond raw GPU metrics. Standard operating system utilities such as nvidia-smi report high GPU utilization percentages whenever compute cores or memory controllers are active, masking critical inefficiencies in memory access, communication, and kernel scheduling. A serving node running single-stream autoregressive decoding can report 100% GPU utilization while operating at less than 2% of the hardware's theoretical arithmetic throughput.

Modern LLM inference systems alternate between two fundamentally distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound decode phase. Diagnosing hardware bottlenecks and optimizing latency-throughput trade-offs requires formal performance models. By combining classical Roofline analysis with Model FLOPs Utilization (MFU) and Model Bandwidth Utilization (MBU), performance engineers can pinpoint exact resource constraints and implement targeted serving optimizations.

The LLM Roofline Model and Arithmetic Intensity

The Roofline model establishes an upper bound on attainable performance by modeling the interaction between compute capability and memory bandwidth. The model relies on operational or arithmetic intensity, defined as the ratio of floating-point operations executed to the number of bytes transferred between High Bandwidth Memory (HBM) and on-chip SRAM:

$$\text{Arithmetic Intensity } (I) = \frac{\text{FLOPs}}{\text{Bytes Transferred}}$$

Every hardware accelerator possesses a critical arithmetic intensity threshold, often called the hardware knee:

$$I_{\text{crit}} = \frac{\text{Peak Arithmetic Throughput (FLOPs/s)}}{\text{Peak Memory Bandwidth (Bytes/s)}}$$

On an NVIDIA H100 SXM5 accelerator with 989 TFLOPs of non-sparse BF16/FP16 Tensor Core throughput and 3.35 TB/s of HBM3 memory bandwidth, the critical arithmetic intensity is approximately 295 FLOPs/Byte. When executing in FP8 precision (1,979 TFLOPs), the critical threshold rises to roughly 590 FLOPs/Byte.

The attainable floating-point throughput ($P$) for any kernel is governed by the minimum of the hardware's compute ceiling and the memory-bandwidth limit:

$$P = \min\left(\text{Peak FLOPs/s}, I \times \text{Peak Memory Bandwidth}\right)$$

Workloads with an arithmetic intensity below $I_{\text{crit}}$ fall into the memory-bandwidth-bound regime: throughput scales linearly with memory bandwidth, leaving compute units idle. Workloads with an arithmetic intensity above $I_{\text{crit}}$ reside in the compute-bound regime: execution speed is capped by the floating-point units.

Prefill vs. Decode: Two Divergent Hardware Regimes

Serving LLMs in production involves two distinct phases that occupy opposite ends of the Roofline spectrum:

1. Prefill Phase (Compute-Bound)

During prompt processing, the model ingests all input tokens simultaneously. Matrix multiplications take the form of dense General Matrix-Matrix Multiplication (GEMM). For an input sequence length $S$, batch size $B$, and model parameter count $P$, the floating-point operations scale as $2 B S P$, while parameter memory loading remains constant at $2 P$ bytes in 16-bit precision.

As sequence length and batch size increase, arithmetic intensity grows proportionally:

$$I_{\text{prefill}} \approx \frac{2 B S P}{2 P + 2 B S H L} \approx B S$$

where $H$ is hidden dimension and $L$ is layer count. For prompts exceeding several hundred tokens, arithmetic intensity easily surpasses $I_{\text{crit}}$, driving the workload firmly into the compute-bound region.

2. Decode Phase (Memory-Bandwidth-Bound)

Autoregressive token generation proceeds sequentially, producing one token per sequence per forward step. For a batch size of $B = 1$, generating a single token requires streaming every model weight from HBM to on-chip SRAM to perform General Matrix-Vector (GEMV) operations.

For a 70-billion parameter model in FP16 precision (140 GB of weights), generating one token requires loading 140 GB of weights to execute 140 billion FLOPs. The resulting arithmetic intensity is:

$$I_{\text{decode}} = \frac{2 \times 70 \times 10^9 \text{ FLOPs}}{140 \times 10^9 \text{ Bytes}} = 1.0 \text{ FLOP/Byte}$$

Because $1.0 \text{ FLOP/Byte} \ll 295 \text{ FLOPs/Byte}$, single-stream decoding is constrained by memory bandwidth. On an H100 SXM5 GPU capable of 3.35 TB/s, reading 140 GB takes at minimum 41.8 milliseconds per token, yielding a theoretical ceiling of roughly 24 tokens per second regardless of Tensor Core compute capacity.

Core Efficiency Metrics: MFU, HFU, and MBU

Relying solely on hardware counters can introduce reporting errors due to unoptimized kernel overheads. Production systems rely on three standardized efficiency metrics:

Model FLOPs Utilization (MFU)

Originally formulated by Google in the PaLM architecture analysis, Model FLOPs Utilization measures the ratio of theoretical floating-point operations required by the pure transformer architecture to the hardware's theoretical peak capacity:

$$\text{MFU} = \frac{\text{Theoretical Model FLOPs per Token} \times \text{Tokens Processed per Second}}{\text{Theoretical Peak Hardware FLOPs/s}}$$

For standard decoder-only models, theoretical FLOPs per token during forward inference is approximated as $2 P$ (where $P$ is active parameter count). MFU isolates pure mathematical progress from framework-level inefficiencies. High MFU values (typically 40% to 55% in optimized production prefill) signify that the underlying compute hardware is effectively saturated.

Hardware FLOPs Utilization (HFU)

Hardware FLOPs Utilization measures the actual FLOPs executed by the physical hardware, including non-GEMM operations such as layer normalizations, softmax, activation functions, and attention recomputation:

$$\text{HFU} = \frac{\text{Actual Executed Hardware FLOPs/s}}{\text{Theoretical Peak Hardware FLOPs/s}}$$

The difference $\text{HFU} - \text{MFU}$ highlights operator overheads and architectural inefficiencies. In well-optimized serving engines, this gap remains minimal.

Model Bandwidth Utilization (MBU)

Introduced in LLM inference performance engineering literature, Model Bandwidth Utilization evaluates how close the system comes to saturating the physical memory bus during memory-bound decoding:

$$\text{MBU} = \frac{(\text{Active Model Weight Bytes} + \text{KV Cache Bytes Loaded per Step}) \times \text{Tokens/s}}{\text{Peak Hardware Memory Bandwidth (Bytes/s)}}$$

An MBU score approaching 75% to 85% indicates near-optimal kernel efficiency during autoregressive generation. At this threshold, further latency reductions cannot be achieved through kernel-level tuning; they require reducing total bytes transferred (such as weight quantization or KV cache pruning) or increasing batch sizes to raise arithmetic intensity.

The Production Profiling Toolchain

Accurately capturing MFU, MBU, and latency distributions in production serving clusters requires multi-level observability:

1. Kernel-Level Profiling (NVIDIA Nsight Compute / ncu)

Nsight Compute inspects individual CUDA kernels to identify hardware instruction stalls:

• Memory Subsystem Analysis: Measures HBM throughput, L2 cache hit rates, and shared memory bank conflicts.
• Warp State Diagnostics: Identifies whether warps are stalled on memory throttles (stall_long_scoreboard), execution dependencies, or math pipe availability.
• Roofline Visualizer: Automatically maps captured kernels against the accelerator's theoretical Roofline ceilings.

2. Timeline and Host Diagnostics (NVIDIA Nsight Systems / nsys)

Nsight Systems captures timeline traces spanning CPU orchestration and GPU execution:

• Host Launch Latency: Pinpoints Python runtime overhead and CPU thread contention that create gaps (bubbles) between sequential GPU kernel launches.
• Inter-GPU Communication: Profiles NCCL all-reduce and all-gather collectives to detect Tensor Parallelism synchronization delays across PCIe or NVLink fabrics.
• CUDA Graph Replay: Verifies that static execution graphs eliminate kernel launch latencies during fixed-batch decoding loops.

3. Serving-Engine Telemetry (vLLM and SGLang)

Production serving runtimes expose structured operational metrics via Prometheus endpoints:

• vllm:time_to_first_token_seconds: Tracks prefill latency distribution.
• vllm:time_per_output_token_seconds: Measures decode iteration latency.
• vllm:gpu_cache_usage_factor: Monitors physical PagedAttention block utilization.
• vllm:num_requests_waiting: Detects admission queue saturation and prefill preemption.

Diagnostic Decision Framework and Remediation

Performance bottlenecks can be diagnosed systematically by evaluating the relationship between MFU, MBU, and concurrency metrics:

Scenario A: Memory-Bandwidth-Bound Decode (High MBU > 75%, Low MFU < 5%)

• Symptoms: Time per Output Token (ITL) increases linearly with model parameter size; GPU Tensor Cores sit largely idle while HBM memory controllers operate at near capacity.
• Root Cause: Low concurrency or single-stream decoding where every parameter must be fetched for every generated token.
• Remediation:

1. Continuous Batching: Increase the operational batch size to amortize weight reading across multiple concurrent requests.
2. Weight and KV Quantization: Compress weights and KV cache to FP8, INT8, or INT4 (via AWQ, GPTQ, or FP8 E4M3 formats), cutting the byte payload transferred over the bus per token.
3. Attention Architecture Optimization: Leverage Grouped-Query Attention (GQA) or Multi-Head Latent Attention (MLA) to reduce KV cache memory footprints.
4. Speculative Decoding: Deploy small draft models (or parallel verification schemes like EAGLE-2) to verify multiple candidate tokens per memory-read cycle, converting memory-bound GEMVs into compute-bound small GEMMs.

Scenario B: Compute-Bound Prefill (High MFU > 45%, Low MBU)

• Symptoms: Time to First Token (TTFT) scales quadratically or cubically with input context length; Tensor Core utilization approaches maximum rated throughput.
• Root Cause: High-density matrix multiplications processing long input prompts.
• Remediation:

1. Chunked Prefill: Break large prompts into discrete chunks (e.g., 512 or 1024 tokens) and interleave them with decode iterations to prevent decode request starvation.
2. IO-Aware Attention: Ensure FlashAttention-3 or FlashInfer kernels are active to maintain online softmax calculation and minimize round trips between SRAM and HBM.
3. Precision Scaling: Transition prefill computations from BF16 to FP8 GEMMs, doubling peak theoretical FLOP/s capacity.

Scenario C: Communication and Interconnect Stalls (Low MFU, Low MBU, High Inter-GPU Latency)

• Symptoms: GPU compute and memory controllers both show low utilization; Nsight Systems timelines reveal large NCCL all_reduce execution blocks.
• Root Cause: Over-partitioning via Tensor Parallelism across slow interconnects (such as standard PCIe links rather than high-bandwidth NVLink meshes) or configuring high Tensor Parallel degrees ($TP > 4$) across dual-socket boundaries.
• Remediation:

1. Re-evaluate Parallelism Topology: Restrict Tensor Parallelism to GPUs connected via full-bandwidth NVLink intra-node fabrics; use Pipeline Parallelism or Expert Parallelism across node boundaries.
2. Disaggregated Prefill and Decode: Decouple prefill nodes (compute-heavy, high TP) from decode nodes (bandwidth-heavy, memory-capacity-optimized) to isolate interconnect overheads.

Scenario D: Host Launch Latency and Scheduling Bubbles (Frequent GPU Idle Gaps)

• Symptoms: Individual kernels execute rapidly, but timeline traces show millisecond-scale pauses between consecutive kernel invocations.
• Root Cause: Python interpreter overhead, dynamic batch scheduling computation on the CPU host, or eager-mode PyTorch launch latency.
• Remediation:

1. CUDA Graph Capture: Capture static decode execution paths into CUDA Graphs, allowing the host CPU to launch the entire multi-layer forward pass with a single API call.
2. C++ Runtime Offloading: Migrate scheduler loops and token sampling to native C++/Rust runtimes to eliminate GIL contention.

Continuous performance profiling transforms LLM infrastructure management from reactive guessing into systematic engineering. By measuring arithmetic intensity against hardware Rooflines and tracking MFU alongside MBU, engineering teams can maximize inference throughput, cut hardware costs, and enforce predictable serving SLAs.

Sources

• Roofline: An Insightful Visual Performance Model for Multicore Architectures
• PaLM: Scaling Language Modeling with Pathways (Model FLOPs Utilization Formulation)
• Efficient and Economic Large Language Model Inference with Attention Offloading (Model Bandwidth Utilization)
• RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis
• vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention