The speed of local inference is determined by memory bandwidth—the number of bytes per second the machine can stream from memory to compute units—rather than just total memory capacity. Because generation (decoding) is sequential and requires reading weights layer-by-layer, higher bandwidth directly increases the theoretical maximum tokens per second. Factors like quantization precision and Mixture-of-Experts (MoE) architectures can help mitigate these bandwidth limits.
The standard advice for running open-weights models at home is to buy memory. Buy the card with the most gigabytes, the Mac with the biggest unified pool, the mini PC with 128 GB. That advice gets you a machine where a model fits. It does not get you a machine where a model runs fast, because the speed of local inference is set by a different number: how many bytes per second the machine can stream from memory to its compute units.
Between those two constraints sits every purchasing decision in consumer AI hardware in late 2026. This is the ladder, with the measured results that show how far the spec sheets get you.
Why token speed is a bandwidth problem
Generation is the slow half of inference. Prefilling a prompt is parallel and compute-heavy, but decoding is sequential: every output token must pass through every layer of the model, and at each layer the GPU has to read that layer's active weights from memory before it can do anything. The arithmetic per token is tiny compared to the data movement. As Tom's Hardware put it in its comparison of the Mac Studio, the NVIDIA GB10 and AMD's Strix Halo, the speed of the decode process "basically becomes dependent on how fast those model weights can be streamed in from GPU memory."
That gives you a usable ceiling. A dense model at 4-bit quantization moves roughly half a byte per parameter for every token. Divide the machine's memory bandwidth by the bytes per token and you have the theoretical maximum tokens per second. The RTX 5090's 1,792 GB/s against a 4-bit 8B model (about 4.5 GB of weights) yields a ceiling near 400 t/s. Nobody observes that. Real hardware lands at 40 to 60 percent of the ceiling because the KV cache, the attention reads and imperfect memory-controller efficiency eat the difference. But the formula predicts the ranking correctly, and that is what makes bandwidth the number to shop on.
The same logic is why datacenter GPUs are built the way they are. An H100 SXM pairs its enormous FLOPS with 3.35 TB/s of HBM3; its roofline ridge point, the ratio of compute to bandwidth, works out to about 591 FLOPs per byte. Consumer hardware now moves within striking distance of old datacenter bandwidth, the 5090's 1.79 TB/s is close to an A100's 1.55 TB/s, but the top HBM parts have run ahead: 4.8 TB/s on H200, 8 TB/s on B200, around 20 TB/s on the HBM4 generation. A research note on large-batch inference found that even at big batch sizes decoding stays memory-bound, with over half of attention kernel cycles stalled on data access. Bandwidth is the wall everywhere.
The ladder, late 2026
For hardware you can put on a desk, the bandwidth ranking is roughly:
- NVIDIA GeForce RTX 5090: 32 GB GDDR7 on a 512-bit bus, 1,792 GB/s. List price $1,999, street prices well above it since the memory shortage.
- Apple M3 Ultra (Mac Studio): up to 512 GB unified, 819 GB/s.
- AMD EPYC server workstations (12-channel DDR5): up to about 576 GB/s, with capacity in the hundreds of gigabytes at commodity RAM prices.
- Apple M4 Max (Mac Studio): up to 128 GB unified, 546 GB/s on the 16-core variant.
- NVIDIA DGX Spark (GB10): 128 GB unified LPDDR5x on a 256-bit bus, 273 GB/s.
- AMD Ryzen AI Max+ 395 (Strix Halo): up to 128 GB unified LPDDR5x-8000 on a 256-bit bus, 256 GB/s.

Notice the structure. The fast machines hold little; the capacious machines are slow. The 5090 streams weights three times faster than any unified-memory box, but its 32 GB cannot even load a dense 70B model at 4-bit, which needs on the order of 40 GB of weights plus room for context. A 128 GB Strix Halo will hold that 70B and stream it at something like 5 tokens per second, per one hardware-comparison site's estimate; a 64 GB Mac Studio in the same class lands in a similar range, and an M4 Max pushes about 11 t/s on the same model. The fastest way to run a big model locally is not to buy bandwidth at all, it is to make the model's per-token byte count small.
Where the spec sheet lies: MoE, precision, and batching
Three escapes from the bandwidth ladder are doing most of the work in current local-AI buying decisions.
Mixture-of-experts models shrink the numerator. Bytes per token track the active parameters, not total parameters. A community benchmark run on a 128 GB Strix Halo mini PC logged 72 tokens per second generating from Qwen3 30B-A3B at 4-bit, a model with a far smaller active-to-total parameter ratio than a dense model has. Tom's Hardware found the pattern across platforms: on the dense Gemma 12B, the Mac Studio's throughput scaled close to its bandwidth advantage over GB10 and Strix Halo, around 1.8x and 2.3x. On a sparse 35B-A3B MoE, the machine with double the bandwidth was only about 25 percent faster. MoE models make slow hardware usable, and they are why a $2,000 to $4,000 box can feel responsive on a 120B-class model it "should not" run.
Low precision shrinks it further. The DGX Spark's NVFP4 support halves the bytes per parameter versus 8-bit. That is why LMSYS's review of the machine called bandwidth "the key bottleneck in AI inference performance" while still demonstrating it serving GPT-OSS 120B: a later review measurement put the Spark at 38.6 t/s on that model, against 124 t/s on a three-RTX-3090 rig that can barely fit it. Capacity and bandwidth trade against each other in both directions.
Batching is the escape hatch you do not have. A server that answers fifty requests at once loads each weight matrix from memory once and multiplies it against fifty activation vectors, amortizing the stream. That is how datacenter GPUs cash their bandwidth checks, per the roofline analysis above. A single developer prompting a local model has a batch size of one. The bandwidth wall is exactly as high for you as the formula says, which is why consumer "AI workstations" benchmarked one user at a time land where they land.
What this means for the machine in your cart
The decision is still capacity first, then speed, but the honest version of it:
- You run sub-10B dense models or 30B-class MoE at Q4 and want maximum responsiveness. A 5090-class discrete GPU is the right buy. It is the only consumer option where 8B models stream at triple-digit token rates, and its problem is capacity, which smaller models do not have.
- You want a 70B dense or 100 to 235B MoE model to fit at all. You want 128 GB of unified memory, from whichever vendor's bandwidth and ecosystem you can get. In late 2026 that is the uncomfortable part of the market: the DGX Spark went from $3,999 to $4,699 after NVIDIA raised the Founders Edition price on memory supply constraints, AMD's own 128 GB Ryzen AI Halo lands at $3,999, Framework and other OEM Strix Halo desktops undercut both near $2,000 to $3,400, and Apple cut Mac Studio RAM options and lengthened lead times as the DRAM shortage bit. The 128 GB boxes differ in bandwidth by only 273 versus 256 versus 546 GB/s, so software stack, precision support and resale matter more than the spec sheet at this tier.
- You want the frontier model itself, locally. Short of a 512 GB M3 Ultra, nothing on a desk holds a trillion-parameter-class open model at usable speeds. One comparison of current hardware put it plainly: above 128 GB the field empties out. If the model quality requires the size, the bandwidth ladder ends and the rental market begins; our provider-by-provider H100 pricing walkthrough covers what that costs per hour.
- You care about first-token latency for interactive use. Local unified-memory boxes beat any API on distance: a Mac Studio in the same room returns first tokens in tens of milliseconds against a few hundred milliseconds of network before the model even starts. Prompt processing (prefill) on these boxes is the weak spot, not decode, so long-context prompt-heavy workloads feel worse than tokens-per-second numbers suggest.
The uncomfortable summary of late-2026 local hardware: the bandwidth ceiling for a single user is a physics problem the industry is trying to route around with sparser models and lower precision rather than solve with faster desktop memory. Buy for the model you intend to run, verify the bytes per token, divide, and treat anything above that number in a marketing slide as fiction.
Sources
- Tom's Hardware, Exploring Apple Silicon's local AI performance with the Mac Studio and M4 Max
- NVIDIA, DGX Spark User Guide: Hardware Overview
- NVIDIA, DGX Spark product page
- AMD, Ryzen AI Max+ 395 and Ryzen AI Halo Developer Platform
- TechPowerUp, Ryzen AI Max+ 395 specs
- LMSYS Org, NVIDIA DGX Spark In-Depth Review
- IntuitionLabs, DGX Spark Review: 2026 Price & Benchmarks
- Level1Techs, Strix Halo LLM Benchmark Results
- vdf.ai, Best GPU for Local LLM (2026)
- Ocolo, GPU VRAM vs Memory Bandwidth for AI & LLMs
- Spheron, AI's Memory Wall Problem
- arXiv, Mind the Memory Gap: GPU Bottlenecks in Large-Batch LLM Inference
- Macyou, Best Hardware for Local LLMs in 2026
- BirJob, Local LLM Inference on Consumer Hardware in 2026
- Presenc AI, Local AI & Hardware benchmarks
Frequently asked questions
Why does my local LLM run slowly despite having large VRAM?
Large VRAM only ensures the model fits; it does not ensure speed. Inference speed is set by memory bandwidth (bytes per second), as the GPU must sequentially read weights from memory during the decoding phase.
How do I calculate the theoretical max token speed?
Divide the machine's memory bandwidth (GB/s) by the bytes moved per token. For a dense model at 4-bit quantization, this is approximately half a byte per parameter per token.
Which hardware has the highest memory bandwidth for local LLMs?
Among consumer hardware listed, the NVIDIA GeForce RTX 5090 leads with 1,792 GB/s using GDDR7, followed by the Apple M3 Ultra at 819 GB/s and AMD EPYC workstations at roughly 576 GB/s.
How do MoE models affect local inference speed?
Mixture-of-Experts (MoE) models shrink the numerator of bytes per token by using fewer active parameters than total parameters. This allows slower bandwidth hardware to achieve higher token speeds on sparse models compared to dense ones.
