Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters.

Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Engine (ANE) share a single pool of high-speed LPDDR5/LPDDR5X memory. With unified memory capacities reaching 128 GB on M4 Max and 192 GB to 512 GB on Ultra-series chips, a single desktop machine can host 70B parameter models in 8-bit precision or 671B Mixture-of-Experts (MoE) architectures in 4-bit quantization without discrete multi-GPU interconnects.

However, token generation speeds on unified memory systems vary substantially across execution runtimes. Benchmarks reveal distinct performance characteristics across Apple MLX, llama.cpp, and Ollama, driven by memory management implementations, kernel compilation strategies, and runtime abstraction layers.

The Memory Bandwidth Ceiling in Autoregressive Serving

During autoregressive decoding, Large Language Models generate text sequentially, producing one token per forward pass. Because every layer's weight matrices must be read from memory for each generated token, the decoding stage is strictly memory-bandwidth bound.

The theoretical upper bound for single-batch token throughput is governed by the ratio of system memory bandwidth to active model footprint:

Theoretical Max Tokens/Sec = Memory Bandwidth (GB/s) / Active Model Size (GB)

On an Apple M4 Max with 546 GB/s of unified memory bandwidth:

• A 4-bit quantized 70B model occupying approximately 40 GB of memory has a theoretical ceiling of 13.65 tokens per second per stream.
• A 4-bit quantized 8B model occupying approximately 5 GB has a theoretical ceiling of 109.2 tokens per second.

On an M2 Ultra or M3 Ultra platform with 800 GB/s of unified memory bandwidth, those upper bounds increase proportionally to 20 tokens per second for 70B models and 160 tokens per second for 8B models.

By default, macOS allocates up to 75 percent of physical RAM to the GPU. For large model execution, engineers frequently adjust the wired memory limit via kernel sysctl parameters (sysctl iogpu.wired_mem_limit) to allow up to 95 percent of physical RAM to be locked for Metal buffer allocation.

Architectural Execution Profiles: MLX vs. llama.cpp vs. Ollama

The three primary runtime engines on macOS execute LLM workloads through fundamentally different software architectures:

1. Apple MLX and mlx-lm

Apple MLX is an array framework designed natively for Apple Silicon by Apple Machine Learning Research. Built specifically for Metal, MLX incorporates:

• Zero-Copy Tensor Operations: Tensors live in unified memory without intermediate CPU-to-GPU staging buffers.
• Lazy Evaluation and Kernel Fusion: Computations are represented as dynamic directed acyclic graphs (DAGs) and evaluated only when materialized, allowing the runtime to fuse consecutive elementwise operations, RoPE (Rotary Position Embedding) rotations, and attention kernels into single Metal shader dispatches.
• Native Safetensors and Affine Quantization: MLX natively reads standard Safetensors format and applies hardware-aligned 4-bit and 8-bit affine quantization directly in Metal Shading Language (MSL).

2. llama.cpp

The open-source llama.cpp runtime provides a modular C/C++ engine built around the GGML tensor library and GGUF file format:

• GGML Metal Backend: Offloads matrix multiplications and attention operations to Metal compute shaders (ggml-metal.metal) while retaining control flow and orchestration in C++.
• Advanced Quantization Schemes: Supports k-quantization algorithms (Q4_K_M, Q5_K_M, IQ4_XS) that apply non-uniform bit allocations across attention heads and feed-forward layers to preserve perplexity.
• Hybrid Offloading: Allows partial offload across CPU threads and Metal GPU command queues when model footprints exceed available GPU address space.

3. Ollama

Ollama is a local deployment service that wraps the underlying llama.cpp inference engine in a Go application daemon, providing a REST API compatible with OpenAI endpoints, model management tools, and automated system resource discovery.

Empirical Benchmark Findings and Execution Bottlenecks

Systematic comparative studies, including empirical evaluations published in arXiv:2511.05502 and arXiv:2601.19139, demonstrate clear performance divergences across these frameworks:

The Dense Model Gap

On dense architectures such as Llama 3 8B, Qwen 2.5 7B, and Mistral 7B at 4-bit quantization:

• MLX-LM consistently achieves the highest single-stream decoding throughput on Apple Silicon, outperforming llama.cpp by 20 to 45 percent.
• On an Apple M4 Max system, MLX achieves 68 to 88 tokens per second on 8B models, compared to 52 to 68 tokens per second on raw llama.cpp.
• The performance advantage stems from MLX's lazy graph compilation, which eliminates memory copy overhead and minimizes kernel launch latency during iterative token generation.

The Mixture-of-Experts (MoE) Divergence

The throughput gap between MLX and llama.cpp widens significantly on Mixture-of-Experts (MoE) models such as Mixtral 8x7B, Mixtral 8x22B, and Qwen 3.5 MoE:

• On an Apple M4 Pro system running MoE models, MLX achieves up to 130 tokens per second, compared to 43 tokens per second on llama.cpp, representing a 3x throughput differential.
• Because MoE architectures dynamically route tokens to a subset of expert weights on every forward pass, MLX's zero-copy unified memory indexing accesses non-contiguous parameter blocks with zero CPU-GPU serialization penalty. llama.cpp incurs additional tensor slicing overhead when dispatching sparse expert computations across GGML buffers.

The Ollama Wrapper Overhead

A notable finding in local inference profiling is the throughput degradation observed in Ollama compared to raw llama.cpp:

• While raw llama.cpp on an M4 Max reaches nearly 90 tokens per second on standard benchmarks, Ollama running the same GGUF checkpoint on the same machine often drops to 43.5 tokens per second.
• Profiling indicates that Ollama's Go runtime layer, IPC serialization via Cgo bindings, and memory polling overhead consume up to 50 percent of available throughput in sustained generation loops.

Continuous Batching and Multi-Tenant Serving: vLLM-MLX

Traditional local runners process single requests serially. The open-source vllm-mlx framework introduces continuous iteration-level batching and content-based prefix caching natively on top of Apple MLX:

• Research published in arXiv:2601.19139 demonstrates that vllm-mlx delivers 21 percent to 87 percent higher throughput than llama.cpp across models from 0.6B to 30B parameters.
• At 16 concurrent client requests, continuous batching scales aggregate throughput by 4.3x compared to serial execution.
• For multimodal inputs, content-based prefix caching reduces time-to-first-token (TTFT) from 21.7 seconds to 0.78 seconds by hashing vision embeddings in unified memory to prevent redundant image encoder passes.

Scaling Beyond Single Nodes: Clustered Unified Memory

For frontier open-weight models exceeding 128 GB of parameters (such as DeepSeek-V3 671B or Llama 3.1 405B), consumer developers increasingly deploy distributed topologies over high-speed interconnects.

Frameworks like EXO enable peer-to-peer model sharding across multiple Mac Mini or Mac Studio units connected via Thunderbolt 5 RDMA (Remote Direct Memory Access), delivering 80 Gbps bidirectional bandwidth per link:

• A cluster of eight Mac Mini M4 Pro units (64 GB RAM each) pools 512 GB of unified memory to serve DeepSeek-V3 at 5.37 tokens per second.
• Ring-topology tensor parallelism partitions weight matrices across nodes, bypassing traditional cloud GPU rental costs for private on-premise execution.

Framework Selection Matrix

When selecting an on-device LLM serving runtime for Apple Silicon, engineering teams should evaluate operational trade-offs:

• Apple MLX / mlx-lm: Recommended for maximum token throughput on Apple Silicon, fast local fine-tuning (LoRA/QLoRA), and optimal MoE execution.
• llama.cpp: Recommended for heterogeneous hardware deployments requiring identical GGUF artifacts across Linux, Windows, and macOS, or when fine-grained k-quantization formats are mandatory.
• vllm-mlx: Recommended for multi-tenant agent harnesses, continuous batching workloads, and multimodal applications requiring prefix caching.
• Ollama: Recommended for developer desktop tooling and local testing where one-command installation and OpenAI API compatibility outweigh raw generation throughput.

Sources

• Apple MLX Framework: GitHub Repository and Documentation
• Native LLM and MLLM Inference at Scale on Apple Silicon (vllm-mlx): arXiv:2601.19139
• Production-Grade Local LLM Inference on Apple Silicon: arXiv:2511.05502
• llama.cpp Engine: GitHub Repository
• Ollama Engine: GitHub Repository
• EXO Distributed Inference: GitHub Repository