On-Device LLM Inference in Production: Architecture, Runtimes, and Hardware Constraints

Deploying generative language models directly onto edge devices such as smartphones, laptops, embedded systems, and browser sandboxes marks a fundamental shift in AI systems engineering. Moving inference from centralized GPU clusters to client silicon eliminates cloud API costs, cuts network latency to zero, guarantees data privacy by keeping user inputs local, and enables offline functionality.

However, executing modern autoregressive models on resource-constrained client hardware presents strict architectural bottlenecks. Edge inference is fundamentally bounded by memory bandwidth and system memory limits rather than raw compute FLOPS. Deploying Small Language Models (SLMs) such as Llama 3.2 1B and 3B, SmolLM2, and Qwen 2.5 requires specialized runtimes, aggressive low-bit quantization, optimized key-value (KV) cache buffers, and hybrid edge-cloud routing patterns.

The Memory Bandwidth Bottleneck and Hardware Limits

The primary physical constraint in on-device LLM serving is memory bandwidth. While the prefill phase (processing the input prompt) is compute-bound and saturates parallel matrix units, autoregressive token generation (the decode phase) is strictly memory-bound.

During decoding, the runtime must load the entire weight matrix from dynamic RAM (DRAM) into on-chip cache or registers to generate each individual token. For a model with $P$ billion parameters stored in $B$ bytes per parameter, generating one token requires reading at least $P \times B$ gigabytes over the memory bus.

The theoretical maximum generation throughput is governed by the relation:

$\text{Throughput (tokens/sec)} \le \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$

Consider standard mobile and edge hardware configurations:

Flagship Mobile SoCs (e.g., Apple A18 Pro, Qualcomm Snapdragon 8 Elite): Feature unified LPDDR5X memory architectures with bandwidth ranging between 60 GB/s and 85 GB/s. For a 3-billion parameter model quantized to 4 bits (occupying approximately 1.8 GB to 2.0 GB of memory), the theoretical peak decode speed is 30 to 42 tokens per second.
Mid-Tier and Embedded Devices: Often feature LPDDR4X or lower-spec LPDDR5 memory offering 15 GB/s to 35 GB/s of bandwidth. The same 4-bit 3B model is capped at 8 to 18 tokens per second.
System Resource Contention: Memory bus bandwidth is shared across the CPU, GPU, Neural Processing Unit (NPU), display engine, and background OS processes. Practical sustained throughput typically drops by 20% to 35% below theoretical ceilings due to bus arbitration and thermal throttling.

Beyond bandwidth, physical memory capacity is severely constrained. Mobile operating systems enforce aggressive process limits. On iOS, the Jetsam daemon terminates applications that exceed predefined memory allocation ceilings (often 2 GB to 4 GB depending on device generation), while Android's Low Memory Killer (LMK) kills high-footprint background processes. Consequently, the combined footprint of the model weights, KV cache, runtime binary, and application UI must remain within a compact 1.5 GB to 2.5 GB envelope.

Runtime Architectures: ExecuTorch, ONNX Runtime, LiteRT, and WebLLM

Running models efficiently across fragmented edge hardware requires specialized runtime engines that bypass traditional heavy Python dependencies.

ExecuTorch (PyTorch / Meta)

ExecuTorch is PyTorch's unified open-source runtime for on-device inference across mobile and embedded platforms. It separates the execution workflow into two phases: ahead-of-time (AOT) export and a lightweight C++ runtime.

Compilation Flow: Models defined in PyTorch are traced via torch.export, lowered to an intermediate representation, and serialized into .pte bytecode binaries.
Delegate Subsystem: ExecuTorch uses a pluggable delegate backend. Operators can be routed dynamically to:
XNNPACK and Arm KleidiAI: Highly optimized CPU micro-kernels leveraging Armv8/v9 extensions (such as i8mm and dot-product instructions), yielding over 20% throughput gains on Arm Cortex cores according to Arm engineering benchmarks.
Qualcomm QNN: Direct compilation to Qualcomm Hexagon NPUs and Adreno GPUs, achieving 25 to 35 tokens per second on Llama 3.2 1B models on modern Snapdragon platforms per ExecuTorch evaluation papers.
Apple MPS / Metal: GPU-accelerated execution on iOS and macOS hardware.
Zero-Allocation Execution: The C++ runtime allows pre-allocating memory buffers at initialization, preventing memory fragmentation and runtime allocation overhead during token generation.

ONNX Runtime GenAI (Microsoft)

ONNX Runtime GenAI provides a dedicated generative loop wrapper around the core ONNX Runtime engine.

Unified API: Abstracts token generation loops, search algorithms (greedy, top-k, top-p), and tokenizers in native C++, C#, Python, and Java.
Execution Providers (EPs): Supports DirectML on Windows client devices and NPUs, Qualcomm QNN on Snapdragon, CoreML on Apple hardware, and CPU backends via OpenVINO.
Shared KV Cache Buffers: Implements native past-present buffer sharing within Grouped-Query Attention (GQA) operators, preventing duplicate memory allocations across generation steps.

LiteRT (Google / formerly TensorFlow Lite)

Google's LiteRT (the rebranded evolution of TensorFlow Lite and MediaPipe GenAI) provides edge inference optimized for Android and Google hardware.

Hardware Targeting: Compiles to OpenCL/Vulkan GPU shaders and delegates to Google Tensor TPUs and MediaTek APUs.
FlatBuffer Model Format: Loads models via memory-mapped FlatBuffers (.tflite / .bin), minimizing cold-start loading times and virtual memory pressure.

WebLLM and MLC-LLM (Apache TVM & WebGPU)

For browser-based and cross-platform native execution without local binary installation, WebLLM and MLC-LLM leverage Apache TVM compilation to WebGPU and Vulkan.

Shader-Level Execution: Generates optimized WebGPU Shading Language (WGSL) compute shaders that execute directly on client GPUs within modern browsers (Chrome, Edge, Safari).
Buffer Pooling: As detailed in recent WebGPU inference research, WebLLM maintains dynamic buffer pools to bypass browser WebGPU memory allocation limits while preserving multi-precision quantization layouts.

Quantization Strategies for Heterogeneous Edge Silicon

Full 16-bit floating-point (FP16 or BF16) representations are unusable for edge deployment: a 3B parameter model in FP16 consumes over 6 GB of memory, exceeding the total RAM budget of standard mobile devices. Production edge pipelines deploy specific low-bit quantization schemes based on target processor architecture:

Weight-Only Quantization (W4A16 and W4A8)

For CPU and GPU execution, weight-only 4-bit quantization with 16-bit or 8-bit activations is standard:

Mechanism: Model weights are stored in 4-bit integers (INT4) grouped into blocks (typically group size 32, 64, or 128) with per-group FP16 scales and zero-points. Activations remain in FP16 or are dynamically quantized to INT8 during computation.
De-quantization on the Fly: The runtime streams 4-bit weights over the memory bus, dequantizes them to FP16 in on-chip registers, and executes standard floating-point fused multiply-accumulate operations. This cuts memory bandwidth consumption by 75% relative to FP16 with negligible degradation on downstream tasks.

Uniform Low-Bit Quantization for NPUs (W4A4 and W8A8)

Mobile NPUs (such as Qualcomm Hexagon and Apple Neural Engine) possess fixed-point systolic array hardware optimized for integer arithmetic:

Static Quantization Requirements: NPUs cannot tolerate dynamic floating-point dequantization overhead. Both weights and activations must be pre-quantized to fixed-point formats (e.g., INT8/INT8 or INT4/INT8).
Calibration: Requires post-training quantization (PTQ) or quantization-aware training (QAT) with representative calibration datasets to set static activation clamping ranges. Outliers in intermediate activations (activation kurtosis) require per-channel or group-wise scaling to avoid severe perplexity degradation.

KV Cache Management Under Client Constraints

While model weights represent a static memory allocation, the KV cache grows linearly with sequence length:

$\text{KV Cache Size (Bytes)} = 2 \times 2 \times L \times N_{\text{KV}} \times D_{\text{head}} \times S$

where $L$ is layer count, $N_{\text{KV}}$ is the number of key-value heads, $D_{\text{head}}$ is head dimension, and $S$ is total sequence length (context + generated tokens).

On edge devices, uncontrolled KV cache expansion risks sudden out-of-memory (OOM) termination. Systems apply three production techniques:

Grouped-Query Attention (GQA): Modern edge models (e.g., Llama 3.2 1B with 8 KV heads vs 32 query heads, or Qwen 2.5) reduce KV head count by 4x to 8x relative to standard Multi-Head Attention, reducing KV cache memory by up to 75%.
Static Buffer Pre-Allocation: Rather than dynamically reallocating memory as tokens are generated, edge runtimes allocate a fixed circular or pinned buffer sized for the maximum supported sequence length (e.g., 2,048 or 4,096 tokens).
KV Cache Quantization: Storing cached keys and values in INT8 or FP8 format reduces KV memory consumption by 50% compared to FP16, allowing longer conversation context without triggering OS memory termination.

Production Architectures: Hybrid Edge-Cloud Routing

In production systems, on-device models rarely operate in complete isolation. Instead, they serve as the first tier in a hybrid edge-cloud architecture:

Tier 1 (On-Device SLM): Handles latency-critical, privacy-sensitive, and lower-complexity tasks locally:
Text summarization, grammar correction, and local entity extraction.
Intent classification and semantic routing.
Offline interaction and draft generation for speculative editing.
Tier 2 (Cloud Frontier Model): Handles complex multi-step reasoning, large-scale codebase synthesis, and open-domain knowledge retrieval.
Dynamic Escalation Triggers:
Confidence and Entropy Scoring: If the on-device model's token output distribution displays high normalized entropy or low sequence confidence, the query automatically falls back to a cloud model.
Context Length Limits: Prompts exceeding the edge device's supported context window (e.g., >4,096 tokens) route directly to the cloud gateway.
Hardware State Governance: The application client monitors battery level, thermal throttling indicators, and system RAM pressure, dynamically switching between local execution and cloud endpoints.

Deploying on-device language models requires balancing memory bus bandwidth, hardware-specific compilation backends, and quantization schemes. By integrating lightweight runtimes like ExecuTorch and ONNX Runtime GenAI with structured hybrid routing, edge AI systems deliver high-throughput, private, and offline-capable inference within strict mobile resource envelopes.