Subscribe
Topic / page 3

Inference

Articles, page 3

41-60
  1. Hierarchical KV Cache Offloading in Production LLM Serving: Host RAM, Local NVMe, Remote Storage, and LMCache Architecture

    High-concurrency large language model (LLM) serving faces an acute memory capacity bottleneck. While modern GPUs deliver high floating-point compute throughput, High-Bandwidth Memory (HBM) capacity remains severely constrained. In workloads involving multi-turn conversations, agentic coding loops, and long-document retrieval-augmented generation (RAG), Key-Value (KV) cache tensors rapidly consume available GPU memory, forcing engines to either reject requests, preempt running sequences, or recom

    AI
    9 min read
  2. Continuous Batching and Request Scheduling in Production LLM Serving: Comparing Orca, FastServe, Sarathi-Serve, and vLLM Architecture, Preemption Policies, Chunked Prefill Interleaving, and TTFT-TBT Trade-Offs

    Autoregressive large language model serving exhibits a fundamental architectural tension between compute utilization and latency guarantees. Standard deep learning inference pipelines rely on static request-level batching, where incoming queries are grouped into a fixed tensor, executed across forward passes until all sequences finish, and evicted simultaneously. In transformer-based text generation, static batching collapses serving efficiency. Because sequence lengths vary widely and token gen

    AI
    7 min read
  3. Model Routing and Cascades in Production: Comparing RouteLLM, FrugalGPT, Embedding Classifiers, and Verifier Cascades

    Model Routing and Cascades in Production: Comparing RouteLLM, FrugalGPT, Embedding Classifiers, and Verifier Cascades Enterprise LLM deployments face a persistent structural inefficiency: the uniform routing of all incoming queries to flagship frontier models. Commercial API pricing and self-hosted GPU infrastructure costs span two orders of magnitude between lightweight models (such as Llama 3.1 8B, GPT-4o-mini, and Claude 3.5 Haiku at $0.15 to $0.30 per million tokens) and frontier reasoning

    Inference
    9 min read
  4. Test-Time Compute and Inference Search in Production: Comparing Best-of-N, Monte Carlo Tree Search, PRM-Guided Beam Search, and Dynamic Budget Allocation Architecture, Token Economics, and Serving Latency

    In standard production Large Language Model (LLM) deployments, generation compute is fixed: a prompt is processed during prefill, and the model generates an answer via greedy autoregressive decoding or low-temperature sampling. Under this paradigm, accuracy on complex reasoning, mathematical derivation, theorem proving, and code synthesis is bounded by the model parameter count and training data distributions. Recent empirical work demonstrates that scaling inference-time compute can achieve pe

    AI
    11 min read
  5. Semantic Caching in Production LLM Systems: Architecture, Approximate Nearest Neighbor Matching, Cross-Encoder Verification, Invalidation Dynamics, and Serving Economics

    Semantic Caching in Production LLM Systems: Architecture, Approximate Nearest Neighbor Matching, Cross-Encoder Verification, Invalidation Dynamics, and Serving Economics Serving large language models at enterprise scale presents severe latency and cost bottlenecks. While frontier reasoning models and deep autoregressive decoders cost between $2.50 and $60.00 per million tokens and incur time-to-first-token (TTFT) delays ranging from 800 milliseconds to several seconds, a substantial fraction of

    AI
    9 min read
  6. Vision-Language Model Serving in Production: Comparing LLaVA-NeXT, Qwen2-VL, InternVL2, and Pixtral Architecture, Dynamic Resolution Patching, Visual Token Pruning, and Serving Latency Economics

    Deploying Vision-Language Models (VLMs) in production introduces systems challenges distinct from text-only Large Language Models. In standard text serving, input length scales linearly with character and token count. In multimodal systems, a single high-resolution image or short video clip can expand into thousands of visual tokens before textual generation begins. This expansion places immense strain on GPU memory bandwidth, inflates Key-Value (KV) cache allocation, and degrades Time-to-First-

    AI
    8 min read
  7. KV Cache Compression and Dynamic Pruning in Production Serving: Comparing StreamingLLM, H2O, SnapKV, PyramidKV, and DuoAttention Architecture, Memory Bandwidth Reduction, and Needle Retrieval Retention

    As context windows in production large language models expand from 8K tokens to 128K tokens and beyond, the Key-Value (KV) cache replaces model weights as the primary consumer of GPU High Bandwidth Memory (HBM). In high-throughput serving systems, the KV cache grows linearly with sequence length and batch size, creating a dual bottleneck: capacity exhaustion that caps concurrent batch size, and memory bandwidth saturation during autoregressive decoding. While quantization techniques (such as FP

    AI
    11 min read
  8. Prompt Compression in Production: Comparing Selective Context, LLMLingua, LongLLMLingua, and LLMLingua-2 Architecture, Token-Level Information Density, and Serving Economics

    In high-throughput production LLM deployments, prompt length dominates both serving latency and inference costs. For multi-turn conversational agents, long-document retrieval-augmented generation (RAG), and multi-step agentic workflows, input contexts routinely scale from 8,000 to over 64,000 tokens. Because the prefill phase scales quadratically in raw attention FLOPs and linearly in key-value (KV) cache allocation, long prompts drive up Time To First Token (TTFT) and consume disproportionate G

    AI
    9 min read
  9. Speculative Decoding in Production Serving: Comparing Small Draft Models, Medusa, EAGLE-2, and Lookahead Decoding Architecture, Verification Tree Overhead, and Throughput Economics

    Large language model inference during autoregressive decoding is structurally memory-bandwidth bound. During generation, each forward pass loads the model weight matrices (tens to hundreds of gigabytes) from High-Bandwidth Memory (HBM) into on-chip SRAM to produce a single token. Because the arithmetic intensity is close to zero, modern accelerators like the NVIDIA H100 and B200 spend the vast majority of their compute cycles stalled on memory bus transfers rather than executing matrix multiplic

    AI
    7 min read
  10. OpenAI Reinstates 5-Hour Codex and Work Limits for ChatGPT Plus Subscribers

    OpenAI has reinstated a rolling five-hour rate limit on Codex and ChatGPT Work for ChatGPT Plus subscribers. The change ends a temporary multi-week period during which the short-term window was suspended and users were constrained only by overall weekly quota ceilings. OpenAI engineering lead Thibault Sottiaux confirmed the reinstatement, stating that re-enforcing the five-hour window is necessary to smooth peak compute load across OpenAI's inference fleet as developer adoption of complex multi

    OpenAI
    1 min read
  11. Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics

    Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches. While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill a

    Inference
    11 min read
  12. Grammar-Constrained Decoding in Production: Comparing Outlines, llguidance, XGrammar, and LM-Format-Enforcer Architecture, Token Masking Overhead, and JSON Schema Enforcement

    Deploying Large Language Models into production software workflows requires deterministic adherence to structural formats such as JSON schemas, Pydantic data models, SQL queries, and tool-call signatures. Unconstrained autoregressive generation relies entirely on prompt instructions and few-shot examples, frequently resulting in malformed JSON syntax, trailing commas, type mismatches, or missing required attributes. Post-hoc repair parsers and retry loops introduce non-deterministic latency spik

    AI
    10 min read
  13. SCX.ai Partners With DDN to Scale Australian Sovereign AI Inference Infrastructure

    Australian sovereign AI infrastructure provider SCX.ai has entered a strategic partnership with data storage systems vendor DDN to expand its domestic AI inferencing cloud. The collaboration combines SCX's ASIC-based compute clusters with DDN's Infinia data management platform, targeting enterprise, government, and scientific research workloads that must remain strictly within Australian jurisdiction. The agreement comes days after SCX completed a $40 million initial public offering on the Aust

    AI
    2 min read
  14. Speech-to-Text Serving in Production: Comparing Faster-Whisper, Moonshine, SenseVoice, and NeMo Canary Architecture, Streaming Latency, and GPU Economics

    In conversational voice AI and real-time agentic workflows, the speech-to-text (STT) layer sets the hard lower bound on system responsiveness. Human conversational cadence expects turn-taking latencies between 200ms and 500ms. When an AI pipeline must accommodate downstream large language model (LLM) time-to-first-token generation (100ms to 250ms) and text-to-speech (TTS) audio synthesis (100ms to 200ms), the automatic speech recognition (ASR) stage cannot exceed 100ms to 150ms of processing ove

    AI
    8 min read
  15. Cross-Model KV Cache Transfer in Production: Architecture, Closed-Form Ridge Projections, and Cascaded Serving Economics

    Modern enterprise LLM serving architectures frequently rely on multi-model pipelines to balance inference cost, generation latency, and output quality. In model routing cascades, lightweight 8B models triage incoming queries and escalate complex reasoning tasks to 70B or MoE models. In speculative decoding pipelines, smaller draft models propose token sequences verified by larger target models. In long-horizon AI agent swarms, sub-agents frequently switch between specialized models across multi-

    LLM
    8 min read
  16. Vector Quantization for LLM Weights in Production: Comparing QuIP#, AQLM, and VPTQ Architecture, Dequantization Kernels, and 2-Bit Serving Economics

    Vector Quantization for LLM Weights in Production: Comparing QuIP#, AQLM, and VPTQ Architecture, Dequantization Kernels, and 2-Bit Serving Economics Scalar post-training quantization methods such as GPTQ and AWQ have become the standard for compressing large language models to 4-bit integer formats (INT4). At 4 bits per parameter, scalar techniques preserve over 98% of baseline 16-bit floating-point (FP16/BF16) model accuracy across common benchmarks. However, pushing scalar quantization below

    AI
    9 min read
  17. NVIDIA Enters Full Production on Groq 3 LPX, Hitting 3,400 Tokens per Second in Benchmarks

    NVIDIA has moved its Groq 3 LPX dedicated inference accelerator into full commercial production. Announced at Hot Chips 2026, the rack-scale accelerator system is designed as a purpose-built extension for NVIDIA's Vera Rubin NVL72 data center platform, targeting the compounding decode latency bottlenecks created by multi-step autonomous AI agents. European neocloud provider Nebius Group N.V. has committed as the first cloud infrastructure customer to deploy the accelerators, integrating them in

    Hardware
    3 min read
  18. LLM Inference on Enterprise CPUs in Production: Architecture, NUMA Topologies, Matrix Extensions, and Serving Economics

    Deploying large language models has historically been treated as an exclusively GPU-centric problem. Accelerators like NVIDIA H100 and A100 GPUs provide multi-terabyte-per-second High Bandwidth Memory (HBM) and tensor cores essential for training and high-concurrency serving. However, the operational economics of enterprise inference often diverge from frontier training requirements. Many enterprise applications, including internal code assistance, document extraction, low-concurrency agents, an

    AI
    6 min read
  19. Tool-Call Failure Recovery in Production AI Agents: Syntactic Repair, Schema Coercion, Parameter Inoculation, and Dynamic Fallback Architectures

    Tool-Call Failure Recovery in Production AI Agents: Syntactic Repair, Schema Coercion, Parameter Inoculation, and Dynamic Fallback Architectures Autonomous language model agents operate by interleaving natural language reasoning traces with structured tool invocations. However, when deployed in multi-turn production environments, raw tool calling exhibits significant fragility. Empirical studies from benchmark suites such as $\tau$-bench (arXiv:2406.12045) and the Berkeley Function Calling Lead

    Agents
    8 min read