llms.blog
inference18 articles

inference

Articles

  • Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

    While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engin

    1 min

  • Alibaba Demonstrates Native Qwen 3.8 27B Inference on XuanTie C950 RISC-V CPU at 30 Tokens per Second

    Alibaba's semiconductor division, T-Head, announced day-zero native inference support for its latest open-weight model, Qwen 3.8 27B, running directly on the XuanTie C950 RISC-V server processor. Operating without discrete graphics processing units, the 64-core RISC-V chip delivered sustained decode throughput of 30 tokens per second alongside a time-to-first-token latency of 1.9 seconds. The benchmark demonstrates how architectural extensions on general-purpose open instruction sets can handle

    1 min

  • Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    1 min

  • Mistral Expands Platform to Host Third-Party Open Weights Starting with GLM-5.2

    Mistral AI has broadened its API platform to host external open-weight foundation models, beginning with Zhipu AI's GLM-5.2. The move marks a strategic shift for the Paris-based AI company from serving only in-house architectures (such as Mistral Small, Mistral Medium, Mistral Large, and Voxtral) toward operating as a sovereign managed inference hub for third-party open weights. The integration introduces GLM-5.2 under the model identifier zai-glm-5-2 in public preview. The model is hosted with

    1 min

  • Multi-Token Prediction: How Future Token Supervision Densifies Representations and Speeds Up LLM Serving

    Standard autoregressive language models are trained under a strict next-token prediction objective. At every sequence position, the model consumes a prefix of tokens and predicts the single immediate successor token using a cross-entropy loss. While this paradigm has scaled language modeling across orders of magnitude, it suffers from an architectural limitation: myopic optimization. By evaluating loss exclusively on the immediate next step, standard training fails to reward representations that

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min

  • Test-Time Compute Scaling in Large Language Models: How Search, Verification, and Reasoning Chains Trade Latency for Accuracy

    Large language model performance has historically been dictated by pre-training compute scaling laws. As formalised by Kaplan et al. and Chinchilla scaling, increasing model parameters, dataset size, and pre-training FLOPs yielded predictable reductions in cross-entropy loss. During inference, however, computation remained strictly linear and deterministic: one forward pass per generated token. The emergence of inference-time reasoning architectures has introduced a third scaling axis: test-tim

    1 min

  • Etched in Talks to Raise 00M Led by Jane Street at 1B Valuation

    AI inference chip startup Etched is in negotiations to raise $700 million in a new financing round led by existing investor Jane Street, according to reporting from The Wall Street Journal. The proposed financing would value the San Jose-based semiconductor company at approximately $21 billion. The planned round follows a $300 million Series C led by Sequoia Capital that valued the startup at $10.3 billion, effectively doubling its valuation within weeks amid intensifying enterprise demand for

    1 min

  • Continuous Batching in Production LLM Serving: Iteration-Level Scheduling, Chunked Prefills, and Throughput Trade-Offs

    Static batching served as the standard execution paradigm for deep learning inference across computer vision and traditional natural language processing for years. In those domains, incoming requests typically feature fixed input dimensions and deterministic execution graphs. Autoregressive large language model serving breaks every assumption underlying static batching. Input prompts vary widely in token length, output generations terminate nondeterministically upon emitting an end-of-sequence t

    1 min

  • Token Sampling in Large Language Models: How Temperature, Top-k, Top-p, and Min-p Govern Generative Decoding

    Large language models are autoregressive next-token predictors. During inference, the neural network processes an input context and outputs a vector of unnormalized scores, known as logits, across every token in its vocabulary. The mathematical pipeline that converts those raw logits into a single chosen token is the sampling and decoding stage. While model architecture and pre-training weights determine what a model knows, sampling algorithms govern how that knowledge is extracted. Small chang

    1 min

  • Disaggregated Prefill and Decode in Production: Architecture, Economics, and KV Transfer Protocols

    In production large language model serving, the fundamental architectural tension lies between two computationally distinct phases: prefill (processing the input prompt) and decode (generating output tokens autoregressively). In standard co-located serving systems, both phases share the same GPU instances, memory pools, and execution batches. This co-location creates head-of-line blocking, degrades Time to First Token (TTFT), inflates Time Per Output Token (TPOT), and limits cluster-wide hardwar

    1 min

  • Anthropic Rolls Out Global AI Text Watermarking for Claude to Comply with EU AI Act

    Anthropic has started deploying model-level text watermarking across Claude to meet the regulatory requirements of the European Union AI Act. The company confirmed that because it currently lacks infrastructure to reliably partition model inference behavior by geographic jurisdiction, watermarking is being applied globally across all Claude web products and API endpoints. The update follows the formalization of the EU Code of Practice on Transparency of AI-Generated Content, signed in July 2026

    1 min

  • Grouped-Query Attention: How Head Sharing Slashed KV Cache Memory and Scaled LLM Serving

    Serving large language models at scale presents a fundamental hardware bottleneck: autoregressive decoding is bound by memory bandwidth rather than compute. While the initial prompt processing phase (prefill) operates as compute-bound matrix multiplications, token-by-token generation requires loading billions of cached attention states from GPU High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. Multi-Head Attention (MHA), introduced in the foundational Vaswani et al.

    1 min

  • LLM Model Routing and Cascades: Architecture, Economics, and Quality Trade-Offs

    In modern enterprise AI systems, uniform model dispatch, sending all incoming user traffic to a single frontier large language model, is one of the most common architectural inefficiencies. Frontier models like GPT-4o and Claude 3.5 Sonnet provide industry-leading reasoning and code generation capabilities, but their inference costs range between $2.50 and $15.00 per million tokens. Conversely, smaller open-weights or distilled models, such as Llama 3.1 8B or GPT-4o-mini, execute at a fraction o

    1 min

  • Speculative Decoding in Production: Architecture, Economics, and Serving Trade-Offs

    Standard autoregressive large language model (LLM) generation produces one token per forward pass. Because modern inference architectures must read tens of gigabytes of model weights from high-bandwidth memory (HBM) into SRAM to process each solitary token at low batch sizes, generation is strictly memory-bandwidth bound rather than compute bound. Speculative decoding alters this hardware equation. By pairing a fast draft mechanism with parallel verification by the primary target model, inferen

    1 min

  • Prompt Caching in Production: Architecture, Economics, and Invalidation Pitfalls

    As multi-turn AI agents, code generation assistants, and retrieval-augmented generation (RAG) pipelines expand in production, prompt sizes routinely exceed tens of thousands of tokens per request. In these architectures, the vast majority of tokens across consecutive API calls are static: system prompts, OpenAPI tool specifications, reference documents, and accumulated conversation history. Without optimization, inference engines must re-evaluate the full attention matrix for every token on eac

    1 min

  • How Modern Inference Engines Tamed Structured Outputs

    Autoregressive language models generate tokens sequentially by sampling from a probability distribution over a vocabulary. While this mechanism excels at natural language generation, it provides no structural guarantees when producing machine-readable data formats such as JSON, SQL, or structured tool calls. A single missing quotation mark, mismatched bracket, or hallucinated enum value invalidates the downstream payload. Prompt engineering, few-shot examples, and setting sampling temperature t

    1 min

  • French startup Kog says smarter software can pull 30x faster inference out of stock GPUs

    The race for faster AI inference has pushed labs toward custom silicon, but a French startup argues the cheapest speedup is already sitting in servers companies bought. Kog says it can reach roughly thirty times faster LLM inference on standard data-center GPUs using software alone. Kog demonstrated three thousand tokens per second per request on AMD MI300X and NVIDIA H200 GPUs in May, running a small two-billion-parameter model it has since open-sourced as Laneformer 2B. Chief executive Gael D

    1 min