Subscribe
Topic / page 6

Inference

Articles, page 6

101-120
  1. Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

    Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at

    AI
    8 min read
  2. Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS

    Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel

    AI
    7 min read
  3. Rotational Quantization: How Orthogonal Transforms and Hadamard Incoherence Enable Outlier-Free Low-Bit Inference

    Large language model serving is governed by two physical bottlenecks: memory bandwidth during autoregressive decoding and compute throughput during prompt prefill. While weight-only post-training quantization (such as GPTQ or AWQ) reduces parameter footprint to 4 bits, it leaves activations in 16-bit floating-point formats. As a result, inference engines cannot utilize high-throughput INT4 tensor cores, and the key-value (KV) cache continues to consume massive memory pools. Attempting to quanti

    AI
    8 min read
  4. Multi-LoRA Serving in Production: Architecture, Dynamic Adapter Swapping, and GPU Memory Management

    Deploying hundreds or thousands of fine-tuned language models across enterprise workflows presents a fundamental infrastructure dilemma. While parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce training compute by freezing base model weights and training compact low-rank matrices, naive deployment strategies fail at scale. Merging adapter weights directly into the base model creates a distinct 7B to 70B parameter checkpoint for every specialized task, requiring

    AI
    8 min read
  5. Contrastive Decoding in Large Language Models: How Comparing Expert and Amateur Logits Suppresses Hallucination and Reasoning Errors

    Contrastive Decoding in Large Language Models: How Comparing Expert and Amateur Logits Suppresses Hallucination and Reasoning Errors Autoregressive large language models operate by predicting the conditional probability distribution of the next token given a sequence of preceding tokens. However, translating these continuous probability vectors into coherent, factual, and logically sound sequences remains one of the fundamental challenges of modern natural language processing. Traditional deco

    AI
    6 min read
  6. Benchmarking LLM Inference in Production: Architecture, Metrics, and Tooling Across AIPerf, GuideLLM, and LLMPerf

    Traditional HTTP load-testing tools such as Apache Bench, wrk, and Locust evaluate systems using uniform request-response cycles. These tools send a payload, wait for the full response, and compute metrics such as requests per second (RPS) and round-trip latency percentiles. For stateless REST APIs, this model aligns directly with user experience. Large language model (LLM) serving fundamentally breaks this abstraction. An inference request is not an atomic computation. It consists of two disti

    AI
    7 min read
  7. Replit Launches Free Mode Powered by OpenAI's GPT-5.6 Luna

    Software development platform Replit announced the rollout of Free Mode, a tier powered by OpenAI's GPT-5.6 Luna model designed to support zero-cost planning, exploration, and codebase assistance. The integration utilizes recent inference cost reductions and efficiency improvements within the GPT-5.6 model family to provide unmetered conversational assistance without drawing from paid compute budgets. Model Routing and Persistent Project Context Replit Free Mode integrates directly into the

    AI
    1 min read
  8. Cache-Aware Load Balancing in Production LLM Serving: Architecture, Prefix Affinity, and Multi-Replica Routing Trade-Offs

    Cache-Aware Load Balancing in Production LLM Serving: Architecture, Prefix Affinity, and Multi-Replica Routing Trade-Offs When scaling large language model inference across multiple GPU worker nodes, standard Layer-4 and Layer-7 load balancing algorithms create an unseen performance cliff. Round-robin, least-connections, and random routing distribute HTTP/gRPC requests uniformly across compute replicas. However, modern LLM inference engines rely on prompt caching mechanisms, such as vLLM Automa

    AI
    6 min read
  9. Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    Architecture
    8 min read
  10. Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving

    Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene

    Speculative Decoding
    7 min read
  11. FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation

    FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation In large language model serving, execution divides into two distinct operational regimes: prompt prefill and autoregressive token generation (decoding). While FlashAttention transformed prefill throughput by eliminating High Bandwidth Memory (HBM) round-trips for intermediate attention matrices, standard FlashAttention algorithms encounter a severe hardware utilization bottleneck during decoding.

    AI
    7 min read
  12. Z.ai Opens GLM-5.3 API Access at $1.40/$4.40 per Million Tokens with Prompt Caching

    Chinese foundation model developer Z.ai (Zhipu AI) has opened public API access to GLM-5.3, offering developers direct endpoint integration following the model's initial release. The company kept base token rates aligned with the prior generation while introducing discounted prompt caching. GLM-5.3 is priced at $1.40 per million input tokens and $4.40 per million output tokens on the Z.ai platform. For workloads utilizing prompt caching, cached input tokens are billed at $0.26 per million, an 8

    LLM
    2 min read
  13. Z Lab Releases DFlash 2 for Qwen 3.8 27B: Block Diffusion Speculative Decoding with Target KV Injection

    Z Lab has released DFlash 2 checkpoints for Alibaba's Qwen 3.8 27B model family, advancing block-diffusion speculative decoding for open-weights LLM serving. By replacing conventional autoregressive draft models with a non-causal diffusion mechanism paired with direct target key-value (KV) cache injection, the framework achieves up to 3x to 4.3x throughput speedups in production inference engines like SGLang and vLLM without altering output token distributions. For where the 27B fits in the line

    Inference
    2 min read
  14. Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture

    Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture Cerebras Systems has announced the CS-4, a rack-scale AI accelerator system designed around three of its next-generation Wafer Scale Engine 3 Turbo (WSE-3 Turbo) chips and a modular hardware architecture dubbed Nexus. Cerebras confirmed that initial customer shipments for the CS-4 are scheduled to begin in the current quarter. The new system marks a structural shift from Cerebras's single-wafer CS

    Hardware
    2 min read
  15. Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

    While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engin

    AI
    5 min read
  16. Alibaba Demonstrates Native Qwen 3.8 27B Inference on XuanTie C950 RISC-V CPU at 30 Tokens per Second

    Alibaba's semiconductor division, T-Head, announced day-zero native inference support for its latest open-weight model, Qwen 3.8 27B, running directly on the XuanTie C950 RISC-V server processor. Operating without discrete graphics processing units, the 64-core RISC-V chip delivered sustained decode throughput of 30 tokens per second alongside a time-to-first-token latency of 1.9 seconds. Update, October 29, 2026: a developer has since gone the other way and compiled a 32-bit RISC-V CPU into th

    Hardware
    2 min read
  17. Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    LLM
    5 min read
  18. Multi-Token Prediction: How Future Token Supervision Densifies Representations and Speeds Up LLM Serving

    Standard autoregressive language models are trained under a strict next-token prediction objective. At every sequence position, the model consumes a prefix of tokens and predicts the single immediate successor token using a cross-entropy loss. While this paradigm has scaled language modeling across orders of magnitude, it suffers from an architectural limitation: myopic optimization. By evaluating loss exclusively on the immediate next step, standard training fails to reward representations that

    AI
    6 min read
  19. Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    AI
    5 min read