Subscribe
Topic

Efficiency

Methods that make LLMs cheaper to run. Posts explain SmoothQuant, which migrates activation outliers into weights for W8A8 quantization, and Mixture-of-Depths, which skips layers to spend compute only where it is needed.

Articles

1-2
  1. SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models

    SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models Serving large language models (LLMs) in production environments presents two distinct hardware bottlenecks. During the autoregressive generation (decode) phase with small batch sizes, inference is memory-bandwidth bound, as billions of parameters must be streamed from High Bandwidth Memory (HBM) to on-chip SRAM for every generated token. Conversely, during the pro

    Quantization
    12 min read
  2. Mixture-of-Depths: How Dynamic Compute Allocation and Layer Skipping Scale LLM Efficiency

    Standard transformer architectures allocate a uniform computational budget to every token in a sequence. Regardless of whether a model is processing a predictable punctuation mark, a common grammatical connective, or a mathematically dense reasoning step, every token undergoes an identical sequence of matrix multiplications across every multi-head attention and multilayer perceptron (MLP) block throughout the network's depth. This static compute distribution is computationally inefficient. Whil

    AI
    6 min read