Subscribe
Topic / page 3

Transformers

Articles, page 3

41-57
  1. Representation Degeneration and Anisotropy in Large Language Models: Why Embeddings Collapse into Narrow Cones

    In neural language modeling, high-dimensional vector spaces are expected to distribute token and sequence representations uniformly across their ambient dimensions. A space that utilizes all available geometric directions provides maximal representational capacity, allowing subtle semantic and syntactic differences between tokens to be resolved cleanly. Empirical analysis across architectures from BERT and GPT-2 to modern frontier foundation models reveals the opposite behavior: learned token a

    AI
    9 min read
  2. The Residual Stream in Transformers: How Linear Additive State, Subspace Projections, and Layer Accumulation Power Modern LLMs

    In standard descriptions of transformer architectures, multi-head attention and feedforward networks receive the vast majority of analytical focus. However, the operational backbone coordinating these sublayers is the residual stream. First introduced in convolutional computer vision models by He et al. (2016) to mitigate vanishing gradients in deep networks, residual skip connections operate in large language models as a persistent, high-dimensional linear communication bus. Rather than proces

    AI
    6 min read
  3. Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training

    Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform

    AI
    10 min read
  4. Multi-Query Attention: How Single Key-Value Head Sharing Slashed Transformer Serving Bottlenecks

    Multi-query attention (MQA) is an architectural modification to the Transformer attention mechanism designed to resolve the memory bandwidth bottleneck during autoregressive token generation. First proposed by Noam Shazeer in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need, MQA alters the ratio of query, key, and value heads by sharing a single key head and a single value head across all query heads in each Transformer layer. While standard multi-head attention (MHA) pr

    AI
    7 min read
  5. Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory

    In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me

    AI
    9 min read
  6. Cross-Attention and Latent Bottlenecks: How Asymmetric Queries Fuse External Modalities in Transformer Architectures

    Standard self-attention mechanisms in Transformer architectures process tokens symmetrically: every token in a sequence queries every other token within the same sequence. While this symmetric formulation drives autoregressive sequence generation and dense contextual representations, it exhibits a quadratic compute and memory profile of O(N^2) relative to sequence length. When scaling to high-resolution sensory inputs (such as raw video frames, high-density audio waveforms, or millions of point

    AI
    8 min read
  7. Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces

    Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces The attention mechanism serves as the computational core of modern Transformer architectures, providing a mechanism for tokens to dynamically exchange information across arbitrary sequence positions without recurrent state transitions or fixed convolutional receptive fields. While single-head attention computes a single set of attention weights per token pair, modern large lan

    AI
    10 min read
  8. State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs

    For years, deep sequence modeling operated under a structural divide. On one side stood the Transformer architecture, anchored by softmax attention. Softmax attention scales quadratically in compute and memory with sequence length (O(T^2)), but its operations map cleanly to dense General Matrix Multiplications (GEMMs), maximizing utilization of GPU Tensor Cores. On the other side stood Structured State Space Models (SSMs), such as S4 and Mamba-1. Structured SSMs scale linearly in time (O(T · N))

    AI
    6 min read
  9. Diffusion Transformers (DiT): How Patchification and adaLN-Zero Replaced U-Nets in Generative AI

    Generative visual models relied for years on convolutional U-Net architectures to execute iterative denoising. From Denoising Diffusion Probabilistic Models (DDPM) and Ablated Diffusion Models (ADM) to Latent Diffusion Models (LDMs) behind Stable Diffusion, convolutional backbones served as the default engine for image synthesis. While convolutional inductive biases provided translation equivariance and local spatial hierarchies, they imposed architectural rigidities that resisted compute scalin

    AI
    8 min read
  10. Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

    Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at

    AI
    8 min read
  11. Attention with Linear Biases (ALiBi): How Static Positional Slopes Enable Zero-Shot Context Extrapolation

    Large language models process sequences by transforming discrete tokens into continuous vector representations. Standard dot-product self-attention is permutation-invariant: without explicit positional information, the attention operation treats a sequence as an unordered bag of tokens. Early transformer architectures addressed this limitation using Absolute Positional Embeddings (APE), either through fixed sinusoidal functions or learned lookup tables added directly to token embeddings. While

    AI
    6 min read
  12. The Logit Lens: How Direct Unembedding Decodes Intermediate Representations in Large Language Models

    Autoregressive large language models process input tokens through deep stacks of transformer blocks, generating a final probability distribution over the vocabulary at the final layer. For years, the intermediate computations occurring within these hidden layers were treated as opaque black boxes. Mechanistic interpretability research has demonstrated that modern transformer architectures operate through a structured, iterative refinement process across their residual stream. The logit lens and

    AI
    6 min read
  13. Layer Normalization and RMSNorm in Large Language Models: How Pre-LN, Scaling Invariance, and QK-Norm Stabilize Deep Transformers

    Training deep autoregressive Transformers requires maintaining numerical stability across dozens or hundreds of stacked attention and feed-forward blocks. As models scale from 7 billion to hundreds of billions of parameters, uncontrolled variance growth along the residual stream or unbounded attention logits can trigger catastrophic loss spikes, gradient underflow, or numerical divergence. Normalization layers act as the primary stabilizing mechanism in modern Large Language Models (LLMs). Whil

    AI
    8 min read
  14. Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    Architecture
    8 min read
  15. Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads

    Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o

    AI
    9 min read
  16. Grouped-Query Attention: How Head Sharing Slashed KV Cache Memory and Scaled LLM Serving

    Serving large language models at scale presents a fundamental hardware bottleneck: autoregressive decoding is bound by memory bandwidth rather than compute. While the initial prompt processing phase (prefill) operates as compute-bound matrix multiplications, token-by-token generation requires loading billions of cached attention states from GPU High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. Multi-Head Attention (MHA), introduced in the foundational Vaswani et al.

    AI
    6 min read