llms.blog
Educational5 articles

Educational

Articles

  • Context Window Extension in Large Language Models: How Position Interpolation, YaRN, and LongRoPE Scale Sequence Lengths

    Large language models are bounded during pretraining by a fixed sequence length, typically between 2,048 and 8,192 tokens. When standard autoregressive transformers attempt to process sequences beyond this pretraining context window, performance degrades immediately. Perplexity rises sharply and the model loses coherence within a few dozen tokens past the training boundary. Extending this context window by training from scratch on long sequences is computationally prohibitive due to the quadrat

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min

  • In-Context Learning in Large Language Models: How Induction Heads and Attention Circuits Learn Without Weight Updates

    Autoregressive large language models demonstrate the ability to adapt to new tasks, follow few-shot demonstrations, and execute algorithmic patterns entirely within their context windows. Unlike traditional fine-tuning, in-context learning occurs at inference time with frozen model parameters, leaving weights completely unchanged ($\Delta \theta = 0$). For several years following the scaling demonstrations in GPT-3, the internal mechanism governing in-context learning remained an empirical blac

    1 min

  • State Space Models in Large Language Models: How Mamba, S4, and Selective Recurrence Challenge Transformer Attention

    Modern large language models rely almost universally on the Transformer architecture. However, the core mechanism powering Transformers, softmax multi-head self-attention, exhibits fundamental scaling limitations. Specifically, standard self-attention requires quadratic time and memory complexity relative to sequence length during prefilling, alongside a linear memory expansion for the key-value (KV) cache during autoregressive token generation. To circumvent these computational bottlenecks, re

    1 min

  • Rotary Position Embeddings: How Geometry Solved Long Context in Modern LLMs

    Rotary Position Embedding (RoPE) has become the standard positional encoding mechanism across modern large language models, including Meta's Llama series, Mistral, Qwen, and DeepSeek. Unlike earlier techniques that added positional vectors directly to token representations or modified attention matrices with relative distance penalties, RoPE encodes position through geometric rotations in the complex plane. This design enables models to compute relative token distances while processing individu

    1 min