llms.blog
State Space Models4 articles

State Space Models

Articles

  • State Space Models and Mamba (Mamba-1 and Mamba-2): Mathematical Foundations, Selective State Spaces, Structured State Space Duality (SSD), and Linear-Time Sequence Modeling

    State Space Models (SSMs) and their modern selective formulations, most notably Mamba-1 and Mamba-2, represent a foundational alternative to the standard Transformer architecture for sequence modeling. While multi-head self-attention scales quadratically with sequence length ($O(T^2)$) and requires an ever-expanding Key-Value (KV) cache during autoregressive generation ($O(T)$), State Space Models achieve linear time complexity ($O(T)$) during training and constant memory footprint ($O(1)$) per

    1 min

  • Selective State Space Models (Mamba): Mathematical Foundations, Discretization Dynamics, and Linear-Time Sequence Modeling

    The dominant paradigm in natural language processing and modern foundation models relies almost exclusively on the Transformer architecture. While standard multi-head self-attention delivers strong expressivity and in-context learning capabilities, its computational requirements present fundamental scaling bottlenecks: training complexity scales quadratically with sequence length $O(L^2)$, and autoregressive token generation requires storing key-value pairs in high-bandwidth memory (HBM), creati

    1 min

  • Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off

    Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off Autoregressive language models face a fundamental tension between inference efficiency and long-context retrieval capacity. Pure Transformer architectures scale quadratic computational complexity during sequence prefill and linear key-value (KV) cache memory consumption during autoregressive token generation. Conversely, pure State Space Models (SSMs) and linear recurrent neural

    1 min

  • State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs

    For years, deep sequence modeling operated under a structural divide. On one side stood the Transformer architecture, anchored by softmax attention. Softmax attention scales quadratically in compute and memory with sequence length (O(T^2)), but its operations map cleanly to dense General Matrix Multiplications (GEMMs), maximizing utilization of GPU Tensor Cores. On the other side stood Structured State Space Models (SSMs), such as S4 and Mamba-1. Structured SSMs scale linearly in time (O(T · N))

    1 min