State Space Models4 articles

State Space Models

Articles

State Space Models and Mamba (Mamba-1 and Mamba-2): Mathematical Foundations, Selective State Spaces, Structured State Space Duality (SSD), and Linear-Time Sequence Modeling
State Space Models (SSMs) and their modern selective formulations, most notably Mamba-1 and Mamba-2, represent a foundational alternative to the standard Transformer architecture for sequence modeling. While multi-head self-attention scales quadratically with sequence length ($O(T^2)$) and requires an ever-expanding Key-Value (KV) cache during autoregressive generation ($O(T)$), State Space Models achieve linear time complexity ($O(T)$) during training and constant memory footprint ($O(1)$) per
August 28, 20261 min
Selective State Space Models (Mamba): Mathematical Foundations, Discretization Dynamics, and Linear-Time Sequence Modeling
The dominant paradigm in natural language processing and modern foundation models relies almost exclusively on the Transformer architecture. While standard multi-head self-attention delivers strong expressivity and in-context learning capabilities, its computational requirements present fundamental scaling bottlenecks: training complexity scales quadratically with sequence length $O(L^2)$, and autoregressive token generation requires storing key-value pairs in high-bandwidth memory (HBM), creati
August 27, 20261 min
Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off
Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off Autoregressive language models face a fundamental tension between inference efficiency and long-context retrieval capacity. Pure Transformer architectures scale quadratic computational complexity during sequence prefill and linear key-value (KV) cache memory consumption during autoregressive token generation. Conversely, pure State Space Models (SSMs) and linear recurrent neural
August 23, 20261 min
State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs
For years, deep sequence modeling operated under a structural divide. On one side stood the Transformer architecture, anchored by softmax attention. Softmax attention scales quadratically in compute and memory with sequence length (O(T^2)), but its operations map cleanly to dense General Matrix Multiplications (GEMMs), maximizing utilization of GPU Tensor Cores. On the other side stood Structured State Space Models (SSMs), such as S4 and Mamba-1. Structured SSMs scale linearly in time (O(T · N))
August 20, 20261 min

State Space Models

Articles

State Space Models and Mamba (Mamba-1 and Mamba-2): Mathematical Foundations, Selective State Spaces, Structured State Space Duality (SSD), and Linear-Time Sequence Modeling

Selective State Space Models (Mamba): Mathematical Foundations, Discretization Dynamics, and Linear-Time Sequence Modeling

Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off

State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs