Structured State Space Models and Mamba: Mathematical Foundations, Continuous Discretization, and Hardware-Aware Associative Scans
The dominant architectural paradigm in modern deep learning relies on Transformer multi-head self-attention. While self-attention achieves expressive sequence modeling by enabling every token to directly route information to and from every preceding token, it introduces steep computational overheads: training and prefill compute scale quadratically with sequence length (O(L^2)), and autoregressive inference requires an uncompressed Key-Value (KV) cache that scales linearly with sequence length (
1 min