Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts
Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at


















