Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and attention heads.

At a context window of 128,000 tokens, storing full FP16 KV states for a 70-billion parameter model requires over 100 GB of GPU memory for a single request, exceeding the memory capacity of an entire NVIDIA H100 GPU before model weights are loaded.

Sliding Window Attention (SWA), introduced in sparse attention research by Child et al. (2019) and popularized by Longformer, Mistral 7B, and Gemma 3, addresses this scaling bottleneck. By restricting token interactions to a fixed local span and bounding the active KV cache with cyclic rolling buffers, SWA drops per-layer attention computation to $O(N \times W)$ and caps local KV cache memory at $O(W)$, where $W$ is the sliding window size.

The Mechanics of Sliding Window Attention

In standard multi-head causal self-attention, each query vector $q_i$ at timestep $i$ computes scaled dot-product attention across all keys from the start of the sequence up to position $i$:

$$\text{Attention}(Q, K, V)_i = \sum_{j=1}^{i} \text{softmax}\left(\frac{q_i k_j^T}{\sqrt{d_k}}\right) v_j$$

Under Sliding Window Attention, each query token attends only to keys and values situated within a sliding window of length $W$ immediately preceding it. Any key position $j < i - W + 1$ is masked out:

$$\text{SWA}(Q, K, V)_i = \sum_{j=\max(1, i-W+1)}^{i} \text{softmax}\left(\frac{q_i k_j^T}{\sqrt{d_k}}\right) v_j$$

This transforms the full triangular attention matrix into a banded diagonal matrix of width $W$. For a sequence of length $N$, the number of computed pairwise attention scores drops from $N(N+1)/2$ down to $N \times W - W(W-1)/2$. When $N = 32,768$ and $W = 4,096$, this constitutes an 87.5% reduction in attention dot-product operations during the prefill phase.

Layer Stacking and Theoretical Receptive Fields

A common misconception is that a model with a window size of $W = 4,096$ cannot comprehend sequences longer than 4,096 tokens. While a single transformer layer with SWA can only look back $W$ positions directly, stacking multiple transformer layers expands the effective receptive field recursively.

Consider a multi-layer transformer where layer $k$ computes hidden state $h_i^{(k)}$ at sequence position $i$:

• Layer 1: Position $i$ attends directly to input tokens in the range $[i - W + 1, i]$.
• Layer 2: Position $i$ attends to hidden states from Layer 1 in the range $[i - W + 1, i]$. Because position $i - W + 1$ at Layer 1 already incorporated information from position $(i - W + 1) - W + 1 = i - 2W + 2$, Layer 2 can access information spanning back $2W - 1$ tokens.
• Layer $L$: Across $L$ stacked transformer layers, position $i$ possesses a theoretical receptive field of:

$$\text{Receptive Field} = L \times (W - 1) + 1 \approx L \times W$$

For a 32-layer model like Mistral 7B with a window size of $W = 4,096$, the theoretical receptive field reaches $32 \times 4,096 = 131,072$ tokens (128K context).

The Information Dilution Bottleneck

While theoretical receptive fields scale linearly with depth, empirical studies demonstrate that stacking pure local sliding window layers creates severe information bottlenecks. As documented by researchers analyzing long-context retrieval in Xiao et al. (2024), multi-hop information transfer across deep local layers suffers from three structural degradation factors:

1. Residual Dilution: At each transformer layer, local representations pass through residual connections ($x + \text{LayerNorm}(\text{Attention}(x)) + \text{MLP}(\dots)$). Distant contextual signals from $k$ layers below become progressively attenuated against dominating local token representations.
2. Representation Compaction: A single token vector at position $i - W$ must act as a lossy summary channel for all $W$ tokens preceding it. Compressing thousands of upstream tokens into a single hidden vector creates severe information loss.
3. Gradient Vanishing in Long Hops: Backpropagating loss gradients over a 30-hop relay of local attention windows introduces optimization friction compared to direct single-hop global attention connections.

As a consequence, pure SWA models often struggle on needle-in-a-haystack retrieval tasks and multi-document synthesis when critical evidence lies dozens of hops away from the generation point.

Hybrid and Interleaved Attention Topologies

To resolve the information dilution bottleneck while preserving the computational and memory advantages of sliding windows, modern foundation models employ hybrid, interleaved attention architectures. Rather than applying SWA uniformly across every layer, architectures interleave constrained local layers with unconstrained global attention layers.

1. Uniform SWA (Mistral 7B & Mixtral 8x7B)

Mistral AI (2023) deployed uniform 4,096-token sliding windows across all 32 transformer layers, combined with Grouped-Query Attention (GQA). This design maximized raw serving throughput on 8K-to-32K sequence lengths by enabling uniform rolling cache allocations.

2. 1:1 Interleaved Attention (Gemma 2)

In Gemma 2 (2024), Google DeepMind implemented an alternating 1:1 scheme across the network:

• Even layers: Local sliding window attention with $W = 4,096$ tokens.
• Odd layers: Full global attention with span up to 8,192 tokens.

This structure ensures that every local layer is immediately followed by a global layer capable of aggregating representations across the entire context window, eliminating multi-hop information loss while cutting attention FLOPs and local KV cache requirements in half.

3. 5:1 Interleaving (Gemma 3)

In the Gemma 3 Technical Report (2025), DeepMind increased the aggressiveness of sliding window interleaving to a 5:1 ratio:

• 5 consecutive local attention layers operating with a tight sliding window of $W = 1,024$ tokens.
• 1 global attention layer operating across the full sequence length (up to 128,000 tokens).

By restricting five out of every six layers to a 1,024-token cache, Gemma 3 slashes overall KV cache memory usage by over 80% compared to standard full-attention models at 128K context, while suffering negligible degradation in core language modeling perplexity.

The Rolling Buffer KV Cache Architecture

In standard autoregressive inference, the KV cache grows by one key vector and one value vector for every generated token. For an attention layer with sequence length $N$, the cache retains all $N$ historical vectors.

Under Sliding Window Attention, any key-value vector corresponding to a token position $j < i - W + 1$ will never be attended to by subsequent queries. This enables replacing the unbounded linear cache with a fixed-capacity circular buffer known as the Rolling Buffer KV Cache.

Logical Sequence Positions: [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9 ]  (W = 4)
Rolling Buffer Allocation:  [Key/Value slots: 0, 1, 2, 3]

Timestep 0 -> slot 0 (0 mod 4)
Timestep 1 -> slot 1 (1 mod 4)
Timestep 2 -> slot 2 (2 mod 4)
Timestep 3 -> slot 3 (3 mod 4)
Timestep 4 -> slot 0 (4 mod 4) [Overwrites token 0]
Timestep 5 -> slot 1 (5 mod 4) [Overwrites token 1]
Timestep 6 -> slot 2 (6 mod 4) [Overwrites token 2]
Timestep 7 -> slot 3 (7 mod 4) [Overwrites token 3]
Timestep 8 -> slot 0 (8 mod 4) [Overwrites token 4]

Modulo Addressing and Index Wraparound

The cache has a fixed physical capacity of $W$ tokens. For any token generated at global timestep $t$, its key and value tensors are written directly into physical slot:

$$\text{Slot Index} = t \pmod W$$

Once the sequence length exceeds $W$, new keys and values overwrite entries that have fallen outside the active sliding window. The physical memory footprint of the local layer caps permanently at $W$ tokens, regardless of whether the prompt contains 10,000 or 1,000,000 tokens.

Relative Positional Encodings (RoPE) in Rolling Buffers

When using Rotary Position Embeddings (RoPE), positional rotations are tied to absolute sequence positions $t$, not the modulo slot index $t \pmod W$. During the attention kernel computation, query vector $q_t$ (rotated by position $t$) computes dot products against keys stored in the circular buffer.

Because keys were rotated with their original absolute position $j$ upon insertion ($R_{\Theta, j} k_j$), the inner product preserves the correct relative distance:

$$\langle R_{\Theta, t} q_t, R_{\Theta, j} k_j \rangle = q_t^T R_{\Theta, j - t} k_j$$

The attention kernel unrolls the physical cyclic buffer into logical chronological order or applies index arithmetic inside fused GPU kernels without requiring costly memory copies.

Chunked Prefill with Rolling Caches

When processing a long prompt that exceeds window size $W$, the prefill stage cannot simply compute full quadratic cross-attention across the entire sequence. Serving frameworks divide the prompt into sequential chunks of size $W$:

1. Chunk 1 ($[0, W-1]$): Computes self-attention and fills cache slots $[0, W-1]$.
2. Chunk 2 ($[W, 2W-1]$): Computes causal attention within Chunk 2 and cross-attention against keys stored in Chunk 1, overwriting the physical cache slots with Chunk 2 representations.
3. Subsequent Chunks: Each chunk accesses the rolling buffer populated by its immediate predecessor chunk.

Kernel Acceleration and Memory Bandwidth

The true serving bottleneck for large language models during generation is GPU memory bandwidth (the Memory Wall). Every newly generated token requires streaming the entire KV cache from high-bandwidth GPU memory (HBM) into on-chip SRAM to compute attention weights.

In a full-attention layer at sequence position $N = 65,536$, generating a single token requires reading:

$$\text{Data Transferred} = 2 \times N \times H_{\text{kv}} \times D \times \text{BytesPerElement}$$

For a model with 8 KV heads, head dimension 128, and FP16 precision (2 bytes), reading 65,536 tokens requires streaming:

$$2 \times 65,536 \times 8 \times 128 \times 2 = 268.4 \text{ MB per layer}$$

Across a 32-layer model, this demands reading 8.59 GB of HBM memory for every generated token.

With Sliding Window Attention set to $W = 4,096$, the memory streamed per local layer drops to:

$$2 \times 4,096 \times 8 \times 128 \times 2 = 16.77 \text{ MB per layer}$$

This achieves a 16x reduction in memory traffic per local layer. In modern GPU kernels such as FlashAttention-2 and FlashAttention-3, the sliding window causal mask allows the kernel to skip out-of-window SRAM block tiles entirely, eliminating both memory transfers and computation.

Architectural Comparison and KV Cache Footprint

To illustrate the concrete impact of sliding window attention topologies, consider a 32-layer transformer model with 8 KV heads and head dimension 128 operating at FP16 precision across different context lengths:

• Full Attention (Standard): Every layer stores all historical tokens. Requires 4.29 GB at 32K context and 17.18 GB at 128K context per concurrent stream.
• Uniform SWA (Mistral-style, $W = 4,096$): All 32 layers use rolling buffers capped at 4,096 tokens. Requires 0.54 GB at 32K context and stays strictly capped at 0.54 GB at 128K context.
• 1:1 Interleaved (Gemma 2-style): 16 local layers ($W = 4,096$) alternating with 16 full global layers. Requires 2.41 GB at 32K context and 8.86 GB at 128K context (a 48% reduction vs full attention).
• 5:1 Interleaved (Gemma 3-style): 27 local layers ($W = 1,024$) and 5 full global layers. Requires 0.81 GB at 32K context and 3.22 GB at 128K context (an 81% reduction vs full attention).

As sequence lengths push into hundreds of thousands of tokens, interleaved architectures provide the optimal compromise: the global layers maintain single-hop access across the entire context history for needle retrieval, while the majority of local layers keep the KV cache memory footprint and memory bandwidth consumption strictly bounded.

Sources

• Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). Generating Long Sequences with Sparse Transformers. OpenAI.
• Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Transformer. Allen Institute for AI.
• Jiang, A. Q., et al. (2023). Mistral 7B. Mistral AI.
• Google DeepMind. (2024). Gemma 2: Improving Open Language Models at a Practical Size. Google DeepMind.
• Google DeepMind. (2025). Gemma 3 Technical Report. Google DeepMind.
• Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024). Efficient Streaming Language Models with Attention Sinks. ICLR 2024.
• Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. Stanford University.