KV Cache3 articles

KV Cache

Articles

KV Cache Compression and Dynamic Pruning in Production Serving: Comparing StreamingLLM, H2O, SnapKV, PyramidKV, and DuoAttention Architecture, Memory Bandwidth Reduction, and Needle Retrieval Retention
KV Cache Compression and Dynamic Pruning in Production Serving: Comparing StreamingLLM, H2O, SnapKV, PyramidKV, and DuoAttention Architecture, Memory Bandwidth Reduction, and Needle Retrieval Retention As context windows in production large language models expand from 8K tokens to 128K tokens and beyond, the Key-Value (KV) cache replaces model weights as the primary consumer of GPU High Bandwidth Memory (HBM). In high-throughput serving systems, the KV cache grows linearly with sequence length
August 25, 20261 min
Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM
Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM Autoregressive large language model inference is heavily constrained by memory bandwidth and the computational overhead of the prefill phase. For workloads such as multi-turn conversations, autonomous agent tool loops, few-shot prompt evaluations, and tree-search decoding, consecutive requests often share substantial token prefixes. In a standard multi-turn ses
August 22, 20261 min
NVIDIA Researchers Map Cross-Model KV Caches via Linear Ridge Regression to Accelerate Model Cascading Up to 25x
In multi-agent systems and tiered model cascades, routing requests across different model sizes is a standard architecture for balancing inference cost and generation quality. However, switching models mid-session has historically imposed a severe compute tax: because each neural architecture maintains its own distinct internal representations, the receiving model cannot read the key-value (KV) states generated by the previous model. The target model must recompute the entire conversation histor
August 21, 20261 min

KV Cache

KV Cache Compression and Dynamic Pruning in Production Serving: Comparing StreamingLLM, H2O, SnapKV, PyramidKV, and DuoAttention Architecture, Memory Bandwidth Reduction, and Needle Retrieval Retention

Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM

NVIDIA Researchers Map Cross-Model KV Caches via Linear Ridge Regression to Accelerate Model Cascading Up to 25x