Long-running agents have a memory bill that grows every turn. Each tool call adds output to the transcript, and every later request has to read the KV cache for all of it. The usual fix is context engineering: once the transcript passes a token threshold, summarize or trim it. A new paper from Tsinghua University researchers, with a co-author at the Chinese University of Hong Kong, Cache Engineering for Agents (posted October 20, 2026), argues that this fix is expensive in a way most harnesses hide, and proposes letting the agent edit its KV cache directly instead of editing its text.
The authors also release a serving engine, AKV, built on SGLang, that supports this. Their headline numbers: across three open-weight models and four agent benchmarks, a simple drop policy cut average API-equivalent cost by 27.7% and raised average benchmark scores by 1.7 points versus threshold-triggered summarization, with up to 1.51x faster P95 request latency in serving tests.
The problem with summarizing an agent's history
Prefix caching only works if the prefix stays the same. When a harness summarizes or trims old messages, it changes the start of the transcript, so every cached KV state after the edit is invalid and the retained text must be prefilled again. The earlier the edit, the more has to be recomputed. That is why common harnesses wait until a length threshold, then compress most of the trajectory at once and keep only a few recent rounds.
The paper measures both sides of that trade with GPT-OSS-120B on the first 100 BrowseComp-Plus (BCP) tasks:
- Trimming to the latest 12 rounds of tool responses cost 1.85x as much as simply keeping the full context, because of re-prefill.
- Summarizing at a 48K-token threshold cut cost by about 7% but lowered accuracy by 5 percentage points.
- In the full cost breakdown (Figure 13), cache reads were 10.8 of the 13.1 cents per task for the Standard baseline on BCP, or 82.4% of its cost.
So the cost is mostly reading old cache, and the standard way to shrink it either re-prefills or loses information. If you have read our piece on why coding agents do better with whole files than compressed context, this is a related warning from the serving side.
Three cache operations: append, drop, repos
Cache engineering leaves the text transcript alone and changes which KV entries later tokens can attend to. The paper defines three operations:
- append: the normal step, adding KV for a new token.
- drop: removes selected KV entries from future attention. Retained entries, their position encodings and the logical token sequence stay as they were, so nothing is recomputed.
- repos: renumbers the positions of retained keys to 0 through r-1 and resets the next position index. For RoPE models this is a rotation of each cached key from its old position to its new one; values are untouched.
Repos matters because drop alone does not reset positions. Indices keep growing with the logical transcript, so an agent with a small active cache can still hit the model's position limit. In the ablation (Table 3), adding repos to tool-response drop raised the BCP pass rate from 55% to 62%, which the authors attribute to letting runs continue past the position limit that would otherwise halt them.
The main policy tested is a tool-response rolling drop: keep KV for all reasoning and for the latest 12 tool responses, and drop older tool-response KV. The choice is grounded in an attention analysis showing that older tokens get less attention than recent ones, and that tool responses get less attention than assistant messages at similar positions. With repos enabled, dropping reasoning cache, alone or together with tool cache, scored lower on both BCP and SWE-Verified than dropping tool-response cache alone.
One subtlety the paper points out: a retained KV entry was computed while the dropped tokens were still visible, so it may still carry some information from the dropped history. That is different from summarization, which recomputes the kept text without it.
Why serving engines need changes
This breaks an assumption behind prefix caching in engines such as vLLM and SGLang: that the token prefix fully determines the cache. With drop, two requests with identical tokens can hold different KV states, depending on when a drop happened relative to later appends. Ordinary prefix matching could then reuse the wrong states.
AKV handles this with what the authors call "stateless interface, stateful serving." The client still sends the full message list on every request, plus an optional cache field listing which message indices to drop after which message, and where to reposition. The client does not need to track what is resident on the server. Inside, AKV uses:
- an extended radix tree with drop nodes and reposition nodes, so reuse matches on both tokens and operation history;
- drop-aware eviction that frees dropped cache once no request needs it, and prefers evicting dropped cache over cache from finished requests, which is more likely to be reused next round;
- segment-batch attention to rebuild evicted states in one prefill pass, splitting only attention at drop points while the FFN runs over the full sequence as one batch.
For background on the radix-tree reuse AKV extends, see our comparison of prompt and context caching, including RadixAttention KV reuse. For older, attention-score-based approaches to the same memory problem, see KV cache eviction and streaming attention.
The results
The setup compares three methods. Standard summarizes at a high threshold (96K tokens for GPT-OSS-120B and MiniMax-M2.7, 192K for MiroThinker-1.7-Mini). Text Engineering summarizes at 48K for all models. AKV uses the 12-tool-response rolling drop, with repos triggered at Standard's threshold. Benchmarks are BCP, xbench-DeepSearch-2510, a 103-task text-only GAIA validation subset, and SWE-bench Verified. Costs are API-equivalent estimates from Groq's GPT-OSS-120B rates and the official rates for the other models, and AKV's repos tokens are charged at the cache-hit rate as a deliberate overestimate.
Averages across the four benchmarks (Table 1):
- MiniMax-M2.7: AKV scored 53.7 at 4.4 cents per task, versus 52.4 at 5.0 cents for Standard and 52.8 at 5.7 cents for Text Engineering.
- GPT-OSS-120B: AKV scored 53.1 at 7.3 cents, versus 51.0 at 11.2 cents and 48.3 at 10.2 cents.
- MiroThinker-1.7-Mini: AKV scored 45.8 at 14.8 cents, versus 44.1 at 20.4 cents and 37.1 at 18.1 cents.
Averaged across all three models and four benchmarks, AKV cost 27.7% less than Standard and 21.9% less than Text Engineering. The paper's cost breakdown says the savings come from fewer cache reads (about 17% lower than Standard) and less re-prefill (cache-miss input costs about 48% lower than Text Engineering).
AKV did not win every cell. On MiniMax-M2.7, Text Engineering scored higher on BCP (62.3 versus 58.9) and Standard scored higher on SWE-Verified (75.2 versus 74.4). On GPT-OSS-120B, Standard edged it on xbench-DS (38.5 versus 37.5). On MiroThinker-1.7-Mini, Standard scored higher on SWE-Verified (7.8 versus 7.4). The authors say the accuracy gain shows up mainly past the baselines' summarization thresholds: on 100 BCP tasks with GPT-OSS-120B, all three methods were comparable below 48K logical tokens.
An appendix run on the smaller GPT-OSS-20B was less clear: AKV had the best average score (45.2 versus 44.0 for Standard) but cost 4.8 cents per task against 4.6 for Text Engineering. The authors attribute the weaker result to the smaller model being more sensitive to cache changes.
Serving speed
Serving tests ran GPT-OSS-120B on eight NVIDIA A800-80GB GPUs (four prefill, four decode, tensor parallelism of four, BF16 weights) on the top 80 SWE-Verified tasks. Comparing each method at its peak-throughput setting, AKV at 14 concurrent requests versus Standard at 12 gave 1.36x and 1.51x speedups in mean and P95 request latency, and 1.18x the output throughput (Table 2). The gap widens at high load: at 16 concurrent requests, Standard's P95 latency was 229.90 seconds against 30.63 for AKV. At the same batch size, the decode speedup comes mostly from attention: full-attention time per decode step fell 59.8% and total decode-step time fell 16.0%.
Agents writing their own cache policy
Two smaller experiments hint at where this could go. With cache management exposed to the agent as a tool, it decided when to cut and what to keep, and scored 61.0% on a 100-task BCP subset with GPT-OSS-120B (versus 52.1% for Standard, at 12.1 versus 13.4 cents) and 64.0% with MiniMax-M2.7 (versus 56.0%, though cost rose from 5.4 to 5.9 cents as it ran more turns). Separately, a recursive self-improvement loop with MiniMax-M2.7, refining policies on 20 BCP tasks and testing on 100 held-out ones, found policies that kept Standard's accuracy while cutting per-task cost by 15.6% and 17.8% relative to Standard.
What engineers should take from it
- This applies to models you serve yourself. The
cachefield is an AKV extension of the serving API, and the experiments use open-weight models on an SGLang-based engine. - If your harness summarizes at a threshold, the paper's numbers suggest checking how much of your bill is cache reads and re-prefill before tuning the summary prompt.
- A low summarization threshold is not a free saving. In these tests, Text Engineering at 48K had the lowest average score on GPT-OSS-120B and MiroThinker-1.7-Mini, and the highest average cost on MiniMax-M2.7.
- The authors list limits themselves: more cache policies to explore, training agents under this scheme, and serving support for linear-attention models are future work. The results come from one research group's benchmarks and cost model, not production traffic.
