Subscribe
4 min read

Million-Token Windows Did Not Kill RAG. The Price List Explains Why

Bigger context windows keep arriving, and so do the claims that retrieval is obsolete. Vendor rate cards and head-to-head benchmarks tell a duller story: long context is a premium tier, and RAG remains the default.

Retro Olivetti brochure poster: an overstuffed shelf spilling papers beside a tidy filing cabinet with a price tag, headline Million-Token Windows Did Not Kill RAG
Retro Olivetti brochure poster: an overstuffed shelf spilling papers beside a tidy filing cabinet with a price tag, headline Million-Token Windows Did Not Kill RAG

Every time a lab ships a bigger context window, the same obituary gets recycled: RAG is dead, just paste the corpus into the prompt. GPT-6 Astra takes 922,000 input tokens against a 1,050,000-token window, according to OpenAI's model documentation. Gemini and Claude sit in the same range. The windows are real. The obituaries are still wrong, and the reason is visible in two places: the rate cards and the benchmarks.

What a full-window request actually costs

Vendor pricing quietly penalizes exactly the behavior the big windows invite.

OpenAI's published rate card for GPT-6 Astra: $10 per million input tokens, $50 per million output. But requests above 272,000 input tokens are billed at 2x the input and cache rates and 1.5x the output rate, for the entire request, not just the excess. A single 300,000-token prompt therefore pays roughly $6 in input alone, by the model page's own numbers. The same task run over 10 retrieved chunks of a few hundred tokens costs about a dime at the published rate. Caching narrows the gap for repeated static context: GPT-6 Astra's cached input is $1 per million against $10 uncached, and Anthropic has said prompt caching cuts repeat-prefix costs by up to 90%. But caching rewards stable prefixes, not a corpus that changes between requests.

A stack of papers at a teller window feeding a register that prints an absurdly long receipt (Olivetti brochure style illustration)

The second tax is attention, not money. Anthropic's engineering guidance frames context as a finite resource with diminishing returns: "context rot" means recall degrades as the window fills, and its cookbook notes that prefill latency and rot scale with how much is in the window, not with the nominal limit. A model advertising 1M tokens does not use 1M tokens equally well; independent long-context evals (we covered the benchmark side in our explainer on context windows) keep finding that effective context is shorter than advertised context.

The research gave long context the win, with an asterisk

To be fair to the obituary writers: the strongest head-to-head study mostly agrees on quality. Google researchers (Li et al., arXiv 2407.16833) benchmarked retrieval-augmented generation against full long-context ("LC") prompting across multiple public datasets and three frontier models. Their finding: when resources are no constraint, LC consistently outperforms RAG on average, with Gemini-1.5-Pro and GPT-4O scoring 7.6% and 13.1% higher than their RAG counterparts. The asterisk is the cost column: RAG achieved its results using about 17% of LC's compute. The same paper proposed Self-Route, which lets the model decide per query whether retrieved snippets suffice or the full context is needed, and reports near-LC quality at a fraction of the cost.

Follow-up work refused to generalize the win. LaRA, an ICML 2025 benchmark from Alibaba NLP with 2,326 test cases across four QA task types and three long-context settings, tested eleven models and concluded there is "no silver bullet": whether retrieval or full context wins depends on model size, task type, context length and retrieval quality, with actionable guidelines rather than a verdict. Smaller open models in particular did better with retrieved snippets than with the same material stuffed into the window.

Retrieval spent the same few years getting better

The comparison people keep re-running is stale on both sides. While windows grew, the retrieval step stopped being the weak link it was in 2023. Anthropic's Contextual Retrieval method, which prepends document-level context to each chunk before embedding, cut failed retrievals by 49%, and by 67% when combined with reranking, in the company's tests. The preprocessing itself costs about $1.02 per million document tokens with prompt caching doing the heavy lifting.

Anthropic's own guidance is worth quoting because it is the most generous version of the long-context argument and still ends in a boundary condition: if your knowledge base is smaller than roughly 200,000 tokens (about 500 pages of material), skip RAG and put the whole thing in the cached prompt. That is a real recommendation, and it is also an admission that above that size the trade flips back to retrieval.

The decision, as it actually looks in late 2026

A mechanical switch routing paper cards either to a tidy index cabinet or to a window stuffed with loose piles (Olivetti brochure style illustration)
  • Whole-window prompting: one document or a corpus that fits under the effective (not advertised) window, tasks that genuinely need synthesis across everything, and prototypes where infrastructure time costs more than tokens. Budget for the long-context rate tier if you cross vendors' thresholds.
  • Retrieval: corpora that outgrow the window, sources that change between requests, anything needing an audit trail from answer to document, and high-volume production where the 17%-of-compute finding is the whole ballgame. Pair a decent embedder with reranking; retrieval quality, not window size, is the variable that decides head-to-head results in LaRA's analysis.
  • Routing/hybrid: the default for teams that have looked at their own evals. Cheap retrieve-first, escalate to full document when snippets disagree, is what Self-Route formalized and what most agent harnesses now do ad hoc.

The honest summary is not that RAG won or lost. It is that window size became a pricing and latency decision rather than a capability one, and pricing decisions are the least glamorous way to kill a feature. Labs keep doubling the advertised window because it sells; rate cards above 272K tokens are the tell that nobody expects you to live there.

Sources