Every conversation with a chatbot, every document you paste into an AI tool, and every line of code an agent reads fits inside one fixed budget. That budget is the context window, and it is the single most misunderstood number on any model's spec sheet. This explainer covers what the window actually is, why it grew from 2,000 tokens to a million, and why a bigger window is not the same as a better memory.
What the context window is
A large language model reads and writes text as tokens, and it can only "see" the tokens that fit inside a fixed-size input buffer at the moment it generates its next word. That buffer is the context window. It holds the system instructions, your prompt, every previous turn of the conversation, the documents or files you attached, the output of any tools the model called, and the answer being generated. When the buffer fills, something has to fall off the front or be thrown away.
The window is not storage. The model does not retain anything outside it. Ask a model about a sentence you deleted from the chat and it genuinely cannot see it, the way you cannot re-read a page you burned. What people call the model's "memory" in a product like ChatGPT or Claude is either a window being re-filled with summaries, or a database outside the model that gets queried and pasted back in.
GPT-3, the 2020 paper by Brown et al. that set the template for modern LLMs, worked with a window of 2,048 tokens: roughly 1,500 words. A single dense page of text was most of the budget. By late 2026, hosted frontier models ship windows of about one million tokens, and Meta's open-weights Llama 4 Scout advertised 10 million when it launched in April 2025.
Why the window has a size limit at all
The limit is not an artificial paywall. It comes from how the transformer architecture, the mechanism behind every modern LLM, computes attention.
In the original 2017 transformer paper, Vaswani et al. noted that self-attention compares every token against every other token, so the compute and memory it requires grow with the square of the input length. Double the window, and the raw attention cost goes up four times. Optimized implementations like FlashAttention make the quadratic term survivable, but they do not delete it.
Serving makes the economics harsher still. During generation, the model caches the keys and values of every token it has already processed, the KV cache, so it does not recompute them each step. That cache grows linearly with sequence length and lives in fast GPU memory, which is the scarcest resource in any inference datacenter. A million-token request is not twice as expensive as a 500,000-token one; it ties up vastly more memory while every other user waits.
Position handling adds a third constraint. Modern models encode token positions with schemes like rotary position embeddings that behave reliably only over lengths similar to what the model trained on. Stretching a window past that range takes deliberate work, and models generally handle positions they were never trained on poorly.
The growth race, briefly
- 2020: GPT-3 ships 2,048 tokens.
- May 2023: Anthropic expands Claude from 9K to a 100K-token window, about 75,000 words, and demonstrates the model finding a single altered line in the full text of The Great Gatsby.
- November 2023: Claude 2.1 doubles the ceiling to 200K; GPT-4 Turbo settles at 128K.
- February 2024: Google's Gemini 1.5 Pro brings a 1-million-token window, with 2 million in experimental preview.
- April 2025: Meta's Llama 4 Scout claims 10 million tokens for open-weights, self-hosted use.
- Late 2026: The hosted frontier has largely stopped growing. Across current Claude, GPT and Gemini models the standard window is about one million tokens (tracked with primary sources here). The doubling era ended; benchmark scores did not keep pace with the spec sheets, and vendors appear to have noticed.

Why a bigger window is not a better memory
Two lines of research say the advertised number is an upper bound on what a model can hold, not on what it can use.
Position matters. Liu et al.'s Lost in the Middle, published in TACL 2024, moved the answer document to different positions in multi-document questions. Performance traced a U: models use information at the very start (primacy) and very end (recency) of a context far better than information in the middle. For GPT-3.5-Turbo with 20 documents, accuracy was 75.8% when the answer came first and 53.8% when it sat tenth, below the model's 56.1% score with no documents at all. Explicitly long-context models, GPT-4 included, showed the same curve, just higher up.
Length itself degrades quality. Chroma's Context Rot technical report (July 2025) held task difficulty constant and varied only input length, across 18 frontier models including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. Every model got worse as input grew, even on copy-and-retrieve tasks that should be trivial. The degradation was non-uniform: extra distractor passages hurt some models far more than others. A related concern, quantified by NVIDIA's RULER benchmark, is that many models' effective context is well below their advertised maximum, with retrieval reliability falling as tasks require genuinely using information across the whole window.
The practical reading: a 1M-token window filled with 1M tokens of meeting notes is a worse place to find an answer than a 10K-token window containing only the five paragraphs that matter.
What eats your window
In real use, the budget drains faster than the raw number suggests:
- System prompts and tool definitions. Agents like Claude Code ship tens of thousands of tokens of instructions and tool schemas before you type a word.
- Conversation history. In the naive case, every turn is re-sent in full, so cost and window use grow quadratically across a long chat.
- Attachments and tool outputs. One scraped web page or build log can dwarf your own question.
- Reasoning traces. Thinking models spend window tokens, and sometimes billed tokens, on internal scratchpad text you never see.
- Tokenization overhead. Prose runs roughly 0.75 words per token, but code, symbols and non-English text consume far more tokens per unit of meaning.
When the budget runs out, products handle it one of three ways: truncate the oldest turns, summarize them to buy space (Anthropic's consumer apps call this automatic compaction), or refuse the request. Summarization is lossy by construction, which is why long agent sessions "forget" decisions made early on unless the agent writes notes to disk outside the window.

Why it matters
The context window is the interface between a stateless statistical model and anything that looks like memory. Nearly every trick in applied LLM work is really a strategy for the window: retrieval-augmented generation fetches only the relevant paragraphs instead of pasting the library; caching systems reuse a prefix so long prompts bill cheaper; agent frameworks compact history and push durable state into files and databases.
The honest summary for late 2026: windows are large enough that fitting the material is rarely the problem anymore, and small-minded enough that attention to it remains one. The spec sheet number is a capacity limit. Whether a model uses the capacity is a different, measured, and frequently disappointing question.
Sources
- Vaswani et al., Attention Is All You Need (2017): https://arxiv.org/abs/1706.03762
- Brown et al., Language Models are Few-Shot Learners (2020): https://arxiv.org/abs/2005.14165
- Anthropic, Introducing 100K Context Windows (2023): https://www.anthropic.com/news/100k-context-windows
- Google, Our next-generation model: Gemini 1.5 (2024): https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
- Meta, Llama 4: https://ai.meta.com/llama/
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024: https://arxiv.org/abs/2307.03172
- Hong, Troynikov, Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma technical report (2025): https://www.trychroma.com/research/context-rot
- Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (2024): https://arxiv.org/abs/2404.11757
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023): https://arxiv.org/abs/2305.13245
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021): https://arxiv.org/abs/2104.09864
- Hidekazu Konishi, LLM Context Window Growth Timeline (2026): https://hidekazu-konishi.com/entry/llm_context_window_growth_timeline.html
