Subscribe
Explainers
7 min read

Lost in the Middle: Why AI Misses Facts in Long Prompts

AI models use facts at the start and end of long prompts well and miss facts in the middle. What the Lost in the Middle study found and how to write around it.

Olivetti-style cover reading Lost in the Middle: Why AI Misses Facts in Long Prompts, with a row of document blocks and a faded middle block on cream paper
Olivetti-style cover reading Lost in the Middle: Why AI Misses Facts in Long Prompts, with a row of document blocks and a faded middle block on cream paper

If you paste a long report into a chatbot and ask about one detail on page 12, the answer can come back wrong even though the fact was right there. A 2023 study led by Stanford researchers gave this a name: "lost in the middle." Language models tend to use information at the start and end of a long prompt well, and information in the middle less well.

This explainer covers what the researchers tested, what they found, why it seems to happen, and what you can do about it when you build prompts, chat with long documents, or run retrieval-augmented generation (RAG).

What is the lost in the middle problem?

The lost in the middle problem is a position effect. When a model gets a long input and the one fact it needs sits in the middle of that input, its accuracy drops compared with when the same fact sits near the beginning or the end.

The name comes from the paper Lost in the Middle: How Language Models Use Long Contexts by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, from Stanford University, UC Berkeley and Samaya AI. It was posted to arXiv in 2023 and published in the journal TACL in 2024.

The authors plot accuracy against the position of the answer and get a U-shaped curve: high at the start, high at the end, low in the middle. They describe the two high ends as primacy bias (favoring the beginning) and recency bias (favoring the end).

What did the researchers test?

The team used two tasks that let them control exactly where the key information sits.

  • Multi-document question answering. The model gets a question and a set of Wikipedia passages. Exactly one passage contains the answer. The others are distractors: passages that a retrieval system (Contriever) judged relevant to the question but that do not contain the answer. Questions come from the NaturalQuestions-Open dataset. Inputs had 10, 20 or 30 documents, and the researchers moved the answer passage to different positions.
  • Key-value retrieval. The model gets a JSON object full of random key-value pairs (each key and value is a random UUID) and must return the value for one given key. This is a minimal test of whether a model can find matching text in its input at all.

They tested open models (MPT-30B-Instruct with an 8,192-token context, and LongChat-13B (16K)) and closed models through their APIs (GPT-3.5-Turbo with 4K tokens, GPT-3.5-Turbo (16K), Claude-1.3 with 8K tokens, and Claude-1.3 (100K)). They also ran GPT-4 (8K) on a subset of the question answering experiments. All runs used greedy decoding.

What did they find?

The main result is the U-shape. Across models, accuracy was often highest when the answer passage was first or last, and it fell when the answer was in the middle. This held even for the models built for long contexts.

The drop can be large. In Section 2.3 of the paper, GPT-3.5-Turbo's multi-document accuracy drops by more than 20% when the answer moves to the middle. In the worst case, in the 20-document and 30-document settings, its accuracy with the answer in the middle was lower than its closed-book accuracy, where it answers from memory with no documents at all.

Table 1 of the paper gives those two reference points for each of the six model versions. Closed-book means no documents. Oracle means only the single passage that holds the answer.

  • LongChat-13B (16K): 35.0% closed-book, 83.4% oracle
  • MPT-30B-Instruct: 31.5% closed-book, 81.9% oracle
  • GPT-3.5-Turbo: 56.1% closed-book, 88.3% oracle
  • GPT-3.5-Turbo (16K): 56.0% closed-book, 88.6% oracle
  • Claude-1.3: 48.3% closed-book, 76.1% oracle
  • Claude-1.3 (100K): 48.2% closed-book, 76.4% oracle

So for GPT-3.5-Turbo, adding 19 or 29 distractor passages around the right answer, and putting that answer in the middle, could leave it worse off than giving it nothing to read.

Schematic U-shaped curve from Liu et al. (2023), not measured data: accuracy is higher when the answer is at the start or end of the prompt and lower in the middle, labeled primacy bias and recency bias

Two more findings matter for anyone choosing a model:

  • Longer context windows did not mean better use of context. The extended-context versions (GPT-3.5-Turbo (16K), Claude-1.3 (100K)) often scored about the same as their shorter siblings when the input fit both. A bigger window lets you send more text. It does not by itself make the model read that text better.
  • The key-value task was not solved by everyone. Claude-1.3 and Claude-1.3 (100K) did nearly perfectly on it at every input length. GPT-3.5-Turbo, GPT-3.5-Turbo (16K) and MPT-30B-Instruct again did worst when the key was in the middle. A task that only requires finding an exact match still showed the position effect for those models.

Why does it happen?

The Liu et al. paper does not give one final cause, but it tests several ideas.

  • Model design. Encoder-decoder models (Flan-T5-XXL and Flan-UL2) were relatively robust to the position of the answer, but only on inputs shorter than the sequence length they were trained on. On longer inputs, they showed the same U-shape.
  • Where the question sits. Putting the question both before and after the documents, which the authors call query-aware contextualization, gave near-perfect results on key-value retrieval. On multi-document question answering, it changed the trend only a little.
  • Instruction tuning. Instruction-tuning data often puts the task at the start of the input, so the team checked whether that taught models to favor the beginning. The base model MPT-30B, before instruction tuning, also showed the U-shape. Instruction tuning alone does not explain it.

The authors compare the pattern to the serial-position effect in human memory: people asked to recall a list tend to remember the first and last items best. They call it surprising in a Transformer, because self-attention can in principle reach any token in the input equally.

A 2024 follow-up, Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization by Cheng-Yu Hsieh and colleagues at the University of Washington, MIT and Google Cloud AI Research, links the effect to attention itself. They report that models show a U-shaped attention bias: tokens at the beginning and end of the input get more attention regardless of whether they are relevant. Models do attend to a relevant passage in the middle, but leading and ending content distracts them. Their calibration method, also called found-in-the-middle, corrects for that bias. They report better RAG performance across tasks, beating existing methods by up to 15 percentage points.

If you want background on how models store and reuse the text they have already read, see our explainer on the KV cache.

Does a bigger context window fix it?

Not on its own. The Liu et al. results already showed that the 16K and 100K versions did not beat their shorter versions at using the same input.

A separate 2024 NVIDIA paper, Ruler: What's the Real Context Size of Your Long-Context Language Models? by Cheng-Ping Hsieh, Simeng Sun and colleagues, points the same way. The team tested 17 long-context models on 13 tasks. Models scored nearly perfectly on the simple needle-in-a-haystack test, which hides one fact in filler text. But almost all of them dropped sharply as the input grew. Every model claimed a context size of 32K tokens or more, and only half kept satisfactory performance at 32K.

The lesson from both papers: the advertised context length tells you how much text a model accepts, not how well it uses that text. We make a related argument in a bigger context window is not a better memory.

What this means for RAG and long documents

RAG systems search a document store and paste the top results into the prompt. That is close to the multi-document setup in the paper, so the findings apply directly.

The paper ran a case study with a standard retriever and reader on NaturalQuestions-Open. As they retrieved more passages, the share of questions with the answer somewhere in the passages (retriever recall) kept rising. The reader's accuracy flattened long before that. Going from 20 retrieved documents to 50 improved accuracy by only about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3.

The authors frame long prompts as a trade-off. More context may contain the answer, but it also adds more text the model must reason over, and that can lower accuracy. This is one reason retrieval does not remove wrong answers, a point we cover in RAG does not end hallucinations.

Practical tips for long prompts

These follow from the research above. None is a guarantee, so measure them on your own task.

  • Put what matters at the edges. Place the key instructions, the most relevant document and the question near the start or the end of the prompt, not buried in the middle.
  • Rerank before you paste. If you use retrieval, order the passages so the most relevant ones come first, and keep weak matches out.
  • Do not pad. More passages raised recall but barely moved accuracy in the paper's case study. Fewer, better passages can be the cheaper and safer choice.
  • Repeat the question after long documents. Asking before and after the documents fixed key-value lookups in the paper. It helped question answering only a little, so treat it as a low-cost habit, not a cure.
  • Test position on purpose. The authors recommend checking long-context models with the relevant information at different positions. Do the same with your own prompts: move the key fact to the start, middle and end and compare the answers.
Five tips for long prompts: put key facts and the question at the start or end, rerank retrieved passages, do not pad the prompt, repeat the question after the documents, test position yourself

What this research does not show

The Liu et al. experiments used models from 2023: GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct and LongChat-13B, plus a GPT-4 subset. The paper does not test the models people use today, so it cannot tell you how strong the effect is in any current model. Ruler and Found in the Middle are from 2024 and also use the models of their time.

What the work does give you is a test method and a known failure mode. The safe assumption is that position can matter, and the safe practice is to check it on your model, with your data, before you trust a long prompt.

Sources

ExplainersResearchLLM