Lost in the Middle: How Language Models Use Long Contexts
Authors: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang
Published July 6, 2023
Read the paper: arXiv 2307.03172 · PDF
Why it matters
A bigger context window tells you how much text a model will accept. It doesn't tell you how well the model will use all of it. The authors moved one relevant document around among distractors in multi-document question answering, and moved one key around in a key-value lookup task. Accuracy depended on position: it was often highest when the answer was near the beginning or the end, and it dropped significantly when the answer was in the middle.
That result is the evidence behind a lot of context engineering advice. If you stuff a prompt with every retrieved chunk, the one that matters can land in the weak spot. The paper also gives evaluation setups you can reuse to test long-context behavior yourself.
How to use it
Put the instruction and the most important material at the start or the end of the prompt, and keep supporting material in between. In RAG, send fewer, better chunks: use a reranker so the strongest passages go first, and drop the weak ones instead of padding the context. When you test a long-context feature, move the key fact to different positions and check that the answer stays right in each one.
Abstract
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.

