Context window

The context window of a large language model (LLM) is the maximum number of tokens that can be used by the model at one time when generating an output. It determines how much information, such as text from a conversation or document, the model can take into account when producing a response. Although context windows have grown substantially as LLMs have developed, research has shown that models do not necessarily use all the information within a large context window equally effectively.

Definition

The context window limits the combined number of input, output and reasoning tokens in a single model request. Context windows are measured in tokens, which are units produced by the model's tokenizer rather than words or characters. In practical terms, the context window determines the information available to the model when producing a response, which can include previous messages in a conversation. Information outside the context window is not directly available to the model, but a process known as retrieval-augmented generation can retrieve information from external sources as additional context. A longer context window can allow a model to work with longer prompts, conversations, documents, codebases, or retrieved passages without first compressing or discarding as much information. It has been compared to working memory or attention span.

Increase in size and context rot

The practical size of context windows has increased rapidly as LLM systems have developed. Some models are limited by the sequence lengths used during training, while attention variants and positional-encoding methods can allow models to operate on longer sequences than those seen during training. Earlier large language models commonly used self-attention mechanisms whose computational requirements increased quadratically with sequence length, making longer context windows more expensive to process. By the mid-2020s, long-context systems had reported context windows ranging from hundreds of thousands to millions of tokens; Google researchers reported Gemini 1.5 evaluations on retrieval tasks at up to 10 million tokens.

Larger context windows tend to mean more coherent answers with fewer errors and hallucinations. However, as context windows grow, there are often diminishing returns as a larger context window does not necessarily mean that a model can use the entire context equally well, sometimes called context rot. In "Lost in the Middle", Liu et al. found that performance on long-context tasks was often worse when relevant information appeared in the middle of an input rather than near the beginning or end. Some studies indicate that the "effective context length" can be smaller than the advertised one, the former being the point in which the performance of the model starts to deteriorate when the answer isn't very easy to spot. Other benchmarks have assessed long-context capability using tasks that go beyond simple retrieval, including multi-document question answering, long-dialogue understanding, code repository understanding, and structured-data reasoning.

There are disadvantages of longer context windows including slower response times due to increased computational requirements, reduced explainability of model outputs and higher costs associated with processing larger numbers of tokens.