Context window

Common card · No. 334486

The context window of a large language model (LLM) is the maximum number of tokens that can be used by the model at one time when generating an output. It determines how much information, such as text from a conversation or document, the model can take into account when producing a response. A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the effective context length that is of interest, i.e. the length of the context that a model can use effectively. Information outside the context window is not directly available to the model, but retrieval-augmented generation can retrieve information from external sources to put into the context window. A longer context window can allow a model to work with longer prompts, conversations, documents, codebases, or retrieved passages without first compressing or discarding as much information. It has been compared to working memory or attention span.

Read on Wikipedia