How does an AI agent remember, and look things up before it answers?

A model remembers only what is inside its context window, and for anything beyond that it needs a way to look things up. The context window is the maximum number of tokens the model can use at one time when generating an output. In practical terms it is the material the model can see while answering; anything outside the window is not directly available unless it is summarized, retrieved or provided again.

The windows have grown. The small GPT-2 model had a context window of only about 1,000 tokens, while by the mid-2020s long-context systems reported windows of hundreds of thousands to millions of tokens. But bigger is not simply better. A paper called Lost in the Middle found that performance is often highest when the relevant information is at the beginning or end of the input, and degrades significantly when it sits in the middle, even for long-context models. Longer windows also bring slower responses and higher costs.

So agents also look things up. Retrieval-augmented generation, or RAG, lets a model retrieve and use new information from external data sources: it first refers to a specified set of documents, then answers. Typically the documents are turned into embeddings, lists of numbers, and stored in a vector database; given a question, a retriever selects the most relevant ones to add to the prompt.

RAG is no cure-all. It lets users check the cited sources, but it does not prevent hallucinations.