The Context Window and the Key-Value Cache | AI Learning Guide

The cache is worth dwelling on, because it explains a lot of everyday behavior. Attention needs the keys and values of every earlier token. Without a cache you would rebuild all of them at each step, so the work would grow with the length of the text already produced. With the cache, you keep them and compute only the new token's contribution. That is why the first token of a response often takes noticeably longer than the rest: the prompt is processed in one pass, and then each following token is cheap. It also explains why very long contexts cost memory, since the cache has to hold keys and values for every token still inside the window.

0:00 / 0:00

Auto-play

The context window is a hard budget

The context window is the maximum number of tokens the model can attend to in a single forward pass. It is a fixed property of the trained model, set by the architecture and the position information it was trained with. Everything the model can see must fit inside it: the system prompt, the conversation history, any retrieved or pasted text, and the tokens generated so far. Once the budget is full, something must be dropped, summarized, or truncated, and whatever is dropped is genuinely invisible to the model. This is why a long conversation can appear to forget an early instruction: the instruction may no longer be inside the window.

Why caching matters

Recall that attention computes a query, key, and value for each position, and each new position attends to the keys and values of all earlier positions. If those were recomputed from scratch at every generation step, step \(n\) would cost work proportional to \(n\), and generating a response of length \(N\) would cost work proportional to \(N^2\). The key-value cache removes that waste: after each step, the keys and values of the processed tokens are stored, so the next step only computes the query, key, and value for the single new token and attends against the stored ones. Each step then costs roughly a constant amount of attention work instead of growing with the prefix length. The cache is why streaming output feels fast and why memory, not just compute, becomes a limiting resource for long contexts: the cache grows with the number of tokens held in the window.

What this means for latency and cost

Two different costs are at work. Prefill processes the prompt in parallel and is fast per token but scales with prompt length; decoding produces one token per step and is inherently sequential, so output length drives latency. Longer prompts and longer outputs both raise cost, and a longer context also enlarges the key-value cache. This is the practical reason a shorter, well-targeted prompt often beats pasting everything into the window.