One Token at a Time | Visual AI Learning Guide

Here the loop runs on its own. On the left, the sequence starts as the prompt's token IDs, and on the right the model returns a probability for every token in the vocabulary. Watch one bar light up, get chosen, and slide onto the end of the sequence; that longer sequence is then fed straight back in as the next input. Notice that only one position is ever predicted at a time, never the whole sentence at once. And because each new distribution is conditioned on everything already chosen, an early pick stays in the context and keeps shaping later steps, until the end-of-sequence token or the length limit finally stops the cycle.

0:00 / 0:00

Auto-play

A language model does not emit a sentence. At each step it receives a sequence of token IDs and returns a probability distribution over the entire vocabulary for the single next position. Generation is the loop that repeatedly applies this function.

Start with the prompt tokenized into IDs, for example \([t_1, t_2, \dots, t_n]\). The model produces a distribution \(P(t_{n+1} \mid t_1, \dots, t_n)\), a decoding rule selects one token from it, and that token is appended. The new sequence \([t_1, \dots, t_n, t_{n+1}]\) becomes the input for the next call, which yields \(P(t_{n+2} \mid t_1, \dots, t_{n+1})\). The loop continues until a special end-of-sequence token is chosen or a length limit is reached.

Two consequences follow. First, the model has no separate plan it executes; the entire response is built from local next-token decisions, and each decision conditions on everything generated so far. Second, an error or an unusual choice early in the sequence becomes part of the context for every later step, which is why generation can drift.

The loop is also why the same prompt can yield different outputs: the distribution is fixed given the input, but the token actually chosen depends on the decoding rule applied to it.