Skip to content
Learn Motion
ExploreHow it worksMembership
Log in
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
Stacking Blocks into a Language Model

From the Last Hidden State to a Next-Token Distribution

2 / 2
Focus on the last position's vector. Because attention is causal, that vector has attended to every earlier position, so it is the only one that summarizes the whole context. The output head is a learned matrix with one row per vocabulary entry. Multiplying it by that vector gives one score per token, and those raw scores are the logits. Logits are unbounded, so they are not probabilities yet. Softmax exponentiates each logit and divides by the sum of all exponentials, which gives a positive number for every token and makes the whole vector sum to one. The largest logit still gets the largest probability, but the distribution keeps the full ranking and the relative confidence. One subtlety: every position produces a distribution over the next token, but only the last position's distribution is used to generate the next token.
0:00 / 0:00

Why the last position carries the prediction

The final block returns one vector per position. Because attention is causal, the vector at the last position has attended to every earlier position, so it is the only one that summarizes the full context. That vector \(h\) is what the output head consumes.

Projection to logits

The output head is a learned matrix \(W_{out}\) with one row per vocabulary entry and \(d\) columns. It maps the hidden vector to one score per token: \[ z = W_{out} h \] The resulting vector \(z\) has length \(|V|\), the vocabulary size. Each entry \(z_i\) is a logit, an unbounded real score for token \(i\).

Softmax turns logits into probabilities

Logits are not probabilities, so softmax normalizes them: \[ p_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)} \] Each \(p_i\) is positive and the probabilities sum to one. The largest logit yields the largest probability, and the gaps between logits control how confident the distribution is.

What the model predicts at each position

Every position produces a distribution over the next token, but only the last position's distribution is used to generate the next token. The earlier positions were trained to predict their own next tokens, and that training signal is what shapes the shared parameters.

Previous2 / 2Next

Learn Motion

Generate a course. Learn it properly.

Operated by Wuhan Daoyin Technology Co., Ltd.

Contact: [email protected]
Privacy PolicyTerms of Service

© 2026 Learn Motion