Why the last position carries the prediction
The final block returns one vector per position. Because attention is causal, the vector at the last position has attended to every earlier position, so it is the only one that summarizes the full context. That vector \(h\) is what the output head consumes.
Projection to logits
The output head is a learned matrix \(W_{out}\) with one row per vocabulary entry and \(d\) columns. It maps the hidden vector to one score per token: \[ z = W_{out} h \] The resulting vector \(z\) has length \(|V|\), the vocabulary size. Each entry \(z_i\) is a logit, an unbounded real score for token \(i\).
Softmax turns logits into probabilities
Logits are not probabilities, so softmax normalizes them: \[ p_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)} \] Each \(p_i\) is positive and the probabilities sum to one. The largest logit yields the largest probability, and the gaps between logits control how confident the distribution is.
What the model predicts at each position
Every position produces a distribution over the next token, but only the last position's distribution is used to generate the next token. The earlier positions were trained to predict their own next tokens, and that training signal is what shapes the shared parameters.