Predicting the Next Token Is Its Own Supervision | AI Learning Guide

The training task needs no human labels. At each position the model predicts a distribution over the next token, and the correct answer is just the token that actually comes next in the text. That is why we call it self-supervised: the text supervises itself. The error is measured by cross-entropy, which is minus the log of the probability the model gave to the correct token. If the model is confident and right, that number is near zero; if it is wrong or unsure, the number is large. Because every position is a training example, a long document creates many prediction problems at once, and the amount of supervision grows with the amount of text.

0:00 / 0:00

Auto-play

Next-token prediction is self-supervised: the label at each position is the token that actually follows in the text, so raw text supplies both input and target.

Cross-entropy loss

For a single position where the correct next token is \(y\) and the model assigns it probability \(p_y\), the loss is \[-\log p_y.\] Averaged over all positions in a batch, this is the cross-entropy loss. It is small when the model puts high probability on the correct token and large when it does not. Minimizing it is the same as maximizing the probability the model assigns to the observed text.

Every position in a document is a training example, so a corpus of length \(n\) provides roughly \(n\) prediction problems with labels already present. The supervision signal grows with the data, which is what makes training on very large text collections feasible.