Self-attention lets each position in a sequence decide which other positions to draw information from. Every input vector \(x_i\) is projected by three learned weight matrices into a query \(q_i = W_Q x_i\), a key \(k_i = W_K x_i\), and a value \(v_i = W_V x_i\). The query of position \(i\) is compared with the key of every position \(j\) by a dot product, \(q_i \cdot k_j\), which measures how relevant position \(j\) is to position \(i\). Dividing by \(\sqrt{d_k}\), where \(d_k\) is the key dimension, keeps the scores from growing with dimension, and a softmax turns the scores into attention weights that are non-negative and sum to 1. The output for position \(i\) is the weighted sum \(\sum_j \alpha_{ij} v_j\): a blend of the value vectors, dominated by the positions with the largest weights. Because the weights are computed from content rather than fixed positions, the same mechanism can link a pronoun to its referent or a verb to its subject. A causal mask sets the weights for future positions to zero, so a position can only attend to itself and earlier positions, which is what makes next-token prediction well defined.
Learn MotionHow Large Language Models Work: From Tokens to Inference
The Transformer Block: Attention and Feed-Forward Processing
Queries, Keys, and Values: How One Position Gathers Information
1 / 3
Start with one position and follow its query. The query is compared with every key by a dot product, and those scores become weights that sum to one. The output is then a weighted blend of the value vectors, so the position keeps most of what it needs and only a little of what it does not. Notice that nothing in this comparison uses position numbers, so on its own attention cannot tell word order.
0:00 / 0:00