A single attention operation produces one weight distribution per position, which forces the model to compress every kind of relationship into one comparison. Multi-head attention instead runs several attention operations in parallel, each with its own learned projections \(W_Q^{(h)}\), \(W_K^{(h)}\), and \(W_V^{(h)}\). Head \(h\) computes its own weights and its own weighted sum, and the head outputs are concatenated and passed through an output projection \(W_O\). The projections are learned, so different heads can specialize: one may link a pronoun to its antecedent, another may track the previous word, another may connect a verb to a distant object. Because each head works in a smaller subspace of dimension \(d_k = d / H\), the total cost stays close to that of a single full-width head. The output projection mixes the heads back into one vector per position, so the rest of the block sees a single representation rather than a set of separate ones.
Learn MotionHow Large Language Models Work: From Tokens to Inference
The Transformer Block: Attention and Feed-Forward Processing
Why One Attention Pattern Is Not Enough
2 / 3
One attention pattern has to serve every purpose at once, which is a tight constraint. Multiple heads relax it: each head gets its own query, key, and value projections, so each can learn a different way of linking positions. Their outputs are concatenated and then mixed by an output projection, so the block still passes forward one vector per position. The practical effect is that a single layer can track several relationships at the same time instead of averaging them into one.
0:00 / 0:00