跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
The Transformer Block: Attention and Feed-Forward Processing

Why One Attention Pattern Is Not Enough

2 / 3
One attention pattern has to serve every purpose at once, which is a tight constraint. Multiple heads relax it: each head gets its own query, key, and value projections, so each can learn a different way of linking positions. Their outputs are concatenated and then mixed by an output projection, so the block still passes forward one vector per position. The practical effect is that a single layer can track several relationships at the same time instead of averaging them into one.
0:00 / 0:00

A single attention operation produces one weight distribution per position, which forces the model to compress every kind of relationship into one comparison. Multi-head attention instead runs several attention operations in parallel, each with its own learned projections \(W_Q^{(h)}\), \(W_K^{(h)}\), and \(W_V^{(h)}\). Head \(h\) computes its own weights and its own weighted sum, and the head outputs are concatenated and passed through an output projection \(W_O\). The projections are learned, so different heads can specialize: one may link a pronoun to its antecedent, another may track the previous word, another may connect a verb to a distant object. Because each head works in a smaller subspace of dimension \(d_k = d / H\), the total cost stays close to that of a single full-width head. The output projection mixes the heads back into one vector per position, so the rest of the block sees a single representation rather than a set of separate ones.

参考资料

  1. [1]Attention Is All You Needarxiv.org
上一页2 / 3下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion