跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
The Transformer Block: Attention and Feed-Forward Processing

Queries, Keys, and Values: How One Position Gathers Information

1 / 3
Start with one position and follow its query. The query is compared with every key by a dot product, and those scores become weights that sum to one. The output is then a weighted blend of the value vectors, so the position keeps most of what it needs and only a little of what it does not. Notice that nothing in this comparison uses position numbers, so on its own attention cannot tell word order.
0:00 / 0:00

Self-attention lets each position in a sequence decide which other positions to draw information from. Every input vector \(x_i\) is projected by three learned weight matrices into a query \(q_i = W_Q x_i\), a key \(k_i = W_K x_i\), and a value \(v_i = W_V x_i\). The query of position \(i\) is compared with the key of every position \(j\) by a dot product, \(q_i \cdot k_j\), which measures how relevant position \(j\) is to position \(i\). Dividing by \(\sqrt{d_k}\), where \(d_k\) is the key dimension, keeps the scores from growing with dimension, and a softmax turns the scores into attention weights that are non-negative and sum to 1. The output for position \(i\) is the weighted sum \(\sum_j \alpha_{ij} v_j\): a blend of the value vectors, dominated by the positions with the largest weights. Because the weights are computed from content rather than fixed positions, the same mechanism can link a pronoun to its referent or a verb to its subject. A causal mask sets the weights for future positions to zero, so a position can only attend to itself and earlier positions, which is what makes next-token prediction well defined.

参考资料

  1. [1]Attention Is All You Needarxiv.org
上一页1 / 3下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion