跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
Embeddings: Turning Tokens into Vectors

From a Token ID to a Row of Numbers

1 / 2
Think of the embedding matrix as a spreadsheet with one row per vocabulary entry and one column per dimension. The token ID is not a quantity to compute with; it is a row number. When the model sees ID four thousand eight hundred twenty-one, it walks to row four thousand eight hundred twenty-one and copies out that entire row of numbers. That copied row is the embedding. So a sentence of ten tokens becomes ten vectors, and a batch of sentences becomes a block of vectors. The numbers in the row start out random, and training nudges them, exactly the way it nudges every other weight in the network. Nobody writes down that column three means animal or column seven means plural. The model only discovers directions that make its later predictions less wrong.
0:00 / 0:00

The embedding matrix \(E\) is a learned table with one row per vocabulary entry and one column per embedding dimension. If the vocabulary has \(V\) entries and the embedding dimension is \(d\), then \(E\) has shape \(V \times d\). Turning a token ID into a vector is a lookup: the ID is a row index, and the vector is that row.

For a token whose ID is \(i\), the embedding is
\[e_i = E[i]\]
where \(E[i]\) denotes row \(i\) of the matrix. Nothing is multiplied or summed during this step; the ID simply selects which row to read. A sequence of token IDs therefore becomes a sequence of vectors, and a batch of sequences becomes a three-dimensional tensor of shape (batch size, sequence length, \(d\)).

The entries of \(E\) start as small random numbers and are adjusted during training by the same gradient-based procedure that adjusts every other parameter. No one assigns meaning to an individual column. What the network learns is a set of directions such that tokens used in similar contexts end up with rows that behave similarly in later computation. The dimension \(d\) is a design choice, commonly in the hundreds or thousands; a larger \(d\) gives the model more room to represent distinctions but costs memory and compute, since \(E\) alone contains \(V \times d\) parameters.

上一页1 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion