Skip to content
Learn Motion
ExploreHow it worksMembership
Log in
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
Embeddings: Turning Tokens into Vectors

Why Nearby Vectors Mean Similar Words, and Why Order Is Missing

2 / 2
Two embeddings are similar when they point in the same direction, and cosine similarity measures exactly that, ignoring how long the vectors are. Words used in interchangeable contexts get pushed toward the same direction, because the model is rewarded for treating them alike when it predicts the next token. That is also why a consistent offset can appear between paired words: the step from one member of a pair to the other often points the same way across different pairs. But notice what the lookup cannot do. The same token always returns the same row, wherever it sits. If you just added the embeddings together, dog bites man and man bites dog would give the same sum, because addition does not care about order. Attention has the same blind spot on its own. So position has to be added in on purpose, and that is what the next chapter builds on.
0:00 / 0:00

Once tokens are points in a continuous space, similarity becomes arithmetic. The dot product \(e_a \cdot e_b\) is large when two vectors point in the same direction, and the cosine similarity \[\cos(e_a, e_b) = \frac{e_a \cdot e_b}{\lVert e_a \rVert \, \lVert e_b \rVert}\] normalizes that value by the vector lengths, giving a number between \(-1\) and \(1\) that depends only on direction. Two words that appear in interchangeable contexts, such as two synonyms, tend to acquire rows with high cosine similarity, because the network is pushed to treat them the same way when predicting what comes next. This is also why directions can carry consistent relations: if the training text uses pairs like king and queen, man and woman in parallel, the difference vector between the two members of a pair often points in a similar direction across pairs.

There is a catch. The lookup is per token, so the same token always produces the same vector no matter where it appears. If the input were simply the sum or the set of embeddings, the sequences "dog bites man" and "man bites dog" would be indistinguishable, because addition is commutative and the lookup has no notion of position. Self-attention, which comes next, compares every position with every other position and is likewise order-agnostic on its own. Position information therefore has to be injected deliberately, for example by adding a position-dependent vector to each token embedding before the block stack. Without that injection, the model would see a bag of tokens rather than a sequence.

Previous2 / 2Next

Learn Motion

Generate a course. Learn it properly.

Operated by Wuhan Daoyin Technology Co., Ltd.

Contact: [email protected]
Privacy PolicyTerms of Service

© 2026 Learn Motion