Once tokens are points in a continuous space, similarity becomes arithmetic. The dot product \(e_a \cdot e_b\) is large when two vectors point in the same direction, and the cosine similarity \[\cos(e_a, e_b) = \frac{e_a \cdot e_b}{\lVert e_a \rVert \, \lVert e_b \rVert}\] normalizes that value by the vector lengths, giving a number between \(-1\) and \(1\) that depends only on direction. Two words that appear in interchangeable contexts, such as two synonyms, tend to acquire rows with high cosine similarity, because the network is pushed to treat them the same way when predicting what comes next. This is also why directions can carry consistent relations: if the training text uses pairs like king and queen, man and woman in parallel, the difference vector between the two members of a pair often points in a similar direction across pairs.
There is a catch. The lookup is per token, so the same token always produces the same vector no matter where it appears. If the input were simply the sum or the set of embeddings, the sequences "dog bites man" and "man bites dog" would be indistinguishable, because addition is commutative and the lookup has no notion of position. Self-attention, which comes next, compares every position with every other position and is likewise order-agnostic on its own. Position information therefore has to be injected deliberately, for example by adding a position-dependent vector to each token embedding before the block stack. Without that injection, the model would see a bag of tokens rather than a sequence.