From a Token ID to a Row of Numbers | Visual AI Learning Guide

Think of the embedding matrix as a spreadsheet with one row per vocabulary entry and one column per dimension. The token ID is not a quantity to compute with; it is a row number. When the model sees ID four thousand eight hundred twenty-one, it walks to row four thousand eight hundred twenty-one and copies out that entire row of numbers. That copied row is the embedding. So a sentence of ten tokens becomes ten vectors, and a batch of sentences becomes a block of vectors. The numbers in the row start out random, and training nudges them, exactly the way it nudges every other weight in the network. Nobody writes down that column three means animal or column seven means plural. The model only discovers directions that make its later predictions less wrong.

0:00 / 0:00

Auto-play

The embedding matrix \(E\) is a learned table with one row per vocabulary entry and one column per embedding dimension. If the vocabulary has \(V\) entries and the embedding dimension is \(d\), then \(E\) has shape \(V \times d\). Turning a token ID into a vector is a lookup: the ID is a row index, and the vector is that row.

For a token whose ID is \(i\), the embedding is \[e_i = E[i]\] where \(E[i]\) denotes row \(i\) of the matrix. Nothing is multiplied or summed during this step; the ID simply selects which row to read. A sequence of token IDs therefore becomes a sequence of vectors, and a batch of sequences becomes a three-dimensional tensor of shape (batch size, sequence length, \(d\)).

The entries of \(E\) start as small random numbers and are adjusted during training by the same gradient-based procedure that adjusts every other parameter. No one assigns meaning to an individual column. What the network learns is a set of directions such that tokens used in similar contexts end up with rows that behave similarly in later computation. The dimension \(d\) is a design choice, commonly in the hundreds or thousands; a larger \(d\) gives the model more room to represent distinctions but costs memory and compute, since \(E\) alone contains \(V \times d\) parameters.