Inside a Tokenizer: Text to IDs and Back | Visual AI Learning Guide

Watch what happens as you edit the sentence. Each subword piece gets its own box, and under each box sits the integer ID that the model actually receives. Click a token and you can see its entry in the vocabulary — the ID is simply its position in that list. Now switch to the reverse direction: the IDs are looked up and joined back into text. Notice that the spacing does not always come back exactly as you typed it, because whitespace is usually glued to the following token. That detail matters later, but the key point here is that the mapping is fixed and works both ways.

0:00 / 0:00

Auto-play

A tokenizer is a fixed, deterministic mapping in both directions. Going forward, it segments the input text into subword pieces and replaces each piece with its integer ID from the vocabulary, producing a token ID sequence. Going backward, it looks up each ID and concatenates the pieces to reconstruct the text.

The vocabulary is the bridge between the two directions: an ordered list of every token the model knows, where the position in the list is the integer ID. Because the mapping is fixed, the same text always produces the same IDs, and the model's parameters never have to deal with raw characters.

One practical consequence is that the reconstruction is not always character-identical. Whitespace is often attached to the following token, and a piece like "ization" carries no leading space, so joining pieces back together requires knowing the tokenizer's spacing convention. The IDs are what the model consumes; the text is only what we read.