Why Text Must Be Cut into Units | AI Learning Guide

The worked split is the clearest way to see the trade-off in action. At the word level, "unbelievably" needs its own vocabulary entry, and any new word or typo would be missing entirely. At the character level it becomes twelve separate units, so the model has to carry meaning across a much longer sequence. The subword split lands in between: un, believ, and ably are pieces that show up again and again across English, so a small vocabulary can still cover almost anything you throw at it. That is the whole reason subword tokenization won out.

0:00 / 0:00

Auto-play

A language model can only work with a finite set of discrete units, because its input and output layers are indexed by integers. Choosing those units is a trade-off between how large the vocabulary is and how long the sequences become.

The two extremes

Character-level

Vocabulary of a few hundred symbols
A 40-word sentence becomes ~200 units
No unknown symbols, but long sequences to track

Word-level

Hundreds of thousands of word forms
Short sequences, one unit per word
New words, names, and typos fall outside the vocabulary

The subword compromise

Subword tokenization keeps frequent words intact and splits rare ones into smaller pieces that recur across many words. The vocabulary stays in the tens of thousands, sequences stay reasonably short, and almost any text can be represented without an unknown-symbol fallback.

Splitting a sentence

Take the sentence "Tokenization is unbelievably useful." A word-level scheme needs a separate entry for "unbelievably"; a character-level scheme turns it into twelve units. A subword scheme might produce something like: Token | ization | is | un | believ | ably | useful | . — eight units, each drawn from a fixed vocabulary, and the pieces un-, believ-, and -ably also appear in countless other words.