跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
From Text to Tokens

Why Text Must Be Cut into Units

1 / 2
The worked split is the clearest way to see the trade-off in action. At the word level, "unbelievably" needs its own vocabulary entry, and any new word or typo would be missing entirely. At the character level it becomes twelve separate units, so the model has to carry meaning across a much longer sequence. The subword split lands in between: un, believ, and ably are pieces that show up again and again across English, so a small vocabulary can still cover almost anything you throw at it. That is the whole reason subword tokenization won out.
0:00 / 0:00

A language model can only work with a finite set of discrete units, because its input and output layers are indexed by integers. Choosing those units is a trade-off between how large the vocabulary is and how long the sequences become.

The two extremes

Character-level

  • Vocabulary of a few hundred symbols
  • A 40-word sentence becomes ~200 units
  • No unknown symbols, but long sequences to track

Word-level

  • Hundreds of thousands of word forms
  • Short sequences, one unit per word
  • New words, names, and typos fall outside the vocabulary

The subword compromise

Subword tokenization keeps frequent words intact and splits rare ones into smaller pieces that recur across many words. The vocabulary stays in the tens of thousands, sequences stay reasonably short, and almost any text can be represented without an unknown-symbol fallback.

Splitting a sentence

Take the sentence "Tokenization is unbelievably useful." A word-level scheme needs a separate entry for "unbelievably"; a character-level scheme turns it into twelve units. A subword scheme might produce something like: Token | ization | is | un | believ | ably | useful | . — eight units, each drawn from a fixed vocabulary, and the pieces un-, believ-, and -ably also appear in countless other words.

1 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion