Skip to content
Learn Motion
ExploreHow it worksMembership
Log in
Learn Motion

How ChatGPT Writes One Token at a Time

1From Your Words to Tokens2Predicting the Next Token3Choosing a Token and Looping4Why the Answer Reads as Coherent Text
From Your Words to Tokens

One Sentence, Many Pieces

1 / 4
Type a sentence into ChatGPT and it looks like a smooth line of words. But that is not what the model receives.
0:00 / 0:00

ChatGPT does not receive letters or whole words. A tokenizer splits the input text into subword tokens, which are the units the model actually works with. A short sentence therefore becomes a sequence of several token pieces, and the number of tokens is usually different from the number of words.

1 / 4Next

Learn Motion

Generate a course. Learn it properly.

Operated by Wuhan Daoyin Technology Co., Ltd.

Contact: [email protected]
Privacy PolicyTerms of Service

© 2026 Learn Motion