跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How Large Language Models Work: From Tokens to Inference

1From Text to Tokens2Embeddings: Turning Tokens into Vectors3The Transformer Block: Attention and Feed-Forward Processing4Stacking Blocks into a Language Model5Training: How the Model Learns to Predict6From Pretraining to Instruction Following7Inference: Generating Text Token by Token
Inference: Generating Text Token by Token

Choosing the Token: Temperature, Top-k, and Top-p

2 / 3
Start with temperature near zero and press sample a few times. The same token comes up every time, because the distribution has collapsed onto the top candidate. Now raise the temperature and watch the bars flatten; the low-probability tokens become reachable and the samples start to vary. Then set k to three and p to about zero point nine. The bars outside the eligible set grey out, and sampling only draws from what remains. The thing to notice is that temperature changes the shape of the distribution, while k and p change which candidates are allowed at all. That is the whole trade-off: tighten these controls when you need one reliable answer, loosen them when you want variety.
0:00 / 0:00

The model's output at each step is a distribution over the vocabulary, not a decision. Decoding is the rule that converts that distribution into one token, and it is the main lever controlling how deterministic or how varied the output is.

Greedy decoding takes the single highest-probability token every time. It is deterministic: the same prompt and the same model give the same output. Its weakness is repetition and blandness, because the most likely continuation is often the safest one.

Temperature rescales the logits before the softmax. With logits \(z_i\), the sampling distribution is
\[ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} \]
where \(T\) is the temperature. As \(T \to 0\) the distribution concentrates on the top token and approaches greedy decoding; as \(T\) grows, the distribution flattens and low-probability tokens become reachable. Temperature changes the shape of the distribution but never removes candidates.

Top-k and top-p change which candidates are eligible. Top-k keeps the \(k\) highest-probability tokens and renormalizes. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose cumulative probability exceeds \(p\), so the candidate pool adapts to how confident the model is: a peaked distribution yields a small pool, a flat one a larger pool. The two are often combined, for example top-p with a moderate temperature.

The practical rule is that consistency and variety pull in opposite directions. Tasks with a defensible single answer, such as extracting a field or writing code that must compile, favor low temperature or greedy decoding. Tasks that benefit from many acceptable phrasings, such as brainstorming or drafting alternatives, favor higher temperature with top-p sampling.

上一页2 / 3下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion