The model's output at each step is a distribution over the vocabulary, not a decision. Decoding is the rule that converts that distribution into one token, and it is the main lever controlling how deterministic or how varied the output is.
Greedy decoding takes the single highest-probability token every time. It is deterministic: the same prompt and the same model give the same output. Its weakness is repetition and blandness, because the most likely continuation is often the safest one.
Temperature rescales the logits before the softmax. With logits \(z_i\), the sampling distribution is \[ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} \] where \(T\) is the temperature. As \(T \to 0\) the distribution concentrates on the top token and approaches greedy decoding; as \(T\) grows, the distribution flattens and low-probability tokens become reachable. Temperature changes the shape of the distribution but never removes candidates.
Top-k and top-p change which candidates are eligible. Top-k keeps the \(k\) highest-probability tokens and renormalizes. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose cumulative probability exceeds \(p\), so the candidate pool adapts to how confident the model is: a peaked distribution yields a small pool, a flat one a larger pool. The two are often combined, for example top-p with a moderate temperature.
The practical rule is that consistency and variety pull in opposite directions. Tasks with a defensible single answer, such as extracting a field or writing code that must compile, favor low temperature or greedy decoding. Tasks that benefit from many acceptable phrasings, such as brainstorming or drafting alternatives, favor higher temperature with top-p sampling.