The model has produced a probability distribution over the vocabulary, but no token has been chosen. Selection means picking one entry from that distribution. In greedy decoding the highest-probability token is always taken. In sampling, a token is drawn at random with probability proportional to its assigned probability, so a token with 40 percent probability is chosen roughly 40 percent of the time. Once selected, the token is appended to the end of the sequence, making the input one token longer.
How ChatGPT Writes One Token at a Time