Temperature, Top-K And Top-P Explained

A model has computed scores for possible next tokens, but a score is not yet a selection. Keep the model's learned weights and current context fixed. What changes if the application adjusts temperature or filters the candidates? Follow the same supplied scores through

temperature scaling, greedy or sampled selection, a candidate filter, and a validation check. These controls change the generation process; we must still inspect what the selected output means for the task. Use 3 vocabulary candidates labelled A, B and C, with supplied logits 2, 1 and 0.

These are unnormalized scores for possible output tokens. Their labels are teaching placeholders, not measured model predictions. The attention lesson used similar numbers over source positions. Here the output being selected is a next token; no value vectors

are being combined by this selection rule. With positive temperature T, a common convention divides each logit by T before softmax. At T equal to 1, the scaled logits remain 2, 1 and 0. The resulting probabilities are approximately 0.665, 0.245 and 0.090. They sum to 1

under this display rounding. Candidate A has the largest probability, but sampling can still select another candidate. At T equal to 2, the scaled logits become 1, 0.5 and 0. Their softmax probabilities are approximately 0.506, 0.307 and 0.186. The rounded display sums to 0.999;

the unrounded distribution sums to 1. A remains the top candidate, while B and C receive larger shares. Raising this temperature flattens the distribution without changing the model's learned parameters or adding a new source to the context. Greedy decoding chooses a highest-scoring available candidate at each step.

For both supplied distributions, that candidate is A. Sampling instead draws according to the specified probabilities, so its selections can vary. A service setting called temperature 0 usually invokes a greedy or limiting convention; dividing logits by 0 is not the calculation to perform.

Inspect the implementation's actual generation mode rather than assuming a universal switch. Filtering changes which candidates remain available for sampling. In a top-k example with k equal to 2, retain the 2 highest-ranked candidates A and B and remove C. Under the supplied temperature-1 probabilities,

renormalize the retained probabilities: about 0.731 for A and 0.269 for B. The new sampling distribution is conditional on this retained set. It is not the unchanged original distribution with a missing row. Top-p, also called nucleus sampling, uses

cumulative probability instead of a fixed count. For a separate supplied sorted distribution 0.50, 0.30, 0.15 and 0.05, use a threshold p equal to 0.75. The leading candidate contributes 0.50, which is insufficient. Adding the next gives 0.80, so keep those 2 candidates

under our stated smallest-set rule. Renormalizing gives 0.625 and 0.375 for sampling within the retained set. The distinction matters when distributions change. A fixed top-k keeps a specified count, while this top-p rule can retain a different number at different generation steps. Implementations also have details about filter order,

ties and minimum retained candidates. Hugging Face's pinned Transformers 4.57.1 documentation describes these controls as generation configuration. That is an inspectable implementation example, not a claim that every service applies identical defaults or operations in the same order. Holtzman and colleagues' study, first released in April 2019 and revised in February 2020,

compared decoding strategies using the same language model and motivated a dynamic nucleus. Its experiments showed that selection strategy affected the generated text under their setup. Use that historical case to ask which generation rule produced an observed result. It does not select a universally best rule for a museum answer,

code output or structured record. For a structured task, validate what is produced. Suppose an application allows exactly 1 label from an approved set. Check membership and required format after generation, then measure task accuracy. A token's high probability is

not the probability that the whole response is factually true. A familiar unsupported continuation can remain the greedy choice. Record model version, context, settings and repeated results; numerical execution or service revisions can affect reproducibility even under nominally deterministic controls. Pause with logits 2, 1 and 0.

Compare the temperature-1 probabilities 0.665, 0.245 and 0.090 with the temperature-2 probabilities 0.506, 0.307 and 0.186. Does the highest-ranked candidate change? Which alternatives become more likely under sampling, and has this setting added factual knowledge? A remains highest-ranked in both cases.

B and C become more likely under the stated sampling distributions. No new factual source or learned parameter has been added. These controls alter selection from the model's scores. Evaluate the resulting task behavior and factual support separately from that distribution change.

Try another filter example. The supplied sorted probabilities are 0.40, 0.25, 0.20 and 0.15. Under the smallest-leading-set top-p rule with p equal to 0.70, which candidates remain? Compare that set with top-k at k equal to 2.

Then say what happens to the retained probabilities before sampling. The cumulative totals are 0.40, 0.65 and 0.85, so top-p retains the 1st 3 candidates. Top-k retains only the 1st 2. For top-p, divide each retained probability by 0.85; for top-k,

divide the 1st 2 by 0.65. Each retained distribution must sum to 1 for sampling. The different sets can lead to different continuations even though the original model scores were identical. You can now separate a model's learned weights, its current token scores

and the rule that selects from them. Temperature reshapes a distribution, filters restrict its available set, and validation checks the produced output against the task. The next lesson asks a different question: whether the information needed for that task fits in context and remains usable when the input becomes long.