Token Sampling in Large Language Models: How Temperature, Top-k, Top-p, and Min-p Govern Generative Decoding

Large language models are autoregressive next-token predictors. During inference, the neural network processes an input context and outputs a vector of unnormalized scores, known as logits, across every token in its vocabulary. The mathematical pipeline that converts those raw logits into a single chosen token is the sampling and decoding stage.

While model architecture and pre-training weights determine what a model knows, sampling algorithms govern how that knowledge is extracted. Small changes to temperature, truncation thresholds, and penalties dictate whether a model produces deterministic code, coherent long-form prose, or degenerate repetition.

Understanding how sampling mechanisms operate mathematically explains why specific decoding configurations fail, how modern inference engines execute them efficiently, and why newer techniques like Min-p have emerged to address structural flaws in classical methods.

The Logit-to-Probability Transformation

At the final layer of a transformer language model, the hidden state vector is projected through an unembedding matrix to produce a logit vector $z \in \mathbb{R}^{|V|}$, where $|V|$ is the vocabulary size (typically between 32,000 and 128,000+ tokens).

The baseline conversion from raw real-valued logits to a normalized probability distribution $P$ over vocabulary tokens $w_i$ is computed via the standard softmax function:

$$P(w_i) = \frac{\exp(z_i)}{\sum_{j=1}^{|V|} \exp(z_j)}$$

Greedy Decoding vs. Stochastic Sampling

The simplest decoding approach is greedy search (or argmax decoding), where the engine always selects the token with the highest predicted probability:

$$w^* = \arg\max_{w_i \in V} P(w_i)$$

Greedy decoding is deterministic and optimal when there is a single correct continuation, such as in arithmetic, structured JSON generation, and specific programming tasks.

However, in open-ended generation, greedy decoding frequently leads to degenerate cycles. As established by Holtzman et al. (2020), neural language models assign high probability to repetitive loops once entered, creating attractor states that standard greedy selection cannot escape. To generate diverse, natural text, systems introduce stochastic sampling, drawing tokens randomly according to modified probability distributions.

Temperature Scaling

Temperature scaling originates in statistical mechanics (the Boltzmann/Gibbs distribution) and was adapted to machine learning for confidence calibration and knowledge distillation by Hinton et al. (2015).

Temperature modifies the logit vector prior to the softmax calculation by dividing each logit $z_i$ by a strictly positive scalar parameter $T$:

$$P(w_i | T) = \frac{\exp(z_i / T)}{\sum_{j=1}^{|V|} \exp(z_j / T)}$$

The behavior of the distribution changes across different temperature ranges:

• Low Temperature ($T \to 0$): As $T$ approaches zero, the difference between the maximum logit and all other logits is magnified exponentially. In the limit, the distribution collapses into a one-hot Dirac delta distribution centered on $\arg\max(z)$, rendering sampling identical to greedy decoding.
• Unit Temperature ($T = 1.0$): The network uses its raw, calibrated training distribution directly without alteration.
• High Temperature ($T > 1.0$): Dividing by a large value compresses the variance among logits, flattening the distribution toward a discrete uniform distribution.
• Infinite Temperature ($T \to \infty$): Every token in the vocabulary becomes equally likely, resulting in completely random token selection.

While higher temperatures increase lexical and conceptual diversity, they also lift low-probability tokens from the unreliable tail into plausible sampling range, causing hallucinations and grammatical breakdown if unconstrained.

Truncation Strategies: Top-k, Top-p, and Min-p

To prevent low-probability tail tokens from being sampled while maintaining generation diversity, inference engines apply truncation filters before drawing the final token.

1. Top-k Sampling

Introduced in the context of neural story generation by Fan et al. (2018), Top-k sampling limits the candidate pool to the $k$ tokens with the highest probabilities.

The algorithm sorts all tokens by probability in descending order, retains the top $k$ entries, sets the logits of all remaining tokens $i > k$ to $-\infty$, and re-normalizes the distribution:

$$V^{(k)} = \{w_1, w_2, \dots, w_k\}$$

$$P'(w_i) = \begin{cases} \frac{P(w_i)}{\sum_{w_j \in V^{(k)}} P(w_j)} & \text{if } w_i \in V^{(k)} \\ 0 & \text{otherwise} \end{cases}$$

Limitations of Top-k

Top-k uses a static threshold that does not adjust to the model's confidence:

• Peaked Distributions (High Confidence): When completing a phrase with an obvious next token (such as "The capital of France is Paris"), the top token may hold 99% probability. A fixed $k=50$ forces 49 near-zero probability tokens into the candidate set, creating an unnecessary risk of sampling an erroneous token.
• Flat Distributions (Low Confidence): When multiple continuations are equally valid (such as following an open-ended narrative prompt), hundreds of plausible tokens may exist. A fixed $k=50$ prematurely truncates valid vocabulary options.

2. Top-p (Nucleus) Sampling

To address the rigidity of Top-k, Holtzman et al. (2020) introduced Nucleus (Top-p) sampling. Instead of fixing the number of candidates, Top-p sets a threshold $p \in (0, 1]$ for the cumulative probability mass.

Tokens are sorted descending by probability, and the smallest subset $V^{(p)}$ is chosen such that their cumulative sum meets or exceeds $p$:

$$\sum_{w_i \in V^{(p)}} P(w_i) \ge p$$

The selected tokens are then re-normalized to sum to 1.0.

Nucleus sampling dynamically adjusts the candidate pool size:

• When the model is confident, the top token or top two tokens exceed $p$, shrinking $|V^{(p)}|$ to 1 or 2 tokens.
• When the model is uncertain, the probability is distributed broadly, expanding $|V^{(p)}|$ to encompass dozens or hundreds of tokens.

The High-Temperature Failure Mode of Top-p

Despite its advantages, Top-p exhibits a specific pathology when paired with high temperatures. Raising $T$ flattens the long tail of the distribution. Because the cumulative sum $p$ must still be reached, hundreds or thousands of low-quality tokens are drawn into the nucleus pool. Consequently, Top-p cannot safely prevent incoherent tokens from being selected under higher exploration settings.

3. Min-p Sampling

To resolve the tail-inflation issue of Top-p, Shwartz-Ziv and Roush (2024) formulated Min-p sampling (subsequently integrated into major serving runtimes including vLLM and Hugging Face Transformers).

Min-p establishes a dynamic minimum probability threshold that is scaled directly relative to the probability of the most likely token $p_{\max} = \max_{w_j} P(w_j)$:

$$p_{\text{threshold}} = p_{\max} \cdot \text{min\_p}$$

A token $w_i$ is included in the candidate pool $V^{(\text{min\_p})}$ if and only if:

$$P(w_i) \ge p_{\text{threshold}}$$

The selected candidates are then re-normalized:

$$P'(w_i) = \begin{cases} \frac{P(w_i)}{\sum_{w_j \in V^{(\text{min\_p})}} P(w_j)} & \text{if } P(w_i) \ge p_{\max} \cdot \text{min\_p} \\ 0 & \text{otherwise} \end{cases}$$

How Min-p Adapts to Confidence

• High Confidence ($p_{\max} = 0.90$, $\text{min\_p} = 0.05$): The threshold is $0.90 \times 0.05 = 0.045$ (4.5%). Only tokens with at least 4.5% probability qualify, discarding nearly all tail noise.
• Low Confidence ($p_{\max} = 0.10$, $\text{min\_p} = 0.05$): The threshold is $0.10 \times 0.05 = 0.005$ (0.5%). Many viable tokens qualify, enabling broad exploration.

Because the cutoff threshold scales with $p_{\max}$, Min-p allows practitioners to raise temperature for creativity without admitting implausible tail tokens.

Penalties and Logit Manipulation

In addition to truncation, production serving systems apply arithmetic modifiers directly to logits prior to softmax calculation.

1. Repetition, Frequency, and Presence Penalties

To prevent repetitive phrasing and degenerate looping, models use penalty modifiers introduced by Keskar et al. (2019):

• Multiplicative Repetition Penalty: Scales logit $z_i$ by a factor $\theta > 1.0$ for all previously generated tokens:

$$z_i' = \begin{cases} z_i / \theta & \text{if } z_i > 0 \\ z_i \cdot \theta & \text{if } z_i \le 0 \end{cases}$$

• Additive Frequency Penalty: Reduces a token's logit proportionally to the number of times $c_i$ it has already appeared in the output:

$$z_i' = z_i - (\alpha_{\text{freq}} \cdot c_i)$$

• Additive Presence Penalty: Applies a fixed reduction if a token has appeared at least once ($c_i > 0$), regardless of exact frequency:

$$z_i' = z_i - (\alpha_{\text{pres}} \cdot \mathbb{I}[c_i > 0])$$

Serving Pitfall

Excessive penalties degrade output quality by penalizing necessary grammatical words (articles, prepositions, conjunctions) and code tokens (such as return, def, class, or common brackets). Penalties should generally remain modest (e.g., frequency penalty $\le 0.2$).

2. Logit Bias

Logit bias applies direct user-specified additive offsets to specific vocabulary IDs:

$$z_i' = z_i + b_i$$

Setting $b_i = -\infty$ guarantees that token $w_i$ cannot be sampled, which is commonly used to suppress forbidden markers, control grammar boundaries, or enforce structural constraints.

Inference Engine Execution: GPU-Accelerated Sampling

In modern high-throughput LLM serving systems such as vLLM, SGLang, and TensorRT-LLM, sampling is executed on the GPU immediately following the forward pass.

Transferring a full logit tensor ($B \times |V|$, where batch size $B = 256$ and vocabulary $|V| = 128,000$, amounting to approximately 131 MB per decode step) from GPU VRAM to host CPU memory would create a severe PCIe bottleneck and introduce high latency.

Instead, inference engines use fused CUDA and Triton kernels that execute logit penalties, temperature scaling, Top-k/Top-p/Min-p sorting, softmax, and multinomial sampling in on-chip GPU SRAM:

1. Penalty application and temperature division are performed element-wise in parallel across threads.
2. Top-k selection uses GPU radix sort or parallel reduction trees (such as Warp-level reductions) to find the largest $k$ values without sorting the full vocabulary.
3. Cumulative distribution calculation (Top-p) runs via parallel prefix scans (inclusive scan).
4. Multinomial sampling draws a uniform random variable $u \sim \mathcal{U}(0, 1)$ on GPU and performs binary search or parallel scan across the cumulative distribution to select the final token ID.

This keeps all intermediate probability arrays on-chip, returning only the single selected 32-bit integer token ID per sequence back to the engine's batch manager.

Sampling Parameter Profiles for Production Workloads

• Code Generation and Math (temperature=0.0): Greedy decoding. Maximizes precision, syntax validity, and deterministic adherence to algorithmic constraints.
• Structured Output and JSON Extraction (temperature=0.1, top_p=0.95): Low temperature suppresses syntax deviations while accommodating minor variations in schema field content.
• Factual Retrieval and Summarization (temperature=0.3, min_p=0.05): Restricts generation to high-confidence tokens while retaining natural phrasing flow.
• Open-Ended Writing and Dialogue (temperature=0.8, min_p=0.05): Allows broad lexical selection while preventing tail degradation and hallucinated artifacts.
• Brainstorming and High-Variance Ideation (temperature=1.1, min_p=0.08): High entropy encourages unexpected associative jumps without admitting corrupted vocabulary fragments.

Sources

• Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. International Conference on Learning Representations (ICLR).
• Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. Association for Computational Linguistics (ACL).
• Shwartz-Ziv, R., & Roush, A. (2024). Min-p Sampling for Creative and Coherent LLM Outputs. arXiv:2407.01082.
• Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019). CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv:1909.05858.
• Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
• vLLM Team. (2024). vLLM Sampling Parameters Documentation.