Every word an AI assistant writes passes through the same narrow gate. Before a language model can say anything, it produces a raw score for every candidate in its vocabulary, then feeds the whole list to one function that converts scores into probabilities. That function is softmax. It sits at the output layer, inside the attention mechanism, inside the routers that steer mixture-of-experts models, and underneath every temperature dial in every API. Once you can see it, half of the modern AI vocabulary (logits, temperature, logit bias, soft targets) turns into plain arithmetic.
From scores to probabilities
The last layer of a language model outputs one number per vocabulary entry. These numbers are called logits. They are unbounded: they can be negative, they can be large, and they do not sum to anything useful. The obvious move, just pick the biggest one, makes a brittle system. Hard max is not differentiable, so it cannot train, and it throws away the model's uncertainty.
Softmax makes the decision soft. The recipe is two lines of arithmetic: exponentiate every logit, then divide each result by the sum of all the exponentials. Out comes a list of numbers between 0 and 1 that sums to exactly 1, which is what a probability distribution has to look like. Larger logits get larger probabilities, and the ordering never changes.
The sharpening is where the interesting behavior lives. For three scores [2, 1, 0], softmax returns 66.5%, 24.5% and 9.0%. Widen the same shape to [8, 5, 0] and it returns 95.2%, 4.7% and 0.03%. The exponential multiplies gaps: a difference of 3 logits becomes a probability ratio of about 20 to 1, because e^3 is roughly 20.1. A difference of 6 becomes about 400 to 1. That is why the function is named the way it is: it behaves like a max, but softly, and the name "softmax" is credited to the neural-network researcher John S. Bridle in two 1989 papers.

Why the exponential, and not just dividing by the sum
A fair question: why not normalize the scores directly and skip the exponent? Three answers, each from a different field.
The first is physics. The identical formula appears in statistical mechanics: the probability that a system occupies a state is proportional to the exponential of negative energy, divided by a sum over all states. That distribution goes back to Ludwig Boltzmann in 1868 and was formalized by J. Willard Gibbs in 1902. The denominator softmax divides by is the same object physicists call the partition function. In machine learning the logits play the role of (negative) energies.
The second is decision theory. In 1959 the mathematical psychologist R. Duncan Luce derived the same functional form from his choice axiom, a constraint on rational preference: the relative odds of choosing option A over option B should not change when an irrelevant option C is added to the menu. The only way to generate choices from scores that obeys that axiom is, up to base, a softmax. Luce's model is still the standard way to describe how animals and humans split choices between options, and softmax policies in reinforcement learning descend from the same idea (Gao and Pavel collect the properties in a 2017 survey).
The third is training. Probabilities produced by softmax pair with cross-entropy loss in a way that no other normalization does, covered below. Once that combination became the default for classification, softmax was self-perpetuating.
Where softmax actually hides in an LLM
The output layer. A model like Llama 3 carries a vocabulary of 128,000 entries; OpenAI's current o200k tokenizer defines roughly 200,000. To generate one token, the network computes 128,000 or 200,000 logits and runs a single softmax over all of them. The sampler then draws from that distribution, or argmaxes it, or rejects from it under a top-p filter. Every token of every answer is one softmax pass.
Inside attention. In the 2017 Transformer paper, each query token computes a compatibility score with every key, and softmax converts those scores into weights that sum to 1, producing a weighted average of value vectors. This is how a model "decides what to read." Causal masking leans on the same arithmetic: positions the model must not see are given hugely negative scores, which exponentiate to essentially zero, so future tokens get no weight at all.
In expert routing. Mixture-of-experts models need to decide how much of the query to send to each expert. Mixtral 8x7B, for example, scores its eight experts, applies softmax, and keeps the top two, using their softmax weights to blend the outputs.
In training-time policies. Reinforcement-learning methods score actions with softmax to get exploration probabilities, the direct descendant of Boltzmann's distribution: try everything, but prefer what looks good.
The temperature knob

Softmax has exactly one dial, and every AI product exposes it. Writing softmax(z/T) instead of softmax(z) divides all logits by a temperature T before exponentiating. A high temperature flattens the distribution toward uniform, so rare tokens get a real chance; a low temperature sharpens it toward a winner-take-all spike; at the limit T approaching 0, sampling becomes greedy argmax. The worked example [2, 1, 0], which softmax renders as 66.5/24.5/9.0 percent, becomes 86.7/11.7/1.6 percent at T = 0.5. The "temperature" parameter in chat APIs is precisely this divisor, and Geoffrey Hinton's 2015 knowledge-distillation paper used the same knob in reverse: sharpen or soften a teacher's distribution by choosing T, then train a student to copy the softened one, because the small probabilities carry information about which wrong answers are nearly right.
Why it always travels with cross-entropy
Training grades the model with cross-entropy loss: negative log of the probability assigned to the correct token. Differentiate that loss with respect to the logits and something unexpectedly clean happens. The log in the loss cancels the exponential in the softmax, and the gradient for every single score collapses to "predicted probability minus what actually happened." For the correct token the error is p minus 1; for every other token it is just p. If the model said 66.5 percent for the right answer, the correction pushes with strength 0.335. If it said 99 percent, the push is nearly nothing. The update is exactly the misprediction, nothing more, which is why softmax plus cross-entropy has been the default output stage of essentially every classifier and language model since. The derivation is in any standard reference (Dive into Deep Learning, section 4.1; Goodfellow et al., Deep Learning, chapter 6).
The implementation gotcha everyone inherits
Naive softmax overflows. Floating-point arithmetic blows up past e^88 in float32, and a single large logit produces an infinity, then a NaN, then a dead training run. The fix exploits a property: softmax is unchanged if you add (or subtract) the same constant from every input, because the constant cancels between numerator and denominator. Real implementations subtract the maximum logit first, so the largest exponentiation is exp(0) = 1 and nothing can overflow. This "max trick" is the standard softmax in PyTorch and TensorFlow. Attention kernels carry it further: FlashAttention computes running maxima and running sums tile by tile (the online-softmax scheme) so it never materializes the full score matrix, which is a large part of why long-context Transformers became affordable.
Known failure modes
Softmax is not free of criticism, and three lines of research matter.
Calibration: Guo et al. (2017) showed modern neural networks are systematically overconfident, the softmax layer being a named suspect, because exponential sharpening rewards near-zero training loss with near-certain predicted probability. Their fix, temperature scaling, is just the dial above, fitted after training.
The softmax bottleneck: Yang et al. (2018) proved that because of the shared normalization, the set of distributions a low-dimensional softmax layer can express is rank-limited, and proposed a mixture of softmaxes to escape it.
Sparse losers: softmax always gives every candidate strictly positive probability, which means a 200,000-token vocabulary spreads mass over tokens that should be impossible. Martins and Astudillo (2016) proposed sparsemax and Peters et al. (2019) generalized it to the entmax family, which assigns exact zeros to low-scoring options and interpolates continuously between softmax (alpha = 1) and hard argmax. They work, occasionally outperforming softmax in attention, but none have displaced it in frontier models.
The takeaway
Softmax is twelve words of arithmetic, inherited from 19th-century physics via 1950s decision theory and 1989 neural networks, that decides how a model turns its internal confidence scores into choices. Temperature, distillation, logit bias, top-p sampling, MoE routing and calibration research are all operations on this one function. When someone says a model "picked" a word, what actually happened is that a vector of hundreds of thousands of numbers was exponentiated, normalized, and rolled against.
Sources
- Softmax function, Wikipedia
- Bridle (1989), Probabilistic Interpretation of Feedforward Classification Network Outputs
- Gao & Pavel (2017), On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
- Vaswani et al. (2017), Attention Is All You Need
- The Llama 3 Herd of Models
- OpenAI cookbook, How to count tokens with tiktoken
- Mixtral of Experts
- Hinton et al. (2015), Distilling the Knowledge in a Neural Network
- Dive into Deep Learning, section 4.1, Softmax Regression
- Goodfellow et al., Deep Learning, chapter 6
- Numerically stable softmax (max-subtraction trick)
- Dao et al. (2022), FlashAttention
- Guo et al. (2017), On Calibration of Modern Neural Networks
- Yang et al. (2018), Breaking the Softmax Bottleneck
- Martins & Astudillo (2016), From Softmax to Sparsemax
- Peters et al. (2019), Sparse Sequence-to-Sequence Models
