Cross-Entropy Loss: The Training Objective Behind Every LLM

Cross-entropy loss is the objective function that guides every large language model during training. Despite its ubiquity, the concept is often treated as a black box. This post explains why cross-entropy loss is the natural choice for training probabilistic models of language, and how it connects to fundamental concepts in information theory and statistics.

The Gradient of Prediction Error

A language model outputs a probability distribution over the vocabulary at each position. For a vocabulary of size V, the model produces a vector q = (q_1, ..., q_V) where q_i represents the model's predicted probability that the next token belongs to token i. These probabilities sum to one: sum_i q_i = 1.

During training, we have a ground truth token y (a single index from 0 to V-1). The model's "error" is the discrepancy between its predicted distribution q and the true distribution p, where p assigns probability 1 to the correct token y and 0 to all others.

Information Theory Foundation

Cross-entropy measures the average number of extra nats (natural units of information) needed to encode samples from the true distribution using a code optimized for the model's predicted distribution. Formally, the cross-entropy between distributions p and q is:

H(p, q) = - sum_i p_i * log(q_i)

Since p is a one-hot vector (p_y = 1, p_i = 0 for i != y), the formula simplifies to:

H(p, q) = -log(q_y)

This is exactly the negative log-likelihood of the correct token under the model's distribution. The logarithm penalizes low probability assignments exponentially: a correct token with probability 0.5 produces a loss of 0.69 nats, while probability 0.1 gives 2.30 nats.

The Connection to KL Divergence

Cross-entropy decomposes into two terms:

H(p, q) = H(p) + KL(p || q)

where H(p) is the entropy of the data distribution and KL(p || q) is the Kullback-Leibler divergence from the model's predictions to the truth. Since H(p) is fixed by the data, minimizing cross-entropy is equivalent to minimizing the KL divergence. This equivalence is the reason cross-entropy works as a learning signal: it makes the model's predicted distribution match the empirical distribution of the training data.

Why Softmax + Cross-Entropy Works

Modern transformer models output unnormalized logits z = (z_1, ..., z_V). To obtain probabilities, we apply softmax:

q_i = exp(z_i) / sum_j exp(z_j)

The combination of softmax followed by cross-entropy loss has a clean gradient. The derivative of the loss with respect to the logits simplifies to:

dL/dz = q - p

This is just the difference between the predicted probabilities and the target. No softmax derivative explicitly appears—the chain rule cancels the softmax Jacobian. This computational convenience explains why classification heads universally use this pair: the gradient computation is one subtraction per logit, numerically stable and fast.

Perplexity: An Interpretable Metric

Perplexity is the exponential of cross-entropy: PPL = exp(H). It measures the "effective vocabulary size" the model faces at each prediction step. A perplexity of 20 means the model is, on average, as uncertain as if it were choosing uniformly among 20 tokens. A perplexity of 1 indicates perfect prediction (probability 1 for the correct token).

If cross-entropy approaches 0, perplexity approaches 1. Higher cross-entropy means higher perplexity. Language modeling papers typically report perplexity because it's more interpretable than raw loss values.

The Link to Maximum Likelihood Estimation

When we minimize cross-entropy loss over a dataset, we are simultaneously maximizing the probability assigned to the observed training tokens. This is maximum likelihood estimation (MLE). The equivalence between minimizing cross-entropy and maximizing likelihood is not coincidental—it's fundamental. Cross-entropy is the negative log-likelihood, so the two objectives are identical up to a sign change.

This connection provides statistical justification for cross-entropy: it is the theoretically principled answer to measuring the quality of a probabilistic model for categorical data with a fixed vocabulary.

PyTorch Implementation

PyTorch's nn.CrossEntropyLoss combines log_softmax and nll_loss (negative log-likelihood loss) internally for numerical stability. It expects raw logits (not probabilities) and integer class indices as targets:

import torch
import torch.nn as nn

loss_fn = nn.CrossEntropyLoss()
logits = torch.randn(4, 10000)  # batch of 4, vocab size 10000
targets = torch.tensor([3, 7, 1500, 9999])  # true token indices

loss = loss_fn(logits, targets)

For language modeling with label smoothing (a regularization technique), you can set the label_smoothing parameter:

loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)

Common Misconceptions

Is cross-entropy the same as negative log-likelihood? For one-hot targets (as in standard classification), yes—both refer to -log(q_y). The confusion arises because PyTorch provides separate modules: CrossEntropyLoss expects logits and applies softmax internally, while NLLLoss expects log probabilities as input.

Why not use mean squared error? MSE with softmax outputs creates vanishing gradients for confident-but-wrong predictions. The model learns slowly when it should learn quickly. Cross-entropy's gradient magnitude naturally scales with prediction confidence.

Does cross-entropy work for imbalanced data? Cross-entropy treats all classes equally, which can be problematic when some tokens are rare. Solutions include class weights, focal loss, or oversampling rare tokens in the training data.

Sources

• Cross-entropy loss: the standard training objective explained | ZeroEntropy.dev | https://zeroentropy.dev/concepts/cross-entropy-loss
• Cross-Entropy Loss: Information Theory for LLM Training | Michael Brenndoerfer | https://mbrenndoerfer.com/writing/cross-entropy-loss-language-models-information-theory
• CrossEntropyLoss — PyTorch documentation | https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
• Entropy, Perplexity and Its Applications | Lei Mao | https://leimao.github.io/blog/Entropy-Perplexity
• Understanding Evaluation Metrics for Language Models | The Gradient | https://www.thegradient.pub/understanding-evaluation-metrics-for-language-models