Supervised Fine-Tuning in Large Language Models: Loss Masking, Sequence Packing, and Alignment Dynamics

Pre-training endows a large language model with broad linguistic patterns, world knowledge, and reasoning primitives by predicting the next token across trillions of uncurated web tokens. However, a raw base model remains a document completer rather than an interactive assistant. Given a prompt such as "Explain how a compiler works," a base model is as likely to generate additional exam questions or web navigation headers as it is to answer the query.

Supervised Fine-Tuning (SFT), frequently termed instruction tuning, serves as the foundational bridge between unstructured pre-training and downstream alignment. By fine-tuning model parameters on curated pairs of instructions and high-quality responses, SFT conditions the network to recognize task boundaries, adopt conversational personas, and follow complex user constraints.

Behind this conceptual simplicity lie crucial engineering choices: prompt loss masking, sequence packing algorithms, attention boundary isolation, and loss normalization across variable-length dialogues.

Illustration of prompt loss masking, block-diagonal attention matrices, and sequence packing in supervised fine-tuning

Mathematical Objective of Supervised Fine-Tuning

In standard causal language modeling pre-training, the model maximizes the likelihood of every token in an input sequence. In contrast, Supervised Fine-Tuning operates on structured dialogue instances consisting of an instruction or context prompt $x = (x_1, x_2, \dots, x_M)$ and an assistant response $y = (y_1, y_2, \dots, y_N)$ .

The full concatenated sequence is $S = (x_1, \dots, x_M, y_1, \dots, y_N)$ with total length $T = M + N$ . SFT optimizes the conditional auto-regressive log-likelihood exclusively over the completion tokens:

$\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t=1}^{N} \log P_\theta(y_t \mid x_1, \dots, x_M, y_1, \dots, y_{t-1})$

In this formulation, the model learns the conditional probability distribution $P(y \mid x)$ rather than the joint distribution $P(x, y)$ .

Prompt Loss Masking

A foundational divergence between continued pre-training and SFT is loss masking. In PyTorch and modern training frameworks, cross-entropy loss functions support an ignore_index argument (conventionally set to -100).

import torch
import torch.nn as nn

# Standard SFT loss setup
criterion = nn.CrossEntropyLoss(ignore_index=-100, reduction='none')

# labels array matches input_ids shape, with prompt indices masked out
# input_ids: [x1, x2, ..., xM, y1, y2, ..., yN]
# labels:    [-100, -100, ..., -100, y1, y2, ..., yN]

Why Prompt Loss Must Be Masked

If prompt tokens are unmasked during training, the loss function forces the model to predict the next token of user prompts. This produces two distinct failure modes:

Gradient Pollution: The model expends optimization capacity learning the arbitrary phrasing and distribution of user questions rather than learning how to generate compliant answers.
Degraded Instruction Adherence: Research documented in Instruction Fine-Tuning: Does Prompt Loss Matter? demonstrates that assigning positive loss weight to prompt tokens degrades downstream task performance on short-completion instruction benchmarks.

The Boundary Transition Condition

Autoregressive models predict token $t+1$ given tokens $1 \dots t$ . Consequently, the loss computed at index $M$ (the final prompt delimiter, such as <|im_start|>assistant\n) evaluates the network's prediction of $y_1$ (the first token of the assistant's reply).

Accurate index alignment during dataset preparation is critical. If masking accidentally covers the boundary token, the model receives zero gradient signal on how to initiate its response, leading to stuttering or generation failure.

Sequence Packing and Padding-Free Training

Instruction datasets exhibit extreme variance in sequence length: a simple math query may comprise 50 tokens, while a detailed code refactoring session spans 4,000 tokens.

In naive batching, every sequence in a micro-batch is padded with <pad> tokens to match the longest sequence in that batch. Because transformer self-attention scales quadratically ( $O(L^2)$ ) with sequence length $L$ , padded tokens consume substantial compute and memory bandwidth while contributing zero gradient information.

Naive Padded Batch (Wasteful):
Sample 1: [Token 1, Token 2, ..., Token 500,  PAD, PAD, ..., PAD (3596 tokens)]
Sample 2: [Token 1, Token 2, ..., Token 4096]
Total tokens processed: 8192 | Useful tokens: 4596 | Efficiency: 56.1%

Contiguous Sequence Packing

To eliminate padding overhead, modern trainers implement sequence packing. Multiple independent conversations are concatenated end-to-end into a single contiguous context buffer of fixed capacity (such as 4,096, 8,192, or 32,768 tokens) using bin-packing algorithms (such as first-fit decreasing).

Packed Contiguous Buffer (Zero Waste):
[Doc 1: 500 tokens][Doc 2: 1200 tokens][Doc 3: 2396 tokens] = 4096 tokens (100% compute efficiency)

Preventing Cross-Sample Contamination

Packing introduces a critical architectural challenge: if tokens from Document 2 attend to tokens from Document 1 within the same attention window, cross-dialogue contamination occurs. The model learns spurious correlations across unrelated user queries.

Two complementary mechanisms isolate packed sequences:

Block-Diagonal Attention Masking: A 2D attention mask ensures that query token $q_i$ belonging to Document $k$ can only compute attention scores against key tokens $k_j$ that also belong to Document $k$ .
FlashAttention Variable-Length Kernels: Rather than instantiating large 2D attention masks in global memory, FlashAttention-2 provides flash_attn_varlen_func. This kernel accepts an unpadded 1D token tensor alongside a cu_seqlens (cumulative sequence lengths) vector:

# Cumulative sequence length boundaries for 3 packed sequences of length 500, 1200, 2396
cu_seqlens = torch.tensor([0, 500, 1700, 4096], dtype=torch.int32, device='cuda')

# FlashAttention computes self-attention strictly within each boundary segment
output = flash_attn_varlen_func(
    q, k, v, 
    cu_seqlens_q=cu_seqlens, 
    cu_seqlens_k=cu_seqlens, 
    max_seqlen_q=2396, 
    max_seqlen_k=2396, 
    causal=True
)

Position ID Resetting

When packing sequences, practitioners choose between two positional encoding strategies:

Continuous Position IDs: Indices increment monotonically from 0 to $L-1$ across the entire packed buffer.
Reset Position IDs: Position indices reset to 0 at the start of each packed conversation segment ( $[0, 1, \dots, 499, 0, 1, \dots, 1199]$ ).

Resetting position IDs maintains consistency with inference conditions, where single conversations always start at position 0. When paired with Rotary Position Embeddings (RoPE), position resets prevent long-context extrapolation distortions on short conversations.

Loss Normalization: Token-Mean vs. Sample-Mean

When computing gradients over a batch of variable-length conversational responses, the reduction strategy significantly influences optimization dynamics.

Token-Level Normalization (Token-Mean)

Token-mean reduction sums the cross-entropy losses across all active (unmasked) completion tokens in the batch and divides by the total count of active tokens:

$\mathcal{L}_{\text{token-mean}} = \frac{\sum_{i=1}^{B} \sum_{t=1}^{N_i} \ell_{i,t}}{\sum_{i=1}^{B} N_i}$

Under token-mean normalization, a response containing 2,000 tokens contributes ten times more gradient weight than a response containing 200 tokens. This introduces an implicit optimization bias toward longer, more verbose generations.

Sample-Level Normalization (Sample-Mean)

Sample-mean reduction computes the average token loss for each dialogue sample individually, and then averages these sample losses across the batch:

$\mathcal{L}_{\text{sample-mean}} = \frac{1}{B} \sum_{i=1}^{B} \left( \frac{1}{N_i} \sum_{t=1}^{N_i} \ell_{i,t} \right)$

Sample-mean normalization assigns equal weight to every instruction regardless of completion length. A concise factual answer is weighted identically to an extensive code refactoring output. Modern alignment frameworks like Hugging Face TRL and NVIDIA NeMo provide configurable loss reduction parameters to balance token versus sample weighting.

Data Dynamics and the Superficial Alignment Hypothesis

A central theoretical milestone in understanding SFT is the Superficial Alignment Hypothesis introduced by LIMA: Less Is More for Alignment (Zhou et al., NeurIPS 2023).

The hypothesis posits that almost all knowledge, factual associations, and reasoning capabilities are acquired during pre-training. Supervised Fine-Tuning acts primarily as a style and format filter, teaching the model which sub-distribution of its existing internal representations to activate when interacting with users.

The authors showed that fine-tuning a 65B parameter LLaMA base model on just 1,000 carefully curated instruction examples produced output quality competitive with models trained on tens of thousands of uncurated examples.

Overfitting and Entropy Collapse

Because instruction datasets are orders of magnitude smaller than pre-training corpora (typically $10^3$ to $10^6$ examples versus $10^{12}$ tokens), SFT models are vulnerable to overfitting:

Epoch Regimes: Full-parameter SFT typically runs for only 1 to 3 epochs. Training beyond 3 epochs often leads to memorization, repetitive phrasing, and degraded generalization.
Entropy Collapse: Over-training on narrow instruction formats sharply reduces output distribution entropy, impairing the model's creative problem-solving and causing catastrophic forgetting of base pre-training capabilities.
Hyperparameter Stability: Standard recipes employ conservative learning rates (between $1 \times 10^{-5}$ and $2 \times 10^{-5}$ for full fine-tuning; $1 \times 10^{-4}$ to $2 \times 10^{-4}$ for LoRA), a linear warmup over 3% to 5% of training steps, cosine learning rate decay, and weight decay of 0.01 to 0.1.

Chat Templates and Special Token Serialization

Modern instruction models rely on structured chat templates (such as ChatML) to parse multi-turn dialogue histories unambiguously into linear token sequences.

{% for message in messages %}
{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}
{% endfor %}
{% if add_generation_prompt %}
{{'<|im_start|>assistant\n'}}
{% endif %}

During serialization, each role turn is enclosed within dedicated delimiter tokens:

<|im_start|>system\n...<|im_end|>
<|im_start|>user\n...<|im_end|>
<|im_start|>assistant\n...<|im_end|>

End-of-Sequence Token Handling

A frequent implementation defect in custom SFT pipelines involves improper handling of the end-of-sequence delimiter (<|im_end|> or <|endoftext|>).

If the <|im_end|> token at the conclusion of an assistant turn is masked out or omitted from label calculations, the model is never penalized for continuing to generate text after completing its answer. During inference, this results in generation runaway, where the model fabricates imaginary user turns and converses with itself indefinitely.

Summary of SFT Implementation Best Practices