Supervised Fine-Tuning

Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations

Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations Supervised fine-tuning pipelines routinely waste 40–70% of FLOPs on padding tokens. Sequence packing eliminates that waste by concatenating multiple training examples into a single long tensor, but it introduces three hard constraints: attention must not leak across example boundaries, loss must be computed only on the intended target spans (typically assistant responses), and the data c