Transformer Architecture2 articles

Transformer Architecture

Articles

Loss Spikes and Training Stability in Large Language Models: How Attention Logit Drift, z-loss, and QK-Norm Prevent Gradient Explosions
During the pre-training of modern large language models, few operational failures are as costly as loss spikes. When training clusters containing thousands of GPUs run for weeks across trillions of tokens, a sudden, discontinuous surge in cross-entropy loss can corrupt optimizer momentum buffers, induce numerical overflow in half-precision representations, and permanently degrade downstream model capabilities. In severe cases, models experience catastrophic divergence, forcing engineering teams
August 21, 20261 min
In-Context Learning in Large Language Models: How Induction Heads and Attention Circuits Learn Without Weight Updates
Autoregressive large language models demonstrate the ability to adapt to new tasks, follow few-shot demonstrations, and execute algorithmic patterns entirely within their context windows. Unlike traditional fine-tuning, in-context learning occurs at inference time with frozen model parameters, leaving weights completely unchanged ($\Delta \theta = 0$). For several years following the scaling demonstrations in GPT-3, the internal mechanism governing in-context learning remained an empirical blac
August 18, 20261 min