In-Context Learning as Implicit Gradient Descent: How Transformers Optimize Models in Forward Activations
When large language models (LLMs) adapt to new tasks from a handful of prompt demonstrations, their static weights remain completely untouched. No backpropagation runs through the network, no optimizer updates parameters, and no gradients are calculated. Yet, the model’s predictions improve steadily as more input-output examples are added to the prompt context. For years, this phenomenon (known as in-context learning, or ICL) was treated as an empirical black box. Recent theoretical and mechani
1 min