Teacher Forcing and Exposure Bias in Autoregressive Models: Mathematical Foundations, Compounding Errors, and Mitigation Strategies

Autoregressive sequence models form the computational backbone of modern language modeling, code synthesis, and multi-step reasoning agents. From decoder-only Transformers like GPT and Claude to recurrent architectures, these systems factorize the joint probability of a sequence of tokens $y = (y_1, y_2, \dots, y_T)$ into a product of conditional probabilities:

$$p_\theta(y_1, \dots, y_T \mid x) = \prod_{t=1}^T p_\theta(y_t \mid x, y_{<t})$$

During standard pre-training and supervised fine-tuning, these models are trained via Maximum Likelihood Estimation (MLE) using an optimization regime known as Teacher Forcing, originally formalized by Ronald Williams and David Zipser in 1989. Under Teacher Forcing, when predicting the token at step $t$, the model is strictly conditioned on the ground-truth prefix $y_{<t} = (y_1, \dots, y_{t-1})$ provided by the dataset, regardless of what tokens the model would have generated itself.

However, during inference and deployment, ground-truth future tokens do not exist. The model must operate in an open-loop, free-running mode where each generated token $\hat{y}_t \sim p_\theta(\cdot \mid x, \hat{y}_{<t})$ is appended to the context and fed back into the model to predict step $t+1$.

This structural discrepancy between training-time conditioning and test-time execution introduces a systemic failure mode known as Exposure Bias. When a model makes even a minor mistake early in generation, it transitions into an out-of-distribution state that it was never exposed to during training. Because the model was never trained to recover from imperfect prefixes, small initial errors compound catastrophically across time steps.

1. Mathematical Formulation of the Train-Test Discrepancy

To understand why Exposure Bias occurs, we examine the divergence in state distributions between training and inference.

Teacher-Forced Training Objective

In standard autoregressive training over a dataset $\mathcal{D} = \{(x^{(i)}, y^{(i)})\}_{i=1}^N$, the model parameters $\theta$ are optimized by minimizing empirical cross-entropy loss:

$$\mathcal{L}_{\text{TF}}(\theta) = -\mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{t=1}^T \log p_\theta(y_t \mid x, y_1, y_2, \dots, y_{t-1}) \right]$$

Because the prefix $y_{<t}$ at every step $t$ is drawn directly from the true empirical data distribution $p_{\text{data}}$, the training process enjoys two critical properties:

1. Full Parallelism: In Transformer architectures, causal attention masking allows all $T$ token predictions and loss computations to be evaluated simultaneously in parallel on modern GPU hardware, avoiding sequential rollout loops during training.
2. Stable Gradients: The model's internal hidden states $h_t$ remain anchored along trajectories defined entirely by human-authored or verified ground-truth sequences.

Free-Running Inference Dynamics

At test time, given a prompt $x$, the sequence generation proceeds iteratively:

$$\hat{y}_1 \sim p_\theta(\cdot \mid x)$$ $$\hat{y}_2 \sim p_\theta(\cdot \mid x, \hat{y}_1)$$ $$\hat{y}_t \sim p_\theta(\cdot \mid x, \hat{y}_1, \dots, \hat{y}_{t-1})$$

Let $d_{\text{train}}^t$ represent the distribution over prefixes $y_{<t}$ observed during training, and let $d_\theta^t$ represent the distribution over generated prefixes $\hat{y}_{<t}$ induced by the model's own policy $\pi_\theta$.

Under Teacher Forcing, $d_{\text{train}}^t = p_{\text{data}}(y_{<t} \mid x)$. At inference time, however, the model samples from $d_\theta^t = p_\theta(\hat{y}_{<t} \mid x)$. As generation proceeds, the total variation distance or Kullback-Leibler divergence between these two distributions increases monotonically:

$$D_{\text{KL}}\left(d_\theta^t \parallel d_{\text{train}}^t\right) \ge 0$$

Once $d_\theta^t$ departs from the support of $d_{\text{train}}^t$, the model encounters hidden representation states $h_t \notin \text{supp}(d_{\text{train}})$. In these uncharted regions of latent space, output logits become uncalibrated, gradient paths are undefined, and prediction accuracy collapses.

2. The Compounding Error Theorem: Quadratic vs. Linear Drift

The severity of Exposure Bias is not merely an intuitive observation; it is governed by formal error-compounding bounds in sequential decision-making.

In their foundational work on imitation learning and structured prediction, Ross, Gordon, and Bagnell (2011) proved that when an agent is trained via standard supervised learning (behavior cloning / teacher forcing) on an expert distribution:

The Quadratic Supervised Bound

Assume an autoregressive model has an expected single-step error probability bounded by $\epsilon$ under the training distribution:

$$\mathbb{E}_{s \sim d_{\text{train}}} \left[ \mathbf{1}\left(\arg\max_y p_\theta(y \mid s) \neq y^* \right) \right] \le \epsilon$$

When this model is evaluated autoregressively over a sequence of length $T$, the probability that the model makes zero errors up to step $t-1$ is $(1 - \epsilon)^{t-1} \approx 1 - (t-1)\epsilon$.

Conversely, the probability that the model has committed at least one error prior to step $t$ is:

$$P(\text{mistake before } t) = 1 - (1 - \epsilon)^{t-1} \le (t - 1)\epsilon$$

If we assume an adversarial condition where a single mistake places the model in an unfamiliar state where all subsequent predictions fail (accumulating a loss of 1 per subsequent step), the total expected sequence-level error $\mathcal{E}(T)$ satisfies:

$$\mathcal{E}_{\text{supervised}}(T) \le \sum_{t=1}^T \left[ (1 - (t-1)\epsilon) \cdot \epsilon + (t-1)\epsilon \cdot 1 \right] \approx \sum_{t=1}^T t\epsilon = \frac{T(T+1)}{2}\epsilon \in \mathcal{O}\left(\epsilon T^2\right)$$

Under teacher-forced supervised learning, sequence errors grow quadratically with sequence length $T$.

The Linear On-Policy Bound

By contrast, if the model is trained on state distributions visited by its own policy (as in online imitation algorithms like DAgger or policy-gradient reinforcement learning), the state distribution mismatch is resolved. The expected sequence error scales linearly:

$$\mathcal{E}_{\text{on-policy}}(T) \le T \epsilon \in \mathcal{O}\left(\epsilon T\right)$$

For a generation horizon of $T = 1000$ tokens or steps, $\mathcal{O}(T^2)$ error compounding represents a three-order-of-magnitude amplification in vulnerability compared to the on-policy $\mathcal{O}(T)$ baseline.

3. Practical Manifestations in Modern LLMs

In contemporary large language models, Exposure Bias manifests across several distinct phenomena:

1. Hallucination Cascades

When an LLM produces an erroneous entity or incorrect intermediate reasoning step at token $t$, that hallucinated token enters the key-value (KV) cache. In subsequent layers, causal self-attention mechanisms compute attention weights over all previous keys:

$$A_{t+k, t} = \text{softmax}\left(\frac{q_{t+k} k_t^T}{\sqrt{d_k}}\right)$$

Because the model was trained on coherent, factually consistent corpora where prior statements are treated as ground truth, the attention mechanism assumes $\hat{y}_t$ is ground truth. The model actively conditions on its own hallucination, generating secondary and tertiary claims to maintain rhetorical consistency with its initial error.

2. Degenerate Repetition Loops

A classic symptom of exposure bias is degenerate repetition, where a model enters an infinite loop repeating sentences or n-gram phrases. Once a slight statistical wobble causes a repeated phrase to appear twice in the context window, the model's self-attention strongly amplifies the recurrence pattern (via induction heads and positional repetition bias), making the probability of generating the phrase a third time approach 1.0.

3. Long-Horizon Agent and Reasoning Failures

In autonomous agent workflows (e.g., executing terminal commands or multi-turn tool calling) and extended chain-of-thought mathematical derivations, exposure bias is acute. A single malformed JSON character, invalid shell flag, or flawed arithmetic sub-step places the agent in an execution state outside its training distribution. Without explicit recovery training, the agent continues executing downstream actions based on false premises.

4. Algorithmic Mitigation Strategies

Over the past decade, researchers have developed multiple distinct paradigms to bridge the train-test divergence.

+-----------------------------------------------------------------------------+
|                      EXPOSURE BIAS MITIGATION SPECTRUM                     |
+-----------------------------------------------------------------------------+
| Supervised / Heuristic               Adversarial / State           RL & Policy Optimization
|                                                                             |
| - Scheduled Sampling (2015)         - Professor Forcing (2016)   - MIXER (2016)
| - Word Dropout / Replacement         - Continuous Latent Align    - PPO / RLHF (2022)
| - DAgger Prefix Augmentation                                      - GRPO / RLVR (2024-2026)
+-----------------------------------------------------------------------------+

1. Scheduled Sampling (Bengio et al., 2015)

In Scheduled Sampling (Bengio et al., NIPS 2015), the training regime is altered by stochastically replacing ground-truth tokens with model predictions during training.

At each step $t$ in training epoch $i$, a Bernoulli random variable is sampled with probability $\epsilon_i \in [0, 1]$. The input token fed to the next step is determined by:

$$y_{t-1}^{\text{input}} = \begin{cases} y_{t-1}^* & \text{with probability } \epsilon_i \\ \arg\max_y p_\theta(y \mid y_{<t-1}^{\text{input}}) & \text{with probability } 1 - \epsilon_i \end{cases}$$

The probability $\epsilon_i$ decays according to a schedule:

• Linear Decay: $\epsilon_i = \max(\epsilon_{\min}, 1 - k \cdot i)$
• Exponential Decay: $\epsilon_i = k^i$ with $k < 1$
• Inverse Sigmoid Decay: $\epsilon_i = \frac{k}{k + \exp(i / k)}$

Theoretical and Practical Limitations: While Scheduled Sampling showed empirical gains in recurrent sequence-to-sequence tasks (machine translation, captioning), it suffers from fundamental flaws:

1. Gradient Inconsistency: Because sampling discrete tokens is non-differentiable, backpropagation cannot pass through the decision to pick $\hat{y}_t$. Gradients are only computed conditionally on the selected token as if it were fixed, yielding biased gradient estimates.
2. Grammar Mismatch: Forcing the model to predict the ground-truth target $y_t^*$ even when the input prefix $y_{<t}^{\text{input}}$ has diverged into an alternative grammatical structure penalizes valid alternative sentence completions.
3. Transformer Incompatibility: Scheduled Sampling breaks parallel causal attention masking during training. Every token prediction requires a sequential forward pass, multiplying training compute by sequence length $T$.

2. Professor Forcing (Goyal et al., 2016)

To avoid discrete token sampling issues, Professor Forcing (Goyal et al., NeurIPS 2016) introduced an adversarial framework operating in hidden activation space rather than token space.

A generator network produces hidden state trajectories under both Teacher Forcing ($h_{\text{TF}}^t$) and Free Running ($h_{\text{FR}}^t$). A discriminator network $D(\cdot)$ is trained to distinguish between sequences of hidden states:

$$\mathcal{L}_{\text{disc}} = -\mathbb{E}\left[\log D(h_{\text{TF}}^{1:T})\right] - \mathbb{E}\left[\log(1 - D(h_{\text{FR}}^{1:T}))\right]$$

The generator is trained simultaneously on MLE cross-entropy and an adversarial loss that forces the distribution of its free-running hidden states to match its teacher-forced hidden states:

$$\mathcal{L}_{\text{gen}} = \mathcal{L}_{\text{TF}} - \lambda \mathbb{E}\left[\log D(h_{\text{FR}}^{1:T})\right]$$

While Professor Forcing eliminated the need for discrete token backpropagation, training adversarial GAN-like objectives over deep sequence dynamics proved unstable and computationally demanding at scale.

3. Policy Optimization and Reinforcement Learning (PPO, GRPO, RLVR)

The most effective and scalable solution to Exposure Bias in modern frontier models is Reinforcement Learning with Policy Optimization.

Originally adapted for sequence models via MIXER (Ranzato et al., 2016) using REINFORCE, modern post-training relies on actor-critic algorithms like Proximal Policy Optimization (PPO) and value-model-free variants like Group Relative Policy Optimization (GRPO).

Under policy gradient objectives, sequence generation is treated as a Markov Decision Process (MDP):

• State $s_t$: The prefix context $(x, \hat{y}_{<t})$.
• Action $a_t$: The selected token $\hat{y}_t \in \mathcal{V}$.
• Transition: Deterministic concatenation $s_{t+1} = [s_t, a_t]$.
• Reward $R(\hat{y})$: Evaluated on the completed sequence by a reward model or a deterministic oracle (e.g., test suites, compiler pass, math verification).

The policy gradient objective optimizes expected reward under the model's own sampling distribution:

$$\mathcal{J}(\theta) = \mathbb{E}_{(x, y) \sim \mathcal{D}, \, \hat{y}_{1:T} \sim p_\theta(\cdot \mid x)} \left[ R(x, \hat{y}_{1:T}) \right]$$

Because rollouts are generated on-policy ($\hat{y} \sim p_\theta$), the model explores states that it naturally generates at inference time. If the model veers into an awkward or sub-optimal trajectory early in the rollout, the policy learns to navigate from that exact compromised state toward a successful terminal reward, fundamentally eliminating the covariate shift of Teacher Forcing.

5. Comparative Matrix: Training Paradigms

| Method | Sampling Distribution | Error Bound | Parallel Training? | Differentiable Gradients? | Primary Modern Domain | | :--- | :--- | :--- | :--- | :--- | :--- | | Teacher Forcing (MLE) | $p_{\text{data}}(y_{<t})$ (Ground Truth) | $\mathcal{O}(\epsilon T^2)$ | Yes (Causal Masking) | Yes (Exact Cross-Entropy) | Pre-training, Initial SFT | | Scheduled Sampling | Mix of $p_{\text{data}}$ and $p_\theta$ | $\mathcal{O}(\epsilon T^{1.5} - \epsilon T^2)$ | No (Sequential Step) | No (Biased Approximation) | Classic RNN Seq2Seq | | Professor Forcing | $p_{\text{data}}$ + Latent Adversarial | $\mathcal{O}(\epsilon T)$ (Latent) | Partial | Yes (W-Loss on Activations) | Continuous Time Series | | DAgger | $p_\theta(\hat{y}_{<t})$ + Expert Labels | $\mathcal{O}(\epsilon T)$ | No (Interactive Oracle) | Yes (Supervised on Rollouts) | Robotics, Imitation Learning | | Policy Gradient / GRPO | $p_\theta(\hat{y}_{1:T})$ (On-Policy) | $\mathcal{O}(\epsilon T)$ | Yes (Batched Rollouts) | Yes (Score Function / PPO) | Reasoning Models, Agent Loops |

6. Exposure Bias in Autoregressive Diffusion and World Models

In recent architectures expanding beyond discrete text into continuous video and physical world models, Exposure Bias has resurfaced in continuous state spaces.

In autoregressive video generation (e.g., generating frame $t$ conditioned on frames $1 \dots t-1$), models trained with pure Teacher Forcing on clean video frames rapidly accumulate visual artifacts, blurriness, and physical hallucination when forced to condition on their own imperfectly rendered historical frames.

Recent frameworks such as Diffusion Forcing and Self Forcing resolve this by training diffusion backbones with independent noise schedules per frame, explicitly conditioning future denoising steps on self-generated, noisy historical rollouts. By exposing the denoising network to its own generation artifacts during training, the model develops intrinsic error-correcting dynamics that maintain visual coherence over trajectories exceeding thousands of frames.

7. Conclusion and Engineering Implications

Teacher Forcing remains irreplaceable for large-scale pre-training due to its computational throughput and mathematical tractability under parallel GPU attention kernels. However, treating Teacher Forcing as sufficient for downstream deployment creates fragile systems prone to catastrophic drift.

For engineers and researchers building frontier LLM applications:

1. Recognize the Quadratic Penalty: In tasks requiring long sequence generations ($T > 1000$), standard SFT models will suffer quadratic error compounding unless supplemented with on-policy training.
2. Deploy On-Policy Alignment for Reasoning: The performance surge observed in frontier reasoning models (e.g., DeepSeek-R1, OpenAI o-series) stems directly from extensive on-policy RL (GRPO/RLVR), which teaches models to backtrack, verify, and self-correct from self-generated intermediate errors.
3. Design Agentic Error Recovery: In AI agent architectures, assume Exposure Bias will occur. Build structural guardrails such as deterministic validation oracles, environment feedback loops, and dynamic backtrack mechanisms to reset compromised context windows before errors cascade into unrecoverable failure.

Sources

• Williams, R. J., & Zipser, D. (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2), 270–280. https://doi.org/10.1162/neco.1989.1.2.270
• Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Proceedings of AISTATS, PMLR 15:627–635. https://arxiv.org/abs/1011.0686
• Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. Advances in Neural Information Processing Systems (NIPS 2015). https://arxiv.org/abs/1506.03099
• Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016). Sequence Level Training with Recurrent Neural Networks. International Conference on Learning Representations (ICLR 2016). https://arxiv.org/abs/1511.06732
• Goyal, A., Lamb, A., Zhang, Y., Zhang, S., Courville, A., & Bengio, Y. (2016). Professor Forcing: A New Algorithm for Training Recurrent Networks. Advances in Neural Information Processing Systems (NeurIPS 2016). https://arxiv.org/abs/1610.09038
• Schmidt, F. (2019). Generalization in Generation: A closer look at Exposure Bias. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. https://aclanthology.org/P19-2049/
• Shao, Z., et al. (2024). DeepSeek-Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300
• Chen, B., et al. (2024). Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. https://arxiv.org/abs/2407.01392
• Hu, Y., et al. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. https://arxiv.org/abs/2506.08009