Machine Unlearning in Large Language Models: How Gradient Ascent, Representation Misdirection, and Negative Alignment Purge Neural Memory

Large language models memorize massive amounts of training data, including copyrighted literature, personally identifiable information (PII), proprietary codebases, and dual-use knowledge spanning cyber exploits and biological hazards. When copyright holders, regulators, or safety researchers demand the removal of specific data, complete retraining from scratch is economically infeasible, often costing tens of millions of dollars in compute.

Machine unlearning aims to remove the influence of targeted data points or domains from a trained model's parameters while leaving general reasoning capabilities intact. Unlike standard fine-tuning or refusal guardrails, true unlearning alters parameter representations so the model behaves as if the targeted training samples were never seen.

The Machine Unlearning Problem

In classical supervised learning, exact unlearning can be achieved through deterministic data partitioning frameworks such as SISA (Sharded, Isolated, Sliced, Aggregated). In massive transformer models, parameters exhibit dense polysemantic superposition: millions of factual associations are distributed across shared attention projections and multi-layer perceptron (MLP) weights. Modifying a weight matrix to forget one specific document often alters adjacent semantic trajectories, causing catastrophic forgetting on unrelated tasks.

Formally, given a pre-trained model parameter set $\theta_0$ , a forget dataset $\mathcal{D}_{forget}$ , and a retain dataset $\mathcal{D}_{retain}$ , the objective of approximate unlearning is to produce updated parameters $\theta^*$ such that:

The probability distribution over $\mathcal{D}_{forget}$ matches that of a model trained strictly on $\mathcal{D}_{retain}$ .
The predictive performance on $\mathcal{D}_{retain}$ and general downstream benchmarks remains unchanged.
Computational cost scales with the size of $\mathcal{D}_{forget}$ rather than the entire pre-training corpus.

Gradient Ascent and Catastrophic Model Collapse

The most direct formulation of unlearning is Gradient Ascent (GA), introduced for language models by Jang et al. (2022). While standard training minimizes the negative log-likelihood (NLL) of target tokens, gradient ascent maximizes the loss on the forget set:

$\mathcal{L}_{GA}(\theta) = -\mathbb{E}_{(x, y) \sim \mathcal{D}_{forget}} \left[ \sum_{t=1}^{T} \log \pi_\theta(y_t \mid x, y_{<t}) \right]$

To prevent parameter drift, practitioners combine gradient ascent with a standard cross-entropy loss or Kullback-Leibler (KL) divergence penalty on the retain dataset $\mathcal{D}_{retain}$ :

$\mathcal{L}_{GA+KL}(\theta) = -\alpha \mathcal{L}_{NLL}(\mathcal{D}_{forget}; \theta) + \beta \mathcal{L}_{NLL}(\mathcal{D}_{retain}; \theta) + \gamma D_{KL}(\pi_{\theta_0}(\cdot \mid x) \parallel \pi_\theta(\cdot \mid x))$

Despite its conceptual simplicity, naive gradient ascent suffers from fundamental optimization failures:

Gradient Explosion and Divergence: As the probability $\pi_\theta(y_t \mid x, y_{<t})$ approaches zero, the gradient of the negative log-likelihood $\nabla_\theta \log \pi_\theta(z)$ grows without bound. This pushes parameters into extreme regions of loss space, inducing numerical instability.
Catastrophic Collapse: Unconstrained ascent frequently corrupts general language fluency. Instead of replacing targeted facts with plausible neutral tokens, the model begins generating repetitive punctuation, syntax errors, or degenerated token loops across all queries.
Surface-Level Suppression: The model rapidly learns to lower the probability of specific surface strings without erasing the underlying semantic knowledge, leaving internal representations accessible to indirect prompting.

Reference-Bounded Alignment: Negative Preference Optimization

To prevent the gradient explosion inherent in unconstrained gradient ascent, Zhang et al. (2024) proposed Negative Preference Optimization (NPO). NPO adapts the Direct Preference Optimization (DPO) framework to single-sided unlearning tasks where only negative (unwanted) completions exist without curated positive pairs.

NPO anchors the unlearning policy $\pi_\theta$ to the pre-unlearning reference model $\pi_{ref}$ (where $\pi_{ref} = \pi_{\theta_0}$ ). The loss function is formulated as:

$\mathcal{L}_{NPO}(\theta) = -\frac{2}{\beta} \mathbb{E}_{z \sim \mathcal{D}_{forget}} \left[ \log \sigma \left( -\beta \log \frac{\pi_\theta(z)}{\pi_{ref}(z)} \right) \right]$

Where $\sigma$ is the sigmoid function and $\beta$ controls the strength of the KL divergence penalty against the reference model.

The gradient of $\mathcal{L}_{NPO}$ introduces an implicit adaptive weighting term $W_\theta(z)$ :

$\nabla_\theta \mathcal{L}_{NPO}(\theta) = -\mathbb{E}_{z \sim \mathcal{D}_{forget}} \left[ W_\theta(z) \nabla_\theta \log \pi_\theta(z) \right]$

$W_\theta(z) = \frac{2 \pi_\theta^\beta(z)}{\pi_\theta^\beta(z) + \pi_{ref}^\beta(z)}$

When the model assigns high probability to the forget sequence relative to the reference model, $W_\theta(z) \approx 1$ , applying full gradient ascent pressure. As the model successfully unlearns the target text and $\pi_\theta(z)$ drops below $\pi_{ref}(z)$ , the weighting factor $W_\theta(z)$ smoothly decays to zero. This dynamic gating prevents gradient explosion, bounds parameter drift, and preserves language coherence without requiring constant manual tuning of gradient ascent step sizes.

Geometric Erasure: Representation Misdirection and Task Vectors

While token-level optimization operates on final output logits, representation-level methods target the intermediate activation manifolds inside transformer layers.

Representation Misdirection for Unlearning (RMU)

Introduced by Li et al. (2024) alongside the Weapons of Mass Destruction Proxy (WMDP) benchmark, Representation Misdirection for Unlearning (RMU) steers intermediate hidden representations rather than modulating cross-entropy loss.

RMU identifies a target intermediate layer $l$ and minimizes a dual-objective mean squared error (MSE) loss:

$\mathcal{L}_{RMU}(\theta) = \mathbb{E}_{x_f \sim \mathcal{D}_{forget}} \left[ \| h_\theta^l(x_f) - u \|_2^2 \right] + \alpha \mathbb{E}_{x_r \sim \mathcal{D}_{retain}} \left[ \| h_\theta^l(x_r) - h_{\theta_0}^l(x_r) \|_2^2 \right]$

Where:

$h_\theta^l(x)$ represents the hidden activation vector at layer $l$ for input $x$ .
$u$ is a fixed, randomly initialized target vector orthogonal to standard semantic trajectories.
$\alpha$ is a retention weighting hyperparameter.

By explicitly forcing the forget representations $h_\theta^l(x_f)$ toward the arbitrary vector $u$ , the network loses its ability to propagate domain-specific semantic cues into subsequent layers. Simultaneously, the retention loss penalizes any displacement of retain-set activations $h_\theta^l(x_r)$ from their baseline states $h_{\theta_0}^l(x_r)$ .

Task Vector Subtraction

Task vector arithmetic, formalized by Ilharco et al. (2022), offers a parameter-space unlearning approach. When a model is fine-tuned on a specific dataset $\mathcal{D}_{task}$ , the parameter displacement vector is defined as:

$\tau_{task} = \theta_{task} - \theta_{base}$

To unlearn the task or domain, the task vector is negated and subtracted from the current model weights with a scaling factor $\lambda$ :

$\theta_{unlearn} = \theta - \lambda \tau_{task}$

While effective for isolated domain fine-tunes where clean checkpoint diffs exist, task vector negation degrades in pre-trained foundation models where targeted knowledge is entangled across millions of interleaved pre-training steps rather than isolated in an additive delta.

Benchmarks and Evaluation Metrics

Evaluating unlearning requires measuring both forget efficacy and retain preservation. Simple accuracy checks on the forget set are insufficient, as a model might memorize standard refusal templates or output garbled text.

The TOFU Benchmark

The Task of Fictitious Unlearning (TOFU), developed by Maini et al. (2024), provides a standardized evaluation suite. TOFU generates 200 completely synthetic author autobiographies consisting of 4,000 question-answer pairs. Because these entities never existed in internet pre-training corpora, researchers can train a model on the synthetic biographies and evaluate unlearning at 1%, 5%, and 10% forget set sizes against an exact ground truth: the "Retain-Only" model trained from scratch without the forget samples.

TOFU evaluates unlearning across four primary metric dimensions:

Forget Quality: Evaluated using the Kolmogorov-Smirnov (KS) test to compare the p-value distribution of model loss on the forget set against the loss distribution of the Retain-Only ground truth model. A high p-value indicates that the unlearned model's loss profile is statistically indistinguishable from a model that never saw the data.
Model Utility: Evaluated on general reasoning benchmarks (MMLU, GSM8K) and world knowledge benchmarks to verify that unrelated capabilities remain intact.
Retain Set Accuracy: Accuracy and perplexity on the subset of author profiles that the model was explicitly instructed to keep.
Real Author Knowledge: Perplexity on biographies of actual historical figures to detect out-of-distribution entity degradation.

The WMDP Benchmark

For hazardous knowledge, the Weapons of Mass Destruction Proxy (WMDP) provides over 4,000 multiple-choice questions assessing actionable biosecurity, cybersecurity, and chemical weapon knowledge. Unlearning success is measured by driving WMDP accuracy down to random guessing (25% on 4-choice questions) while maintaining baseline accuracy on standard STEM benchmarks in MMLU.

The Persistence of Latent Knowledge and Relearning Risks

A primary vulnerability across current unlearning techniques is the distinction between output suppression and true weight erasure.

Latent Representation Probing

Even when an unlearned model outputs "I do not have information on this topic" or random predictions, linear probes trained on intermediate residual stream activations often extract the "forgotten" facts with high accuracy. Lynch et al. (2024) demonstrated that probing intermediate MLP layers of models unlearned via the "Who's Harry Potter?" protocol recovered specific character relationships and plot details that surface-level generations avoided.

The Relearning Vulnerability

When an unlearned model is fine-tuned on a tiny fraction of the original forget set (or semantically related out-of-domain text), the unlearned capabilities often re-emerge within a handful of gradient steps. This indicates that the underlying neural circuits and feature spaces were suppressed rather than dismantled.

For mission-critical safety and legal compliance, machine unlearning remains an active area of research. While reference-bounded methods like NPO and representation steering like RMU provide significant stability improvements over naive gradient ascent, verifiable and irreversible parameter erasure in overparameterized transformers remains an open theoretical challenge.