Data Pruning and Core-Set Selection in Deep Learning: How EL2N, GraNd, and Memorization Dynamics Break Power-Law Scaling

Modern foundation models are trained on tens of trillions of tokens, requiring millions of GPU hours. Yet empirical analyses consistently reveal that massive portions of web-crawled corpora and large-scale vision datasets are either highly redundant, uninformative, or dominated by unlearnable noise. Standard empirical scaling laws (such as those formulated by Kaplan et al. and Chinchilla) model generalization error as a power-law function of total training samples ( $L(N) \propto N^{-\alpha}$ ). However, this scaling behavior assumes that training examples are sampled uniformly at random from an uncurated distribution.

Recent theoretical and empirical advances in data pruning and core-set selection demonstrate that power-law scaling is not a fundamental constant of neural network optimization, but rather an artifact of random data sampling. By scoring individual training examples using early optimization dynamics (such as Gradient Norm Distance, Error L2-Norm, and example forgetting statistics), practitioners can prune 30% to 50% of training data without degradation in generalization, or achieve exponential error scaling under optimal selection metrics.

Sample Valuation via Gradient Norms: GraNd and EL2N

To construct an optimal subset (or core-set) from an initial dataset $\mathcal{D} = \{(x_i, y_i)\ Handel}_{i=1}^N$ , one must quantify the marginal contribution of each training sample to the reduction of generalization error.

+-----------------------------------------------------------------------------+
|                           DATA PRUNING WORKFLOW                             |
|                                                                             |
|  [ Full Dataset D ]                                                         |
|         |                                                                   |
|         v (Early Training: Epochs 1-10)                                     |
|  [ Proxy / Early Checkpoint ] ---> Compute EL2N: || p(x; θ) - y ||_2        |
|         |                                                                   |
|         v (Rank & Filter by Budget)                                         |
|  +--------------------+---------------------+--------------------+          |
|  | Low Budget (Small) | Mid Budget (Medium) | High Budget (Large)|          |
|  | Keep Easy/Moderate | Keep Moderate/Hard  | Keep Hardest Only  |          |
|  +--------------------+---------------------+--------------------+          |
|         |                                                                   |
|         v                                                                   |
|  [ Pruned Core-Set S ] ===> Full Pre-Training (Exponential Error Scaling)   |
+-----------------------------------------------------------------------------+

Gradient Norm Distance (GraNd)

Paul et al. (2021) introduced the Gradient Norm Distance (GraNd) metric, which measures the expected norm of the gradient vector induced by a single training example on the model parameters $\theta_t$ at training step $t$ :

$\chi_t(x_i, y_i) = \mathbb{E}_{\theta_t} \left[ \|\nabla_\theta \mathcal{L}(f(x_i; \theta_t), y_i)\|_2 \right]$

Intuitively, GraNd measures how aggressively a single sample attempts to displace the model weights during optimization. In deep architectures parameterized by cross-entropy loss over pre-activation logits $z_i \in \mathbb{R}^C$ , the gradient with respect to parameter weight matrix $W_L$ in the final linear layer decomposes via the chain rule:

$\nabla_{W_L} \mathcal{L}(f(x_i; \theta_t), y_i) = (p(x_i; \theta_t) - y_i) \, h_{L-1}(x_i)^T$

where $p(x_i; \theta_t) = \text{softmax}(z_i)$ is the predicted class probability distribution, $y_i$ is the one-hot target vector, and $h_{L-1}(x_i)$ is the penultimate feature representation. Taking the Frobenius norm yields:

$\|\nabla_{W_L} \mathcal{L}(f(x_i; \theta_t), y_i)\|_F = \|p(x_i; \theta_t) - y_i\|_2 \, \|h_{L-1}(x_i)\|_2$

The Error L2-Norm (EL2N) Shortcut

While computing GraNd requires evaluating gradients through backpropagation, empirical analysis reveals that the penultimate feature norm $\|h_{L-1}(x_i)\|_2$ tends toward uniform values across samples after initial batch normalization or layer normalization steps. The variation in GraNd is predominantly driven by the residual error vector:

$\text{EL2N}_t(x_i, y_i) = \mathbb{E}_{\theta_t} \left[ \|p(x_i; \theta_t) - y_i\|_2 \right] = \mathbb{E}_{\theta_t} \left[ \sqrt{\sum_{c=1}^C (p_c(x_i; \theta_t) - y_{i,c})^2} \right]$

Paul et al. established that EL2N scores evaluated at very early stages of training (such as between epochs 5 and 10 of a 100-epoch training schedule across 5 to 10 random weight initializations) serve as an effective proxy for full-trajectory training dynamics. Pruning 50% of the lowest-EL2N examples on CIFAR-10 and 25% on CIFAR-100 results in zero loss in final test accuracy when training a fresh model from scratch on the pruned subset.

Forgetting Dynamics and the Memorization Frontier

While static snapshot scores like EL2N capture instantaneous error, tracking optimization trajectories across steps exposes temporal learning dynamics.

Data Pruning and Scaling Laws Transition

Unforgettable vs. Forgettable Examples

Toneva et al. (2019) quantified example difficulty by defining a forgetting event: an instance where a sample correctly classified at step $t$ is incorrectly classified at step $t+1$ following stochastic gradient updates on subsequent mini-batches.

Samples fall into two distinct operational regimes:

Unforgettable Examples: Samples that, once learned during optimization, are never forgotten. These typically reside deep within high-density class regions away from decision boundaries.
Forgettable Examples: Samples that repeatedly cycle between correct and incorrect classifications. These lie along complex decision margins or represent atypical feature combinations.

Toneva et al. proved that up to 30% of unforgettable examples in standard vision and language benchmarks can be completely removed from the training set with zero impact on generalization. The network retains sufficient geometric anchoring from the remaining support points to reconstruct the correct decision boundary.

Feldman's Long-Tail Memorization Theorem

A critical question arises: Can we simply discard all high-loss or high-forgetting examples as uninformative noise?

Feldman (2020) and Feldman and Zhang (2020) proved that on real-world distributions with heavy tails, exact memorization of atypical instances is mathematically necessary for achieving optimal generalization. If a sub-population appears with probability $p \ll 1$ , the optimal Bayes predictor must allocate model capacity to memorize those specific instances rather than abstracting them into a shared prototype.

Consequently, aggressive data pruning based solely on discarding high-difficulty samples degrades out-of-distribution robustness and tail-class recall. Effective core-set algorithms must distinguish between informative rare samples and corrupt label noise.

Statistical Physics of Data Pruning: Breaking Power-Law Scaling

The theoretical boundary of data pruning was formalized by Sorscher, Geirhos, Shekhar, Ganguli, and Morcos (2022). Using the replica method from statistical physics to analyze student-teacher perceptron setups in high dimensions, the authors evaluated how test error $\epsilon$ scales with pruned dataset size $N$ .

The Transition from Power-Law to Exponential Scaling

Under standard uniform random data sampling, the generalization error of linear and non-linear classifiers asymptotically decays as a power law:

$\epsilon(N) \propto N^{-\alpha}$

where $\alpha$ is governed by the dimension of the underlying data manifold.

However, when samples are filtered using a ranking function that scores example difficulty relative to the current student margin, Sorscher et al. proved that generalization error transitions to exponential decay:

$\epsilon(N) \propto \exp(-\beta N)$

where $\beta$ is a rate parameter determined by the quality of the pruning metric.

Generalization Error ε(N)
^
|   |     \  Uniform Random Sampling: ε(N) ∝ N^(-α)  [Power Law]
|       |         |           |             |               |   \             |     \             |       \  Optimal Data Pruning: ε(N) ∝ exp(-β N)  [Exponential Decay]
|         |           \_________
+------------------------------------------------------------>
                                             Dataset Size (N)

The Budget-Aware Pruning Pareto Frontier

A central discovery of Sorscher et al. is that the optimal data pruning strategy is strictly dependent on the available training budget:

Abundant Data ( $N \gg N_{\text{crit}}$ ): Retain highest-difficulty samples. Easy examples yield vanishingly small gradients; hard examples define fine decision margins.
Moderate Data ( $N \approx N_{\text{crit}}$ ): Retain medium-difficulty samples. Maximizes information density while avoiding boundary collapse.
Constrained Data ( $N \ll N_{\text{crit}}$ ): Retain easy and prototypical samples. Retaining only hard samples destabilizes convergence and destroys base feature representations.

Applying a single static pruning rule across varying budget regimes leads to severe performance collapse. When data is scarce, hard examples function as adversarial outliers; when data is abundant, easy examples function as dead compute.

Scaling Core-Set Selection to Trillion-Token Large Language Models

In autoregressive language model pre-training, calculating exact per-sample gradient trajectories across 15 trillion tokens is computationally prohibitive. Modern LLM pipelines utilize scalable proxy valuations.

Selection via Proxy (SVP)

Coleman et al. (2020) introduced Selection via Proxy (SVP), showing that sample difficulty rankings generated by a small, fast proxy model (such as a 100M-parameter transformer trained for 10% of total schedule) transfer directly to target architectures (such as 70B-parameter models). The relative ranking of sample informativeness remains invariant across architectural scale.

Reducible Holdout Loss (RHO-Loss)

Mindermann et al. (2022) introduced Reducible Holdout Loss Selection (RHO-Loss) to address the primary flaw of raw loss filtering in web-scale text.

Raw cross-entropy loss conflates two distinct phenomena:

High Information Content: Syntactically rich, complex factual knowledge that the model has not yet internalized.
Irreducible Noise: Cryptographic keys, garbled character encoding, random token strings, and broken HTML tags that no model can predict.

RHO-Loss computes the difference between the loss under a reference proxy model $\mathcal{M}_{\text{ref}}$ (trained on clean validation data) and the current target model $\mathcal{M}_{\theta}$ :

$\Delta \mathcal{L}(x) = \mathcal{L}(x; \mathcal{M}_{\text{ref}}) - \mathcal{L}(x; \mathcal{M}_{\theta})$

+-------------------+--------------------+------------------------------------+
| Reference Loss    | Target Loss        | RHO-Loss Category & Action         |
+-------------------+--------------------+------------------------------------+
| Low               | High               | High Reducible Loss -> KEEP        |
| Low               | Low                | Already Mastered -> PRUNE (Easy)   |
| High              | High               | Irreducible Noise -> PRUNE (Garbage)|
+-------------------+--------------------+------------------------------------+

By prioritizing tokens where the loss is reducible rather than simply high, RHO-Loss removes web crawl noise while retaining high-signal edge cases. Marion et al. (2023) validated this approach at scale, demonstrating that pruning 30% of pre-training tokens based on proxy scoring matches or exceeds the downstream benchmark performance of full-dataset baselines.

Implementation Architecture and Compute Economics

Integrating data pruning into production pre-training pipelines requires amortizing the scoring overhead against downstream training savings:

Proxy Scoring Pass: A compact reference model (1% to 5% target parameter size) processes candidate shards, calculating token-level or sequence-level EL2N or RHO-loss scores in a single high-throughput forward pass.
Stratified Indexing: Shards are partitioned into difficulty buckets using locality-sensitive indexing or quantile binning.
Dynamic Annealing Mixture: During pre-training, the data loader draws from difficulty buckets according to the optimal budget curve: starting with high-density prototypical data during initial warmup, transitioning to high-difficulty, high-EL2N samples in mid-training, and finishing with high-quality domain-specific tokens.

When training runs exceed $10^{24}$ FLOPs, investing 2% to 4% of total compute in an offline proxy valuation pass reduces pre-training duration by 25% to 40%, yielding substantial compute and latency savings while breaking standard power-law bounds.

Sources

Paul, M., Ganguli, S., & Dziugaite, G. K. (2021). Deep Learning on a Data Diet: Finding Important Examples Early in Training. NeurIPS 2021. arXiv:2107.07075
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. S. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022 Outstanding Paper. arXiv:2206.14486
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., & Gordon, G. J. (2019). An Empirical Study of Example Forgetting during Deep Neural Network Learning. ICLR 2019. arXiv:1812.05159
Feldman, V. (2020). Does Learning Require Memorization? A Short Tale about a Long Tail. STOC 2020. arXiv:1906.05271
Feldman, V., & Zhang, C. (2020). What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. NeurIPS 2020. arXiv:2008.03703
Coleman, C., et al. (2020). Selection via Proxy: Efficient Data Selection for Deep Learning. ICLR 2020. arXiv:1906.11829
Mindermann, S., et al. (2022). Prioritized Training on Points that are Rapidly Learnable, Worth Learning, and Not Yet Learnt. ICML 2022. arXiv:2202.00722
Marion, M., et al. (2023). When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale. NeurIPS 2023. arXiv:2306.14124