Neural Collapse: How Simplex Equiangular Tight Frames Emerge at the Terminal Phase of Training

In classification tasks, deep neural networks exhibit an unexpected geometric simplicity during late-stage optimization. While the internal activations of early training appear high-dimensional and complex, the penultimate layer representations and linear classifiers converge toward an exact, symmetrical geometric structure known as Neural Collapse (NC).

First identified empirically by Papyan, Han, and Donoho (2020), Neural Collapse emerges during the Terminal Phase of Training (TPT). This regime begins when classification training error first vanishes to zero and optimization continues to drive the training loss toward zero.

During TPT, penultimate features across all classes collapse onto their class means, and those class means arrange themselves into a maximally symmetric geometric configuration: a Simplex Equiangular Tight Frame (ETF). Concurrently, the final-layer linear classifier weights align with these class means, rendering the classifier functionally equivalent to a nearest-centroid decision rule.

The Four Interconnected Phenomena

Neural Collapse consists of four mathematically distinct but structurally linked properties that manifest progressively as training loss approaches zero.

+--------------------------------------------------------------------------------+
|                             NEURAL COLLAPSE (NC)                               |
+--------------------------------------------------------------------------------+
|  NC1: Variability Collapse  | Within-class covariance Sigma_W -> 0             |
|  NC2: Simplex ETF Geometry  | Class means achieve equal norm & cos(theta) =    |
|                             | -1/(C - 1) maximal pairwise separation           |
|  NC3: Self-Duality          | Classifier weights align with class means:       |
|                             | W_norm = M_norm^T                                |
|  NC4: Nearest Class Center  | Linear logit argmax reduces to Euclidean argmin  |
|                             | ||h - mu_c||^2                                   |
+--------------------------------------------------------------------------------+

NC1: Within-Class Variability Collapse

Let $h_{c,i} \in \mathbb{R}^d$ denote the penultimate layer activation vector for the $i$ -th sample in class $c \in \{1, \dots, C\}$ , with $N_c$ samples per class. The class mean $\mu_c$ and global mean $\mu_G$ are defined as:

$\mu_c = \frac{1}{N_c} \sum_{i=1}^{N_c} h_{c,i}, \quad \mu_G = \frac{1}{C} \sum_{c=1}^C \mu_c$

The within-class covariance matrix $\Sigma_W$ and between-class covariance matrix $\Sigma_B$ are:

$\Sigma_W = \frac{1}{C} \sum_{c=1}^C \frac{1}{N_c} \sum_{i=1}^{N_c} (h_{c,i} - \mu_c)(h_{c,i} - \mu_c)^\top$

$\Sigma_B = \frac{1}{C} \sum_{c=1}^C (\mu_c - \mu_G)(\mu_c - \mu_G)^\top$

As training proceeds through TPT, within-class variability collapses toward zero relative to between-class variability:

$\text{Tr}(\Sigma_W \Sigma_B^\dagger) \to 0$

where $\Sigma_B^\dagger$ is the Moore-Penrose pseudoinverse. Individual activations $h_{c,i}$ contract onto their respective class centroids $\mu_c$ , eliminating intra-class variance in the penultimate representation space (Papyan et al., 2020).

NC2: Convergence to a Simplex Equiangular Tight Frame (ETF)

Centered class mean vectors $\bar{\mu}_c = \mu_c - \mu_G$ converge to the vertices of a standard Simplex Equiangular Tight Frame in $\mathbb{R}^d$ ( $d \ge C - 1$ ). A Simplex ETF is the unique geometric configuration that maximizes the minimum pairwise distance between $C$ unit vectors in Euclidean space.

Mathematically, the centered class means satisfy three conditions:

Equal Length: All centered class means possess identical Euclidean norms: $\|\bar{\mu}_c\|_2 = \|\bar{\mu}_{c'}\|_2 = \mu^*$ for all $c, c'$ .
Equiangularity: All pairwise angles between distinct class means are identical and maximally obtuse: $\frac{\bar{\mu}_c^\top \bar{\mu}_{c'}}{\|\bar{\mu}_c\|_2 \|\bar{\mu}_{c'}\|_2} = -\frac{1}{C - 1}$ for all $c \ne c'$ .
Tight Frame Condition: Let $M = [\bar{\mu}_1, \dots, \bar{\mu}_C] \in \mathbb{R}^{d \times C}$ . The matrix $M$ satisfies $M M^\top = \frac{C}{C - 1} \mu^{*2} \cdot P_{\text{span}(M)}$ , where $P_{\text{span}(M)}$ is the orthogonal projection operator onto the $(C - 1)$ -dimensional subspace spanned by the class means (Mixon et al., 2020).

NC3: Self-Duality

Let $W = [w_1, \dots, w_C]^\top \in \mathbb{R}^{C \times d}$ denote the weight matrix of the final linear classification layer. Under Neural Collapse, the normalized classifier vectors $\bar{w}_c = w_c / \|w_c\|_2$ align exactly with the normalized centered class means:

$\frac{w_c}{\|w_c\|_2} = \frac{\bar{\mu}_c}{\|\bar{\mu}_c\|_2} \quad \forall c \in \{1, \dots, C\}$

Consequently, the classifier weight matrix $W$ forms the identical Simplex ETF geometry as the representation centroids. The dual relationship between feature representation and linear separation achieves perfect geometric congruence: $W \propto M^\top$ (Papyan et al., 2020).

NC4: Nearest Class Center Simplification

In standard inference, the network assigns input representation $h$ to the class maximizing the linear logit:

$\hat{y}(h) = \arg\max_{c \in \{1, \dots, C\}} (w_c^\top h + b_c)$

Under NC1, NC2, and NC3, the bias terms $b_c$ become uniform ( $b_c = b_{c'}$ ), and the norms $\|w_c\|_2$ become equal. Expanding the squared Euclidean distance reveals:

$\|h - \mu_c\|_2^2 = \|h\|_2^2 - 2 \mu_c^\top h + \|\mu_c\|_2^2$

Because $\|\mu_c\|_2^2$ is constant across all $c$ , minimizing $\|h - \mu_c\|_2^2$ is strictly equivalent to maximizing $\mu_c^\top h \propto w_c^\top h$ . The affine classifier collapses to the non-parametric Nearest Class Center (NCC) decision rule:

$\hat{y}(h) = \arg\min_{c \in \{1, \dots, C\}} \|h - \mu_c\|_2$

Theoretical Foundations: The Unconstrained Features Model

To explain why gradient descent drives diverse architectures toward Simplex ETFs, theoretical work formulated the Unconstrained Features Model (UFM), also referred to as the Layer-Peeled Model (Mixon et al., 2020; Fang et al., 2021).

In the UFM, the complex nonlinear backbone is abstracted away. The penultimate layer features $H = [h_{1,1}, \dots, h_{C,N}] \in \mathbb{R}^{d \times (CN)}$ and the classifier parameters $(W, b)$ are treated as free optimization variables under standard loss objectives with weight decay ( $L_2$ regularization):

$\min_{W, b, H} \frac{1}{CN} \sum_{c=1}^C \sum_{i=1}^N \mathcal{L}(W h_{c,i} + b, y_c) + \frac{\lambda_W}{2} \|W\|_F^2 + \frac{\lambda_H}{2} \|H\|_F^2$

Cross-Entropy and Margin Maximization

Under cross-entropy loss, softmax probabilities encourage logits $z_{c} = w_c^\top h$ for the correct class to approach $+\infty$ while suppressing incorrect logits $z_{j} = w_j^\top h$ ( $j \ne c$ ). Constrained by weight decay ( $\|W\|_F^2 + \|H\|_F^2$ ), the optimization problem maps to maximizing the geometric separation margin on the hypersphere.

Because the sum of all centered vectors in an ETF is zero ( $\sum_{c=1}^C \bar{\mu}_c = 0$ ), the mutual repulsion among all $C$ classes reaches its global geometric equilibrium at $\cos(\theta) = -1/(C - 1)$ . Zhu et al. (2021) proved that under the UFM, every critical point is either a global minimizer satisfying NC1–NC4 or a strict saddle point with negative curvature, allowing standard gradient descent to reliably reach the Simplex ETF configuration.

Mean Squared Error Dynamics

Neural Collapse is not exclusive to cross-entropy loss. Han, Papyan, and Donoho (2022) demonstrated that training classification networks under Mean Squared Error (one-hot target regression) yields identical Neural Collapse properties along a predictable optimization trajectory on the central path.

Practical Implications in Modern Deep Learning

1. The Value of the Terminal Phase of Training

Prior to the discovery of Neural Collapse, continuing training past zero classification error was frequently regarded as superfluous or prone to overfitting. Neural Collapse demonstrates that TPT refines the internal geometric margin: as training loss decays from $10^{-2}$ to $10^{-6}$ , within-class variability continues to contract, and classifier alignment tightens, directly improving test set margin boundaries (Papyan et al., 2020).

2. Linear Probing and Representation Quality

When evaluating foundation models or vision-language backbones, practitioners frequently apply linear probing on frozen penultimate representations. The emergence of NC explains why linear classifiers are effective: representations have already structured class information into linearly separable, maximally distant angular clusters.

3. Class Imbalance and Minority Collapse

When class distributions are non-uniform ( $N_1 \gg N_C$ ), the exact symmetry of the Simplex ETF breaks. Under standard cross-entropy, majority classes consume a disproportionate fraction of the representation space, compressing minority classes into narrow subspaces or causing their centroids to merge with majority centroids. This phenomenon, known as Minority Collapse, explains why standard fine-tuning degrades sharply on long-tailed distributions and motivates class-balanced loss weighting and ETF-constrained classifiers (Fang et al., 2021).

4. Fixed-Classifier Architectures

Because the optimal terminal geometry of the classification layer is fixed to a Simplex ETF, recent architectures in vision and language domain adaptation employ fixed ETF classifiers. Instead of training $W$ via gradient descent, practitioners initialize $W$ as a pre-computed deterministic Simplex ETF matrix and freeze it throughout training. This eliminates classifier parameter redundancy, speeds up convergence, and prevents representation collapse during imbalanced fine-tuning (Yang et al., 2022).

Summary

Neural Collapse reveals an unexpected structural simplicity in deep classification networks. Across architectures including ResNets, Vision Transformers, and MLPs, optimization during the terminal phase of training drives penultimate activations into zero-variance centroids arranged in a Simplex Equiangular Tight Frame, while the final linear layer converges to an exact nearest-centroid matcher.