Representation Degeneration and Anisotropy in Large Language Models: Why Embeddings Collapse into Narrow Cones

In neural language modeling, high-dimensional vector spaces are expected to distribute token and sequence representations uniformly across their ambient dimensions. A space that utilizes all available geometric directions provides maximal representational capacity, allowing subtle semantic and syntactic differences between tokens to be resolved cleanly.

Empirical analysis across architectures from BERT and GPT-2 to modern frontier foundation models reveals the opposite behavior: learned token and sequence embeddings collapse into a narrow, low-dimensional cone. In this degenerate geometry, known as representation degeneration or embedding anisotropy, the majority of vectors point in virtually identical directions, causing pairwise cosine similarities between completely unrelated tokens to exceed 0.80 or approach 0.99 in upper layers.

This geometric collapse degrades text generation diversity, skews likelihood estimates for rare tokens, and impairs semantic retrieval in dense vector pipelines. Understanding why anisotropy arises from cross-entropy loss dynamics, how it propagates through transformer layers, and how post-processing and training regularizers resolve it is foundational to modern representation learning.

Defining Isotropy, Anisotropy, and Representation Degeneration

A vector distribution in $d$-dimensional space $\mathbb{R}^d$ is isotropic if its directional variance is uniformly distributed across all basis vectors on the unit hypersphere $S^{d-1}$. For a normalized dataset of $N$ embeddings $X = \{x_1, x_2, \dots, x_N\} \subset \mathbb{R}^d$ where $\|x_i\|_2 = 1$:

• Isotropic expectation: The mean vector $\mu = \frac{1}{N} \sum_{i=1}^N x_i \approx \mathbf{0}$, and the covariance matrix $\Sigma = \frac{1}{N} X^T X \approx \frac{1}{d} I_d$. Under strict isotropy, the expected cosine similarity between two randomly chosen independent vectors $x_i$ and $x_j$ ($i \neq j$) is zero:

$$\mathbb{E}_{i \neq j} [\cos(x_i, x_j)] = 0$$

• Anisotropic reality: The distribution exhibits a massive directional bias. The mean vector $\mu$ has a large norm ($\|\mu\|_2 \gg 0$), and the covariance matrix $\Sigma$ is dominated by a tiny fraction of its eigenvalues. The expected random pairwise cosine similarity is strictly positive and often close to unity:

$$\mathbb{E}_{i \neq j} [\cos(x_i, x_j)] \gg 0$$

Representation degeneration, formulated formally by Gao et al. (2019), describes this specific pathology where learned word embeddings and hidden states concentrate in a narrow cone centered around a shared direction vector, severely contracting the effective dimensionality of the representation space.

The Mathematical Mechanics of Representation Degeneration

Representation degeneration is not an incidental training artifact; it is an intrinsic consequence of optimizing standard cross-entropy loss with maximum likelihood estimation (MLE) over imbalanced natural language vocabularies.

1. The Cross-Entropy Push-Pull Dynamics

In standard autoregressive language modeling, given a context hidden state $h_t \in \mathbb{R}^d$ at step $t$, the probability of predicting token $w \in V$ from vocabulary $V$ is parameterized by a pre-softmax unembedding matrix $W \in \mathbb{R}^{|V| \times d}$ with token vectors $w_i \in \mathbb{R}^d$:

$$P(w_t = k \mid h_t) = \frac{\exp(h_t^T w_k)}{\sum_{j=1}^{|V|} \exp(h_t^T w_j)}$$

The cross-entropy loss for the ground-truth token $k$ at step $t$ is:

$$\mathcal{L}_t = -\ln P(w_t = k \mid h_t) = -h_t^T w_k + \ln \left( \sum_{j=1}^{|V|} \exp(h_t^T w_j) \right)$$

Computing the gradients of $\mathcal{L}_t$ with respect to the target embedding $w_k$ and non-target embeddings $w_j$ ($j \neq k$) illustrates the directional forces:

$$\frac{\partial \mathcal{L}_t}{\partial w_k} = -(1 - P(w_t = k \mid h_t)) h_t$$

$$\frac{\partial \mathcal{L}_t}{\partial w_j} = P(w_t = j \mid h_t) h_t \quad (j \neq k)$$

When updating the parameters with gradient descent (step size $\eta$):

• Target token $w_k$: The embedding is pushed in the direction of the hidden state $+h_t$.
• All non-target tokens $w_j$ ($j \neq k$): Every other token in the entire vocabulary is pushed in the opposite direction, $-h_t$, with a force proportional to its current predicted probability $P(w_t = j \mid h_t)$.

2. Zipf's Law and Vocabulary Imbalance

Natural language follows a heavy-tailed Zipfian distribution. A small number of frequent words (punctuation, determiners, conjunctions) account for the vast majority of target occurrences, while the long tail of vocabulary words appears infrequently.

For a rare or moderately frequent word $w_r$:

1. It is selected as the ground truth token $k$ only a negligible fraction of the time.
2. It serves as a non-target token $j$ on almost every single training step across the entire dataset.

Consequently, the cumulative gradient updates for $w_r$ are dominated by negative updates:

$$\Delta w_r = -\eta \sum_{t=1}^T \frac{\partial \mathcal{L}_t}{\partial w_r} \approx -\eta \sum_{t=1}^T P(w_t = r \mid h_t) h_t$$

Because the probabilities $P(w_t = r \mid h_t)$ are strictly positive, $w_r$ is driven into the negative cone generated by the convex hull of the hidden states $\{h_t\}$.

3. The Convex Hull Theorem

Gao et al. (2019) established that if there exists a subset of tokens that never appear (or rarely appear) in a training partition, minimizing the training loss forces their embeddings to lie within the intersection of negative half-spaces defined by all contextual hidden states:

$$\mathcal{H}^- = \{ v \in \mathbb{R}^d \mid v^T h_t < 0, \; \forall h_t \in \mathcal{C} \}$$

where $\mathcal{C} = \text{conv}(\{h_t\}_{t=1}^T)$ is the convex hull of hidden states. As the training set grows, the convex hull $\mathcal{C}$ expands, causing the dual negative cone $\mathcal{H}^-$ to shrink. All rare and tail tokens are compressed into this vanishingly narrow directional cone.

Layer-Wise Propagation Across Transformers

Anisotropy is not limited to input and output embedding matrices; it compounds throughout the depth of transformer networks.

In a landmark empirical study, Ethayarajh (2019) analyzed the layer-by-layer geometry of BERT, ELMo, and GPT-2, measuring the average cosine similarity between randomly sampled words across different layers:

• Input Layers: Embeddings start with moderate directional diversity (average cosine similarity between 0.15 and 0.35).
• Intermediate Layers: Self-attention layers contextualize representations by mixing token embeddings with attention-weighted sums of surrounding tokens, pulling representations toward common subspace directions.
• Top Layers: In autoregressive models like GPT-2, the average cosine similarity of random word pairs reached 0.95 to 0.99 in the final layer. Two entirely unrelated words sampled from distinct contexts were nearly collinear.

This phenomenon explains why contextualized word representations exhibit high intra-sentence similarity and low self-similarity across contexts after adjusting for anisotropy: upper layers become dominated by static positional and grammatical bias vectors that dwarf token-specific semantic components.

Measuring Representation Degeneration and Anisotropy

Several formal metrics quantify the severity of representation collapse in deep models:

1. Average Pairwise Cosine Similarity

The simplest direct measure of directional concentration is the empirical mean cosine similarity across $M$ randomly sampled vector pairs:

$$\text{AvgCos}(X) = \frac{1}{M} \sum_{(i,j), i \neq j}^M \frac{x_i^T x_j}{\|x_i\|_2 \|x_j\|_2}$$

In a fully isotropic space, $\text{AvgCos}(X) \approx 0$. In severely anisotropic spaces, $\text{AvgCos}(X) \to 1$.

2. Singular Value Spectrum and Spectral Decay

Let $H \in \mathbb{R}^{N \times d}$ be the centered matrix of hidden representations, with singular value decomposition $H = U \Sigma V^T$, where $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_d \ge 0$ are the singular values.

In anisotropic representations:

• The first singular value $\sigma_1$ is disproportionately large compared to subsequent values ($\sigma_1^2 \gg \sum_{i=2}^d \sigma_i^2$).
• The first 1 to 3 principal components account for over 50% to 80% of total variance.
• The singular values decay exponentially rather than exhibiting a flat, uniform spectrum.

3. Effective Rank ($R_{\text{eff}}$)

Effective rank measures how many orthogonal dimensions are actively utilized by the representation matrix:

$$p_i = \frac{\sigma_i}{\sum_{j=1}^d \sigma_j}, \quad R_{\text{eff}}(H) = \exp \left( -\sum_{i=1}^d p_i \ln p_i \right)$$

For an embedding dimension of $d = 4096$:

• An isotropic representation utilizes nearly all dimensions ($R_{\text{eff}} \approx 4000$).
• A degenerated representation collapses to a low-dimensional manifold ($R_{\text{eff}} < 50$).

Downstream Consequences on Model Performance

Embedding anisotropy introduces critical failure modes across NLP and LLM workloads:

1. Cosine Similarity Distortion:

In dense retrieval, vector search algorithms (HNSW, Flat IP, IVF) rely on cosine similarity or dot products. When all vectors share a large constant offset component $\mu$, the cosine metric is dominated by $\mu$, compressing the dynamic range of semantic similarity scores into a tiny band (e.g., [0.82, 0.88]). Minor semantic variations are masked by directional noise.

1. Repetition and Degenerate Text Generation:

In autoregressive generation, narrow cone geometries in the unembedding matrix $W$ mean that multiple candidate tokens produce nearly identical logits for a given hidden state. When temperature sampling or greedy decoding is applied, the model struggles to sharply separate contextually appropriate tokens from generic, high-frequency tokens, promoting repetitive loops.

1. Poor Zero-Shot Sentence Transfer:

Averaging unadjusted token embeddings from BERT or RoBERTa produces poor sentence representations for Semantic Textual Similarity (STS) benchmarks, often performing worse than simple GloVe averaging unless isotropy is restored.

Algorithmic and Architectural Mitigations

Several distinct techniques have been developed to counteract representation degeneration, ranging from post-processing transforms to loss regularizers and contrastive objectives.

1. Post-Processing: Centering and Whitening

For frozen, pre-trained models, post-processing can restore isotropy without retraining:

• Mean Centering: Subtract the global empirical mean $\mu = \frac{1}{N} \sum_{i=1}^N x_i$:

$$\tilde{x}_i = x_i - \mu$$ This eliminates the shared bias direction that shifts all vectors into a positive orthant.

• BERT-Whitening (Su et al., 2021): Transforms representations to have zero mean and an identity covariance matrix:

$$\tilde{x}_i = (x_i - \mu) W_{\text{whiten}}, \quad \text{where } W_{\text{whiten}} = U \Lambda^{-1/2}$$ Here, $\Sigma = U \Lambda U^T$ is the eigendecomposition of the covariance matrix. Whitening equalizes variance across all directions, directly boosting STS benchmark scores by 10 to 30 percentage points over raw averaged embeddings.

2. Regularization: Cosine Spectrum Control

During pre-training, Gao et al. (2019) proposed Cosine Spectrum Control, which adds an explicit regularization penalty to prevent singular value collapse in the embedding matrix $W$:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{MLE}} + \lambda \sum_{i=1}^{|V|} \sum_{j \neq i}^{|V|} \max \left( 0, \cos(w_i, w_j) - \gamma \right)$$

Alternatively, Wang et al. (2020) proposed singular value regularization to penalize the ratio $\frac{\sigma_1}{\sigma_d}$, ensuring that energy remains distributed across higher-order singular components during training.

3. Contrastive Learning: SimCSE and InfoNCE Alignment

Gao, Yao, & Chen (2021) demonstrated that unsupervised contrastive learning (SimCSE) directly solves anisotropy by optimizing the InfoNCE loss:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(h_i, h_i^+) / \tau)}{\sum_{j=1}^B \exp(\text{sim}(h_i, h_j^+) / \tau)}$$

Wang and Isola (2020) showed that InfoNCE optimizes two competing geometric properties:

1. Alignment: Pulls semantically identical pairs ($h_i, h_i^+$) close together.
2. Uniformity: Pushes all negative pairs ($h_i, h_j$) apart, asymptotically forcing the embedding distribution to match a uniform distribution on the unit hypersphere $S^{d-1}$.

Unsupervised SimCSE uses standard dropout masks inside the transformer encoder as data augmentation, completely eliminating anisotropy in sentence embeddings and establishing high semantic discriminability.

4. Architectural Adjustments

Modern large language models employ architectural patterns that reduce degeneration:

• Untied Input and Output Embeddings: Decoupling the input embedding matrix $W_{\text{embed}}$ from the output projection head $W_{\text{unembed}}$ prevents gradient conflicts between vocabulary input lookups and softmax target classification.
• Pre-LayerNorm and RMSNorm: Applying normalization immediately before each attention and MLP block stabilizes activation magnitudes, preventing the runaway growth of dominant directional components.
• Rotational Coordinate Invariance (QuaRot / SpinQuant): Randomized orthogonal rotations ($X' = X R$, where $R^T R = I$) preserve inner products and cosine distances while scattering anisotropic outlier channels across all coordinates, facilitating outlier-free low-bit quantization.

Comparison of Anisotropy Mitigation Strategies

Evaluating the primary approaches across key operational parameters:

• Mean Centering:
• Operational Stage: Post-processing (Inference only)
• Compute Overhead: Negligible ($O(d)$ per vector)
• Training Required: No
• Primary Mechanism: Subtracts global centroid vector $\mu$
• Limitations: Does not equalize non-uniform variance across orthogonal dimensions
• BERT-Whitening (ZCA / SVD):
• Operational Stage: Post-processing (Inference only)
• Compute Overhead: Very low ($O(d^2)$ projection)
• Training Required: No
• Primary Mechanism: Linearly transforms covariance matrix to identity $I_d$
• Limitations: Sensitive to estimation noise in covariance matrices of out-of-domain data
• Cosine Spectrum Control:
• Operational Stage: Pre-training / Fine-tuning
• Compute Overhead: Moderate (pairwise comparisons or mini-batch SVD)
• Training Required: Yes
• Primary Mechanism: Penalizes high cosine similarity or steep singular value decay
• Limitations: Requires hyperparameter tuning of regularization weight $\lambda$ and threshold $\gamma$
• Contrastive Learning (SimCSE / InfoNCE):
• Operational Stage: Post-training / Continual Fine-tuning
• Compute Overhead: Moderate ($O(B^2 d)$ batch contrastive loss)
• Training Required: Yes
• Primary Mechanism: Explicitly optimizes hyperspherical uniformity and alignment
• Limitations: Requires dedicated contrastive fine-tuning pass on unlabelled or paired text
• Untied Embeddings and RMSNorm:
• Operational Stage: Model Architecture Pre-training
• Compute Overhead: Negligible during training; slight parameter increase
• Training Required: Yes (Pre-training architecture design)
• Primary Mechanism: Decouples input/output gradient dynamics and normalizes residual streams
• Limitations: Must be specified prior to model pre-training

Sources

• Gao, J., He, D., Tan, X., Qin, T., Wang, L., & Liu, T. Y. (2019). Representation Degeneration Problem in Training Natural Language Generation Models. International Conference on Learning Representations (ICLR).
• Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., & Gu, Q. (2020). Improving Neural Language Generation with Spectrum Control. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
• Su, J., Cao, J., Liu, W., & Ou, Y. (2021). Whitening Sentence Representations for Better Semantic Vector Space. arXiv preprint arXiv:2103.15316.
• Gao, T., Yao, X., & Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Li, B., Zhou, H., He, J., Wang, M., Yang, Y., & Li, L. (2020). On the Sentence Embeddings from Pre-trained Language Models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Wang, T., & Isola, P. (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. International Conference on Machine Learning (ICML).