Representation Degeneration and Anisotropy in Large Language Models: Why Embeddings Collapse into Narrow Cones
In neural language modeling, high-dimensional vector spaces are expected to distribute token and sequence representations uniformly across their ambient dimensions. A space that utilizes all available geometric directions provides maximal representational capacity, allowing subtle semantic and syntactic differences between tokens to be resolved cleanly. Empirical analysis across architectures from BERT and GPT-2 to modern frontier foundation models reveals the opposite behavior: learned token a
















