The Curse of Multilinguality in Large Language Models: Capacity Dilution, Tokenizer Fertility, and Representation Interference

Training a single transformer foundation model to process dozens or hundreds of languages is one of the central goals of modern natural language processing. In theory, massive multilingual pre-training unlocks positive cross-lingual transfer: low-resource languages gain syntactic, factual, and reasoning capabilities from the rich supervision available in high-resource languages such as English and Chinese.

In practice, scaling language coverage introduces severe architectural and optimization trade-offs. Beyond a certain threshold of languages, empirical performance across both high-resource and low-resource languages begins to deteriorate under fixed parameter budgets. This phenomenon, known as the "curse of multilinguality," represents a structural bottleneck in foundation model scaling.

Understanding the curse of multilinguality requires examining three interconnected mechanisms: parameter capacity dilution, tokenizer fertility imbalances, and representation interference caused by conflicting gradients during joint pre-training.

1. The Transfer-Dilution Trade-Off

The curse of multilinguality was formally characterized in massively multilingual machine translation (Aharoni et al., 2019) and large-scale cross-lingual representation learning with XLM-R (Conneau et al., 2020).

When a model is trained on a small number of related languages, joint training yields strong positive transfer. Shared vocabulary tokens, morphological regularities, and universal syntactic representations allow low-resource languages to leverage the semantic abstractions learned from high-resource data.

However, as the number of target languages N increases under a fixed parameter budget C_model, two competing forces emerge:

1. Positive Cross-Lingual Transfer: Generalization improvements gained by sharing parameter representations across semantically aligned corpora.
2. Capacity Dilution: The reduction in effective parameters dedicated to the specific idioms, grammar, and specialized vocabulary of any single language.

       Performance
           ▲
           │          Optimal Language Capacity
           │                 ┌───────┐
           │                ╱         ╲
           │   Positive    ╱           ╲    Capacity Dilution &
           │   Transfer   ╱             ╲   Negative Interference
           │  Dominates  ╱               ╲  Dominates
           │            ╱                 ╲
           │           ╱                   ▼
           └──────────┴──────────────────────────►
                     Low                  High
                            Number of Languages (N)

At low language counts, positive transfer outweighs capacity dilution. Beyond an inflection point, adding more languages forces the network to compress divergent syntactic systems and disjoint lexicons into the same finite weight matrices. The result is negative interference, where accuracy on high-resource languages drops without corresponding gains in low-resource performance.

2. Mathematical Mechanics of Capacity Dilution

In a dense transformer, every forward pass routes tokens through the same shared self-attention projections and multi-layer perceptron (MLP) weights. If a model with fixed capacity C_model must represent N distinct language distributions D_1, D_2, ..., D_N, the effective parameter capacity per language scales inversely:

$$C_{\text{eff}} \approx \frac{C_{\text{model}}}{N}$$

Because web-scale text corpora are heavily skewed toward a small fraction of dominant languages, naive empirical risk minimization over a multilingual corpus causes the model to allocate almost all capacity to English and high-resource European or East Asian languages.

Temperature-Scaled Sampling

To prevent low-resource languages from being completely starved during pre-training, multilingual datasets use temperature-scaled multinomial sampling. Given language proportions p_i = |D_i| / sum(|D_j|), the probability q_i of sampling a batch from language i is modified using a temperature parameter alpha in (0, 1]:

$$q_i = \frac{p_i^\alpha}{\sum_{j=1}^N p_j^\alpha}$$

• When alpha = 1, sampling matches the natural corpus distribution, heavily penalizing low-resource languages.
• When alpha approaches 0, sampling becomes completely uniform across all N languages.
• In practice, values of alpha between 0.2 and 0.7 (such as alpha = 0.3 in XLM-R and mBERT variants) are chosen to upsample tail languages.

While temperature scaling provides necessary gradient updates for underrepresented languages, it introduces a severe optimization dilemma:

1. Overfitting on Tail Corpora: Low-resource datasets are often small, containing repetitive web crawls or synthetic text. Upsampling them with low alpha forces the model to repeatedly cycle through low-quality data, memorizing noise and hallucinating patterns.
2. Under-Training on High-Resource Corpora: Downsampling high-resource text deprives the model of rich, high-quality reasoning and factual supervision, dragging down top-end benchmark scores in dominant languages.

3. The Tokenizer Fertility Tax and Vocabulary Allocation

A major physical bottleneck in multilingual language modeling is subword vocabulary allocation. In transformer architectures, the embedding matrix W_emb and the final unembedding projection layer scale linearly with vocabulary size V.

When V is bounded (historically between 32,000 and 64,000 tokens in early models like mBERT and GPT-2), the tokenizer must distribute merge rules across multiple scripts and alphabets (Latin, Cyrillic, Arabic, Devanagari, Hanzi, Kana, Hangul, Ge'ez, and others).

The Fertility Metric

Tokenizer efficiency for a language is measured by fertility (F), defined as the average number of subword tokens (T) required to represent a single standard word (W):

$$F = \frac{T}{W}$$

Because Byte-Pair Encoding (BPE) and WordPiece algorithms greedily select the most frequent byte sequences from the training corpus, English and dominant Latin-script languages capture whole words or long stems in single tokens (F_en between 1.1 and 1.3).

Non-Latin and morphologically rich languages suffer severe token fragmentation:

• Devanagari, Tamil, Bengali: Often split into individual Unicode bytes or character fragments (F between 3.5 and 7.0).
• Arabic and Hebrew: Semitic root-and-pattern morphology is fragmented across non-contiguous subwords (F between 2.5 and 4.5).
• Burmese, Khmer, Amharic: Frequently fall back to raw byte-level representation (F > 8.0).

System-Level Consequences of High Fertility

As shown by Petrov et al. (2023), high tokenizer fertility imposes compounding penalties across the entire deployment lifecycle:

1. Quadratic Compute Inflation: Transformer self-attention complexity scales as O(L^2), where L is sequence length. A sentence that requires 100 tokens in English may take 400 tokens in Hindi, requiring 16 times more attention FLOPs to process the exact same semantic content.
2. Effective Context Window Collapse: In a model with an 8,192-token context window, an English prompt can fit approximately 6,500 words of source text. A high-fertility language may fit fewer than 1,500 words before exhausting the active KV cache.
3. Inference Latency and Generation Slowdown: Autoregressive decoding generates text one token per step. High-fertility languages require proportionally more sequential decoding steps, multiplying time-to-last-token (TTLT) latency.
4. Economic Disparity: Commercial API providers charge per token. A user querying an LLM in an underrepresented language pays three to ten times more for identical semantic workflows compared to an English user.

4. Representation Interference and Gradient Conflict

Beyond capacity and tokenization constraints, multilingual joint training suffers from intrinsic optimization friction known as representation interference.

When optimizing model parameters theta across a batch containing diverse languages, the total loss gradient is the sum of per-language gradients:

$$g = \sum_{i=1}^N q_i \nabla_\theta \mathcal{L}_i(\theta)$$

Languages exhibit distinct syntactic structures (such as Subject-Verb-Object in English vs. Subject-Object-Verb in Japanese or Turkish), grammatical inflection paradigms, and semantic associations. When the model updates weights to minimize loss on language i, the computed gradient g_i can point in the opposite direction of the gradient g_j for language j:

$$\langle g_i, g_j \rangle = g_i^T g_j < 0$$

               g_i (Language i: SVO / Analytic)
                    ▲
                    │
                    │   θ_shared
                    └──────────►
                    │
                    │
                    ▼
               g_j (Language j: SOV / Agglutinative)
               
          Gradient Conflict: ⟨g_i, g_j⟩ < 0
          Causes parameter oscillation and feature erasure.

When gradients conflict, parameter updates intended to improve one language directly degrade the internal representations learned for the other. This causes optimization instability, parameter oscillation, and representational drift:

• Attention Head Specialization Conflict: Attention heads that attempt to track long-range subject-verb dependencies in German or Turkish are overwritten by local positional patterns learned from English or French.
• Polysemy and False Cognate Collisions: Identical surface strings across languages (such as "gift" meaning "present" in English but "poison" in German) collide in shared token embedding space unless sufficient contextual routing capacity is available in early transformer layers.

5. Architectural and Algorithmic Mitigations

Modern LLM research employs several engineering strategies to circumvent the curse of multilinguality.

1. Massive Vocabulary Scaling

Modern frontier models allocate substantially larger token budgets to mitigate the fertility tax. Moving from 32,000 tokens (Llama 1 and 2) to 128,000 tokens (Llama 3) or 151,000+ tokens (Qwen 2.5) directly reduces fertility for non-Latin scripts:

• Llama 2 (32,000 tokens): Baseline fertility (F ≈ 2.8 in CJK, F ≈ 6.2 in Indic scripts).
• Llama 3 (128,256 tokens): Approximately 50% fertility reduction in CJK and 60% reduction in Indic scripts.
• Qwen 2.5 (151,643 tokens): Approximately 65% fertility reduction across East Asian and South Asian languages.
• Gemma 2 (256,000 tokens): Up to 75% fertility reduction, bringing non-Latin script efficiency significantly closer to Latin benchmarks.

Expanding vocabulary size increases the static parameter footprint of the embedding tables, but it dramatically lowers runtime sequence lengths and preserves context window depth.

2. Sparse Mixture of Experts (MoE)

The most effective architectural defense against capacity dilution is sparse routing (such as Switch Transformers, DeepSeek-V3, and Mixtral).

In an MoE architecture with E experts, a learned router activates top-k experts per token:

$$y = \sum_{m \in \text{Top-}k} G(x)_m \cdot E_m(x)$$

MoE decouples total parameter capacity from per-token compute FLOPs. A model can maintain 300+ billion total parameters to house language-specific morphological and factual knowledge, while executing only 30 billion active parameters per token. Empirical analysis of multilingual MoEs demonstrates that gating networks naturally dedicate specific expert subsets to distinct language families and script groups, reducing gradient conflict in dense feed-forward blocks.

3. Modular Adapters and Language-Specific Subnetworks

Frameworks such as MAD-X (Pfeiffer et al., 2020) isolate language-specific representations using modular parameter blocks. Instead of sharing all weights across all languages:

• A frozen, highly capable base transformer captures universal semantic and syntactic logic.
• Lightweight, language-specific bottleneck adapters are trained independently for each language.
• At inference time, the host system loads the appropriate language adapter, eliminating negative cross-lingual interference by construction.

4. Continual Pre-Training with Vocabulary Swapping

Rather than forcing a single foundation model to achieve zero-shot perfection across 200 languages during initial pre-training, practitioners increasingly use targeted continual post-training.

A high-capacity English or Chinese base model is adapted to a target language by initializing new embedding rows for target-language subwords, aligning representations via parallel translation objectives, and performing causal pre-training on high-quality monolingual target corpora. This decoupled approach preserves the strong reasoning foundation of the base model while avoiding the capacity tax of universal joint optimization.

Sources

• Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020. arXiv:1911.02116
• Aharoni, R., Johnson, M., & Firat, O. (2019). Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges. NAACL 2019. arXiv:1907.05019
• Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. arXiv:2305.15425
• Pfeiffer, J., et al. (2020). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. EMNLP 2020. arXiv:2005.00052
• Rust, P., et al. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. ACL 2021. arXiv:2012.15613