Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora

Pre-training data quality governs the downstream reasoning, factual accuracy, and sample efficiency of foundation large language models (LLMs). While early pre-training efforts relied on uncurated or lightly processed dumps from web repositories like Common Crawl, modern state-of-the-art models depend on multi-stage data curation pipelines that filter out noise, purge redundant sequences, eliminate toxic artifacts, and score educational value. Unfiltered web corpora contain severe pathologies: repetitive SEO spam, machine translation artifacts, tracking scripts, duplicated templates, and contaminated evaluation benchmarks. Training directly on raw data degrades reasoning benchmarks, triggers loss spikes, and increases verbatim memorization by orders of magnitude.

Modern data curation operates as a multi-tier pipeline that transforms raw petabyte-scale web crawls into high-density, multi-trillion-token training datasets. Understanding this pipeline requires examining the mechanical stages through which raw text passes: text extraction, heuristic filtering, exact and fuzzy deduplication, semantic clustering, model-based quality scoring, and benchmark decontamination.

1. Text Extraction and Heuristic Filtering

The first phase of pre-training curation converts raw web crawl archives (such as Common Crawl WARC files) into clean text documents and removes non-linguistic noise.

+------------------+     +--------------------+     +---------------------+
| Raw Web Crawl    | --> | Text Extractor     | --> | Heuristic Filters   |
| (WARC / HTML)    |     | (Trafilatura/HTML) |     | (Length, Punct, Char|
+------------------+     +--------------------+     +---------------------+
                                                               |
                                                               v
+------------------+     +--------------------+     +---------------------+
| Deduplication    | <-- | Perplexity Filter  | <-- | Language ID         |
| (Exact/MinHash)  |     | (KenLM / CCNet)    |     | (FastText > 0.65)   |
+------------------+     +--------------------+     +---------------------+

Document Extraction

Raw HTML parsing significantly alters downstream corpus quality. Early datasets often utilized default Common Crawl WET files (pre-extracted text), which frequently retain boilerplate navigation menus, broken layout fragments, and advertising banners. Contemporary curation pipelines employ dedicated extraction engines such as Trafilatura or Resiliparse. As demonstrated in the FineWeb technical report by Penedo et al. (2024), utilizing Trafilatura directly on WARC records produces higher linguistic coherence and structural fidelity than standard WET extractions.

Document and Line-Level Heuristics

Heuristic filters apply deterministic thresholding to eliminate low-quality documents without the computational overhead of running neural models over petabytes of text. Key heuristic filters include:

• Length Constraints: Documents with fewer than a minimum word count (typically 50 to 100 words) or exceeding maximum character thresholds are discarded. Extremely short documents lack sufficient discourse structure, while unnaturally long single documents often represent database dumps or concatenated log files.
• Character and Word Distributions: Documents are filtered if the ratio of symbols to words exceeds specific bounds, or if the mean word length falls outside human linguistic norms (e.g., mean word length below 3 or above 10 characters). Documents with excessive uppercase ratios (such as >20% capitalized characters) or high digit ratios are discarded to remove phone directories, financial tables, and machine-generated lists.
• Line-Level Filtering: Following rules formalized in C4 (Raffel et al., 2020) and refined in RefinedWeb (Penedo et al., 2023), lines that do not end in terminal punctuation marks (periods, question marks, exclamation points), lines composed entirely of bullet points or ellipsis sequences, and lines matching common web boilerplate ("Cookie Policy", "Terms of Service", "All Rights Reserved") are stripped before document assembly.
• Code and Markup Residue: Text containing unbalanced HTML tags, excessive curly braces, or JavaScript fragments is purged unless designated for code-specific pipelines.

Language Identification

Documents must be classified by natural language to avoid polluting monolingual or balanced multilingual corpora. Most pipelines utilize linear classifiers like FastText (Joulin et al., 2016) or Compact Language Detector (CLD2/CLD3). Documents where the primary target language probability falls below a strict threshold (commonly $p < 0.65$) are rejected.

Perplexity Filtering (CCNet)

Introduced by Wenzek et al. (2019) in CCNet, perplexity filtering uses an n-gram language model (typically a 5-gram KenLM model) trained on a curated target domain (such as Wikipedia) to compute the normalized cross-entropy of web documents:

$$\text{Perplexity Score} = \exp\left( -\frac{1}{N} \sum_{i=1}^N \log P(w_i \mid w_{i-n+1}^{i-1}) \right)$$

Documents exhibiting extreme perplexity are pruned:

• Excessively High Perplexity: Represents gibberish, OCR errors, broken character encodings, or ungrammatical text.
• Excessively Low Perplexity: Frequently flags synthetic boilerplate, repetitive keyword lists, or short templated phrases that do not reflect natural prose.

2. Deduplication Architectures: Exact, Fuzzy, and Semantic

Deduplication is among the most impactful curation steps for foundation models. In a landmark study, Lee et al. (2021) analyzed public NLP datasets like C4 and discovered that 1.68% of documents were near-duplicates, with some exact 50-token spans repeated over 60,000 times.

Duplicate text harms LLMs in three ways:

1. Memorization and Privacy: Models trained on duplicated sequences memorize text exponentially faster than on unique sequences (Carlini et al., 2022).
2. Training Instability: Batches containing repeated high-frequency tokens cause sudden gradient spikes and attention collapse.
3. Compute Waste: Training on redundant tokens burns GPU FLOPs without expanding the model's semantic frontier.

+-------------------------------------------------------------------------+
|                        DEDUPLICATION SPECTRUM                           |
+-------------------------------------------------------------------------+
| Exact Substring      | Suffix Arrays / Hashing | Exact matching >= 50t  |
| Near-Duplicate (LSH) | MinHash + Banding (LSH) | Jaccard similarity >=0.8|
| Semantic (SemDeDup)  | Bi-Encoder + Clustering | Cosine distance < eps  |
+-------------------------------------------------------------------------+

Exact Substring Deduplication (ExactSubstr)

Exact deduplication identifies and removes identical verbatim token sequences across the entire dataset. Lee et al. introduced ExactSubstr, an algorithm using memory-mapped Suffix Arrays:

• The corpus is concatenated into a single continuous token array.
• A suffix array is constructed to index all suffixes lexicographically.
• The algorithm scans the suffix array to locate maximal repeated substrings matching or exceeding a chosen threshold (typically $\ge 50$ tokens or 100 characters).
• Redundant substrings are excised while preserving at least one canonical occurrence.

Near-Duplicate Detection via MinHash LSH

Near-duplicate deduplication targets documents that share substantial content but vary by minor insertions, header changes, or formatting differences. This is achieved using MinHash Locality-Sensitive Hashing (LSH):

1. Shingling: Each document $D$ is decomposed into a set of overlapping n-grams (typically 5-grams or 13-grams):

$$S(D) = \{ \text{ngram}_1, \text{ngram}_2, \dots, \text{ngram}_m \}$$

1. MinHash Signatures: Using $k$ distinct universal hash functions $h_1, h_2, \dots, h_k$ (where $k$ is commonly 128 or 256), a signature vector $v(D)$ is formed by recording the minimum hash value observed across all shingles:

$$v(D)_j = \min_{s \in S(D)} h_j(s)$$ The probability that two documents share the same minimum hash for a random hash function equals their Jaccard similarity: $$P(h_j(A) = h_j(B)) = J(A, B) = \frac{|S(A) \cap S(B)|}{|S(A) \cup S(B)|}$$

1. Banding and LSH Indexing: The signature vector of length $k$ is partitioned into $b$ bands, each containing $r$ rows ($k = b \times r$). Two documents are treated as candidate duplicates if their hash sub-vectors match across all $r$ rows in at least one band. The probability of collision is:

$$P(\text{Candidate Pair}) = 1 - (1 - J(A, B)^r)^b$$ This yields an S-shaped selection curve where pairs with Jaccard similarity above a chosen threshold (e.g., $J \ge 0.8$) are matched with high probability while dissimilar pairs are discarded without pairwise comparison.

1. Per-Snapshot vs. Global Deduplication:

In the development of FineWeb, Penedo et al. tested both global MinHash (deduplicating across 96 Common Crawl snapshots spanning a decade) and per-snapshot MinHash (deduplicating within each monthly crawl independently). Interestingly, per-snapshot deduplication produced stronger downstream model performance on knowledge benchmarks. Global deduplication aggressively purged recurring factual statements across years, reducing diversity, whereas per-snapshot deduplication eliminated intra-batch redundancy while allowing natural repetition of historical concepts across time.

MinHash Signature: [ h1, h2, h3, h4 | h5, h6, h7, h8 | ... | hk-3, hk-2, hk-1, hk ]
                     \____________/   \____________/         \________________/
                         Band 1           Band 2                   Band b

Semantic Deduplication (SemDeDup)

Traditional MinHash LSH relies on surface-level n-gram overlap and fails to detect paraphrases, summaries, or documents expressing identical information in different vocabularies. To address this, Abbas et al. (2023) developed SemDeDup (NeurIPS 2023):

1. Embedding Generation: Every pre-filtered document is passed through a lightweight dense encoder (such as OPT-125M or Contriever) to generate a dense semantic embedding vector $e_i \in \mathbb{R}^d$.
2. K-Means Partitioning: To avoid quadratic $\mathcal{O}(N^2)$ pairwise distance calculations across billions of documents, the embedding space is clustered into $K$ centroids using spherical K-Means.
3. Intra-Cluster Pairwise Pruning: Within each cluster, cosine similarities between all document pairs are evaluated:

$$\text{sim}(e_i, e_j) = \frac{e_i \cdot e_j}{\|e_i\| \|e_j\|}$$ If $\text{sim}(e_i, e_j) \ge 1 - \epsilon$ (where $\epsilon$ is a tight distance threshold), document $j$ is flagged as a semantic duplicate and pruned.

SemDeDup demonstrates that up to 50% of web-scale datasets can be removed without degrading downstream model accuracy, allowing LLMs to achieve equal or superior perplexity with half the training tokens.

3. Model-Based Quality Filtering

Heuristic and deduplication pipelines eliminate obvious junk and redundancy, but they cannot evaluate pedagogical value, factual depth, or explanatory clarity. Modern pipelines incorporate learned neural classifiers to rank and filter candidate documents.

FastText and GBDT Quality Classifiers

Early approaches (such as the GPT-3 data pipeline by Brown et al., 2020 and RefinedWeb by Penedo et al., 2023) trained binary classifiers to distinguish between high-quality reference text and raw web crawls:

• Positive Reference Distribution: Curated datasets consisting of Wikipedia, peer-reviewed arXiv papers, selected books, and curated web links (e.g., Reddit posts with $\ge 3$ karma).
• Negative Distribution: Unfiltered Common Crawl web documents.
• Classifier Architecture: FastText linear models or Gradient Boosted Decision Trees (GBDT / LightGBM) trained on document features (perplexity, character ratios, vocabulary richness). Documents scoring below a tuned threshold are dropped.

Synthetic Annotation and LLM-as-a-Judge Distillation

The current state of the art in quality filtering uses frontier LLMs to annotate data quality, followed by distillation into high-throughput classifiers for corpus-wide filtering.

+--------------------------+
| 500k Sample Documents   |
+--------------------------+
             |
             v
+--------------------------+     Prompt: Rate educational value (0 to 5)
| Teacher LLM              | --> based on clarity, substance, structure,
| (e.g. Llama-3-70B-Inst.) |     and pedagogical utility.
+--------------------------+
             |
             v
+--------------------------+
| High-Throughput Student  | --> Scores 15+ Trillion Tokens
| (DeBERTa / FastText)     |     Retains Top Educational Tier (FineWeb-Edu)
+--------------------------+

1. Teacher LLM Scoring: In FineWeb-Edu, researchers prompted Llama-3-70B-Instruct to rate 500,000 web documents on a 0 to 5 educational score scale. The rubric assessed whether a page presented structured, verifiable, educational explanations versus conversational filler or shallow product listings.
2. Classifier Training: The synthetic annotations served as ground-truth labels to fine-tune a small encoder model (such as a DeBERTa or ModernBERT classifier).
3. Corpus-Scale Inference: The distilled classifier evaluated all 15 trillion tokens in FineWeb at high throughput. Filtering the dataset to retain documents with predicted educational scores $\ge 3$ produced FineWeb-Edu (1.3 trillion tokens).
4. Empirical Results: Models trained on the 1.3T-token FineWeb-Edu subset substantially outperformed identical models trained on 15T tokens of unfiltered web data on knowledge-intensive benchmarks including MMLU, ARC, and OpenBookQA.

Similarly, the DataComp-LM (DCLM) benchmark by Li et al. (NeurIPS 2024) systematically evaluated dozens of curation techniques across a 240T token pool. DCLM proved that model-based filtering was the single most decisive factor in pre-training efficiency: a 7B parameter model trained from scratch on 2.6T tokens of DCLM-Baseline reached 64% 5-shot accuracy on MMLU, matching models trained on over 6x more compute on inferior data mixtures.

4. Benchmark Decontamination and Mixture Blending

The final stage of data curation guarantees evaluation integrity and balances domain representation.

+---------------------+     +----------------------+     +----------------------+
| Filtered Documents  | --> | Benchmark            | --> | Domain Mixture       |
| (High Quality Tier) |     | Decontamination      |     | Blending & Weighting |
+---------------------+     +----------------------+     +----------------------+

Benchmark Decontamination

To ensure zero-shot and few-shot evaluation results reflect genuine reasoning rather than memorization, training datasets undergo rigorous decontamination audits against standard benchmarks (MMLU, GSM8K, HumanEval, MATH, ARC):

• N-gram Overlap Filtering: Candidate training documents are checked for exact matching 8-gram or 13-gram sequences with benchmark question-answer pairs.
• Substring Masking / Removal: When an overlap occurs, the entire document is either pruned from the training corpus or the matching span is redacted.
• Embedding Inversion Checks: Semantic similarity checks identify benchmark prompts that have been rephrased or translated in training dumps.

Domain Mixture Optimization

A foundation model corpus is rarely a single homogeneous dataset. Instead, it is blended from distinct domains:

• General Web Text (Educational/Curated): ~50-60%
• Source Code and Technical Repositories: ~15-25%
• Mathematics and STEM Formulations: ~10-15%
• Books and Long-Form Literature: ~5-10%
• Synthetic Instruction and Dialogue Data: ~5%

Techniques such as DoReMi (Xie et al., 2023) and RegMix (Liu et al., 2024) optimize domain weights automatically. DoReMi trains a small proxy model to identify domain loss discrepancies and runs minimax optimization to reweight domains, maximizing worst-case performance across evaluation tasks.

Architectural Summary of Pre-Training Data Curation

• Extraction (Trafilatura / Resiliparse): Structured WARC and HTML parsing that eliminates raw web boilerplate, JavaScript leftovers, and broken navigation trees.
• Heuristics (Rule-Based Thresholds): Character distributions, length bounds, punctuation checks, and line-level filters to discard unformatted spam and non-prose lists.
• Language Identification (FastText / CLD3): Linear n-gram classification discarding documents below target language confidence ($p < 0.65$).
• Perplexity Filtering (KenLM 5-gram / CCNet): Domain cross-entropy scoring against clean anchors (e.g. Wikipedia) to prune noisy gibberish and low-entropy repetitive boilerplate.
• Exact Deduplication (Suffix Arrays / ExactSubstr): Verbatim substring matching to remove duplicate spans $\ge 50$ tokens across the corpus, reducing memorization and stabilizing loss.
• Fuzzy Deduplication (MinHash LSH): 5-gram shingling and banded signature hashing to identify near-duplicate documents ($J \ge 0.8$) and prune templated variations.
• Semantic Deduplication (SemDeDup): Dense embedding clustering and cosine similarity pruning to eliminate paraphrased and semantically redundant passages.
• Quality Scoring (LLM Distillation / DeBERTa): Synthetic 0 to 5 educational scoring distilled from frontier models to isolate high-information, pedagogical tokens.
• Decontamination (8/13-gram Substring Matching): Exact overlap scanning against benchmark test splits to preserve evaluation validity.

Modern foundation model pre-training is as much an exercise in data engineering and algorithmic curation as it is in neural network architecture. By chaining heuristic filters, multi-tier deduplication, and neural quality classifiers, frontier labs convert raw, noisy web crawls into compact, high-density token streams that maximize model reasoning per GPU FLOP.

Sources

• Penedo, G., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557. https://arxiv.org/abs/2406.17557
• Lee, K., et al. (2021). Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499. https://arxiv.org/abs/2107.06499
• Abbas, A., et al. (2023). SemDeDup: Data-efficient learning at web-scale through semantic deduplication. arXiv:2303.09540. https://arxiv.org/abs/2303.09540
• Li, J., et al. (2024). DataComp-LM: In search of the next generation of training sets for language models. arXiv:2406.11794. https://arxiv.org/abs/2406.11794
• Wenzek, G., et al. (2019). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. arXiv:1911.00359. https://arxiv.org/abs/1911.00359
• Penedo, G., et al. (2023). The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv:2306.01116. https://arxiv.org/abs/2306.01116
• Carlini, N., et al. (2022). Quantifying Memorization Across Neural Language Models. arXiv:2202.07646. https://arxiv.org/abs/2202.07646
• Xie, S. M., et al. (2023). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. arXiv:2305.10429. https://arxiv.org/abs/2305.10429
• Liu, Z., et al. (2024). RegMix: Data Mixture as a Regression Problem. arXiv:2407.01492. https://arxiv.org/abs/2407.01492