Subscribe
educational
7 min read

What Is BLEU? The 2002 Formula That Graded Machine Translation for a Generation

BLEU, introduced in 2002 by IBM researchers, measures machine translation quality by counting matching n-grams against human references. Despite its historical dominance, decades of criticism highlight flaws like paraphrase blindness, leading the field to adopt neural metrics like COMET for better…

Diagram explaining the four components of the BLEU machine translation evaluation formula including n-gram precision and brevity penalty
Diagram explaining the four components of the BLEU machine translation evaluation formula including n-gram precision and brevity penalty

The BLEU metric, introduced in a 2002 ACL paper by IBM researchers, automatically evaluates machine translation quality by measuring the fraction of word sequences that match human reference translations. Although it standardized reproducibility for two decades, it is currently criticized for ignoring semantic meaning, prompting the field to adopt neural alternatives like COMET and chrF.

For about twenty years, the standard way to announce a translation model was a sentence like this: our system improves English-to-French by 1.8 BLEU. The formula behind that number appeared in 2002, beat out every serious rival for the next decade, and then watched the research community slowly stop trusting it. Today BLEU sits in an odd position. The WMT conference that made its name has moved to other metrics, independent studies over two decades have documented its failures, and Google Cloud still scores custom translation models with BLEU as the only metric its console supports.

This is what the number is, how it is computed, what researchers found wrong with it, and what replaced it.

Counting matches, not meaning

BLEU, short for Bilingual Evaluation Understudy, was described in a 2001 IBM research report by Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu, then published at the ACL conference in 2002. It answers one narrow question: how many word sequences in the machine's translation also appear in a human reference translation.

The computation has four moving parts.

N-gram precision. The candidate translation and the reference are broken into contiguous word sequences of length 1, 2, 3 and 4. The metric measures what fraction of the candidate's n-grams occur in the reference. A unigram check rewards getting the words right; longer n-grams reward getting word order right. The original paper found 4-gram depth to correlate best with human judgments, with the shorter lengths tracking adequacy and the longer ones fluency.

Clipping. Raw precision is gameable, and the paper's own example shows why. A candidate consisting of seven copies of the word "the" scores perfect unigram precision against the reference "the cat is on the mat", because all seven tokens appear in the reference. Clipping caps each n-gram's count at its maximum occurrence in any reference, so "the" can contribute at most a couple of matches, and the sentence scores near zero at longer lengths.

The geometric mean. The four precision scores are combined multiplicatively, weighted equally by default. Because a geometric mean collapses when any component is zero, a candidate that misses every matching 4-gram is punished severely even with good word overlap.

The brevity penalty. A system that outputs only its three safest words would have high precision, so BLEU multiplies in an exponential penalty when the candidate is shorter than the references.

The result ranges from 0 to 1, usually quoted times 100. The paper itself cautioned that a score of 1 is not the bar: even professional human translators do not reliably reproduce a reference word for word, and two competent human translations of the same text typically score in the 60s or 70s at best. The metric is also corpus-level by design, computed over hundreds or thousands of sentences at once. Its use on single sentences is one of the misuses researchers have complained about since the early 2000s.

Two rows of tiles joined by matching lines: BLEU scores overlap between machine output and human reference, nothing more

Why it took over anyway

Before BLEU, evaluating machine translation meant human panels, which are slow and expensive, or word error rate borrowed from speech research, which punishes harmless reordering. The IBM authors explicitly said they fashioned their metric after word error rate, then fixed its weaknesses by counting matches with multiple human references instead of editing distance to one.

What won the field over was the economics. Human evaluation varies between annotators and costs dollars per sentence. BLEU is a script. Run the same output twice and you get the same number, so labs could iterate daily, and reviewers could check each other's claims. Within a few years it was attached to the WMT shared tasks, and by the era of neural translation it appeared in essentially every machine translation paper. Later model families inherited the habit: the GPT-3, PaLM, Whisper, FLAN and NLLB evaluations all include BLEU scores, according to Benjamin Marie's surveys of the metric's use and limits.

What two decades of criticism found

The objections are not opinion, they are measured results, and they stack up.

  • Paraphrase blindness. Translation has many valid renderings; BLEU only recognizes the ones the reference happened to use. A fluent, correct translation that chooses different words scores as if it were wrong. Callison-Burch and colleagues showed at EACL 2006 that millions of candidate variants receive identical BLEU scores, some clearly better translations than others, and concluded that BLEU improvement "is not sufficient to reflect a genuine improvement in translation quality".
  • Reachable peaks. Smith and colleagues (EAMT 2016, "Climbing Mont BLEU") built a system that optimized directly against the reference and produced translations that scored far above ordinary systems while being barely intelligible word salads.
  • Correlation artifacts. Mathur and colleagues won an ACL 2020 outstanding paper award for BLiMP, which showed that the strong BLEU-to-human correlations reported in earlier years were partly inflated by garbage outlier systems in the test data, and called for retiring BLEU as the default metric.
  • Quality moving in the opposite direction. Freitag and colleagues (2022) optimized a system for a newer learned metric, got translations that human raters judged significantly better, and watched their BLEU score drop by 8.9 points doing it.
  • Preprocessing sensitivity. Tokenization conventions move scores by more than many real system improvements. Matt Post's sacreBLEU (WMT 2018) exists because published BLEU scores were effectively incomparable, and standardizes tokenization plus a version signature. Ehud Reiter's 2018 structured review in Computational Linguistics collected the validity failures into one place.
A gauge above a conveyor, a hand pulling one item off the line: modern evaluation leans on learned metrics and human comparison, not a single dial

What replaced it

The field's answer was to learn the metric instead of hand-coding it. COMET, BLEURT and MetricX are neural models trained on large collections of human quality judgments; they read the source, the reference and the machine output, and score meaning rather than surface overlap. chrF, which counts character n-grams, survives as a cheap middle ground that forgives morphology and tokenization differences better than BLEU.

WMT itself switched. Its recent shared tasks rank submissions with neural metrics, and its 2024 metrics survey, which scored dozens of automatic metrics against human judgments, put BLEU ninth on average correlation at about 0.59, below chrF at 0.61, while the best learned metrics exceeded 0.72. LLM-based judges in the GEMBA family ranked near the top of the same comparison, and the 2025 and 2026 metrics tasks are dominated by reference-free neural scoring and model judges.

Vendors reflect the split. Google Cloud's console evaluates custom translation models with BLEU and nothing else, per its documentation, while the translation market's quality marketing has moved to blind comparative human evaluation: DeepL publishes a whitepaper whose thesis is that metrics like BLEU and COMET fall short of what matters in production and human comparison is what it actually stands behind.

How to read a BLEU number now

Three rules survive the criticism. A BLEU delta means something only inside one pipeline, one tokenizer, one language pair; across those boundaries it is not comparable. Differences of a point or two on a small test set are usually noise, which is why Marie and others push significance testing. And no absolute score means quality: 40 on news English-to-German says nothing about whether output is fit for support tickets.

Where BLEU remains useful is as a tripwire. Because its behavior is well understood and its cost is zero, a sudden two-point slide in a weekly regression run is a cheap signal that something broke in the data or the decoding, even if nobody would use that number to rank vendors. The metric did its job. It made translation evaluation reproducible at a time when the alternative was a panel of bilingual readers, and the field outgrew it once it could afford better. The number you still see in product slides is a fossil, and a careful reader now treats it like one.

Sources

Frequently asked questions

How is the BLEU score calculated?

BLEU combines four components: unigram to 4-gram precision, which counts matching word sequences; clipping, which caps matches at the reference count to prevent gaming; a geometric mean to combine these scores equally; and a brevity penalty that punishes outputs shorter than references.

Why is BLEU considered flawed?

Critics note that BLEU suffers from 'paraphrase blindness,' where correct translations with different wording score poorly. It also correlates poorly with human judgment when outlier systems are present, and optimizing for higher BLEU scores can sometimes result in lower human-perceived quality.

What has replaced BLEU in research?

The field has largely moved toward learned neural metrics such as COMET, BLEURT, and MetricX, which are trained on human quality judgments rather than simple word overlap. Metrics like chrF and LLM-based judges are also gaining prominence for their superior correlation with human evaluation.

Does Google Cloud use other metrics besides BLEU?

Google Cloud’s console currently supports only BLEU for evaluating custom translation models, unlike vendors that prioritize human comparative evaluation or other automated standards.

educational