What is the BLEU Score (Bilingual Evaluation Understudy)?
The BLEU (Bilingual Evaluation Understudy) score is an algorithm used for evaluating the quality of text that has been machine-translated from one natural language to another. It was introduced by Kishore Papineni and colleagues at IBM in the paper "BLEU: a Method for Automatic Evaluation of Machine Translation," presented at the 2002 Annual Meeting of the Association for Computational Linguistics (ACL), and is one of the first metrics to claim a high correlation with human judgments of quality.
The BLEU score is calculated by comparing the machine-translated text (candidate) with one or more professionally human-translated texts (references). The quality is considered to be the correspondence between a machine's output and that of a human. The central idea behind BLEU is that "the closer a machine translation is to a professional human translation, the better it is".
How is the bleu score calculated?
The BLEU score assesses machine translation quality by measuring how closely it matches human translation. It combines n-gram precision and a brevity penalty for a comprehensive evaluation. N-gram precision counts the shared n-grams (1 to 4 words in sequence) between the machine and human translations, divided by the total n-grams in the machine translation. The brevity penalty applies when the machine translation is shorter than the reference, with a penalty factor of 1 if the machine translation is equal to or longer than the reference.
The score is calculated for individual translated segments—generally sentences—and then averaged over the whole corpus to reach an estimate of the translation's quality. Intelligibility or grammatical correctness are not taken into account.
The BLEU score is a number between 0 and 1, though it is commonly reported as a percentage on a 0–100 scale. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 (or 100) representing more similar texts. A BLEU score of 1 (or 100) means that the candidate sentence perfectly matches one of the reference sentences. However, even human translators do not achieve a perfect score.
What is a good BLEU score?
A good BLEU score varies based on the dataset, language pair, and preprocessing choices. Scores are most meaningful for comparing systems on the same test set under the same evaluation setup. Higher is generally better, but even strong BLEU scores can miss semantic errors or reward n-gram overlap that does not reflect true translation quality.
There is no universal threshold for a good BLEU score. Interpretation depends on domain, language pair, tokenization, smoothing choices, and the number of reference translations. Teams often pair BLEU with human review or metrics like COMET or chrF because BLEU does not directly measure meaning or factual accuracy.
What are its benefits and limitations?
The BLEU score offers several advantages, such as providing a quantitative measure for comparing machine and human translations, being computationally efficient, and incorporating n-gram precision to reflect certain linguistic structures.
However, it has limitations, including a lack of semantic analysis, potential overrating of grammatically correct but contextually inaccurate translations, dependency on reference translations that may not cover all correct possibilities, and not always being the most suitable metric for every application.
Additionally, the selection of n-gram length and the method for applying the brevity penalty can introduce subjectivity into the scoring process.