BLEU vs Learned Metrics for Translation Quality
Compare BLEU with learned metrics (COMET, BLEURT, BERTScore) for translation QA; combine scores with deterministic checks and human review.
In this article
- Beyond COMET: New Approaches to Evaluating Machine Translation Quality in Game Localization
- BLEU and Learned Metrics: Inputs, Strengths, and Costs
- BLEU: Track Lexical Changes Across Releases
- COMET, BLEURT, and BERTScore: Assess Meaning
- Errors BLEU and Learned Metrics Can Miss
- Test Meaning-Changing Errors Separately
- Check Document Structure and Completeness
- Build an Auditable Translation Review Workflow
- Calibrate Scores Against Domain Review
- Route Exceptions and Record Approval Evidence
- Assign Owners and Test System Changes
- Conclusion: Combine Metrics with Controlled Acceptance
- FAQs
- What should we do when BLEU and learned metrics disagree?
- How much reviewer-rated data do we need to set thresholds?
- How often should we recalibrate translation quality checks?
- Frequently asked questions
- What should we do when BLEU and learned metrics disagree?
- How much reviewer-rated data do we need to set thresholds?
- How often should we recalibrate translation quality checks?
Originally published October 25, 2026.
I use BLEU to track wording changes and learned metrics to help flag meaning errors - not to approve translations. Even one wrong number or a switch from “must not” to “may” can change an instruction despite a high score.
Here’s how I compare the options:
- BLEU: Checks wording overlap against references; fast on CPUs and best for tracking results across a document set.
- COMET: Uses the source and translation, plus a reference for reference-based models, to help assess meaning.
- BLEURT: Uses a translation and reference to predict quality.
- BERTScore: Compares contextual word representations to allow for paraphrasing; it is not a supervised quality predictor.
Quick Comparison
| Metric | Reference needed? | Main use | Processing |
|---|---|---|---|
| BLEU | Yes | Wording-change tracking | CPU; no model |
| COMET | Depends on model | Meaning-based review triage | Neural model |
| BLEURT | Yes | Quality prediction | Neural model |
| BERTScore | Yes | Similarity despite paraphrasing | Transformer model |
I set separate thresholds using bilingual reviewers’ ratings for each language pair and document type. Scores from different metrics or checkpoints are not interchangeable.
Before release, I pair scoring with checks for terminology, values, omissions, and document structure. I keep versioned review records, retest system changes, and require human approval for safety-critical content regardless of score.
Beyond COMET: New Approaches to Evaluating Machine Translation Quality in Game Localization
BLEU and Learned Metrics: Inputs, Strengths, and Costs
Choose a metric based on what you need to detect: changes in wording, shifts in meaning, or similarity despite paraphrasing. The required inputs vary: BLEU needs one or more reference translations, while reference-free COMET uses only the source and candidate.[3][4]
| Metric | Required inputs | Strength and appropriate use | Limits | Runtime and infrastructure |
|---|---|---|---|---|
| BLEU | Candidate and one or more references | Lexical comparison for corpus-level release monitoring | Measures overlap, not meaning | Fast CPU processing; no model checkpoint |
| COMET | Source and candidate; reference for reference-based models | Adequacy assessment for ranking and review triage | A single score rarely explains the error | Neural inference; GPU batching |
| BLEURT | Candidate and reference | Semantic and fluency signals for reference-based review | Limited explanation of individual scores | Neural inference; model memory |
| BERTScore | Candidate and reference | Similarity comparison that tolerates paraphrasing | Token alignment does not explain errors | Transformer inference; encoder-dependent cost |
Resource needs depend on implementation and model size, not just the metric name. Sequence length, batch size, hardware, and evaluation volume also affect latency and memory use. Before budgeting infrastructure, benchmark the selected checkpoint on representative documents. Neural metrics have no universal cost ranking.
Use these differences to select a metric for release monitoring, ranking, or human review triage - not acceptance.
BLEU: Track Lexical Changes Across Releases
BLEU measures clipped n-gram precision, usually over 1- to 4-grams, and applies a brevity penalty. That penalty does not check completeness. For low-cost release tracking, use trusted references and aggregate results at the corpus level.
Keep tokenization, normalization, smoothing, the reference set, and the implementation fixed. Short sentences offer few matches, making sentence-level BLEU unreliable. Valid paraphrases can also lose a large amount of credit.[2][4][5][7]
COMET, BLEURT, and BERTScore: Assess Meaning
These metrics help when wording can change but meaning must stay the same.
Reference-based COMET uses the source, candidate, and reference to predict quality based on training against human judgments. Reference-free variants, including COMET-QE and Kiwi, use only the source and candidate. Treat those results as a separate evaluation stream.
BLEURT adapts a pretrained encoder through synthetic-data training and fine-tuning on human ratings. Standard BERTScore instead matches contextual token embeddings. It uses learned representations but is not, by design, a supervised quality predictor.
Choose checkpoints suited to your language pair and domain. Training-data bias and domain mismatch can weaken results. A strong correlation on benchmark data also does not prove that a metric will catch critical technical errors.[8][9]
Do not compare scores across metrics or checkpoints.[3][4][6] Keep separate thresholds and record checkpoint and software versions. Treat changes to encoders, tokenization, reference sets, or aggregation as a break in the comparison history.
Errors BLEU and Learned Metrics Can Miss
Scores can hide critical defects. Use them to help find the errors that matter most in regulated translation, but pair them with targeted checks.
| Error category | BLEU | Learned metrics | Blind spot | Supplemental control |
|---|---|---|---|---|
| Terminology | Favors reference wording | Can miss glossary violations | Glossary noncompliance | Domain lexicon/glossary check |
| Numbers, units, dates, and identifiers | Limited sensitivity to digit shifts | Better at assessing meaning, but unstable on short strings | Wrong value, unit, date, or part ID | Rule-based regex and unit validation |
| Negation and modality | Sensitive to word order | Assess semantic adequacy | Negation reversal or changed obligation | Meaning-changing error tests |
| Fluency and coherence | Assess surface features only | Align more closely with human judgment | Document-level coherence | Document-level context review |
| Formatting and document fidelity | No structural awareness | No structural awareness | Markup or layout failure | Structural and markup checks |
Test Meaning-Changing Errors Separately
In regulated content, value changes are the highest-risk misses. Test critical values and instructions separately from overall quality. Include controlled changes to:
- Tolerances
- Part identifiers
- Substitutions between “must not” and “may”
Check whether each defect gets flagged. An aggregate score alone won't tell you whether a check caught a specific error.
Preserve source units and values during comparison, and explicitly validate dates and identifiers. Match each value to its surrounding instruction or field. A matching term elsewhere in the document isn't enough.
Check Document Structure and Completeness
Regulated documents must preserve both meaning and structure, including required fields. Check table alignment, required sections, signatures, and workflow fields separately. Problems with structure or markup can hide missing or misaligned content.
Accuracy, completeness, and approval readiness need separate checks. Run structural validation before approval.
Build an Auditable Translation Review Workflow
Auditable Translation Review Workflow
When scoring flags a risk, send the result through a controlled review workflow.
Calibrate Scores Against Domain Review
Create versioned evaluation sets for each regulated document type. Track BLEU against saved references, and pair it with a learned metric validated for the language pair and domain.
Set routing thresholds using bilingual reviewers’ severity ratings. Combine those scores with deterministic checks for terminology, numbers, units, dates, identifiers, and omissions.
Route Exceptions and Record Approval Evidence
Connect intake, translation, evaluation, exception routing, and approval in Microsoft 365 or your existing enterprise systems. Route failed checks and conflicting metric signals to bilingual reviewers. Add subject-matter review based on document risk and policy.
Safety-critical content must pass explicit human review gates, regardless of score. Preserve evidence for each decision, including document and reference versions, metric versions, configurations, scores, thresholds, decisions, overrides, and unresolved exceptions.
Tie approval to the exact version reviewed. Any edits should reopen the required checks. Apply role-based access controls, and assess data residency and permitted data use before sending content to external evaluators.
Assign Owners and Test System Changes
Assign owners for evaluation sets, thresholds, exception queues, and release approval. Define clear escalation rules so each owner knows when and where to send an issue.
Before changing a translation system or metric, assign an owner to validate the change and record the result.
Conclusion: Combine Metrics with Controlled Acceptance
Use BLEU to track lexical regressions and learned metrics such as COMET to prioritize semantic review. Add deterministic checks for terminology, numbers, units, dates, identifiers, and omissions. Each score should guide routing - not grant approval.
Run a pilot on representative documents before rollout. After launch, keep versioned audit evidence and periodically sample auto-approved translations to check for drift. Expand only when sampling confirms that the workflow still aligns with reviewer judgment.
FAQs
What should we do when BLEU and learned metrics disagree?
Treat scores as routing signals, not final verdicts. BLEU measures surface overlap without checking the source, so it can miss meaning errors that source-aware learned metrics detect.
For regulated workflows, combine structural checks, expected terminology, and established quality thresholds. Don’t rely on a single metric. When results remain unclear, prioritize human review - especially for safety-critical documents or those with strict compliance requirements. Use feedback from disagreements to refine quality gates.
How much reviewer-rated data do we need to set thresholds?
There’s no fixed amount. Start by reviewing your current workflow. Compare historical metric scores with past human quality ratings to set a baseline and check how closely those scores track translation quality [1].
Treat thresholds as settings you can adjust as machine translation performance and content types change. Periodically sample auto-approved content to check whether metrics still predict quality. Review thresholds on a regular schedule, such as quarterly [1].
How often should we recalibrate translation quality checks?
Review your translation quality thresholds on a set schedule - ideally every quarter - because machine translation engine performance and your content change over time [1]. Sample auto-approved content periodically to check whether your metrics still predict quality accurately [1]. Use that feedback to keep automated quality checks aligned with your changing business needs and quality standards [1].
Frequently asked questions
What should we do when BLEU and learned metrics disagree?
Treat scores as routing signals, not final verdicts. BLEU measures surface overlap without checking the source, so it can miss meaning errors that source-aware learned metrics detect. For regulated workflows, combine structural checks, expected terminology, and established quality thresholds. Don’t rely on a single metric. When results remain unclear, prioritize human review - especially for safety-critical documents or those with strict compliance requirements. Use feedback from disagreements to refine quality gates.
How much reviewer-rated data do we need to set thresholds?
There’s no fixed amount. Start by reviewing your current workflow. Compare historical metric scores with past human quality ratings to set a baseline and check how closely those scores track translation quality [1]. Treat thresholds as settings you can adjust as machine translation performance and content types change. Periodically sample auto-approved content to check whether metrics still predict quality. Review thresholds on a regular schedule, such as quarterly [1].
How often should we recalibrate translation quality checks?
Review your translation quality thresholds on a set schedule - ideally every quarter - because machine translation engine performance and your content change over time [1]. Sample auto-approved content periodically to check whether your metrics still predict quality accurately [1]. Use that feedback to keep automated quality checks aligned with your changing business needs and quality standards [1].
