One mistranslated dosage instruction can do more damage than a hundred misplaced commas. Yet many translation quality reports still treat every error the same way, or skip the counting altogether and settle for “looks good to us.” As AI translation moves into everyday production, buyers need a way to compare quality claims on equal terms.
That is what MQM translation quality evaluation provides. Multidimensional Quality Metrics (MQM) gives reviewers a shared vocabulary for classifying errors and a transparent way to turn them into a number. At Localizera, it is the framework our linguists use to judge machine translation output. Here is how it works, math included.
TL;DR
- What it is: MQM (Multidimensional Quality Metrics) is an open framework that classifies translation errors by type and severity and converts them into a quality score.
- How it works: Three steps: define the scope and error types, have trained annotators mark each error, then calculate the score.
- The math: Common default weights are neutral (0), minor (1), major (5), and critical (25 or automatic fail). A widely used formula is Score = (1 − penalty points ÷ words evaluated) × 100.
- MQM vs. BLEU / COMET: Automatic metrics are fast and cheap for tracking engines; MQM is slower but tells you what is wrong. The two work best together.
- Thresholds: There is no universal pass mark. Set thresholds and sample sizes by content risk, and only compare scores built on the same setup.
What Is MQM (Multidimensional Quality Metrics)?
MQM (Multidimensional Quality Metrics) is an open framework for identifying, classifying, and scoring translation errors so that quality can be measured consistently and compared fairly. Instead of asking “Is this translation good?”, it asks “What went wrong, where, and how much does it matter?”
MQM grew out of European translation-technology research in the early 2010s and was later harmonized with the TAUS Dynamic Quality Framework as the DQF-MQM error typology. Freitag et al. (2021), “Experts, Errors, and Context,” showed that MQM annotations by professional translators were more reliable than crowd ratings, and MQM has since served as the human benchmark in the WMT metrics shared tasks. The newer ISO 5060 standard for evaluating translation output builds on the same error-based approach to translation quality assessment.
Localization managers, MT program leads, procurement teams, and linguists running LQA (linguistic quality assurance) use it for three reasons: a shared vocabulary, comparable results across vendors and engines, and actionable feedback that shows whether to fix the glossary, retrain the engine, or brief post-editors.
If you are new to how engines produce these errors in the first place, our guide to machine translation types, use cases, and best practices is a useful primer.
How MQM Scoring Works: Define, Annotate, Score
MQM scoring follows three steps:
- Define the scope. Decide what content is being evaluated, how large the sample is, and which error types matter for this project. A legal contract and a mobile game need different checklists.
- Annotate. Trained annotators, ideally professional native-speaker linguists, mark every error by category and severity.
- Calculate the score. Penalty points are totaled and normalized by the number of words evaluated to produce a quality score.
MQM Error Categories and Severity: The Error Types
MQM offers a large catalog of error types, and teams pick only those that matter for the content. The core categories are:
|
Category |
What it catches |
Example |
|---|---|---|
|
Accuracy |
Meaning that is added, omitted, or mistranslated |
“Do not exceed 2 tablets” rendered as “Take 2 tablets” |
|
Fluency |
Grammar, spelling, punctuation, readability |
Wrong verb agreement |
|
Terminology |
Wrong or inconsistent use of approved terms |
Using “cart” where the glossary says “basket” |
|
Style |
Tone, register, brand voice |
Overly formal copy on a casual app screen |
|
Locale conventions |
Dates, numbers, currency, units, address formats |
03/04 read as March 4 vs. April 3 |
Many setups also include verity (content that is not true or appropriate for the target market), design and markup (truncation, broken tags), and internationalization issues in the source. Locale conventions are where many “accurate” translations still fail, which is why making content feel truly local deserves its own place in the checklist.
The Severity Levels of MQM Quality
Severity is what separates MQM from simple error counting. Each error receives one of four levels, each with a penalty weight:
|
Severity |
Meaning |
Common default weight |
|---|---|---|
|
Neutral |
A preference or note, not a real error |
0 |
|
Minor |
Noticeable, but does not affect meaning or usability |
1 |
|
Major |
Changes meaning or confuses the reader |
5 |
|
Critical |
Could cause legal, safety, financial, or reputational harm |
25 (or automatic fail) |
These are commonly used defaults, not fixed rules. Teams can customize the weights, but they must keep them identical across any tests they intend to compare. Getting MQM error categories and severity right is the foundation of every score that follows.
Calculating the MQM Translation Quality Score (Worked Example)
Formulas vary by provider. A widely used version is:
Score = (1 − total penalty points ÷ words evaluated) × 100
Illustrative example only. Suppose a reviewer evaluates a 1,000-word sample of machine-translated product content and records the following:
|
# |
Category |
Severity |
Penalty points |
|---|---|---|---|
|
1 |
Fluency (punctuation) |
Minor |
1 |
|
2 |
Fluency (word order) |
Minor |
1 |
|
3 |
Style (register) |
Minor |
1 |
|
4 |
Style (awkward phrasing) |
Minor |
1 |
|
5 |
Locale conventions (date format) |
Minor |
1 |
|
6 |
Terminology (inconsistent term) |
Minor |
1 |
|
7 |
Accuracy (omitted condition) |
Major |
5 |
|
8 |
Terminology (wrong product term) |
Major |
5 |
|
Total |
6 minor + 2 major |
16 |
Score = (1 − 16 ÷ 1,000) × 100 = 98.4
Now add one critical error, such as a wrong safety warning. With a weight of 25, the total becomes 41 points and the score drops to 95.9. Many programs go further and treat any critical error as an automatic fail, regardless of the number. That is deliberate: a sample that scores 99 but contains one dangerous mistranslation should never pass.
What the Research Says About Machine Translation Quality Evaluation
- Readable is not the same as correct. According to Nasution (2022), “Machine Translation in Website Localization: Assessing its Translation Quality”, a machine translation quality evaluation of Google Translate output for a private-jet charter website (English to Indonesian), rated against a human translation on a three-point readability scale, found that weaker sentences ignored Indonesian stylistic norms, skipped cultural adaptation, and copied the source structure. The study concludes that human post-editing remains essential for website localization.
Our insight: even sentences rated fully readable contained an untranslated English term and a literal rendering of “internal flights.” A readability score lets those through; MQM logs them as terminology, style, and locale conventions errors.
- Evaluation and estimation are two different jobs. Based on the Machine Translation Quality Assessment (MTQA) project at Tilburg University, led by Dimitar Shterionov with Javad Pourmostafa Roshan Sharami and the language service provider Aglatech14, MT quality work has two stages: human evaluation before deployment decides whether an engine is production-ready, and quality estimation (QE) after deployment guides post-editors and approximates post-editing effort.
Our insight: this mirrors translation quality assessment in practice. MQM is the gold standard for the first stage and supplies the human-labeled data needed to check whether a QE model can be trusted in the second.
MQM vs. BLEU / COMET and Other Automatic Metrics
|
Metric |
What it measures |
Needs a human reference? |
Speed and cost |
Best used for |
|---|---|---|---|---|
|
MQM |
Human-annotated errors by type and severity |
No (needs trained annotators) |
Slow, higher cost |
Diagnosing quality, vendor comparison, final acceptance |
|
BLEU |
Word overlap with a reference translation |
Yes |
Instant, near-free |
Quick engine tracking; weak on meaning |
|
TER |
Edits needed to match a reference |
Yes |
Instant, near-free |
Estimating post-editing distance |
|
COMET |
Neural prediction of human quality judgments |
Usually yes |
Fast, low cost |
Ranking engines and systems at scale |
|
Quality estimation (QE) |
Predicted quality with no reference |
No |
Fast, low cost |
Routing content, flagging risky segments |
The key message: automatic metrics are excellent for tracking engines at scale, but they cannot tell you why a translation fails or whether a single error is dangerous. MQM can. Mature Language AI workflows use automatic scores to monitor and triage, then use MQM to diagnose and approve.
LLMs and research systems such as xCOMET can now pre-annotate errors in MQM format. This speeds up review, but human validation is still needed, especially for critical errors- one of many ways AI and automation are reshaping localization without removing the human check.
How We Apply MQM to Machine Translation at Localizera
- Representative samples. We draw samples from real project content across each content type and language pair, rather than hand-picking easy segments.
- Qualified annotators. Reviews are carried out by professional native-speaker linguists trained in MQM.
- Project-specific error types and weights. We agree on error categories and severity weights with the client before the review starts, and keep them fixed across rounds so results stay comparable.
- Results that change the workflow. Findings feed back into glossaries and style guides, engine selection and tuning, and briefs for our post-editors.
This is what connects LQA (linguistic quality assurance) to real decisions. A pattern of terminology errors points to the glossary; recurring accuracy errors may mean the engine is not ready for that content type, so it is routed to full machine translation post-editing instead of light review. The same process supports our language-specific work, from Japanese MTPE services, where honorifics and register drive many style errors, to German MTPE services, where compound terminology and formal address are common trouble spots.
Want to know how your current MT output really scores? Request a quote for MTPE translation with MQM-based quality review.
What Is a Good MQM Translation Quality Score? Thresholds and Sample Sizes
There is no universal pass mark. The right threshold depends on what a mistake would cost.
|
Content risk |
Examples |
Illustrative threshold |
|---|---|---|
|
High |
Medical, legal, financial, safety content |
99+ and zero critical errors |
|
Medium |
Marketing, UI, customer-facing product content |
97–98, zero critical errors |
|
Low |
Internal docs, support articles, user-generated content |
95–96 |
On sampling, a recommended practice (not a standard) is a representative sample of roughly 1,000 to 2,000 words per content type and language pair, drawn from across the job. Small samples swing wildly: one major error moves a 300-word sample by more than 1.5 points.
Most importantly, scores are only comparable when the same error types, weights, formula, and thresholds are used. A 98 from one vendor and a 98 from another can mean very different things.
Common MQM Scoring Mistakes to Avoid
- Untrained or uncalibrated annotators score the same text differently.
- Samples that are too small cannot represent a large project.
- Changing weights between tests makes trend lines meaningless.
- Ignoring severity hides the one error that matters.
- Reading only the number: always check the category breakdown alongside routine translation QA.
Conclusion
Translation quality stops being a matter of opinion once the method is shared and transparent. MQM provides that method: agreed error categories, severity weights, a clear formula, and risk-based thresholds. Paired with COMET and QE for scale, it lets you compare engines and vendors honestly and catch the errors that actually matter.
The practical takeaway is simple: before you trust any MQM translation quality claim, ask which error types, weights, formula, and sample size produced it. If the answers are vague, so is the score.
If you want machine translation that is fast and verified by people, explore Localizera’s human-reviewed translation services and MTPE solutions.
Frequently Asked Questions
What does MQM stand for in translation?
MQM stands for Multidimensional Quality Metrics, an open framework for classifying translation errors by category and severity and turning them into a quality score.
What is a good MQM score?
It depends on content risk. High-risk medical or legal content typically needs 99+ with zero critical errors; internal content can accept less.
How is MQM different from BLEU and COMET?
MQM vs. BLEU / COMET is human diagnosis vs. automated estimation: BLEU and COMET score output automatically, while MQM uses trained linguists to identify and weight each error.
Who created MQM?
MQM came out of European translation-technology research in the 2010s and was later harmonized with TAUS DQF as DQF-MQM.
How many words should be sampled for an MQM review?
A common recommended practice is 1,000 to 2,000 words per content type and language pair, drawn from across the project. This is guidance, not a formal standard.
Can AI do MQM scoring?
Partly. LLMs and models such as xCOMET can pre-annotate likely errors and speed up MQM scoring, but human validation is still needed, especially to confirm major and critical errors.