Skip links

How We Measure Machine Translation Quality (MQM Explained)

One mistranslated dosage instruction can do more damage than a hundred misplaced commas. Yet many translation quality reports still treat every error the same way, or skip the counting altogether and settle for “looks good to us.” As AI translation moves into everyday production, buyers need a way to compare quality claims on equal terms.

That is what MQM translation quality evaluation provides. Multidimensional Quality Metrics (MQM) gives reviewers a shared vocabulary for classifying errors and a transparent way to turn them into a number. At Localizera, it is the framework our linguists use to judge machine translation output. Here is how it works, math included.

TL;DR

  • What it is: MQM (Multidimensional Quality Metrics) is an open framework that classifies translation errors by type and severity and converts them into a quality score.
  • How it works: Three steps: define the scope and error types, have trained annotators mark each error, then calculate the score.
  • The math: Common default weights are neutral (0), minor (1), major (5), and critical (25 or automatic fail). A widely used formula is Score = (1 − penalty points ÷ words evaluated) × 100.
  • MQM vs. BLEU / COMET: Automatic metrics are fast and cheap for tracking engines; MQM is slower but tells you what is wrong. The two work best together.
  • Thresholds: There is no universal pass mark. Set thresholds and sample sizes by content risk, and only compare scores built on the same setup.

What Is MQM (Multidimensional Quality Metrics)?

MQM (Multidimensional Quality Metrics) is an open framework for identifying, classifying, and scoring translation errors so that quality can be measured consistently and compared fairly. Instead of asking “Is this translation good?”, it asks “What went wrong, where, and how much does it matter?”

MQM grew out of European translation-technology research in the early 2010s and was later harmonized with the TAUS Dynamic Quality Framework as the DQF-MQM error typology. Freitag et al. (2021), “Experts, Errors, and Context,” showed that MQM annotations by professional translators were more reliable than crowd ratings, and MQM has since served as the human benchmark in the WMT metrics shared tasks. The newer ISO 5060 standard for evaluating translation output builds on the same error-based approach to translation quality assessment.

Localization managers, MT program leads, procurement teams, and linguists running LQA (linguistic quality assurance) use it for three reasons: a shared vocabulary, comparable results across vendors and engines, and actionable feedback that shows whether to fix the glossary, retrain the engine, or brief post-editors.

If you are new to how engines produce these errors in the first place, our guide to machine translation types, use cases, and best practices is a useful primer.

How MQM Scoring Works: Define, Annotate, Score

MQM scoring follows three steps:

  1. Define the scope. Decide what content is being evaluated, how large the sample is, and which error types matter for this project. A legal contract and a mobile game need different checklists.
  2. Annotate. Trained annotators, ideally professional native-speaker linguists, mark every error by category and severity.
  3. Calculate the score. Penalty points are totaled and normalized by the number of words evaluated to produce a quality score.

MQM Error Categories and Severity: The Error Types

MQM offers a large catalog of error types, and teams pick only those that matter for the content. The core categories are:

Category

What it catches

Example

Accuracy

Meaning that is added, omitted, or mistranslated

“Do not exceed 2 tablets” rendered as “Take 2 tablets”

Fluency

Grammar, spelling, punctuation, readability

Wrong verb agreement

Terminology

Wrong or inconsistent use of approved terms

Using “cart” where the glossary says “basket”

Style

Tone, register, brand voice

Overly formal copy on a casual app screen

Locale conventions

Dates, numbers, currency, units, address formats

03/04 read as March 4 vs. April 3

Many setups also include verity (content that is not true or appropriate for the target market), design and markup (truncation, broken tags), and internationalization issues in the source. Locale conventions are where many “accurate” translations still fail, which is why making content feel truly local deserves its own place in the checklist.

The Severity Levels of MQM Quality

Severity is what separates MQM from simple error counting. Each error receives one of four levels, each with a penalty weight:

Severity

Meaning

Common default weight

Neutral

A preference or note, not a real error

0

Minor

Noticeable, but does not affect meaning or usability

1

Major

Changes meaning or confuses the reader

5

Critical

Could cause legal, safety, financial, or reputational harm

25 (or automatic fail)

These are commonly used defaults, not fixed rules. Teams can customize the weights, but they must keep them identical across any tests they intend to compare. Getting MQM error categories and severity right is the foundation of every score that follows.

Calculating the MQM Translation Quality Score (Worked Example)

Formulas vary by provider. A widely used version is:

Score = (1 − total penalty points ÷ words evaluated) × 100

Illustrative example only. Suppose a reviewer evaluates a 1,000-word sample of machine-translated product content and records the following:

#

Category

Severity

Penalty points

1

Fluency (punctuation)

Minor

1

2

Fluency (word order)

Minor

1

3

Style (register)

Minor

1

4

Style (awkward phrasing)

Minor

1

5

Locale conventions (date format)

Minor

1

6

Terminology (inconsistent term)

Minor

1

7

Accuracy (omitted condition)

Major

5

8

Terminology (wrong product term)

Major

5

 

Total

6 minor + 2 major

16

Score = (1 − 16 ÷ 1,000) × 100 = 98.4

Now add one critical error, such as a wrong safety warning. With a weight of 25, the total becomes 41 points and the score drops to 95.9. Many programs go further and treat any critical error as an automatic fail, regardless of the number. That is deliberate: a sample that scores 99 but contains one dangerous mistranslation should never pass.

What the Research Says About Machine Translation Quality Evaluation

  1. Readable is not the same as correct. According to Nasution (2022), “Machine Translation in Website Localization: Assessing its Translation Quality”, a machine translation quality evaluation of Google Translate output for a private-jet charter website (English to Indonesian), rated against a human translation on a three-point readability scale, found that weaker sentences ignored Indonesian stylistic norms, skipped cultural adaptation, and copied the source structure. The study concludes that human post-editing remains essential for website localization.

Our insight: even sentences rated fully readable contained an untranslated English term and a literal rendering of “internal flights.” A readability score lets those through; MQM logs them as terminology, style, and locale conventions errors.

  1. Evaluation and estimation are two different jobs. Based on the Machine Translation Quality Assessment (MTQA) project at Tilburg University, led by Dimitar Shterionov with Javad Pourmostafa Roshan Sharami and the language service provider Aglatech14, MT quality work has two stages: human evaluation before deployment decides whether an engine is production-ready, and quality estimation (QE) after deployment guides post-editors and approximates post-editing effort.

Our insight: this mirrors translation quality assessment in practice. MQM is the gold standard for the first stage and supplies the human-labeled data needed to check whether a QE model can be trusted in the second.

MQM vs. BLEU / COMET and Other Automatic Metrics

 

Metric

What it measures

Needs a human reference?

Speed and cost

Best used for

MQM

Human-annotated errors by type and severity

No (needs trained annotators)

Slow, higher cost

Diagnosing quality, vendor comparison, final acceptance

BLEU

Word overlap with a reference translation

Yes

Instant, near-free

Quick engine tracking; weak on meaning

TER

Edits needed to match a reference

Yes

Instant, near-free

Estimating post-editing distance

COMET

Neural prediction of human quality judgments

Usually yes

Fast, low cost

Ranking engines and systems at scale

Quality estimation (QE)

Predicted quality with no reference

No

Fast, low cost

Routing content, flagging risky segments

The key message: automatic metrics are excellent for tracking engines at scale, but they cannot tell you why a translation fails or whether a single error is dangerous. MQM can. Mature Language AI workflows use automatic scores to monitor and triage, then use MQM to diagnose and approve.

LLMs and research systems such as xCOMET can now pre-annotate errors in MQM format. This speeds up review, but human validation is still needed, especially for critical errors- one of many ways AI and automation are reshaping localization without removing the human check.

How We Apply MQM to Machine Translation at Localizera

  • Representative samples. We draw samples from real project content across each content type and language pair, rather than hand-picking easy segments. 
  • Qualified annotators. Reviews are carried out by professional native-speaker linguists trained in MQM. 
  • Project-specific error types and weights. We agree on error categories and severity weights with the client before the review starts, and keep them fixed across rounds so results stay comparable. 
  • Results that change the workflow. Findings feed back into glossaries and style guides, engine selection and tuning, and briefs for our post-editors. 

This is what connects LQA (linguistic quality assurance) to real decisions. A pattern of terminology errors points to the glossary; recurring accuracy errors may mean the engine is not ready for that content type, so it is routed to full machine translation post-editing instead of light review. The same process supports our language-specific work, from Japanese MTPE services, where honorifics and register drive many style errors, to German MTPE services, where compound terminology and formal address are common trouble spots.

Want to know how your current MT output really scores? Request a quote for MTPE translation with MQM-based quality review. 

What Is a Good MQM Translation Quality Score? Thresholds and Sample Sizes

There is no universal pass mark. The right threshold depends on what a mistake would cost. 

Content risk

Examples

Illustrative threshold

High

Medical, legal, financial, safety content

99+ and zero critical errors

Medium

Marketing, UI, customer-facing product content

97–98, zero critical errors

Low

Internal docs, support articles, user-generated content

95–96

On sampling, a recommended practice (not a standard) is a representative sample of roughly 1,000 to 2,000 words per content type and language pair, drawn from across the job. Small samples swing wildly: one major error moves a 300-word sample by more than 1.5 points.

Most importantly, scores are only comparable when the same error types, weights, formula, and thresholds are used. A 98 from one vendor and a 98 from another can mean very different things.

Common MQM Scoring Mistakes to Avoid

  • Untrained or uncalibrated annotators score the same text differently.
  • Samples that are too small cannot represent a large project.
  • Changing weights between tests makes trend lines meaningless.
  • Ignoring severity hides the one error that matters.
  • Reading only the number: always check the category breakdown alongside routine translation QA.

Conclusion

Translation quality stops being a matter of opinion once the method is shared and transparent. MQM provides that method: agreed error categories, severity weights, a clear formula, and risk-based thresholds. Paired with COMET and QE for scale, it lets you compare engines and vendors honestly and catch the errors that actually matter.

The practical takeaway is simple: before you trust any MQM translation quality claim, ask which error types, weights, formula, and sample size produced it. If the answers are vague, so is the score.

If you want machine translation that is fast and verified by people, explore Localizera’s human-reviewed translation services and MTPE solutions.

Frequently Asked Questions

What does MQM stand for in translation?

MQM stands for Multidimensional Quality Metrics, an open framework for classifying translation errors by category and severity and turning them into a quality score.

What is a good MQM score?

It depends on content risk. High-risk medical or legal content typically needs 99+ with zero critical errors; internal content can accept less.

How is MQM different from BLEU and COMET?

MQM vs. BLEU / COMET is human diagnosis vs. automated estimation: BLEU and COMET score output automatically, while MQM uses trained linguists to identify and weight each error.

Who created MQM?

MQM came out of European translation-technology research in the 2010s and was later harmonized with TAUS DQF as DQF-MQM.

How many words should be sampled for an MQM review?

A common recommended practice is 1,000 to 2,000 words per content type and language pair, drawn from across the project. This is guidance, not a formal standard.

Can AI do MQM scoring?

Partly. LLMs and models such as xCOMET can pre-annotate likely errors and speed up MQM scoring, but human validation is still needed, especially to confirm major and critical errors.