E.g., 04/07/2020
E.g., 04/07/2020

MQM: Bringing sanity to translation quality metrics

By: Andrzej Zydron, CTO - XTM International

11 September 2014

For many years within the global labelling/marketing/content departments of enterprises worldwide and indeed within the translation and localization industry, the quality of delivered translations has been very difficult to define. Choosing translation providers or individual translators that claim the highest quality processes or had the strongest reputations has not always proved to deliver the desired and expected results. Variable requirements and complexity make it difficult to consistently evaluate language deliveries. The ability to rate quality in an effective and uniform manner has long been a requirement in general in the industry but of crucial importance especially for high risk life science translation projects and translations for high-risk patient safety medical device labelling. For example, a translation project may have a negative reaction from an end in-country recipient client. But there can be several reasons;

  • Low quality of the source content
  • Translation process issues
  • Subjective views of end country reviewer
  • Incomplete or low quality source or target terminology
  • Software internationalization issues
  • Low quality translation

Process owners need to be able to rate the performance of various translation resources at their disposal for a given language in a consistent and uniform manner. Another example a translation process owner needs to assess is the relative quality of a number of translators for a given language pair.  A particularly high sensitivity project may demand the highest level translation resources. A historical record from a pool of translators assessed in a uniform and managed methodology would significantly help to choose those with a consistently high quality of output. QTLaunchPad is an EU FP7 funded project designed to help improve translation quality. The key to any form of translation (be it human or machine translation or machine translation post-editing) quality is the ability to provide consistent and measurable metrics. In the past initiatives such as the LISA QA model or JAE2450 have tried to approach this, but without a great deal of success in terms of industry acceptance, mainly due to the fact that they adopted a ‘one size fits all’ approach to all translation projects, whereas in reality there is a very wide spectrum of diversity within the type of translation task being undertaken. In addition these models have not kept up with the rapid pace of adoption of machine translation and subsequent post-editing nature of translation in this field. The main goal of the QTLaunchPad project is establishing the concept of Measurable Quality Metrics (MQM). MQM is at the heart of QTLaunchPad and is a completely open standard.   These standards were designed by a committee of industry experts and are open to public scrutiny as part of their development. The key participants in this project were the German Research Center for Artificial Intelligence (DFKI), Dublin City University, the University of Sheffield and the Athena Institute for Language and Speech Processing. All leading academic institutions with access to world renowned experts in the filed including Arle Lommel, who was one of the major architects of the LISA QA model. MQM principles MQM is based on the following key principles:

  1. Flexibility: MQM eschews the ‘one size fits all’ model of previous QA initiatives. It allows adapting metrics to the specific nature of the translation project at hand. Translation projects do not all have the same quality requirements or complexity. Documentation destined for automotive technicians in the workshop may have a quality requirement of ‘fit for purpose’ (technicians never read the manuals anyway. Well, only as a matter of last resort when all other attempts including hitting the affected item hard with a hammer have failed), whereas owner glove box instructions for a car must have no errors whatsoever. Metrics must be ‘tuneable’ along different dimensions, e.g. domain, purpose, audience. MQM provides a flexible catalogue of issues types that can be extended and adapted as required for the project to hand.
  2. Fairness: MQM allows for the defining the true cause of the problem. If the source text is badly and ambiguous, then the blame should not be laid at the feet of the translator. If the source text is of a low quality then it must be possible to determine this.
  3. Suitability: MQM is suitable for all types of translation, production methods and technologies, as well as allowing extensibility for new categories of checks.
  4. Comparability: MQM results must be comparable with each other even if the assessments tasks are not checking the same thing.
  5. Standards: MQM is based on ISO 11669 for defining QA check dimensions.
  6. Granularity: MQM supports various levels of granularity as required by the given project to hand.

MQM features MQM offers the following features:

  1. Open Standard: MQM is a completely open and unencumbered standard.
  2. Flexibility: MQM provides a comprehensive and extensible list of issues types.
  3. Separation: MQM provides for identifying both source and target issues.
  4. Granularity: MQM allows for a hierarchical view of issues from high level to finely grained.
  5. Dimensions: MQM provides for up to eleven dimensions of QA issues, based on ISO/TS-11669, to guide users in choosing the appropriate types of issue for QA tasks.
  6. Backwards Compatibility: MQM supports previous QA models such as the LISA QA model or SAE J2450.
  7. Extensibility: MQM allows a project manager to add additional or more finely grained issue types.

The importance of MQM Using MQM, a project manager can now provide an unambiguous, non-subjective, and systematic quality assessment of the work of a translator. This takes into account any shortcomings of the original source text and can be used over time to build up a complete and comprehensive objective assessment of a linguist’s quality of work. The functionality offers benefits for both – linguist and a project manager alike: one being able to work with only the best, and the other one being fully appreciated for all the hard work he’s done. MQM provides a significant improvement based on a fair and open standard of the quality of the work of individual translators. XTM Cloud and MQM MQM provides a significant improvement based on a fair and open standard of the quality of translation deliveries. The functionality so far has only been introduced into one commercial translation management technology – the latest release of XTM Cloud.   It is available under the LQA (Linguist Quality Assurance) section and is a predefined optional workflow step in XTM.  

Andrzej Zydroń has worked in IT since 1976. His experience has covered all aspects of computing, with in depth knowledge of software engineering, XML, encoding methodologies and translation memory. A member of the former LISA OSCAR Steering Committee, and currently Rapporteur for ETSI LIS, he was the lead author of the GMX-V and xml:tm standards and active in the development of other OSCAR standards. GALA is currently working in partnership with DFKI to develop translate5, a browser-based open source system for editing and analyzing translations, based on the MQM framework. In translate5, users can edit, comment, filter, and sort translations. The tool supports supports terminology tagging, relay languages, and reference files, and includes components for workflow, task management, and user administration. Watch Marc Mittag's recent webinar on the current state of translate5 and future features.  

Andrzej Zydron

Andrzej Zydroń has worked in IT since 1976 and has been responsible for major successful projects at Xerox, SDL, Oxford University Press, Ford of Europe, DocZone and Lingo24. He is extremely involved in a wide variety of standards projects and open standard technical committees. He is currently head of the OASIS OAXAL (Open Architecture for XML Authoring and Localization) technical committee.