E.g., 03/23/2019
E.g., 03/23/2019

Bridging the Gap in QT21: A Harmonized Approach to Quality Metrics

By: Attila Görög (TAUS), Arle Lommel (DFKI), and Hanz Uszkoreit (DFKI) - TAUS


07 May 2015

Learn how the translation industry's two independent efforts to measure and standardize translation quality, the TAUS Dynamic Quality Framework (DQF) and Multidimensional Quality Metrics (MQM), are being “harmonized” to form an industry-wide quality standard. 

The past few years have seen an explosion of interest in translation quality assessment methods and metrics. In particular, two independent efforts have risen to prominence in the localization industry, the TAUS Dynamic Quality Framework (DQF) and Multidimensional Quality Metrics (MQM), which was developed in the European Union-funded QTLaunchPad Project, which ended last year,

Both of these systems have seen widespread interest and use. Since 2012 TAUS has provided an online suite of tools to be adopted in the near future through the DQF API by a number of translation technology users and providers such as VMware, SDL, Memsource, Translated and many more.  MQM has been adopted by a number of tools, including XTM, the open-source translate5, and a number of in-house tools, soon to be joined by a light-weight open source “scorecard” tool. These tools are also now joined by a proposal at the ASTM F43 standards committees called LQA (officially WK46397 "Standard Guide for Public LQA methodology"), spearheaded by Logrus Corporation, to define methods for quality assessment.

You can add to the mix a lot of legacy use of the LISA QA Model (usually modified), SAE J2450, and any number of in-house, proprietary quality assessment methods. All of these efforts show just how important quality assessment has become, but the diversity of methods can leave end users wondering just which one to use and why they should have to choose. The developers of MQM and DQF, in particular, have regularly been asked which one should be used and what the difference is, and we realized that having two separate efforts in this central area does not do anyone a service.

As a part of the new EU-funded QT21 project, TAUS and DFKI, which was the primary driver behind MQM, were tasked with harmonizing their respective metrics. In a series of meetings, the two groups have done just that, with important changes made to both frameworks to bring them into harmony. When the new versions of MQM and DQF are released this summer, users will no longer have to choose between the two because they will share the same underlying structure. The following points will apply:

  • For the time being MQM and DQF will remain separate projects, with TAUS continuing to offer the DQF suite of online tools.
  • DQF’s “analytic” method and the MQM hierarchy of translation quality issues have both been modified to share the same basic structure. DQF will use a subset of the full MQM hierarchy based on the experience of TAUS members, while MQM will continue to maintain a broader set of issue types designed to capture and describe the full range of quality assessment metrics currently in use. Users of the DQF analytic method will therefore be guaranteed to be compliant with MQM as well.
  • MQM will add simple support for the other holistic methods in DQF so that DQF scores and quality information can be exchanged with MQM tools as well.
  • Via MQM, DQF will gain an unambiguous mapping to the ITS 2.0 localization quality issue types that have been implemented in Ocelot and other ITS 2.0-compliant tools. This mapping will allow both MQM and DQF to interoperate with this W3C recommendation.

Some of the important changes made to MQM in the process are the following:

  • Two new “dimensions” (top-level categories) were added: Terminology and Locale convention. Terminology collects all terminology-related issues, which had previously been divided in other branches, and Locale convention covers many localization-related issues such as incorrect date or number formats.
  • A number of categories were added to support DQF issue types.
  • The Internationalization dimension was filled in with 48 issue types based on the “LISA Internationalization Readiness” project, which had released some results publicly before LISA ceased operation.

On the DQF side, the following changes were made:

  • MQM names were accepted for many issue types
  • Some issues were moved within the DQF hierarchy to match the MQM hierarchy
  • A few redundancies were eliminated

While the above description covers the harmonization of MQM and DQF, LQA represents a new standardization effort. It differs from DQF and MQM in that it does not provide a listing of error types, but rather provides a method for applying holistic and analytic quality analysis, with an emphasis on quality assessment in crowd-sourced environments. At the Eighth MultilingualWeb Workshop in Riga (29 April, 2015) an agreement was reached to work towards harmonizing LQA with the joint MQM/DQF project. Accordingly, a few issue types have been added to support features of LQA, and the projects will work to harmonize terminology and concepts. Because LQA is focused primarily on method, it does not otherwise require substantial changes to the harmonized error typology. But the basic concepts will be aligned between DQF, MQM, and LQA.

The newest version of the harmonized MQM portion is currently undergoing final internal review and will be published for public comment within the next month. To be notified of its release, please contact Arle Lommel and Attila Görög. The latest public version of MQM is available at http://qt21.eu/mqm-definition/. When the newer version (1.0-beta) is ready for public comment, it will be published at this URL.

In keeping with a long-standing goal for standardization, the harmonized results will be submitted to a standards body for further development and maintenance by the broader community. The standardization process is likely to result in some changes, but will result in the best representation of community consensus and practice.

To learn more and to be included in future announcements about this topic, please send an email to Arle Lommel. We will keep interested parties updated via a mailing list.

(The project QT21 has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 645452.)

Attila Görög has been involved in various national and international projects on language technology in the past 10 years. He has a solid background in PE, terminology, QA, standardization and linguistics. As a product manager at TAUS, he is responsible for the TAUS Evaluation platform also referred to as the Dynamic Quality Framework or DQF.

Arle Lommel is a senior consultant at the German Research Center for Artificial Intelligence (DFKI)’s Language Technology Lab in Berlin. At DFKI he works on standardization of translation-related metadata, translation quality, and other related projects. The former director of open standards at LISA, he holds advanced degrees in linguistics and ethnography.

Hanz Uszkoreit is a Professor of Computational Linguistics at the Dept. of Computational Linguistics and Phonetics of Saarland University at Saarbrücken. He is the Scientific Director at the German Research Center for Artificial Intelligence (DFKI) and Head of DFKI Language Technology Lab.

randomness