E.g., 04/11/2020
E.g., 04/11/2020

Bridging the Gap in QT21: A Harmonized Approach to Quality Metrics

By: QT21 Project, title - TAUS &

Authors:  Attila Görög (TAUS), Arle Lommel (DFKI), and Hans Uszkoreit (DFKI)

Watch the webinar, "Harmonizing Quality—TAUS DQF and MQM Become a Single Quality Framework"  to learn more.

The past few years have seen an explosion of interest in translation quality assessment methods and metrics. In particular, two independent efforts have risen to prominence in the localization industry, the TAUS Dynamic Quality Framework (DQF) and Multidimensional Quality Metrics (MQM), which was developed in the European Union-funded QTLaunchPad Project, which ended last year.

These tools are also now joined by a proposal at the ASTM F43 standards committees called LQA (officially WK46397 "Standard Guide for Public LQA methodology"), spearheaded by Logrus Corporation, to define methods for quality assessment.

You can add to the mix a lot of legacy use of the LISA QA Model (usually modified), SAE J2450, and any number of in-house, proprietary quality assessment methods. All of these efforts show just how important quality assessment has become, but the diversity of methods can leave end users wondering just which one to use and why they should have to choose.

As a part of the new EU-funded QT21 project, TAUS and DFKI, which was the primary driver behind MQM, were tasked with harmonizing their respective metrics. In a series of meetings, the two groups have done just that, with important changes made to both frameworks to bring them into harmony.

When the new versions of MQM and DQF are released this summer, users will no longer have to choose between the two because they will share the same underlying structure.

The following points will apply:

  • For the time being MQM and DQF will remain separate projects, with TAUS continuing to offer the DQF suite of online tools.
  • DQF’s “analytic” method and the MQM hierarchy of translation quality issues have both been modified to share the same basic structure. MQM will add simple support for the other holistic methods in DQF so that DQF scores and quality information can be exchanged with MQM tools as well.
  • Via MQM, DQF will gain an unambiguous mapping to the ITS 2.0 localization quality issue types that have been implemented in Ocelot and other ITS 2.0-compliant tools. This mapping will allow both MQM and DQF to interoperate with this W3C recommendation.

Changes to MQM include:

  • Two new “dimensions” (top-level categories) were added: Terminology and Locale convention.
  • A number of categories were added to support DQF issue types.
  • The Internationalization dimension was filled in with 48 issue types based on the “LISA Internationalization Readiness” project.

Changes to DQF include:

  • MQM names were accepted for many issue types.
  • Some issues were moved within the DQF hierarchy to match the MQM hierarchy.
  • A few redundancies were eliminated.

While the above description covers the harmonization of MQM and DQF, LQA represents a new standardization effort. At the Eighth MultilingualWeb Workshop in Riga (29 April, 2015) an agreement was reached to work towards harmonizing LQA with the joint MQM/DQF project. Accordingly, a few issue types have been added to support features of LQA, and the projects will work to harmonize terminology and concepts.

The newest version of the harmonized MQM portion is currently undergoing final internal review and will be published for public comment within the next month. To be notified of its release, please contact Arle Lommel and Attila Görög. The latest public version of MQM is available at http://qt21.eu/mqm-definition/. When the newer version (1.0-beta) is ready for public comment, it will be published at this URL.

In keeping with a long-standing goal for standardization, the harmonized results will be submitted to a standards body for further development and maintenance by the broader community. The standardization process is likely to result in some changes, but will result in the best representation of community consensus and practice.

To learn more and to be included in future announcements about this topic, please send an email to Arle Lommel. We will keep interested parties updated via a mailing list.

(The project QT21 has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 645452.)  

Attila Görög has been involved in various national and international projects on language technology in the past 10 years. He has a solid background in PE, terminology, QA, standardization and linguistics. As a product manager at TAUS, he is responsible for the TAUS Evaluation platform also referred to as the Dynamic Quality Framework or DQF.

Arle Lommel is a senior consultant at the German Research Center for Artificial Intelligence (DFKI)’s Language Technology Lab in Berlin. At DFKI he works on standardization of translation-related metadata, translation quality, and other related projects. The former director of open standards at LISA, he holds advanced degrees in linguistics and ethnography.

Hanz Uszkoreit is a Professor of Computational Linguistics at the Dept. of Computational Linguistics and Phonetics of Saarland University at Saarbrücken. He is the Scientific Director at the German Research Center for Artificial Intelligence (DFKI) and Head of DFKI Language Technology Lab.

QT21 Project

Quality Translation 21 is a machine translation project which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 645452.