E.g., 04/10/2020
E.g., 04/10/2020

Only 15% of Translation Professionals are Satisfied with their QA Process: 77% demand high quality machine translation

By: Stephen Doherty (Centre for Next Generation Localisation)

06 June 2013

QTLaunchPad is a European Commission-funded collaborative research initiative dedicated to overcoming quality barriers in translation and language technologies. With the support of GALA, almost 500 translation and localization buyers and vendors voiced their opinions on translation quality methods and technologies in a recent online survey. This article presents results from QTLaunchPad’s mapping of the trends and needs of the translation and localization industries. 

As translation and localization is becoming increasingly more technology-focused, how ubiquitous is machine translation (MT)?
Out of almost 500 respondents, over one-third of the respondents reported that they are currently using MT, while a slightly higher percentage stated that their businesses are not using MT at the moment, but that they are planning to do so, either within one year (13%) or in the longer term (22%). Conversely, 28% of the respondents said they do not use MT and have no plans to start doing so.

Fig. 1. Current MT Usage

For those who have adopted MT, which approaches are they taking?
The survey also investigated the type of MT systems used by the above participants. Just over half of the MT adopters use statistical systems (SMT); hybrid MT is also a popular choice (36%), followed by rule-based systems (22%). Of great interest is also the finding that a third of all MT adopters use free external online MT services such as Google Translate, BabelFish, or Bing in their work.

In terms of using off-the-shelf systems, the majority (84%) stated that they had implemented some sort of customization of their MT systems. Popular modifications lie in areas of terminology (61%), in the use of additional domain-specific corpora (32%), and by providing tailor-made linguistic rules (21%). Interestingly, 16% indicated the adoption of controlled authoring or linguistic pre-processing to enhance MT output quality, while another 16% had not carried out any customization of their MT system in any way.

Is MT able to meet the demands of high-quality translated content?
The quality of MT is always of great concern. For most, the successful implementation of MT represents a huge leap and long-term investment in resources and expertise before return on investment becomes apparent. Larger buyers and providers with more resources (and typically more content) still struggle with quality issues and delivering quantifiable results to their customers in a meaningful way. As technologies improve, the case for MT only being good for gisting is being challenged, but what about real-world usable and high-quality outbound translation?

Concerning how much MT output is intended for dissemination purposes, 69% of users felt that less than half of their outbound translation requirements were satisfied with MT, where 12% used MT for more than half of their disseminated translated content, i.e. high-quality published content. Interestingly, 4% of respondents stated that they use MT for all their translation content. Additionally, the majority of those already using MT (77%) witness increasing requests for high-quality MT as part of their work, as opposed to gist or information only quality MT.

A related question elicited opinions on the quality of the MT output provided by the systems used by the respondents. A predominantly positive assessment was given where most responses rated MT output as fair (43%), good (41%), and even excellent (2%).  Only, 7% of the answers indicated that MT output was poor.

How do we know if translation quality is good (enough)?
The evaluation of translation quality represents an area where the absence of reliable and meaningful standardization and evaluation methods for buyers, vendors, MT adopters, etc. is particularly serious. A crucial part of the survey focused on issues concerning translation quality from the point of view of the user, starting with their preferred approaches to assessment.

In general, human translation is quality checked always (38%) or regularly (37%) using internal (44%) or tool-specific (32%) assessment models, eventually in combination with standards such as EN 15038 (30%), ISO 9000 (27%) or LISA QA (15%).

The most popular choice in evaluating MT quality is human evaluation (69%), e.g. having internal staff or external experts assess the fluency and adequacy of MT output. Twenty-two percent indicated the use of state-of-the-art automatic evaluation metrics, such as BLEU, METEOR and TER, as another frequent choice, and 13% reported the adoption of in-house or internally developed automatic evaluation methods. Finally, 35% of the respondents opt for a combination of human and automatic evaluation methods, while 7% did not actually perform any formal quality assessment on their MT output.

Moving beyond the current practice and state-of-the-art
In sum, a clear trend is emerging that MT uptake is becoming more commonplace. Interesting steps are being taken to customize state-of-the-art MT systems to improve their output, yet many users may still face a knowledge and resource gap in the face of this advancement. Translation quality assessment is an area where there is a need to develop reliable and meaningful standardized methods of evaluation, and MT is no exception. Meaningful results in terms of high-quality MT output can then be substantiated with evidence of productivity gains (e.g. in post-editing scenarios), improvements in resource management (e.g. terminology management), and, savings in resources in the face of a growing diversity and volume of content.

QTLaunchPad focuses on identifying and overcoming the barriers to high-quality translation in a range of applied scenarios. Of particular relevance to this focus are evaluation metrics, quality estimation, and optimization of language resources. In this way, we’re preparing the groundwork for a large-scale community-based translation quality initiative. To learn more about these surveys and other QTLaunchPad activities:

  1. Visit the project homepage for more detailed information, accessible reportage, and listings of upcoming events: www.qt.21.eu/launchpad
  2. Follow the discussions and highlights on our LinkedIn group: http://www.linkedin.com/groups?gid=4807518 or connect with us on Twitter @qtlaunchpad and Facebook www.facebook.com/qtlaunchpad

Stephen Doherty, BA, HDip, PhD, MBPsS, is a post-doctoral researcher in the Centre for Next Generation Localisation in Dublin City University. He conducts research on topics of language and cognition, human-computer interaction, machine translation, and translation technologies. He is currently working on QTLaunchPad, a collaborative European research initiative dedicated to overcoming barriers in machine translation and language technologies.