Automatic Evaluation of Machine Translation: Moving Away from Word Matching Metrics
Thursday, 08 September, 2016
Automatic evaluation has been a critical component in the development of machine translation (MT) approaches. Hundreds of metrics have been created, mainly for evaluation during development (e.g. optimization of parameters in statistical MT systems) and for MT system comparison. The vast majority of these metrics (e.g. BLEU) are based on matching words or short sequences of words between the MT output and a pre-generated human translation.
However, with the recent widespread use of MT, other automated ways of assessing quality are necessary, given that human translations are not available for comparison in real-world settings and that the concept of quality in the translation industry goes beyond simple word matching. This webinar introduces trained metrics (a.k.a quality estimation metrics), which can be fine-tuned to specific quality needs and used to assess any number of new translations from an MT system. QuEst++, an open source toolkit to build quality estimation models, is also explained.
This webinar is part of the Research and Innovation Action "Quality Translation 21 (QT21)." This project has received funding from the European Union's Horizon 2020 program for ICT under grant agreement no. 645452.
Lucia Specia is Professor of Language Engineering in the Department of Computer Science of the University of Sheffield. Her research focuses on various aspects of data-driven approaches to multilingual language processing, with particular emphasis on machine translation and its evaluation. She is the recipient of an ERC Starting Grant on Multimodal Machine Translation and is currently involved in various other funded research projects, including the European initiatives QT21 (Quality Translation 21), Cracker (Cracking the Language Barrier) and EXPERT (Empirical Methods for Machine Translation). Before joining the University of Sheffield in 2012, she was Senior Lecturer at the University of Wolverhampton, UK (2010-2011), and research engineer at the Xerox Research Centre, France (2008-2009). She received a PhD in Computer Science from the University of São Paulo, Brazil, in 2008.
Most machine translation output is fine for basic Western LSP needs, as they usually translate within European...
Machine translation is great when it works, but it fails often and in unpredictable ways. What causes bad machine...