CMT — A New Machine Translation Quality Indicator Developed by CM Localization Center
CM Localization Center developed a machine translation (MT) quality indicator that allows for the assessment of machine translation quality without having to compare the machine and human translations. This is a real breakthrough in MT quality assessment, as so far MT assessment was only possible “post factum” after the translation was completed and edited by a human. Our solution enables one to decide whether machine translation really shortens the project turnaround time, or whether using MT is unprofitable.
A machine translation quality indicator that allows users to assess the quality of such translation without any comparative material in the form of a finished human translation has been the subject of numerous research efforts, and some call it the “Holy Grail of MT”. At the present it is impossible to develop an algorithm for completely automatic calculation of such an indicator, as this requires practical understanding of the text by a computer. On the other hand, having a human read and assess the quality of machine translation is both time and cost-consuming. Since such indicator is absolutely necessary, we attempted a hybrid approach. In our solution, humans only assess several dozen randomly selected sentences, but without reading them carefully; instead, one of the 4 options is selected: “Correct”, “Almost correct”, “Acceptable”, or “Unacceptable”. A computer then statistically processes the results and calculates the indicator that takes values in the range between 0 and 100 — as in the case of existing indicators. The task takes around 5 minutes for a human to complete, irrespective of the volume of text.
This January, we completed comparative tests on a sample consisting of 60 projects, comparing the CMT value obtained with the two known indicators: BLEU and METEOR. It appeared that the results are on average consistent in more than 90% of cases. Such high correlation proves that the method employed is correct and the CMT may be used for assessing the quality of any machine translation, even if no comparative material exists. This method will allow companies to decide whether machine translation could help in translating a specific text or whether using it is unprofitable.
For more information, please visit http://www.cmlocalization.eu/EN/
CM Localization Center is one of the leading language service providers operating in the Central and Eastern European market. CM is carrying out advanced works on machine translation (MT), human-aided machine terminology management and quality control tools. The company is a member of international and domestic trade associations, including GALA, TAUS Data Association, and PSBT.