How Much Can Machine Translation Help You? Industry Data Trends
One of the perks of cloud translation platforms is that data is centralized. This means it is possible to get a bird's eye view of certain language industry trends such as Machine Translation (MT) usage and effectiveness. At the moment, 'big data' in translation tools is only beginning to emerge, and developers are still learning how to use it. I believe that in the future, all cloud tools will develop insights and notifications based on user statistics, similar to how freelance marketplace Upwork identifies the top 10% of freelancers in a given field. However, what we can already do is spot useful trends, confirm hypotheses, and bust myths.
As a researcher, I was overjoyed when we introduced data tools into Memsource earlier this year. The following are some of the compelling pieces of information we have gleaned from this collection of industry data.
MT Leverage: Only 5-20% of MT Output is "Good to Go"
For the first study, my colleagues and I decided to look at machine translation (MT) to find out to what extent exactly it can help professional translators. Surveys by industry research bodies suggest that up to 30% of translation companies use MT. This is understandable given how low the EBIT margins are and how every cent saved counts toward company profitability. At the same time, in any translation conversation forum, the rage of translators against the machine is so hot it practically melts keyboards.
I took a look at the collected data to see if it could provide any basis to claims on either side.
Results of the analysis showed that only 5-20% of suggestions from generic MT engines are good enough to use without any changes. Up to 40% of the suggestions are okay after edits, and for 80% of segments MT can provide data to autocomplete.
In the chart above (click image to enlarge), the match number indicates the similarity between the translation and the suggestion from the MT engine:
- Match 100: segments where the professional human translation is identical to the MT suggestion
- Match 85-95: MT suggestions are close enough to use after edits
- Match 50-75: MT is useful for autocompletion of individual words, but not whole segments
- Match 0 represents segments with 0-49 correlation to human translation
Please note: It was only possible to track projects where users first enabled machine translation, and then performed a post-editing analysis. This means the sample here is about 38 million words in size.
Machine Translation Works Better with French, English, Spanish, and Portuguese
Results from the data indicate that French, Portuguese, Spanish, and English MT engines have the highest rates of potential MT leverage. English to French stands out in particular, with more than 20% of the translations being a complete match to MT suggestions and almost 90% of segments having at least some coherence with the MT.
In comparison, Russian, Polish, and Korean have much lower leverage rates of less than 40%, or even 20%, fuzzy matches and 5% complete matches.
This difference is probably due to the morphological typology of the languages. French, Portuguese, Spanish, and English are analytic languages, meaning they rely on word order and auxiliary words such as “are” or “will” to convey meaning. Russian, Polish, and Korean on the other hand are synthetic, which means they use inflections much more. Machine translation is still struggling with those nuances.
MT is "On" for 32% of Projects
In Memsource, just like in other CAT-tools, users can add a machine translation engine to get quick suggestions as they go through the text segment by segment. This process is called “interactive MT post-editing,” and the difference from "classic" post-editing is that translators are not obliged to use MT suggestions for each segment. If a suggestion is too far off they can simply ignore it and write from scratch.
For the MT option, translators commonly utilize generic engines such as Google Translate or Microsoft Translator. This study therefore represents the effect of these engines first and foremost, and does not represent customized engines that have been fine-tuned for specific terminology and subject area domains.
As described, not everyone is a fan of machine translation. Translators and language service companies believe it sometimes compromises quality or interferes with the mental focus and self-discipline of the linguist which are necessary to deliver a perfect translation. Only about 32% of projects in Memsource have MT support enabled. Even this figure is only this high because the "on" option is set as a default.
About two years ago we reported that 48% of projects come with some MT support. Since that time, the average throughput in our system has grown from 100 million words per month to 600-800 million, while the share of MT-enabled projects has dropped. I predict this means that our userbase is becoming more representative of the translation industry as a whole. Memsource used to be a tool where volume was dominated by a few tech-savvy companies, whereas today our users come from all walks of the translation industry. This helps explain the drop in percentage.
What Comes Next?
Will MT usage increase? No doubt. Online conversation today calls for a fast-paced translation. Customers need faster turnaround times more than they need high quality. “Done is better than perfect,” says the famous slogan of Facebook.
At the same time, MT accuracy is constantly improving. It’s not yet human-level, but it is getting there. Hence, there are ever more arguments for enabling MT.
These are just some of the findings from the collected data. We look forward to sharing more over the coming months!