How to Analyze Localization Quality Using Reviews

Black screen with the writing Game over in red color

Before discussing how we use reviews to improve on quality, we want to break down what localization quality is exactly and why it should be evaluated.

Projects are localized in order to ensure players get more enjoyment from a game, and developers get more profit by capturing new markets. This means that reviews from pleased (or angry) players can be used as the basis for evaluating the quality of our work.

We have an internal localization quality control system that we use to digitize the results of our linguists’ work and objectively evaluate them. This is not enough, however. It is possible to make a translation that is perfect from a technical point of view but leaves players unsatisfied. This is where working with review analysis comes into play.

Collection and Analysis of Reviews

Working with player reviews is done in several steps.

First, the reviews are downloaded from open sources (Google Play, Appstore, Steam).
From there, the results are combined into a single document and sorted. We are only interested in reviews of localization, so only segments that contain phrases like “localization,” “translation,” “language,” etc. are taken for further analysis. Every language has its own set of keywords.
Then the reviews we have collected are translated into English using machine translation. It’s not a perfect method, of course, but it suits these purposes.
Finally, a “sentiment” analysis is done to find out what the player wanted to say with their review. They might be content with the localization, upset by the low quality of the translation, or hoping for a chance to play the game in their native language.
The analysis results in a document containing relevant localization reviews, their translation and rating. The project team and client can then use the data.

What To Do With Results

So, now we’ve received a selection of relevant localization reviews sorted into various categories. This is not the final result. We want to use this data to determine a plan of action.

Positive Reviews

screenshot of a user review who gave 5 stars to a localized video game

Translation: "Very interactive and realistic game. The translation was really good."

First of all, getting reviews like these is a cause for celebration. Second of all, we note that the translation team working on this project (in this case, translating into Brazilian Portuguese) is skilled at translating games of this type. In the future, we will employ these linguists to work on similar projects.

Negative Reviews

Negative reviews are ranked within the group. Some reviews are written with strong emotions - the user isn’t satisfied with the game in its entirety (or a specific mechanic) and is “taking it out” on the translation.

Translation: "I’m giving a 4 not 5 because the combat and the translation into Russian aren’t very good. But at least there is one. Cool graphics, cool story, and character creation. The game is good but kinda boring."

This is not an indication of poor-quality localization, but we want to keep it in mind. During the next round of review collection (on average, this kind of work is done once every three months) we will need to do an especially thorough analysis of player opinion of localization into the mentioned language.

It’s also possible to receive more than one negative localization review. If a clear trend emerges, then it is a cause to start trying to figure out what’s the matter.

In a perfect world, we would ask the developer to get in touch with players and get examples of specific in-game errors (preferably with screenshots), then validate the errors and make adjustments to the localization. In the real world, the developer has about a thousand more important things to do, so to begin with, we try to locate the problem on our own. For example, we can send the game for LQA (during which it could easily turn out that properly translated phrases just don’t fit in the fields, or line breaks aren’t formatted properly).

Sometimes (not as often as we’d like, unfortunately) the error is described in the review itself. This makes our work easier and smooths over the sad realization that there is in fact an error.

Translation: "I play in Japanese, but sometimes it shows up as Hangul and I can’t read it. There are a lot of incorrect translations of the things I’m looking for, and in other areas. The translation of “accessories” is too abstract, and the translation of “flask” is incorrect. I really don’t know what “pin,” “glare,” “gloves,” and “gummy fish” are. The popular name, calla lily, should be used for calla lilies. I’m glad some things were fixed in the update, but Lily Cross was turned into Iris Cross and now it makes no sense."

If there are actually errors in the translation, we fix them and send the client an updated translation (this is done for free, as we give a lifetime guarantee for our work). If the problem isn’t in the quality of the translation, then we send an email explaining the current problems and suggest ways to fix them. For example, players might be dissatisfied with the level of formality or being addressed by the wrong gender.

Translation: "I think the game is good. I’ve met a lot of cool people, but the German translation is kind of weird, like it was done with Google Translate. And I’m always addressed as “he” even though I’m female."

This is not a translation error, but by removing issues that frustrate gamers, you can achieve a more loyal community and significantly boost engagement.

Requests for Translation

It’s common to see these types of reviews for games:

Translation: "The game needs to add support for Turkish, it’s not fun playing without it."

Does this mean that the game needs to be translated into Turkish or Russian ASAP? Not always. First, you should evaluate the financial capacity of residents of the target region (to help you do this, we publish annual research of games market on our blog), their current income, number of downloads and average purchase amount. But what we can say for sure is that requests like these are another reason to consider adding a new language.

It’s worth checking the number of requests before deciding whether to expand into a new market. For example, if a developer has a game that’s translated into EFIGS+CJK which is generating large profits, they start looking for growth points. One of these growth points is expanding into new language markets, and aside from metrics, the presence of requests for localization is a signal to choose that market in particular.

For our current projects, we conduct review analysis every three months. This is one of the ways to keep our finger on the pulse, perform quality control and make localization that players will like, and that will work to benefit the developer.