Looking Under the Wrong Lamppost: The Limits of Automated Translation Quality Estimation

Automated Quality Estimation (QE) promises to cut translation costs by routing "good" segments past human review. Yet segment-level QE operates without the extra-segmental context—domain, discourse, terminology, and communicative intent—that determines translation correctness.

Drawing on our peer-reviewed study of over 104,700 MQM-annotated segments across European Commission, European Parliament, and benchmark corpora, we test QE as a ranking signal, probability estimate, and safety gate.

The speakers will discuss that correlation does not justify review bypass: serious errors occur across all score bands, probability calibration fails, and thresholds validated on one dataset do not transfer to another—even within the same language pair.

Key Takeaways

• Context determines correctness—and segment-level QE cannot see it: Isolated source–target segments lack the domain, discourse, and communicative context required to judge fitness for purpose, forcing segment-level QE to evaluate translations against a moving target.

• Correlation is not a safety gate: Statistical correlation or modest ranking lift (median ROC-AUC 0.59–0.63) does not validate bypassing human review—serious errors appear across the entire score range, and normalized QE scores are not calibrated error probabilities.

• Thresholds do not transfer across datasets: A cutoff calibrated to catch 90%+ of errors on one corpus fails on another, even within the exact same language pair (for example, dropping from 91.1% recall on DGT English–Greek to 66.5% on European Parliament data at the same 0.890 cutoff).

Target Audience

Enterprise localization buyers, localization program managers, LSP executives, MT/AI engineers, language quality managers, and operations leads responsible for designing AI-enabled translation workflows, quality gates, and risk management policies.