Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation environments. Such scenarios can be improved if quality predictions are accompanied by a measure of uncertainty. However, models in this task are traditionally evaluated only in terms of point estimate metrics, which do not take prediction uncertainty into account. We investigate probabilistic methods for Quality Estimation that can provide well-calibrated uncertainty estimates and evaluate them in terms of their full posterior predictive distributions. We also show how this posterior information can be useful in an asymmetric risk scenario, which aims to capture typical situations in translation workflows.
This paper considers estimation and testing using location measures for regression m odels that are based on an asymmetric least-squares criterion functio n. These estimators have properties that are analogous to regression quantiles, but are easier to calculate, as are the corresponding test statistics. Asymmetric least-squares tests of homoskedasticity and s ymmetry compare quite favorably with other tests of these hypotheses in terms of asymptotic relative efficiency. Consequently, asymmetric least-squares estimation provides a convenient and relatively efficie nt method of characterizing the conditional distributi on of a dependent variable given some regressors. Copyright 1987 by The Econometric Society.