While objective metrics are valuable, subjective speech assessment by human raters is still considered the “golden standard" of speech quality assessment [1]} since human perception of audio quality is highly subjective [2]}. Subjective assessments are often conducted by aggregating perceptual ratings from multiple raters ranging from 1 (worst) to 5 (best) into a single MOS score per utterance. In contrast to objective metrics, raters can recognize and evaluate degraded utterances easily with minimal reference.
ITU-T P.808 [3]} defines a framework to reproducibly collect MOS-annotated data by providing tools for ensuring inter-rater reliability and spam filtering. Unfortunately, ratings can be noisy due to inherent differences in human perception even when adhering to current best practices [4]}. Moreover, subjective speech assessment is costly, complex and time consuming, severely limiting its scalability.
