A Crowdsourcing Extension of the ITU-T Recommendation P.835 with Validation

The quality of the speech communication systems, which include noise suppression algorithms, are typically evaluated in laboratory experiments according to the ITU-T Rec. P.835. In this paper, we introduce an open-source implementation of the ITU-T Rec. P.835 for the crowdsourcing approach following the ITU-T Rec. P.808 on crowdsourcing recommendations. The implementation is an extension of the P.808 Toolkit and is highly automated to avoid operational errors. To assess our evaluation method's validity, we compared the Mean Opinion Scores (MOS), calculate using ratings collected with our implementation, and the MOS values from a standard laboratory experiment conducted according to the ITU-T Rec, P.835. Results show a high validity in all three scales (average PCC = 0.961). Results of a round-robin test showed that our implementation is a highly reproducible evaluation method (PCC=1.00). Finally, we investigated the performance of five models deep noise suppression models using our P.835 implementation and show what insights can be learned.


INTRODUCTION
Speech calls can be carried out with various devices in different environments, commonly with a nonoptimal acoustic surrounding. Therefore, noise suppression algorithms are widely integrated into the communication chain to enhance the quality of the speech communication system. Those systems are typically evaluated in laboratory-based listening tests according to the ITU-T Rec. P.835 [1] in which separate rating scales are used to independently estimate the quality of the Background noise, the Speech Signal and the Overall quality alone. Separate scales are used as the higher noise suppression level often adversely affects the speech or the signal component. Consequently, in a regular listening-only test, with a singlerating scale according to the ITU-T Rec. P.800, participants can often become confused as to what they should consider in rating the overall "quality". Accordingly, each individual determines her overall quality rating by weighting the signal and background components. Such a process introduces additional error variance in the overall quality ratings and reduces their reliability [1].
In addition, laboratory-based speech quality experiments are more and more replaced by crowdsourcing-based online tests, which are carried out by paid participants. Crowdsourcing offers a faster, scalable, and cheaper approach than traditional laboratory tests. Still, with its challenges: the test participants take part in the test in their working environment using their hardware without a test moderator's direct supervision. The ITU-T Rec, P.808 [2] addresses those challenges and provide methods to collect reliable and valid data in the crowdsourcing practice. In this work, we followed methods described in the ITU-T Rec. P.808 and implemented the P.835 test procedure adapted to the crowdsourcing approach, which we will show is a valid and reliable evaluation method.
In the following, we introduce the open-source implementation of the P.835 and report our evaluations' results. Section 2 describes the toolkit's implementation and different components; Section 3 reports the validity and Section 4 the reproducibility studies we conducted; Section 5 reports the evaluation of different models from INTERSPEECH 2020 DNS challenge and the relation between the three scales used in the P.835 test. Finally, Section 6 discusses the findings and proposes steps for future work.

IMPLEMENTATION
We have extended the open-source P.808 Toolkit [3] with methods for evaluating speech communication systems that include noise suppression algorithm 1 . We followed the ITU-T Rec. P.835 [1] and adapted it for the crowdsourcing approach based on the ITU-T Rec. P.808 [2].
The P.808 Toolkit contains scripts for creating the crowdsourcing test (generate the HTML file, trapping stimuli, input URLs, etc.) and also a script for processing the submitted answers (i.e. data screening and aggregating the reliable ratings). The test includes several sections. In the Qualification section relevant demographic questions are asked and the hearing ability of test participants are examined using a digittriplet test. In Setup section, usage of both ear-pods and suitability of the participant's environment are evaluated using the modified just-noticeable difference in quality test method [4]. In the training section, the test participant is introduced to the rating procedure and familiarized with the rating scale by rating a predefined set of stimuli. The stimuli in the training set should cover the entire range of the scales. The last section is the ratings section in which the participant listens to a set of stimuli and rate them on the given scales. As the crowdsourcing task should be short, therefore, it is recommended to use about ten stimuli in the rating section. The participant can perform one or more tasks. The qualification section only appears once and if the participant passes the test it will not be shown in the next tasks. The setup and training sections appear once a while (every 30 and 60 minutes, respectively) when the worker passed them successfully. We kept the structure of the P.808 Toolkit the same; further details about the P.808 Toolkit can be found in [3] and details on validation of the ITU-T Rec. P.808 in [5].
In the ITU-T Rec. P.835 subjective test, participants are ask to successively attend to and rate the stimulus on the speech signal (from 1-Very distorted to 5-Not distorted), the background noise (from 1-Very intrusive to 5-Not noticeable) and the overall quality (from 1-Bad to 5-Excellent) scales. In our implementation, one stimulus is used for all three ratings [1]. Participants are forced to listen to the stimuli again before rating on each scale attending only to the scale's specific aspect 2 . The presentation order of speech signal and background noise scales are randomized in each task to avoid any order effect. The overall quality scale is always the last rating in the sequence.
In every crowdsourcing task, it is recommended to include trapping and gold questions [6,7]. The trapping stimuli is an obvious quality control mechanism [7] which asks the participant to select a specific response to show their attention. For the P.835 extension, the trapping question asks the participant to select a specific score rather the category's label.

Reference Conditions
The reference conditions should be used in every subjective test, evaluating noisy speech, to independently vary the signal and background ratings through their entire range of scale values [1]. In the ITU-T Rec. P.835, Speech-to-Noise ratio (SNR) is used for varying the background noise (from 0 to 40 dB) and the Modulated Noise Reference Unit (MNRU) [8] for varying the signal rating (from 8 to 40 dBQ). Overall, 12 reference conditions are recommended. However the previous body of works showed that MNRU processing is not appropriate as a reference system for the signal rating scale primary because the degradation in the speech signal by the noise canceller is very different to the degradation resulting from the MNRU processing [9]. Preliminary results from our crowdsourcing test and also expert review showed that software based MNRU degradation 3 leads to a higher MOS rating than what is reported in the recommendation. Therefore, we applied the twelve reference conditions as propose in ETSI TS 103 281 [11] (Table 1) in which the spectral subtraction based distortion is used for degrading the speech signal. It is based on the Wiener filter and leads to signal distortions similar to the one created by the noise cancellers. We used tools provided in [12], clean signals from [13] and noise signals from [11] to create the reference conditions.

VALIDATION
We conducted a crowdsourcing test using our P.835 extension of the P.808 Toolkit and compared its results with tests conducted in the laboratory according to the ITU-T Rec. P.835. In the test, we used four fullband speech files (2 male, 2 female) from the ITU-T P.501 Annex C [13] and applied the above-mentioned twelve reference conditions on them. On average we have collected 86.17 valid votes per test condition. Our results show high correlations to the openly available auditory results conducted in a laboratory from [14] (c.f. Table 2). Figure 2 illustrates the results of the crowdsourcing test. The overall quality ratings tend to be close to the minimum of signal and background noise rating.

REPRODUCIBILITY STUDY
In [15], the authors carried four subjective listening tests in three different laboratories to investigate inter-and intra-lab test result repeatability of P.835 methodology. The Pearson correlation between tests was high and above 0.97 in all cases.
We conducted a reproducibility study using 5 DNS models and an unprocessed dataset using the DNS Challenge blind test set [16]. The blind test set has 700 clips. A P.835 run was done N=2 times 4 , on separate days and with mutually exclusive raters. The results are shown in Figure 3, and show very good reproducibility (c.f. Table 4).

DEEP NOISE SUPPRESSION STUDY
Hu et al. [18] estimated the relationship between the background noise (BAK), the speech signal (SIG), and the overall quality (OVRL) as following using the NOIZEUS dataset.
with adjusted R 2 = 0.99 and ρ = 0.99, which is similar to the relation [18] estimated with a different dataset. We  We next look at 5 representative DNS models from the previous set of 13. The results of P.835 for these models are reported in Table 4. Each model was evaluated with N ∼ 2600 ratings and the resulting 95% confidence interval was 0.04 for MOS values of BAK, SIG, and OVRL. These results show that the best overall performing model, Model5, achieves OVRL DMOS =0.87 performance by very significant noise suppression (BAK DMOS =1.94) and with effectively no speech improvement (SIG DMOS =-0.02). The Model5 also did not add extra impairment to the speech signal compare to the other models. There is an additional BAK DMOS =0.65 improvement that can be made in noise suppression (comparing to the reference condition i01, i.e. no degradation, with BAK MOS =4.88). Using the Eq. 2 for BAK MOS =5 and SIG MOS =3.92, the predicted OVLR MOS is 3.98, which is an estimate of the best DNS we can do without improving signal quality. To get significantly better performance the speech signal must be improved, such as through dereverberation or capture device distortion removal.

DISCUSSION AND CONCLUSION
We have provided an open-source implementation of the ITU-T Rec.P.835 for evaluating noise suppression algorithms using subjective tests conducted in the crowdsourcing approach. We followed the ITU-T Rec. P.808 and applied the recommended hearing, device usage, and environment suitability tests, as well as the gold standard and trapping questions in the crowdsourcing tasks to ensure the reliability of the estimations. We provide our P.835 implementation as an extension for the P.808 Toolkit. Following the same structure, the toolkit is highly automated to avoid operational errors. We conducted a validity study in which we observed high correlations between MOS values of the three scales calculated on ratings collected by our toolkit and the MOS values from the standard ITU-T Rec. P.835 laboratory-based tests (average PCC = 0.961).
We examined the reproducibility of the subjective ratings collected by our implementation. We collected ratings for 4200 clips, including clips processed by five DNS models and the unprocessed ones, in two runs in separate days with mutually exclusive raters. Results show a very good reproducibility (average PCC = 1.00, SPCC = .98, SPCC after transformation = 0.97).
We also evaluated 13 DNS models using our P.835 implementation. Results show that the best model increases the background noise quality by 1.94 MOS, reaching to 4.23 MOS, without adding extra distortion to the signal quality. Consequently, significant improvement in the performance of the state-of-the-art DNS models can only be achieved by speech signal enhancement.
We observed that the P.835 test duration is significantly longer than the P.808 ACR test, as participants need to listen to the speech clips three times. The cost of the P.835 increases by 2X compared to P.808, which is less than 3X due to the common qualification overhead in both P.808 and P.835.