highlight a cross-dataset generalization error issue that we observed.

We present the individual results of our final models along with the benchmarks against two recent works on our development set, and attach the ConferencingSpeech 2022 challenge results.
The challenge provides two baseline models. The models are trained on random splits of the challenge datasets where the specification of train and development split of the datasets are not available. As a result, we are unable to compare our models directly with the baselines with the available data. Hence, we benchmark the latest NISQA [1]} model (NISQA v2.0) and the DNSMOS [2]} model on our development datasets to provide a fair comparison with our proposed methods. We evaluate our results using average Pearson’s correlation coefficient (PCC) and root mean squared error (RMSE).
<TABLE>Ablation Study
<TABLE>Context-aware features
In Table REF , we perform ablation studies on the context-aware features by replacing them with frame-wise inputs, while keeping the remaining model configuration identical to Model 1. Compared to the results without context-aware features in Table REF , Model 1 in Table  shows across the board performance improvements with PCC scores up and RMSE down by an average of 0.015 and 0.045, respectively, across the development sets.

Generalization error across datasets
In our benchmarks of two previously published works [1]}, [2]}, we notice significant performance inconsistencies across our development datasets. As shown in Table , the NISQA model performs noticeably better on the NISQA test datasets, whereas the DNSMOS struggles most on them (i.e. NISQA shows the highest PCC at 0.891 on the NISQA FOR testset, but an average of around 0.702 on the two challenge testsets; DNSMOS scores below 0.586 PCC on both NISQA testsets). This raises concerns about model generalization across varied domains.
We conduct further ablation studies by removing the PSTN training data in the training of Model 1, the result of which is shown in Table REF . We notice the performance remains similar in the other three development sets but drops off significantly in the PSTN development set, where PCC decreases from 0.754 to 0.654.
Although a thorough discussion of generalization error [5]} in MOS prediction is beyond the scope of this work, we highlight the fact that model performance can be highly dataset sensitive in this task, even for corpora with outwardly similar content.

Convolutional Context-aware Transformer
Moving on to our proposed models, the individual results for the three selected models are presented in Table , we note that despite Model 3 outperforming Model 1 and Model 2 on all development datasets, the ensemble prediction further improves overall robustness in both PCC and RMSE.
Comparing to the work of NISQA and DNSMOS, we are able to achieve significantly better performance on the two challenge datasets, with PCC around 0.901 and RMSE around 0.452 for the Tencent corpus and PCC around 0.779 and RMSE around 0.559 for the PSTN corpus. However, NISQA retains the best performance on its own test corpus.
Finally, we present the results for the challenge evaluation test set in Table REF . The baseline model is a simplified version of the state-of-art NISQA model[1]}. This model uses a mel-spectral input to LSTM layers followed by several fully connected layers. An average pooling layer then used to average the predicted frame-level MOS and output the final predicted MOS score. The metrics also include the epsilon-insensitive RMSE after a 3rd order polynomial monotonic mapping (RMSE*3rd) [7]}, [8]} which takes into account the confidence interval of the individual MOS scores.
The results indicate that our proposed model shows better performance across all three datasets under three different evaluation metrics. Overall, our CCAT model, reaches a 0.697 PCC while the baseline model only reach 0.53 PCC, indicating that the our model has a higher correlation to subjective speech assessment ratings. The RMSE and RMSE*3rd over all eval datasets for our proposed model is also lower than the baseline model. The proposed model reaches a 0.57 RMSE and 0.401 RMSE*3rd compare to 0.768 RMSE and 0.497 RMSE*3rd from the baseline model. Therefore, we can conclude that our proposed CCAT model shows promising results in the robust prediction of MOS scores under varied conditions.
<TABLE>
Conclusions
Subjective quality assessment by human raters is the best method available for assessing speech quality. However, it is time consuming and costly to conduct. In this paper, we proposed a novel end-to-end non-intrusive speech quality assessment model called convolutional context-aware transformer network or CCAT. The proposed model architecture provides an accurate MOS prediction across both real and simulated conditions. The overall performance of our model is further improved by using hyper-parameter tuning and model ensembling techniques.
We also highlight the issue of poor model generalization across MOS corpora. We observe this behavior in our model as well as several other models from recent literature trained on cross-domain dataset splits. We hypothesize that this is likely a byproduct of the limited scale and variety of current MOS prediction datasets and may be exacerbated by the inherently noisy nature of subjective MOS ratings. As future work, we believe the robustness of the proposed model could be further improved with a larger and more diverse datasets as well as methods for cross-domain regularization.
