Despite the widespread use of supervised deep learning methods for affect recognition from speech, they are severely limited by the lack of a sufficient amount of labelled speech data. Considering the abundant availability of unlabelled data, this paper proposed a semi-supervised model that can effectively utilise the unlabelled data in multi-task learning way in order to improve the performance of speech emotion recognition. The proposed model adversarialy learns a shared representation for two auxiliary tasks along with emotion identification as the main task. We consider speaker and gender identification as auxiliary tasks in order to operate the model on any large audio corpus. We demonstrate that in a scenario with limited labelled training samples, one can significantly improve the performance of a supervised classification task by simultaneously training with additional auxiliary tasks having an availability of large amount of data. The proposed model is rigorously evaluated for both categorical and dimensional emotion classification tasks. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on two publicly available datasets.
Another stream of research, instead of using different emotional attributes as auxiliary tasks, has utilised other available attributes, such as speaker and gender to improve the performance of SER @cite_52 . For instance, @cite_33 used gender and naturalness (natural or acted corpus) recognition as auxiliary tasks to improve the performance of emotion recognition using different emotional databases. @cite_21 used MTL approach to investigate the influence of domain (whether the expression is spoken or sung), corpus, and gender on the generalisability of cross-corpus emotion recognition systems. The authors used six emotional databases and showed that the performance of SER system is increased with the number of training emotional corpora.
@cite_21 @cite_52 @cite_33
2598545578 2749894918 2322208196 2016839396
There is growing interest in emotion recognition due to its potential in many applications. However, a pervasive challenge is the presence of data variability caused by factors such as differences across corpora, speaker’s gender, and the “domain” of expression (e.g., whether the expression is spoken or sung). Prior work has addressed this challenge by combining data across corpora and or genders, or by explicitly controlling for these factors. In this work, we investigate the influence of corpus, domain, and gender on the cross-corpus generalizability of emotion recognition systems. We use a multi-task learning approach, where we define the tasks according to these factors. We find that incorporating variability caused by corpus, domain, and gender through multi-task learning outperforms approaches that treat the tasks as either identical or independent. Domain is a larger differentiating factor than gender for multi-domain data. When considering only the speech domain, gender and corpus are similarly influential. Defining tasks by gender is more beneficial than by either corpus or corpus and gender for valence, while the opposite holds for activation. On average, cross-corpus performance increases with the number of training corpora. The results demonstrate that effective cross-corpus modeling requires that we understand how emotion expression patterns change as a function of non-emotional factors. One of the challenges in Speech Emotion Recognition (SER) "in the wild" is the large mismatch between training and test data (e.g. speakers and tasks). In order to improve the generalisation capabilities of the emotion models, we propose to use Multi-Task Learning (MTL) and use gender and naturalness as auxiliary tasks in deep neural networks. This method was evaluated in within-corpus and various cross-corpus classification experiments that simulate conditions "in the wild". In comparison to Single-Task Learning (STL) based state of the art methods, we found that our MTL method proposed improved performance significantly. Particularly, models using both gender and naturalness achieved more gains than those using either gender or naturalness separately. This benefit was also found in the high-level representations of the feature space, obtained from our method proposed, where discriminative emotional clusters could be observed. In this letter, a novel cross-corpus speech emotion recognition (SER) method using domain-adaptive least-squares regression (DaLSR) model is proposed. In this method, an additional unlabeled data set from target speech corpus is used to serve as an auxiliary data set and combined with the labeled training data set from source speech corpus for jointly training the DaLSR model. In contrast to the traditional least-squares regression (LSR) method, the major novelty of DaLSR is that it is able to handle the mismatch problem between source and target speech corpora. Hence, the proposed DaLSR method is very suitable for coping with cross-corpus SER problem. For evaluating the performance of the proposed method in dealing with the cross-corpus SER problem, we conduct extensive experiments on three emotional speech corpora and compare the results with several state-of-the-art transfer learning methods that are widely used for cross-corpus SER problem. The experimental results show that the proposed method achieves better recognition accuracies than the state-of-the-art methods. One of the ever-present bottlenecks in Automatic Emotion Recognition is data sparseness. We therefore investigate the suitability of unsupervised learning in cross-corpus acoustic emotion recognition through a large-scale study with six commonly used databases, including acted and natural emotion speech, and covering a variety of application scenarios and acoustic conditions. We show that adding unlabeled emotional speech to agglomerated multi-corpus training sets can enhance recognition performance even in a challenging cross-corpus setting; furthermore, we show that the expected gain by adding unlabeled data on average is approximately half the one achieved by additional manually labeled data in leave-one-corpus-out validation.