Despite the widespread use of supervised deep learning methods for affect recognition from speech, they are severely limited by the lack of a sufficient amount of labelled speech data. Considering the abundant availability of unlabelled data, this paper proposed a semi-supervised model that can effectively utilise the unlabelled data in multi-task learning way in order to improve the performance of speech emotion recognition. The proposed model adversarialy learns a shared representation for two auxiliary tasks along with emotion identification as the main task. We consider speaker and gender identification as auxiliary tasks in order to operate the model on any large audio corpus. We demonstrate that in a scenario with limited labelled training samples, one can significantly improve the performance of a supervised classification task by simultaneously training with additional auxiliary tasks having an availability of large amount of data. The proposed model is rigorously evaluated for both categorical and dimensional emotion classification tasks. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on two publicly available datasets.
Another stream of research, instead of using different emotional attributes as auxiliary tasks, has utilised other available attributes, such as speaker and gender to improve the performance of SER @cite_52 . For instance, @cite_33 used gender and naturalness (natural or acted corpus) recognition as auxiliary tasks to improve the performance of emotion recognition using different emotional databases. @cite_21 used MTL approach to investigate the influence of domain (whether the expression is spoken or sung), corpus, and gender on the generalisability of cross-corpus emotion recognition systems. The authors used six emotional databases and showed that the performance of SER system is increased with the number of training emotional corpora.
@cite_21 @cite_52 @cite_33
2598545578 2963929227 2749894918
There is growing interest in emotion recognition due to its potential in many applications. However, a pervasive challenge is the presence of data variability caused by factors such as differences across corpora, speaker’s gender, and the “domain” of expression (e.g., whether the expression is spoken or sung). Prior work has addressed this challenge by combining data across corpora and or genders, or by explicitly controlling for these factors. In this work, we investigate the influence of corpus, domain, and gender on the cross-corpus generalizability of emotion recognition systems. We use a multi-task learning approach, where we define the tasks according to these factors. We find that incorporating variability caused by corpus, domain, and gender through multi-task learning outperforms approaches that treat the tasks as either identical or independent. Domain is a larger differentiating factor than gender for multi-domain data. When considering only the speech domain, gender and corpus are similarly influential. Defining tasks by gender is more beneficial than by either corpus or corpus and gender for valence, while the opposite holds for activation. On average, cross-corpus performance increases with the number of training corpora. The results demonstrate that effective cross-corpus modeling requires that we understand how emotion expression patterns change as a function of non-emotional factors. Long short-term memory (LSTM) is normally used in recurrent neural network (RNN) as basic recurrent unit. However, conventional LSTM assumes that the state at current time step depends on previous time step. This assumption constraints the time dependency modeling capability. In this study, we propose a new variation of LSTM, advanced LSTM (A-LSTM), for better temporal context modeling. We employ A-LSTM in weighted pooling RNN for emotion recognition. The A-LSTM outperforms the conventional LSTM by 5.5 relatively. The A-LSTM based weighted pooling RNN can also complement the state-of-the-art emotion classification framework. This shows the advantage of A-LSTM. One of the challenges in Speech Emotion Recognition (SER) "in the wild" is the large mismatch between training and test data (e.g. speakers and tasks). In order to improve the generalisation capabilities of the emotion models, we propose to use Multi-Task Learning (MTL) and use gender and naturalness as auxiliary tasks in deep neural networks. This method was evaluated in within-corpus and various cross-corpus classification experiments that simulate conditions "in the wild". In comparison to Single-Task Learning (STL) based state of the art methods, we found that our MTL method proposed improved performance significantly. Particularly, models using both gender and naturalness achieved more gains than those using either gender or naturalness separately. This benefit was also found in the high-level representations of the feature space, obtained from our method proposed, where discriminative emotional clusters could be observed.