The performance of speech emotion recognition is affected by the differences in data distributions between train (source domain) and test (target domain) sets used to build and evaluate the models. This is a common problem, as multiple studies have shown that the performance of emotional classifiers drops when they are exposed to data that do not match the distribution used to build the emotion classifiers. The difference in data distributions becomes very clear when the training and testing data come from different domains, causing a large performance gap between development and testing performance. Due to the high cost of annotating new data and the abundance of unlabeled data, it is crucial to extract as much useful information as possible from the available unlabeled data. This study looks into the use of adversarial multitask training to extract a common representation between train and test domains. The primary task is to predict emotional-attribute-based descriptors for arousal, valence, or dominance. The secondary task is to learn a common representation, where the train and test domains cannot be distinguished. By using a gradient reversal layer, the gradients coming from the domain classifier are used to bring the source and target domain representations closer. We show that exploiting unlabeled data consistently leads to better emotion recognition performance across all emotional dimensions. We visualize the effect of adversarial training on the feature representation across the proposed deep learning architecture. The analysis shows that the data representations for the train and test domains converge as the data are passed to deeper layers of the network. We also evaluate the difference in performance when we use a shallow neural network versus a deep neural network and the effect of the number of shared layers used by the task and domain classifiers.
Song al @cite_35 proposed a couple of methods based on (NMF) that utilized data from both train and test domains. The proposed methods aimed to represent a matrix formed by data from both domains as two non-negative matrices whose product is an approximation of the original matrix. The factorization was regularized by (MMD) to ensure that the differences in the feature distributions of the two corpora were minimized. The proposed methods aim to learn a robust low dimensional feature representation using either unlabeled data or labels as hard constraints on the problem. Abdelwahab and Busso @cite_34 proposed creating an ensemble of classifiers, where each classifier focuses on a different feature space (each classifier maximized the performance for a given emotion category). The features were selected over the labeled data from the target domain obtained with active learning. This semisupervised approach learned discriminant features for the target domain, increasing the robustness against shift in the data distributions between domains.
@cite_35 @cite_34
2503045345 2661967629
In this paper, we investigate the NMF algorithms for cross-corpus speech emotion recognition.We propose a transfer NMF scheme for cross-corpus speech emotion recognition, in which the NMF and MMD algorithms are jointly optimized.Two novel transfer NMF approaches, called transfer graph regularized NMF (TGNMF) and transfer constrained NMF (TCNMF), are presented, respectively.The optimization algorithms for TGNMF and TCNMF approaches are presented in detail. Automatic emotion recognition from speech has received an increasing amount of interest in recent years, and many speech emotion recognition methods have been presented, in which the training and testing procedures are often conducted on the same corpus. However, in practice, the training and testing speech utterances are collected from different conditions or devices, which will have adverse effects on recognition performance. To address this problem, in this paper, a novel cross-corpus speech emotion recognition method, called transfer non-negative matrix factorization (TNMF) is proposed. Specifically, the NMF approach, which is popular in computer vision and pattern recognition fields, is utilized to obtain low dimensional representations of emotional features. Meanwhile, the discrepancies between source and target data sets are considered, and the maximum mean discrepancy (MMD) algorithm is used for similarity measurement. Then, the TNMF method, which jointly optimizes the NMF and MMD algorithms, is presented. Moreover, to further improve the recognition performance, two variants of TNMF, called transfer graph regularized NMF (TGNMF) and transfer constrained NMF (TCNMF), are proposed, respectively. Several experiments are carried out on three popular emotional databases, and the results demonstrate the effectiveness and robustness of our scheme. When emotion recognition systems are used in new domains, the classification performance usually drops due to mismatches between training and testing conditions. Annotations of new data in the new domain is expensive and time demanding. Therefore, it is important to design strategies that efficiently use limited amount of new data to improve the robustness of the classification system. The use of ensembles is an attractive solution, since they can be built to perform well across different mismatches. The key challenge is to create ensembles that are diverse. This paper proposes the use of active learning along with feature selection to build a diverse ensemble that performs well in the new domain. The diversity and accuracy of the ensemble are achieved by (1) training emotional classifiers with bias toward specific emotions, (2) eliminating overlap in the feature sets of the ensemble, and (3) conducting feature selection by maximizing the performance over the new labeled data. We study various data selection criteria, and different sample sizes to determine the best approach toward building a stable diverse ensemble that generalize well on new domains.