minimizing the residual between \(\mathbf {z_d}\)  and \(\mathbf {\hat{z}_d}\)  . Therefore, the loss factor corresponding to the slope between \(\mathbf {z_d}\)  and \(\mathbf {l_d}\)  is given as:
\(\mathcal {L}_{\text{sl}} = \left\Vert  \frac{\mathbf {\hat{z}_d}(a_1)-\mathbf {\hat{z}_d}(a_2)}{\mathbf {{l}_d}(a_1)-\mathbf {{l}_d}(a_2)} -1\right\Vert _2,\) 
where \(a_1, a_2\)  are two arbitrary data instances.
Furthermore, the residual component is the standard mean squared error:
\(\mathcal {L}_{\text{res}} = \mathbb {E}\Vert \mathbf {z_d}-\mathbf {\hat{z}_d} \Vert _2^2.\) 
Therefore, our proposed total metric loss is composed of:
\(\mathcal {L}_{\text{met}} = \mathcal {L}_{\text{res}} + \mathcal {L}_{\text{sl}}\) 
The final equation for the DAE with metric learning is given as:
\(\begin{aligned}\text{arg}\min _{f_\theta , g_\phi } \quad \mathcal {L}_{\text{rec}}+\mathcal {L}_{\text{met}},\end{aligned}\) 
where \(\mathcal {L}_{\text{rec}}\)  has the same formulation as in Eq.  REF
Pair-wise distance preservation
<TABLE>To obtain an insight on the effectiveness of the proposed metric loss, we study the correlation between \(\mathbf {z_d}\)  and \(\mathbf {l_d}\)  by using ordinary least squares (OLS) to model \(\mathbf {l_d}\)  as a function of \(\mathbf {z_d}\)  as follows: \(\mathbf {l_d} = c+\beta _1\mathbf {z_{d_1}}+\beta _2\mathbf {z_{d_2}}\) . The assumption of a linear relation between the pair-wise point distances in the latent space and the labels is motivated from our proposed metric loss. The \(R^2\)  adjusted is presented in Table REF .

Experimental setup
In this section, we describe the dataset and the input feature space, the architecture of the models, the pre-processing steps followed by the description of the methods of comparison and our evaluation setup.
Datasets and input features:
IEMOCAP, an audio-visual affect data set, is used to train and validate the models [1]}. The data set comprises of annotations representing both the categorical and dimensional emotional model [2]}. The models are trained with data from the emotional categories neutral (N), sad (S), happy (H), angry (A). We use the extended Geneva minimalistic acoustic parameter set (eGeMAPS) [3]}, specifically the functionals of lower-level features. Each speech sample yields a feature vector comprising of 88 features and we use the OpenSmile toolkit to extract the features [4]}. To study how the latent representations are transferred between corpora and languages, we test classification accuracy over the following transfer datasets :
the Surrey Audio-Visual Expresses Emotion (SAVEE) database that is primarily English and consists of male speakers only,

the Berlin Database of Emotional Speech (Emo-DB) recorded in German and,

the Canadian French Emotional (CAFE) speech database comprising of French audio samples [5]}, [6]}, [7]},

the URDU-dataset consisting of Urdu speech, and

the Acted Emotional Speech Dynamic (AESD) Database comprising of Greek speech [8]}, [9]}. Note that AESD does not comprise of speech samples from the neutral class, whereby evaluation includes AESD speech samples from three and two classes only. For correlation analysis, due to the lack of activation and valence labels, we annotate the following datasets:

EMO-DB,

URDU

AESD and

CAFE . These labels are then used to test the baseline DAE-unsupervised, and the proposed DAE-metric-act and DAE-metric-val models.

System architecture:
Past works using AEs and variants have mostly addressed novel network architectures for better classification accuracy [10]}, [11]}, [12]}, [13]}, [14]}. However, since the focus of this work is to investigate the potential of metric learning in obtaining a more transferable embedding space for emotion recognition, we employ a simple architecture for the DAE baseline with performance that is comparable to existing methods. The proposed methods share the same architecture as the baseline model, illustrated in Fig. REF . The size of the input features is 88 and we compress the latent space to two dimensions. As described above, the input feature vector is comprised of the descriptive statistics of each speech signal, without temporal correlation. Therefore, in the encoder and decoder we employ fully connected NNs, instead of convolutional or recurrent NNs. The fully connected layers are followed by rectified linear units (ReLU) to incorporate non-linearity within the model.


Preprocessing:
Prior to using the data sets for training and testing, we remove the outliers by computing the z-score and eliminating the data samples that have a z-score, \(-10>z>10\) . We chose a threshold of 10 instead of the standard value of 3 because the goal of this work is to understand the behavior of the models for both typical and atypical rendition of emotions in speech. Therefore, we only remove the extreme outliers. For run-time evaluation over the transfer datasets, we employ statistics from 20% of the transfer data set to standardize the remaining 80%.


Proposed methods:
We train and validate the following models:
DAE-Unsupervised is used as a baseline,

DAE-Metric-Act that utilizes the proposed semi-supervision via the activation labels from Sec. REF , and

DAE-Metric-Val utilizes the proposed semi-supervision via the valence labels (Sec. REF ). 
The input features to the DAE is corrupted by a noise component, \(x_{\text{input}}=x_{\text{true}}+N\)  and \(N\in \mathcal {N}(0, 1)\) . We use the mean squared error (MSE) to optimize the baseline model as shown in Eq: REF . To study the consistency of the results, 5-fold cross-validation is used on the IEMOCAP database while the transfer data sets are identical over the iterations and are used for model testing only. The models were trained over 50 epochs with a batch size of 64, and we used the Adam optimizer with the learning rate set to 1e-3. Additional methods used for reference are listed in the following section.
Reference methods:
In addition to comparing the performance of the DAE models with the proposed metric loss to the unsupervised DAE, we employ the following as reference methods to gauge the relative performance of the models developed in this work:


1. 
Since the DAE-unsupervised model is employed as a baseline to evaluate the effectiveness of the metric-loss, we compared its classification performance with similar methods from literature as presented in Table REF .


2.
We implemented the support vector classifier, SVC, trained separately on all data sets towards the downstream task of classifying the input eGeMAPS features into target classes. The results are shown in Table REF . We use the supervised SVC to study the upper-bound of the performance limits of the SER task. However, the work in this paper is focused on enabling reliable SER for languages with few or no labelled data, and supervised learning is therefore inapplicable within this scope.


3.
We used the SUPERB: Speech processing Universal PERformance Benchmark for emotion classification on the transfer datasets [15]}. The method employs the HuBERT (Hidden-Unit BERT), specifically hubert-large-ll60k model, as the base model to first extract speech representations [16]}. The speech representations are then given as input to a linear downstream task to classify the speech signal into emotions. The base model is trained on English data sets and the linear downstream model for emotion classification is trained on the IEMOCAP dataset. We employ the model for classification only as the model is specifically trained for that. The results are shown in Table REF . Relative to the complexity of the models developed in this paper (\(<4\times 10^2\)  parameters), note that the considered model is highly complex (\(>3\times 10^8\)  parameters).


4.
Lastly, besides comparing the correlation between the embedding and the labels for the proposed models and the unsupervised model, we employ supervision to train models with the proposed metric losses. In other words, we use the labels from the transfer data sets in the metric loss during training. We do this to obtain insights on the performance limits of the proposed method and the results are presented in Table REF  as metric-act (supervised) and metric-val (supervised).


Experiments for evaluation:
To investigate the efficacy of the proposed methods, in the following parts we investigate the quality of the latent embedding by using them as the input features for classification and correlation downstream tasks. Towards that, we evaluate the models in terms of
the correlation coefficient, and

the classification accuracy.

1. Correlation analysis
: We study the correlation between the latent dimensions and dimensional variables (activation, valence). With the main focus to study the effectiveness of the proposed continuous metric learning method using the dimensional variables, we evaluate how large a proportion of the labels can be explained by the latent dimensions. Therefore, as described in Sec. REF , we compute the correlation coefficients between \(\mathbf {z}\)  and \(l\) , via the adjusted \(R^2\) . The mean and standard deviation over 5-folds for \(R^2\)  for the models are shown in Table REF ; Note that the \(R^2\) , where the p-value of the F-statistic \(>0.05\) , are indicated by an asterisk. Models with larger \(R^2\)  are more effective in preserving the distance between samples in the embedding space with respect to the dimensional variables.


2. Classification of emotion classes
: We inspect the performance of the methods by classifying the speech samples into emotional categories using the support vector classifier (SVC) with a linear kernel. For evaluation, we use balanced accuracy for the 4-class (N-S-H-A) and 3-class (N-S-A) scenarios to account for imbalanced classes. Furthermore, supervised training on eGeMAPS using SVC and the SUPERB model are employed as references, as described above. The results are presented in Table REF .

Results and discussion
<TABLE><TABLE><TABLE>In Table. REF , we list current models similar to the DAE-unsupervised baseline in terms of the architecture, network size and input-output format. We observe that the performance of the developed unsupervised model is comparable to state of the art. Therefore, we consider the DAE+Linear-SVM model a reasonable baseline to gauge the performance of the proposed metric-loss in this paper.

Correlation analysis:
From Table REF , we observe that the mean and standard deviation of \(R^2\)  is consistently higher for the proposed methods relative to the unsupervised baseline, over the transfer data sets. However, DAE-unsupervised is as good as DAE-metric-act in terms of the correlation between \(\mathbf {z}\)  and the activation for EMO-DB. With supervision, as anticipated metric-act and metric-val models seem to have a higher \(R^2\)  relative to the unsupervised and semi-supervised models, specifically for the activation variable.
To summarize the observations, the proposed methods show higher adjusted \(R^2\)  between the modeled \(\mathbf {z}\)  and the label considered. However, we observe that \(R^2\)  corresponding to the valence variable is lower than the activation variable. An unaccounted non-linear relation between valence variable and the latent space could be a reason for the observation. Nevertheless, both metric-act and metric-val seem to effectively order and arrange data samples in the latent space, relative to the activation label. This is also evident from Fig. REF , wherein we can observe that the samples in the latent space are better distributed for metric-act and metric-val then for the unsupervised model.


Classification of emotion classes:
We observe that for both 3- and 4-class scenarios, the proposed methods have higher accuracy than the baseline unsupervised method. Furthermore, while SUPERB outperforms all the methods for IEMOCAP and SAVEE, it seems to have lower accuracy than the baseline and the proposed models for the remaining data sets, specifically for the 3-class classification task. Lastly, although classification accuracy of the supervised SVC is superior and implies that there is class-discriminating information in the data sets, how much of that information corresponds to the paralinguistic aspects of speech and emotion is worth investigating.
The balanced accuracy scores (Table REF ) for metric-val and metric-act is consistently better than the unsupervised baseline over the transfer datasets, indicating that incorporating a distance preservation metric in the loss function aids in shaping the distribution of features in the latent space that is more consistent over languages. It is also interesting to note that while metric-val did not show a large difference in \(R^2\)  with respect to the baseline (Table REF ), in terms of classification accuracy it is often better than metric-act. Furthermore, the proposed models are unable to effectively differentiate between anger and happy, whereby its performance drops in 4-class classification. A similar trend was observed for the reference SUPERB model (lower 3-class classification accuracy), despite its much higher complexity. This indicates that more complex models do not necessary lead to learning meaningful representations of the more subtle aspects of speech.


Effectiveness of metric loss function:
From Table REF  we observe that metric-act, i.e., the DAE trained with semi-supervision from the activation labels of the IEMOCAP dataset, shows the highest adjusted \(R^2\)  value between \(\mathbf {l_d}\)  and \(\mathbf {z_d}\) , \(\mathbf {l_d}\)  corresponding to the activation labels. This suggests that the proposed formulation of the continuous metric loss (Eq. REF ) is effective when activation labels are employed for semi-supervision. In contrast, for metric-val wherein \(\mathbf {l_d}\)  corresponds to valence, the \(R^2\)  value is similar to the baseline. A potential reason for this could be that the relation between valence and the embedding is inherently non-linear. This implies that different approaches are necessary to model valence and activation variables. Investigating and including aspects of the true relation between valence, activation and the embedding will be addressed in future work. In conclusion, we can state that using the dimensional variables, activation and valence, to learn emotion representations yield embeddings that are more transferable to unseen datasets and new languages. Furthermore, the proposed continuous metric loss with semi-supervision enables us to incorporate information on the dimensional variables within the model, hence aiding transferability.


Conclusion

In this work, we proposed a method for continuous metric learning, such that the difference between two feature points is continuous. We apply the method for speech emotion recognition, wherein we model a DAE that is semi-supervised using the activation and valence labels and the continuous metric loss. Our results show that the embedding from the proposed method is generally more consistent and thereby more transferable to different languages. The proposed methods are evaluated in terms of classification performance, and the proposed models outperform the baseline method on all the transfer datasets. Furthermore, to investigate the correspondence of the latent space to activation and valence variables, we compute the adjusted \(R^2\)  that indicates how much variation in the labels can be explained by a linear combination of the latent space. While the \(R^2\)  with respect to valence is generally lower than that for activation, between the methods, the proposed models outperform the baseline method for the datasets. Addressing the lack of labels corresponding to activation and valence variables in the transfer dataset, the annotated transfer dataset is the second contribution of this work.
