Domain Adapting Deep Reinforcement Learning for Real-world Speech Emotion Recognition

Computers can understand and then engage with people in an emotionally intelligent way thanks to speech-emotion recognition (SER). However, the performance of SER in cross-corpus and real-world live data feed scenarios can be significantly improved. The inability to adapt an existing model to a new domain is one of the shortcomings of SER methods. To address this challenge, researchers have developed domain adaptation techniques that transfer knowledge learnt by a model across the domain. Although existing domain adaptation techniques have improved performances across domains, they can be improved to adapt to a real-world live data feed situation where a model can self-tune while deployed. In this paper, we present a deep reinforcement learning-based strategy (RL-DA) for adapting a pre-trained model to a real-world live data feed setting while interacting with the environment and collecting continual feedback. RL-DA is evaluated on SER tasks, including cross-corpus and cross-language domain adaption schema. Evaluation results show that in a live data feed setting, RL-DA outperforms a baseline strategy by 11% and 14% in cross-corpus and cross-language scenarios, respectively.


INTRODUCTION
E Motion-aware interaction can significantly improve human-computer interaction. Consequently, there has been much research on automatic speech emotion recognition (SER). While SER within the same corpus is possible with reasonable accuracy, cross-corpus SER performance needs significant improvement [1], [2], [3]. Cross-corpus SER is crucial to achieving emotion-aware interactions in real-life applications, as speech signals obtained from different devices, recording background, spoken language, and acoustic signal conditions can be different with the training dataset and real-world implementation [1].
Researchers have explored domain adaptation techniques to improve the cross-corpus SER performance [4], [5], [6], [7]. Domain adaptation is a transfer learning approach where a trained model undergoes training aiming to optimise for a different distribution other than the previously trained distribution. The key limitation of the existing domain adaptation methods is that they cannot be adapted to the dynamic nature of the real-life setting. In such a setting, it will be beneficial if an intelligent agent interacting with a customer/user could dynamically update itself when it wrongly categorises speech emotion. In this paper, we address this challenge. We develop a domain adaptation technique based on reinforcement learning (RL), where we exploit the dynamically updating feature of RL to develop a cross-corpus SER technique that can be dynamically up- Rajapakshe  dated. Reinforcement Learning has been used to optimise realworld interactions with robots and machines after training in a simulated environment [8], [9], however, it has not been used in the field of SER. In a typical RL scenario, an agent takes actions in an environment, which is interpreted into a reward and a representation of the state, which are fed back into the agent. This does not trivially match a domain adaptation scenario of SER. To the best of our knowledge, we are the first to develop and evaluate techniques for using RL for this task.
A possible use case of the proposed technique is shown in Figure 1. A base deep learning model can be pre-trained with a labelled source dataset before starting Reinforcement Learning based Domain Adaptation (RL-DA). Then, the pretrained model is optimised to the target domain with RL. During this optimisation, the data feed can come from a live data source, where the model gets feedback from the customer regarding its interpretation of emotion in the customer's voice. Feedback can, for example, be implemented by a thumbs up or down button on the customer console. An example application demonstrating the above use case can be found online 1 . This application consumes recorded audio and infers the emotion from that audio utterance while users are able to provide feedback on the inferred emotion. This feedback is used for RL optimisation.
Using widely used speech corpora, we evaluate the model in cross-corpus and cross-language scenarios and show that we achieve better SER performance than fully supervised benchmark models. Moreover, we simulate a real-life (live) data feed scenario and show that our proposed RL model significantly outperforms fully supervised benchmark models.  [10], [11], [12], recommendation systems [13], and robotics [14]. However, the use of RL for speech-based applications is slowly getting traction [15].
In this section, we discuss the existing literature in two groups: one group combines literature on Domain Adaptation in SER, and the other group consolidates the previous work done with RL for domain adaptation.

Domain Adaptation
Domain adaptation was introduced to overcome the "corpus bias" issue. Since deep learning methods are powerful in extracting non-linear features in the input, domain adaptation can easily be implemented on deep learning-based platforms, yielding more robust and performing models [16], [17].
Mao et al. experimented on a two-class task for domain adaptation by sharing priors between source and target domains. They pre-trained a two-layer neural network with unsupervised learning and then shared the common classifier parameters between source and target [4]. Our approach follows the RL paradigm for domain adaptation instead of unsupervised learning, and we pre-train the source model with a labelled source dataset.
Gharib et al. proposed unsupervised adversarial domain adaptation and pre-trained the model with two conditional sets achieving nearly 10 % increase in accuracy [24].
Ahn et al. proposed a Few-shot learning methodology in unsupervised domain adaptation for cross-corpus SER tasks [23]. They used multiple corpora to optimise the emotion recognition robustness to unseen samples.
Abdelwahab and Busso studied the best adaptation technique for speech emotion recognition models out of supervised domain adaptation and online training. They also studied how the size of the labelled data affects the performance of the resulting model [25].
proposed using adversarial training techniques to improve cross-lingual domain adaption [29]. This is an unsupervised learning method that utilises two distinct auto-encoders for source and target, as well as a single discriminator, to determine whether an audio utterance is fake or real. Although we focus on cross-lingual domain adaptation, our main target is enabling domain adaptation in real-world applications.

Reinforcement Learning for Domain Adaptation
The RL agent gains knowledge via experiences, and these experiences are obtained by exploring and exploiting. The RL agent uses the policies to choose the best action given the present state -the method is called exploitation. Since the RL agent does not have enough experience to update the policy in the initial phase, it explores the environment. Exploration is done by randomly selecting actions without considering the policy, and the experience is stored in the memory buffer with a reward. The main challenge in this exploration is the unsafe choice of actions in some states [22].
Prior knowledge can be fed to the RL agent before releasing it to the real world in many ways. Demonstration based learning [30], [31], and pre-training and domain adaptation [22] are some of the methods to nourish the RL agent with knowledge. This reduces the amount of exploration time and the risk of unsafe exploration. Previous studies also have shown that pre-training has reduced the training time to achieve higher performance [32], [33], [34].
Koo et al. consider the feature extractor for the target domain as a generator where the parameters are similar to the latent feature set in the source and are trained with adversarial training loss. They proposed this to enhance the coherent training of the target domain feature extractor [21]. Though their research proposes an RL-based approach in domain adaptation, they have not studied speech emotion recognition.
Leonetti et al. have introduced a new RL algorithm, "DARLING", which can adapt to the new environment and improve the reliability of the decisions [18].
Hazara and Kyrki proposed a transfer approach that captures a simulated system's core features and rationalises the dynamics combined with incremental learning. They demonstrated their approach by applying it to a robot with a basketball task and showed the target model has improved task generalisation capability over direct usage [35]. In summary: 1) Literature on domain adaptation for SER is very rich, many approaches involve deep learning techniques. 2) Very few studies involve RL for speech emotion recognition tasks such as EmoRL [19]. But there is lack of research in the literature that uses RL for domain adaptation for SER. 3) Table 1 provides a brief overview of the literature on RL for domain adaptation and speech emotion classification, which attests that there has been a lack of literature that combines RL with DA for the problem of SER, which is the focus of this paper.  [4] Carr et al, 2018 [12] Lakomkin et al, 2018 [19] Hossain and Muhammad, 2019 [20] Koo et al, 2019 [21] Arndt et al, 2020 [22] Ahn et al, 2021 [23] This paper

Background
Reinforcement Learning follows the principle of behaviourist psychology and learns similar to a child who learns to perform a new task. There are two main components in an RL problem: the Environment and the Agent. The agent performs actions on an environment, and the environment returns a reward regarding the action performed along with the next state of the environment. Figure 2 shows the concise architecture of general RL methodology.
A Markov Decision Process can interpret an RL problem, which has the parameters state space S, action space A, and reward R. A state s ∈ S is the representation of the current environment, action a ∈ A is a single task that can be performed on the environment, and reward r ∈ R is the feedback value returned from the environment after executing the task a for the state s. RL agent learns a policy π(a, s) that represents a probability of taking action a for a specific state s. The Q-value or quality value represents the expected reward when action a is performed on an environment with state s. The optimum Q-value Q * (a, s) can be written as in Equation 1; Q-values are estimated using Q-tables in q-learning, while a deep neural network called Deep Q Network (DQN) is used to estimate the Q values in deep q-learning. The loss function in Equation 2 is used to calculate the loss and Q target can be as of Equation 3.
Combining the two equations 2 and 3, the loss function can be re-written as in Equation 4.
The loss L should be minimised in the training phase of RL.

Domain Adapting Reinforcement Learning for Speech Emotion Recognition
In this study, RL is used as an emotion recognising game where an RL agent attempts to recognise the correct emotion (action a) for a given audio utterance (state s). A reward r is returned by the environment. The reward value is obtained either by the feedback from the user in a real-world scenario or by comparing the ground truth and inferred emotion by the RL agent when training with labelled datasets. A policy π is learnt by the RL agent to maximise the reward gained at each episode. The state space S is defined as the distribution of speech audio, the Action space A is defined as the discrete emotion classes, and action selection is the inference of an emotion to a given audio utterance. According to Equation 4 to minimise L, Q(s t , a t ) should be maximised whereas the Reward R t+1 should be maximal at L * . Pre-training the DQN model prior to utilising an RL agent has proven to increase the performance of the model [33], [34]. Here, RL is used to optimise the pre-trained model to the target domain. The RL agent with a pre-trained DQN model interacts with the environment and tries to optimise the DQN to adapt to the target domain.

Baseline
Though RL methodologies do not report performance based on accuracy since there is no labeled data associated with RL experiments, we are using a supervised learning (SL) approach as the baseline to compare the performance in our methodology. We use a DNN architecture similar to the DNN used in the RL agent (Deep Q Network in Figure 2). We first train the model with the Source Dataset to be aligned with the pre-training in the RL approach and obtain a Base Model as in Figure 1. Then, the Base Model is trained with the Target dataset to match with the RL-DA. This allows us to compare the two approaches RL-DA and Supervised Learning-based Domain Adaptation, with similar parameters. We use four labelled datasets, and section 4.1 describes the datasets used in this study. A fraction of 20 % of each dataset is reserved as a testing dataset, and the remaining subsection is used for either pretraining or RL domain optimisation. Section 4 describes the different kinds of experiments carried out within this study to see the effectiveness of our approach.

Datasets
To measure the performance of the proposed RL based Domain Adaptation (RL-DA) approach, we use four widely used publicly available datasets for our experiments: MSP-IMPROV [36], IEMOCAP [37], ESD [38] and EmoDB [39]. The selection of datasets covers the ability to use them for Cross-Corpus (CC) experiments as well as Cross-Language (CL) experiments. Since the datasets do not contain a balanced number of utterances under each emotion class, we use an audio augmentation technique to generate audio utterances. We use Vocal Tract Length Perturbation (VLTP) [40] to augment audio utterances balancing utterances under each emotion class when preparing the testing subsets. All the four speech emotion datasets we use contain categorical emotional labels, and we use four emotions widely used in the literature: 'happiness', 'sadness', 'anger', and 'neutral'.

IEMOCAP (IEM)
IEM is a popular dataset containing 12 hours of multispeaker acted audio-visual data. The IEMOCAP dataset contains dyadic sessions with both improvised and scripted scenarios. It contains categorical emotion labels happiness, sadness, anger, and neutrality as well as dimensional labels valence, dominance, and activation [37]. This study uses the audio data modality from improvised scenarios and categorical labels.

MSP-IMPROV (MSP)
MSP is an acted audio-visual database that is popular in multi-modal speech emotion recognition research. The database was initially developed for a study in audiovisual emotional perception, but this dataset has also been used in speech emotion recognition studies [23], [41], [42], [43] which makes it a candidate for our study as well. The dataset was recorded in a controlled environment and consists of 20 determined scripts covering the primary four emotions: happiness, sadness, anger, and neutral [36].

Emotional Speech Dataset (ESD)
ESD is a public database of speech data that was initially developed for speech synthesis and voice conversion. It contains utterances from 20 speakers in two languages (10 Mandarin and 10 English) categorically labelled into five emotion classes: happy, surprise, neutral, angry, and sad [38]. We use utterances from 10 English speakers under the four emotions happiness, sadness, anger, and neutral.

Berlin EmoDB (EmoDB)
EmoDB is an actor recorded database of emotional speech utterances in the German language. 10 actors (5 male and 5 female) in the age range between 21 years and 35 years speak 10 scripted texts in 7 different emotions such as anger, boredom, disgust, fear, happiness, sadness, and neutral [39]. We use EmoDB as the target dataset for cross-language experiments. We choose the utterances of the four emotion classes: anger, happiness, sadness, and neutral.

Diverse Environments Multichannel Acoustic Noise Database (DEMAND)
The DEMAND dataset is a popular real-world background noise dataset. The database contains background noises from 18 different environments under six categories. An array of 16 microphones is used to record the audio in each environment stored in 16 channels. The DEMAND dataset contains versions of both 16 kHz and 48 kHz sampling rates, and we use 48 kHz and a down-sampled to 22 kHz version to match with the other datasets used in this study.
A summary of the datasets used in this study is shown in Table 2

Input feature
We use Mel Frequency Cepstral Coefficients (MFCC) as the input features [44], [45]. We use 2, 048 as the frame length and the hop length as 512 and extract 40 MFCCs using the Librosa python library for audio and music analysis [46].

Model configuration
The functionality of the DNN model is to classify the emotion embedded in speech utterances, and the study uses MFCCs of speech utterances as the input for the model. We use a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) combined model since it motivates the capability to learn both frequency and temporal components in the speech signal [34], [47]. We stack CNN, LSTM, and fully connected layers, respectively, for the model to learn the discriminative features [45]. As shown in Figure 2, the feature matrix is fed into two layers of 2D convolution layers of filter sizes 5 and 3, and the output of the first 2D convolution layer is batch normalised. Hyperparameter batch size is set to 128. The output from the second 2D convolution layer is passed to the LSTM layer of 16 cells after flattening and then to a fully connected layer of 256 units. A dropout rate of 0.3 is applied before the last Dense output layer. The number of units in the output layer is four, equal to the number of emotions to be classified. The linear activation function of the output layer is used when the model is used in the RL agent since it outputs Q-values for a specific state, and they should not be normalised. The input shape of the model is 40 × 87, where 40 MFCCs are used in the input and 87 is the number of MFCC frames.
The model is optimised using the Adam Optimiser with a learning rate of 2.5 × 10 −4 . We use the popular Deep Learning API Keras [48] with Tensorflow [49] (version 2.1.0) as the back-end for modelling and training purposes in this study.

Experiments
We design and run several experiments to evaluate the proposed RL based domain adaptation approach. For evaluation, we develop three scenarios. (1) Pre-training with a source dataset and domain optimisation with a target dataset separately; (2) Pre-training with a subset of the source dataset and domain optimisation with a mix of the remaining source and target datasets; (3) Domain adaptation in a recreated real-world scenario. Details of each scenario and results are explained in Sections 5.1, 5.2, and 5.3 respectively.
The Unweighted Average Recall rate (UAR %) is one of the popular metrics used by researchers to measure the performance of speech based machine learning models. The recall is calculated for each label of the classification problem, and the UAR is calculated by taking the unweighted average of the recall values of each label The Python code repository of the experiments and data preparation is hosted in public GitHub repository "RL-DomainAdaptation" 2

EVALUATIONS
This section reports the results and description obtained through the experiments in section 4. We use the IEMO-CAP and MSP-Improv datasets as source datasets, while 2. https://github.com/jayaneetha/RL-DomainAdaptation the ESD dataset is used as the target dataset in the crosscorpus schema, and EmoDB is used as the target dataset in the cross-language schema. We use Max Boltzman Policy (Max.B) [50] as the RL policy in the RL agent.

Experiment with separate datasets for source and target
Here, we evaluate the proposed RL-DA approach for separate source and target datasets. We do this for both the crosscorpus and cross-language schemas. Table 3 and Figure 3 (a) show the datasets used for each schema. First, the model is pre-trained using the training subset of the source dataset, obtaining the Base model. Then, the pre-trained base model parameters are transferred onto the DQN of the RL agent. Afterwards, the RL deep Q learning algorithm is executed to optimise the DQN model applying the target dataset in the environment which outputs the domain optimised DNN model. To measure the performance of the domain optimised model, we use a Supervised Learning approach. We take the testing subset of the Target dataset and infer the emotion of each testing utterance and record the inference emotion with the labelled emotion for the utterance. The UAR value for each experiment is calculated using the recorded data about inference emotions and ground truth.
Supervised Learning approach based performance is also measured to compare the performance of our methodology as the baseline (SL-DA -sep). We create a DNN model that is similar in architecture as used in DQN (the Deep Q Network in Figure 2) and is pre-trained by the Source dataset. Then, the same model is trained by the training subset of the Target dataset. Subsequently, the resulting model is tested with the testing subset of the Target dataset to obtain the testing accuracy.
The UAR of the baseline (SL-DA -sep) model after training with supervised learning and the UAR of the model after training with the RL-DA approach (RL-DA) are reported in Table 4. We find that the accuracy of RL-DA surpasses the baseline UAR by 0.45 %, 2.92 %, and 8.83 %, respectively, except for in the IEM-EmoDB combination. Additionally, while comparing the SL-DA method and the RL-DA method, the cross-language schema's standard deviation has decreased from 8.26 to 0.71, respectively, demonstrating that the model performances are more consistent.
This outcome is noteworthy because, unlike SL approaches, which feed labelled data to train the model, the RL methods only provide feedback rewards that show the accuracy of inference made during the RL optimisation phase. So, achieving on par SL accuracy also means that  We calculate UAR to compare the performance of our models. We use the model architecture, source, and the target dataset as the parameters of similarity in the literature on cross-corpus speech emotion recognition to compare our results. The results are represented in Table 5.

Experiment mixing the source and target datasets
This experiment is carried out to evaluate our proposed methodology while including a subset of the source dataset in the RL optimisation phase. The aim of such mixing is to preserve generalisable features of the source dataset. This experiment is performed similar to the previous experiment mentioned in 5.1, but with the following modifications: Only a subset of the training set of the source dataset is used in pre-training, and a mix of the remaining training set of the source dataset and the training set of the target dataset is used in the RL optimisation. Table 6 and Figure 3 (b) show how the datasets are used in this experiment. We measure the performance by using SL as a baseline (SL-DA -mix) to compare with our proposed approach. Results of the experiment by mixing source data into the target dataset as mentioned above are presented in Table 7. The accuracy of the RL-DA approach is compared with the Supervised Learning approach -SL-DA -mix. The RL-DA approach outperforms SL-DA -mix at least by 8 % in each experiment. Since the target dataset includes components from the source dataset, the representations of the model learnt during the pre-training stage are kept while the model adapts to the new domain.
We compare the performance of RL-DA in two scenarios where the target dataset is separated (S) with the source dataset (Section 5.1), and the target dataset is mixed (M) with utterances from the source dataset (Section 5.2) and visualised in Figure 4.
From the above visualisation, we note that the mixing of source and target datasets yields better performance than experiments with separate datasets except for the experiment for MSP-ESD combination. The mixing of datasets increases the diversity of the training utterances which leads the model to learn generalised feature representations when training.  8 Testing Accuracy of SL-DA -rw and the proposed RL-DA approach of each experiment mimicking a real-world scenario for each cross-corpus (CC) and cross-language (CL) schema with (w/) and without (w/o) background noise. Background Noise is added to simulate real-world conditions.

Experiments simulating a live data feed scenario
We experiment to quantify the performance difference between SL and RL optimisation in a simulated live data feed scenario. Testing in a real live data feed scenario would require integrating the system with a call centre or similar setting, which is problematic. However, we closely simulate the live scenario by continually pushing the data from the target dataset. RL-DA uses this data to continually update itself, whereas the SL-DA method uses the static model, which is trained using the source dataset. The methodology of the RL optimisation is similar to the experiment described in 5.1. The baseline model (SL-DA -live) is first trained with a training subset of the source dataset, and then, the testing accuracy is measured with testing subsets of the target dataset. We also measure the performance of RL-DA with realworld audio. We mix background audio with our target datasets to create a new variation of the dataset with background sound. Audio from the kitchen environment of the DEMAND dataset is used as the background sound to mix with target datasets (ESD and EmoDB). We use -5 db as the Signal-to-Noise ratio when mixing the background audio to the speech utterances.
The testing accuracy of the models trained in three different ways (SL-DA -live, RL-DA -target dataset without background noise (RL-DA w/o noise), and RL-DA -target dataset with background noise (RL-DA w/ noise)) in a realworld scenario mentioned in 5.3 are tabulated in Table 8.
The ∆ column shows the difference in accuracy between our approach and the baseline, where one has to use a model pre-trained only in the source dataset.
Observing the ∆ column in Table 8, it is clear that RLbased domain adaptation (RL-DA) outperforms the baseline SL strategy in the live data feed situation (SL-DA -live). Supervised learning methodologies do not have the possibility to adapt to the deviation of a domain distribution without a labelled dataset to retrain during inference time. On the other hand, RL based learning methodologies constantly receive feedback from the environment and optimise the agent model for the domain deviations on the go. One can argue that for the supervised method, one can retrain the model. But this requires manual intervention, whereas applying RL, this can be reached without any manual intervention. Therefore, we suggest that RL is more suitable in domain adaptation to be used in real-world scenarios because of the adaptability during the optimisation phase.  Figure 5 shows the comparison between the UAR of the Baseline, RL-DA without noise, and RL-DA with noise after executing the 'simulated live data feed' experiments. We observe that both RL-DA experiments (with and without background audio) show better results than that of the baseline approach.

CONCLUSION
This study investigated the possibility of using RL for domain adaptation in SER. While several strategies have been used for domain adaptation for SER in the past, we were the first to propose RL-technique for domain adaptation herein. Evaluations of this research suggest that using RL can improve domain adaptability in a real-world scenario compared to a SL approach. An accuracy increment of 11 % in cross-corpus and 14 % in cross-language schema was observable in the results when using the proposed RL-DA approach compared to the baseline strategy.
We discussed that the main reason for the improved domain adaptability in a real-world setting is the ability of the RL agent to continually incorporate feedback. On the other hand, supervised learning does not have such a feedback mechanism unless the model is retrained using a manual process.
The outcome of this study is likely to lead the community in this new direction and help eventuate real-world emotion-aware applications that can adapt to the users' emotional expression under a new domain. Sara Khalifa is currently a senior research scientist at the Distributed Sensing Systems research group, Data61-CSIRO. She is also an honorary adjunct lecturer at the University of Queensland and conjoint lecturer at the University of New South Wales. Her research interests rotate around the broad aspects of mobile and ubiquitous computing, mobile sensing, and Internet of Things