A Deep Neural Framework for Contextual Affect Detection

A short and simple text carrying no emotion can represent some strong emotions when reading along with its context, i.e., the same sentence can express extreme anger as well as happiness depending on its context. In this paper, we propose a Contextual Affect Detection (CAD) framework which learns the inter-dependence of words in a sentence, and at the same time the inter-dependence of sentences in a dialogue. Our proposed CAD framework is based on a Gated Recurrent Unit (GRU), which is further assisted by contextual word embeddings and other diverse hand-crafted feature sets. Evaluation and analysis suggest that our model outperforms the state-of-the-art methods by 5.49% and 9.14% on Friends and EmotionPush dataset, respectively.


Introduction
It becomes quite natural for us to gauge the emotion of a person if they explicitly mention that they are angry, sad or excited or even if they use the corresponding emojis at the end of the sentence, but what if there happens a drift in emotion while having a series of conversation between two people? And what if they stop using emotional emojis after a certain point of time even though they continue to be in the same state of emotion. Even human annotators may be confused if they do not consider context. Given that, even face-to-face conversation is confusing, sometimes, it should not be a matter of surprise if there could be a misinterpretation in textual conversations. The situation can get worse if there is a multi-party 1 conversation. In this scenario, emotion of one speaker can change due to the utterance of the second speaker, and can again be switched due to the intervening of the third speaker. We have to be attentive to every speaker in the conversation or else our context would be lost.
Although a significant amount of research has been carried out for emotion analysis, only in the recent time there is a trend for performing emotion analysis Oh no-no-no, give me some specifics. Non-Neutral Chandler I love the specifics, the specifics were the best part! Non-Neutral Joey Hey, what about the scene with the kangaroo? Did-did you like that part? Neutral Chandler I was surprised to see a kangaroo in a World War I epic.
Non-Neutral Joey You fell asleep!! Anger Joey There was no kangaroo! Anger of the dialogues in order to build an effective and human-like conversational agent.
Our current work focuses on detecting emotions in a textual dialogue system. We aim at labeling each utterance of a dialogue with one of the eight emotions, which comprises of Ekman's [4] six basic emotion tags, i.e., anger, fear, sadness, happiness, disgust and surprise plus neutral and non-Neutral 2 emotion. An example instance is depicted in Table 1. If we look at the last utterance in the table i.e., There was no kangaroo!, it can be considered as neutral but while we consider its previous context, it should be assigned with the anger class.
Our deep neural network framework follows a stacking structure which utilizes bidirectional gated recurrent unit (Bi-GRU)arranged in an hierarchical form. The lower Bi-GRU produces utterance level embeddings, and the upperlevel Bi-GRU makes use of these embeddings to capture the contextual information in the dialogue. Some handcrafted features are incorporated at the different levels of the model to capture more linguistic evidences, which eventually found to be effective compared to the other models. We evaluate our proposed system on the benchmark dataset of EmotionLines 2018 [1]. We observe that our proposed framework attains better performance compared to the state-of-art model.

Related Work
Emotion detection and Sentiment classification have always been a hot research topic in the area of Natural Language Processing (NLP). Existing research on emotion detection have mostly focused on textual contents. In recent times, deep neural nets are being used very extensively to perform emotion analysis in a variety of domains, mediums and languages. Some of the most widely used models for capturing emotions include Convolutional Neural Network (CNN) [11] and Recurrent Neural Networks (RNN) [14] like Long-Short Term Memory (LSTM) [12] and Gated Recurrent Unit (GRU) [2]. All these works focus on to classifying emotions at the sentence level or utterance level, and thus cannot capture the context and inter-dependence among the utterances in dialogue. [17] proposed a network which made use of bidirectional long contextual short-term memory to detect the emotion of an utterance in dialogue and named it as bcLSTM. Later on, [6] improved bcLSTM by introducing a memory network which makes use of speaker information as well for context modeling. The authors in [13] used Bi-LSTM to capture word dependencies and to extract relevant features for detecting various emotions. On top of that, they applied selfattention [21] to capture the inter-dependencies between the utterances of a dialogue. The work reported in [18] uses hierarchical attention network model [22] to embed contextual information among the utterances in a dialogue. [8] used a bidirectional gated recurrent unit (Bi-GRU) [3] fused with self-attention [21] and its word embeddings to efficiently utilize word-level as well as utterance-level information.
Our proposed model differs from the existing models in the sense that we derive deep contextualized representation of each word in an utterance, and then incorporate it into the model as a feature along with the pre-trained Glove word embedding [15]. We acquire these word embedding from the pre-trained ELMo [16] model. These representations take the entire context into account. Being character based, they allow the network to use morphological cues to form robust representations for out-of-vocabulary words unseen in training. We use hierarchical Bi-GRU to learn the context in a dialogue fused with various handcrafted features obtained through transfer learning over similar tasks.

Proposed Methodology
In this section, we describe our proposed framework in details. Unlike the previous models, which performed exceptionally well in predicting some emotions follows complex architecture.. Evaluation shows that a model which performs very well for a specific emotion class compensates with the lower performance in other emotion classes (i.e. performs at the cost of other classes). In contrast, our model is straightforward and efficient, and at the same time outperforms every different models with significant margins. It consists of two layers of hierarchythe lower layer is for encoding the utterances (named as utterance-encoder ), and the upper layer is for encoding the dialogues (named as dialogue-encoder ).
Given a batch/dialogue 3 , firstly the individual utterances are passed through utterance-encoder which comprises of a recurrent layer (biGRU) followed by a max-pooling layer. GRU learns the contextual representation of each word in an utterance, i.e., the representation of each word is learned based on the sequence of words in the utterance. Subsequently, we apply max-pooling over the hidden representation of each utterance to capture the most important features over time. The obtained utterance representations form the input to the dialogueencoder, which again comprises of biGRU to capture the contextual information of each utterance in a dialogue. Since the task is to classify each utterance of a dialogue, the hidden representations of biGRU over the time are passed through fully connected layer followed by Softmax to obtain the corresponding emotion label. Further, the inputs to utterance encoder and dialogue encoder are assisted by a diverse set of hand-crafted features (c.f. Section 3.1) for the final prediction. Figure 1 depicts our proposed architecture.

Hand-crafted Features
We make use of transfer learning to capture important evidences obtained from the various state-of-the-art pre-trained deep learning models. Following subsections explain these models in details: -DeepMoji [5]: DeepMoji performs distant supervision on a very large dataset [20] (1.2 billion tweets) comprising of noisy labels (emojis). By incorporating transfer learning on various downstream tasks, they were able to outperform the state-of-the-art results on 8 benchmark datasets on 3 NLP tasks across the five domains. DeepMoji can give an excellent representation of text which can be incorporated in any sentiment or emotion detection model to improve the performance. Since our target task is closely related to this, we adopt this for our domain and extract the embeddings of 2304 dimension from the attention layer, which acts as the utterance embedding feature for upper-level (dialogue-encoder) of the model. -ELMo [16]: Unlike traditional word embedding techniques, ELMo makes use of bidirectional LSTM network to create word representations. This biLSTM is trained with a coupled language model objective on a large text corpus. Thus we can say, each word representation is a function of the entire sentence. It analyses the words within the context that they are used, thus capturing the syntactic as well as semantic characteristics of the words and also take care of variance across the linguistic contexts. Also, being character based, we can have representations of out-of-vocabulary words as well. Since it is proven that the addition of ELMo representations can improve the performance of any NLP model, we incorporate it into our model along with the pre-trained Glove word embedding. To reduce the processing time, we extracted the 1024 dimensional ELMo word embeddings beforehand instead of creating it during the process.

Word Embedding
Embedding matrix is generated from the pre-processed text using a combination of pre-trained and deep contextualized word embedding: 1. Pre-trained GloVe embeddings for tweets [15]: We use 300-dimensional pre-trained GloVe word embeddings, trained on the Twitter corpus, for the experiments. The glove is a count-based model that captures the count of the word, i.e., how frequently it appears in a context. 2. Deep contextualized ELMo embeddings [16]: We extract 2304 dimensional word vectors from ELMo embedding model, which learns the embedding from an internal state of biLSTM and represents word-level contextual features of the input text.
We finally concatenate the word representation of both the embeddings, which act as input to the utterance-encoder (lower level).

Dataset Description
For experiments, we use the benchmark dataset of EmotionLines 2018 [1], which is an emotion annotated corpus of multi-party conversation. The dataset comprises of two individual corpus of dialogue set extracted from two different sources, one from the famous TV show scripts named Friends, and the other from human to human chat logs on Facebook messenger through an application called EmotionPush.
1. Friends TV series data : The Friends script was crawled, and each scene of an episode was treated as a dialogue. Thus each dialogue consists of multiple speakers. [1] separated the dialogues based on its window size 4 of [5,9], [10,14], [15,19], and [20,24]. Finally, they randomly collected 250 dialogues from each-thus creating a corpus of 1000 dialogues, which are further splitted up into 720, 80, and 200 dialogues for training, validation and testing, respectively. 2. EmotionPush Chat log data: This data was collected by crawling the private conversation among the friends on facebook messenger with the help of EmotionPush app. To protect the private information of the users like names, organizations, locations, email address, and phone numbers, they used a two-step masking procedure. They treated the conversations lasting not more than 30 minutes as a dialogue. Finally, they make use of the same procedure for sampling and categorizing as they used for Friends TV script and collected 1000 dialogues which were again divided in the same ratio for training, validation, and testing. Table 2 shows the distribution of both Friends and EmotionPush datasets in terms of the number of dialogues, the number of utterances, and the number of emotion labels. To compare our model we follow the setup of [7] and evaluate its performance only on four emotions, i.e., anger, joy, sadness and neutral on both Friends and EmotoinPush datasets, and excluding all the other emotion classes during training of our model. We ignore other emotion classes by setting their corresponding loss weights to zero. Fig 2 depicts the distribution of emotion classes into train, validation, and test set for both Friends and EmotionPush datasets.

Pre-processing
Friends data consists of scene snippets containing multi-speaker conversation, while EmotionPush data includes Facebook messenger chats between two individuals. Both datasets contain some incomplete sentences and excessive use of punctuations. In addition to it, EmotionPush data also contains emoticons which are absent in the Friends data. We believe that the reason behind it is that Friends data is the script which is collected by converting audio to text. We perform the following pre-processing steps:  remove extra space, emoticons and the newline character. Finally, we perform word tokenization to construct the vocabulary of words and symbols. The tokens, thus, collected are mapped to their corresponding 300-dimensional Glove vectors.

Experiments
We pad each utterance to a maximum length of 50 words. The GRU dimension is set to 300 for both lower and upper level. We employ 300 plus 1024 dimensional word embeddings and 300 plus 2304 dimensional sentence embeddings for the experiments. We use Tanh activation and set the Dropout [19] as 0.5 in order to prevent the model from overfitting. We optimize our model using Adam optimizer along with weighted categorical cross-entropy loss functions for emotion classification. From Table 2, it is clear that EmotionLines data suffers from class imbalance issue. We follow [9] to prevent our model from getting biased towards more frequently occurring emotion classes-thereby providing larger weights to the losses corresponding to less frequently occurring classes and vice-versa.
We conduct our experiments on the Pytorch framework 5 . We adopt the official evaluation metric of EmotionX 2018 [7] shared task, i.e. weighted accuracy (WA) (eq.1) and un-weighted accuracy (UWA) (eq.2), for measuring the performance of our model. We train our model for the maximum 50 epochs with early stopping criteria on validation accuracy, having the patience of 10. We initialize the learning rate by 0.00025 with the decaying factor of 0.5 on every 15 epochs.
where a c refers to the accuracy of emotion class c and p c refers to the percentage of emotion class c in the test set.  Table 3 shows the evaluation results of the top-performing existing models so far compared with that of our proposed framework. In the table, HiGRU and HiGRU-sf are proposed by [8]. Results show that our model outperforms all the other models with a significant margin. Further, we observe the improvement to be statistically significant with 95% and 99% confidence on Friends and Emo-tionPush dataset, i.e., the p-value is less than 0.05 for paired T-test [10] of both the datasets. On Friends dataset, our model reports un-weighted and weighted accuracy of 74.39% and 75.94% as compared to that of 68.9% and 74.4% of the state-of-art models, thus improving them by 5.49% and 1.54%. On the other hand, our CAD framework performs better on EmotionPush dataset as well, reporting the unweighted and weighted accuracies of 80.18% and 86.24%, respectively, as compared to that of 70.2% and 77.1% in the state-of-art model, thus giving an increment of 9.14% and 9.98%. It is worth noticing that the accuracy of anger in EmotionPush dataset increases by 17.38 points without compensating with the other classes. This implies that for all the emotion classes our framework performs reasonably competent with the accuracies being balanced and not biased to any specific class.
To test the correctness of our prediction, we adopt some examples where the emotion drift occurs over an extended context. It can be seen from the Table  4 that rare emotions are predicted correctly. Thus we can say that our model captures the contextual information in the dialogue pretty well.

Error Analysis
In Figure 3, we show the confusion matrices for both the datasets (i.e., Friends and EmotionPush) for quantitative analysis. We find that joy and sad emotions are mostly confused with neutral, possibly due to the absence of affective words and large number of neutral class. Most of the neutral emotions get confused with joy due to the presence of some positive sentiment words. Majority of anger emotion in EmotionPush dataset are missclassified as neutral (mostly due to absence of any sentiment bearing words) while that of Friends dataset are misclassified as joy due to presence of exclaimation mark which shows strong emotion such as joy or anger. The presence of positive sentiment enforce it to predict joy.  We also perform a qualitative analysis to get better insights about the strength and weakness of the system. We found that, apart from dominating the effect of higher distribution class, few other error cases are present such as: -Loss of emotion drift: When a strong emotion such as anger is encountered, it continues to be predicted. The reason behind it might be because we did not consider the speaker information while extracting contextual information from the utterances. An example is shown in the first part of Table  5. -Usage of expression like lol, haha, etc.: Conversations involving such expressions have led to misclassifications of prediction by the model for some of the instances. An example is given in the second part of Table 5. The sentences are originally labeled Sad. However, due to the presence of a smiling expression-related term (haha and lol ), the model has predicted it as Joy.

Conclusion
In this paper, we have presented a hybrid deep neural network framework (CAD) for detecting emotions in a dialogue. We propose a hierarchical BiGRU network which takes the assistance of various hand-crafted feature set on different levels of architecture to learn the contextual information of dialogue. The learned representation is fed to a fully connected layer over the time steps followed by softmax layer for the predictions. We have evaluated our model on the benchmark datasets of EmotionLines-2018, which consists of two corpora i.e., Friends and EmotionPush data. The evaluation suggests that our CAD framework obtains improved results against the state-of-art model so far. This model can be applied to various other similar datasets as well to improve their results.

Acknowledgement
Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya Ph.D. scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).