---
abstract: 'Emotions widely affect human decision-making. This fact is taken into account by affective computing with the goal of tailoring decision support to the emotional states of individuals. However, the accurate recognition of emotions within narrative documents presents a challenging undertaking due to the complexity and ambiguity of language. Performance improvements can be achieved through deep learning; yet, as demonstrated in this paper, the specific nature of this task requires the customization of recurrent neural networks with regard to bidirectional processing, dropout layers as a means of regularization, and weighted loss functions. In addition, we propose *sent2affect*, a tailored form of transfer learning for affective computing: here the network is pre-trained for a different task (i.e. sentiment analysis), while the output layer is subsequently tuned to the task of emotion recognition. The resulting performance is evaluated in a holistic setting across 6 benchmark datasets, where we find that both recurrent neural networks and transfer learning consistently outperform traditional machine learning. Altogether, the findings have considerable implications for the use of affective computing.'
address:
- 'ETH Zurich, Weinbergstr. 56/58, 8092 Zurich, Switzerland'
- 'National Institute of Informatics, 2-1-2 Hitotsubashi, 101-8430 Tokyo, Japan'
author:
- Bernhard Kratzwald
- Suzana Ilić
- Mathias Kraus
- Stefan Feuerriegel
- Helmut Prendinger
title: 'Deep learning for affective computing: text-based emotion recognition in decision support'
---

Affective computing ,Emotion recognition ,Deep learning ,Natural language processing ,Text mining ,Transfer learning

Introduction
============

Emotions drive the ubiquitous decision-making of humans in their everyday lives [@Oatley2011; @Greene2002; @Schwarz2000]. Furthermore, emotional states can implicitly affect human communication, attention, and the personal ability to memorize information [@Derakshan2010; @Dolan2002]. While the recognition and interpretation of emotional states often comes naturally to humans, these tasks pose severe challenges to computational routines [[e.g.]{}, @Poria2017; @Tausczik2010a]. As such, the term *affective computing* refers to techniques for detecting, recognizing, and predicting human emotions ([e.g.]{}, joy, anger, sadness, trust, surprise, anticipation) with the goal of adapting computational systems to these states [@Picard1997]. The resulting computer systems are not only capable of exhibit empathy [@Picard1995] but can also provide decision support tailored to the emotional state of individuals.

Emotional information is conveyed through a multiplicity of physical and physiological characteristics. Examples of such indicators include vital signs such as heart rate, muscle activity or sweat production on the surface of the skin [[e.g.]{}, @Lux2015; @Tao2005]. A different stream of research tries to infer emotions from the content and its mode of communication. These approaches to affective computing are primarily categorized by the modality of the message, [i.e.]{}, whether it takes the form of speech, gesture, or written information [@Calvo2010]. In this terminology, affective computing can comprise both unimodal and multimodal analyses. For instance, videos allow for the recognition of facial expressions and vocal tone [@Chen2017; @ElAyadi2011; @Shan2009].

The focus of this work is on the unimodal analysis of written materials in English. This choice reflects the prominence of textual materials as a widespread basis for decision-making [@Hogenboom2016]. Illustrative examples are as follows (a detailed review is given later in ). For instance, the use of affective language as a proxy for emotional closeness can be used to measure the strength of interpersonal ties in social networks [@Marsden2012]. Similarly, marketing utilizes the recognition of emotional states in order to predict the purchase intentions of customers [@Ang2000], satisfaction with services [@Greaves2013], and even to measure the overall brand reputation [@Al-Hajjar2015]. In a related context, decision support can leverage affective signals in financial materials in order to suggest trading decisions [@Gilbert2010] or forecast the economic climate [@Nyman2015]. Furthermore, affect can also improve processes and decision-making in the provision of healthcare [@Spiro.2016] or education [@Rodriguez2012].

Previous research on affective computing has merely utilized methods from traditional machine learning, while recent advances from the field of deep learning – namely, recurrent neural networks and transfer learning – have been widely overlooked. However, their use promises further improvements. In fact, techniques from deep learning have become prominent in various decision support activities involving sequential data [[e.g.]{}, @Evermann2017] and especially linguistic materials [[e.g.]{}, @Kraus2017; @Mahmoudi.2018], where deep learning was able to enhance the performance when deriving decisions from unstructured data. One of the inherent advantages of deep learning is that it can successfully model highly non-linear relationships.

This work draws upon existing solution techniques from the realm of deep learning [@Kraus2017] and applies them to a problem domain different from that of our research objective. First and foremost, we extend existing techniques from the discipline of deep learning to the task of text-based emotion recognition in order to expand the body of knowledge. Following @Kraus2017, we also utilize long short-term memory networks (LSTMs) that can make predictions based on running texts of varying lengths. However, affective computing differs substantially from related tasks due to the high number of often imbalanced target labels. Thus, this task requires both customized network architectures and procedures. Hence, its applicability is only made possible through the several methodological innovations that we summarize in the following.

In order to handle class imbalances in affective computing, we propose the following modifications beyond @Kraus2017: (i) bidirectional processing of the text, (ii) dropout layers as a means of regularization, and (iii) a weighted loss function. The latter becomes especially critical due to the imbalanced distribution of labels. In fact, without the weighted loss function, the network ends up resembling merely a majority class vote.

We further propose an extension of transfer learning called *sent2affect*. That is, the network is first trained on the basis of sentiment analysis and, after exchanging the output layer, is then tuned to the task of emotion recognition. To the best of our knowledge, this presents a novel strategy for better affective computing as the inductive knowledge transfer is not merely based on a different *dataset*, but a different *task*.

Even though affective computing has gained great traction over the past several years [@Ribeiro2016], there is a scarcity of widely-accepted datasets for text-based emotion recognition that can be used for benchmarking and that facilitate fair comparisons. A relatively small, but more common, dataset was provided by SemEval-2007 and consists of annotated news headlines [@Strapparava2007]. A significantly larger, but underutilized, corpus is composed of affect-labeled literary tales [@Alm2008]. Our literature review notes considerable differences across datasets that vary in their linguistic style, domain, affective dimensions, and the structure of the outcome variable. With regard to the latter, the majority of datasets involve a classification task in which exactly one affective category is assigned to a document, while others request a numerical score across multiple dimensions, [i.e.]{}, a regression task. Hence, it is a by-product of this research to contribute a holistic comparison that benchmarks different methods across datasets used in prior research. For this purpose, we conducted an extensive search for affect-labeled datasets that serves as the foundation for our computational experiments. As a result, we find that deep learning consistently outperforms the baselines from traditional machine learning. In fact, we observe performance improvements of up to in F1-score as part of classification tasks and in mean squared error as part of regression tasks.

The findings of this work have direct implications for management, practice, and research. As such, various application areas of decision support – such as customer support, marketing, or recommender systems – can be improved considerably through the use of affective computing. Similarly, all systems with human-computer interactions ([e.g.]{}chatbots and personal assistants) could further benefit from emotion recognition and a deeper understanding of empathy. In fact, emotion detection could significantly impact and refine all use cases in which sentiment analysis ([i.e.]{}, only positive/negative polarity) has already proved to be a valuable approach, since these lend themselves to a more fine-grained analysis and decision-making beyond only one dimension. In academia, text-based emotion recognition supports the cognitive and social sciences as a new approach to measuring and interpreting individual and collective emotional states.

The rest of this paper is structured as follows. reviews earlier works on text-based emotion recognition, including the underlying affect theories, datasets used for benchmarking, and computational approaches. This reveals a research gap with regard to both deep neural networks and transfer learning within the field of affective computing. As a remedy, introduces our methods rooted in deep learning, which are then evaluated in . Based on our findings, we detail implications for both research and management in , while concludes.

Background {#sec:background}
==========

We specifically point out that the terms “sentiment analysis” and “affective computing” are often used interchangeably [@Munezero2014]. However, comprehensive surveys [@Pang2006; @Yadollahi2017] recognize clear differences that distinguish each concept: sentiment analysis measures the subjective polarity towards entities in terms of only two dimensions, namely, positivity and negativity. Conversely, affective computing concerns the identification of explicit emotional states and, hence, this approach is also referred to as emotion recognition. The choice of emotional dimensions depends on the underlying affect theory and involves a wide range of mental states such as happiness, anger, sadness, or fear. For reasons of clarity, we strictly distinguish between the aforementioned concepts in our terminology.

Accordingly, this section first provides an overview of prevalent emotion models as specified by affect theories and, based on their dimensions, reviews computational methods for inferring affective information from natural language. This gives rise to a variety of use cases, which are detailed subsequently.

Affect theory {#sec:affecttheory}
-------------

In the field of psychology, there is no consensus regarding a universal classification of emotions [@Frijda1988; @Izard2009], as physiological arousal in the proposed theories varies with causes, cognitive appraisal processes, and context. Yet a conventional approach is to distinguish emotions based on how the underlying constructs are defined. On the one hand, emotions can be defined as a set of discrete states with mutually-exclusive meanings, while, on the other hand, emotions can also be characterized by a combination of numerical dimensions, each associated with a rating of intensity. The categorization into either a discrete set or a combination of intensity labels yields later benefits with regard to computational implementation, as it directly aids in formalizing the different machine learning models.

Categorical emotion models involve a variety of prevalent examples, including the so-called basic emotions. These introduce a discrete set of emotions with innate and universal characteristics [@tomkins1962; @Izard1992]. One of the first attempts by @Ekman1987 to classify emotions led to the categorization of six discrete items labeled as basic: namely, anger, disgust, fear, happiness, sadness, and surprise. The model was later extended by @averill1980theories to include trust and anticipation, resulting in eight basic emotions. An alternative categorization by Tomkins [@tomkins1962; @tomkins1963] classifies nine primary affects into positive (enjoyment, interest), neutral (surprise), and negative (anger, disgust, dissmell, distress, fear, shame) expressions.

Dimensional models of emotion locate constructs in a two- or multi-dimensional space [@Poria2017]. Here the assumption of disjunct categories is relaxed such that the magnitude along each dimension can be measured separately [@Russell1980], yielding continuous intensity scores. Different variants have been proposed, out of which we summarize an illustrative subset in the following. One of the earliest examples is Russell’s circumplex model [@Russell1980], consisting of bivariate classifications into valence and arousal. Depending on the strength of each component, certain regions in the two-dimensional space are given explicit interpretations (such as tense, aroused, excited) according to 28 emotional states. The Wheel of Emotions is an extension of the circumplex model whereby eight primary emotion dimensions are represented as four pairs of opposites: joy versus sadness, anger versus fear, trust versus disgust, and surprise versus anticipation [@Plutchik2001]. Recent approaches introduce complex hybrid emotion models, such as the Hourglass of Emotions [@Cambria2012], which represents affective states through both discrete categories and four independent, but concomitant, affective dimensions. However, neither the Wheel of Emotions nor the Hourglass of Emotions has yet found its way into common datasets for affective computing.

Datasets for benchmarking {#sec:datasets}
-------------------------

provides a holistic overview of datasets used for text-based affective computing. These datasets exhibit fundamentally different characteristics and challenges, as they vary in size, domain, linguistic style and underyling affect theory. We summarize key observations in the following.

In terms of text source, the datasets refer to tasks that utilize narrative materials from classic literature [@Alm2008], while others are based on traditional media [@Strapparava2007], and even Twitter or Facebook posts [@Preotiuc-Pietro2016]. Social media, in particular, tends to be informal and subject to variable levels of veracity, especially in comparison with more formal linguistic sources such as newspaper headlines. Similar variations become apparent in terms of where the annotations originate from. For instance, emotion labels can rely upon self-reporting of emotional experiences [@Wallbott.1986] or stem from ex post labeling efforts via crowdsourcing [@Mohammad2015].

The majority of datasets were annotated based on categorical emotion models, thereby defining a discrete set of labels. The chosen emotions largely follow suggestions from the different affect theories and predominantly focus on basic emotions (or subsets thereof) due to their prevalence. Even though the number and choice of emotions differ, one can identify four emotions that are especially common as they appear in almost all categorical models: anger, joy (happiness), fear, and sadness. Some emotions occur more often than others in the usual routines of humans [@Plutchik2001; @Ekman1987] and one thus obtains datasets  [[e.g.]{}, @Strapparava2007; @Mohammad2015] wherein the relative frequency of emotions is highly unbalanced. This imposes additional computational challenges as classifiers tend to overlook infrequent classes.

In contrast, dimensional models of emotions appear less frequently. Only one dataset, composed of newspaper headlines [@Strapparava2007], provides a score for each of the six emotion categories. From a methodological point of view, this categorization into dimension-based models requires different prediction models. While categorical models refer to machine learning with single-label classification tasks in the sense that we identify the appropriate item based on a discrete label, dimensional models allow for regression tasks in the sense that we predict a score for every item and emotion.

Computational methods
---------------------

The automatic recognition of text-based emotions relies upon different computational techniques that comprise lexicon-based methods and machine learning. Due to wealth of approaches, we can only summarize the predominant streams of research in the following and refer to @Calvo2010 [@Poria2017] for detailed methodological surveys.

### Lexicon-based methods

Lexicon-based approaches utilize pre-defined lists of terms that are categorized according to different affect dimensions [@Mohammad2012]. On the one hand, these lexicons are often compiled manually, a fact which can later be exploited for keyword matching. For instance, the Harvard IV dictionary (inside the General Inquirer software) and LIWC provide such lists with classification by domain experts [@Tausczik2010a]. These were not specifically designed for affective computing, but still include psychological dimensions ([e.g.]{}, pleasure, arousal and emotion in the case of Harvard IV; anxiety, anger, and sadness for LIWC). The NRC Word-Emotion Association lexicon was derived analogously but with the help of crowdsourcing rather than involving experts from the field of psychology research [@Mohammad2013]. The latter dictionary includes 10 granular categories such as anticipation, trust, and anger.

In order to overcome the need for manual dictionary creation, heuristics have been proposed to construct affect-related wordlists. Common examples include the WordNet-Affect dictionary, which starts with a set of seed words labeled as affect and then assigns scores to all other words based on their proximity to the seed words [@Strapparava2004a]. However, the resulting affect dictionary includes only general categories of mood- or emotion-related words, rather than further distinguishing the type of emotion. More recent methods operate, for instance, via mixture models [@Bandhakavi2017], fuzzy clustering [@Poria2014], or by incorporating word embeddings [@Li2017]. The precision of dictionaries can further be improved by embedding these in linguistic rules that adjust for the surrounding context.

Dictionary-based approaches are generally known for their straightforward use and out-of-the-box functionality. However, manual labeling is error-prone, costly, and inflexible as it impedes domain customization. Conversely, the vocabulary from the heuristics is limited to a narrow set of dimensions that were selected a priori and, as a result, this procedure has difficulties when generalizing to other emotions [[cf.]{} @Agrawal2012].

### Machine learning

Machine learning can infer decision rules for recognizing emotions based on a corpus of training samples with explicit labels [@Danisman2008; @Chaffar2011]. This can overcome the aforementioned limitations of lexicon-based methods concerning scalability and domain customization. Moreover, it can also learn implicit signals of emotions, since findings from a comprehensive, comparative study suggest that affect is rarely communicated through emotionally-charged lexical cues but rather via implicit expressions [@Balahur2012a].

Previous research has experimented with different models for inferring affect from narrative materials. Examples include methods that explicitly exploit the flexibility of machine learning, such as random forests [[e.g.]{}, @Gordeev2016] and support vector machines [[e.g.]{}, @Chatzakou2017; @Danisman2008], both of which have commonly been deployed in literature. Studies have shown that random forests tends to compute faster, while support vector machines yield superior performance [@Chatzakou2017]. These classifiers are occasionally, but infrequently, restricted to the subset of affect cues from emotion lexicons [@Bandhakavi2017]. However, the more common approach relies upon general linguistic features, [i.e.]{}, bag-of-words with subsequent tf-idf weighting [@Alm2005; @Strapparava2007]. Consistent with these works, we later draw upon machine learning models ([i.e.]{}, random forest and support vector machine) together with tf-idf features as our baseline.

### Deep learning

In the following, we discuss the few attempts at applying deep learning to affective computing, but find that actual performance evaluations are scarce. The approach in @Gordeev2016 predicts aggression expressed through natural language using convolutional neural networks with a sliding window and subsequent max-pooling. However, this approach is subject to several limitations as the network is designed to handle only a single dimension ([i.e.]{}, aggression) and it is thus unclear how it generalizes across multi-class predictions or even regression tasks that appear in dimensional emotion models. Even though the approach utilizes a network, its network architecture can only handle texts of predefined size, analogous to traditional machine learning. In this respect, it differs from recurrent networks, which iterate over sequences and thus can handle texts of arbitrary size.

The work in @Felbo2017a utilizes an LSTM that is pretrained with tweets based on the appearance of emoticons; however, this work does not report a comparison of their LSTM against a baseline from traditional machine learning. A different approach [@Gupta2017] utilizes a custom LSTM architecture in order to assign emotion labels to complete conversations in social media. However, this approach is tailored to the specific characteristics and emotions of this type of conversational-style data. In addition, the conclusion from their numerical experiments cannot be generalized to affective computing, since the authors labeled their dataset through a heuristic procedure and then reconstructed this heuristic with their classifier. Closest to our approach are experiments that include an LSTM for intensity estimation of emotions [@Goel.2017; @Lakomkin.2017; @Meisheri.2017; @Zhang.2017], but the results are limited to regression tasks where the presence of specific affective dimensions is given a priori.

Up to this point, the potential performance gains from using recurrent neural networks as the state of the art in deep learning have not yet been studied in relation to text-based emotion recognition. This fact was also noted in a recent literature survey [@Poria2017].

Transfer learning
-----------------

Transfer learning is a technique whereby knowledge from a source domain is leveraged in order to improve performance in a (possibly different) target domain. It is often used to overcome the constraints of limited training data, as well as for tasks that are sensitive to overfitting [@Pan.2010]. A straightforward approach to transferring knowledge in natural language applications is to draw upon pretrained word embeddings [@Kraus2017]. This approach merely requires an additional dataset without labels as it operates in unsupervised fashion. However, it only facilitates the representation of words and fails to help learning parameters inside the neural network.

More complex strategies can even utilize labels and perform transfer learning from a source to a target dataset. The underlying transfer can occur either concurrently or sequentially:

-   The former trains two networks concurrently on both the source and the target task with shared parameters. For instance, one network learns to translate sentences, while the other recognizes named entities [@Mou.2016]. This is known to help the network concentrate on a shared understanding and, in practice, puts emphasis on more abstract relationships.

-   The latter sequential procedure first trains a network on a source dataset and, in a second step, applies the network to the target dataset in order to fine-tune the network parameters [[e.g.]{}, @Kratzwald.2018]. This is often accompanied by minor modifications to network architectures ([e.g.]{}, by replacing the prediction layer). While such an approach seems intriguing, it is impeded by the heterogeneous nature of baseline datasets for emotion recognition.

However, natural language applications often lack suitable source datasets [@Mou.2016]. As a remedy, we propose sent2affect: that is, we employ not only a different *dataset* but also a different *task* (namely, sentiment analysis). To the best of our knowledge, this presents the first work on affective computing that attempts to accomplish an inductive knowledge transfer across tasks.

Methods {#sec:methods}
=======

This section presents our methods for inferring emotional states from narrative contents. We first summarize our baselines from traditional machine learning and deep learning, while the inherent nature of affective computing requires us to come up with multiple innovations concerning the network architecture. Our proposed advances are detailed in . Finally, we detail our novel approach to transfer learning, called sent2affect, whereby knowledge from the related task of sentiment analysis is applied to emotion recognition. illustrates this pipeline.

![Illustrative pipeline for inferring affective states from narrative materials. This can either happen through (i) traditional machine learning with feature engineering or, as proposed in this work, (ii) deep recurrent neural networks, optionally in conjunction with our proposed sent2affect transfer learning.[]{data-label="fig:pipeline"}](deeplearning.pdf){width="100.00000%"}

Benchmarks
----------

Baselines from traditional machine learning
-------------------------------------------

Traditional machine learning can only learn from a fixed-size vector of features and, for this purpose, features for machine learning are commonly built upon bag-of-words. The frequencies are further weighted by the tf-idf scheme in order to measure the relative importance of terms to a document within a corpus. Mathematically, the measure of term importance is obtained by computing the product of the term frequency and the inverse document frequency. This approach serves as a widely-accepted benchmark against which algorithms for natural language processing are evaluated.

The aforementioned features are then fed into the actual predictive models from traditional machine learning. Here we chose two approaches for both classification and regression as our baseline models: namely, random forest and support vector machine ([i.e.]{}, a support vector regression for predictive numerical scores). These are known for their superior performance in previous studies [[e.g.]{}, @Chatzakou2017]. Moreover, both approaches entail high flexibility when modeling non-linear relationships and demonstrate high accuracy even in settings where the number of potential features exceeds the number of observations.

### Baselines from na[ï]{}ve deep learning

Deep learning has triggered a paradigm shift in machine learning [@Kraus.2018] since it has yielded unprecedented performance results, especially for natural language processing. The theoretical argument for this is that recurrent neural networks from deep learning can iterate over the individual words of a sequence with arbitrary length. Here the input directly consists of words $x_1, \ldots, x_N$ and thus circumvents the need for feature engineering ([e.g.]{}, creating bag-of-words with tf-idf) as used in traditional machine learning. As a result, recurrent neural networks store a lower-dimensional representation of the input sequence that encodes the whole document and can even maintain the actual word order with long-ranging semantics [@Kraus.2018]. For this reason, recurrent neural networks differ from traditional machine learning, which can only adapt to short texts due to the use of $n$-grams.

We draw upon @Kraus2017 as the basis for our deep neural network architecture. This basic model consists of three layers: (a) an embedding layer that maps words in one-hot encoding onto low-dimensional vectors, (b) a recurrent layer to pass information on between words, and (c) a final dense layer for making the actual prediction. All three layers are described in detail in the online appendix. We experimented with this approach, but found that its performance is almost identical to a majority class vote. Therefore, we refrain from reporting the exact results; instead, we focus on the following improvements.

Proposed deep neural networks for affective computing {#sec:new_deep_learning}
-----------------------------------------------------

Using the aforementioned deep learning architectures is non-trivial for the following reasons. First, they are not suited to the small datasets from affective computing and typically lead to severe overfitting. Hence, we propose the use of a dropout layer as a form of regularization. Second, our task involves complex, open-domain language, which benefits further from bidirectional processing. Third, severe class imbalances are addressed by a weighted loss function. This loss function treats each class equally in order to avoid biases towards certain classes. Altogether, these extensions were necessary for using deep learning in our research setting.

### Dropout layer

Deep neural networks can easily consist of up to millions of free parameters and, consequently, these models run the risk of overfitting. This is especially a problem when the training data is scarce. As a remedy, the weights in the network are regularized by randomly dropping out a certain share of neurons in order to improve the generalizability of the network. This prevents the neurons from co-adapting too much during training [@Srivastava.2014]. We use dropout within the recurrent layer; that is, we randomly drop out connections between the recurrent LSTM cells. Dropout is disabled, [i.e.]{}, all neurons are used, during test time in order to leverage the full predictive power of the learned parameters (cf. the online appendix for a detailed specification). Furthermore, we apply dropout between the output of the recurrent layer and the input to the prediction layer.

### Bidirectional processing

To further improve the predictive performance of the base model, we draw upon so-called bidirectional recurrent layers, which have shown success in various other domains. That is, we use not only one but two LSTM layers to read the text. While one layer processes the text from left to right, a second one processes the text from right to left. More formally, let $h_1$ determine the hidden state of the LSTM network that processes the input in the forward direction and $h_2$ the hidden state of the LSTM that reads the text backwards. We then use the concatenation of both hidden states, [e.g.]{}, $[h_1, h_2]$, as input for the final prediction layer. Thus we are able to cover long- and short-term dependencies in both directions. We later abbreviate the bidirectional LSTM via BiLSTM and additionally run separate experiments for comparing the performance across the LSTM and BiLSTM.

### Weighted loss functions for unbalanced data

Affective computing commonly involves multiple, highly imbalanced target labels. Using a na[ï]{}ve loss function in this case would optimize towards the majority class and thus result in a performance similar to a majority vote. Such problems are typically addressed by over- or undersampling, yet these approaches yielded only marginal improvements in our experiments. As an alternative, we suggest the use of a weighted loss function. This multiplies the error of each data point with a weight that is the inverse size of the corresponding class.

Assume a training sample $x_i$ with ground-truth label $y_i$, and $p_{ik}$ denoting the output of the prediction layer, [e.g.]{}, the probability of $x_i$ belonging to class $k$. Then the weighted loss for $x_i$ is calculated via $$\mathcal{L}(x_i, \theta) = w_{i}  \sum_{k=1}^{K} \mathds{1}_{y_i=k} \log p_{ik}$$ with $\mathds{1}$ denoting the indicator function. The weight $w_{i}$ for input $x_i$ depends solely on its ground truth label $y_i$ and, similar to @King.2001, is calculated as $$w_{i} = \frac{N}{K \sum_{j} \mathds{1}_{y_j=y_i}} ,$$ where $K$ denotes the total number of classes and $N$ the number of samples.

Sent2affect approach to transfer learning across tasks
------------------------------------------------------

Due to the large number of degrees-of-freedom, training deep neural networks is often associated with challenges ([e.g.]{}, overfitting, ineffective generalization). In practice, this is encountered by large datasets in order to prevent overfitting and, hence, a different strategy is often applied when handling smaller datasets such as those in our experiments. Here the idea is to implement transfer learning, [i.e.]{}, the inductive transfer of knowledge from a different, yet related, task to the problem under investigation. In our case, we develop a novel approach, sent2affect, as detailed in the following.

The choice of the source task is non-trivial and it is mainly tasks of a semantically similar nature that result in the transferability of the network. For this purpose, we suggest the use of sentiment analysis as a related task, since it shares a certain similarity in the sense that positive and negative polarity is inferred from linguistic materials; however, sentiment analysis differs from affective computing, as it does not address affective dimensions or emotional states. The relatedness between both tasks enables the network to infer similar representation for both.

Formally, our approach to transfer learning optimizes the weights of a neural network for a target task $\mathcal{T}$ and dataset $\mathcal{D_T}$ based on a different, yet related, source task $\mathcal{S}$ with dataset $\mathcal{D_S}$. After optimizing the parameters of our network for $\mathcal{S}$ on $\mathcal{D_S}$ we replace the task-specific prediction layer of the network to yield predictions for our target task $\mathcal{T}$. Therefore, we utilize the estimated parameters as an initial value for further optimization with the help of the actual dataset $\mathcal{D_T}$ [@Pan.2010]. The pseudocode of the overall process is stated in .

In our experiments, we utilize a large-scale, public dataset[^1] as a basis for knowledge induction. This dataset finds widespread application in sentiment analysis and includes about samples labeled according to positive or negative sentiment. We then optimize the deep neural network with the goal of predicting the underlying sentiment scores. The resulting coefficients of the network are further trained with an actual dataset from affective computing. Here the differences in the data type of the prediction outcome ([i.e.]{}, computing a positivity/negativity score versus affective dimensions) are handled by removing the dense layer and, instead, amending a new prediction layer that targets the new output. As a result, the majority of weights benefits from transfer learning, while only the neurons in the prediction layer are training after a random initialization. The intuition of this approach is as follows: deep neural networks generally contain multiple layers, where layers closer to the final prediction layers are supposed to encode the original input at a higher level of abstraction.

Given training data $\mathcal{D_T}$ for the affective computing task $\mathcal{T}$ and additional corpus $\mathcal{D_S}$ for sentiment analysis $\mathcal{S}$ $m \gets $ Initialize recurrent neural network ([i.e.]{}consisting of recurrent layer $f$, dense layer $\psi$, …) $m \gets $ Estimate parameters w.r.t. $\mathcal{S}$ using $\mathcal{D_S}$ $\psi \gets $ Replace dense layer with randomly-initialized dense layer according to the dimensions of $\mathcal{T}$ $\psi \gets $ Fine-tune $\psi$ w.r.t. $\mathcal{T}$ using $\mathcal{D_T}$ **return** Recurrent neural network $m$

Model estimation
----------------

Consistent with previous research [@manning1999foundations], we tokenize each document, convert all characters to lower-case, and remove punctuation, numbers, and stop words. Moreover, we perform stemming, which maps inflected words onto a base form; [e.g.]{}, ** and ** are both mapped onto **. We conducted all pre-processing operations to yield bag-of-words representations by using the natural language tookit NLTK.

For those datasets with no designated test set, we introduced a random $80/20$ split in training and test data. For the random forest classifier, we manually optimized over the number of trees, maximum number of features for every split, and the depth. For the support vector classifier, we conducted an extensive grid-search over the hyperparameters following @hsu2003practical. In detail, we experimented with linear, radial basis function, and sigmoid kernels, optimizing the cost $C$ over $2^{-5}, 2^{-3},\ldots,2^{15}$ and the radius parameter $\gamma$ over $2^{-15},2^{-13},\ldots,2^3$. For unbalanced datasets, we weighted the loss function by class frequency in order to prevent models from predicting the majority classes only.

We used different deep learning models. Depending on the specification, we used pre-trained GloVe[^2] embeddings or randomly-initialized embeddings (which are learned jointly during the training phase). The models were trained using the Adam optimizer, whereby the process was stopped once we noted an increase in the validation error. For reasons of reproducibility, we report the performance metrics averaged over 10 independent runs.

Evaluation {#sec:evaluation}
==========

This section reports our computational experiments evaluating the improvements gained by using deep neural networks (and especially transfer learning) for affective computing. Here we draw upon all datasets from and, according to the type of the underlying affect theory, we divide the performance measurements into classification and regression tasks.

Classification according to categorical emotion models
------------------------------------------------------

We begin with classification tasks according to categorical emotion models, where the objective is to predict the predominant emotion(s). We follow previous literature [[e.g.]{}, @Chatzakou2017; @Danisman2008] and analogously choose two baselines prevalent in traditional machine learning: namely, the random forest classifier and the support vector machine. Both are fed with bag-of-words with tf-idf weighting, whereas the proposed deep neural networks circumvent the need for feature engineering. Here we compare variants that extend the LSTM[^3] with bidirectional encodings and pretrained word embeddings. The resulting performance is listed in , where we account for unbalanced distributions of labels by using the weight-averaged F1-score. The F1-score for a single class is given by the harmonic mean of precision and recall, [i.e.]{}, $$\text{F1} = 2 \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} .$$ In addition, we report sensitivity and specificity scores. The sensitivity of a single class equals the recall, while the specificity measures the fraction of true negatives. Similar to the F1-score, we calculate both independently for each class, [i.e.]{}, $$\text{sensitivity} = \mathit{TP}/(\mathit{TP}+\mathit{FN}) 
\qquad \text{and} \qquad
\text{specificity} = \mathit{TN}/(\mathit{TN}+\mathit{FP}),$$ where the number of true positives and true negatives is denoted by $\mathit{TP}$ and $\mathit{TN}$, and the number of false positives and false negatives is denoted by $\mathit{FP}$ and $\mathit{FN}$. For the final scores, we average over all classes weighted by the class size.

Our results in consistently reveal superior performance through the use of deep learning. We observe that, regardless of the architecture, models with pre-trained GloVe embeddings outperform their counterparts with randomly-initialized word embeddings. In fact, the use of pre-trained word embeddings yields performance improvements over the best baseline in $9$ out of $10$ experiments. An explanation stems from the fact that embeddings which have not been pre-trained result in considerably more degrees-of-freedom and thus a greater chance of overfitting. Our initial expectations are met as the imposed dropout layers and loss-weighting successfully diminish the problem of overfitting. Furthermore, our imposed architectural enhancements surpass the performance of previous deep learning architectures, such as that proposed by [@Kraus2017]. As such, the bidirectional recurrent layers outperform the variant with a unidirectional layer in four out of five experiments, yielding the only architecture that consistently outperforms the traditional baseline on all datasets, with improvements between and across the datasets. We experimented with the naïve network from @Kraus2017, but it failed in three out of five datasets resulting in merely predicting the majority class; hence, we omitted the results.

The performance gains from our proposed architectural improvements are a result of the class imbalance and the language noise of the source. For instance, the highest relative improvement over traditional machine learning is achieved in the case of the dataset of headlines [@Strapparava2007], constructed of four equally-sized classes and proper English. On the other hand, the dataset of election tweets [@Mohammad2015], which is composed of highly unbalanced classes and considerable language noise, yields the lowest improvement.

reports sensitivity and specificity scores as an additional robustness check. The results confirm our findings, [i.e.]{}, we witness the largest performance improvements for datasets with less noise. For the election tweet dataset [@Mohammad2015], the best bidirectional LSTM model achieves a sensitivity of $56.9$, while the best baseline achieves a slightly better score of $57.1$. We can significantly strengthen our results for this challenging dataset by applying transfer learning, as reported in \[sec:results\_transfer\_learning\].

Regression according to dimensional affect models
-------------------------------------------------

Depending on the affect theory, one can also model emotional categories according to dimensional ratings and, as a result, this is implemented as a regression task, where the intensity of emotional states is predicted. We choose the same baselines as in the previous experiments and compare them to deep neural networks. All models are evaluated based on the mean squared error (MSE).

reports our results. These show a consistent improvement of up to as a result of using deep learning as compared to traditional machine learning. Similar to the classification task, our findings identify the BiLSTM with pre-trained word embeddings as the superior method in all seven experiments. We further note that the BiLSTM appears to outperform the unidirectional LSTM in all experiments. The relative performance increases vary between the different affective dimensions.

Transfer learning via sent2affect {#sec:results_transfer_learning}
---------------------------------

The previous experiments revealed consistent improvements through the use of deep learning; however, several benchmark datasets entail only a fairly small set of samples, which could impede the training of deep neural networks. For instance, the dataset of inferring emotions from election tweets [@Mohammad2015] comprises only 1,646 samples for training. A potential remedy is utilizing large-scale datasets from other tasks and then inducing knowledge to affective computing. More precisely, we now experiment with the potential performance improvements to be gained by additionally applying our transfer learning approach . By inducing network parameters from sentiment analysis to affective computing, we benefit from the considerably larger datasets that are used in sentiment analysis, since the sentiment dataset consists of about tweets that are associated with positive and negative labels.

compares our transfer learning approach against two baselines: (i) a na[ï]{}ve BiLSTM and (ii) the transfer learning approach of @Kraus2017, where only GloVe word-embeddings are pre-trained. We choose the election tweets [@Mohammad2015] and general tweets [@SemEval2018Task1] datasets to demonstrate how we can transfer the knowledge from thousands of sentiment-labeled tweets to the task of emotion recognition. Furthermore, na[ï]{}ve deep learning alone yields an inferior performance. While the BiLSTM with pre-trained word embeddings has previously represented the best-performing architecture, we still observe that transfer learning yields additional improvements. These amount to for the election tweets and for the general tweets. Evidently, transfer learning can successfully benefit from the large-scale dataset for sentiment analysis and, as a result, optimizes the neuron weights such that these find a more generalizable representation of emotion-laden materials.

Discussion {#sec:discussion}
==========

Comparison
----------

Our series of experiments reveals considerable and consistent performance improvements over default implementations of deep learning through the use of our customized networks. This points towards the need to customize deep neural networks according to the unique characteristics of the underlying task.

In this paper, we refrained from evaluating performance on the basis of a single dataset and, instead, perform a holistic analysis, demonstrating that our customized networks outperformed the baselines in all experiments by up to . Interestingly, our proposed modifications, such as with regard to regularization, were even able to learn the underlying relationships from the rather small datasets of merely 1,000 observations. However, we observe an overall pattern whereby the performance improvements tend to be higher when there is less language noise. In addition, we observe further improvement through the use of word embeddings, as these reduce the high-dimensional vectors with terms as one-hot encoding to lower-dimensional spaces.

In the majority of experiments, the superior results stem from using a bidirectional LSTM as compared to a simple LSTM. We note that not only traditional machine learning but all network architectures required extensive training in order to ensure that embeddings and dropout layer functioned well together. Finally, the task of emotion recognition in affective computing is related to sentiment analysis, which infers a positive/negative polarity from linguistic materials. Hence, it is interesting to study whether one can further improve performance through an inductive transfer of knowledge from a different task (rather than a different dataset), despite the distinct objective, linguistic style, and annotation scheme. As a result, our sent2affect implementation of transfer learning establishes additional improvements of up to .

Deep-learning-based affect computing for decision support in social media
-------------------------------------------------------------------------

As a proof of concept, we utilize our bidirectional LSTM to support the notoriously difficult task of classifying news into factual and non-factual. This demonstrates how affective computing can eventually facilitate decision support for social media platforms seeking to recognize and prevent the spread of . We utilize the dataset of [@Shu.2017] and predict whether a news item is factual. The prediction model is given by a logistic regression that is fed with the output of our affect prediction layer. Our approach achieves an accuracy of when using the affective dimensions of the headlines and when using separate affective dimensions of both headlines and content. This almost matches the reported baseline performance from prior research [@Rubin.2015], where a content-based classifier was used to detect fabricated news items. However, we refrain from learning towards certain linguistic devices or individual stories. Instead, our approach ensures generalizability by identifying highly polarizing language as part of its decision support.

Further use cases of deep-learning-based affective computing for better decision support {#sec:applications}
----------------------------------------------------------------------------------------

Text-based affective computing drives decision support in a variety of application areas in which understanding the emotional state of individuals is crucial. provides an overview of interesting examples from research, as well as actual use cases from businesses. This table is intended to give an overview of areas where decision support could potentially be improved through the use of our deep-learning-based models for affective computing. It is evident that affective computing facilitates decision-making in all operational areas of businesses, such as management, marketing, and finance. For instance, firms can infer the perceived emotion of customers from online product reviews and base managerial decisions on this data in order to support product development [@Ullah2016] and advertising [@Ang2000]. In a financial context, emotional media content has been identified as a driver in the decision-making of investors [@Prollochs2016], which can thus serve as a decision rule for stock investments [@Gilbert2010].

Beyond that, deep learning for emotion recognition could also facilitate public decision support with respect to politics and even education, as well as healthcare for individuals. For instance, affective computing can infer emotion concerning personal health conditions [@Anderson2011; @Desmet2013; @Greaves2013; @VanDerZanden2014] and during learning processes [@Rodriguez2012]. Notably, all of the prior references engage in affect-aware decision-making, but have not yet evaluated the use of deep learning.

Implications for management and practice
----------------------------------------

Even though deep learning has gained considerable traction lately, its use cases outside of academia remain scarce. A possible reason is located in the complexity of operationalizing deep neural networks. While recurrent architectures have previously been applied to sentiment analysis, the task of emotion recognition requires several modifications in order to obtain a better-than-random performance. This specifically applies to the proposed bidirectional processing of texts, regularization, and loss functions that can handle highly imbalanced datasets. As a direct recommendation for use cases of affective computing, we propose a shift towards customized network architectures, even for fairly small datasets of around 1,000 training samples, as in our case. Altogether, this highlights the need for a thorough understanding by practitioners of the available tools in order to benefit from deep learning.

Affective computing for linguistic materials yields new opportunities for business models and consumer-centered services [@Li.2011; @Doucet.2012; @Dai.2015; @Yin.2014]. Detecting and subsequently responding to the emotional states of users, customers, patients, and employees has the potential to significantly accelerate and improve management processes and optimize human-computer interactions. Here text remains a critical form of communication, while attempts have also been made to apply affective computing to speech or other multimodal input [@Calvo2010], including visual data [@Chen2017; @ElAyadi2011; @Shan2009]. Management should assess potential use cases in critical areas of operations from their own organizations. Our overview in provides illustrative examples, while further applications are likely to arise with recent methodological innovations.

Implications for research
-------------------------

The process of improving the performance of affective computing would benefit considerably from a rigorous suite of baseline datasets. In the status quo, a variety of datasets with distinct goals and purposes is commonly used for benchmarking methodological innovations for affective computing. For instance, our literature survey identified four different strategies for annotating, including simple labels, multi-class labels, and numerical scores. Moreover, the set of affective dimensions varied between two ([i.e.]{}, valence, arousal without explicitly naming emotions) and a set of 8 emotions ([e.g.]{}, anger, disgust, surprise). However, this directly links to challenges concerning comparability and generalizability. In this sense, a network architecture that has been found effective for one annotation scheme might not work out for other datasets. On top of that, different labels prohibit transfer learning and thus impede performance. We therefore suggest a standardized approach to annotations.

According to our literature review, datasets for affective computing vary in size from 1,000 instances to 7,902, and yet all of them remain fairly small when compared to other applications of deep learning. As a result, this is known to limit the performance of bidirectional LSTMs and other deep neural network architectures, which generally require large-scale datasets. For instance, datasets for sentiment analysis, such as the one used for our transfer learning approach, consist of up to labeled samples. Future research should thus aim at creating larger datasets in order to enable the effective exploitation of deep learning.

Conclusion {#sec:conclusion}
==========

Affective computing allows one to infer individual and collective emotional states from textual data and thus offers an anthropomorphic path for the provision of decision support. Even though deep learning has yielded considerable performance improvements for a variety of tasks in natural language processing, na[ï]{}ve network architectures struggle with the task of emotion recognition. As a remedy, several modifications are presented in this paper: namely, bidirectional processing, dropout regularization, and weighted loss functions in order to cope with imbalances in the datasets.

Our computational experiments span categorical and dimensional emotion models, which require tailored algorithmic implementations involving, [e.g.]{}, multi-class classification, as well as regression tasks and transfer learning. Our results show that pre-trained bidrectional LSTMs consistently outperform the baseline models from traditional machine learning. The performance improvements can even range up to in F1-score for classification and in MSE for regression. We propose sent2affect, a customized strategy of transfer learning that draws upon the different task of sentiment analysis (as opposed to different datasets, as is usually the case), which is responsible for further performance improvements of between and .

Acknowledgements {#acknowledgements .unnumbered}
================

The authors gratefully acknowledge the financial support for Suzana Ilić from Prof. Kotaro Nakayama and Prof. Yutaka Matsuo, Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.

[^1]: Kaggle: Twitter sentiment analysis, retrieved from <https://www.kaggle.com/c/twitter-sentiment-analysis2>, March 21, 2018.

[^2]: The pre-trained word embeddings can be retrieved from <http://nlp.stanford.edu/data/glove.6B.zip>.

[^3]: We use the acronym LSTM when referring to the unidirectional model. Whenever we refer to the bidirectional LSTM model, we use the explicit designation BiLSTM.
