Past shared tasks on emotions use data with both overt expressions of emotions (I am so happy to see you!) as well as subtle expressions where the emotions have to be inferred, for instance from event descriptions. Further, most datasets do not focus on the cause or the stimulus of the emotion. Here, for the first time, we propose a shared task where systems have to predict the emotions in a large automatically labeled dataset of tweets without access to words denoting emotions. Based on this intention, we call this the Implicit Emotion Shared Task (IEST) because the systems have to infer the emotion mostly from the context. Every tweet has an occurrence of an explicit emotion word that is masked. The tweets are collected in a manner such that they are likely to include a description of the cause of the emotion - the stimulus. Altogether, 30 teams submitted results which range from macro F1 scores of 21 to 71 . The baseline (MaxEnt bag of words and bigrams) obtains an F1 score of 60 which was available to the participants during the development phase. A study with human annotators suggests that automatic methods outperform human predictions, possibly by honing into subtle textual clues not used by humans. Corpora, resources, and results are available at the shared task website at this http URL
One of the first corpora annotated for emotions is that by who analyze sentences from fairy tales. annotate news headlines with emotions and valence, annotate tweets on elections, and tweets of a stance dataset . The SemEval-2018 Task 1: Affect in Tweets @cite_7 includes several subtasks on inferring the affectual state of a person from their tweet: emotion intensity regression, emotion intensity ordinal classification, valence (sentiment) regression, valence ordinal classification, and multi-label emotion classification. In all of these prior shared tasks and datasets, no distinction is made between implicit or explicit mentions of the emotions. We refer the reader to for a more detailed overview of emotion classification datasets.
@cite_7
2250812182
Past work on emotion processing has focused solely on detecting emotions, and ignored questions such as ‘who is feeling the emotion (the experiencer)?’ and ‘towards whom is the emotion directed (the stimulus)?’. We automatically compile a large dataset of tweets pertaining to the 2012 US presidential elections, and annotate it not only for emotion but also for the experiencer and the stimulus. We then develop a classifier for detecting emotion that obtains an accuracy of 56.84 on an eight-way classification task. Finally, we show how the stimulus identification task can also be framed as a classification task, obtaining an F-score of 58.30.