This paper is the basis paper for the accepted IJCNN challenge One-Minute Gradual-Emotion Recognition (OMG-Emotion) by which we hope to foster long-emotion classification using neural models for the benefit of the IJCNN community. The proposed corpus has as the novelty the data collection and annotation strategy based on emotion expressions which evolve over time into a specific context. Different from other corpora, we propose a novel multimodal corpus for emotion expression recognition, which uses gradual annotations with a focus on contextual emotion expressions. Our dataset was collected from Youtube videos using a specific search strategy based on restricted keywords and filtering which guaranteed that the data follow a gradual emotion expression transition, i.e. emotion expressions evolve over time in a natural and continuous fashion. We also provide an experimental protocol and a series of unimodal baseline experiments which can be used to evaluate deep and recurrent neural models in a fair and standard manner.
In the related field of sentiment analysis, the Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos (MOSI) @cite_17 has in-the-wild data. This corpus contains 2199 video segments obtained from Youtube by a data crawler for the purpose of sentiment analysis. The authors of this corpus provide extensive annotations based on sentiment analysis of visual gestures and verbal information. The dataset was annotated using the Amazon Mechanical Turk tool, and for each video segment, five different annotators evaluate the video based on a scale from strongly positive to strongly negative, and four head gestures (smile, head nod, frown and head shake). Although a very challenging dataset, it does not contain contextual information, unlike the IEMOCAP, and thus cannot be used to train emotional models that make use of changes in contextual information. The use of real-world information makes it more suitable for generalized emotion recognition, but the lack of emotional annotation, be it categorical or dimensional, reduces the applicability of the corpus for training automatic emotion recognition systems.
@cite_17
2465534249
People are sharing their opinions, stories and reviews through online video sharing websites every day. Studying sentiment and subjectivity in these opinion videos is experiencing a growing attention from academia and industry. While sentiment analysis has been successful for text, it is an understudied research question for videos and multimedia content. The biggest setbacks for studies in this direction are lack of a proper dataset, methodology, baselines and statistical analysis of how information from different modality sources relate to each other. This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI). The dataset is rigorously annotated with labels for subjectivity, sentiment intensity, per-frame and per-opinion annotated visual features, and per-milliseconds annotated audio features. Furthermore, we present baselines for future studies in this direction as well as a new multimodal fusion approach that jointly models spoken words and visual gestures.