Emotion recognition plays an important role in human-computer interaction (HCI) and has been extensively studied for decades. Although tremendous improvements have been achieved for posed expressions, recognizing human emotions in "close-to-real-world" environments remains a challenge. In this paper, we proposed two strategies to fuse information extracted from different modalities, i.e., audio and visual. Specifically, we utilized LBP-TOP, an ensemble of CNNs, and a bi-directional LSTM (BLSTM) to extract features from the visual channel, and the OpenSmile toolkit to extract features from the audio channel. Two kinds of fusion methods, i,e., feature-level fusion and model-level fusion, were developed to utilize the information extracted from the two channels. Experimental results on the EmotiW2018 AFEW dataset have shown that the proposed fusion methods outperform the baseline methods significantly and achieve better or at least comparable performance compared with the state-of-the-art methods, where the model-level fusion performs better when one of the channels totally fails.
Facial activity plays a significant role in emotion recognition. One of the major steps is to extract the most discriminative features that are capable of capturing appearance and geometry facial changes. These features can be roughly divided into two categories: human designed and learned features. Most recently, CNNs @cite_2 @cite_20 @cite_48 @cite_26 @cite_39 @cite_35 @cite_28 @cite_31 have achieved promising recognition performance under real-world conditions as demonstrated in the recent EmotiW2017 @cite_11 @cite_13 @cite_32 and EmotiW2018 @cite_27 challenges.
@cite_35 @cite_26 @cite_28 @cite_48 @cite_32 @cite_39 @cite_27 @cite_2 @cite_31 @cite_13 @cite_20 @cite_11
2811322499 2561355506 2548264631 2293804193
Facial expression recognition is a topical task. However, very little research investigates subtle expression recognition, which is important for mental activity analysis, deception detection, etc. We address subtle expression recognition through convolutional neural networks (CNNs) by developing multi-task learning (MTL) methods to effectively leverage a side task: facial landmark detection. Existing MTL methods follow a design pattern of shared bottom CNN layers and task-specific top layers. However, the sharing architecture is usually heuristically chosen, as it is difficult to decide which layers should be shared. Our approach is composed of (1) a novel MTL framework that automatically learns which layers to share through optimisation under tensor trace norm regularisation and (2) an invariant representation learning approach that allows the CNN to leverage tasks defined on disjoint datasets without suffering from dataset distribution shift. To advance subtle expression recognition, we contribute a Large-scale Subtle Emotions and Mental States in the Wild database (LSEMSW). LSEMSW includes a variety of cognitive states as well as basic emotions. It contains 176K images, manually annotated with 13 emotions, and thus provides the first subtle expression dataset large enough for training deep CNNs. Evaluations on LSEMSW and 300-W (landmark) databases show the effectiveness of the proposed methods. In addition, we investigate transferring knowledge learned from LSEMSW database to traditional (non-subtle) expression recognition. We achieve very competitive performance on Oulu-Casia NIR&Vis and CK+ databases via transfer learning. Facial emotion recognition is the most significant parameter for an efficacious Human Machine Interaction (HMI). It plays a crucial role in interpreting and communicating with the people who have speaking impairments as well as a medium to understand and communicate with infants who cannot emote their feelings verbally. In this paper, we propose a hybrid feature extraction technique consisting of Discrete Wavelet Transform (DWT) accompanied by Shape Feature Extraction Technique (SFET).The features extracted were tested on standard classifiers such as Support Vector Machine (SVM) and K-Nearest Neighbourhood (KNN) classifiers. Facial images from JAFFE and Cohn-Kennedy databases were utilized for training as well as testing. The work shows a very high facial emotion recognition rate of 93.94 and 91.8 with the proposed method for JAFFE and Cohn-Kanade databases respectively. In the past three years, Emotion Recognition in the Wild (EmotiW) Grand Challenge has drawn more and more attention due to its huge potential applications. In the fourth challenge, aimed at the task of video based emotion recognition, we propose a multi-clue emotion fusion (MCEF) framework by modeling human emotion from three mutually complementary sources, facial appearance texture, facial action, and audio. To extract high-level emotion features from sequential face images, we employ a CNN-RNN architecture, where face image from each frame is first fed into the fine-tuned VGG-Face network to extract face feature, and then the features of all frames are sequentially traversed in a bidirectional RNN so as to capture dynamic changes of facial textures. To attain more accurate facial actions, a facial landmark trajectory model is proposed to explicitly learn emotion variations of facial components. Further, audio signals are also modeled in a CNN framework by extracting low-level energy features from segmented audio clips and then stacking them as an image-like map. Finally, we fuse the results generated from three clues to boost the performance of emotion recognition. Our proposed MCEF achieves an overall accuracy of 56.66 with a large improvement of 16.19 with respect to the baseline. The Emotion Recognition in the Wild (EmotiW) Challenge has been held for three years. Previous winner teams primarily focus on designing specific deep neural networks or fusing diverse hand-crafted and deep convolutional features. They all neglect to explore the significance of the latent relations among changing features resulted from facial muscle motions. In this paper, we study this recognition challenge from the perspective of analyzing the relations among expression-specific facial features in an explicit manner. Our method has three key components. First, we propose a pair-wise learning strategy to automatically seek a set of facial image patches which are important for discriminating two particular emotion categories. We found these learnt local patches are in part consistent with the locations of expression-specific Action Units (AUs), thus the features extracted from such kind of facial patches are named AU-aware facial features. Second, in each pair-wise task, we use an undirected graph structure, which takes learnt facial patches as individual vertices, to encode feature relations between any two learnt facial patches. Finally, a robust emotion representation is constructed by concatenating all task-specific graph-structured facial feature relations sequentially. Extensive experiments on the EmotiW 2015 Challenge testify the efficacy of the proposed approach. Without using additional data, our final submissions achieved competitive results on both sub-challenges including the image based static facial expression recognition (we got 55.38 recognition accuracy outperforming the baseline 39.13 with a margin of 16.25 ) and the audio-video based emotion recognition (we got 53.80 recognition accuracy outperforming the baseline 39.33 and the 2014 winner team's final result 50.37 with the margins of 14.47 and 3.43 , respectively).