In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative local motion patterns feature, with three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition. And, the third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variation and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-rank state-of-the-art methods for macro expression recognition.
Concerning deep learning approaches, we underline important contrasts. On one hand, deep learning approaches provide good results for macro expression recognition (see * lines in Table ). Deep learning approaches are based on auto-encoded features optimized for specific datasets. For example, Breuer and Kimmel @cite_4 employ Ekman's facial action coding system (FACS) in order to boost the performances of their approach. On the other hand, deep learning results are clearly lower than handcrafted approaches in micro expressions recognition (Table ). Furthermore, most of the time, in deep learning approaches, there is an augmentation of the initial data available (flip, rotate, Gaussian blur, etc.) in order to provide sufficient training data. Hence, comparison between deep learning approaches (using augmented data) and handcraft approaches (using only the original data) must be handled with care.
@cite_4
2060312700 2112480082 2963958000 2910652840
This paper exploits the discriminative power of manifold learning in conjunction with the parsimonious power of sparse signal representation to perform robust facial expression recognition. By utilizing an l1 reconstruction error and a statistical mixture model, both accuracy and tolerance to occlusion improve without the need to perform neutral frame subtraction. Initially facial features are mapped onto a low dimensional manifold using supervised Locality Preserving Projections. Then an l1 optimization is employed to relate surface projections to training exemplars, where reconstruction models on facial regions determine the expression class. Experimental procedures and results are done in accordance with the recently published extended Cohn-Kanade and GEMEP-FERA datasets. Results demonstrate that posed datasets overemphasize the mouth region, while spontaneous datasets rely more on the upper cheek and eye regions. Despite these differences, the proposed method overcomes previous limitations to using sparse methods for facial expression and produces state-of-the-art results on both types of datasets. For one-shot learning gesture recognition, two important challenges are: how to extract distinctive features and how to learn a discriminative model from only one training sample per gesture class. For feature extraction, a new spatio-temporal feature representation called 3D enhanced motion scale-invariant feature transform (3D EMoSIFT) is proposed, which fuses RGB-D data. Compared with other features, the new feature set is invariant to scale and rotation, and has more compact and richer visual representations. For learning a discriminative model, all features extracted from training samples are clustered with the k-means algorithm to learn a visual codebook. Then, unlike the traditional bag of feature (BoF) models using vector quantization (VQ) to map each feature into a certain visual codeword, a sparse coding method named simulation orthogonal matching pursuit (SOMP) is applied and thus each feature can be represented by some linear combination of a small number of codewords. Compared with VQ, SOMP leads to a much lower reconstruction error and achieves better performance. The proposed approach has been evaluated on ChaLearn gesture database and the result has been ranked amongst the top best performing techniques on ChaLearn gesture challenge (round 2). Deep learning based approaches have been dominating the face recognition field due to the significant performance improvement they have provided on the challenging wild datasets. These approaches have been extensively tested on such unconstrained datasets, on the Labeled Faces in the Wild and YouTube Faces, to name a few. However, their capability to handle individual appearance variations caused by factors such as head pose, illumination, occlusion, and misalignment has not been thoroughly assessed till now. In this paper, we present a comprehensive study to evaluate the performance of deep learning based face representation under several conditions including the varying head pose angles, upper and lower face occlusion, changing illumination of different strengths, and misalignment due to erroneous facial feature localization. Two successful and publicly available deep learning models, namely VGG-Face and Lightened CNN have been utilized to extract face representations. The obtained results show that although deep learning provides a powerful representation for face recognition, it can still benefit from preprocessing, for example, for pose and illumination normalization in order to achieve better performance under various conditions. Particularly, if these variations are not included in the dataset used to train the deep learning model, the role of preprocessing becomes more crucial. Experimental results also show that deep learning based representation is robust to misalignment and can tolerate facial feature localization errors up to 10 of the interocular distance. The automatic recognition of spontaneous facial micro-expressions becomes prevalent as it reveals the actual emotion of humans. However, handcrafted features employed for recognizing micro-expressions are designed for general applications and thus cannot well capture the subtle facial deformations of micro-expressions. To address this problem, we propose an end-to-end deep learning framework to suit the particular needs of micro-expression recognition (MER). In the deep model, re- current convolutional networks are utilized to learn the representation of subtle changes from image sequences. To guarantee the learning of deep model, we present a temporal jittering procedure to greatly enrich the training samples. Through performing the experiments on three spontaneous micro-expression datasets, i.e., SMIC, CASME, and CASME2, we verify the effectiveness of our proposed MER approach.