In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative local motion patterns feature, with three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition. And, the third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variation and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-rank state-of-the-art methods for macro expression recognition.
Although most micro expression recognition studies have considered LBP-TOP, several authors investigate alternative methods. @cite_28 propose spatio-temporal completed local quantized pattern (STCLQP) and obtain promising performances. The reason may be that STCLQP provides more useful information for micro expression recognition, as it extracts jointly information characterizing magnitudes and orientations. @cite_0 employ temporal interpolation and motion magnification to counteract the low intensity of micro expressions. They show that longer interpolated sequences do not lead to improve performances, because the movement tends to be diluted, and interpolating micro expression segments using only 10 frames is enough. Recently, @cite_27 design a feature for micro expression recognition based on a robust optical flow method, and extract Main Directional Mean Optical-flow (MDMO). They showed that the magnitude is more discriminant than the direction when working with micro expression.Furthermore, several deep-learning methods have been proposed to deal with micro expression @cite_4 @cite_43 @cite_19 and they all present low performances.
@cite_4 @cite_28 @cite_0 @cite_43 @cite_27 @cite_19
2156489769 2782644834 2526853616 2501714289
Spontaneous facial micro-expression analysis has become an active task for recognizing suppressed and involuntary facial expressions shown on the face of humans. Recently, Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) has been employed for micro-expression analysis. However, LBP-TOP suffers from two critical problems, causing a decrease in the performance of micro-expression analysis. It generally extracts appearance and motion features from the sign-based difference between two pixels but not yet considers other useful information. As well, LBP-TOP commonly uses classical pattern types which may be not optimal for local structure in some applications. This paper proposes SpatioTemporal Completed Local Quantization Patterns (STCLQP) for facial micro-expression analysis. Firstly, STCLQP extracts three interesting information containing sign, magnitude and orientation components. Secondly, an efficient vector quantization and codebook selection are developed for each component in appearance and temporal domains to learn compact and discriminative codebooks for generalizing classical pattern types. Finally, based on discriminative codebooks, spatiotemporal features of sign, magnitude and orientation components are extracted and fused. Experiments are conducted on three publicly available facial micro-expression databases. Some interesting findings about the neighboring patterns and the component analysis are concluded. Comparing with the state of the art, experimental results demonstrate that STCLQP achieves a substantial improvement for analyzing facial micro-expressions. HighlightsWe propose spatiotemporal completed local quantized pattern for micro-expression analysis.We propose to use three useful information, including the sign-based, magnitude-based and orientation-based difference of pixels for LBP.We propose to use an efficient vector quantization and discriminative codebook selection to make LBP-TOP more discriminative and compact.We evaluate the framework on three publicly available facial micro-expression databases.We evaluate the influence of parameters, different components and codebook selection to spatiotemporal completed local quantized pattern. Micro-expressions are difficult to be observed by human beings due to its low intensity and short duration. Recently, several works have been developed to resolve the problems of micro-expression recognition caused by subtle intensity and short duration. One of them, Local binary pattern from three orthogonal planes (LBP-TOP) is primarily used to recognize micro-expression from the video recorded by high-speed camera. Several variances of LBP-TOP have also been developed to promisingly improve the performance of LBP-TOP for microexpression recognition. However, these variances of LBP-TOP including LBP-TOP cannot well extract the subtle movements of micro-expression so that they have the low performance. In this paper, we propose spontaneous local radon-based binary pattern to analyze micro-expressions with subtle facial movements. Firstly, it extracts the sparse information by using robust principal component analysis since micro-expression data are sparse in both temporal and spatial domains caused by short duration and low intensity. These sparse information can provide much motion information to dynamic feature descriptor. Furthermore, it employs radon transform to obtain the shape features from three orthogonal planes, as radon transform is robustness to the same histogram distribution of two images. Finally, one-dimensional LBP is employed in these shape features for constructing the spatiotemporal features for microexpression video. Intensive experiments are conducted on two available published micro-expression databases including SMIC and CASME2 databases for evaluating the performance of the proposed method. Experimental results demonstrate that the proposed method achieves promising performance in microexpression recognition. Micro-expressions are brief facial movements characterized by short duration, involuntariness and low intensity. Recognition of spontaneous facial micro-expressions is a great challenge. In this paper, we propose a simple yet effective Main Directional Mean Optical-flow (MDMO) feature for micro-expression recognition. We apply a robust optical flow method on micro-expression video clips and partition the facial area into regions of interest (ROIs) based partially on action units. The MDMO is a ROI-based, normalized statistic feature that considers both local statistic motion information and its spatial location. One of the significant characteristics of MDMO is that its feature dimension is small. The length of a MDMO feature vector is @math , where @math is the number of ROIs. Furthermore, to reduce the influence of noise due to head movements, we propose an optical-flow-driven method to align all frames of a micro-expression video clip. Finally, a SVM classifier with the proposed MDMO feature is adopted for micro-expression recognition. Experimental results on three spontaneous micro-expression databases, namely SMIC, CASME and CASME II, show that the MDMO can achieve better performance than two state-of-the-art baseline features, i.e., LBP-TOP and HOOF. Recently, there are increasing interests in inferring mirco-expression from facial image sequences. Due to subtle facial movement of micro-expressions, feature extraction has become an important and critical issue for spontaneous facial micro-expression recognition. Recent works usually used spatiotemporal local binary pattern for micro-expression analysis. However, the commonly used spatiotemporal local binary pattern considers dynamic texture information to represent face images while misses the shape attribute of face images. On the other hand, their works extracted the spatiotemporal features from the global face regions, which ignore the discriminative information between two micro-expression classes. The above-mentioned problems seriously limit the application of spatiotemporal local binary pattern on micro-expression recognition. In this paper, we propose a discriminative spatiotemporal local binary pattern based on an improved integral projection to resolve the problems of spatiotemporal local binary pattern for micro-expression recognition. Firstly, we develop an improved integral projection for preserving the shape attribute of micro-expressions. Furthermore, an improved integral projection is incorporated with local binary pattern operators across spatial and temporal domains. Specifically, we extract the novel spatiotemporal features incorporating shape attributes into spatiotemporal texture features. For increasing the discrimination of micro-expressions, we propose a new feature selection based on Laplacian method to extract the discriminative information for facial micro-expression recognition. Intensive experiments are conducted on three availably published micro-expression databases. We compare our method with the state-of-the-art algorithms. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.