In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative local motion patterns feature, with three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition. And, the third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variation and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-rank state-of-the-art methods for macro expression recognition.
According to @cite_0 , micro expressions are much more difficult to detect without temporal information. Thus, researchers use spatio-temporal features for micro expression analysis. @cite_49 extend LBP-TOP using optical strain information as a weighting function to find smaller motions. @cite_20 propose an extension of LBP-TOP based on the three intersecting lines crossing over the center point of the three histograms. They provide more compact and lightweight representation by minimizing the redundancy in LBP-TOP. @cite_22 propose a new spatio-temporal LBP on an improved integral projection combining the benefit of texture and shape.
@cite_0 @cite_20 @cite_22 @cite_49
2782644834 2076617402 2156489769 234798790
Micro-expressions are difficult to be observed by human beings due to its low intensity and short duration. Recently, several works have been developed to resolve the problems of micro-expression recognition caused by subtle intensity and short duration. One of them, Local binary pattern from three orthogonal planes (LBP-TOP) is primarily used to recognize micro-expression from the video recorded by high-speed camera. Several variances of LBP-TOP have also been developed to promisingly improve the performance of LBP-TOP for microexpression recognition. However, these variances of LBP-TOP including LBP-TOP cannot well extract the subtle movements of micro-expression so that they have the low performance. In this paper, we propose spontaneous local radon-based binary pattern to analyze micro-expressions with subtle facial movements. Firstly, it extracts the sparse information by using robust principal component analysis since micro-expression data are sparse in both temporal and spatial domains caused by short duration and low intensity. These sparse information can provide much motion information to dynamic feature descriptor. Furthermore, it employs radon transform to obtain the shape features from three orthogonal planes, as radon transform is robustness to the same histogram distribution of two images. Finally, one-dimensional LBP is employed in these shape features for constructing the spatiotemporal features for microexpression video. Intensive experiments are conducted on two available published micro-expression databases including SMIC and CASME2 databases for evaluating the performance of the proposed method. Experimental results demonstrate that the proposed method achieves promising performance in microexpression recognition. In this paper, we propose a new, compact, 4D spatio-temporal “Nebula” feature to improve expression and facial movement analysis performance. Given a spatio-temporal volume, the data is voxelized and fit to a cubic polynomial. A label is assigned based on the principal curvature values, and the polar angles of the direction of least curvature are computed. The labels and angles for each feature are used to build a histogram for each region of the face. The concatenated histograms from each region give us our final feature vector. This feature description is tested on the posed expression database BU-4DFE and on a new 4D spontaneous expression database. Various region configurations, histogram sizes, and feature parameters are tested, including a non-dynamic version of the approach. The LBP-TOP approach on the texture image as well as on the depth image is also tested for comparison. The onsets of the six canonical expressions are classified for 100 subjects in BU-4DFE, while the onset, offset, and non-existence of 12 Action Units (AUs) are classified for 16 subjects from our new spontaneous database. For posed expression recognition, the Nebula feature approach shows improvement over LBPTOP on the depth images and significant improvement over the non-dynamic 3D-only approach. Moreover, the Nebula feature performs better for AU classification than the compared approaches for 11 of the AUs tested in terms of accuracy as well as Area Under Receiver Operating Characteristic Curve (AUC). Spontaneous facial micro-expression analysis has become an active task for recognizing suppressed and involuntary facial expressions shown on the face of humans. Recently, Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) has been employed for micro-expression analysis. However, LBP-TOP suffers from two critical problems, causing a decrease in the performance of micro-expression analysis. It generally extracts appearance and motion features from the sign-based difference between two pixels but not yet considers other useful information. As well, LBP-TOP commonly uses classical pattern types which may be not optimal for local structure in some applications. This paper proposes SpatioTemporal Completed Local Quantization Patterns (STCLQP) for facial micro-expression analysis. Firstly, STCLQP extracts three interesting information containing sign, magnitude and orientation components. Secondly, an efficient vector quantization and codebook selection are developed for each component in appearance and temporal domains to learn compact and discriminative codebooks for generalizing classical pattern types. Finally, based on discriminative codebooks, spatiotemporal features of sign, magnitude and orientation components are extracted and fused. Experiments are conducted on three publicly available facial micro-expression databases. Some interesting findings about the neighboring patterns and the component analysis are concluded. Comparing with the state of the art, experimental results demonstrate that STCLQP achieves a substantial improvement for analyzing facial micro-expressions. HighlightsWe propose spatiotemporal completed local quantized pattern for micro-expression analysis.We propose to use three useful information, including the sign-based, magnitude-based and orientation-based difference of pixels for LBP.We propose to use an efficient vector quantization and discriminative codebook selection to make LBP-TOP more discriminative and compact.We evaluate the framework on three publicly available facial micro-expression databases.We evaluate the influence of parameters, different components and codebook selection to spatiotemporal completed local quantized pattern. Micro-expression recognition is still in the preliminary stage, owing much to the numerous difficulties faced in the development of datasets. Since micro-expression is an important affective clue for clinical diagnosis and deceit analysis, much effort has gone into the creation of these datasets for research purposes. There are currently two publicly available spontaneous micro-expression datasets—SMIC and CASME II, both with baseline results released using the widely used dynamic texture descriptor LBP-TOP for feature extraction. Although LBP-TOP is popular and widely used, it is still not compact enough. In this paper, we draw further inspiration from the concept of LBP-TOP that considers three orthogonal planes by proposing two efficient approaches for feature extraction. The compact robust form described by the proposed LBP-Six Intersection Points (SIP) and a super-compact LBP-Three Mean Orthogonal Planes (MOP) not only preserves the essential patterns, but also reduces the redundancy that affects the discriminality of the encoded features. Through a comprehensive set of experiments, we demonstrate the strengths of our approaches in terms of recognition accuracy and efficiency.