In this paper, we proposed a novel Probabilistic Attribute Tree-CNN (PAT-CNN) to explicitly deal with the large intra-class variations caused by identity-related attributes, e.g., age, race, and gender. Specifically, a novel PAT module with an associated PAT loss was proposed to learn features in a hierarchical tree structure organized according to attributes, where the final features are less affected by the attributes. Then, expression-related features are extracted from leaf nodes. Samples are probabilistically assigned to tree nodes at different levels such that expression-related features can be learned from all samples weighted by probabilities. We further proposed a semi-supervised strategy to learn the PAT-CNN from limited attribute-annotated samples to make the best use of available data. Experimental results on five facial expression datasets have demonstrated that the proposed PAT-CNN outperforms the baseline models by explicitly modeling attributes. More impressively, the PAT-CNN using a single model achieves the best performance for faces in the wild on the SFEW dataset, compared with the state-of-the-art methods using an ensemble of hundreds of CNNs.
Facial expression recognition has been extensively studied as elaborated in the recent surveys @cite_2 @cite_7 . One of the major steps in facial expression recognition is to extract features that capture the appearance and geometry changes caused by facial behavior, from either static images or dynamic sequences. These features can be roughly divided into two main categories: hand-crafted and learned features. Recently, features learned by deep CNNs have achieved promising results, especially in more challenging settings. Most of these approaches were trained from all training data, whereas attribute-related and expression-related facial appearances are intertwined in the learned features. While progress has been achieved in choices of features and classifiers, the challenge posed by subject variations remains for person-independent recognition.
@cite_7 @cite_2
2811322499 2281723898 1996958556 2104563967
Facial expression recognition is a topical task. However, very little research investigates subtle expression recognition, which is important for mental activity analysis, deception detection, etc. We address subtle expression recognition through convolutional neural networks (CNNs) by developing multi-task learning (MTL) methods to effectively leverage a side task: facial landmark detection. Existing MTL methods follow a design pattern of shared bottom CNN layers and task-specific top layers. However, the sharing architecture is usually heuristically chosen, as it is difficult to decide which layers should be shared. Our approach is composed of (1) a novel MTL framework that automatically learns which layers to share through optimisation under tensor trace norm regularisation and (2) an invariant representation learning approach that allows the CNN to leverage tasks defined on disjoint datasets without suffering from dataset distribution shift. To advance subtle expression recognition, we contribute a Large-scale Subtle Emotions and Mental States in the Wild database (LSEMSW). LSEMSW includes a variety of cognitive states as well as basic emotions. It contains 176K images, manually annotated with 13 emotions, and thus provides the first subtle expression dataset large enough for training deep CNNs. Evaluations on LSEMSW and 300-W (landmark) databases show the effectiveness of the proposed methods. In addition, we investigate transferring knowledge learned from LSEMSW database to traditional (non-subtle) expression recognition. We achieve very competitive performance on Oulu-Casia NIR&Vis and CK+ databases via transfer learning. A key issue regarding feature extraction is the capability of a technique to extract distinctive features to represent facial expressions while requiring a low computational complexity. In this study, the authors propose a novel approach for appearance-based facial feature extraction to perform the task of facial expression recognition on video sequences. The proposed spatiotemporal texture map (STTM) is capable of capturing subtle spatial and temporal variations of facial expressions with low computational complexity. First, face is detected using Viola–Jones face detector and frames are cropped to remove unnecessary background. Facial features are then modelled with the proposed STTM, which uses the spatiotemporal information extracted from three-dimensional Harris corner function. A block-based method is adopted to extract the dynamic features and represent the features in the form of histograms. The features are then classified into classes of emotion by the support vector machine classifier. The experimental results demonstrate that the proposed approach shows superior performance compared with the state-of-the-art approaches with an average recognition rate of 95.37, 98.56, and 84.52 on datasets containing posed expressions, spontaneous micro-expressions, and close-to-real-world expressions, respectively. They also show that the proposed algorithm requires low computational cost. Facial expressions are widely used in the behavioral interpretation of emotions, cognitive science, and social interactions. In this paper, we present a novel method for fully automatic facial expression recognition in facial image sequences. As the facial expression evolves over time facial landmarks are automatically tracked in consecutive video frames, using displacements based on elastic bunch graph matching displacement estimation. Feature vectors from individual landmarks, as well as pairs of landmarks tracking results are extracted, and normalized, with respect to the first frame in the sequence. The prototypical expression sequence for each class of facial expression is formed, by taking the median of the landmark tracking results from the training facial expression sequences. Multi-class AdaBoost with dynamic time warping similarity distance between the feature vector of input facial expression and prototypical facial expression, is used as a weak classifier to select the subset of discriminative feature vectors. Finally, two methods for facial expression recognition are presented, either by using multi-class AdaBoost with dynamic time warping, or by using support vector machine on the boosted feature vectors. The results on the Cohn-Kanade (CK+) facial expression database show a recognition accuracy of 95.17 and 97.35 using multi-class AdaBoost and support vector machines, respectively. Images of facial expressions are often captured from various views as a result of either head movements or variable camera position. Existing methods for multiview and or view-invariant facial expression recognition typically perform classification of the observed expression using either classifiers learned separately for each view or a single classifier learned for all views. However, these approaches ignore the fact that different views of a facial expression are just different manifestations of the same facial expression. By accounting for this redundancy, we can design more effective classifiers for the target task. To this end, we propose a discriminative shared Gaussian process latent variable model (DS-GPLVM) for multiview and view-invariant classification of facial expressions from multiple views. In this model, we first learn a discriminative manifold shared by multiple views of a facial expression. Subsequently, we perform facial expression classification in the expression manifold. Finally, classification of an observed facial expression is carried out either in the view-invariant manner (using only a single view of the expression) or in the multiview manner (using multiple views of the expression). The proposed model can also be used to perform fusion of different facial features in a principled manner. We validate the proposed DS-GPLVM on both posed and spontaneously displayed facial expressions from three publicly available datasets (MultiPIE, labeled face parts in the wild, and static facial expressions in the wild). We show that this model outperforms the state-of-the-art methods for multiview and view-invariant facial expression classification, and several state-of-the-art methods for multiview learning and feature fusion.