Facial expressions are combinations of basic components called Action Units (AU). Recognizing AUs is key for developing general facial expression analysis. In recent years, most efforts in automatic AU recognition have been dedicated to learning combinations of local features and to exploiting correlations between Action Units. In this paper, we propose a deep neural architecture that tackles both problems by combining learned local and global features in its initial stages and replicating a message passing algorithm between classes similar to a graphical model inference approach in later stages. We show that by training the model end-to-end with increased supervision we improve state-of-the-art by 5.3 and 8.2 performance on BP4D and DISFA datasets, respectively.
@cite_2 proposed a multi-task approach to learn a common kernel representation that describes AU correlations. @cite_30 adopted a latent variable Conditional Random Field (CRF) to jointly detect multiple AUs from predesigned features. While existing methods capture local pairwise AU dependencies, @cite_12 proposed a restricted Boltzmann machine that captures higher-order AU interactions. Together with patch-learning, @cite_23 used positive and negative competitions among AUs to model a discriminative multi-label classifier. @cite_6 placed a CRF on top of deep representations learned by a CNN. Both components are trained iteratively to estimate AU intensity. @cite_1 used a Restricted Boltzman Machine that captures joint probabilities between facial landmark locations and AUs. More recently, @cite_8 proposed a loss combining the recognition of isolated and groups of AUs.
@cite_30 @cite_8 @cite_1 @cite_6 @cite_23 @cite_2 @cite_12
2964088293 2609450230 2606784655 2780309588
Recent works have shown that exploiting multi-scale representations deeply learned via convolutional neural networks (CNN) is of tremendous importance for accurate contour detection. This paper presents a novel approach for predicting contours which advances the state of the art in two fundamental aspects, i.e. multi-scale feature generation and fusion. Different from previous works directly considering multi-scale feature maps obtained from the inner layers of a primary CNN architecture, we introduce a hierarchical deep model which produces more rich and complementary representations. Furthermore, to refine and robustly fuse the representations learned at different scales, the novel Attention-Gated Conditional Random Fields (AG-CRFs) are proposed. The experiments ran on two publicly available datasets (BSDS500 and NYUDv2) demonstrate the effectiveness of the latent AG-CRF model and of the overall hierarchical framework. Although both feature dependencies and label dependencies are crucial for facial action unit (AU) recognition, little work addresses them simultaneously till now. To address this limitation, we propose a 4-layer Restrict Boltzmann Machine (RBM) to simultaneously capture feature level and AU level dependencies to recognize multiple AUs. Specifically, the bottom two layers of the RBM model capture dependencies among image features, while the top two layers capture the high order dependencies among AU labels. An efficient learning algorithm is introduced to jointly learn all layers to leverage the interactions among different layers. Experiments on two benchmark databases demonstrate the effectiveness of the proposed approach in modelling complex AU relationships from both features and labels jointly, and its improved performance over the existing methods. We consider the task of automated estimation of facial expression intensity. This involves estimation of multiple output variables (facial action units --- AUs) that are structurally dependent. Their structure arises from statistically induced co-occurrence patterns of AU intensity levels. Modeling this structure is critical for improving the estimation performance; however, this performance is bounded by the quality of the input features extracted from face images. The goal of this paper is to model these structures and estimate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning. To this end, we propose a novel Copula CNN deep learning approach for modeling multivariate ordinal variables. Our model accounts for @math structure in output variables and their @math - @math dependencies via copula functions modeled as cliques of a CRF. These are jointly optimized with deep CNN feature encoding layers using a newly introduced balanced batch iterative training algorithm. We demonstrate the effectiveness of our approach on the task of AU intensity estimation on two benchmark datasets. We show that joint learning of the deep features and the target output structure results in significant performance gains compared to existing deep structured models for analysis of facial expressions. Most previous algorithms for the recognition of Action Units (AUs) were trained on a small number of sample images. This was due to the limited amount of labeled data available at the time. This meant that data-hungry deep neural networks, which have shown their potential in other computer vision problems, could not be successfully trained to detect AUs. A recent publicly available database with close to a million labeled images has made this training possible. Image and individual variability (e.g., pose, scale, illumination, ethnicity) in this set is very large. Unfortunately, the labels in this dataset are not perfect (i.e., they are noisy), making convergence of deep nets difficult. To harness the richness of this dataset while being robust to the inaccuracies of the labels, we derive a novel global-local loss. This new loss function is shown to yield fast globally meaningful convergences and locally accurate results. Comparative results with those of the EmotioNet challenge demonstrate that our newly derived loss yields superior recognition of AUs than state-of-the-art algorithms.