Smile is one of the key elements in identifying emotions and present state of mind of an individual. In this work, we propose a cluster of approaches to classify posed and spontaneous smiles using deep convolutional neural network (CNN) face features, local phase quantization (LPQ), dense optical flow and histogram of gradient (HOG). Eulerian Video Magnification (EVM) is used for micro-expression smile amplification along with three normalization procedures for distinguishing posed and spontaneous smiles. Although the deep CNN face model is trained with large number of face images, HOG features outperforms this model for overall face smile classification task. Using EVM to amplify micro-expressions did not have a significant impact on classification accuracy, while the normalizing facial features improved classification accuracy. Unlike many manual or semi-automatic methodologies, our approach aims to automatically classify all smiles into either spontaneous' or posed' categories, by using support vector machines (SVM). Experimental results on large UvA-NEMO smile database show promising results as compared to other relevant methods.
Recently, more attention has been paid to dynamical properties of smiles such as the duration, amplitude, speed, and acceleration instead of static features like smile symmetry or the AUs. To analyze these properties, the smile is generally broken up into three different phases - onset, apex, and offset. Spontaneous smiles tend to have a smaller amplitude, a slower onset @cite_35 , and a shorter total duration @cite_40 . The eye region is analyzed as well - the eyebrow raise in posed smiles have a higher maximum speed, larger amplitude and shorter duration than spontaneous ones @cite_19 . Most techniques extract dynamic properties of smiles that are known to be important factors in classifying smiles @cite_30 @cite_38 . Apart from these properties of smiles, facial dynamics can reveal other useful information for the classification of smiles, such as the subject's age.
@cite_30 @cite_35 @cite_38 @cite_19 @cite_40
2156503193 2173219554 2014590766 2144697550
We investigated movement differences between deliberately posed and spontaneously occurring smiles and eyebrow raises during a videotaped interview that included a facial movement assessment. Using automated facial image analysis, we quantified lip corner and eyebrow movement during periods of visible smiles and eyebrow raises and compared facial movement within participants. As in an earlier study, maximum speed of movement onset was greater in deliberate smiles. Maximum speed and amplitude were greater and duration shorter in deliberate compared to spontaneous eyebrow raises. Asymmetry of movement did not differ within participants. Similar patterns contrasting deliberate and spontaneous movement in both smiles and eyebrow raises suggest a common pattern of signaling for spontaneous facial displays. Almost all work in automatic facial expression analysis has focused on recognition of prototypic expressions rather than dynamic changes in appearance over time. To investigate the relative contribution of dynamic features to expression recognition, we used automatic feature tracking to measure the relation between amplitude and duration of smile onsets in spontaneous and deliberate smiles of 81 young adults of Euro- and African-American background. Spontaneous smiles were of smaller amplitude and had a larger and more consistent relation between amplitude and duration than deliberate smiles. A linear discriminant classifier using timing and amplitude measures of smile onsets achieved a 93 recognition rate. Using timing measures alone, recognition rate declined only marginally to 89 . These findings suggest that by extracting and representing dynamic as well as morphological features, automatic facial expression analysis can begin to discriminate among the message values of morphologically similar expressions. We investigated the value of the Duchenne (D) smile as a spontaneous sign of felt enjoyment. Participants either smiled spontaneously in response to amusing material (spontaneous condition) or were instructed to pose a smile (deliberate condition). Similar amounts of D and non-Duchenne (ND) smiles were observed in these 2 conditions (Experiment 1). When subsets of these smiles were presented to other participants, they generally rated spontaneous and deliberate D and ND smiles differently. Moreover, they distinguished between D smiles of varying intensity within the spontaneous condition (Experiment 2). Such a differentiation was also made when seeing the upper or lower face only (Experiment 3), but was impaired for static compared with dynamic displays (Experiment 4). The predictive value of the D smile in these judgment studies was limited compared with other features such as asymmetry, apex duration, and nonpositive facial actions, and was only significant for ratings of the upper face and static displays. These findings raise doubts about the reliability and validity of the D smile and question the usefulness of facial descriptions in identifying true feelings of enjoyment. Smiling is an indispensable element of nonverbal social interaction. Besides, automatic distinction between spontaneous and posed expressions is important for visual analysis of social signals. Therefore, in this paper, we propose a method to distinguish between spontaneous and posed enjoyment smiles by using the dynamics of eyelid, cheek, and lip corner movements. The discriminative power of these movements, and the effect of different fusion levels are investigated on multiple databases. Our results improve the state-of-the-art. We also introduce the largest spontaneous posed enjoyment smile database collected to date, and report new empirical and conceptual findings on smile dynamics. The collected database consists of 1240 samples of 400 subjects. Moreover, it has the unique property of having an age range from 8 to 76 years. Large scale experiments on the new database indicate that eyelid dynamics are highly relevant for smile classification, and there are age-related differences in smile dynamics. Each smile is unique: one person surely smiles in different ways (e.g. closing opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single embedding is then exploited by a conditional recurrent network which generates a landmark embedding sequence conditioned on a specific expression (e.g. spontaneous smile). Next, the generated landmark embeddings are fed into a multi-mode recurrent landmark generator, producing a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. Our experimental results demonstrate the effectiveness of our CMM-Net in generating realistic videos of multiple smile expressions.