Smile is one of the key elements in identifying emotions and present state of mind of an individual. In this work, we propose a cluster of approaches to classify posed and spontaneous smiles using deep convolutional neural network (CNN) face features, local phase quantization (LPQ), dense optical flow and histogram of gradient (HOG). Eulerian Video Magnification (EVM) is used for micro-expression smile amplification along with three normalization procedures for distinguishing posed and spontaneous smiles. Although the deep CNN face model is trained with large number of face images, HOG features outperforms this model for overall face smile classification task. Using EVM to amplify micro-expressions did not have a significant impact on classification accuracy, while the normalizing facial features improved classification accuracy. Unlike many manual or semi-automatic methodologies, our approach aims to automatically classify all smiles into either spontaneous' or posed' categories, by using support vector machines (SVM). Experimental results on large UvA-NEMO smile database show promising results as compared to other relevant methods.
Recently, more attention has been paid to dynamical properties of smiles such as the duration, amplitude, speed, and acceleration instead of static features like smile symmetry or the AUs. To analyze these properties, the smile is generally broken up into three different phases - onset, apex, and offset. Spontaneous smiles tend to have a smaller amplitude, a slower onset @cite_35 , and a shorter total duration @cite_40 . The eye region is analyzed as well - the eyebrow raise in posed smiles have a higher maximum speed, larger amplitude and shorter duration than spontaneous ones @cite_19 . Most techniques extract dynamic properties of smiles that are known to be important factors in classifying smiles @cite_30 @cite_38 . Apart from these properties of smiles, facial dynamics can reveal other useful information for the classification of smiles, such as the subject's age.
@cite_30 @cite_35 @cite_38 @cite_19 @cite_40
2156503193 2173219554 2014590766 2144697550
We investigated movement differences between deliberately posed and spontaneously occurring smiles and eyebrow raises during a videotaped interview that included a facial movement assessment. Using automated facial image analysis, we quantified lip corner and eyebrow movement during periods of visible smiles and eyebrow raises and compared facial movement within participants. As in an earlier study, maximum speed of movement onset was greater in deliberate smiles. Maximum speed and amplitude were greater and duration shorter in deliberate compared to spontaneous eyebrow raises. Asymmetry of movement did not differ within participants. Similar patterns contrasting deliberate and spontaneous movement in both smiles and eyebrow raises suggest a common pattern of signaling for spontaneous facial displays. Almost all work in automatic facial expression analysis has focused on recognition of prototypic expressions rather than dynamic changes in appearance over time. To investigate the relative contribution of dynamic features to expression recognition, we used automatic feature tracking to measure the relation between amplitude and duration of smile onsets in spontaneous and deliberate smiles of 81 young adults of Euro- and African-American background. Spontaneous smiles were of smaller amplitude and had a larger and more consistent relation between amplitude and duration than deliberate smiles. A linear discriminant classifier using timing and amplitude measures of smile onsets achieved a 93 recognition rate. Using timing measures alone, recognition rate declined only marginally to 89 . These findings suggest that by extracting and representing dynamic as well as morphological features, automatic facial expression analysis can begin to discriminate among the message values of morphologically similar expressions. We investigated the value of the Duchenne (D) smile as a spontaneous sign of felt enjoyment. Participants either smiled spontaneously in response to amusing material (spontaneous condition) or were instructed to pose a smile (deliberate condition). Similar amounts of D and non-Duchenne (ND) smiles were observed in these 2 conditions (Experiment 1). When subsets of these smiles were presented to other participants, they generally rated spontaneous and deliberate D and ND smiles differently. Moreover, they distinguished between D smiles of varying intensity within the spontaneous condition (Experiment 2). Such a differentiation was also made when seeing the upper or lower face only (Experiment 3), but was impaired for static compared with dynamic displays (Experiment 4). The predictive value of the D smile in these judgment studies was limited compared with other features such as asymmetry, apex duration, and nonpositive facial actions, and was only significant for ratings of the upper face and static displays. These findings raise doubts about the reliability and validity of the D smile and question the usefulness of facial descriptions in identifying true feelings of enjoyment. Smiling is an indispensable element of nonverbal social interaction. Besides, automatic distinction between spontaneous and posed expressions is important for visual analysis of social signals. Therefore, in this paper, we propose a method to distinguish between spontaneous and posed enjoyment smiles by using the dynamics of eyelid, cheek, and lip corner movements. The discriminative power of these movements, and the effect of different fusion levels are investigated on multiple databases. Our results improve the state-of-the-art. We also introduce the largest spontaneous posed enjoyment smile database collected to date, and report new empirical and conceptual findings on smile dynamics. The collected database consists of 1240 samples of 400 subjects. Moreover, it has the unique property of having an age range from 8 to 76 years. Large scale experiments on the new database indicate that eyelid dynamics are highly relevant for smile classification, and there are age-related differences in smile dynamics. Each smile is unique: one person surely smiles in different ways (e.g. closing opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single embedding is then exploited by a conditional recurrent network which generates a landmark embedding sequence conditioned on a specific expression (e.g. spontaneous smile). Next, the generated landmark embeddings are fed into a multi-mode recurrent landmark generator, producing a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. Our experimental results demonstrate the effectiveness of our CMM-Net in generating realistic videos of multiple smile expressions. Recognizing smiles is of much importance for detecting happy moods. Gabor features are conventionally widely applied to facial expression recognition, but the number of Gabor features is usually too large. We proposed to use Pyramid Histogram of Oriented Gradients (PHOG) as the features extracted for smile recognition in this paper. The comparisons between the PHOG and Gabor features using a publicly available dataset demonstrated that the PHOG with a significantly shorter vector length could achieve as high a recognition rate as the Gabor features did. Furthermore, the feature selection conducted by an AdaBoost algorithm was not needed when using the PHOG features. To further improve the recognition performance, we combined these two feature extraction methods and achieved the best smile recognition rate, indicating a good value of the PHOG features for smile recognitions. Kinship verification from facial appearance is a difficult problem. This paper explores the possibility of employing facial expression dynamics in this problem. By using features that describe facial dynamics and spatio-temporal appearance over smile expressions, we show that it is possible to improve the state of the art in this problem, and verify that it is indeed possible to recognize kinship by resemblance of facial expressions. The proposed method is tested on different kin relationships. On the average, 72.89 verification accuracy is achieved on spontaneous smiles. Autism spectrum disorder (ASD) is a neurodevelopmental disability with atypical traits in behavioral and physiological responses. These atypical traits in individuals with ASD may be too subtle and subjective to measure visually using tedious methods of scoring. Alternatively, the use of intrusive sensors in the measurement of psychophysical responses in individuals with ASD may likely cause inhibition and bias. This paper proposes a novel experimental protocol for non-intrusive sensing and analysis of facial expression, visual scanning, and eye-hand coordination to investigate behavioral markers for ASD. An institutional review board approved pilot study is conducted to collect the response data from two groups of subjects (ASD and control) while they engage in the tasks of visualization, recognition, and manipulation. For the first time in the ASD literature, the facial action coding system is used to classify spontaneous facial responses. Statistical analyses reveal significantly (p <0.01) higher prevalence of smile expression for the group with ASD with the eye-gaze significantly averted (p<0.05) from viewing the face in the visual stimuli. This uncontrolled manifestation of smile without proper visual engagement suggests impairment in reciprocal social communication, e.g., social smile. The group with ASD also reveals poor correlation in eye-gaze and hand movement data suggesting deficits in motor coordination while performing a dynamic manipulation task. The simultaneous sensing and analysis of multimodal response data may provide useful quantitative insights into ASD to facilitate early detection of symptoms for effective intervention planning. Reliable detection of ordinary facial expressions (e.g. smile) despite the variability among individuals as well as face appearance is an important step toward the realization of perceptual user interface with autonomous perception of persons. We describe a rule-based algorithm for robust facial expression recognition combined with robust face detection using a convolutional neural network. In this study, we address the problem of subject independence as well as translation, rotation, and scale invariance in the recognition of facial expression. The result shows reliable detection of smiles with recognition rate of 97.6 for 5600 still images of more than 10 subjects. The proposed algorithm demonstrated the ability to discriminate smiling from talking based on the saliency score obtained from voting visual cues. To the best of our knowledge, it is the first facial expression recognition model with the property of subject independence combined with robustness to variability in facial appearance. We introduce a simple framework for identifying biases of a smiling attribute classifier. Our method poses counterfactual questions of the form: how would the prediction change if this face characteristic had been different? We leverage recent advances in generative adversarial networks to build a realistic generative model of face images that affords controlled manipulation of specific image characteristics. We introduce a set of metrics that measure the effect of manipulating a specific property of an image on the output of a trained classifier. Empirically, we identify several different factors of variation that affect the predictions of a smiling classifier trained on CelebA. Asymmetries of the smiling facial movement were more frequent in deliberate imitations than spontaneous emotional expressions. When asymmetries did occur they were usually stronger on the left side of the face if the smile was deliberate. Asymmetrical emotional expressions, however, were about equally divided between those stronger on the left side of the face and those stronger on the right. Similar findings were obtained for the actions involved in negative emotions, but a small data base made these results tentative. Smile is not only a visual expression. When it occurs together with speech, it also alters its acoustic realization. Being able to synthesize speech altered by the expression of smile can hence be an important contributor for adding naturalness and expressiveness in interactive systems. In this work, we present a first attempt to develop a Hidden Markov Model (HMM)-based synthesis system allowing to control the degree of smile in speech. It relies on a model interpolation technique, enabling speech-smile sentences with various smiling intensities to be generated. Sentences synthesized using this approach have been evaluated through a perceptual test. Encouraging results are reported here. Based on the assumptions that relational messages are multidimensional and that they are frequently communicated by nonverbal cues, this experiment manipulated five nonverbal cues -eye contact, proximity, body lean, smiling, and touch - to determine what meanings they convey along four relational message dimensions. Subjects (N= 150) observed 2 out of 40 videotaped conversational segments in which a male-female dyad presented various combinations of the nonverbal cues. High eye contact, close proximity, forward body lean, and smiling all conveyed greater intimacy, attraction, and trust. Low eye contact, a distal position, backward body lean, and the absence of smiling and touch communicated greater detachment. High eye contact, close proximity, and smiling also communicated less emotional arousal and greater composure, while high eye contact and close proximity alone conveyed greater dominance and control. Effects of combinations of cues and sex-differences are also reported. In this paper we present the AmuS database of about three hours worth of data related to amused speech recorded from two males and one female subjects and contains data in two languages French and English. We review previous work on smiled speech and speech-laughs. We describe acoustic analysis on part of our database, and a perception test comparing speech-laughs with smiled and neutral speech. We show the efficiency of the data in AmuS for synthesis of amused speech by training HMM-based models for neutral and smiled speech for each voice and comparing them using an on-line CMOS test. We study the problem of facial analysis in videos. We propose a novel weakly supervised learning method that models the video event (expression, pain etc.) as a sequence of automatically mined, discriminative sub-events (eg. onset and offset phase for smile, brow lower and cheek raise for pain). The proposed model is inspired by the recent works on Multiple Instance Learning and latent SVM HCRF- it extends such frameworks to model the ordinal or temporal aspect in the videos, approximately. We obtain consistent improvements over relevant competitive baselines on four challenging and publicly available video based facial analysis datasets for prediction of expression, clinical pain and intent in dyadic conversations. In combination with complimentary features, we report state-of-the-art results on these datasets. The increasing adoption of social media provides unprecedented opportunities to gain insight into human nature at vastly broader scales. Regarding the study of population-wide sentiment, prior research commonly focuses on text-based analyses and ignores a treasure trove of sentiment-laden content: images. In this paper, we make methodological and computational contributions by introducing the Smile Index as a formalized measure of societ al happiness. Detecting smiles in 9 million geo-located tweets over 16 months, we validate our Smile Index against both text-based techniques and self-reported happiness. We further make observational contributions by applying our metric to explore temporal trends in sentiment, relate public mood to societ al events, and predict economic indicators. Reflecting upon the innate, language-independent aspects of facial expressions, we recommend future improvements and applications to enable robust, global-level analyses. We conclude with implications for researchers studying and facilitating the expression of collective emotion through socio-technical systems. Conventional mobile social services such as Loopt and Google Latitude rely on two classes of trusted relationships: participants trust a centralized server to manage their location information and trust between users is based on existing social relationships. Unfortunately, these assumptions are not secure or general enough for many mobile social scenarios: centralized servers cannot always be relied upon to preserve data confidentiality, and users may want to use mobile social services to establish new relationships. To address these shortcomings, this paper describes SMILE, a privacy-preserving "missed-connections" service in which the service provider is untrusted and users are not assumed to have pre-established social relationships with each other. At a high-level, SMILE uses short-range wireless communication and standard cryptographic primitives to mimic the behavior of users in existing missed-connections services such as Craigslist: trust is founded solely on anonymous users' ability to prove to each other that they shared an encounter in the past. We have evaluated SMILE using protocol analysis, an informal study of Craigslist usage, and experiments with a prototype implementation and found it to be both privacy-preserving and feasible. A central role in the development process of children is played by self-exploratory activities Through a playful interaction with the surrounding environment, they test their own capabilities, explore novel situations, and understand how their actions affect the world During this kind of exploration, interesting situations may be discovered By learning to reach these situations, a child incrementally develops more and more complex skills Inspired by studies from psychology, neuroscience, and machine learning, we designed SMILe (Self-Motivated Incremental Learning), a learning framework that allows artificial agents to autonomously identify and learn a set of abilities useful to face several different tasks, through an iterated three phase process: by means of a random exploration of the environment (babbling phase), the agent identifies interesting situations and generates an intrinsic motivation (motivating phase) aimed at learning how to get into these situations (skill acquisition phase) This process incrementally increases the skills of the agent, so that new interesting configurations can be experienced We present results on two gridworld environments to show how SMILe makes it possible to learn skills that enable the agent to perform well and robustly in many different tasks. We present a multi-purpose algorithm for simultaneousface detection, face alignment, pose estimation, genderrecognition, smile detection, age estimation and face recognitionusing a single deep convolutional neural network (CNN). Theproposed method employs a multi-task learning framework thatregularizes the shared parameters of CNN and builds a synergyamong different domains and tasks. Extensive experimentsshow that the network has a better understanding of face andachieves state-of-the-art result for most of these tasks We present a multi-purpose algorithm for simultaneous face detection, face alignment, pose estimation, gender recognition, smile detection, age estimation and face recognition using a single deep convolutional neural network (CNN). The proposed method employs a multi-task learning framework that regularizes the shared parameters of CNN and builds a synergy among different domains and tasks. Extensive experiments show that the network has a better understanding of face and achieves state-of-the-art result for most of these tasks.