Automated deception detection (ADD) from real-life videos is a challenging task. It specifically needs to address two problems: (1) Both face and body contain useful cues regarding whether a subject is deceptive. How to effectively fuse the two is thus key to the effectiveness of an ADD model. (2) Real-life deceptive samples are hard to collect; learning with limited training data thus challenges most deep learning based ADD models. In this work, both problems are addressed. Specifically, for face-body multimodal learning, a novel face-focused cross-stream network (FFCSN) is proposed. It differs significantly from the popular two-stream networks in that: (a) face detection is added into the spatial stream to capture the facial expressions explicitly, and (b) correlation learning is performed across the spatial and temporal streams for joint deep feature learning across both face and body. To address the training data scarcity problem, our FFCSN model is trained with both meta learning and adversarial learning. Extensive experiments show that our FFCSN model achieves state-of-the-art results. Further, the proposed FFCSN model as well as its robust training strategy are shown to be generally applicable to other human-centric video analysis tasks such as emotion recognition from user-generated videos.
. Deception detection is closely related to emotion recognition: deception could be considered as a specific emotion state of humans, albeit it is much more subtle and harder to detect than others such as happy and angry. Although emotion recognition with still face images has been well studied in previous works, emotion recognition from user-generated videos @cite_17 is still a challenging problem. In particular, because of the complicated and unstructured nature of user-generated videos and the sparsity of video frames that express the emotion content, it is often hard to understand emotions conveyed in user-generated videos. To address this challenging problem, multi-modal fusion and knowledge transfer approaches have been proposed in recent works @cite_40 @cite_2 @cite_42 @cite_5 . In this paper, we show that our FFCSN model can be easily extended to emotion recognition from user-generated videos, with state-of-the-art results achieved.
@cite_42 @cite_40 @cite_2 @cite_5 @cite_17
2571743746 565148957 2251550834 2773980160
Deception is a very common phenomenon and its detection can be beneficial to our daily lives. Compared with other deception cues, micro-expression has shown great potential as a promising cue for deception detection. The spotting and recognition of micro-expression from long videos may significantly aid both law enforcement officers and researchers. However, database that contains both micro-expression and macro-expression in long videos is still not publicly available. To facilitate development in this field, we present a new database, Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME) @math ), which provides both macro-expressions and micro-expressions in two parts (A and B). Part A contains 87 long videos that contain spontaneous macro-expressions and micro-expressions. Part B includes 300 cropped spontaneous macro-expression samples and 57 micro-expression samples. The emotion labels are based on a combination of action units (AUs), self-reported emotion for every facial movement, and the emotion types of emotion-evoking videos. Local Binary Pattern (LBP) was employed for the spotting and recognition of macro-expressions and micro-expressions and the results were reported as a baseline evaluation. The CAS(ME) @math database offers both long videos and cropped expression samples, which may aid researchers in developing efficient algorithms for the spotting and recognition of macro-expressions and micro-expressions. One of important cues of deception detection is micro-expression. It has three characteristics: short duration, low intensity and usually local movements. These characteristics imply that micro-expression is sparse. In this paper, we use the sparse part of Robust PCA (RPCA) to extract the subtle motion information of micro-expression. The local texture features of the information are extracted by Local Spatiotemporal Directional Features (LSTD). In order to extract more effective local features, 16 Regions of Interest (ROIs) are assigned based on the Facial Action Coding System (FACS). The experimental results on two micro-expression databases show the proposed method gain better performance. Moreover, the proposed method may further be used to extract other subtle motion information (such as lip-reading, the human pulse, and micro-gesture etc.) from video. Deception detection has been receiving an increasing amount of attention from the computational linguistics, speech, and multimodal processing communities. One of the major challenges encountered in this task is the availability of data, and most of the research work to date has been conducted on acted or artificially collected data. The generated deception models are thus lacking real-world evidence. In this paper, we explore the use of multimodal real-life data for the task of deception detection. We develop a new deception dataset consisting of videos from reallife scenarios, and build deception tools relying on verbal and nonverbal features. We achieve classification accuracies in the range of 77-82 when using a model that extracts and fuses features from the linguistic and visual modalities. We show that these results outperform the human capability of identifying deceit. We present a system for covert automated deception detection in real-life courtroom trial videos. We study the importance of different modalities like vision, audio and text for this task. On the vision side, our system uses classifiers trained on low level video features which predict human micro-expressions. We show that predictions of high-level micro-expressions can be used as features for deception prediction. Surprisingly, IDT (Improved Dense Trajectory) features which have been widely used for action recognition, are also very good at predicting deception in videos. We fuse the score of classifiers trained on IDT features and high-level micro-expressions to improve performance. MFCC (Mel-frequency Cepstral Coefficients) features from the audio domain also provide a significant boost in performance, while information from transcripts is not very beneficial for our system. Using various classifiers, our automated system obtains an AUC of 0.877 (10-fold cross-validation) when evaluated on subjects which were not part of the training set. Even though state-of-the-art methods use human annotations of micro-expressions for deception detection, our fully automated approach outperforms them by 5 . When combined with human annotations of micro-expressions, our AUC improves to 0.922. We also present results of a user-study to analyze how well do average humans perform on this task, what modalities they use for deception detection and how they perform if only one modality is accessible. Our project page can be found at this https URL .