In this paper we present a non-invasive ambient intelligence framework for the semi-automatic analysis of non-verbal communication applied to the restorative justice field. In particular, we propose the use of computer vision and social signal processing technologies in real scenarios of Victim-Offender Mediations, applying feature extraction techniques to multi-modal audio-RGB-depth data. We compute a set of behavioral indicators that define communicative cues from the fields of psychology and observational methodology. We test our methodology on data captured in real world Victim-Offender Mediation sessions in Catalonia in collaboration with the regional government. We define the ground truth based on expert opinions when annotating the observed social responses. Using different state-of-the-art binary classification approaches, our system achieves recognition accuracies of 86 when predicting satisfaction, and 79 when predicting both agreement and receptivity. Applying a regression strategy, we obtain a mean deviation for the predictions between 0.5 and 0.7 in the range [1-5] for the computed social signals.
The participants involved in these conversational settings usually appear either sitting or with some parts of their body occluded. Therefore, from a computer vision point of view, it might be best just to focus only on the upper body regions. Then, visual feature extraction techniques can focus on the most significant sources of information coming from the region of interest, which might be the face or hands, for example. These regions provide discriminative behavioral information, or adaptors, which are movements, such as head scratching, indicative of attitude, anxiety level and self-confidence @cite_0 ; or beat gestures, which are small baton-like movements of the hands used to emphasize important parts of speech with respect to the larger discourse @cite_14 . However, as explained in @cite_36 @cite_7 , body posture is also found to be an important indicator of a person's emotional state. Additionally, another potential source of information is provided by facial expressions @cite_40 @cite_49 @cite_42 @cite_23 .
@cite_14 @cite_7 @cite_36 @cite_42 @cite_0 @cite_40 @cite_49 @cite_23
1995790706 2062024333 2121947860 2745980686
We present a novel human vision inspired framework that can recognize facial expressions very efficiently and accurately. We propose to computationally process small, salient region of the face to extract features as it happens in human vision. To determine which facial region(s) is perceptually salient for a particular expression, we conducted a psycho-visual experimental study with an eye-tracker. A novel feature space conducive for recognition task is proposed, which is created by extracting Pyramid Histogram of Orientation Gradients features only from the salient facial regions. By processing only salient regions, proposed framework achieved two goals: (a) reduction in computational time for feature extraction (b) reduction in feature vector dimensionality. The proposed framework achieved automatic expression recognition accuracy of 95.3 on extended Cohn-Kanade (CK+) facial expression database for six universal facial expressions. Summary In highly social species such as humans, faces have evolved to convey rich information for social interaction, including expressions of emotions and pain [1–3]. Two motor pathways control facial movement [4–7]: a subcortical extrapyramidal motor system drives spontaneous facial expressions of felt emotions, and a cortical pyramidal motor system controls voluntary facial expressions. The pyramidal system enables humans to simulate facial expressions of emotions not actually experienced. Their simulation is so successful that they can deceive most observers [8–11]. However, machine vision may be able to distinguish deceptive facial signals from genuine facial signals by identifying the subtle differences between pyramidally and extrapyramidally driven movements. Here, we show that human observers could not discriminate real expressions of pain from faked expressions of pain better than chance, and after training human observers, we improved accuracy to a modest 55 . However, a computer vision system that automatically measures facial movements and performs pattern recognition on those movements attained 85 accuracy. The machine system's superiority is attributable to its ability to differentiate the dynamics of genuine expressions from faked expressions. Thus, by revealing the dynamics of facial action through machine vision systems, our approach has the potential to elucidate behavioral fingerprints of neural control systems involved in emotional signaling. Previous studies of eye gaze have shown that when looking at images containing human faces, observers tend to rapidly focus on the facial regions. But is this true of other high-level image features as well? We here investigate the extent to which natural scenes containing faces, text elements, and cell phones - as a suitable control - attract attention by tracking the eye movements of subjects in two types of tasks - free viewing and search. We observed that subjects in free-viewing conditions look at faces and text 16.6 and 11.1 times more than similar regions normalized for size and position of the face and text. In terms of attracting gaze, text is almost as effective as faces. Furthermore, it is difﬁcult to avoid looking at faces and text even when doing so imposes a cost. We also found that subjects took longer in making their initial saccade when they were told to avoid faces text and their saccades landed on a non-face non-text object. We reﬁne a well-known bottom–up computer model of saliency-driven attention that includes conspicuity maps for color, orientation, and intensity by adding high-level semantic information (i.e., the location of faces or text) and demonstrate that this signiﬁcantly improves the ability to predict eye ﬁxations in natural images. Our enhanced model’s predictions yield an area under the ROC curve over 84 for images that contain faces or text when compared against the actual ﬁxation pattern of subjects. This suggests that the primate visual system allocates attention using such an enhanced saliency map. The face conveys a blend of verbal and nonverbal information playing an important role in daily interaction. While speech articulation mostly affects the orofacial areas, emotional behaviors are externalized across the entire face. Considering the relation between verbal and nonverbal behaviors is important to create naturalistic facial movements for conversational agents (CAs). Furthermore, facial muscles connect areas across the face, creating principled relationships and dependencies between the movements that have to be taken into account. These relationships are ignored when facial movements across the face are separately generated. This paper proposes to create speech-driven models that jointly capture the relationship not only between speech and facial movements, but also across facial movements. The input to the models are features extracted from speech that convey the verbal and emotional states of the speakers. We build our models with bidirectional long-short term memory (BLSTM) units which are shown to be very successful in modeling dependencies for sequential data. The objective and subjective evaluations of the results demonstrate the benefits of joint modeling of facial regions using this framework.