In this paper we present a non-invasive ambient intelligence framework for the semi-automatic analysis of non-verbal communication applied to the restorative justice field. In particular, we propose the use of computer vision and social signal processing technologies in real scenarios of Victim-Offender Mediations, applying feature extraction techniques to multi-modal audio-RGB-depth data. We compute a set of behavioral indicators that define communicative cues from the fields of psychology and observational methodology. We test our methodology on data captured in real world Victim-Offender Mediation sessions in Catalonia in collaboration with the regional government. We define the ground truth based on expert opinions when annotating the observed social responses. Using different state-of-the-art binary classification approaches, our system achieves recognition accuracies of 86 when predicting satisfaction, and 79 when predicting both agreement and receptivity. Applying a regression strategy, we obtain a mean deviation for the predictions between 0.5 and 0.7 in the range [1-5] for the computed social signals.
The participants involved in these conversational settings usually appear either sitting or with some parts of their body occluded. Therefore, from a computer vision point of view, it might be best just to focus only on the upper body regions. Then, visual feature extraction techniques can focus on the most significant sources of information coming from the region of interest, which might be the face or hands, for example. These regions provide discriminative behavioral information, or adaptors, which are movements, such as head scratching, indicative of attitude, anxiety level and self-confidence @cite_0 ; or beat gestures, which are small baton-like movements of the hands used to emphasize important parts of speech with respect to the larger discourse @cite_14 . However, as explained in @cite_36 @cite_7 , body posture is also found to be an important indicator of a person's emotional state. Additionally, another potential source of information is provided by facial expressions @cite_40 @cite_49 @cite_42 @cite_23 .
@cite_14 @cite_7 @cite_36 @cite_42 @cite_0 @cite_40 @cite_49 @cite_23
1708911126 2069462093 2123701509 2118163921 2139942200 2097128017 2138206939
Gesturing is such an integral yet unconscious part of communication that we are mostly oblivious to it. But if you observe anyone in conversation, you are likely to see his or her fingers, hands, and arms in some form of spontaneous motion. Why? David McNeill, a pioneer in the ongoing study of the relationship between gesture and language, set about answering this question in "Gesture and Thought" with an unlikely accomplice - Tweety Bird. McNeill argues that gestures are active participants in both speaking and thinking. He posits that gestures are key ingredients in an "imagery-language dialectic" that fuels speech and thought; gestures are the "imagery" and also the components of "language," rather than mere consequences. The smallest unit of this dialectic is the "growth point," a snapshot of an utterance at its beginning psychological stage. Enter Tweety Bird. In "Gesture and Thought", the central growth point comes from a cartoon. In his quest to eat Tweety Bird, Sylvester the cat first scales the outside of a rain gutter to reach his prey. Unsuccessful, he makes a second attempt by climbing up the inside of the gutter. Tweety, however, drops a bowling ball down the gutter; Sylvester swallows the ball. Over the course of twenty-five years, McNeill showed this cartoon to numerous subjects who spoke a variety of languages. A fascinating pattern emerged. Those who remembered the exact sequence of the cartoon while retelling it all used the same gesture to describe Sylvester's position inside the gutter. Those who forgot, in the retelling, that Sylvester had first climbed the outside of the gutter did not use this gesture at all. Thus that gesture becomes part of the "growth point" - the building block of language and thought. An ambitious project in the ongoing study of the relationship of how we communicate and its connection to thought, "Gesture and Thought" is a work of such consequence that it will influence all subsequent linguistic and evolutionary theory on the subject. Nonverbal communication plays an important role in many aspects of our lives, such as in job interviews, where vis-α-vis conversations take place. This paper proposes a method to automatically detect body communicative cues by using video sequences of the upper body of individuals in a conversational context. To our knowledge, our work brings novelty by explicitly addressing the recognition of visual activity in a seated, conversational setting from monocular video, compared to most existing work in video-based motion capture, which targets full-body with lower limb activities. We first detect the person hands in the sequence by searching for the higher speed parts along the whole video. Then, aided by training a set of typical conversational movements, we infer the approximate 3D upper body pose, that we transfer to a low-dimensionality space in order to perform action recognition. We test our system in the context of job interviews, with several new databases that we make publicly available. In this paper, we present a fully-automatic Spatio-Temporal GrabCut human segmentation methodology that combines tracking and segmentation. GrabCut initialization is performed by a HOG-based subject detection, face detection, and skin color model. Spatial information is included by Mean Shift clustering whereas temporal coherence is considered by the historical of Gaussian Mixture Models. Moreover, full face and pose recovery is obtained by combining human segmentation with Active Appearance Models and Conditional Random Fields. Results over public datasets and in a new Human Limb dataset show a robust segmentation and recovery of both face and pose using the presented methodology. What is the relation between gestures and speech? In terms of symbolic forms, of course, the spontaneous and unwitting gestures we make while talking differ sharply from spoken language itself. Whereas spoken language is linear, segmented, standardized, and arbitrary, gestures are global, synthetic, idiosyncratic, and imagistic. In Hand and Mind, David McNeill presents a bold theory of the essential unity of speech and the gestures that accompany it. This long-awaited, provocative study argues that the unity of gestures and language far exceeds the surface level of speech noted by previous researchers and in fact also includes the semantic and pragmatic levels of language. In effect, the whole concept of language must be altered to take into account the nonsegmented, instantaneous, and holistic images conveyed by gestures. McNeill and his colleagues carefully devised a standard methodology for examining the speech and gesture behavior of individuals engaged in narrative discourse. A research subject is shown a cartoon like the 1950 Canary Row--a classic Sylvester and Tweedy Bird caper that features Sylvester climbing up a downspout, swallowing a bowling ball and slamming into a brick wall. After watching the cartoon, the subject is videotaped recounting the story from memory to a listener who has not seen the cartoon. Painstaking analysis of the videotapes revealed that although the research subjects--children as well as adults, some neurologically impaired--represented a wide variety of linguistic groupings, the gestures of people speaking English and a half dozen other languages manifest the same principles. Relying on data from more than ten years of research, McNeill shows thatgestures do not simply form a part of what is said and meant but have an impact on thought itself. He persuasively argues that because gestures directly transfer mental images to visible forms, conveying ideas that language cannot always express, we must examine language and gesture This paper introduces social signal processing (SSP), the domain aimed at automatic understanding of social interactions through analysis of nonverbal behavior. The core idea of SSP is that nonverbal behavior is machine detectable evidence of social signals, the relational attitudes exchanged between interacting individuals. Social signals include (dis-)agreement, empathy, hostility, and any other attitude towards others that is expressed not only by words but by nonverbal behaviors such as facial expression and body posture as well. Thus, nonverbal behavior analysis is used as a key to automatic understanding of social interactions. This paper presents not only a survey of the related literature and the main concepts underlying SSP, but also an illustrative example of how such concepts are applied to the analysis of conflicts in competitive discussions. The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing. We propose a method for head-pose invariant facial expression recognition that is based on a set of characteristic facial points. To achieve head-pose invariance, we propose the Coupled Scaled Gaussian Process Regression (CSGPR) model for head-pose normalization. In this model, we first learn independently the mappings between the facial points in each pair of (discrete) nonfrontal poses and the frontal pose, and then perform their coupling in order to capture dependences between them. During inference, the outputs of the coupled functions from different poses are combined using a gating function, devised based on the head-pose estimation for the query points. The proposed model outperforms state-of-the-art regression-based approaches to head-pose normalization, 2D and 3D Point Distribution Models (PDMs), and Active Appearance Models (AAMs), especially in cases of unknown poses and imbalanced training data. To the best of our knowledge, the proposed method is the first one that is able to deal with expressive faces in the range from @math to @math pan rotation and @math to @math tilt rotation, and with continuous changes in head pose, despite the fact that training was conducted on a small set of discrete poses. We evaluate the proposed method on synthetic and real images depicting acted and spontaneously displayed facial expressions.