In this paper we present a non-invasive ambient intelligence framework for the semi-automatic analysis of non-verbal communication applied to the restorative justice field. In particular, we propose the use of computer vision and social signal processing technologies in real scenarios of Victim-Offender Mediations, applying feature extraction techniques to multi-modal audio-RGB-depth data. We compute a set of behavioral indicators that define communicative cues from the fields of psychology and observational methodology. We test our methodology on data captured in real world Victim-Offender Mediation sessions in Catalonia in collaboration with the regional government. We define the ground truth based on expert opinions when annotating the observed social responses. Using different state-of-the-art binary classification approaches, our system achieves recognition accuracies of 86 when predicting satisfaction, and 79 when predicting both agreement and receptivity. Applying a regression strategy, we obtain a mean deviation for the predictions between 0.5 and 0.7 in the range [1-5] for the computed social signals.
Once data from the environment have been acquired and processed to define a set of behavioral features, they serve as the basis for modelling a set of communication indicators. For instance, in @cite_25 , the authors outline a system for real-time tracking of the human body with the objective of interpreting human behavior. However, in the context of conversations we are particularly interested in behavioral traits belonging to social signals captured in the communication and in the interactions between the participants in the VOM sessions. In this sense, levels of agitation (or energy), activity, stress, and engagement are analyzed not only from their body movements, but also from their speech, facial expressions, or gaze directions, in order to predict behavioral responses.
@cite_25
1571434535 2077594926 2526567570 79159144
Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA , a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone , accelerometer , bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals’ personality as well as their position , head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at h ttp: tev.fbk.eu salsa. We studied the behavioral and emotional dynamics displayed by two people trying to resolve a conflict. 59 groups of two people were asked to talk for 20 minutes to try to reach a consensus about a topic on which they disagreed. The topics were abortion, affirmative action, death penalty, and euthanasia. Behavior data were determined from audio recordings where each second of the conversation was assessed as proself, neutral, or prosocial. We determined the probability density function of the durations of time spent in each behavioral state. These durations were well fit by a stretched exponential distribution, with an exponent, , of approximately 0.3. This indicates that the switching between behavioral states is not a random Markov process, but one where the probability to switch behavioral states decreases with the time already spent in that behavioral state. The degree of this “memory” was stronger in those groups who did not reach a consensus and where the conflict grew more destructive than in those that did. Emotion data were measured by having each person listen to the audio recording and moving a computer mouse to recall their negative or positive emotional valence at each moment in the conversation. We used the Hurst rescaled range analysis and power spectrum to determine the correlations in the fluctuations of the emotional valence. The emotional valence was well described by a random walk whose increments were uncorrelated. Thus, the behavior data demonstrated a “memory” of the duration already spent in a behavioral state while the emotion data fluctuated as a random walk whose steps did not have a “memory” of previous steps. This work demonstrates that statistical analysis, more commonly used to analyze physical phenomena, can also shed interesting light on the dynamics of processes in social psychology and conflict management. During face-to-face interactions, people naturally integrate nonverbal behaviors such as facial expressions and body postures as part of the conversation to infer the communicative intent or emotional state of their interlocutor. The interpretation of these nonverbal behaviors will often be contextualized by interactional cues such as the previous spoken question, the general discussion topic or the physical environment. A critical step in creating computers able to understand or participate in this type of social face-to-face interactions is to develop a computational platform to synchronously recognize nonverbal behaviors as part of the interactional context. In this platform, information for the acoustic and visual modalities should be carefully synchronized and rapidly processed. At the same time, contextual and interactional cues should be remembered and integrated to better interpret nonverbal (and verbal) behaviors. In this article, we introduce a real-time computational framework, MultiSense, which offers flexible and efficient synchronization approaches for context-based nonverbal behavior analysis. MultiSense is designed to utilize interactional cues from both interlocutors (e.g., from the computer and the human participant) and integrate this contextual information when interpreting nonverbal behaviors. MultiSense can also assimilate behaviors over a full interaction and summarize the observed affective states of the user. We demonstrate the capabilities of the new framework with a concrete use case from the mental health domain where MultiSense is used as part of a decision support tool to assess indicators of psychological distress such as depression and post-traumatic stress disorder (PTSD). In this scenario, MultiSense not only infers psychological distress indicators from nonverbal behaviors but also broadcasts the user state in real-time to a virtual agent (i.e., a digital interviewer) designed to conduct semi-structured interviews with human participants. Our experiments show the added value of our multimodal synchronization approaches and also demonstrate the importance of MultiSense contextual interpretation when inferring distress indicators. We have developed a system, allowing real-time monitoring of human gestures, which can be used for the automatic recognition of behavioural correlates of psychological stress. The system is based on a low-cost camera (Microsoft Kinect), which provides video recordings capturing the subject’s upper body activity. Motion History Images (MHIs) are calculated in real-time from these recordings. Appropriate algorithms are thereafter applied over the MHIs, enabling the real-time calculation of activity-related behavioural parameters. The system’s efficiency in real-time calculation of behavioural parameters has been tested in a pilot trial, involving monitoring of behavioural parameters during the induction of mental stress. Results showed that our prototype is capable to effectively calculate simultaneously eight different behavioural parameters in real-time. Statistical analysis indicated significant correlations between five of these parameters and self-reported stress. The preliminary findings suggest that our approach could potentially prove useful within systems targeting automatic stress detection, through unobtrusive monitoring of subjects.