Job interview simulation with a virtual agents aims at improving people's social skills and supporting professional inclusion. In such simulators, the virtual agent must be capable of representing and reasoning about the user's mental state based on social cues that inform the system about his her affects and social attitude. In this paper, we propose a formal model of Theory of Mind (ToM) for virtual agent in the context of human-agent interaction that focuses on the affective dimension. It relies on a hybrid ToM that combines the two major paradigms of the domain. Our framework is based on modal logic and inference rules about the mental states, emotions and social relations of both actors. Finally, we present preliminary results regarding the impact of such a model on natural interaction in the context of job interviews simulation.
In order to be able to reason on the affective dimension of the interaction, conveyed by the non-verbal behaviour of both interlocutors, several models rely on the cognitive structure of emotions and appraisal theories such as CPM @cite_4 or OCC @cite_7 . These theories provide domain-independant descriptions of triggering conditions of emotions, that are required for the development the affective aspect of the ToM reasoner. For instance, @cite_14 @cite_13 are BDI-based implementation of the OCC theory. FAtiMA's double appraisal model @cite_18 , although not implemented using a BDI framework, also encodes the OCC model. However, in these models, the inference mechanism itself encodes the chosen Appraisal Theory. On the contrary, in our model, we propose a theory-independant ToM reasoner. While our experiments were conducted using an OCC-based model, the corresponding rules (described in equation ) could be easily replaced by another theory.
@cite_18 @cite_14 @cite_4 @cite_7 @cite_13
1600101713 1995641269 2164186291 2078969644
We report the implementation and evaluation of a Simulation Theory (ST) approach to the Theory of Mind in intelligent graphical agents driven by an affective agent architecture FAtiMA. The existing cognitive appraisal mechanism is adapted to produce a second appraisal cycle, a double appraisal, in order to evaluate the emotional impact of possible actions. The action with the greatest emotional impact is selected as a means of producing more interesting dramatic actions. A variant in which the actual minds of characters present are used is also implemented and evaluated. Results show that these mechanisms do produce more interesting stories. We developed and evaluated a multimodal affect detector that combines conversational cues, gross body language, and facial features. The multimodal affect detector uses feature-level fusion to combine the sensory channels and linear discriminant analyses to discriminate between naturally occurring experiences of boredom, engagement flow, confusion, frustration, delight, and neutral. Training and validation data for the affect detector were collected in a study where 28 learners completed a 32- min. tutorial session with AutoTutor, an intelligent tutoring system with conversational dialogue. Classification results supported a channel × judgment type interaction, where the face was the most diagnostic channel for spontaneous affect judgments (i.e., at any time in the tutorial session), while conversational cues were superior for fixed judgments (i.e., every 20 s in the session). The analyses also indicated that the accuracy of the multichannel model (face, dialogue, and posture) was statistically higher than the best single-channel model for the fixed but not spontaneous affect expressions. However, multichannel models reduced the discrepancy (i.e., variance in the precision of the different emotions) of the discriminant models for both judgment types. The results also indicated that the combination of channels yielded superadditive effects for some affective states, but additive, redundant, and inhibitory effects for others. We explore the structure of the multimodal linear discriminant models and discuss the implications of some of our major findings. Past research in analysis of human affect has focused on recognition of prototypic expressions of six basic emotions based on posed data acquired in laboratory settings. Recently, there has been a shift toward subtle, continuous, and context-specific interpretations of affective displays recorded in naturalistic and real-world settings, and toward multimodal analysis and recognition of human affect. Converging with this shift, this paper presents, to the best of our knowledge, the first approach in the literature that: 1) fuses facial expression, shoulder gesture, and audio cues for dimensional and continuous prediction of emotions in valence and arousal space, 2) compares the performance of two state-of-the-art machine learning techniques applied to the target problem, the bidirectional Long Short-Term Memory neural networks (BLSTM-NNs), and Support Vector Machines for Regression (SVR), and 3) proposes an output-associative fusion framework that incorporates correlations and covariances between the emotion dimensions. Evaluation of the proposed approach has been done using the spontaneous SAL data from four subjects and subject-dependent leave-one-sequence-out cross validation. The experimental results obtained show that: 1) on average, BLSTM-NNs outperform SVR due to their ability to learn past and future context, 2) the proposed output-associative fusion framework outperforms feature-level and model-level fusion by modeling and learning correlations and patterns between the valence and arousal dimensions, and 3) the proposed system is well able to reproduce the valence and arousal ground truth obtained from human coders. The process of eliciting client language toward change (change talk [CT]) is implicated as a causal mechanism in motivational interviewing (MI) and brief motivational interventions (BMI). We investigated the articulation of counselor behaviors and CT during BMI with young men. We coded 149 sessions using the Motivational Interviewing Skill Code and summarized these codes into three counselor categories (MI-consistent [MICO], MI-inconsistent [MIIN], other) and three client categories (CT, counter CT [CCT], follow neutral [F N]). We then computed immediate transition frequencies and odds ratios using sequential analysis software. CT was significantly more likely following MICO behaviors, whereas MIIN behaviors only led to CCT and F N. This strongly supports the use of MI skills to elicit CT during BMI with young men, whose speech also predicted counselor behaviors (particularly CT to MICO and CCT to MIIN). Additional analyses showed that among MICO behaviors, reflective listening may be a particularly powerful technique to elicit CT.