Job interview simulation with a virtual agents aims at improving people's social skills and supporting professional inclusion. In such simulators, the virtual agent must be capable of representing and reasoning about the user's mental state based on social cues that inform the system about his her affects and social attitude. In this paper, we propose a formal model of Theory of Mind (ToM) for virtual agent in the context of human-agent interaction that focuses on the affective dimension. It relies on a hybrid ToM that combines the two major paradigms of the domain. Our framework is based on modal logic and inference rules about the mental states, emotions and social relations of both actors. Finally, we present preliminary results regarding the impact of such a model on natural interaction in the context of job interviews simulation.
In order to be able to reason on the affective dimension of the interaction, conveyed by the non-verbal behaviour of both interlocutors, several models rely on the cognitive structure of emotions and appraisal theories such as CPM @cite_4 or OCC @cite_7 . These theories provide domain-independant descriptions of triggering conditions of emotions, that are required for the development the affective aspect of the ToM reasoner. For instance, @cite_14 @cite_13 are BDI-based implementation of the OCC theory. FAtiMA's double appraisal model @cite_18 , although not implemented using a BDI framework, also encodes the OCC model. However, in these models, the inference mechanism itself encodes the chosen Appraisal Theory. On the contrary, in our model, we propose a theory-independant ToM reasoner. While our experiments were conducted using an OCC-based model, the corresponding rules (described in equation ) could be easily replaced by another theory.
@cite_18 @cite_14 @cite_4 @cite_7 @cite_13
1600101713 1995641269 2164186291 2078969644 2798536775 2049199149 2117645142 1992605069 2092623161 2168240343 2168053878 2103184652 2554340554 2022950268 2150150736 1801368039 2044807399 1963753144 2963609017 2592385986
We report the implementation and evaluation of a Simulation Theory (ST) approach to the Theory of Mind in intelligent graphical agents driven by an affective agent architecture FAtiMA. The existing cognitive appraisal mechanism is adapted to produce a second appraisal cycle, a double appraisal, in order to evaluate the emotional impact of possible actions. The action with the greatest emotional impact is selected as a means of producing more interesting dramatic actions. A variant in which the actual minds of characters present are used is also implemented and evaluated. Results show that these mechanisms do produce more interesting stories. We developed and evaluated a multimodal affect detector that combines conversational cues, gross body language, and facial features. The multimodal affect detector uses feature-level fusion to combine the sensory channels and linear discriminant analyses to discriminate between naturally occurring experiences of boredom, engagement flow, confusion, frustration, delight, and neutral. Training and validation data for the affect detector were collected in a study where 28 learners completed a 32- min. tutorial session with AutoTutor, an intelligent tutoring system with conversational dialogue. Classification results supported a channel × judgment type interaction, where the face was the most diagnostic channel for spontaneous affect judgments (i.e., at any time in the tutorial session), while conversational cues were superior for fixed judgments (i.e., every 20 s in the session). The analyses also indicated that the accuracy of the multichannel model (face, dialogue, and posture) was statistically higher than the best single-channel model for the fixed but not spontaneous affect expressions. However, multichannel models reduced the discrepancy (i.e., variance in the precision of the different emotions) of the discriminant models for both judgment types. The results also indicated that the combination of channels yielded superadditive effects for some affective states, but additive, redundant, and inhibitory effects for others. We explore the structure of the multimodal linear discriminant models and discuss the implications of some of our major findings. Past research in analysis of human affect has focused on recognition of prototypic expressions of six basic emotions based on posed data acquired in laboratory settings. Recently, there has been a shift toward subtle, continuous, and context-specific interpretations of affective displays recorded in naturalistic and real-world settings, and toward multimodal analysis and recognition of human affect. Converging with this shift, this paper presents, to the best of our knowledge, the first approach in the literature that: 1) fuses facial expression, shoulder gesture, and audio cues for dimensional and continuous prediction of emotions in valence and arousal space, 2) compares the performance of two state-of-the-art machine learning techniques applied to the target problem, the bidirectional Long Short-Term Memory neural networks (BLSTM-NNs), and Support Vector Machines for Regression (SVR), and 3) proposes an output-associative fusion framework that incorporates correlations and covariances between the emotion dimensions. Evaluation of the proposed approach has been done using the spontaneous SAL data from four subjects and subject-dependent leave-one-sequence-out cross validation. The experimental results obtained show that: 1) on average, BLSTM-NNs outperform SVR due to their ability to learn past and future context, 2) the proposed output-associative fusion framework outperforms feature-level and model-level fusion by modeling and learning correlations and patterns between the valence and arousal dimensions, and 3) the proposed system is well able to reproduce the valence and arousal ground truth obtained from human coders. The process of eliciting client language toward change (change talk [CT]) is implicated as a causal mechanism in motivational interviewing (MI) and brief motivational interventions (BMI). We investigated the articulation of counselor behaviors and CT during BMI with young men. We coded 149 sessions using the Motivational Interviewing Skill Code and summarized these codes into three counselor categories (MI-consistent [MICO], MI-inconsistent [MIIN], other) and three client categories (CT, counter CT [CCT], follow neutral [F N]). We then computed immediate transition frequencies and odds ratios using sequential analysis software. CT was significantly more likely following MICO behaviors, whereas MIIN behaviors only led to CCT and F N. This strongly supports the use of MI skills to elicit CT during BMI with young men, whose speech also predicted counselor behaviors (particularly CT to MICO and CCT to MIIN). Additional analyses showed that among MICO behaviors, reflective listening may be a particularly powerful technique to elicit CT. Automatic understanding of human affect using visual signals is of great importance in everyday human–machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion) constitute popular and effective representations for affect. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge (Aff-Wild Challenge) that was recently organized in conjunction with CVPR 2017 on the Aff-Wild database, and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional and recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW 2017 datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http: ibug.doc.ic.ac.uk resources first-affect-wild-challenge. This article discusses a formal belief, desire, intention (BDI)-based agent model for theory of mind (ToM). The model uses BDI concepts to describe the reasoning process of an agent that reasons about the reasoning process of another agent, which is also based on BDI concepts. We discuss three different application areas and illustrate how the model can be applied to each of them. We explore a case study for each of the application areas and apply our model to it. For each case study, a number of simulation experiments are described, and their results are discussed. This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, the affective technologies being developed are rife with theoretical assumptions that impact their effectiveness. Hence, an informed and integrated examination of emotion theories from multiple areas will need to become part of computing practice if truly effective real-world systems are to be achieved. This survey discusses theoretical perspectives that view emotions as expressions, embodiments, outcomes of cognitive appraisal, social constructs, products of neural circuitry, and psychological interpretations of basic feelings. It provides meta-analyses on existing reviews of affect detection systems that focus on traditional affect detection modalities like physiology, face, and voice, and also reviews emerging research on more novel channels such as text, body language, and complex multimodal systems. This survey explicitly explores the multidisciplinary foundation that underlies all AC applications by describing how AC researchers have incorporated psychological theories of emotion and how these theories affect research questions, methods, results, and their interpretations. In this way, models and methods can be compared, and emerging insights from various disciplines can be more expertly integrated. We develop a rule-based system that trigger emotions based on the emotional model.We extract the corresponding cause events in fine-grained emotions.We get the proportions of different cause components under different emotions.The language features and Bayesian probability are used in this paper. Emotion analysis and emotion cause extraction are key research tasks in natural language processing and public opinion mining. This paper presents a rule-based approach to emotion cause component detection for Chinese micro-blogs. Our research has important scientific values on social network knowledge discovery and data mining. It also has a great potential in analyzing the psychological processes of consumers. Firstly, this paper proposes a rule-based system underlying the conditions that trigger emotions based on an emotional model. Secondly, this paper extracts the corresponding cause events in fine-grained emotions from the results of events, actions of agents and aspects of objects. Meanwhile, it is reasonable to get the proportions of different cause components under different emotions by constructing the emotional lexicon and identifying different linguistic features, and the proposed approach is based on Bayesian probability. Finally, this paper presents the experiments on an emotion corpus of Chinese micro-blogs. The experimental results validate the feasibility of the approach. The existing problems and the further works are also present at the end. Current theories of emotion suggest that threat-related stimuli are first processed via an automatically engaged neural mechanism, which occurs outside conscious awareness. This mechanism operates in conjunction with a slower and more comprehensive process that allows a detailed evaluation of the potentially harmful stimulus (LeDoux, 1998). We drew on the Halgren and Marinkovic (1995) model to examine these processes using event-related potentials (ERPs) within a backward masking paradigm. Stimuli used were faces with fear and neutral (as baseline control) expressions, presented above (supraliminal) and below (subliminal) the threshold for conscious detection. ERP data revealed a double dissociation for the supraliminal versus subliminal perception of fear. In the subliminal condition, responses to the perception of fear stimuli were enhanced relative to neutral for the N2 "excitatory" component, which is thought to represent orienting and automatic aspects of face processing. By contrast, supraliminal perception of fear was associated with relatively enhanced responses for the late P3 "inhibitory" component, implicated in the integration of emotional processes. These findings provide evidence in support of Halgren and Marinkovic's temporal model of emotion processing, and indicate that the neural mechanisms for appraising signals of threat may be initiated, not only automatically, but also without the need for conscious detection of these signals. We propose a joint model for unsupervised induction of sentiment, aspect and discourse information and show that by incorporating a notion of latent discourse relations in the model, we improve the prediction accuracy for aspect and sentiment polarity on the sub-sentential level. We deviate from the traditional view of discourse, as we induce types of discourse relations and associated discourse cues relevant to the considered opinion analysis task; consequently, the induced discourse relations play the role of opinion and aspect shifters. The quantitative analysis that we conducted indicated that the integration of a discourse model increased the prediction accuracy results with respect to the discourse-agnostic approach and the qualitative analysis suggests that the induced representations encode a meaningful discourse structure. The ability of a computer to detect and appropriately respond to changes in a user's affective state has significant implications to human-computer interaction (HCI). In this paper, we present our efforts toward audio-visual affect recognition on 11 affective states customized for HCI application (four cognitive motivational and seven basic affective states) of 20 nonactor subjects. A smoothing method is proposed to reduce the detrimental influence of speech on facial expression recognition. The feature selection analysis shows that subjects are prone to use brow movement in face, pitch and energy in prosody to express their affects while speaking. For person-dependent recognition, we apply the voting method to combine the frame-based classification results from both audio and visual channels. The result shows 7.5 improvement over the best unimodal performance. For person-independent test, we apply multistream HMM to combine the information from multiple component streams. This test shows 6.1 improvement over the best component performance More than 15 years after the early studies in Affective Computing (AC), [1] the problem of detecting and modeling emotions in the context of human-computer interaction (HCI) remains complex and largely unexplored. The detection and modeling of emotion is, primarily, the study and use of artificial intelligence (AI) techniques for the construction of computational models of emotion. The key challenges one faces when attempting to model emotion [2] are inherent in the vague definitions and fuzzy boundaries of emotion, and in the modeling methodology followed. In this context, open research questions are still present in all key components of the modeling process. These include, first, the appropriateness of the modeling tool employed to map emotional manifestations and responses to annotated affective states; second, the processing of signals that express these manifestations (i.e., model input); and third, the way affective annotation (i.e., model output) is handled. This paper touches upon all three key components of an affective model (i.e., input, model, output) and introduces the use of deep learning (DL) [3], [4], [5] methodologies for affective modeling from multiple physiological signals. A common phenomenon in everyday life is that, when a strange event occurs or is announced, a regular crowd can completely change, showing different intense emotions and sometimes uncontrollable and violent emerging behavior. These emotions and behaviors that disturb the organization of a crowd are of concern in our study, and we attempt to predict these suspicious circumstances and provide help in making the right decisions at the right time. Furthermore, most of the models that address crowd disasters belong to the physical or the cognitive approaches. They study pedestrian flow and collision avoidance, etc., and they use walking speed and angle of vision. However, in this work, based on a behavioral rules approach, we aim to model emergent emotion, behavior and influence in a crowd, taking into account particularly the personality of members of the crowd. For this purpose, we have combined the OCEAN Openness, Consciousness, Extraversion, Agreeableness, and Neuroticism personality model with the OCC Ortony, Clore, and Collins emotional model to indicate the susceptibility of each of the five personality factors to feeling every emotion. Then we proposed an approach that uses first fuzzy logic for the emotional modeling of critical emotions of members of the crowd at the announcement or the presence of unusual events, in order to quantify emotions. Then, we model the behavior and the tendency towards actions using probability theory. Finally, the influence among the members of the crowd is modeled using the neighborhood principle and cellular automata. The basic idea of our anticipatory approach to perception is to avoid the common separation of perception and generation of behavior and to fuse both aspects into a consistent neural process. Our approach tries to explain the phenomenon of perception, in particular, of perception at the level of sensorimotor intelligence, from a behavior-oriented point of view. Perception is assumed to be a generative process of anticipating the course of events resulting from alternative sequences of hypothetically executed actions. By means of this sensorimotor anticipation, it is possible to characterize a visual scenery immediately in categories of behavior, i.e. by a set of actions which describe possible methods of interaction with the objects in the environment. Thus, the competence to perceive a complex situation can be understood as the capability to anticipate the course of events caused by different action sequences. Starting from an abstract description of anticipatory perception and the essential biological evidence for internal simulation, we present two biologically motivated computational models that are able to anticipate and evaluate hypothetically sensorimotor sequences. Both models consider functional aspects of those cortical and subcortical systems that are assumed to be involved in the process of sensory prediction and sensorimotor control. Our first approach, the Model for Anticipation based on Sensory IMagination (MASIM), realizes a sequential search in sensorimotor space using a simple model of lateral cerebellum as sensory predictor. We demonstrate the efficiency of this model approach in the light of visually guided local navigation behaviors of a mobile system. The second approach, the Model for Anticipation based on Cortical Representations (MACOR), is actually still at a conceptual level of realization. We postulate that this model allows a completely parallel search at the neocortical level using assemblies of spiking neurons for grouping, separation, and selection of sensorimotor sequences. Both models are intended as general schemes for anticipation based perception at the level of sensorimotor intelligence. Signals from peripheral physiology (e.g., ECG, EMG, and GSR) in conjunction with machine learning techniques can be used for the automatic detection of affective states. The affect detector can be user-independent, where it is expected to generalize to novel users, or user-dependent, where it is tailored to a specific user. Previous studies have reported some success in detecting affect from physiological signals, but much of the work has focused on induced affect or acted expressions instead of contextually constrained spontaneous expressions of affect. This study addresses these issues by developing and evaluating user-independent and user-dependent physiology-based detectors of nonbasic affective states (e.g., boredom, confusion, curiosity) that were trained and validated on naturalistic data collected during interactions between 27 students and AutoTutor, an intelligent tutoring system with conversational dialogues. There is also no consensus on which techniques (i.e., feature selection or classification methods) work best for this type of data. Therefore, this study also evaluates the efficacy of affect detection using a host of feature selection and classification techniques on three physiological signals (ECG, EMG, and GSR) and their combinations. Two feature selection methods and nine classifiers were applied to the problem of recognizing eight affective states (boredom, confusion, curiosity, delight, flow -engagement, surprise, and neutral). The results indicated that the user-independent modeling approach was not feasible; however, a mean kappa score of 0.25 was obtained for user-dependent models that discriminated among the most frequent emotions. The results also indicated that k-nearest neighbor and Linear Bayes Normal Classifier (LBNC) classifiers yielded the best affect detection rates. Single channel ECG, EMG, and GSR and three-channel multimodal models were generally more diagnostic than two--channel models. To explain observations from nonmonotonic background theories, one often needs removal of some hypotheses as well as addition of other hypotheses. Moreover, some observations should not be explained, while some are to be explained. In order to formalize these situations, extended abduction was introduced by Inoue and Sakama (1995) to generalize traditional abduction in the sense that it can compute negative explanations by removing hypotheses and antidexplanations to unexplain negative observations. In this paper, we propose a computational mechanism for extended abduction. When a background theory is written in a normal logic program, we introduce its transaction program for computing extended abduction. A transaction program is a set of nonddeterministic production rules that declaratively specify addition and deletion of abductive hypotheses. Abductive explanations are then computed by the fixpoint of a transaction program using a bottomdup model generation procedure. The correctness of the proposed procedure is shown for the class of acyclic covered abductive logic programs. In the context of deductive databases, a transaction program provides a declarative specification of database update. Research in affective computing requires ground truth data for training and benchmarking computational models for machine-based emotion understanding. In this paper, we propose a large video database, namely LIRIS-ACCEDE, for affective content analysis and related applications, including video indexing, summarization or browsing. In contrast to existing datasets with very few video resources and limited accessibility due to copyright constraints, LIRIS-ACCEDE consists of 9,800 good quality video excerpts with a large content diversity. All excerpts are shared under creative commons licenses and can thus be freely distributed without copyright issues. Affective annotations were achieved using crowdsourcing through a pair-wise video comparison protocol, thereby ensuring that annotations are fully consistent, as testified by a high inter-annotator agreement, despite the large diversity of raters’ cultural backgrounds. In addition, to enable fair comparison and landmark progresses of future affective computational models, we further provide four experimental protocols and a baseline for prediction of emotions using a large set of both visual and audio features. The dataset (the video clips, annotations, features and protocols) is publicly available at: http: liris-accede.ec-lyon.fr . Abstract The purpose of this paper is twofold: (i) to investigate the emotion representation models and find out the possibility of a model with minimum number of continuous dimensions and (ii) to recognize and predict emotion from the measured physiological signals using multiresolution approach. The multimodal physiological signals are: Electroencephalogram (EEG) (32 channels) and peripheral (8 channels: Galvanic skin response (GSR), blood volume pressure, respiration pattern, skin temperature, electromyogram (EMG) and electrooculogram (EOG)) as given in the DEAP database. We have discussed the theories of emotion modeling based on i) basic emotions, ii) cognitive appraisal and physiological response approach and iii) the dimensional approach and proposed a three continuous dimensional representation model for emotions. The clustering experiment on the given valence, arousal and dominance values of various emotions has been done to validate the proposed model. A novel approach for multimodal fusion of information from a large number of channels to classify and predict emotions has also been proposed. Discrete Wavelet Transform, a classical transform for multiresolution analysis of signal has been used in this study. The experiments are performed to classify different emotions from four classifiers. The average accuracies are 81.45 , 74.37 , 57.74 and 75.94 for SVM, MLP, KNN and MMC classifiers respectively. The best accuracy is for ‘Depressing’ with 85.46 using SVM. The 32 EEG channels are considered as independent modes and features from each channel are considered with equal importance. May be some of the channel data are correlated but they may contain supplementary information. In comparison with the results given by others, the high accuracy of 85 with 13 emotions and 32 subjects from our proposed method clearly proves the potential of our multimodal fusion approach. Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications. We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths. We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization. Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X). We quantitatively show that training with the textual explanations not only yields better textual justification models, but also better localizes the evidence that supports the decision. We also qualitatively show cases where visual explanation is more insightful than textual explanation, and vice versa, supporting our thesis that multimodal explanation models offer significant benefits over unimodal approaches. We aim to apply sparse autoencoder-based unsupervised feature learning to emotional semantic analysis for textile images. To tackle the problem of limited training data, we present a cross-domain feature learning scheme for emotional textile image classification using convolutional autoencoders. We further propose a correlation-analysis-based feature selection method for the weights learned by sparse autoencoders to reduce the number of features extracted from large size images. First, we randomly collect image patches on an unlabeled image dataset in the source domain and learn local features with a sparse autoencoder. We then conduct feature selection according to the correlation between different weight vectors corresponding to the autoencoder’s hidden units. We finally adopt a convolutional neural network including a pooling layer to obtain global feature activations of textile images in the target domain and send these global feature vectors into logistic regression models for emotional image classification. The cross-domain unsupervised feature learning method achieves 65 to 78 average accuracy in the cross-validation experiments corresponding to eight emotional categories and performs better than conventional methods. Feature selection can reduce the computational cost of global feature extraction by about 50 while improving classification performance.