Equipping social and service robots with the ability to perceive human emotional intensities during an interaction is in increasing demand. Most of existing work focuses on determining which emotion(s) participants are expressing from facial expressions but largely overlooks the emotional intensities spontaneously revealed by other social cues, especially body languages. In this paper, we present a real-time method for robots to capture fluctuations of participants' emotional intensities from their body poses. Unlike conventional joint-position-based approaches, our method adopts local joint transformations as pose descriptors which are invariant to subject body differences as well as the pose sensor positions. In addition, we use a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) architecture to take the specific emotion context into account when estimating emotional intensities from body poses. The dataset evaluation suggests that the proposed method is effective and performs better than baseline method on the test dataset. Also, a series of succeeding field tests on a physical robot demonstrates that the proposed method effectively estimates subjects emotional intensities in real-time. Furthermore, the robot equipped with our method is perceived to be more emotion-sensitive and more emotionally intelligent.
Emotional intensity is defined as the psychological states of being activated by some evoking stimulus @cite_16 , which can be measured by bio-electric signals, such as heart rate, blood pressure, skin conductance etc @cite_33 @cite_10 @cite_31 . For example, Kulic and Croft leveraged heart rate, perspiration rate, and facial muscle contraction to estimate participants' emotional intensities in terms of anxiety, calm and surprise towards robot arm motions @cite_20 ; Swangnetr and Kaber used heart rate and galvanic skin response signals to estimate the intensity of joy and excitement for the elderly patients during medical Patient-Robot Interactions @cite_0 . Saulnier exploited the natural muscle tensions to estimate participants' stress levels for domestic robots @cite_25 . However, all these studies require putting sensor on participants, e.g., via special wearable devices @cite_19 or electro-physiological monitoring devices @cite_20 @cite_6 , it may be difficult and sometimes impossible to deploy such biological methods for social or service robots in real-time HRI.
@cite_33 @cite_6 @cite_0 @cite_19 @cite_31 @cite_16 @cite_10 @cite_25 @cite_20
2470957930 2019771222 1963753144 1543168978
Emotion is expressed in multiple modalities, yet most research has considered at most one or two. This stems in part from the lack of large, diverse, well-annotated, multimodal databases with which to develop and test algorithms. We present a well-annotated, multimodal, multidimensional spontaneous emotion corpus of 140 participants. Emotion inductions were highly varied. Data were acquired from a variety of sensors of the face that included high-resolution 3D dynamic imaging, high-resolution 2D video, and thermal (infrared) sensing, and contact physiological sensors that included electrical conductivity of the skin, respiration, blood pressure, and heart rate. Facial expression was annotated for both the occurrence and intensity of facial action units from 2D video by experts in the Facial Action Coding System (FACS). The corpus further includes derived features from 3D, 2D, and IR (infrared) sensors and baseline results for facial expression and action unit detection. The entire corpus will be made available to the research community. Due to a major shortage of nurses in the U.S., future healthcare service robots are expected to be used in tasks involving direct interaction with patients. Consequently, there is a need to design nursing robots with the capability to detect and respond to patient emotional states and to facilitate positive experiences in healthcare. The objective of this study was to develop a new computational algorithm for accurate patient emotional state classification in interaction with nursing robots during medical service. A simulated medicine delivery experiment was conducted at two nursing homes using a robot with different human-like features. Physiological signals, including heart rate (HR) and galvanic skin response (GSR), as well as subjective ratings of valence (happy-unhappy) and arousal (excited-bored) were collected on elderly residents. A three-stage emotional state classification algorithm was applied to these data, including: (1) physiological feature extraction; (2) statistical-based feature selection; and (3) a machine-learning model of emotional states. A pre-processed HR signal was used. GSR signals were nonstationary and noisy and were further processed using wavelet analysis. A set of wavelet coefficients, representing GSR features, was used as a basis for current emotional state classification. Arousal and valence were significantly explained by statistical features of the HR signal and GSR wavelet features. Wavelet-based de-noising of GSR signals led to an increase in the percentage of correct classifications of emotional states and clearer relationships among the physiological response and arousal and valence. The new algorithm may serve as an effective method for future service robot real-time detection of patient emotional states and behavior adaptation to promote positive healthcare experiences. Abstract The purpose of this paper is twofold: (i) to investigate the emotion representation models and find out the possibility of a model with minimum number of continuous dimensions and (ii) to recognize and predict emotion from the measured physiological signals using multiresolution approach. The multimodal physiological signals are: Electroencephalogram (EEG) (32 channels) and peripheral (8 channels: Galvanic skin response (GSR), blood volume pressure, respiration pattern, skin temperature, electromyogram (EMG) and electrooculogram (EOG)) as given in the DEAP database. We have discussed the theories of emotion modeling based on i) basic emotions, ii) cognitive appraisal and physiological response approach and iii) the dimensional approach and proposed a three continuous dimensional representation model for emotions. The clustering experiment on the given valence, arousal and dominance values of various emotions has been done to validate the proposed model. A novel approach for multimodal fusion of information from a large number of channels to classify and predict emotions has also been proposed. Discrete Wavelet Transform, a classical transform for multiresolution analysis of signal has been used in this study. The experiments are performed to classify different emotions from four classifiers. The average accuracies are 81.45 , 74.37 , 57.74 and 75.94 for SVM, MLP, KNN and MMC classifiers respectively. The best accuracy is for ‘Depressing’ with 85.46 using SVM. The 32 EEG channels are considered as independent modes and features from each channel are considered with equal importance. May be some of the channel data are correlated but they may contain supplementary information. In comparison with the results given by others, the high accuracy of 85 with 13 emotions and 32 subjects from our proposed method clearly proves the potential of our multimodal fusion approach. We conducted a 19 participant study using a system comprised of wireless galvanic skin response (GSR), heart rate (HR), activity sensors and a mobile phone for aggregating sensor data and enabling affect logging by the user. Each participant wore the sensors daily for five days, generating approximately 900 hours of continuous data. We found that analysis of emotional events was highly dependent on correct windowing and report results on synthesized windows around annotated events. Where raters agreed on the timing and quality of the emotion we were able to recognize 85 of the high and low energy emotions and 70 of the positive and negative emotions. We also gained many insights regarding participant's perception of their emotional state and the complexity of emotion in real life.