In recent years, the use of bio-sensing signals such as electroencephalogram (EEG), electrocardiogram (ECG), etc. have garnered interest towards applications in affective computing. The parallel trend of deep-learning has led to a huge leap in performance towards solving various vision-based research problems such as object detection. Yet, these advances in deep-learning have not adequately translated into bio-sensing research. This work applies novel deep-learning-based methods to various bio-sensing and video data of four publicly available multi-modal emotion datasets. For each dataset, we first individually evaluate the emotion-classification performance obtained by each modality. We then evaluate the performance obtained by fusing the features from these modalities. We show that our algorithms outperform the results reported by other studies for emotion valence arousal liking classification on DEAP and MAHNOB-HCI datasets and set up benchmarks for the newer AMIGOS and DREAMER datasets. We also evaluate the performance of our algorithms by combining the datasets and by using transfer learning to show that the proposed method overcomes the inconsistencies between the datasets. Hence, we do a thorough analysis of multi-modal affective data from more than 120 subjects and 2,800 trials. Finally, utilizing a convolution-deconvolution network, we propose a new technique towards identifying salient brain regions corresponding to various affective states.
Table shows that in almost all the cases EEG has been the preferred bio-sensing modality while vision modality i.e. the use of the frontal videos to analyze facial expressions has not been commonly used on these datasets. The classification accuracy for all emotion classes as per the circumplex model rather than only for arousal valence are rarely reported. In other cases such as @cite_61 @cite_6 , where the analysis of emotions is reported, the goal seems to be clustering the complete dataset into four classes rather than having a distinct training and testing partition for evaluation.
@cite_61 @cite_6
2536442339 2222581876
Recognition of discriminative brain functional network pattern and regions corresponding to emotions are important in understanding the neuron functional network underlying the human emotion process. Emotion models mapping onto brain is possible with the help of emotion-specific network patterns and its corresponding brain regions. This paper presents a method to identify emotion related functional connectivity pattern and their distinctive associated regions using EEG phase synchrony (phase locking value (PLV)) connectivity analysis. The emotion-specific channel pairs, reactive band, and synchrony related locations are identified based on the network dissimilarities between emotion and rest tasks. With the most reactive pairs identified, the emotion-specific functional network is formed. The proposed method is validated on ‘database for emotion analysis using physiological signals (DEAP)’ that confirms the distinct nature of identified functional connectivity pattern and the regions corresponding to the emotion. The brain functional network perspective forms the basis to relate mechanisms of brain functions. This work analyzes the network mechanisms related to human emotion based on synchronization measure - phase-locking value in EEG to formulate the emotion specific brain functional network. Based on network dissimilarities between emotion and rest tasks, most reactive channel pairs and the reactive band corresponding to emotions are identified. With the identified most reactive pairs, the subject-specific functional network is formed. The identified subject-specific and emotion-specific dynamic network pattern show significant synchrony variation in line with the experiment protocol. The same network pattern are then employed for classification of emotions. With the study conducted on the 4 subjects, an average classification accuracy of 62 was obtained with the proposed technique.