Computer Science Biology Medicine
Computer Science-2021.gz:143294
Computer Science
200bed87f66d0266132c051119854977a2808c4e
Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition

In recent years, emotion recognition based on electroencephalography (EEG) has received growing interests in the brain-computer interaction (BCI) field. The neuroscience researches indicate that the left and right brain hemispheres demonstrate activity differences under different emotional activities, which could be an important principle for designing deep learning (DL) model for emotion recognition. Besides, owing to the nonstationarity of EEG signals, using convolution kernels of a single size may not sufficiently extract the abundant features for EEG classification tasks. Based on these two angles, we proposed a model termed Multi-Scales Bi-hemispheric Asymmetric Model (MSBAM) based on convolutional neural network (CNN) structure. Evaluated on the public DEAP and DREAMER datasets, MSBAM achieved over 99% accuracy for the two-class classification of low-level and high-level states in each of four emotional dimensions, i.e., arousal, valence, dominance and liking, respectively. This study further demonstrated the promising potential to design the DL model from the multi-scale characteristics of the EEG data and the neural mechanisms of the emotion cognition.


Introduction
Emotion is a kind of physiological and psychological phenomenon playing a significant role in daily lifeDolan (2002) Cabanac (2002). In recent years, emotion recognition based on machine learning and deep learning(DL) methods has attracted growing attention of the research community Torres, Torres, Hernández-Álvarez and Yoo (2020). Emotions can be detected from different modalities, such as facial expression Zhao, Tao, Zhang, Xu, Zhang, Hao and Chen (2021b), speech El Ayadi, Kamel and Karray (2011), and physiological data Li, Liu, Si, Li, Li, Zhu, Huang, Zeng, Yao, Zhang and Xu (2019a), etc. Among these, the physiological signals are hard to fake and show a decided advantage in measurement of spontaneous mental activity under different emotional states.
Human physiological signals can be measured by different imaging modalities, such as functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), functional near-infrared spectroscopy (fNIRS), and electroencephalography (EEG), et al. Benefiting from properties such as high temporal resolution, non-invasiveness, low cost, and the portability of devices, EEG has been widely employed in the field of emotion recognition Li, Huan, Hou, Tian, Zhang and Song (2021b).
In the field of emotion recognition, research objects can be defined in many ways. From the emotional modeling point of view, common theories include discrete emotional model and emotional circumplex model, which correspond to different evaluation criteria. For instance, Berke Kılıç and Serap Aydın proposed a method based on support vector machine (SVM) and graph theoretical network measures to classify five pairs of discrete emotions. Their method achieved the best accuracy of 80.65% Kılıç and Aydın (2022). In addition, some researchers tried to explore other related indicators using the emotional EEG. For example, Serap Aydın proposed an emotional complexity marker, i.e. phase space trajectory matrix. They employed this marker to classify the gender of subjects and distinguish nine emotional states from baseline state Aydın (2020). In this study, we mainly focus on discussing the classification between low-level and high-level of the evaluative dimensions in the emotional circumplex model.
To differentiate emotional states, the conventional algorithms usually include a feature extractor and a classifier. Various hand-crafted features have been employed to extract the differences between different emotional states. For instance, Wen et al. utilized Pearson correlation coefficient (PCC) to estimate the correlation between all channel pairs. They extracted the PCC feature and convolutional neural networks (CNN) features parallelly to classify the emotional states. Their method achieved average accuracies of 77.98% and 72.98% in valance and arousal of the DEAP dataset, respectively Wen, Xu and Du (2017). Zheng et al. extracted five hand-crafted features, i.e. power spectral density (PSD), differential entropy (DE), differential asymmetry, rational asymmetry, asymmetry and differential causality features, to recognize the emotion using SVM and graph regularized extreme learning machine (GELM) classifier, respectively. The DE features and GELM classifier obtained 69.67% accuracy for four classification task on valance-arousal space Zheng, Zhu and Lu (2019). Moon et al. adopted the PCC, phaselocking value (PLV) and transfer entropy (TE) to calculate the hand-crafted features, and employed CNN to extract advanced features to classify the emotional states. Their method achieved 87.75% accuracy in valance dimension with the data length of 3 seconds Moon, Chen, Hsieh, Wang and Lee (2020).
With the rapid development of deep learning, increasing researchers are pursuing end-to-end solutions to replace the conventional classification methods based on handcraft features. Yang et al. proposed a parallel convolutional recurrent neural network model by combining CNN and Long-Short Term Memories neural network(LSTM). The CNN is adopted to extract the inter-channel correlation, and the LSTM is designed to mine the temporal contextual correlation. Their method achieved 90.80% and 91.03% performance on valance and arousal of DEAP with data length of 1 second, respectively Yang, Wu, Qiu, Wang and Chen (2018a). Ma et al. proposed a method that applied residual structure to LSTM. Multimodal signal were input into two residual LSTM networks that are shared part weight to extract the emotion related high-level features. This method obtained 92.30% and 92.87% accuracies on the valance and arousal dimensions of DEAP dataset with data length of 1 second, respectively Ma, Tang, Zheng and Lu (2019b). Yin et al. proposed a method by fusing graph convolutional neural networks (GCNN) and LSTM. Several GCNNs were employed to extract features of graph domain. LSTM are used to extract the temporal features and detect their changes. This method obtained 90.45% and 90.60% accuracy on DEAP with data length of 6 seconds, respectively Yin, Zheng, Hu, Zhang and Cui (2021a).
For the past few years, increasing studies have started to design the DL models by considering the physiological mechanisms of emotions. Bi-hemispheric discrepancy under different emotion states is a vital neural mechanism, which has been used in several studies with DL models for emotion recognition Li, Zheng, Zong, Cui, Zhang and Zhou (2021c) ;Huang, Chen, Liu, Zheng, Tian and Jiang (2021). For instance, Li et al. proposed a method using LSTM to capture the high-level discrepancy features from bi-hemispheric hand-crafted features. They input those features into a domain adaption model to classify the emotional states. This model yielded 92.38% and 84.14% accuracy on the subject dependent and independent task on the SEED dataset, respectively Li et al. (2021c). In another study, Li et al. proposed a model, termed R2G-STNN, to extract regional features according to the spatial region concerning physiological function, and fused the regional and global features. This method achieved 93.38% and 84.16% accuracy on the SEED dataset Li, Zheng, Wang, Zong and Cui (2019b).
EEG signals are nonstationary, using convolution kernels of a single size may not sufficiently extract the abundant features for EEG classification tasks. Previous studies have demonstrate that using various convolution kernels of different sizes could learn multi-scale EEG features that are beneficial for different EEG classification tasks Ko, Jeon, Jeong and Suk (2021). For example, Li et al. proposed a multi-scale fusion CNN model based on attention mechanism for motor imagery classification Li, Xu, Wang, Fang and Ji (2020). The experimental results indicated that the model achieved a better performance compared with the baseline methods. The multi-scale CNN has also been introduced in emotion recognition. Phan et al. proposed a 2D CNN model with convolution kernels of different sizes for arousal and valence binary classification. They used kernel size of 5 × 5 and 7 × 7 to extract spatial features to describe the short-range and long-range relations between EEG channels. This method achieved 98.27% and 98.36% accuracies on the valance and arousal dimensions of DEAP dataset Phan, Kim, Yang and Lee (2021).
However, the physiological mechanisms and multi-scale features of EEG have not been simultaneously considered in a DL model for emotion recognition. Based on this consideration, we proposed a DL model termed Multi-Scales Bihemispheric Asymmetric Model (MSBAM) to explore the multi-scale asymmetric information. MSBAM is composed of the parallel spatial domain feature extractor and temporal domain feature extractor, followed by a fully connected layer classifier. We conducted extensive experiments on the public DEAP dataset and DREAMER dataset to evaluate the performance of MSBAM and the compared baseline methods. The results indicate that our MSBAM achieve better performance than the baseline methods.
The remainder of this paper is organized as follows. Section 2 introduces materials and methods. Section 3 describes the settings and results of extensive experiments, comparisons between MSBAM and the baseline methods are also be provided in this section. Section 4 and 5 present the discussions and conclusion.

Datasets
The public DEAP and DREAMER dataset were adopted to validate the performance of our MSBAM, which are widely used for emotion recognition researches.
The DEAP dataset is a multimodal dataset presented by Koelstra et al. Koelstra, Muhl, Soleymani, Lee, Yazdani, Ebrahimi, Pun, Nijholt and Patras (2012). Thirty-two healthy subjects participated in the experiment. They were watching 40 one-minute pieces of music videos. The EEG and peripheral physiological signals were recorded when they were watching videos. Forty electrodes (32 for EEG and 8 for peripheral physiological signals) were used for the EEG recording. Participants were asked to rate a score for each video from 1 to 9 to evaluate the levels of four emotional dimensions, i.e., arousal, valence, dominance and liking, respectively. Each trial contains 3 s baseline data and 60 s task data. For the offline analysis, the EEG signals were first downsampled to 128 Hz from 512 Hz, and re-referenced to the common average reference. Electrooculogram (EOG) artifacts were removed. Then, the EEG data was filtered by a bandpass filter with 4-45 Hz. The details can be found in the reference Koelstra et al. (2012).
The DREAMER dataset was presented by Katsigiannis et al. Katsigiannis and Ramzan (2018). Twenty-three subjects (9 females and 14 males) were recruited in the experiments. The dataset was collected with 14 electrodes at a sampling rate of 128 Hz during the subjects watching 18 emotional film clips. The electrode placement was according to the international 10-20 system. The duration of the film clips are between 65 and 393 seconds. Before watching each emotional film clip, subjects are asked to watch a neutral film clip that is regarded as having no valance to record the baseline signals. After the EEG experiments, the subjects were asked to rate a score for each clip from 1 to 5 for arousal, valence and dominance by self-assessment, respectively.

The classification tasks for emotional states
In this work, MSBAM was designed to distinguish between low-level and high-level emotional states according to the circumplex model with the EEG data. The classification tasks were conducted independently in each of the four emotion dimensions, i.e., arousal, valence, dominance and liking. To generate the two-class data in each emotion dimension, a threshold of self-rating scores was chosen to label each trial with a low-level or high-level emotional state. For example, in the valance dimension, supposing the threshold was 5, the trials with self-rating score higher than 5 were labeled as high valance, other trials were labeled as low valance. The values of the threshold for the DEAP and DREAMER datasets were different according to previous studies. Specifically, for the four dimensions of the DEAP dataset, the threshold was set as 5 as in the previous studies Liu, Zheng and Lu (2016); Li, Chai, Wang, Yang and Du (2021a). Similarly, for the DREAMER dataset, the threshold was set as 3 because the score range is from 1 to 5 Li et al. (2021a); Cui, Liu, Zhang, Chen, Wang and Chen (2020).

Data processing
In order to reduce the noise and improve the stability of the EEG data, baseline correction and z-score normalization were implemented on the two datasets, which were commonly pre-processing procedures for EEG Huang et al. (2021); Cui et al. (2020). Previous studies demonstrated that these operations can improve emotion recognition accuracies by reducing the interference of basic emotional state before the task period Yang et al. (2018a). The diagram of baseline correction is illustrated in the Fig. 1.
Take for instance the DEAP dataset. First, the 3-s baseline data in each trial were divided into three segments of 1 s length without overlap. These segments were further averaged to obtain the baseline signal of 1 s length, called resting-state segment. Second, the 60 s task data in each trial were divided into 60 segments of 1 s length without overlap, termed task-state segments. Third, each task-state segment subtracted the resting-state segment. After these steps, we concatenated all the processed task-state segments to construct the new task-state data of 60 s. At the last, z-score normalization was utilized on each channel of the task-state data. For the DREAMER dataset, we selected the baseline data of the last 3 s in each trial to calculate the resting-state segment for each trial. After the pre-processing procedure, the new task-state data will be segmented with Wnd seconds window without overlap. Finally, we obtained samples ∈ ℝ × , where denotes the number of channels and = * represents the number of sample points.
denotes the sampling rate of 128 Hz. For the parameter , we set it to be 1 in the following experiments as that in the previous studies Huang et al. (2021); Ma et al. (2019b); Li et al. (2021a).
Overall, we have obtained 32(subjects) × 40(trials) × 60(segments per trial) =76800 samples in total for DEAP dataset. Similarly, 23(subjects) × 3728(segments per subject) = 85744 samples are obtained for DREAMER dataset. Traditionally, EEG data is represented as 2D matrix with the shape of channels × sample points for most algorithms. Under the 2D representation, the data from all used channels at a sampling time point are arranged into a column vector, the topological information among different channels would be lost. In recent years, the 3D representation has been introduced for EEG data Zhao, Zhang, Zhu, You, Kuang and Sun (2019a). In this way, the data from all used channels at one sample point are arranged into a 2D matrix according to distribution of the international 10-20 system, which can retain the spatial information among channels to some extent. We denote this operation as spatial transformation. The schematic diagram is shown in Fig. 2. We implemented the spatial transformation at all sample points to obtain the 3D representation of the EEG data. This 3D representation can preserve both temporal information and spatial information of EEG data, which has been adopted for various EEG classification tasks Cui et al. (2020); Zhang, Cai, Nie, Xu, Zhao and Guan (2021); Zhao et al. (2019a).

3D representation of the EEG data
As the input of our model, each sample ∈ ℝ × is transformed to a 3D spatial-temporal matrix ∈ ℝ × × . In current study, = 32 and 14 for DEAP and DREAMER respectively. = 128, and and were set to 9. Then, each sample was in a 3D shape of size 9 × 9 × 128. Previous studies indicate the significance of physiological mechanisms and multi-scale features of EEG for emotion recognition Li et al. (2021c); Phan et al. (2021). However, the correlations and differences of asymmetric features in various time scales have not been considered in a DL model for emotion recognition. Based on this consideration, our MSBAM contains three parts, i.e., a spatial feature extractor block, a temporal feature extractor block, and a feature classification block. The structure and details of MSBAM are shown in Figs. 3, 4 and 5, respectively.

Spatial feature extractor block
For the spatial feature extractor block, as shown in Fig. 4, it contains three convolution(Conv) operations and one fully connected layer (FC). The first Conv operation, denoted as 1 , contains a convolution ( 1 ) with 16 kernels of size 1 × 1 × 128, and a Scaled Exponential Linear Units (SELU) activation function, which is expressed as: where denotes the SELU function. This procedure is able to compress the data from 3 D into 2 D, and all values in the time dimension are transformed to single feature value. The remaining two Conv operations are similar to the 1 except for the convolution with 32 kernels of size 3 × 3 and padding size of (1, 1). Through these Conv operations successively, a feature ∈ ℝ 9×9×1×16 is obtained to represent the spatial feature of the EEG data.
Then the feature is flattened to a vector and input into a FC layer followed by a batch normalization layer and softmax activation to obtain the normalized feature , which could be described as: where W is the weight matrix, b is the bias. The procedures in the formulas (3)-(6) are denoted as the -( ), and will be utilized in the temporal feature extractor block.

Temporal feature extractor block
For the temporal feature extractor block, as is shown in Fig. 5(a), it contains branches, each of which contains a convolution operation named , a flattenconcatenation operation, and a FNL. Multiple branches that using temporal convolution kernels of different lengths could learn multi-scale EEG features which is beneficial for emotion recognition tasks.
In each branch, the EEG data D is split into two parts in spatial dimension. As shown in Fig. 5(b), the first part, denoted as ∈ ℝ 9×5×128 , comes from the columns of [1, 2, 3, 4, 5] of the 9 × 9 matrix as shown in Fig. 2, which represents the data from left hemisphere. Similarly, the second part, denoted as ∈ ℝ 9×5×128 , comes from the columns of [5,6,7,8,9], which represents the data from right hemisphere. To remain the unified location of electrodes between and , a horizontal flip is implemented on , which rearranges the columns as [9,8,7,6,5]. After the process, and are first input into the same block, which is illustrated in the following: where denotes -th branch, and is SELU activation function. The conducts a 3D convolution with 16 kernels of size 9 × 5 × and the stride of size 9 × 5 × . The is the time scale of the extractor, and the is equal to //2.
After the block, the and are obtained and utilized to calculate discrepant features, which are denoted as ∈ ℝ 1×1×̂ ×16 . The calculation is described as follows: Then, these three kinds of features, i.e., , and , are first flattened to vectors ( ∈ ℝ̂ ×16 , ∈ ℝ̂ ×16 , and ∈ ℝ̂ ×16 ), and further combined into a vector ∈ ℝ̂ ×16×3 . The feature is input into FNL to achieve unified representation. We denoted the − ℎ feature from − ℎ branch as ∈ ℝ 40 . Finally, all the features from all the branches were concatenated as one vector : After these procedures above, we obtain the asymmetric features on multiple time scales, . The would be input into the feature concatenation and classification block along with .

Feature classification block
Through the previous procedures, we obtain two groups of features, i.e., and . These features are first concatenated to a integrated vector , which is called final feature map. The final feature map will be used to visualize the model in the following analysis. Through a dropout with the rate of 0.7, the feature is input into a FC layer with two neurons. The output feature is regarded as the possibility( ( | ), = 0, 1) that the EEG data belongs to each class. The predicted label is that of the class which has maximal possibility. The procedure could be described as: = arg max ( | ), = 0, 1.
where the ( | ) is the possibility of D belonging to the c-th class.

Baseline methods
To verify the performance of our model, we compared the MSBAM with ten baseline methods, which were evaluated on the DEAP or DREAMER dataset. These methods are introduced briefly as follows: The method of Liu-2019. Liu et al. introduced a method named deep canonical correlation analysis(DCCA) Liu, Qiu, Zheng and Lu (2019). The raw EEG signals and peripheral physiological features were transformed by different nonlinear networks to obtain the two groups of features. These features were fused by a weighted sum method after being regularized with the traditional CCA method. Then, the fusion features were used to train a linear SVM classifier for emotion recognition. Evaluated on DEAP, the DCCA method achieved accuracies of 84.33% and 85.62% on arousal and valance dimensions.
The method of Qiu-2018. Qiu et al. proposed a multimodal emotion recognition method named correlated attention network (CAN) Qiu, Li and Hu (2018). They extracted features with two Bidirectional Gated Recurrent Unit neural networks, and applied a canonical correlation analysis to calculate the correlation. In the end, the attention mechanism was utilized to extract the features that are important to represent emotional states. Finally, a softmax layer is applied to complete the emotion classification task. Their method achieved accuracies of 84.79% and 86.45% on the arousal and valance dimensions of DEAP dataset.
The method of Yin-2021. Yin et al. proposed an ECL-GCNN model that fused LSTM and GCNN for emotion classification Yin et al. (2021a). In the GCNN, the EEG channels and functional connections between two channels were denoted as the vertex nodes and edges, respectively. The greater value of the edge means the closer relationship between two channels. The features extracted by the GCNNs were input into LSTM networks to extract the higher level features for emotion classification tasks. They attained the accuracies of 90.45% and 90.60% on the valance and arousal dimensions of DEAP dataset.
The method of Yang-2018. Yang et al. introduced a preprocessing method for baseline correction, and validated it with their new model termed as parallel convolutional recurrent neural network (PCRNN) Yang et al. (2018a). In this model, the pre-processed 2D EEG signals were transformed into a 3D representation according to the spatial topology of the electrodes. A group of CNNs and LSTMs were then utilized to extract the spatial and temporal features, respectively. To classify the emotion states, these features were concatenated and input into a fully connected layer. This method yields accuracies of 90.80% and 91.03% on the valance and arousal dimensions of DEAP dataset.
The method of Liao-2020. Liao et al. proposed a multimodal emotion recognition method Liao, Zhong, Zhu and Cai (2020). They transformed the raw EEG signal into 3D representation as in the studied by Yang et al Yang et al. (2018a). 2D-CNN kernels were employed to extract the spatial features of EEG signals, and LSTM were utilized to extract the temporal features of peripheral physiological signals. The concatenated spatial and temporal features were fed into a softmax classifier to predict the emotion states. This method achieved accuracies of 91.95% and 93.06% on the valance and arousal dimensions of DEAP dataset.  Ma et al. (2019b). MMResLSTM contained four LSTMs, and the last three of them had a residual structure. Multimodal data were respectively fed into two MMResLSTM modules that shared parameters to extract the high-level features, and then concatenated to predict the emotion state by a softmax layer. MMResLSTM attained accuracies of 92.30% and 92.87% for valance and arousal of DEAP dataset.
The method of Huang-2021. Huang et al. released a method based on the discrepancy of emotional response between two hemispheres, which was named bi-hemisphere discrepancy convolutional neural network (BiDCNN) Huang et al. (2021). Three different matrices were constructed and input into the BiDCNN to extract the spatial and temporal features, including the discrepancy features of emotional responses between left and right hemispheres. In the end, all the features were concatenated and fed into a series of Conv layers and FC layers to predict the emotion state. Evaluated on DEAP, Their method achieved accuracies of 94.38% and 94.72% on arousal and valance dimensions.
The method of Li-2021. Li et al. proposed a method named dilated fully convolutional networks (DFCN) Li et al. (2021a). They filtered the raw data into four frequency bands, and calculated three kind of features, i.e., Kurtosis feature, Power feature, and DE feature, in each frequency band. Those features were rearranged to a 3D representation. The DFCN contains two convolution layers, three dilated convolution layers and two linear layers. Besides, they introduced Spectral Norm Regularization (SNR) to reduce the sensitivity of distribution. This method yields 94.59%, 95.32%, 94.78% and 95.19% accuracies on the valance arousal dominance and liking dimensions of DEAP dataset, and achieved accuracies of 93.15%, 91.30% and 92.04% on the valance, arousal and dominance dimensions of DREAMER dataset.
The method of Cui-2020. Cui et al. introduced a method named RACNN Cui et al. (2020). They employed continuous 1-D convolution to learn temporal representations. Then two parallel branches were implemented. The first branch was designed to capture the regional information between adjacent channels, and the second one was to capture the discriminative feature between the two hemispheres of the brain. At last, they concatenated the features to recognize the emotion state. They conducted the experiments on valance and arousal dimensions of both DEAP and DREAMER datasets, and yields average accuracies of 96.65%/97.11% and 95.55%/97.01% respectively.

Model implementation
For the MSBAM, the cross-entropy was employed as the loss function. Adam optimizer was utilized to minimize the loss function with 0.001 learning rate initially. The experiment performed 50 epochs, and the learning rate was reduced to 0.0001 at the 30th epoch. During implementing the MSBAM model, the number of branches was set to 2, and the parameters of time scales ( ) in the three branches in temporal feature extractor were set to 128 and 64, respectively.
The experiments were independently conducted in each emotion dimension of the two datasets. The average accuracy in each dimension was used to evaluate the performance of MSBAM and the compared baseline models. The experimental steps are described as follows, and the flow-chart is illustrated in Fig. 6 • Randomly split the EEG segments of each subjects into 10 equal parts.
• Implement 10-fold cross validation experiments for each subject based on the data generated in the above step, and obtain the accuracy of -th subjects, denotes as 1 , 2 , ..., 10 .
• Compute the average accuracy over all subjects as , where is the number of subjects in the dataset.

Results
We respectively conducted the experiments in the dimensions of valence, arousal, dominance and liking for the DEAP dataset, and valence, arousal, dominance for DREAMER dataset with MSBAM and the baseline methods. The experimental results of all methods are summarized in Table 1 and Table 2. In Table 2, the symbol "*" denotes the result reproduced by us, due to the results on current dataset are not provided in the original paper (similarly hereinafter). We used the hyper-parameters provided by the original paper and empirically set parameters that were not provided when we reproduced the model. We could find that our MSBAM yields better performance than the baseline methods. Compared with the best baseline model, MSBAM could improve average recognition accuracy by 2.71%, 2.26%, 4.61% and 4.27% in valence, arousal, dominance and liking dimensions of DEAP, and 1.04%, 0.94% and 0.72% in valence, arousal and dominance dimensions of DREAMER, respectively. In addition, MSBAM achieves smaller standard deviation than the best baseline method. It indicates that MSBAM has better robustness than the baseline methods. These experiments were conducted to verify the effectiveness of the MSBAM, and compare its performance with the baseline methods. To observe the performance of proposed MSBAM of each subject, two line charts are presented in Fig. 7. In the charts, we present the results on DEAP dataset from four baseline methods, i.e., − 2020 and − 2020, − 2021 and − 2021. We could find that all the methods yield analogical tendencies as the subject changes. The MSBAM shows better performance than other models. It could yield the better accuracies on all subjects.
To further investigate the results of MSBAM, the features before the last FC layer(final feature map) in Valence dimension of DEAP dataset were visualized by the -SNE Van der Maaten and Hinton (2008). In this work, we reproduced three baseline methods to compare with the proposed MSBAM. Six subjects with lowest accuracies were selected. The features of -SNE are shown in Fig. 8. The first row in the figure illustrated the features obtained from the original data. The second to the last row presented the final feature maps of test set in the first fold of each subject. We could observe that the features extracted in the proposed MSBAM have better intra-category similarity and intercategory separability than the baseline method, which can facilitate the classification.

Discussions
In current study, we proposed a method MSBAM based on the physiological mechanisms of emotion and the characteristics of EEG to classify the two-class classification in four emotional dimensions, i.e., arousal, valence, dominance and liking, respectively. In each dimension, the EEG data was assigned a low-level or high-level states based on a threshold of the self-rating scores. The 10-fold cross validation was used in intra-subject classification experiments.
The experimental results demonstrate that the proposed MSBAM model outperforms the compared baseline methods. This may be attributed to the model structure that extracts the features of multi-scale and bi-hemispheric asymmetry, simultaneously. To further verify the rationality of the MSBAM, several ablation experiments were conducted on the two datasets. In the following discussion, we will take the DEAP dataset as an example. First, we removed all FNL in the model to verify the effectiveness of the normalization method before feature fusion. Then, we further checked whether the spatial feature extractor and two branches in temporal feature extractor block were indispensable. The results of several ablation experiments are shown in Table 3. MSBAM-ri means that the i-th branch in the temporal feature extractor block was removed, and the MSBAM-rSpat means the spatial feature extractor was removed, in the original MSBAM.
As shown in Table 3, we could find that FNL played an important role in the model. One possible explanation is that the FNL is able to scale the features of each branch into the same shape and value domain. Because of the discrepancy of convolution kernels, the feature maps of each branch have different length and value range. These feature maps were transformed to 40 features with the range of 0 to 1 after being processed by the FNL. This procedure may reduce the influence of the weight bias introduced by the discrepancy. In brief, removing all parts of the branches in temporal feature extractor block, or the spatial feature extractor block will reduce the accuracies obtained by the original MSBAM. These results indicate that MSBAM performs better than its variant.
To further evaluate the performance of MSBAM, we further conducted the multiple classification tasks of the four categories of emotions, i.e., high valence and high arousal, high valence and low arousal, low valence and high arousal, and low valence and low arousal, as in previous study Cao, Hao, Wang, Gao and Xiang (2020). The MSBAM yielded the accuracy of 99.20 ± 0.86.
Although the MSBAM shows better performance than all the baseline methods, some limitations should be mentioned. In the Table 1, we could find that all methods can obtain accuracy above 80%. However, there exists a problem of data leakage because of the approach to obtaining the training data and testing data for all the trial-dependent method, which is a common phenomenon in existing DL models Ding, Robinson, Zhang, Zeng and Guan (2022); Wang, Liu, Qi, Deng and Li (2021). A few studies try to evaluate the DL models on more challenging scenarios, trial-independent classification within subject Ding et al. (2022); Wang et al. (2021) and subject-independent classification Hu, Wang, Jia, Bu, Sutcliffe and Feng (2021); Zhao, Yan and Lu (2021a). Those two scenarios should be adopted in future studies when we evaluate the DL models to avoid data leakage and biased evaluation.

Conclusion
In this paper, we propose an emotion recognition method MSBAM to extract the multi-scale and bi-hemispheric asymmetric EEG features for emotion classification. The proposed MSBAM could achieve average accuracies of 99.36%, 99.37%, 99.39% and 99.46% in Valence, Arousal, Dominance and Liking dimensions on DEAP dataset and 99.69%, 99.76% and 99.79% in Valence, Arousal, and Dominance dimensions on DREAMER dataset, respectively, which outperformed all the baseline methods in the subjectdependent classification scenario. Although some limitations should be addressed in future studies, current study demonstrates that exploring the multi-scale features and utilizing the neural mechanism of the emotions such as the bi-hemispheric asymmetry to design the DL model, are beneficial for emotion recognition.