A hierarchical approach with feature selection for emotion recognition from speech

We examine speaker independent emotion classification from speech, reporting experiments on the Berlin database across six basic emotions. Our approach is novel in a number of ways: First, it is hierarchical, motivated by our belief that the most suitable feature set for classification is different for each pair of emotions. Further, it uses a large number of feature sets of different types, such as prosodic, spectral, glottal flow based, and AM-FM ones. Finally, it employs a two-stage feature selection strategy to achieve discriminative dimensionality reduction. The approach results to a classification rate of 85%, comparable to the state-of-the-art on this dataset.


Introduction
Automatic emotion recognition from speech has attracted significant interest in recent years, aiming at improved human-computer interaction. Several approaches have been proposed in the literature, varying in the types and number of features employed, the classifier used, and the system developed. For example, there have been works using traditional features from the speech recognition literature or higher level voice quality ones (He et al., 2010), some adopting flat and simple systems, and others that employ complex classifiers and systems of a hierarchical form (Mao and Zhan, 2010;Shaukat and Chen, 2008). Up to now, most of the highest performing speaker independent emotion recognition systems use large feature sets and rather complex classifiers (Schuller et al., 2006;Lee and Narayanan, 2005). For example, Schuller et al. (2006) extract a large set of 4k features, achieving an average recognition rate of 87% on the Berlin database of emotional speech (Burkhardt et al., 2005). The proposed approach in this paper achieves comparable performance with the best research efforts by using only 112 features in total. We accomplish this by designing a system that is based on smaller and more specially trained sub-systems that focus on pairs of emotions. We also use a two-stage feature selection scheme, in contrast to the stateof-the-art simple sequential selection. We finally involve feature sets that are not typically employed in the emotion recognition literature, such as glottal flow features and AM-FM ones. Thus, although the total number of features remains small, there is much variety in their types. The rest of the paper is organized as follows: First, in Section 2, we describe the different feature sets considered, together with their basic theoretical background. In Section 3, we present our two-stage feature selection approach, and, in Section 4, we explain how the final system works. Following these, in Section 5, we present our experiments and associated results. Finally, we conclude the paper with a short summary in Section 6.

Feature Extraction
A number of feature sets are considered in our approach. In more detail they fall within the following four categories:

Prosodic Features
Such features are strongly related to the emotional state of the speaker and are extensively used in the literature. In this paper, we extract two features relevant to prosody intonation and intensity. First, we calculate the pitch contour of the utterance using the RAPT algorithm (Talkin, 1995), and, second, we compute signal energy to obtain information about speech intensity.

Spectral Features
As in the state-of-the-art in speech recognition, we employ the Mel frequency cepstral coefficients (MFCCs), together with their first-order derivatives (Young et al., 2002). In addition, we compute the zero crossing rate (ZCR) of each frame (Young et al., 2002).

Glottal Flow Features
The volume velocity of air-flow through the glottis is the excitation source for voiced speech. The glottal flow is related with several voice quality features, such as breathiness, harshness, and creakiness, and therefore it provides useful information about the emotional state of the speaker. The estimation of the glottal flow is based on Fant's sourcefilter theory, according to which the voice excitation and the vocal tract are linearly separable. In this manner, speech production can be modeled by a cascade of linearly separable filters. In order to obtain the glottal flow, we perform inverse filtering by using the IAIF algorithm, which employs the discrete all pole (DAP) method to model the vocal tract and then cancels it iteratively to obtain an estimate of the glottal flow waveform (Airas et al., 2005). Once the glottal flow is obtained, we extract time and frequency based features from its waveform, such as the: [ 112 ] [ 40 ] [ 10 ]

Feature Selection ( 2 stages )
MRMR selection backward feature selection wrapper Figure 1: The two-stage feature selection scheme employed here. Numbers in brackets depict feature dimensionality.
• Open quotient, which is the ratio of the time in which the vocal folds are open and the whole pitch period duration.
• Speed quotient, defined as the ratio of rise and fall time of the glottal flow.
• Normalized amplitude quotient that is used to parametrize the glottal closing phase.
• Harmonic richness factor, which is the ratio of higher harmonics to the first harmonic.
In total, we extract 18 time based and 4 frequency based features.

AM-FM Features
The AM-FM model considers decomposing the speech signal into a series of a few instantaneous frequency and amplitude signals. These signals can be considered as timefrequency distributions, containing acoustic information that is not captured by the linear speech model (Potamianos and Maragos, 1999). Following the approach in (Dimitriadis and Maragos, 2005), we model each speech sound by six AM-FM signals, estimating their parameters by the energy separation algorithm (ESA). At the end, twelve (2×6) parameters are obtained for each utterance frame.
We should note that, for all aforementioned feature sets, we only utilize their first and second order statistics (mean and variance).

Feature Selection Strategy
Feature selection is a very crucial step in pattern recognition problems to counter the curse of dimensionality. It does so, avoiding feature transformations such as PCA or LDA, obtaining instead a subset from the initial set of features that is most relevant to the classification problem at hand. In general, this requires a search strategy to select candidate subsets and an objective function to evaluate such candidates. Depending on their objective function, feature selection algorithms can be divided into filters and wrappers. Filters evaluate feature subsets by their information content, typically statistical dependence or informationtheoretic measures. Wrappers, in contrast, evaluate the subsets by their classification rate on test data. One can claim that filters have better generalization properties, as they are not related to any classifier. On the other hand, wrappers can interact with a specific classifier and find a subset that is also appropriate for the problem at hand. Two of the simplest and most used wrappers belong to sequential feature selection algorithms, and are forward feature selection (FFS) and backward feature selection (BFS). FFS starts from the empty set and sequentially adds the feature that results in the highest recognition rate, when combined with the features that have already been selected. In contrast, BFS starts from the full feature set and sequentially removes the feature that leaves a subset with the highest recognition rate. Both constitute greedy approaches and may be trapped in local minima. In our experiments, we compare these two wrappers and show that BFS performs better, especially when the initial set of features is large. In our system we use a two-step feature selection scheme that takes advantage of both generalization properties of filter algorithms and classifier-adaptive properties of wrappers. As a filter we employ the maximum relevanceminimum redundancy (MRMR) algorithm (Peng et al., 2005). This method tries to select a feature set that has maximum relevance with the two emotions involved in each two-class emotion classification sub-problem (see Section 4), as well as minimum intra-redundancy in terms of mutual information between its features. In more detail, if S denotes a set of selected features within the set of all possible features Ω, then a measure of its redundancy is given by where I(i, j) represents the mutual information between features i and j, and |S| denotes the number of features in set S. The minimum redundancy criterion seeks the set of features S that minimizes W S . Next, if c ∈ C denotes the class of interest (for the two-class emotion classification problem we will have C = {c 1 , c 2 }), we can calculate the relevance of feature set S as The maximum relevance criterion seeks feature set S that maximizes V C, S for the specific classification problem C.
The MRMR algorithm then tries to achieve both of the two previous criteria by maximizing the following quotient: .
Obtaining the optimal solution to the above through exhaustive search is clearly intractable. In practice, one proceeds with a sequential, incremental, non-optimal approach, by first selecting as the first feature the one that maximizes the relevance criterion, and subsequently adding happiness anger neutral sadness fearanxiety boredom Figure 3: The five sub-systems for recognizing "happiness", denoted by the five lines connecting the circles.
one feature at a time, similarly to the FSS approach mentioned earlier. Assuming that the selected set at the current iteration is S, then feature i ∈ Ω − S will be selected as The process terminates, once we arrive at the desired feature set size. We should also note that, in the above, all mutual information quantities are computed after appropriate discretization of all continued-valued features. The two-stage feature selection scheme employed in our approach is depicted in Figure 1.

Final System
Our emotion classification system has a hierarchical structure, as it is composed from multiple specialized systems. Each one of the smaller systems is trained to distinguish between only one pair of emotions (e.g., fear vs. happiness, anger vs. happiness, etc.), and it is trained separately, as depicted in Figures 2 and 3. Since, in this paper, we concentrate on six basic emotions (see Section 5), the proposed approach gives rise to 15 sub-systems. Each such sub-system is implemented based on a simple linear support vector machine (SVM) classifier. Majority voting over the sub-system classification results is then used to provide the final classification. In addition, we use the a-priori knowledge of gender information, as such has been shown to play an important role in emotion recognition in the literature.

Experiments and Results
In our experiments we used the well-known Berlin database of emotional speech. The corpus has been recorded at the Technical University of Berlin, and it consists of 493 utterances by 10 professional actors (5 male and 5 female). It contains seven emotions (acted), namely anger, happiness, sadness, fear, boredom, neutral, and disgust (Burkhardt et al., 2005), of which we concentrate on the first six in accordance with similar work in the literature. We follow a leaveone-speaker-out experimental paradigm to provide speaker independence. In the following tables we present results in the form of confusion matrices for several different experiments. From these, we can immediately observe the superiority of the gender dependent vs. the gender independent approach. Clearly, a-priori gender knowledge significantly improves   algorithm performance. Furthermore, we observe the gradual improvement of recognition accuracy of almost all emotion classes, first when applying the BFS scheme in place of FSS, and subsequently when employing the proposed twostage approach that includes the MRMR algorithm. One should note that these results are in par with human emotion perception experiments reported at 84.3% by Schuller et al. (2007). Next, in Figure 4, we show some results on the several subsystems performance with different initial features sets. As we can observe, the addition of glotal and AM-FM features lead to better recognition results. In a few sub-systems performance is somewhat better when using only MFCC and prosodic features. This demonstrates that such features are sufficient for the specific sub-systems. Finally, concerning the types of features selected by the proposed algorithm, it should be noted that different feature combinations are selected for each subsystem, a fact that backs our hierarchical approach. For example, when one of the two classes of interest is fear, more glottal features are selected.

Conclusions
In this paper, we proposed a hierarchical classification system that is based on the discriminative power of its appropriately trained sub-systems. These sub-systems employ feature sets of various types appropriately selected through a two-stage feature selection algorithm. Finally we took advantage of the well-established method of gender dependent systems to achieve better results.
Our final system achieved an overall classification accuracy of 85.18% that is comparable to the state-of-the-art in the field. Both glottal flow and AM-FM features were selected by the feature selection scheme employed and indeed improved recognition results in comparison to the state-of-   Table 4: Confusion matrix for the gender independent experiment with the final proposed feature selection scheme (backward selection followed by MRMR). The overall accuracy is 80.09%.
the-art systems that employ only MFCCs and prosodic features. Finally, the combination of the MRMR filter based algorithm with the BFS wrapper based one outperformed simple sequential feature selection.
In future work, we will investigate the use of temporal feature information (feature contours), as opposed to just their means and variances. Furthermore, we will consider larger datasets and spontaneous speech data.