A novel database of Children's Spontaneous Facial Expressions (LIRIS-CSE)

Computing environment is moving towards human-centered designs instead of computer centered designs and human's tend to communicate wealth of information through affective states or expressions. Traditional Human Computer Interaction (HCI) based systems ignores bulk of information communicated through those affective states and just caters for user's intentional input. Generally, for evaluating and benchmarking different facial expression analysis algorithms, standardized databases are needed to enable a meaningful comparison. In the absence of comparative tests on such standardized databases it is difficult to find relative strengths and weaknesses of different facial expression recognition algorithms. In this article we present a novel video database for Children's Spontaneous facial Expressions (LIRIS-CSE). Proposed video database contains six basic spontaneous facial expressions shown by 12 ethnically diverse children between the ages of 6 and 12 years with mean age of 7.3 years. To the best of our knowledge, this database is first of its kind as it records and shows spontaneous facial expressions of children. Previously there were few database of children expressions and all of them show posed or exaggerated expressions which are different from spontaneous or natural expressions. Thus, this database will be a milestone for human behavior researchers. This database will be a excellent resource for vision community for benchmarking and comparing results. In this article, we have also proposed framework for automatic expression recognition based on convolutional neural network (CNN) architecture with transfer learning approach. Proposed architecture achieved average classification accuracy of 75% on our proposed database i.e. LIRIS-CSE.


Introduction
Computing paradigm has shifted from computer-centered computing to humancentered computing [1]. This paradigm shift has created tremendous opportunity for computer vision research community to propose solution to existing problems and invent ingenious applications and products which were not though of before. One of the most important property of human-centered computing interfaces is the ability of machines to understand and react to social and affective or emotional signals [2,3].
Mostly humans express their emotion via facial channel, also known as facial expressions [3]. Humans are blessed with the amazing ability to recognize facial expression robustly in real-time but for machines it still is a difficult task to decode facial expressions. Variability in pose, illumination and the way people show expressions across cultures are some of the parameters that make this task more difficult [4].
Another problem that hinders the development of such system for real world applications is the lack of databases with natural displays of expressions [5]. There are number of publicly available benchmark databases with posed displays of the six basic emotions [6] i.e. happiness, anger, disgust, fear, surprise and sadness, exist but there is no equivalent of this for spontaneous / natural basic emotions. While, it has been proved that spontaneous facial expressions differ substantially from posed expressions [7].
Another issue with most of publicly available databases is absence of children in recorded videos or images. Research community has put lot of efforts to built databases of emotional videos or images but almost all of them contain adult emotional faces [8,9]. By excluding children's stimuli in publicly available databases vision research community not only restricted itself to application catering only adults but also produced limited study for the interpretation of expressions developmentally [10].

Publicly available databases of children's emotional stimuli
To the best of our knowledge there are only three publicly available databases that contains children emotional stimuli / images. They are: 1. The NIMH Child Emotional Faces Picture Set (NIMH-ChEFS) [11] 2. The Dartmouth Database of Childrens Faces [12] 3. The Child Affective Facial Expression (CAFE) [10] The NIMH Child Emotional Faces Picture Set (NIMH-ChEFS) [11] database has 482 emotional frames containing expressions of "fear", "anger", "happy" and "sad" with two gaze conditions: direct and averted gaze. Children that posed for this database were between 10 and 17 years of age. The databases is validated by 20 adult raters.
The Dartmouth Database of children Faces [12] contains emotional images (six basic emotions) of 40 male and 40 female Caucasian children between the Six universal expressions: first row show example images from the Dartmouth database [12]. Second row show emotional images from the Child Affective Facial Expression (CAFE) [10], while last row show emotional images from the movie clip of our proposed database, LIRIS-CSE. First column corresponds to expression of "happiness", second column corresponds to expression of "surprise", third column corresponds to expression of "sadness", fourth column corresponds to expression of "anger", fifth column corresponds to expression of "fear" and last column corresponds to expression of "disgust". Expressions in our proposed database are spontaneous and natural and can easily be differentiated from posed / exaggerated expressions of the other two database.
ages of 6 and 16 years. All facial images in the database were assessed by at least human 20 raters for facial expression identifiability and intensity. Expression of happy was most accurately identified while fear was least accurately identified by human raters. Human raters correctly classified 94.3% of the happy faces while expression of fear was correctly identified in 49.08% of the images, least identifiable by human raters. On average human raters correctly identified expression in 79.7% of the images. Refer Figure 1 for examples images from the database.
The Child Affective Facial Expression (CAFE) database [10] is composed of 1192 emotional images (six basic emotions and neutral) of 2 to 8 years old children. Children that posed for this database were ethnically and racially diverse. Refer Figure 1 for examples frames from the database.

Weaknesses of publicly available databases of children's emotional stimuli
Although above describe children expression databases are diverse in terms of pose, camera angles and illumination but have following drawbacks: 1. Above mentioned databases contain posed expressions and as mentioned before that spontaneous or natural facial expressions differ substantially from posed expressions as they exhibit real emotion whereas, posed expressions are fake and disguise inner feelings [7,13].
2. All of these databases contains only static images / mug shots with expression at peak intensity. According to study conducted by psychologist Bassili [14] it was concluded that facial muscle motion/movement is fundamental to the recognition of facial expressions. He also concluded that human can robustly recognize expressions from video clip than by just looking at mug shot. 3. All of the above mentioned databases for children facial expressions present few hundred to maximum 1200 static frames. In current era where requirement of amount of data for learning a concept / computational model (deep learning [15]) has increased exponentially, only few hundred static images are not enough. Generally, for evaluating and benchmarking different facial expression analysis algorithms, standardized databases are needed to enable a meaningful comparison. In the absence of comparative tests on such standardized databases it is difficult to find relative strengths and weaknesses of different facial expression recognition algorithms. Thus, it is utmost important to develop natural / spontaneous emotional database contains children movie clip / dynamic images. This will allow research community to built robust system for children's natural facial expression recognition. Thus, our contributions in this study are two-fold : 1. We are presenting a novel emotional database (LIRIS-CSE) that contains 208 movie clip / dynamic images of 12 ethnically diverse children showing spontaneous expressions in two environments, i.e. 1) lab / classroom environment 2) home environment (refer Figure 2). 2. We have also proposed a framework for automatic facial expression recognition based on convolutional neural network (CNN) architecture with transfer learning approach. Proposed architecture achieved average classification accuracy of 75% on our proposed database.

Novelty of proposed database (LIRIS-CSE)
To overcome above mentioned drawbacks of databases of children's facial expression (refer Section 2.1), we are presenting a novel emotional database that contains movie clip / dynamic images of 12 ethnically diverse children. This unique database contains spontaneous / natural facial expression of children in diverse settings (refer Figure 2 to see variations in recording scenarios) showing six universal or prototypic emotional expressions ("happiness", "sadness", "anger", "surprise", "disgust" and "fear") [16,17]. Children are recorded in constraint free environment (no restriction on head movement, no restriction on hands movement, free sitting setting, no restriction of any sort) while they watched specially built / selected stimuli. This constraint free environment allowed us to record spontaneous / natural expression of children as they occur. The database has been validated by 22 human raters. Details of recording parameters are presented in Table 2. In comparison with above mentioned databases for children facial expressions that have only few hundred images, our database (LIRIS-CSE) contains 26 thousand (26,000) frames of emotional data, refer Section 4.3 for details.
The spontaneity of recorded expressions can easily be observed in Figure  1. Expressions in our proposed database are spontaneous and natural and can easily be differentiated from posed / exaggerated expressions of the other two databases. Figure 3 shows facial muscle motion / transition for different spontaneous expressions.

Participants
In total 12 (five male and seven female children) ethnically diverse children between the ages of 6 and 12 years with mean age of 7.3 years participated in our database recording session. 60 % of recordings are done in classroom / lab environment, while 40% of the clips in the database are recorded in home conditions. Recording children in two different environments has been done to have different background and illumination conditions in the recorded database. Refer Figure 2 for example images with different backgrounds and illumination conditions.

Database acquisition details
First step for the creation of proposed spontaneous expressions database was the selection of visual stimuli that can induce emotions in children. Considering ethical reasons and young age of children we carefully selected stimuli and removed any stimuli that can have long term negative impact on the children. Due to these ethical reasons we did not include emotion inducer clips for the negative expression of "anger" and selected very few clips to induce emotion of "fear" and "sadness". The same has been practiced before by Valstar et. al [18]. Due to this very reason the proposed database contains more emotional clips of expressions of "happiness" and "surprise". Although there were no emotion inducer clips for the expression of "anger" but still database contains very few clips where children show expression of "anger" (refer Figure 1) due to the fact that young children use expressions of "disgust" and "anger" interchangeably [19].

Emotion inducing stimuli
We either selected only animated cartoon / movies or small video clips of kids doing funny actions to stimuli list. The reasons for selecting videos to induce emotions in children are as follows: 1. All the selected videos for inducing emotions contains audio as well. Video stimuli along with audio gives immersive experience, thus is powerful emotion inducer [20].
2. Video stimuli provides more engaging experience then static images, restricting undesirable head movement.
3. Video stimuli can evoke emotions for a longer duration. This helped us in recording and spotting children facial expressions.
List of stimuli selected as emotion inducers are presented in Table 1. Total running length of selected stimuli is 17 minutes and 35 seconds. One of the consideration for not selecting more stimuli is to prevent children's lose of interest or disengagement over time [21].

Recording setup
Inspired by Li et al. [20], we setup high speed webcam, mounted at the top of laptop with speaker output, at a distance of 50 cm. As explained above, The audio output enhanced the visual experience of a child, thus helping us induce emotions robustly. Recording setup is illustrated in Figure 4. As mentioned in

Video segmentation
After recording video for each child we carefully examined the recoding and removed any unnecessary recorded part, usually at the beginning and at the end of video recording. As the video (with audio) stimuli that children watched was the combination of different emotional videos (refer Section 4.1 for the details of visual stimuli), our recorded video contained whole spectrum of expressions in one single video. We then manually segmented one single video recording of each child into segments of small video chunks / clips such that each video clip show one pronounced expression. Refer Figure 3 to see results after segmentation process. It can be observed from the referred figure that each small video clip contains neutral expression at the beginning, then shows onset of an expression, and finishes when expression is visible at its peak along with some frames after peak expression frame. Total number of small video clips, each containing specific expression, present in our database (LIRIS-CSE) are 208. Total running length of segmented clips in presented database (videos having children facial expressions) is seventeen minutes and twenty-four seconds. That makes total of around 26 thousand (26,000) frames of emotional data, considering recording is done at 25 frames / second, refer Table 2. Figure 5: Blended Expressions. Example images that show co-occurrence of more than one expressions in a single clip of the database. First row present frames from a clip that show occurrence of expressions of "sadness" and "anger". Similarly, second row shows co-occurrence of expressions of "surprise" and "happiness".
There are seventeen (17) video clips present in this database that have two labels, for example "happily surprised", " Fear surprise" etc. This is due to the fact that for young children different expressions co-occur / blended expressions [22,19] or a visual stimuli was so immersive that transition from one expression to another expression was not pronounced. Refer Figure 5 to see example images from segmented video segments that show blended expressions.

Database validation
The database has been validated by 22 human raters / evaluators between the ages of 18 and 40 years with mean age of 26.8 years. 50% of database raters / evaluators were in the age bracket of [18 -25] years and rest of 50% were in the age bracket of [26 -40] years. Human evaluators who were in the age bracket of [18 -25] years were university students and other group of evaluators were university faculty members. Human raters / evaluators were briefed about the experiment before they started validating the database.
For database validation purpose we built software that played segmented video (in random order) and records human evaluator choice of expression label. The screen capture of the software is presented in Figure 6. If required, evaluator can play video multiple times before recoding their choice for any specific video.
In summary, instructions given to human raters / evaluators were following: 1. Watch carefully each segmented video and select expression that is shown in the played segmented video.
2. If played video did not show any visible / pronounced expression, selected an option of "undecided". Each video can be played multiple times without any upper bound on number of times video to be played.
3. Once expression label / response is submitted for a played segmented video then this label can not be edited.

Validation data analysis
After validation data collection, we performed statistical analysis on the gathered data and calculated confusion matrix. Refer Figure 7 to see calculated confusion matrix. Rows in the referred confusion matrix show induced intended expressions (average%), while columns show expression label given by human raters / evaluators (average %). Diagonal values represent agreement between induced intended expressions and expression label given by human evaluators, with darker colors representing greater agreement.
As per calculated results expression of "happy" was most correctly spotted by evaluators, with average accuracy of 73.2%. On the other hand expression of " fear" was least correctly identified by evaluators, with average accuracy of 40.1%. These results are consistent with results from [12,16]. We did not include expression of "anger" in analysis as there is only one movie clip with anger expression in the database.
Expression of "fear" which is least identified, is often perceptually mixed with expressions of "surprise" and "disgust". As mentioned above, this is due to the fact that for young children different expressions co-occur (blended expressions) [22,19] or a visual stimuli was so immersive that transition from one expression to another expression was not pronounced. Refer Figure 5 to see example images from segmented video segments that show blended expressions.
Overall average accuracy of human evaluators / raters is 55.3%. As per study published by Matsumoto et al. [23] human's usually can spot expressions correctly 50% of the time and the easiest expression for human's to identify are "happy" and "surprise". These results conforms well with the results that we obtained from human evaluators as expression of "happy" was most correctly identified while average accuracy of human evaluators raters is also around 50% (55.3% to be exact).

Database availability
The novel database of Children's Spontaneous Expressions (LIRIS-CSE) is available for research purposes only. It can be downloaded by researcher / lab after signing End User License Agreement (EULA). Website to download LIRIS-CSE database is: https://childrenfacialexpression.projet.liris.cnrs.fr/.

Automatic recognition of affect, a transfer learning based approach
In order to provide benchmark machine learning / automatic classification of expression results on our database (LIRIS-CSE), we have done experiment based on transfer learning paradigm. Usually machine learning algorithms make prediction on data that is similar to what algorithm is trained on; training and test data are drawn from same distribution. On the contrary transfer learning allows distributions used in training and testing to be different [24]. We used transfer learning approach in our experiment due to following facts: 1. We wanted to benefit from deep learning model that has achieved high accuracy on recognition tasks that take image as input i.e. ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) [25], and is available for research purposes.
2. It requires large database to train deep learning model from the scratch [26]. In a given scenario, we can not train deep learning model from the very beginning.
The architecture of CNN was first proposed by LeCun [15]. It is a multi-stage or multi-layer architecture. This essentially means there are multiple stages in CNN for feature extraction. Every stage in the network has an input and output which is composed of arrays known as feature maps. Every output feature map consists of patterns or features extracted on locations of the input feature map. Every stage is made up of layers after which classification takes place [35,36,37]. Generally, these layers are: 1. Convolution layer: This layer makes use of filters, which are convolved with the image, producing activation or feature maps. 2. Feature Pooling layer: This layer is inserted to reduce the size of the image representation, to make the computation efficient. The number of parameters is also reduced which in turn controls over-fitting. 3. Classification layer: This is the fully connected layer. This layer computes the probability / score learned classes from the extracted features from convolution layer in the preceding steps.

VGGNet architecture and transfer learning
Since 2012, deep Convolution Networks (ConvNets) have become a focus of computer vision scientists. Various architectures were proposed to achieve higher accuracy for a given tasks. For example, best submissions to ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) [25], [39,40] proposed to use a smaller receptive window size / smaller filter size and smaller stride in the first convolutional layer. Generally, ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has served as a platform for advancements in deep visual recognition architectures.
The best proposed ConvNets architectures for ILSVRC 2014 competition were GoogleNet (a.k.a. Inception V1) from Google [41] and VGGNet by Simonyan and Zisserman [38]. GoogleNet contains 1 x 1 Convolution at the middle of the network and global average pooling was used at the end of the network instead of using fully connected layers, refer Section 6.1 for discussion on different types of layers. VGGNet consists of 16 convolutional layers (VGG16). It is one of the most appealing framework because of its uniform architecture, refer  VGG16 architecture (refer Figure 8) takes image of 224 x 224 with the receptive field size of 3 x 3. The convolution stride is 1 pixel and padding is 1 (for receptive field of 3 x 3). There are two fully connected layers with 4096 units each, the last layer is a softmax classification layer with x units (representing x classes / x classes to recognize) and the activation function is the rectified linear unit (ReLU). The only downside of VGG16 architecture is its huge number of trainable parameters. VGG16 consists of 138 million parameters.

Experimental framework and results
As discussed earlier, CNN requires large database to learn concept [42,26], making it impractical for different applications. This bottleneck is usually avoided using transfer learning technique [43]. Transfer learning is a machine learning approach that focuses on ability to apply relevant knowledge from previous learning experiences to a different but related problem. We have used transfer learning approach to built framework for expression recognition using our proposed database (LIRIS-CSE) as the size of our database is not sufficiently large to robustly train all layers of CNN from the very beginning. We used pretrained VGG model (VGG16, a 16 layered architecture) [38], which is a deep convolutional network trained for object recognition [32]. It is developed and trained by Oxford University's Visual Geometry Group (VGG) and shown to achieve robust performance on the ImageNet dataset [44]for object recognition. Refer Section 6.2 for discussion on VGG16. We replaced last fully connected layer of VGG16 pre-trained model with dense layer having five outputs. This makes 5005 trainable parameters. Number of output of last dense layer corresponds to number of classes to be recognized, in our experiment we learned concept of five classes i.e. five expression to be recognized (out of six universal expression, expression of "anger" was not included in this experiment as there are few (one) clip(s) for "anger", for explanation see Section 4). We trained last dense layer with images (frames from videos) from our proposed database using softmax activation function and ADAM optimizer [45].
Our proposed database consists of video but for this experiment we extracted frames from videos and fed them to above described ConvNet architecture. We used 80% of frames for training and 10% of frames for validation process. With above mentioned parameters, proposed CNN achieved average expression accuracy of 75% on our proposed database (five expressions). Model accuracy and loss curves are shown in Figure 9.

Conclusion
In this article we presented novel database for Children's Spontaneous Expressions (LIRIS-CSE). The database contains six universal spontaneous expression shown by 12 ethnically diverse children between the ages of 6 and 12 years with mean age of 7.3 years. There were five male and seven female children. 60% of recordings were done in classroom / lab environment and 40% of the clips in the database were recorded in home conditions. The LIRIS-CSE database contains 208 small video clips (on average each clip is 5 seconds long), with each clip containing one specific expression. Clips have neutral expression / face at the beginning of clip, then it show onset of an expression, and finishes when expression is visible at its peak along with some frames after peak expression frame. The database has been validated by 22 human raters / evaluators between the ages of 18 and 40 years.
To the best of our knowledge, this database is first of its kind as it records and shows six (rather five as expression of "anger" is spotted/recorded only once) universal spontaneous expressions of children. Previously there were few image databases of children expressions and all of them show posed or exaggerated expressions which are different from spontaneous or natural expressions. Thus, this database will be a milestone for human behavior researchers. This database will be a excellent resource for vision community for benchmarking and comparing results.
For benchmarking automatic recognition of expression we have also provided result using Convolutional Neural Network (CNN) architecture with transfer learning approach. Proposed approach obtained average expression accuracy of 75% on our proposed database, LIRIS-CSE (five expressions).