2D Face Recognition on CNN with Half-Average-Face

As the size of training dataset of face recognition models becomes larger and larger, we are interested in a method called Average-Half-Face (AHF), which could halve the size of training samples. The AHF method divides a full face into two halves and then averages them together (reversing the columns of one of the halves). We preprocess the dataset with the method of AHF, and train them on two different models, Eigenfaces and Convolutional Neural Network (CNN). We compare the prediction results with those models trained on the original dataset. Previous researches showed that AHF is superior to Full-Face (FF), while our experiment results further showed that in some cases AHF also boosts CNN. The application of AHF can bring both saving in storage and reduction on training cost time.


Introduction
Face recognition has been extensively researched since 1960s [1]. For years of development, it has become one of the most well-known applications currently. Large industries in different countries are benefiting from this technology, from mobile phone human-face detection, to automatic surveillance in public places. However, digital images in recent years are larger and larger, up to several MB. For devices limited in processing power and storage, it is necessary to develop an optimized algorithm to carry on face recognition. The idea of Average-Half-Face (AHF) [2] was proposed to make use of the inherent bilateral symmetry of human faces. Due to the symmetry of human face, the left and right sides of a face have lots of overlapped features, which could be eliminated by averaging the two half-faces. Though this concept initially focused on extracting facial profiles, it has the side effect of decreasing size of images and accelerating training cost time, while giving an even better output than the traditional Full-Face (FF) approach.
However, the dataset [3] for experiments in that paper is relatively symmetric, which is ideal when comparing to real world cases. To simulate a more realistic situation, we employ the AHF and Eigenfaces algorithm on a more asymmetric dataset, the ORL Database of Faces [4]. Figure 1 displays a face image on the ORL Database, and the AHF expression of this image. (b) AHF expression. Figure 1. One example of a face image from the ORL database and its output after being preprocessed by AHF algorithm Figure 1 (b) shows how AHF averages two half faces, which may not align concisely. If the full face tilts, or the face expression or the face features (like a crooked nose) are asymmetric, the AHF expression would have ghosting. It raises a question: whether the loss of the information due to AHF would lead to a decrease in accuracy? According to the experiment result of Satone's work [5], the answer might be 'no'. AHF approach still performs a little better than FF approach in terms of accuracy. As the creator of AHF mentioned, it was still not clear what other factors were leading to the increase in accuracy [2]. As well, using only left or right half-faces, the outputs are worse than that of Full-Face [6]. Therefore, we can assume that the reason behind the increase in accuracy is far from the loss of information due to dimension decrease.
On the other hand, considering the development of CNN's applications on face recognition, we are interested in the performance of AHF based on CNN. To discover it, we did a cross-validation both horizontally and vertically on PCA and CNN, AHF and FF. That is, we built four models, PCA+AHF, PCA+FF, CNN+AHF, CNN+FF, and then compare PCA+AHF with PCA+FF, PCA+AHF with CNN+AHF, for example. By these experiments, we evaluate the application of AHF on face recognition CNN.
In this paper, we first give a short description on AHF and our face recognition approaches. Next, we carry out related experiments, and then present results that demonstrate the accuracy gains of AHF.

Average-Half-Face
The idea of face recognition using a half face comes from the fact the human face is roughly symmetrical. Zhao and Chellappa [7] developed a face recognition system based on face symmetry, which alleviates the effect of variations in illumination. Ramanathan et al. [8] used the notion of 'half-faces' to circumvent the problem of non-uniform illumination on measuring face similarity. Chen [9] combined the idea of half-face with face-template. In 2008, Harguess [2] came up with the idea of AHF. He derived it from the use of the Symmetry Preserving Singular Value Decomposition (SPSVD) [10].
Generally, AHF is featured by this 4-step preprocess: 1. Centralize the face. This is done by finding the location of the tip of the nose. If the face tilts, we centralize the face by finding a straight line that can divide the face into two halves of the same amount of pixels.
2. Partition the image into two equal left and right halves. 3. Reverse the columns of the left half-face. 4. Add two half faces and average them. This preprocess aims to decrease the number of columns of an image, while the loss of information is relatively little. Due to the fact that the two sides of a human face are symmetric, the mean-squared error of the difference of two half faces is low. To minimize this error, we need to centralize the image first. Then, we reverse the left half-face so that two halves are aligned before averaging. After that, we carry out the operation of averaging, which was proved a better way than just keeping the left half or the right half [2].
After preprocessing, everything is quite the same as what are indicated in a traditional Eigenfaces algorithm [11].
Satone [5] modified the plain AHF algorithm by adding steps of wavelet transform. She achieved an impressive promotion on error rate on the ORL database.

Eigenfaces
Eigenfaces [11] came up early and has played an important role in face recognition and detection. Egienfaces is considered as a use of principal components analysis (PCA). It projects face images onto the principal components of the original set of training images, in order to extract features and decrease dimension. Then face images of individuals are compared (by Euclidean distance, typically) on this subspace of principal components.
Eigenfaces has been proved a powerful tool to carry out face recognition. Lawrence [12] performed experiments using eigenfaces algorithm on the ORL database and reached a 10.5% recognition error rate when the number of eigenfaces he used was between 175 and 199.
Several extensions or related methods are available. Multilinear principal components analysis (MPCA) [13] applies PCA to tensors or multilinear arrays. Furthermore, MPCA could be extended by using Linear Discriminant Analysis (LDA) [14] on the projected multilinear arrays to perform feature selection. Independent component analysis (ICA) [15], a generalization of PCA, tries to identify high-order statistical relationships between pixels to form a better set of basis vectors.

Convolutional Neural Network
Though Neural Network (NN) begin to catch everyone's eyes not long ago, its researches of application on face recognition started at an early time. In an early comparison [16] between eigenfaces, elastic matching [17] and NN, NN did not perform well, and was upper bounded by eigenfaces. However, with an approach of convolutional neural network (CNN) [12], NN did have an astonishing performance on the ORL database, beating eigenfaces and Hidden Markov models [18].
After years of development, CNN has become a standard method in computer vision. Due to the availability of very large scale training datasets [19], and computational resources eligible to train a multi-layers CNN, CNN performs much better than the past. With the help of the Deep Neural Network (DNN) architecture, the state-of-the-art of face recognition has been significantly advanced [20]. In recent years, Deep Convolutional Neural Network has been applied on mobile devices [21], showing the necessity to improve the processing power of mobile devices and to optimize face recognition algorithms.

ORL database
The ORL database [4] we used in this paper, contains a set of faces taken between April 1992 and April 1994 at the Olivetti Research Laboratory in Cambridge, U.K. There are totally 400 images, while there are 40 individuals and 10 images for each one. Unlike the Yale face database [3], there are more variations, including facial expression, illumination and face angles. All the images were taken against a dark homogeneous background. Each image is greyscale with a resolution of 92 X 112.

Input Resizing
Each image in ORL database would be firstly resized from 92 X 112 to 46 X 56 (halved width and halved height), in order to accelerate training. Then, a half face image, as its name indicates, has a halved width 46 * 0.5 = 23. The size of it is 23 X 56. To compare with half face directly, we will also scale the FF image to 32 X 40, keeping the aspect ratio while the amount of pixels is almost the same as that of half face images. The 23 X 56 input and 32 X 40 input have a similar training speed.

Eigenfaces
There is no special detail in our Eigenfaces code. It projects face images onto the principal components of the original set of training images, in order to extract features and decrease dimension. Then face images of individuals are compared (by Euclidean distance, typically) on this subspace of principal components. The document of OpenCV [22] is referred to be checked.

CNN
Our model is quite the same as keras's exemplary CNN model to recognize the MNIST database [23], which has a good performance on simple small-size image datasets. Table 1 is an overlook of this model.

Results and Discussions
As we mentioned, there are 400 training images totally, and 10 for each of 40 subjects. From the 10 images of each subject, we randomly take as trainset, and as (10 − ) testset. We trained several models on the trainset and tested them on the testset. Table 2 shows the accuracy of these models.
On CNN, we fix the learning rate at 1.0, and both train 90 epochs. The evaluation metric is accuracy. Table 2 shows the accuracy of each models, while x alters in the range from 3 to 9( = 3, = 5, = 7, = 9). Every model was cross-validated. 980 Generally, with the increase of , the accuracy increases on every model. Due to the limited size of testset, the accuracy does not look consistent on some . For example, when = 3, CNN+FF surpasses CNN+FF (half_size). However, when = 5, CNN+FF (half_size) is better. In fact, their performances are almost the same, and the accuracy fluctuates because the small size of testset. Overall, all have good performance when is big. When ≥ 7, every model has an accuracy score more than 0.950. However, we can see that the HAF-related models are quite stable. No matter it is Eigenfaces+HAF or CNN+HAF, it has a higher accuracy score on every than its counterparts (FF and FF (half_size)). We also noticed that only the CNN+HAF performed not worse than its Eigenfaces counterpart. The other two, FF and FF (half_size), were beaten by their Eigenfaces versions. CNN+HAF even predicted better than Eigenfaces+HAF when = 9 , proving its potential to recognize faces in an efficient way.
On the other hand, CNN often has a large scale of parameters, and the time spent on training/testing the CNN is usually long. In a CNN, the theoretical time complexity of all convolutional layers given by a related work [24] is: (1) Here is the index of a convolutional layer, and is the depth (number of convolutional layers). is the number of filters in the ℎ layer. −1 is also known as the number of input channels of the ℎ layer.
is the spatial size (length) of the filter. is the spatial size of the output feature map. Reducing the size of input by half, the time costed on training/testing convolutional layers would be four times less if the computational overhead is not omitted.
In our experiments, Table 3 shows how many times were cost on every model. HAF-related models have the least time cost, and FF-related (half_size) models are the second. The FF-related models are the worst. CNN+HAF took about 43.1% time compared with CNN+FF. It could be concluded that the time cost and the input size are positively related. Therefore, we can see the advantage of adapting traditional full-size images to halved ones to train models. In spite of shorter training time, HAF performs better than FF. It is inspiring because we are able to train on some certain datasets with less time but get better performance.  Figure 2 shows the learning curves of CNN on different input shapes. We can see that FF, HAF and FF (half_size) are all about to converge around 60 epochs.

Conclusion
In this paper, we presented an experimental evaluation of the adaptation of AHF on CNN. The models were evaluated using different number of training images and test images by cross-validation.The experiment showed the idea of Half-Average-Face could also work well on CNN, and it brought improvements on both prediction accuracy and training time.