An approach for automating the design of convolutional neural networks

Image recognition is an independent field of the computer science nowadays. Image classification is one of its main domains, in which investigated objects can be represented by an image or a video stream. The objective of the image classification is correct assigning of objects to corresponding classes, and there exist many effective approaches for solving this problem. One of the most popular approaches is artificial neural networks, which are a method from the field of machine learning. Despite the fact that neural networks cover a wide range of machine learning problems, they are also able to solve the problem of the image classification. However, there is one more specific approach for neural networks-based images classification that applies the deep learning conception. The best-known deep learning algorithm is called the convolutional neural network (CNN). The CNN uses a principle of using the same parts of a neural network to manipulate with different local parts of an input image. As well as the standard neural network architecture, the convolutional neural network should be fine-tuned for solving a certain problem. Because of the CNN’s depth and complexity, the tuning process usually is complex and needs huge computational efforts. In this study, we have proposed an approach for creating ensembles of previously trained convolutional neural networks. The approach allows to increase the performance of the image classification. The results of experiments for image classification problems are presented and discussed. The experiments show that the proposed approach is able to outperform the standard perceptron and single convolutional neural network.


Introduction
Image recognition is one of the most complicated problem for machine learning techniques. For many real-world applications a solution to the problem cannot be described as a set of instructions for a machine learning algorithm. Instead of that it is needed to design a model which will be able to estimate the instances (input data) by itself. For many machine learning techniques, both the structure of the model and its parameters are needed to be fine-tuned. In a case of simple models, the tuning process can be done manually by a human-expert, but for complex image recognition problems simple models do not perform well, thus an automated designing of more complicated structures is preferable.
Starting with the LeNet-5 [1], convolutional neural networks (CNNs) usually had the standard structure included lots of layers ( Figure 1). This structure shows perfect results working with MNIST, CIFAR and ImageNet benchmark problems [2,3]. For many real-world problems, there is no stand-alone algorithm which can achieve a high performance for the given problem. In this case one of the most powerful technique for obtaining the best possible results for the problem is creating an ensemble model. Ensemble models consist of pooling together the predictions of different algorithms, to produce better predictions [4]. There are some popular methods of designing ensemble models, such as the simple voting and the weighted voting. In this study, we will use the simple voting approach in order to take into consideration all predictions from all CNN models. The output of the ensemble will be a label, which polled the majority of the votes.
In this study we will automate the process of CNNs designing by defining some special filters which are used as kernels in the CNN structure. Moreover, we will use an ensemble of previously trained CNN with the identic subsampling layers. The results of experiments are compared with fully connected NN with the same structure as in the fully-connected layers in CNNs.
The rest of the paper is organized as follows. Section 2 describes related work. Section 3 describes the proposed approach. In Section 4 the results of numerical experiments are discussed. In the Conclusion the results and further research are discussed.

Related work
The term "neural network" has its origins in attempts to find mathematical representations of information processing in biological systems (McCulloch and Pitts, 1943;Widrow and Hoff, 1960;Rosenblatt, 1962;Rumelhart et al., 1986). Indeed, it has been used very broadly to cover a wide range of different models [5].
Before deep learning appeared, David Hubel and Torsten Wiesel performed one of the most important research about striate cortex [6]. They explored parts of the visual cortex which are responsible for detecting edges. They found that some neurons activated when there were vertical lines, other neurons activated when there were horizontal lines. There were also neurons which were responsible for detecting other particular angles. A few later scientists explored that visual cortex is organized as a layered structure. Exactly these ideas affected on designing convolutional neural nets.
Deep Learning and its subfield CNNs appeared not so long ago on a scale of machine learning. Now we can claim that the first CNN was called Neocognitron and it was designed by Fukushima K. in 1979Fukushima K. in -1980. Today there is designed a wide range of CNNs architectures. And despite the fact that they show excellent results, the majority of them uses the same ideas. On the other hand, some other approaches become more popular. And one of these approaches is ensemble modeling. Ensemble models usually consist only of CNNs. There are some researches that proves its higher efficiency on various image recognition problems, such as MNIST [9] or FMOW [10].

Proposed Approach
Each CNN has three types of layers: convolutional layers, subsampling layers and fully-connected layers. First two layers differ this algorithm from the others. Convolutional layer represents the list of kernels, where each kernel is a special filter applied to the input image. These kernels aimed to find some graphical patterns in original image and in the images, which were processed by the previous layers. The output of this layer is represented by the feature maps. Subsampling layer created to reduce the dimensionality of the problem and the feature maps. This layer pools closely-spaced pixels. Let's discuss each layer in details.

Convolutional layer.
As it was mentioned, one of the most important part of convolutional layer is kernel. Kernel represents the process of filter applying to the input image. In practice, various types of kernels are used. What is more, some training techniques are implemented to tune the kernels. Usually the adapted backpropagation algorithm is used. Each filter follows its own purpose. However, the majority of them is used to increase the quality of pixel image. In this paper the following filters were considered:  Gradient filters (directions: north, south, west, and east). Belong to difference linear filters. These filters are also called edge detection filters.  Laplacians (kernel = 3x3 pixels, 5x5 pixels). These filters commonly used to increase the sharpness and applied to the images which were previously blurred in case to reduce the noise.  Line detection filters (incrementation of an angle degree = 45°).  Sobel operator (lines: horizontal, vertical). Discrete differentiation operator used to create an image with emphasizing edges.  Sharpening filters (2 levels of sharpness). These filters are used to highlight fine detail in an image or to enhance detail that has been blurred.  Smoothing filter. These filters are used to clean noised images. Kernel is just a special kind linear operator applied to the input data. Let's define the following variables: -feature map of -th layer, kernel size , -weight matrix, where = . The result of convolution with kernel of size can be represented as follows: , where -the result of convolution at the -th layer, and -the output of the whole the previous layer. In other words, to get the component of the next layer, linear operator is applied to the previous layer, so scalar multiplication of pixel values and kernel vector is performed. The example of filter applying is shown in Figure 2.

Subsampling layer.
In recognition problems it is possible to pool feature pixels in some cases. First of all, the information about some features existing is more valuable knowledge than the concrete location of feature border pixels. What is more, the dimensionality of some problems is too high and it is needed to reduce the dimensionality. The example of pooling operator is shown in Figure 3. There are some functions can be used, such as finding maximum, minimum, median, etc. And one more parameter which needed to be defined is step. In many papers maximum function is used. For example, in object detection [11] and human pose detection [12] problems this operator shows better results. Moreover, some researches explored that max-pooling is more efficient function [13].
After applying listed layers and training fully-connected layers, we need to design ensemble models. For choosing the general prediction simple voting is used. There is not a concrete strategy for uniting different CNNs, because each filter determines unique features, and it is hard to predict which combination of extracted features works better. What it more, as we shall see throughout the paper, the CNNs combinations are different in all problems.

Experimental Settings and Results
To estimate the proposed approach performance, we have used 3 image recognition problems.

CIFAR -10.
It is an object recognition problem. Number of examples = 60000. Dimensionality of the problem: 1024 (each image has a resolution = 32x32 pixels). Number of classes = 10. Each class has 6000 images, so it is a balanced dataset. The classes are following: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The classes are completely mutually exclusive and there is no overlap between automobiles and trucks, for example. Color model: RGB.

The ORL Database of Faces.
It is a face recognition problem. This dataset contains a set of face images taken between April 1992 and April 1994 at the lab. There are ten different images of each of 40 distinct subjects. For some subjects, the images were taken at different times, varying the lighting, facial expressions (open / closed eyes, smiling / not smiling) and facial details (glasses / no glasses). All the images were taken against a dark homogeneous background with the subjects in an upright, frontal position (with tolerance for some side movement). Dimensionality of the problem: 10304 (each image has a resolution = 92x112 pixels). Number of classes = 40. Color model: grayscale.

Experimental settings.
Firstly, all the filters listed in Section 3 were used to create and train different CNNs (each CNN had a one specific filter). Results of some filters applying are shown in Figures 5 -7. After that, the ensemble models were created, based on previously trained CNNs. Highest possible number of CNNs in one ensemble model equals 3.

Experimental Results.
For each experiment 10 independent runs were computed. Performance shows mean value of these runs. Performance of fully-connected NN for all considered problems is presented in Table 1 Performance of the best single CNNs using one filter is presented in Tables 2-4.   Performance of the best ensemble models is presented in Tables 5 -7.    The experimental results show that creating ensemble models from previously trained CNNs can improve the performance, comparing with a single NN. Ensemble models also outperforms single CNNs, designed only with one filter and give more reliability. What is more, there is not a statistical significance between these algorithms.

Conclusions
In this study, an automation of creating an ensemble of various CNNs is considered. All CNNs were designed using different types of image filters for increasing the quality of bitmap images and for detecting graphical patterns in images. The feedforward NN stage in all CNNs is the same for providing the equal conditions of the experiments. The experimental results have demonstrated that the performance of the proposed approach is comparable with the state-of-the-art techniques. Furthermore, the dimensionality of input data has been decreased, thus it is required smaller computational capabilities for implementing the CNN algorithm.
In further work, we will implement the proposed approach using the CUDA parallel computation framework for improving the computational performance and for carrying out more experiments. We will explore a wider range of existing image filters and will try to design new ones. More complicated structures of CNNs and ensemble models will be investigated with various image recognition problems.