A Survey on the New Generation of Deep Learning in Image Processing <s> A. IMAGE CLASSIFICATION <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. IMAGE CLASSIFICATION <s> Deep learning has recently exhibited good performance in many applications. The convolution neural network is an often-used architecture for deep learning and has been widely used in computer vision and audio recognition, and outperformed other related handcraft designed feature in recent years. These techniques compared to other artificial intelligence algorithms and handcraft features need extremely much more time in training and testing and then were not widely used in the early days. Our study is about the impacts of different factors used in the convolution neural network. The considered factors are network depth, numbers of filters, and filter sizes. The used data set is the CIFAR dataset. According to our experiments, some suggestions about those factors are recommended in this study. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. IMAGE CLASSIFICATION <s> Abstract Classification plays an important role in many fields of synthetic aperture radar (SAR) image understanding and interpretation. Many scholars have devoted to design features to characterize the content of SAR images. However, it is still a challenge to design discriminative and robust features for SAR image classification. Recently, the deep learning has attracted much attention and has been successfully applied in many fields of computer vision. In this paper, a novel feature learning approach that is called discriminant deep belief network (DisDBN) is proposed to learning high-level features for SAR image classification, in which the discriminant features are learned by combining ensemble learning with a deep belief network in an unsupervised manner. Firstly, some subsets of SAR image patches are selected and marked with pseudo-labels to train weak classifiers. Secondly, the specific SAR image patch is characterized by a set of projection vectors that are obtained by projecting the SAR image patch onto each weak decision space spanned by each weak classifier. Finally, the discriminant features are generated by feeding the projection vectors to a DBN for SAR image classification. Experimental results demonstrate that better classification performance can be achieved by the proposed approach than the other state-of-the-art approaches. <s> BIB003
Deep learning thrives with large neural networks and large datasets. One key ingredient for success of deep learning in image classification is the use of convolutional architectures  - BIB002 . For example, deep CNNs has shown state-of-art classification performance on datasets such as ImageNet, a large visual database designed for use in visual object recognition software research. In addition, a dramatic 2012 breakthrough in solving the imageNet challenge BIB001 is widely considered to be the beginning of the deep learning revolution (see in Fig.24 ). From then on, many innovative deep frameworks based on CNNs was proposed for imagenet classification tasks, such as AlexNet, GoogleNet, VGG, and ResNet, etc. In other words, ImageNet large scale visual recognition challenge (ILSVRC) greatly promotes the development of deep CNNs. Meanwhile, scientific researchers have put forward various training techniques one after another. For example, to reduce over-fitting in the globally connected layers, a new regularization method dropout that proved to be very useful. Furthermore, BN is a very effective regularization method that can accelerate the training of large conglomerate networks many times. Further, the obtained higher accuracy can implement by updating the residual module to use identity mappings, and so on. ELM has demonstrated better generalization performance with extreme fast learning speed in many benchmarks and real applications. Compared with the performance of support vector machine (SVM), the classification performance of ELM shows that ELM has better generalization performance with much less training time on majority cases than SVM for both feature extraction methods. Furthermore, the improvement algorithms, which based on ELM-RVFL, have a significant improvement in classification accuracy rate. For examples, the improvement algorithm H-ELM and BLS not only achieves state-of-art results but also shortens the training time from days (spent by deep learning) to several minutes (by ELM) in MNIST OCR dataset, traffic sign recognition and 3D graphics application, Norb, Fashion MNIST, SVHN, etc. However, on some complex datasets (such as CIFAR 10/100 and ImageNet datasets), ELM series algorithms still need further improvement and combined with the prior of an image to achieve excellent generalization performance, there is still a long way to go. Though deep neural networks are robust, they have some apparent deficiencies. One is that a massive amount of training data usually required for training deep neural networks, which can disable few-shot learning. The other is that deep neural networks are very complicated models and powerful computational facilities traditionally needed for the training process. More importantly, deep neural networks are with too many hyperparameters, and the learning performance depends seriously on the careful tuning of them BIB003 . For small-scale datasets, the GAN series can generate more different style samples to promote the generalization performance of the deep discriminative model for classification tasks. At the same time, it also provides a practical solution to the small sample problem. For the enormous hyperparameters, deep forest, which is a novel decision tree ensemble method, can achieve better classification performance by much fewer hyperparameters. However, the deep forest still needs further improvement on more complex datasets. In short, facing a variety of classification applications, a specific deep method is not omnipotent for classification tasks. Two ways motivate the continuous development of deep learning, one is application tasks, and data characteristics, more training tricks (including BN, dropout, early stopping, momentum, and so on) and suitable deep models, such as CV-CNN, and the other is the enhancement of model expressive ability that is independent of data, such as CapsuleNet. In TABLE 4, We summarize some benchmark datasets and corresponding deep learning methods commonly used in image classification. Compared with the classical models, these new generation models are intended to highlight novel framework/architecture design and relatively excellent generalization performance.