A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Abstract In this paper we explore and discuss the learning and generalization characteristics of the random vector version of the Functional-link net and compare these with those attainable with the GDR algorithm. This is done for a well-behaved deterministic function and for real-world data. It seems that ‘ overtraining ’ occurs for stochastic mappings. Otherwise there is saturation of training. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Abstract It is clear that the learning speed of feedforward neural networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: (1) the slow gradient-based learning algorithms are extensively used to train neural networks, and (2) all the parameters of the networks are tuned iteratively by using such learning algorithms. Unlike these conventional implementations, this paper proposes a new learning algorithm called e xtreme l earning m achine (ELM) for s ingle-hidden l ayer f eedforward neural n etworks (SLFNs) which randomly chooses hidden nodes and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide good generalization performance at extremely fast learning speed. The experimental results based on a few artificial and real benchmark function approximation and classification problems including very large complex applications show that the new algorithm can produce good generalization performance in most cases and can learn thousands of times faster than conventional popular learning algorithms for feedforward neural networks. 1 <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> In this invited paper, my overview material on the same topic as presented in the plenary overview session of APSIPA-2011 and the tutorial material presented in the same conference [1] are expanded and updated to include more recent developments in deep learning. The previous and the updated materials cover both theory and applications, and analyze its future directions. The goal of this tutorial survey is to introduce the emerging area of deep learning or hierarchical learning to the APSIPA community. Deep learning refers to a class of machine learning techniques, developed largely since 2006, where many stages of non-linear information processing in hierarchical architectures are exploited for pattern classification and for feature learning. In the more recent literature, it is also connected to representation learning, which involves a hierarchy of features or concepts where higher-level concepts are defined from lower-level ones and where the same lower-level concepts help to define higher-level ones. In this tutorial survey, a brief history of deep learning research is discussed first. Then, a classificatory scheme is developed to analyze and summarize major work reported in the recent deep learning literature. Using this scheme, I provide a taxonomy-oriented survey on the existing deep architectures and algorithms in the literature, and categorize them into three classes: generative, discriminative, and hybrid. Three representative deep architectures – deep autoencoders, deep stacking networks with their generalization to the temporal domain (recurrent networks), and deep neural networks (pretrained with deep belief networks) – one in each of the three classes, are presented in more detail. Next, selected applications of deep learning are reviewed in broad areas of signal and information processing including audio/speech, image/vision, multimodality, language modeling, natural language processing, and information retrieval. Finally, future directions of deep learning are discussed and analyzed. <s> BIB003 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Deep learning is a rather new approach to machine learning that has achieved remarkable results in a large number of different image processing applications. Lately, application of deep learning to detect and classify spectral and spatio-spectral signatures in hyperspectral images has emerged. The high dimensionality of hyperspectral images and the limited amount of labelled training data makes deep learning an appealing approach for analysing hyperspectral data. Auto-Encoder can be used to learn a hierarchical feature representation using solely unlabelled data, the learnt representation can be combined with a logistic regression classifier to achieve results in-line with existing state-of-the-art methods. In this paper, we compare results between a set of available publications and find that deep learning perform in line with state-of-the-art on many data sets but little evidence exists that deep learning outperform the reference methods. <s> BIB004 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> The health care sector is totally different from any other industry. It is a high priority sector and consumers expect the highest level of care and services regardless of cost. The health care sector has not achieved society’s expectations, even though the sector consumes a huge percentage of national budgets. Mostly, the interpretations of medical data are analyzed by medical experts. In terms of a medical expert interpreting images, this is quite limited due to its subjectivity and the complexity of the images; extensive variations exist between experts and fatigue sets in due to their heavy workload. Following the success of deep learning in other real-world applications, it is seen as also providing exciting and accurate solutions for medical imaging, and is seen as a key method for future applications in the health care sector. In this chapter, we discuss state-of-the-art deep learning architecture and its optimization when used for medical image segmentation and classification. The chapter closes with a discussion of the challenges of deep learning methods with regard to medical imaging and open research issue. <s> BIB005 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> The recently proposed Multi-Layer Convolutional Sparse Coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of Convolutional Neural Networks (CNNs). Under this framework, the computation of the forward pass in a CNN is equivalent to a pursuit algorithm aiming to estimate the nested sparse representation vectors -- or feature maps -- from a given input signal. Despite having served as a pivotal connection between CNNs and sparse modeling, a deeper understanding of the ML-CSC is still lacking: there are no pursuit algorithms that can serve this model exactly, nor are there conditions to guarantee a non-empty model. While one can easily obtain signals that approximately satisfy the ML-CSC constraints, it remains unclear how to simply sample from the model and, more importantly, how one can train the convolutional filters from real data. In this work, we propose a sound pursuit algorithm for the ML-CSC model by adopting a projection approach. We provide new and improved bounds on the stability of the solution of such pursuit and we analyze different practical alternatives to implement this in practice. We show that the training of the filters is essential to allow for non-trivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers. Last, but not least, we demonstrate the applicability of the ML-CSC model for several applications in an unsupervised setting, providing competitive results. Our work represents a bridge between matrix factorization, sparse dictionary learning and sparse auto-encoders, and we analyze these connections in detail. <s> BIB006 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Computerized microscopy image analysis plays an important role in computer aided diagnosis and prognosis. Machine learning techniques have powered many aspects of medical investigation and clinical practice. Recently, deep learning is emerging as a leading machine learning tool in computer vision and has attracted considerable attention in biomedical image analysis. In this paper, we provide a snapshot of this fast-growing field, specifically for microscopy image analysis. We briefly introduce the popular deep neural networks and summarize current deep learning achievements in various tasks, such as detection, segmentation, and classification in microscopy image analysis. In particular, we explain the architectures and the principles of convolutional neural networks, fully convolutional networks, recurrent neural networks, stacked autoencoders, and deep belief networks, and interpret their formulations or modelings for specific tasks on various microscopy images. In addition, we discuss the open challenges and the potential trends of future research in microscopy image analysis using deep learning. <s> BIB007 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Since the proposal of big data analysis and Graphic Processing Unit (GPU), the deep learning technology has received a great deal of attention and has been widely applied in the field of imaging processing. In this paper, we have an aim to completely review and summarize the deep learning technologies for image denoising proposed in recent years. Morever, we systematically analyze the conventional machine learning methods for image denoising. Finally, we point out some research directions for the deep learning technologies in image denoising. <s> BIB008 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> A. MOTIVATIONS <s> Deep learning constitutes a recent, modern technique for image processing and data analysis, with promising results and large potential. As deep learning has been successfully applied in various domains, it has recently entered also the domain of agriculture. In this paper, we perform a survey of 40 research efforts that employ deep learning techniques, applied to various agricultural and food production challenges. We examine the particular agricultural problems under study, the specific models and frameworks employed, the sources, nature and pre-processing of data used, and the overall performance achieved according to the metrics used at each work under study. Moreover, we study comparisons of deep learning with other existing popular techniques, in respect to differences in classification or regression performance. Our findings indicate that deep learning provides high accuracy, outperforming existing commonly used image processing techniques. <s> BIB009
In recent years, researchers have published many reviews/ surveys papers related to deep learning technology in the field of image processing, such as the works in the fields of microscopic image analysis BIB007 , hyperspectral image analysis BIB004 and medical image analysis BIB005 etc. Besides, the work of Tian et al. summarize the related processing ideas of deep learning in specific image denoising tasks BIB008 . Deng et al. summarize the deep stacked model under the semi-supervised learning paradigm from the design and application points of view BIB003 . These works have positive enlightenment for the improvement of existing methods and model selection of specific application background. However, the types of deep models mentioned in these works are not abundant, none of them can provide a mathematical principle of the models, and they lack some of the most recent deep learning models. Compared with the above reviews/surveys, an initial motivation of this survey is trying to summarize the application of these new-generation deep learning algorithms and frameworks in image processing and to evaluate the effectiveness of these new generations of deep network framework and acquire more inspiration for deep network's design and optimization tricks BIB009 . In particular, the summary of the new generation of deep learning models will follow along the four main lines, 1) Deep CNN and its improved deep network structure  , the classic framework include CV-CNN, SRCNN, Mask R-CNN, FCN, U-Net, DenseNet, FractalNet, etc. 2) GAN  and its improved deep network structure, the classic framework includes DCGAN, PTGAN, InfoGAN, LAPGAN, TAC-GAN, SegAN, CoGAN etc. 3) ELM BIB002 or RVFL BIB001 (denote as ELM-RVFL) and its improved deep network structure, the classic framework includes C-ELM, H-ELM, BLS, F-BLS, etc. 4) Other typical new generation deep network structure includes Deep Forest, ADMM-Net, CapsuleNet, ML-CSC BIB006 , VAE, PCANet, DDL, etc. More concretely, these can be summarized as TABLE 2  and TABLE 3 . In this survey, our main intention is to focus on a new generation of deep learning models and their mathematical principles. It is worth pointing out that we have three criterions for choosing several representative models. One is that this network has a novel topological structure. Second, excellent generalization performance can achieve in specific image application tasks. Third, the research results generally accepted by scientific researchers, and there are further development and improvement for a deep model. Most of the other deep learning models can be variants of these four deep architectures' main lines. In the following parts, we review the four typical deep learning series models in detail. Also, we will mainly describe the generalization performance of the corresponding new generation of deep learning algorithm from the perspective of the application task, which covers various topics, such as image classification, style transfer, object recognition, super-resolution, image compression, image segmentation, change detection, image denoising. Specifically, the framework structure flow diagram of this survey shown in Fig.2 .