Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day. <s> BIB001 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these "Stepped Sigmoid Units" are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors. <s> BIB002 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> During the last decade, the data sizes have grown faster than the speed of processors. In this context, the capabilities of statistical machine learning methods is limited by the computing time rather than the sample size. A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. The large-scale case involves the computational complexity of the underlying optimization algorithm in non-trivial ways. Unlikely optimization algorithms such as stochastic gradient descent show amazing performance for large-scale problems. In particular, second order stochastic gradient and averaged stochastic gradient are asymptotically efficient after a single pass on the training set. <s> BIB003 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> We propose a novel deep network structure called "Network In Network" (NIN) to enhance model discriminability for local patches within the receptive field. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to abstract the data within the receptive field. We instantiate the micro neural network with a multilayer perceptron, which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner as CNN; they are then fed into the next layer. Deep NIN can be implemented by stacking mutiple of the above described structure. With enhanced local modeling via the micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers. We demonstrated the state-of-the-art classification performances with NIN on CIFAR-10 and CIFAR-100, and reasonable performances on SVHN and MNIST datasets. <s> BIB004 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision. <s> BIB005 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection. <s> BIB006 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB007 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> With the aim of developing a fast yet accurate algorithm for compressive sensing (CS) reconstruction of natural images, we combine in this paper the merits of two existing categories of CS methods: the structure insights of traditional optimization-based methods and the speed of recent network-based ones. Specifically, we propose a novel structured deep network, dubbed ISTA-Net, which is inspired by the Iterative Shrinkage-Thresholding Algorithm (ISTA) for optimizing a general $$ norm CS reconstruction model. To cast ISTA into deep network form, we develop an effective strategy to solve the proximal mapping associated with the sparsity-inducing regularizer using nonlinear transforms. All the parameters in ISTA-Net (e.g. nonlinear transforms, shrinkage thresholds, step sizes, etc.) are learned end-to-end, rather than being hand-crafted. Moreover, considering that the residuals of natural images are more compressible, an enhanced version of ISTA-Net in the residual domain, dubbed ISTA-Net+, is derived to further improve CS reconstruction. Extensive CS experiments demonstrate that the proposed ISTA-Nets outperform existing state-of-the-art optimization-based and network-based CS methods by large margins, while maintaining fast computational speed. Our source codes are available: http://jianzhang.tech/projects/ISTA-Net. <s> BIB008 </s> Deep Learning on Image Denoising: An overview <s> Convolutional neural networks for image denoising <s> Recently, deep convolutional neural networks (CNNs) have been attracting considerable attention in single image super-resolution. Some CNN-based methods, such as VDSR verified that residual learning can speed up the training and significantly improve the performance of accuracy. However, with very deep networks, convergence speed is still a critical issue in training due to the cost of requiring enormous parameters. In order to deal with this issue, we redesign the residual networks based on dilated networks. In this paper, we propose symmetrical dilated residual convolution networks (FDSR) to tackle image super-resolution problems. Our network is on the basis of the dilated convolutions supported exponential expansion of the receptive field without loss of resolution and coverage. This means that FDSR can speed up the training and improve the performance of accuracy without increasing the model’s depth or complexity. Meanwhile, we attempt to combine the image pre-processing approach of VGG-net with mean squared error (MSE) to enhance the performance. The experimental results demonstrate that the training time-consuming proposed model achieves nearly a half with even superior restoration quality. Further, we present a novel network with less layers and parameters can achieve real-time performance on a generic CPU and still maintain superior performance. <s> BIB009
Due to plug-and-play network architectures, CNNs have obtained great success in image processing BIB008 BIB009 . As pioneers of CNNs, LeNet BIB001 used convolutional kernels of different sizes to extract features and obtain good performance in image classification. However, due to the activation function, Sigmoid, LeNet had a slow convergence speed, which was a shortcoming for real applications. After LeNet, the proposed AlexNet BIB007 was a milestone for deep learning. Its success had the following reasons. Firstly, graphics processing unit (GPU)  provided strong computational ability. Secondly, random clipping (i.e. dropout) can solve the overfitting problem. Thirdly, ReLU BIB002 can improve the speed of stochastic gradient descent (SGD) rather than Sigmoid BIB003 . Fourthly, data augmentation method can further address the overfitting problem. Although AlexNet has obtained good performance, it could result in high memory due to big convolutional kernels. That limited applications in the real world, such as smart cameras. After that, deeper network architectures with small filters were preferred to improve the performance and reduce computational costs from 2014 to 2016. Specifically, VGG BIB005 used stacked more convolutions with of small kernel size to win the ImageNet large scale visual recognition (LSVR) challenge 2014 via stacked more convolutions with of small kernels. To give an example, we use Fig. 3 to visually show the network architecture. On the top of deeper networks, increasing the width was very popular. GoogleNet BIB006 increased the width to improve the performance for image applications. Moreover, the GoogleNet transformed a big convolutional kernel into two small convolution kernels to reduce the number of parameters and computational cost. Additionally, the GoogLeNet used the inception module BIB004 as well as Inception 1. Its visual network figure was descripted in Fig. 4 .