Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> Summary form only given, as follows. A backpropagation algorithm is presented that varies the number of hidden units. The algorithm is expected to escape local minima and makes it no longer necessary to decide on the number of hidden units. Exclusive-OR training and 8*8 dot alphanumeric font training using this algorithm are explained. In exclusive-OR training, the probability of being trapped in local minima is reduced. In alphanumeric font training, the network converted two to three times faster than the conventional backpropagation algorithm. > <s> BIB001 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> Abstract An extended tanh-function method is proposed for constructing multiple travelling wave solutions of nonlinear partial differential equations (PDEs) in a unified way. The key idea of this method is to take full advantages of a Riccati equation involving a parameter and use its solutions to replace the tanh function in the tanh-function method. It is quite interesting that the sign of the parameter can be used to exactly judge the numbers and types of these travelling wave solutions. In addition, by introducing appropriate transformations, it is shown that the extended tanh-function method still is applicable to nonlinear PDEs whose balancing numbers may be any nonzero real numbers. Some illustrative equations are investigated by this means and new travelling wave solutions are found. <s> BIB002 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data. <s> BIB003 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. <s> BIB004 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (≫ 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%). <s> BIB005 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> Image denoising can be described as the problem of mapping from a noisy image to a noise-free image. The best currently available denoising methods approximate this mapping with cleverly engineered algorithms. In this work we attempt to learn this mapping directly with a plain multi layer perceptron (MLP) applied to image patches. While this has been done before, we will show that by training on large image databases we are able to compete with the current state-of-the-art image denoising methods. Furthermore, our approach is easily adapted to less extensively studied types of noise (by merely exchanging the training data), for which we achieve excellent results as well. <s> BIB006 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. <s> BIB007 </s> Deep Learning on Image Denoising: An overview <s> Neural networks for image denoising <s> Correlation filter (CF) based trackers generally include two modules, i.e., feature representation and on-line model adaptation. In existing off-line deep learning models for CF trackers, the model adaptation usually is either abandoned or has closed-form solution to make it feasible to learn deep representation in an end-to-end manner. However, such solutions fail to exploit the advances in CF models, and cannot achieve competitive accuracy in comparison with the state-of-the-art CF trackers. In this paper, we investigate the joint learning of deep representation and model adaptation, where an updater network is introduced for better tracking on future frame by taking current frame representation, tracking result, and last CF tracker as input. By modeling the representor as convolutional neural network (CNN), we truncate the alternating direction method of multipliers (ADMM) and interpret it as a deep network of updater, resulting in our model for learning representation and truncated inference (RTINet). Experiments demonstrate that our RTINet tracker achieves favorable tracking accuracy against the state-of-the-art trackers and its rapid version can run at a real-time speed of 24 fps. The code and pre-trained models will be publicly available at https://github.com/tourmaline612/RTINet. <s> BIB008
Neural networks are on the basis of machine learning methods, which are pioneer of deep learning techniques BIB007 . Most of neural networks comprise neurons, input X, activation function f , weights W = [W 0 , W 1 , ..., W n−1 ] and biases b = [b 0 , b 1 , ..., b n ]. The activation functions such as Sigmoid   and Tanh BIB005 BIB002 can convert the linear input into non-linearity through W and b as follows. f (X; It is noted that if the neural network has multiple layers, it is regarded as multilayer perceptron (MLP) BIB006 . Also, the middle layers are treated as hidden layers beside the input and output layers. This process can be expressed as where n is the final layer of the neural network. To make readers easier understand the neural network, we use a visual example to show the principle of the neural network as shown in Fig. 2 . The two-layer fully connected neural network includes two layers: hidden layer and output layer (input layer is not regarded as a layer of neural network in general). There are parameters to be defined: x 1 , x 2 , x 3 and o 1 represent inputs and output of this neural network, respectively. w 1 , w 2 , ..., w 11 , w 12 and b 1 , b 2 , b 3 , b 4 are the weights and biases, respectively. To give an example, output of one neuron h 1 via Eqs. (3) and (4) is obtained as follows: Input Layer Hidden Layer Output Layer  First, the output of the network o 1 is obtained. Then, the network uses back propagation (BP) BIB001 and loss function to learn parameters. That is, when the loss value is within specified limitation, the trained model is considered as well trained. It should be noted that if the number of the layers of a neural network is over 3, it is also referred to as a deep neural network. Specifically, stacked auto-encoder (SAR) BIB003 and deep belief network (DBN) BIB004  are typical deep neural networks. They used stacked layers in an unsupervised manner to train the models and obtain good performance. However, these networks were not simple to implement and need a lot of manual settings to achieve an optimal model. Owing to this reason, end-to-end connected networks, especially CNN, were proposed BIB008 . CNNs have wide applications in the field in image processing, especially image denoising, and they are presented in detail in next section.