Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> We explore an innovative strategy for image denoising by using convolutional neural networks (CNN) to learn similar pixel-distribution features from noisy images. Many types of image noise follow a certain pixel-distribution in common, such as additive white Gaussian noise (AWGN). By increasing CNN's width with larger reception fields and more channels in each layer, CNNs can reveal the ability to extract more accurate pixel-distribution features. The key to our approach is a discovery that wider CNNs with more convolutions tend to learn the similar pixel-distribution features, which reveals a new strategy to solve low-level vision problems effectively that the inference mapping primarily relies on the priors behind the noise property instead of deeper CNNs with more stacked nonlinear layers. We evaluate our work, Wide inference Networks (WIN), on AWGN and demonstrate that by learning pixel-distribution features from images, WIN-based network consistently achieves significantly better performance than current state-of-the-art deep CNN-based methods in both quantitative and visual evaluations. \textit{Code and models are available at \url{this https URL}}. <s> BIB001 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> While variational methods have been among the most powerful tools for solving linear inverse problems in imaging, deep (convolutional) neural networks have recently taken the lead in many challenging benchmarks. A remaining drawback of deep learning approaches is their requirement for an expensive retraining whenever the specific problem, the noise level, noise type, or desired measure of fidelity changes. On the contrary, variational methods have a plug-and-play nature as they usually consist of separate data fidelity and regularization terms. In this paper we study the possibility of replacing the proximal operator of the regularization used in many convex energy minimization algorithms by a denoising neural network. The latter therefore serves as an implicit natural image prior, while the data term can still be chosen independently. Using a fixed denoising neural network in exemplary problems of image deconvolution with different blur kernels and image demosaicking, we obtain state-of-the-art reconstruction results. These indicate the high generalizability of our approach and a reduction of the need for problem-specific training. Additionally, we discuss novel results on the analysis of possible optimization algorithms to incorporate the network into, as well as the choices of algorithm parameters and their relation to the noise level the neural network is trained on. <s> BIB002 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> We propose to leverage denoising autoencoder networks as priors to address image restoration problems. We build on the key observation that the output of an optimal denoising autoencoder is a local mean of the true data density, and the autoencoder error (the difference between the output and input of the trained autoencoder) is a mean shift vector. We use the magnitude of this mean shift vector, that is, the distance to the local mean, as the negative log likelihood of our natural image prior. For image restoration, we maximize the likelihood using gradient descent by backpropagating the autoencoder error. A key advantage of our approach is that we do not need to train separate networks for different image restoration tasks, such as non-blind deconvolution with different kernels, or super-resolution at different magnification factors. We demonstrate state of the art results for non-blind deconvolution and super-resolution using the same autoencoding prior. <s> BIB003 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> In this paper, we propose a deep CNN to tackle the image restoration problem by learning the structured residual. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a "residual formatting layer" to format the residual to structured information, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method outperforms existing approaches quantitatively and qualitatively. <s> BIB004 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> In this paper, we propose a novel convolutional neural network (CNN) for image denoising, which uses exponential linear unit (ELU) as the activation function. We investigate the suitability by analyzing ELU's connection with trainable nonlinear reaction diffusion model (TNRD) and residual denoising. On the other hand, batch normalization (BN) is indispensable for residual denoising and convergence purpose. However, direct stacking of BN and ELU degrades the performance of CNN. To mitigate this issue, we design an innovative combination of activation layer and normalization layer to exploit and leverage the ELU network, and discuss the corresponding rationale. Moreover, inspired by the fact that minimizing total variation (TV) can be applied to image denoising, we propose a TV regularized L2 loss to evaluate the training effect during the iterations. Finally, we conduct extensive experiments, showing that our model outperforms some recent and popular approaches on Gaussian denoising with specific or randomized noise levels for both gray and color images. <s> BIB005 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Given a large unlabeled set of images, how to efficiently and effectively group them into clusters based on extracted visual representations remains a challenging problem. To address this problem, we propose a convolutional neural network (CNN) to jointly solve clustering and representation learning in an iterative manner. In the proposed method, given an input image set, we first randomly pick k samples and extract their features as initial cluster centroids using the proposed CNN with an initial model pre-trained from the ImageNet dataset. Mini-batch k-means is then performed to assign cluster labels to individual input samples for a mini-batch of images randomly sampled from the input image set until all images are processed. Subsequently, the proposed CNN simultaneously updates the parameters of the proposed CNN and the centroids of image clusters iteratively based on stochastic gradient descent. We also proposed a feature drift compensation scheme to mitigate the drift error caused by feature mismatch in representation learning. Experimental results demonstrate the proposed method outperforms start-of-the-art clustering schemes in terms of accuracy and storage complexity on large-scale image sets containing millions of images. <s> BIB006 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> The salt and pepper noise, especially the one with extremely high percentage of impulses, brings a significant challenge to image denoising. In this paper, we propose a non-local switching filter convolutional neural network denoising algorithm, named NLSF-CNN, for salt and pepper noise. As its name suggested, our NLSF-CNN consists of two steps, i.e., a NLSF processing step and a CNN training step. First, we develop a NLSF pre-processing step for noisy images using non-local information. Then, the pre-processed images are divided into patches and used for CNN training, leading to a CNN denoising model for future noisy images. We conduct a number of experiments to evaluate the effectiveness of NLSF-CNN. Experimental results show that NLSF-CNN outperforms the state-of-the-art denoising algorithms with a few training images. <s> BIB007 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Many image restoration problems are ill-posed in nature, hence, beyond the input image, most existing methods rely on a carefully engineered image prior, which enforces some local image consistency in the recovered image. How tightly the prior assumptions are fulfilled has a big impact on the resulting task performance. To obtain more flexibility, in this work, we proposed to design the image prior in a data-driven manner. Instead of explicitly defining the prior, we learn it using deep generative models. We demonstrate that this learned prior can be applied to many image restoration problems using an unified framework. <s> BIB008 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Convolutional neural networks offer state-of-the-art performance for medical image denoising. However, their architectures are manually designed for different noise types. The realistic noise in medical images is usually mixed and complicated, and sometimes unknown, leading to challenges in creating effective denoising neural networks. In this paper, we present a Genetic Algorithm (GA)-based network evolution approach to search for the fittest genes to optimize network structures. We expedite the evolutionary process through an experience-based greedy exploration strategy and transfer learning. The experimental results on computed tomography perfusion (CTP) images denoising demonstrate the capability of the method to select the fittest genes for building high-performance networks, named EvoNets, and our results compare favorably with state-of-the-art methods. <s> BIB009 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Deep neural networks provide state-of-the-art performance for image denoising, where the goal is to recover a near noise-free image from a noisy observation. The underlying principle is that neural networks trained on large datasets have empirically been shown to be able to generate natural images well from a low-dimensional latent representation of the image. Given such a generator network, a noisy image can be denoised by i) finding the closest image in the range of the generator or by ii) passing it through an encoder-generator architecture (known as an autoencoder). However, there is little theory to justify this success, let alone to predict the denoising performance as a function of the network parameters. In this paper we consider the problem of denoising an image from additive Gaussian noise using the two generator based approaches. In both cases, we assume the image is well described by a deep neural network with ReLU activations functions, mapping a $k$-dimensional code to an $n$-dimensional image. In the case of the autoencoder, we show that the feedforward network reduces noise energy by a factor of $O(k/n)$. In the case of optimizing over the range of a generative model, we state and analyze a simple gradient algorithm that minimizes a non-convex loss function, and provably reduces noise energy by a factor of $O(k/n)$. We also demonstrate in numerical experiments that this denoising performance is, indeed, achieved by generative priors learned from data. <s> BIB010 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Loss to followup is a significant issue in healthcare and has serious consequences for a study's validity and cost. Methods available at present for recovering loss to followup information are restricted by their expressive capabilities and struggle to model highly non-linear relations and complex interactions. In this paper we propose a model based on overcomplete denoising autoencoders to recover loss to followup information. Designed to work with high volume data, results on various simulated and real life datasets show our model is appropriate under varying dataset and loss to followup conditions and outperforms the state-of-the-art methods by a wide margin (≥ 20% in some scenarios) while preserving the dataset utility for final analysis. <s> BIB011 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> We propose a dual-stage convolutional neural network, augmented with adversarial training, to address the shortcoming of current convolutional neural networks in image denoising. Our dual-stage approach, coupled with feature matching, is especially effective in recovering fine detail under high noise level. First, we use residual learning denoising to output a preliminary denoised reference image. Then, an image reconstruction denoiser uses a multi-scale feature selection layer, which deploys skip-connections and ResNet blocks to recover the image detail based on the noisy image and the reference image. This dual-stage denoising is augmented with the feedback from a discriminator, which forms an adversarial training framework and guides the denoising towards a clean image construction. The feature matching process embedded in the discriminator ensures that the framework can be generalized to a diverse collection of image content. Experimental results show better denoising performance in public benchmark datasets compared with the state-of-the-art approaches. <s> BIB012 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> In this paper, a new generative adversarial networks (GAN) is proposed for image denoising. The proposed GAN has a new generator network to produce denoised images with noisy images as input, and the entire network is trained using a new loss to represent the distance between the data distribution of clean images and denoised images. Based on quantitative and qualitative evaluating criteria, we made comparisons between our method and other denoising methods which shows the superiority of our approach. <s> BIB013 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Wavefront coding (WFC) is a prosperous technology for extending depth of field (DOF) in the incoherent imaging system. Digital recovery of the WFC technique is a classical ill-conditioned problem by removing the blurring effect and suppressing the noise. Traditional approaches relying on image heuristics suffer from high frequency noise amplification and processing artifacts. This paper investigates a general framework of neural networks for restoring images in WFC. To our knowledge, this is the first attempt for applying convolutional networks in WFC. The blur and additive noise are considered simultaneously. Two solutions respectively exploiting fully convolutional networks (FCN) and conditional Generative Adversarial Networks (CGAN) are presented. The FCN based on minimizing the mean squared reconstruction error (MSE) in pixel space gets high PSNR. On the other side, the CGAN based on perceptual loss optimization criterion retrieves more textures. We conduct comparison experiments to demonstrate the performance at different noise levels from the training configuration. We also reveal the image quality on non-natural test target image and defocused situation. The results indicate that the proposed networks outperform traditional approaches for restoring high frequency details and suppressing noise effectively. <s> BIB014 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> As an integral component of blind image deblurring, non-blind deconvolution removes image blur with a given blur kernel, which is essential but difficult due to the ill-posed nature of the inverse problem. The predominant approach is based on optimization subject to regularization functions that are either manually designed, or learned from examples. Existing learning based methods have shown superior restoration quality but are not practical enough due to their restricted model design. They solely focus on learning a prior and require to know the noise level for deconvolution. We address the gap between the optimization-based and learning-based approaches by learning an optimizer. We propose a Recurrent Gradient Descent Network (RGDN) by systematically incorporating deep neural networks into a fully parameterized gradient descent scheme. A parameter-free update unit is used to generate updates from the current estimates, based on a convolutional neural network. By training on diverse examples, the Recurrent Gradient Descent Network learns an implicit image prior and a universal update rule through recursive supervision. Extensive experiments on synthetic benchmarks and challenging real-world images demonstrate that the proposed method is effective and robust to produce favorable results as well as practical for real-world image deblurring applications. <s> BIB015 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Recently, deep learning based video super-resolution (SR) methods have achieved promising performance. To simultaneously exploit the spatial and temporal information of videos, employing 3-dimensional (3D) convolutions is a natural approach. However, straight utilizing 3D convolutions may lead to an excessively high computational complexity which restricts the depth of video SR models and thus undermine the performance. In this paper, we present a novel fast spatio-temporal residual network (FSTRN) to adopt 3D convolutions for the video SR task in order to enhance the performance while maintaining a low computational load. Specifically, we propose a fast spatio-temporal residual block (FRB) that divide each 3D filter to the product of two 3D filters, which have considerably lower dimensions. Furthermore, we design a cross-space residual learning that directly links the low-resolution space and the high-resolution space, which can greatly relieve the computational burden on the feature fusion and up-scaling parts. Extensive evaluations and comparisons on benchmark datasets validate the strengths of the proposed approach and demonstrate that the proposed network significantly outperforms the current state-of-the-art methods. <s> BIB016 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> We propose a novel image denoising method based on a convolutional neural network (CNN), which uses the separable convolution and the gradient prior to reduce the computational complexity while enhancing the denoising performance. The proposed method converts the existing convolution filter in the conventional CNN denoiser to cascaded vertical and horizontal separable convolutions and reduces the number of feature channels between these convolutions by analyzing the distribution of convolution weights. The proposed separable convolution with feature dimension shrinking can greatly reduce the number of multiplications for CNN while minimizing the degradation of denoising quality. In addition, gradients of a given image are used as input for the proposed CNN denoiser by exploiting the relation between an anisotropic diffusion-based denoiser and a residual CNN denoiser to improve the quality of the image denoising. The simulation results showed that the proposed method provided comparable denoising quality while reducing the number of multiplications to 41% compared to the existing state-of-the-art CNN denoiser. <s> BIB017 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> In this paper, we study design of deep neural networks for tasks of image restoration. We propose a novel style of residual connections dubbed "dual residual connection", which exploits the potential of paired operations, e.g., up- and down-sampling or convolution with large- and small-size kernels. We design a modular block implementing this connection style; it is equipped with two containers to which arbitrary paired operations are inserted. Adopting the "unraveled" view of the residual networks proposed by Veit et al., we point out that a stack of the proposed modular blocks allows the first operation in a block interact with the second operation in any subsequent blocks. Specifying the two operations in each of the stacked blocks, we build a complete network for each individual task of image restoration. We experimentally evaluate the proposed approach on five image restoration tasks using nine datasets. The results show that the proposed networks with properly chosen paired operations outperform previous methods on almost all of the tasks and datasets. <s> BIB018 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> This paper presents an integrating concept of de-noising convolutional neural networks (DnCNN) with quadrature amplitude modulation (QAM) for symbol denoising. DnCNN is used to estimate and denoise the Gaussian noise from the received constellation symbols of QAM with unknown noise level. Proposed system shows a significant gain in terms of peak signal-to-noise ratio, system throughput and bit-error rate; in comparison with conventional QAM systems. The basic concept, system level integration, and simulated performance gains are presented to elucidate the concept. <s> BIB019 </s> Deep Learning on Image Denoising: An overview <s> The combination of optimization method and CNN/NN for AWNI denoising <s> Both interclass variances and intraclass similarities are crucial for improving the classification performance of discriminative dictionary learning (DDL) algorithms. However, existing DDL methods often ignore the combination between the interclass and intraclass properties of dictionary atoms and coding coefficients. To address this problem, in this paper, we propose a discriminative Fisher embedding dictionary learning (DFEDL) algorithm that simultaneously establishes Fisher embedding models on learned atoms and coefficients. Specifically, we first construct a discriminative Fisher atom embedding model by exploring the Fisher criterion of the atoms, which encourages the atoms of the same class to reconstruct the corresponding training samples as much as possible. At the same time, a discriminative Fisher coefficient embedding model is formulated by imposing the Fisher criterion on the profiles (row vectors of the coding coefficient matrix) and coding coefficients, which forces the coding coefficient matrix to become a block-diagonal matrix. Since the profiles can indicate which training samples are represented by the corresponding atoms, the proposed two discriminative Fisher embedding models can alternatively and interactively promote the discriminative capabilities of the learned dictionary and coding coefficients. The extensive experimental results demonstrate that the proposed DFEDL algorithm achieves superior performance in comparison with some state-of-the-art dictionary learning algorithms on both hand-crafted and deep learning-based features. <s> BIB020
It is known that machine learning uses optimization techniques BIB006 BIB020 and discriminative learning methods BIB016 BIB001 to deal with image applications in general. Although optimization methods have good performance on different low-level vision tasks, these methods need manual setting parameters, which were time-consuming. The discriminative learning methods have fast speed in image restoration. However, they are not flexible for various low-level vision tasks. To make a tradeoff between efficiency and flexibility, discriminative learning optimization-based method BIB002 BIB003 was presented for image applications, such as image denoising. The CNN with prior knowledge via regular term of loss function is common method in image denosing  , which can mainly divide two categories to filter the noise: 1) improvement of denoising speed. 2) Improvement of denoising performance. For improving denoising speed, optimization method cooperated CNN was a good tool to rapidly find optimal solution in image denoising BIB017 BIB007 . For example, a GAN with maximum a posteriori (MAP) was used to estimate the noise and deal with other tasks, such as image inpainting and super-resolution BIB008 . An experience-based greedy and transfer learning strategies with CNN can accelerate genetic algorithm to obtain a clean image BIB009 . Noisy image and noise level mapping were as inputs of CNN, which had faster execution in predicting the noise  . For improving the denoising performance, CNN combined optimization methods to make noisy image smooth BIB010 BIB011 BIB004 . CNN with total variation (TV) reduced the effect of noise pixels BIB005 . Spliting Bregman iteration algorithm and CNN  can enhance pixels through image depth to obtain the latent clean image. A dual-stage CNN with feature matching can better recover the detailed information of the clean image, especially noisy images BIB012 . The GAN with nearest neighbor had good effect between noisy and clean images, and filtered the noisy image BIB013 . Wavefront coding jointed CNN to enhance pixels of latent clean image via transform domain BIB014 . Additionally, there are other excellent denoising methods as shown in BIB018 BIB019 BIB015 . Table  3 shows that detailed information of the combination of optimization method and CNN/NN in AWNI denoising.