Deep neural networks provide state-of-the-art performance for image denoising, where the goal is to recover a near noise-free image from a noisy image.
 The underlying principle is that neural networks trained on large datasets have empirically been shown to be able to generate natural images well from a low-dimensional latent representation of the image.
 Given such a generator network, or prior, a noisy image can be denoised by finding the closest image in the range of the prior.
 However, there is little theory to justify this success, let alone to predict the denoising performance as a function of the networks parameters.
 In this paper we consider the problem of denoising an image from additive Gaussian noise, assuming the image is well described by a deep neural network with ReLu activations functions, mapping a k-dimensional latent space to an n-dimensional image.
 We state and analyze a simple gradient-descent-like iterative algorithm that minimizes a non-convex loss function, and provably removes a fraction of (1 - O(k/n)) of the noise energy.
 We also demonstrate in numerical experiments that this denoising performance is, indeed, achieved by generative priors learned from data. We consider the image or signal denoising problem, where the goal is to remove noise from an unknown image or signal. In more detail, our goal is to obtain an estimate of an image or signal y˚P R n from y " y˚`η, where η is unknown noise, often modeled as a zero-mean white Gaussian random variable with covariance matrix σ 2 {nI.Image denoising relies on modeling or prior assumptions on the image y˚. For example, suppose that the image y˚lies in a k-dimensional subspace of R n denoted by Y. Then we can estimate the original image by finding the closest point in 2 -distance to the noisy observation y on the subspace Y. The corresponding estimate, denoted byŷ, obeys DISPLAYFORM0 with high probability (throughout, }¨} denotes the 2 -norm). Thus, the noise energy is reduced by a factor of k{n over the trivial estimateŷ " y which does not use any prior knowledge of the signal. The denoising rate (1) shows that the more concise the image prior or image representation (i.e., the smaller k), the more noise can be removed. If on the other hand the prior (the subspace, in this example) does not include the original image y˚, then the error bound (1) increases as we would remove a significant part of the signal along with noise when projecting onto the range of the signal prior. Thus a concise and accurate prior is crucial for denoising.Real world signals rarely lie in a priori known subspaces, and the last few decades of image denoising research have developed sophisticated and accurate image models or priors and algorithms. Examples include models based on sparse representations in overcomplete dictionaries such as wavelets (Donoho, 1995) and curvelets (Starck et al., 2002) , and algorithms based on exploiting self-similarity within images BID4 . A prominent example of the former class of algorithms is the BM3D BID4 algorithm, which achieves state-of-the-art performance for certain denoising problems. However, the nuances of real world images are difficult to describe with handcrafted models. Thus, starting with the paper (Elad & Aharon, 2006 ) that proposes to learn sparse representation based on training data, it has become common to learn concise representation for denoising (and other inverse problems) from a set of training images.In 2012, Burger et al. BID2 applied deep networks to the denoising problem, by training a deep network on a large set of images. Since then, deep learning based denoisers (Zhang et al., 2017) have set the standard for denoising. The success of deep network priors can be attributed to their ability to efficiently represent and learn realistic image priors, for example via autodecoders (Hinton & Salakhutdinov, 2006) and generative adversarial models (Goodfellow et al., 2014) . Over the last few years, the quality of deep priors has significantly improved (Karras et al., 2017; Ulyanov et al., 2017) . As this field matures, priors will be developed with even smaller latent code dimensionality and more accurate approximation of natural signal manifolds. Consequently, the representation error from deep priors will decrease, and thereby enable even more powerful denoisers.As the influence of deep networks in inverse problems grows, it becomes increasingly important to understand their performance at a theoretical level. Given that most optimization approaches for deep learning are first order gradient methods, a justification is needed for why they do not get stuck in local minima. The closest theoretical work to this question is BID1 , which solves a noisy compressive sensing problem with generative priors by minimizing empirical risk. Under the assumption that the network is Lipschitz, they show that if the global optimizer can be found, which is in principle NP-hard, then a signal estimate is recovered to within the noise level. While the Lipschitzness assumption is quite mild, the resulting theory does not provide justification for why global optimality can be reached.The most related work that establishes theoretical reasons for why gradient methods would not get stuck in local minima, when using deep generative priors for solving inverse problems, is Hand & Voroninski (2018) . In it, the authors establish global favorability for optimization of the noiseless empirical risk function. Specifically, they show existence of a descent direction outside a ball around the global optimizer and a negative multiple of it in the latent space of the generative model. This work does not provide a specific algorithm which provably estimates the global minimizer, nor does it provide an analysis of the robustness of the problem with respect to noise.In this paper, we propose the first algorithm for solving denoising with deep generative priors that provably finds an approximation of the underlying image. Following the lead of Hand & Voroninski (2018), we assume an expansive Gaussian model for the deep generative network in order to establish this result.Contributions: The goal of this paper is to analytically quantify the denoising performance of deep-prior based denoisers. Specifically, we characterize the performance of a simple and efficient algorithm for denoising based on a d-layer generative neural network G : R k Ñ R n , with k ă n, and random weights. In more detail, we propose a gradient method with a tweak that attempts to minimize the least-squares loss f pxq " 1 2 }Gpxq´y} 2 between the noisy image y and an image in the range of the prior, Gpxq. While f is non-convex, we show that the gradient method yields an estimatex obeying DISPLAYFORM1 with high probability, where the notation À absorbs a constant factor depending on the number of layers of the network, and its expansitivity, as discussed in more detail later. Our result shows that the denoising rate of a deep prior based denoiser is determined by the dimension of the latent representation.We also show in numerical experiments, that this rate-shown to be analytically achieved for random priors-is also experimentally achieved for priors learned from real imaging data. Loss surface f pxq " }Gpxq´Gpx˚q}, x˚" r1, 0s, of an expansive network G with ReLu activation functions with k " 2 nodes in the input layer and n 2 " 300 and n 3 " 784 nodes in the hidden and output layers, respectively, with random Gaussian weights in each layer. The surface has a critical point near´x˚, a global minimum at x˚, and a local maximum at 0.
By analyzing an algorithms minimizing a non-convex loss, we show that all but a small fraction of noise can be removed from an image using a deep neural network based generative prior.
{"src": "scitldr-AIC_train", "paper_id": "SklcFsAcKX"}