Neural networks have reached outstanding performance for solving various ill-posed inverse problems in imaging. However, drawbacks of end-to-end learning approaches in comparison to classical variational methods are the requirement of expensive retraining for even slightly different problem statements and the lack of provable error bounds during inference. Recent works tackled the first problem by using networks trained for Gaussian image denoising as generic plug-and-play regularizers in energy minimization algorithms. Even though this obtains state-of-the-art on many tasks, heavy restrictions on the network architecture have to be made if provable convergence of the underlying fixed point iteration is a requirement. More recent work has proposed to train networks to output descent directions with respect to a given energy function with a provable guarantee of convergence to a minimizer of that energy. However, each problem and energy requires the training of a separate network. In this paper we consider the combination of both approaches by projecting the outputs of a plug-and-play denoising network onto the cone of descent directions to a given energy. This way, a single pre-trained network can be used for a wide variety of reconstruction tasks. Our show improvements compared to classical energy minimization methods while still having provable convergence guarantees. In many image processing tasks an observed image f is modeled as the of the transformation of a clean imageû under a known (linear) operator A and unknown noise ξ, f = Aû + ξ. (In most cases, the problem of reconstructingû from f and A is ill-posed and can thus not be solved by a simple inversion of A, giving rise to the field of regularization theory with iterative or variational methods, see e.g. for an overview. In recent years neural networks were very successful in learning a direct mapping G(f) ≈û for a variety of problems such as deblurring, denoising, super-resolution, demosaicing and MRI-or CT-reconstruction. Even though this works well in practice, there are rarely any guarantees on the behaviour of neural networks on unseen data, making them difficult to use in safety-critical applications. Moreover, for each problem and type of noise a separate network has to be trained. In contrast, classical variational methods try to find the solution by the minimization of a suitable energy function of the formû where H f is a data fidelity term, for example commonly chosen as H f (u) = 1 2 ||Au − f || 2, and R is a regularization function that models prior knowledge about the solution, e.g. the popular total variation (TV) regularization, R(u) = ∇u 1,. While minimizers of come with many desirable theoretical guarantees, regularizations like the TV often cannot perfectly capture the complex structure of the space of natural images. To combine the advantages of powerful feed-forward networks and model-based approaches like, authors have considered various hybrid models like learning regularizers (e.g. ), designing networks architectures that resemble the structure of minimization algorithms or differential equations, e.g., interleaving networks with classical optimization steps, or using the parametrization of networks as a regularization for, see e.g.. A particularly flexible approach arises from, where proximal operators with respect to the regularizer are replaced by arbitrary denoising operators, with recent works focusing on the use of denoising networks. While such approaches allow to tackle different inverse problems with the same neural network, the derivation of theoretical guarantees -even in terms of the convergence of the ing algorithmic scheme -remains difficult, see or some discussion in, unless the denoiser satisfies particular properties. The starting point of the above-mentioned algorithmic schemes that utilize denoising networks to regularize model-based inverse problems are methods for the minimization of. While most works focus on primal-dual / ADMM approaches, their convergence analysis is quite delicate even in a setting in which one still minimizes (nonconvex) energies, such that we turn to two simpler methods, gradient descent and proximal gradient methods, where Following the idea of, considering either a gradient descent or a proximal step on the regularization as a generic denoising operation gives rise to the following two algorithmic schemes, where G denotes any kind of denoiser, e.g. a convolutional neural network, and we define ρ(u for the sake of brevity of notation. We refer to for a more detailed derivation. Algorithmic schemes like or combine the model-based flexibility of energy minimization methods (i.e. explicit modelling of H f) with the expressive power of deep neural networks G. Unfortunately -despite their success in various practical applications -schemes like or remain dangerous to be used: Figure 1 shows the of running the iteration with H f = 0 on a noisy input image f = u 0 for 100 and 800 iterations using a DnCNN preimplemented in Matlab as the denoiser G. As we can see the image gets completely distorted. Even more strikingly, the range of the image increased from values in to an interval of [−185, 218] within the first 1000 iterations. Clearly, the algorithmic scheme diverges. A natural condition for the provable convergence of a scheme like (at least along subsequences) would be a 1-Lipschitz continuous operator G. There has been previous work on computing upper bounds for the best Lipschitz constant of a network and using it to enforce a user defined Lipschitz constant L during training time but we found that enforcing non-expansiveness drastically decreased the denoising performance. The problem of computing the best Lipschitz constant, in hope of improving those , was recently proved to be NP-hard and thus is infeasible. Therefore, we adapt the recent idea proposed in to safeguard neural networks by forcing them to predict a descent direction to a given model-based energy, such that it can be used within a line search algorithm to guarantee convergence. More precisely, at any given estimate u and model-based energy E the authors use the Euclidean projection onto the half space as the last layer of their network. Even though the ing algorithm converges to the minimizer of E, experiments showed significantly higher peaks of the PSNR value in early iterations compared to classical gradient descent on E. Intuitively, the descent direction proposed by the network pushes the iteration closer towards the distribution of the training data than a usual gradient descent step. While the approach of has to train a separate network for each inverse problem and each type of noise, we investigate the combination of the flexible algorithmic schemes and with the idea from to project onto the half-space of descent directions to safeguard the underlying algorithm. In the following G will always refer to a generic denoising network, like DnCNN. We assume that E(u) = H f (u) + R(u) is a continuously differentiable, strictly convex and coercive energy function. As a first step, we simply rewrite the algorithmic schemes and in such a way that they resemble a gradient descent iteration, i.e., such that we can interpret as "update directions" of the respective algorithmic schemes. Because the plain iterations and can easily be divergent, we safeguard them by projecting them onto the half-space of descent directions C(γ, ∇E(u k)), i.e., Note that we replaced the averaging of the gradient descent and denoising step in by an abitrary convex combination using a parameter α to determine the respective influence of the data term and the denoising more flexibly. After computing the above directions d k, we update our iterates using with a step size t k chosen based on a backtracking line-search mechanism similar to. Under weak additional conditions, the latter guarantees the convergence of the proposed scheme to the minimizer of E. Such a minimizer could of course be determined by any classical algorithm, but we hope for to yield a better path towards the true minimizer, and consider a discrepancy principle for stopping the iteration before convergence. More precisely, we terminate as soon as for H f (û) being an estimate on the (data-term-dependent measure of the) noise level of the considered problem, and β being a scaling factor (typically close to 1). We tested our implementation with the image reconstruction tasks of Gaussian deblurring with standard deviation 1 and 4× single image super resolution. In both cases we added Gaussian noise with standard deviation 0.02 to the corrupted image. We chose the PyTorch implementation 1 of DnCNN pre-trained on a noise level of 0.1 as our denoising network. Our surrogate energy uses a TV regularization with Huber-norm instead of the 1 -norm. As our data term we choose H f (u) = operator. The best hyperparameters for all methods were found with a grid search. In all experiments, for scheme (conv), α = 0 was the best choice for any τ, indicating that the gradient descent step on the data term does not yield much additional information, assumably because the projection onto C(γ, ∇E(u)) which depends on the gradient of the data term anyway. When using (prox), we empirically found τ = 30 to be the best choice. For the projection onto the half-space of descent directions, we we used γ = 5 for both methods for deblurring, and γ = 50 in (conv) and γ = 1 in (prox) for super resolution. For fairness, the classical gradient descent was also implemented using backtracking line search. Figure 2 shows the reconstruction quality of the current iterate compared to ground truth over a span of 500 iterations. The PSNR quickly peaks before slowly converging to the fixed point of the surrogate energy which is consistent with the of. Notably, the convex combination method peaks earlier but not as high as the prox method. Tables 1 and 2 show using early stopping using a discrepancy principle. On all test images our prox scheme beats gradient descent. Table 1: PSNR values for deblurring for varying images and stopping criteria. The algorithm was stopped when H f (u k) < βH f (û). best refers to the highest PSNR over 500 iterations and a "*" means that the stopping criterion was not triggered such that the last iteration was used instead. Table 2: PSNR values for super resolution for varying images and stopping criteria. The algorithm was stopped when H f (u k) < βH f (û). best refers to the highest PSNR over 500 iterations. We combine deep learning and energy minimization methods for solving inverse problems in image reconstruction into a provably convergent algorithmic scheme. Still, our approach is able to generalize to different problems with a single denoising network and without the need to retrain if that problem changes. We were able to reach better than the energy minimization baseline in our experiments, and are happy to elaborate on the above aspects in the NeurIPS workshop.
We use neural networks trained for image denoising as plug-and-play priors in energy minimization algorithms for image reconstruction problems with provable convergence.