In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage. Image restoration, including image denoising, deblurring, inpainting, etc., is one of the most important areas in imaging science. Its major purpose is to obtain high quality reconstructions of images corrupted in various ways during imaging, acquisiting, and storing, and enable us to see crucial but subtle objects that reside in the images. Image restoration has been an active research area. Numerous models and algorithms have been developed for the past few decades. Before the uprise of deep learning methods, there were two classes of image restoration approaches that were widely adopted in the field: transformation based approach and PDE approach. The transformation based approach includes wavelet and wavelet frame based methods BID11 BID3 , dictionary learning based methods BID0 , similarity based methods BID2 BID10 , low-rank models BID21 BID18 , etc. The PDE approach includes variational models BID31 BID35 BID1 , nonlinear diffusions BID33 BID6 BID38 , nonlinear hyperbolic equations BID32 , etc. More recently, deep connections between wavelet frame based methods and PDE approach were established BID4 BID12 .One of the greatest challenge for image restoration is to properly handle image degradations of different levels. In the existing transformation based or PDE based methods, there is always at least one tuning parameter (e.g. the regularization parameter for variational models and terminal time for nonlinear diffusions) that needs to be manually selected. The choice of the parameter heavily relies on the degradation level.Recent years, deep learning models for image restoration tasks have significantly advanced the state-of-the-art of the field. BID20 proposed a convolutional neural network (CNN) for image denoising which has better expressive power than the MRF models by BID22 . Inspired by nonlinear diffusions, BID9 designed a deep neural network for image denoising and BID40 improves the capacity by introducing a deeper neural network with residual connections. use the CNN to simulate a wide variety of image processing operators, achieving high efficiencies with little accuracy drop. However , these models cannot gracefully handle images with varied degradation levels. Although one may train different models for images with different levels, this may limit the application of these models in practice due to lack of flexibility.Taking blind image denoising for example. BID40 designed a 20-layer neural network for the task, called DnCNN-B, which had a huge number of parameters. To reduce number of parameters, BID24 proposed the UNLNet 5 , by unrolling a projection gradient algorithm for a constrained optimization model. However, BID24 also observed a drop in PSNR comparing to DnCNN. Therefore, the design of a light-weighted and yet effective model for blind image denoising remains a challenge. Moreover, deep learning based models trained on simulated gaussian noise images usually fail to handle real world noise, as will be illustrated in later sections.Another example is JPEG image deblocking. JPEG is the most commonly used lossy image compression method. However, this method tend to introduce undesired artifacts as the compression rate increases. JPEG image deblocking aims to eliminate the artifacts and improve the image quality. Recently, deep learning based methods were proposed for JPEG deblocking BID13 BID40 . However, most of their models are trained and evaluated on a given quality factor. Thus it would be hard for these methods to apply to Internet images, where the quality factors are usually unknown.In this paper, we propose a single image restoration model that can robustly restore images with varied degradation levels even when the degradation level is well outside of that of the training set. Our proposed model for image restoration is inspired by the recent development on the relation between deep learning and optimal control. The relation between supervised deep learning methods and optimal control has been discovered and exploited by BID39 ; BID26 BID7 BID16 . The key idea is to consider the residual block x n+1 = x n + f (x n ) as an approximation to the continuous dynamicsẊ = f (X). In particular, BID26 BID16 demonstrated that the training process of a class of deep models (e.g. ResNet by BID19 , PolyNet by BID42 , etc.) can be understood as solving the following control problem: DISPLAYFORM0 Here x 0 is the input, y is the regression target or label,Ẋ = f (X, w) is the deep neural network with parameter w(t), R is the regularization term and L can be any loss function to measure the difference between the reconstructed images and the ground truths.In the context of image restoration, the control dynamicẊ = f (X(t), ω(t)), t ∈ (0, τ ) can be, for example, a diffusion process learned using a deep neural network. The terminal time τ of the diffusion corresponds to the depth of the neural network. Previous works simply fixed the depth of the network , i.e. the terminal time, as a fixed hyper-parameter. However BID30 showed that the optimal terminal time of diffusion differs from image to image. Furthermore, when an image is corrupted by higher noise levels, the optimal terminal time for a typical noise removal diffusion should be greater than when a less noisy image is being processed. This is the main reason why current deep models are not robust enough to handle images with varied noise levels. In this paper, we no longer treat the terminal time as a hyper-parameter. Instead, we design a new architecture (see Fig. 3 ) that contains both a deep diffusion-like network and another network that determines the optimal terminal time for each input image. We propose a novel moving endpoint control model to train the aforementioned architecture. We call the proposed architecture the dynamically unfolding recurrent restorer (DURR).We first cast the model in the continuum setting. Let x 0 be an observed degraded image and y be its corresponding damage-free counterpart. We want to learn a time-independent dynamic systeṁ X = f (X(t), w) with parameters w so that X(0) = x and X(τ ) ≈ y for some τ > 0. See Fig. 2 for an illustration of our idea. The reason that we do not require X(τ ) = y is to avoid over-fitting . For varied degradation levels and different images, the optimal terminal time τ of the dynamics may vary. Therefore, we need to include the variable τ in the learning process as well. The learning of the dynamic system and the terminal time can be gracefully casted as the following moving endpoint control problem: DISPLAYFORM1 Different from the previous control problem, in our model the terminal time τ is also a parameter to be optimized and it depends on the data x. The dynamic systemẊ = f (X(t), w) is modeled by a recurrent neural network (RNN) with a residual connection, which can be understood as a residual network with shared weights BID25 . We shall refer to this RNN as the restoration unit. In order to learn the terminal time of the dynamics, we adopt a policy network to adaptively determine an optimal stopping time. Our learning framework is demonstrated in Fig. 3 . We note that the above moving endpoint control problem can be regarded as the penalized version of the well-known fixed endpoint control problem in optimal control BID15 , where instead of penalizing the difference between X(τ ) and y, the constraint X(τ ) = y is strictly enforced.In short, we summarize our contribution as following:• We are the first to use convolutional RNN for image restoration with unknown degradation levels, where the unfolding time of the RNN is determined dynamically at run-time by a policy unit (could be either handcrafted or RL-based).• The proposed model achieves state-of-the-art performances with significantly less parameters and better running efficiencies than some of the state-of-the-art models.• We reveal the relationship between the generalization power and unfolding time of the RNN by extensive experiments. The proposed model, DURR, has strong generalization to images with varied degradation levels and even to the degradation level that is unseen by the model during training (Fig. 1 ).• The DURR is able to well handle real image denoising without further modification. Qualitative results have shown that our processed images have better visual quality, especially sharper details compared to others. In this paper, we proposed a novel image restoration model based on the moving endpoint control in order to handle varied noise levels using a single model. The problem was solved by jointly optimizing two units: restoration unit and policy unit. The restoration unit used an RNN to realize the dynamics in the control problem. A policy unit was proposed for the policy unit to determine the loop times of the restoration unit for optimal results. Our model achieved the state-of-the-art results in blind image denoising and JPEG deblocking. Moreover, thanks to the flexibility of the given policy, DURR has shown strong abilities of generalization in our experiments.
We propose a novel method to handle image degradations of different levels by learning a diffusion terminal time. Our model can generalize to unseen degradation level and different noise statistic.
{"src": "scitldr-AIC_train", "paper_id": "SJfZKiC5FX"}