Image Restoration by Solving IVP

Recent research on image restoration have achieved great success with the aid of deep learning technologies, but, many of them are limited to dealing SR with realistic settings. To alleviate this problem, we introduce a new formulation for image super-resolution to solve arbitrary scale image super-resolution methods. Based on the proposed new SR formulation, we can not only super-resolve images with multiple scales, but also find a new way to analyze the performance of super-resolving process. We demonstrate that the proposed method can generate high-quality images unlike conventional SR methods.


Introduction
Super-resolution (SR) task is used to increase the image resolution by estimating its underlying high frequency details. However, SR is a highly ill-posed problem since for any lowresolution input, there are multiple high-resolution solutions, which makes it a challenging problem. Therefore, considerable methods have been studied to solve SR problem. First, interpolation-based methods are simple and efficient, and thus they are widely used in many applications. However, these naive approaches have a clear performance limitation. Deep-learning based methods are being very successful in generating high quality images from lowresolution image, and provide quantitatively promising results. Moreover, several generative SR networks which are composed of a highresolution image generator and discriminator, can generate visually more plausible results with the help of numerous perceptual losses (e.g. VGG and adversarial losses). However, these previous learning-based SR approaches are limited to fixed scaling factors (e.g., x2, x3, and x4) to allow quick inference. Therefore, there has been several researches to solve SR problem with an arbitrary scaling factor. Recently, there has been increasing attempts to model differential equations with neural network. In particular, Neural ordinary differential equations (Neural ODE), allows to formulate a differential path from low-resolution input to highresolution output with a neural network. However, this Neural ODE based SR approach uses the Neural ODE as an intermediate layer of HR generator without clear reasoning in the process of modeling a differential equation. Therefore, in this work, we tackle this problem and present a new SR approach with Neural ODE, and our contributions can be summarized as follows: • We propose a new formulation to perform superresolution on arbitrary scale.
• We find a way to analyze the differential path from lowresolution to high-resolution.
• Our method achieves performance close to existing state-of-the-art methods.
2 Related Works

Image Restoration
Recent works on single image restoration focus on learning mapping functions between degraded image and original image. SRCNN [Dong et al., 2015] proposed to learn the nonlinear mapping from LR image to HR image using a CNN model for the first time. VDSR [Kim et al., 2016b] increased the depth of CNN to model more complex LR-HR mappings. Recent studies have applied different kinds of skip connections to ease the optimization process [Lim et al., 2017;Zhang et al., 2018]. Parallel to devising improved feed-forward CNN architectures, many attempts have been done to develop SISR methods that can be better applied to real-world situations. Re-alSR [Cai et al., 2019] proposed a more realistic degradation model to make more natural training dataset and presented a new method to cope with the given degradation settings. Meta-SR [Hu et al., 2019] proposed Meta-Upscale module to handle arbitrary scale factors for SISR.

Gradual Image Restoration
gradual approaches have been proven to perform well in lowlevel vision tasks. In addition to learning a complex nonlinear mapping of a low-quality image to a high-quality one, these approaches decompose this process into multiple steps and iteratively refine the output images [Kim et al., 2016a;Tai et al., 2017;Haris et al., 2018;Li et al., 2019]. DBPN [Haris et al., 2018] proposed a dedicated neural network design that provides an iterative error correcting mechanism to address mutual dependencies of LR and HR images. SRFBN [Li et al., 2019] proposed an efficient recurrent neural network that employs the feedback mechanism, which iteratively improves the input of the network in each step. However, these methods do not have a clear underlying formulation or theoretical analysis on the gradual image restoration process, which consequently require a large amount of engineering to develop the neural architectures [Li et al., 2019;Haris et al., 2018] and specialized training strategies [Li et al., 2019;Kim et al., 2016a]. LapSRN [Lai et al., 2017] can produce large SR results (e.g., x8) with intermediate SR results (e.g., x2, x4), but it can only handle the pre-determined discrete scale factors such as x2, x4, and x8.

Neural Ordinary Differential Equation
Recently, many attempts have been done to integrate differential formulations and deep learning methodologies. These attempts have led to neural ordinary differential equations (NODE) . NODE is a new family of deep neural network models that parameterizes a differential form using a neural network and produces the output by using an ODE solver. Meanwhile, differential equations have often been involved in image restoration task by modeling nonlinear reaction-diffusion and total variation schemes [Chen and Pock, 2016;Rudin et al., 1992]. Such is also the case in deep-learning-based image restoration methods. [He et al., 2019] designed new neural network architectures inspired by the differential equation solving process, such as Leapfrog and Runge-Kutta approaches. In particular, to solve the SR problem, [Scao, 2020] utilized NODE as a part of neural architecture with an integration. However, the interval of the integral is not fixed (open) and they did not present any concrete formulation on the intermediate images involved in the gradual SR process. Consequently, they need to empirically speculate the optimal neural architecture and the integration interval.

Proposed Method
In this section, we present a new neural approach for the gradual SR reconstruction. We first formulate the SR problem as a gradual SR process with an ODE. We then elaborate how to perform SR with the proposed formulation and how to train it.

gradual Super-Resolution Formulation
Existing SR methods utilizing gradual SR process [Lai et al., 2017;Haris et al., 2018;Li et al., 2019] are based on iterative multi-stage approaches and can be viewed as variants of the following: where n denotes the iteration step, I 0 denotes the given initial input LR image, and I n is the iteratively refined image from its previous state I n−1 . These approaches typically produce multiple intermediate HR images during the refinement, and the rendered image at the last N -th iteration [Tai et al., 2017;Li et al., 2019]  To alleviate these problems, we formulate the gradual SR process with a differential equation. This allows us to implement and train the SR networks in an established way while outperforming the performance of conventional gradual SR process. Assume that (I HR ) ↓ t is a downscaled version of a groundtruth clean image I HR using a traditional SR kernel (e.g., bicubic) with a scaling factor 1 t . We then define I(t) by upscaling (I HR ) ↓ t using that SR kernel with a scaling factor t so that I HR and I(t) have the same spatial resolution (see the illustration of "Generating LR image" in (a). Note that t ≥ 1, and I(1) denotes the ground-truth clean image I HR .
To model a gradual SR process,we first estimate the highfrequency image residual with a neural network. Specifically, when t is a conventional discrete-scaling factor (e.g., x2, x3, and x4), image residual between I(t) and I(t − 1) can be modeled using a neural network f discrete as: (2) Notably, I(t − 1) includes more high-frequency details than I(t) without loss of generality. In our method, we model the slightest image difference to formulate a continuously gradual SR process. Therefore, we take the scale factor t to continuous domain, and reformulate (2) as an ODE with a neural network f as: where θ denotes the trainable parameter of the network f . Using this formulation, we can predict the high-frequency image detail required to slightly enhance I(t) with the network f . (Note that we can obtain I(t) with any rational number t by adding padding to the border of image before resizing and then center cropping the image.) As existing SR neural networks have been proven to be successful at predicting the high-frequency residual image [Kim et al., 2016b], we can use conventional SR architectures as our network f without major changes.

Single Image Super-Resolution with Neural
Ordinary Differential Equation In this section, we explain how to super-resolve a given LR image with a continuous scaling factor using our ODE-based SR formulation. First, we obtain I(t 0 ) by upscaling the given LR input image ("Test time LR image" in (a) using the bicubic SR kernel to a desired output resolution with a scaling factor t 0 . Next, we solve the ODE initial value problem with the initial condition I(t 0 ) by integrating the neural network f from t 0 to 1 to acquire the high-quality image I(1) as follows: As the neural network f is modeled to predict desired highfrequency details, our formulation gradually adds the predicted fine details from the input LR image I(t 0 ) through the integration shown as the solid orange line in (a). Thus, our SR approach becomes a gradual SR process which adds the high-frequency details gradually. To compute the integration with f in the proposed formulation, we use conventional ODE solvers to numerically calculate the output image I(1). Specifically, we approximate the high-quality image I(1) given a fully trained neural network f , network parameter θ, initial condition I(t 0 ), and integral interval [t 0 , 1] using an ODE solver (ODESolve()) as: Thus, our method does not need to consider the stop condition (i.e., the number of feedback iterations) of the gradual SR process unlike conventional approaches [Tai et al., 2017;Li et al., 2019]. Notably, we can use conventional ODE solvers to render the desired outputs for the inference, but the solutions should be differentiable to train the network f through the backpropagation scheme. We compare the SR performance with different ODE solvers (e.g., Runge-Kutta and Euler methods) in our experiments. Our formulation is made upon a continuous context, allows a continuous scale factor t 0 where t 0 ≥ 1. This makes our method able to handle the arbitrary-scale SR problem. But unlike conventional multi-scale SR methods [Kim et al., 2016b;Lim et al., 2017;Hu et al., 2019] that successfully learn multi-scale SR tasks by sharing common features across various scales, we explicitly learn the relationship between images with different scales in image domain itself rather than the feature space.

Training
To train the deep neural network f , and learn the parameter θ in (3), we minimize the loss summed over scale factors t using the L1 loss function as: By minimizing the proposed loss function, our network parameter θ is trained to estimate the image detail to be added into the network input. Notably, during the training phase, we need to employ an ODE solver which allows end-to-end training using backpropagation with other components such as the neural net-

Experimental Results
In this section, we carry out extensive experiments to demonstrate the superiority of the proposed method, and add various quantitative and qualitative comparison results. We also provide detailed analysis of our experimental results. We will release our source code upon acceptance. For each CNN architecture, we change the first layer to feed the scale factor t as an additional input. To be specific, we extend the input channel from 3 to 4, and the pixel locations of the newly concatenated channel (4-th channel) are filled with a scalar value t as shown in (b). In addition, for RDN, we remove the last upsampling layer so that input and output resolutions are the same in our work. Note that, no extra parameters are added except for the first layers of the networks.
To train and infer the proposed SR process, we use the Python torchdiffeq library  to employ Runge-Kutta (RK4) method as our ODE solver in (4), which requires only 6 additional lines of code with PyTorch.
For simplicity, our approaches with VDSR and RDN backbones are called vdsr and RDN in the remaining parts of the experiments, respectively.
Dataset and evaluation. We use the DIV2K [Agustsson and Timofte, 2017] dataset to train our vdsr and RDN. During the training phase, we augment the dataset using random cropping, rotating, and flipping.
During the test phase, we evaluate the SR results in terms of PSNR and SSIM metrics on the standard benchmark datasets (Set14 [Zeyde et al., 2010], B100 (BSD100) [Martin et al., 2001], and Urban100 [Huang et al., 2015]). To be consistent with previous works, quantitative results are evaluated on the Y (luminance) channel in the YCbCr color space.
Training setting. We train the network by minimizing the L1 loss in (4) with the Adam optimizer (β 1 = 0.9, β 2 = 0.999, = 10 −8 ) [Kingma and Ba, 2015]. The initial learning rate is set as 10 −4 , which is then decreased by half every 100k gradient update steps, and trained for 600k iterations in total. The mini-batch size of vdsr is 16 (200x200 patches), but our vdsr takes 8 patches as a mini-batch (130x130 patches) owing to the memory limit of our graphic units. Similar to the training settings in Meta-SR [Hu et al., 2019], we train the network f by randomly changing the scale factor t in (4) from 1 to 4 with a stride of 0.1 (i.e., t ∈ {1.1, 1.2, 1.3, ..., 4}).

Comparison with gradual SR Methods
First, we compare our RDN with several state-of-the-art gradual SR methods: DRCN [Kim et al., 2016a], LapSRN [Lai et al., 2017], DRRN [Tai et al., 2017], D-DPBN [Haris et al., 2018, and SRFBN [Li et al., 2019]. As in [Lim et al., 2017], self-ensemble method is used to further improve RDN (denoted as RDN+). Note that, our RDN and RDN+ can handle multiple scale factors t including non-integer scale factors (e.g., x1.5) using the same network parameter. In contrast, other approaches are required to be trained for certain discrete integer scale factors (x2, x3, and x4) separately, resulting in a distinct parameter set for each scale factor. Nevertheless, quantitative restoration results show that our RDN, RDN+ consistently outperforms conventional gradual SR methods for the discrete integer scaling factors (x2, x3, and x4) in terms of PSNR. We We observe that DRRN and SRFBN fail to gradually refine patches with high-frequency details, while our RDN can gradually improve the intermediate images and render promising results at the final states.

Comparison with Multi-scale SR Methods
Our approach can handle a continuous scale factor for the SR task, thus we compare ours with existing multiscale SR methods that can handle continuous scale factors: VDSR [Kim et al., 2016b] and Meta-SR [Hu et al., 2019]. Notably, Meta-SR implemented using RDN (i.e., Meta-RDN) is the current state-of-the-art SR approach.
In Table 1, we show quantitative results compared to existing SR methods (VDSR, RDN, and Meta-RDN). Note that, VDSR+t and RDN+t are modified versions of VDSR and RDN to take the scale factor t as an additional input of the networks and have the same input and output resolutions as in our network f . We also compare our method with these new baselines (VDSR+t and RDN+t) for fair comparisons.
We evaluate the SR performance on the B100 benchmark dataset by increasing the scaling factor from 1.1 to 4.
Interestingly, we observe that vdsr outperforms VDSR and VDSR+t at every scale by a large margin although VDSR and VDSR+t have similar network architecture to our vdsr. Similarly, RDN shows better performance than Meta-RDN and RDN+t. We also provide qualitative comparison results with Meta-SR in Figure 1, and we see that our RDN recovers much clearer edges than Meta-RDN.

Detailed Analysis
Interpolation and extrapolation. We experiment our method on various scale factors that are not shown during the training phase. In Figure 2, we plot PSNR values from vdsr and RDN by changing the scale factor on the B100 dataset. We see that our method learns an interpolation ability and can successfully deal with unseen scales between 1 and 4 (e.g., 1.15, 1.25, ... 3.95). Moreover, ours also learns an extrapolation ability and handles unseen scale factors larger than 4 (e.g., 4.1, 4.2, ... , 4.5). To sum up, our proposed SR process has a power of generalization (i.e., interpolation and extrapolation abilities), even the network is trained with only a limited number of scale factors. SR performance with different ODE solvers. We experiment our method with different ODE solvers (e.g., Euler and RK4 methods). Note that Euler method is computationally cheaper than RK4, but RK4 provides more accurate approximation results generally. Similarly, in Table 2, we see that vdsr trained with RK4 shows slightly better SR performance than vdsr trained with Euler method on the B100 and Set5 datasets. This result suggests that, we can employ conventional ODE solvers to solve our own SR problem, but the quality of the predicted HR images are relying on the performance of the employed ODE solver. Visual output of the network f . In Figure 3, to see the intermediate results by the network f during the gradual SR   procedure at the test stage, we compute absolute value of f (Î(t), t, θ) where t decreases from 4 to 1, and the initial condition is I(t 0 = 4).
Interestingly, on the sharp patch corresponding to the eye (red box), the absolute values are higher when t is small. While, on the homogeneous patch corresponding to the cheek (green box), the absolute values are higher when t is large. Recall that I(t) becomes close to the groundtruth image when t gets small, and the image difference dI (t) dt (≈ f (Î(t), t, θ)) includes more high-frequency components. Therefore, the absolute values of the network at the eye region which includes high-frequency detail becomes large when t is small, while the absolute values at the homogeneous region which does not require high-frequency detail becomes small when t is small.

Conclusion
In this work, we proposed a novel differential equation for the SR task to gradually enhance a given input LR image, and allow continuous-valued scale factor. Image difference between images over different scale factors is physically modeled with a neural network, and formulated as a NODE. To restore a high-quality image, we solve the ODE initial value problem with the initial condition given as an input LR image. The main difference with existing gradual SR methods is that our formulation is based on the physical modeling of the intermediate images, and adds fine high-frequency details gradually. The analysis on the intermediate states during the SR process gives us more insight on the gradual SR reconstruction. Detailed experimental results show that our method achieves superior performance compared to state-of-the-art SR approaches.