We describe techniques for training high-quality image denoising models that require only single instances of corrupted images as training data. Inspired by a recent technique that removes the need for supervision through image pairs by employing networks with a "blind spot" in the receptive field, we address two of its  shortcomings: inefficient training and poor final denoising performance. This is achieved through a novel blind-spot convolutional network architecture that allows efficient self-supervised training, as well as application of Bayesian distribution prediction on output colors. Together, they bring the self-supervised model on par with fully supervised deep learning techniques in terms of both quality and training speed in the case of i.i.d. Gaussian noise. Denoising, the removal of noise from images, is a major application of deep learning. Several architectures have been proposed for general-purpose image restoration tasks, e.g., U-Nets BID13 , hierarchical residual networks BID11 , and residual dense networks BID17 . Traditionally, the models are trained in a supervised fashion with corrupted images as inputs and clean images as targets, so that the network learns to remove the corruption. BID9 introduced NOISE2NOISE training, where pairs of corrupted images are used as training data. They observe that when certain statistical conditions are met, a network faced with the impossible task of mapping corrupted images to corrupted images learns, loosely speaking, to output the "average" image. For a large class of image corruptions, the clean image is a simple per-pixel statistic -such as mean, median, or mode -over the stochastic corruption process, and hence the restoration model can be supervised using corrupted data by choosing the appropriate loss function to recover the statistic of interest.While removing the need for clean training images, NOISE2NOISE training still requires at least two independent realizations of the corruption for each training image. While this eases data collection significantly compared to noisy-clean pairs, large collections of (single) poor images are still much more widespread. This motivates investigation of self-supervised training: how much can we learn from just looking at bad data? While foregoing supervision would lead to the expectation of some regression in performance, can we make up for it by making stronger assumptions about the corruption process? In this paper, we show that under the assumption of additive Gaussian noise that is i.i.d. between pixels, no concessions in denoising performance are necessary.We draw inspiration from the recent NOISE2VOID (N2V) training technique of BID7 . The algorithm needs no image pairs, and uses just individual noisy images as training data, assuming that the corruption is zero-mean and independent between pixels. The method is based on blind-spot networks where the receptive field of the network does not include the center pixel. This allows using the same noisy image as both training input and training target -because the network cannot see the correct answer, using the same image as target is equivalent to using a different noisy realization. This approach is self-supervised in the sense that the surrounding context is used to predict the value of the output pixel without a separate reference image BID3 .The networks used by BID7 do not have a blind spot by design, but are trained to ignore the center pixel using a masking scheme where only a few output pixels can contribute to the loss function, reducing training efficiency considerably. We remedy this with a novel architecture that allows efficient training without masking. Furthermore , the existence of the blind spot leads to poor denoising quality. We derive a scheme for combining the network output with data in the blind spot, bringing the denoising quality on par with conventionally trained networks. In our blind-spot network architecture, we effectively construct four denoiser network branches, each having its receptive field restricted to a different direction. A single-pixel offset at the end of each branch separates the receptive field from the center pixel. The results are then combined by 1×1 convolutions. In practice, we run four rotated versions of each input image through a single receptive field -restricted branch, yielding a simpler architecture that performs the same function. This also implicitly shares the convolution kernels between the branches and thus avoids the four-fold increase in the number of trainable weights. For the baseline experiments, as well as for the backbone of our blind-spot networks, we use the same U-Net BID13 architecture as BID9 , see their appendix for details. The only differences are that we have layers DEC CONV1A and DEC CONV1B output 96 feature maps like the other convolution layers at the decoder stage, and layer DEC CONV1C is removed. After combining the four receptive field restricted branches, we thus have 384 feature maps. These are fed into three successive 1×1 convolutions with 384, 96, and n output channels, respectively, where n is the number of output components for the network. All convolution layers except the last 1×1 convolution use leaky ReLU with α = 0.1 (Maas et al., 2013). All networks were trained using Adam with default parameters BID6 , learning rate λ = 0.0003, and minibatch size of 4. As training data, we used random 256×256 crops from the 50K images in the ILSVRC2012 (Imagenet) validation set. The training continued until 1.2M images were shown to the network. All training and test images were corrupted with Gaussian noise, σ = 25. Table 1 shows the denoising quality in dB for the four test datasets used. From the BSD300 dataset we use the 100 validation images only. Similar to BID7 , we use the grayscale version of the BSD68 dataset -for this case we train a single-channel (c = 1) denoiser using only the luminance channel of the training images. All our blind-spot noise-to-noise networks use the convolutional architecture (Section 2) and are trained without masking. In BSD68 our simplified L2 variant closely matches the original NOISE2VOID training, suggesting that our network with an architecturally enforced blind spot is approximately as capable as the masking-based network trained by BID7 . We see that the denoising quality of our Full setup (Section 3) is on par with baseline results of N2N and N2C, and clearly surpasses standard blind-spot denoising (L2) that does not exploit the information in the blind spot. Doing the estimation separately for each color BID9 and BID7 . Full is our blind-spot training and denoising method as described in Section 3. Per-comp. is an ablated setup where each color component is treated as an independent univariate Gaussian, highlighting the importance of expressing color outputs as multivariate distributions. L2 refers to training using the standard L2 loss function and ignoring the center pixel when denoising. Columns N2N and N2C refer to NOISE2NOISE training of BID9 and traditional supervised training with clean targets (i.e., noise-to-clean), respectively. Results within 0.05 dB of the best result for each dataset are shown in boldface. channel (Per-comp.) performs significantly worse, except in the grayscale BSD68 dataset where it is equivalent to the Full method. FIG1 shows example denoising results. Our Full setup produces images that are virtually identical to the N2N baseline both visually and in terms of PSNR. The ablated Per-comp. setup tends to produce color artifacts, demonstrating the shortcomings of the simpler per-component univariate model. Finally, the L2 variant that ignores the center pixel during denoising produces visible checkerboard patterns, some of which can also be seen in the result images of BID7 . We have shown that self-supervised training -looking at noisy images only, without the benefit of seeing the same image under different noise realizations -is sufficient for learning deep denoising models on par with those that make use of another realization as a training target, be it clean or corrupted. Currently this comes at the cost of assuming pixel-wise independent noise with a known analytic likelihood model.
We learn high-quality denoising using only single instances of corrupted images as training data.
{"src": "scitldr-AIC_train", "paper_id": "H1e7g4edw4"}