Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB001 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Underwater images often suffer from color distortion and visibility degradation due to the light absorption and scattering. Existing methods utilize various assumptions/constrains to achieve reasonable solutions for underwater image enhancement. However, these methods share the common limitation that the adopted assumptions may not work for some particular scenes. To address this problem, this paper proposes an end to end framework for underwater image enhancement, where a CNN-based network called UIE-Net is presented. The UIE-net is trained with two tasks, color correction and haze removal. This unified training approach enables learning a strong feature representation for both tasks simultaneously. For better extracting the inherent features in local patches, a pixels disrupting strategy is exploited in the proposed learning framework, which significantly improves the convergent speed and accuracy. To handle the training of UIE-net, we synthesize 200000 training images based on the physical underwater imaging model. Experiments on benchmark underwater images for cross-scenes show that UIE-net achieves superior performance over existing methods. <s> BIB002 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> This letter reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high-resolution images to map the seafloor; however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects, such as absorption and scattering, which cause attenuation of light at different rates for different wavelengths. While this physical process is well described theoretically, the model depends on many parameters intrinsic to the water column as well as the structure of the scene. These factors make recovery of these parameters difficult without simplifying assumptions or field calibration; hence, restoration of underwater images is a nontrivial problem. Deep learning has demonstrated great success in modeling complex nonlinear systems but requires a large amount of training data, which is difficult to compile in deep sea environments. Using WaterGAN, we generate a large training dataset of corresponding depth, in-air color images, and realistic underwater images. These data serve as input to a two-stage network for color correction of monocular underwater images. Our proposed pipeline is validated with testing on real data collected from both a pure water test tank and from underwater surveys collected in the field. Source code, sample datasets, and pretrained models are made publicly available. <s> BIB003 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Underwater vision is subject to effects of underwater light propagation that act to absorb, scatter, and attenuate light rays between the scene and the imaging platform. Backscattering has been shown to have a strong impact on underwater images. As light interacts with particulate matter in the water column, it is scattered back towards the imaging sensor, resulting in a hazy effect across the image. A similar effect occurs in terrestrial applications in images of foggy scenes due to interaction with the atmosphere. Prior work on multi-image dehazing has relied on multiple cameras, polarization filters, or moving light sources. Single image dehazing is an ill-posed problem; proposed solutions rely on strong priors of the scene. This paper presents a novel method for underwater image dehazing using a light field camera to capture both the spatial and angular distribution of light across a scene. First, a 2D dehazing method is applied to each sub-aperture image. These dehazed images are then combined to produce a smoothed central view. Lastly, the smoothed central view is used as a reference to perform guided image filtering, resulting in a 4D dehazed underwater light field image. The developed method is validated on real light field data collected in a controlled in-lab water tank, with images taken in air for reference. This dataset is made publicly available. <s> BIB004 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Images taken underwater often suffer color distortion and low contrast because of light scattering and absorption. An underwater image can be modeled as a blend of a clear image and a background light, with the relative amounts of each determined by the depth from the camera. In this paper, we propose two neural network structures to estimate background light and scene depth, to restore underwater images. Experimental results on synthetic and real underwater images demonstrate the effectiveness of the proposed method. <s> BIB005 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Improving the quality of underwater image has a significant impact on many signal processing and computer vision applications, while haze-effect and color shift are main handicaps need to be surmounted. Due to the complexity of the underwater environmental factors, most existing image enhancement techniques cannot be directly applied to address this task. In this work, we develop a novel framework to jointly performing residual learning on transmission and image domains for underwater scene entrenchment. Indeed, our deep model consists of a data-driven residual architecture for transmission estimation and a knowledge-driven scene residual formulation for underwater illumination balance. Therefore, we can aggregate the prior knowledge and data information to investigate the underlying underwater image distribution. Moreover, by introducing adaptive exposure map, image colors will also be corrected accordingly. Experimentally, both quantitative and qualitative analysis can indicate outstanding effectiveness of the proposed algorithm, against state-of-the-art approaches. <s> BIB006 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> In an underwater scene, wavelength-dependent light absorption and scattering degrade the visibility of images, causing low contrast and distorted color casts. To address this problem, we propose a convolutional neural network based image enhancement model, i.e., UWCNN, which is trained efficiently using a synthetic underwater image database. Unlike the existing works that require the parameters of underwater imaging model estimation or impose inflexible frameworks applicable only for specific scenes, our model directly reconstructs the clear latent underwater image by leveraging on an automatic end-to-end and data-driven training mechanism. Compliant with underwater imaging models and optical properties of underwater scenes, we first synthesize ten different marine image databases. Then, we separately train multiple UWCNN models for each underwater image formation type. Experimental results on real-world and synthetic underwater images demonstrate that the presented method generalizes well on different underwater scenes and outperforms the existing methods both qualitatively and quantitatively. Besides, we conduct an ablation study to demonstrate the effect of each component in our network. <s> BIB007 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Underwater vision suffers from severe effects due to selective attenuation and scattering when light propagates through water. Such degradation not only affects the quality of underwater images but limits the ability of vision tasks. Different from existing image enhancement and restoration methods which either ignore the wavelength dependency of the attenuation or assume a specific spectral profile, we tackle color distortion problem of underwater image from a new view. In this letter, we propose a weakly supervised color transfer method to correct color distortion. Inspired by Cycle-Consistent Adversarial Networks, we design a multi-term loss function including adversarial loss, cycle consistency loss, and SSIM (Structural Similarity Index Measure) loss. In this way, we translate the color of underwater image as if it is taken in the air, and preserve the content and structure of original underwater image. Experiments on underwater images captured under diverse scenes show that the proposed method can produce visually pleasing results, even outperforms the art-of-the-state methods. Besides, our method can improve the performance of vision tasks. <s> BIB008 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Underwater images suffer from color distortion and low contrast, because light is attenuated while it propagates through water. Attenuation under water varies with wavelength, unlike terrestrial images where attenuation is assumed to be spectrally uniform. The attenuation depends both on the water body and the 3D structure of the scene, making color restoration difficult. Unlike existing single underwater image enhancement techniques, our method takes into account multiple spectral profiles of different water types. By estimating just two additional global parameters: the attenuation ratios of the blue-red and blue-green color channels, the problem is reduced to single image dehazing, where all color channels have the same attenuation coefficients. Since the water type is unknown, we evaluate different parameters out of an existing library of water types. Each type leads to a different restored image and the best result is automatically chosen based on color distribution. We also contribute a dataset of 57 images taken in different locations. To obtain ground truth, we placed multiple color charts in the scenes and calculated its 3D structure using stereo imaging. This dataset enables a rigorous quantitative evaluation of restoration algorithms on natural images for the first time. <s> BIB009 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Turbid underwater environment poses great difficulties for the applications of vision technologies. One of the biggest challenges is the complicated noise distribution of the underwater images due to the serious scattering and absorption. To alleviate this problem, this work proposes a deep pixel-to-pixel networks model for underwater image enhancement by designing an encoding-decoding framework. It employs the convolution layers as encoding to filter the noise, while uses deconvolution layers as decoding to recover the missing details and refine the image pixel by pixel. Moreover, skip connection is introduced in the networks model in order to avoid low-level features losing while accelerating the training process. The model achieves the image enhancement in a self-adaptive data-driven way rather than considering the physical environment. Several comparison experiments are carried out with different datasets. Results show that it outperforms the state-of-the-art image restoration methods in underwater image defogging, denoising and colour enhancement. <s> BIB010 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Underwater image enhancement has received much attention in underwater vision research. However, raw underwater images easily suffer from color distortion, underexposure, and fuzz caused by the underwater scene. To address the above-mentioned problems, we propose a new multiscale dense generative adversarial network (GAN) for enhancing underwater images. The residual multiscale dense block is presented in the generator, where the multiscale, dense concatenation, and residual learning can boost the performance, render more details, and utilize previous features, respectively. And the discriminator employs computationally light spectral normalization to stabilize the training of the discriminator. Meanwhile, nonsaturating GAN loss function combining $L_1$ loss and gradient loss is presented to focus on image features of ground truth. Final enhanced results on synthetic and real underwater images demonstrate the superiority of the proposed method, which outperforms nondeep and deep learning methods in both qualitative and quantitative evaluations. Furthermore, we perform an ablation study to show the contributions of each component and carry out application tests to further demonstrate the effectiveness of the proposed method. <s> BIB011 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Network Specifics <s> Abstract Underwater image restoration, which is the keystone to the underwater vision research, is still a challenging work. The key point of underwater image restoration work is how to remove the turbidity and the color distortion caused by the underwater environment. In this paper, we propose an underwater image restoration method based on transferring an underwater style image into a recovered style using Multi-Scale Cycle Generative Adversarial Network (MCycle GAN) System. We include a Structural Similarity Index Measure loss (SSIM loss), which can provide more flexibility to model the detail structural to improve the image restoration performance. We use dark channel prior (DCP) algorithm to get the transmission map of an image and design an adaptive SSIM loss to improve underwater image quality. We input this information into the network for multi-scale calculation on the images, which achieves the combination of DCP algorithm and Cycle-Consistent Adversarial Networks (CycleGAN). By compared the quantitative and qualitative with existing state-of-the-art approaches, our method shows a pleasing performance on the underwater image dataset. <s> BIB012
After reviewing current deep learning-based underwater image enhancement algorithms, we emphasize the different aspects of the above-mentioned deep models. First, we summarize the network specifics of different models in Table 1 and then further analyze network loss, depth, parameters, and input patch size. Network Loss Network loss plays an integral part in learning the task underhand. Here, we discuss the losses employed in deep underwater image enhancement. The most popular type of loss functions are to minimize the per-pixel error between the ground-truth image and the predicted image, commonly known as 1 and 2 . For example, the UIE-Net BIB002 , UIR-Net BIB005 , P2P Net BIB010 , and URCNN BIB006 only use 2 to optimize their networks. Usually, other losses such as SSIM, gradient etc., are combined with the ones mentioned earlier to improve the performance of the networks, e.g. UWCNN BIB007 . On the other hand, GANs rely on adversarial loss and perceptual loss to enhance the perceptual quality of the enhanced images, such as DenseGAN BIB011 , UWGAN BIB008 , etc. Network Depth and Paramters The network depth and the number of parameters are related. The deeper the network, the more the number of parameters. Unlike other image classification BIB001 and enhancement tasks  where the network depth has exponentially increased and even consists of hundreds of convolutional layers, the underwater image enhancement networks are still very shallow composed of less than 45 layers (deepest network is the WaterGAN BIB003 with 42 layers); hence comprised of very less number of parameters 2 . Input Patch Size Contrary to low-level vision tasks, most of the underwater image enhancement algorithms operate on full-size images. The reason may be to incorporate the wavelength dissipation of red, green, and blue channels. Furthermore, some algorithms reduce the image to predefined size, which requires upsampling Haze-line BIB009 ULFID BIB004 UIEBD  Fig . 3 Representative images: Three sample images from Haze-line BIB009 , ULFID BIB004 , and UIEBD  datasets to show the diversity of the underwater images. as a post-processing step, such as MCycleGAN BIB012 , DenseGAN BIB011 , and UWGAN BIB008 .