Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA and MS COCO-VQA and show that it produces the best reported results in both cases. <s> BIB001 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> This letter reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high-resolution images to map the seafloor; however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects, such as absorption and scattering, which cause attenuation of light at different rates for different wavelengths. While this physical process is well described theoretically, the model depends on many parameters intrinsic to the water column as well as the structure of the scene. These factors make recovery of these parameters difficult without simplifying assumptions or field calibration; hence, restoration of underwater images is a nontrivial problem. Deep learning has demonstrated great success in modeling complex nonlinear systems but requires a large amount of training data, which is difficult to compile in deep sea environments. Using WaterGAN, we generate a large training dataset of corresponding depth, in-air color images, and realistic underwater images. These data serve as input to a two-stage network for color correction of monocular underwater images. Our proposed pipeline is validated with testing on real data collected from both a pure water test tank and from underwater surveys collected in the field. Source code, sample datasets, and pretrained models are made publicly available. <s> BIB002 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> Underwater images suffer from color distortion and low contrast, because light is attenuated while it propagates through water. Attenuation under water varies with wavelength, unlike terrestrial images where attenuation is assumed to be spectrally uniform. The attenuation depends both on the water body and the 3D structure of the scene, making color restoration difficult. Unlike existing single underwater image enhancement techniques, our method takes into account multiple spectral profiles of different water types. By estimating just two additional global parameters: the attenuation ratios of the blue-red and blue-green color channels, the problem is reduced to single image dehazing, where all color channels have the same attenuation coefficients. Since the water type is unknown, we evaluate different parameters out of an existing library of water types. Each type leads to a different restored image and the best result is automatically chosen based on color distribution. We also contribute a dataset of 57 images taken in different locations. To obtain ground truth, we placed multiple color charts in the scenes and calculated its 3D structure using stereo imaging. This dataset enables a rigorous quantitative evaluation of restoration algorithms on natural images for the first time. <s> BIB003 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> In an underwater scene, wavelength-dependent light absorption and scattering degrade the visibility of images, causing low contrast and distorted color casts. To address this problem, we propose a convolutional neural network based image enhancement model, i.e., UWCNN, which is trained efficiently using a synthetic underwater image database. Unlike the existing works that require the parameters of underwater imaging model estimation or impose inflexible frameworks applicable only for specific scenes, our model directly reconstructs the clear latent underwater image by leveraging on an automatic end-to-end and data-driven training mechanism. Compliant with underwater imaging models and optical properties of underwater scenes, we first synthesize ten different marine image databases. Then, we separately train multiple UWCNN models for each underwater image formation type. Experimental results on real-world and synthetic underwater images demonstrate that the presented method generalizes well on different underwater scenes and outperforms the existing methods both qualitatively and quantitatively. Besides, we conduct an ablation study to demonstrate the effect of each component in our network. <s> BIB004 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> Underwater vision suffers from severe effects due to selective attenuation and scattering when light propagates through water. Such degradation not only affects the quality of underwater images but limits the ability of vision tasks. Different from existing image enhancement and restoration methods which either ignore the wavelength dependency of the attenuation or assume a specific spectral profile, we tackle color distortion problem of underwater image from a new view. In this letter, we propose a weakly supervised color transfer method to correct color distortion. Inspired by Cycle-Consistent Adversarial Networks, we design a multi-term loss function including adversarial loss, cycle consistency loss, and SSIM (Structural Similarity Index Measure) loss. In this way, we translate the color of underwater image as if it is taken in the air, and preserve the content and structure of original underwater image. Experiments on underwater images captured under diverse scenes show that the proposed method can produce visually pleasing results, even outperforms the art-of-the-state methods. Besides, our method can improve the performance of vision tasks. <s> BIB005 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> Abstract Underwater image restoration, which is the keystone to the underwater vision research, is still a challenging work. The key point of underwater image restoration work is how to remove the turbidity and the color distortion caused by the underwater environment. In this paper, we propose an underwater image restoration method based on transferring an underwater style image into a recovered style using Multi-Scale Cycle Generative Adversarial Network (MCycle GAN) System. We include a Structural Similarity Index Measure loss (SSIM loss), which can provide more flexibility to model the detail structural to improve the image restoration performance. We use dark channel prior (DCP) algorithm to get the transmission map of an image and design an adaptive SSIM loss to improve underwater image quality. We input this information into the network for multi-scale calculation on the images, which achieves the combination of DCP algorithm and Cycle-Consistent Adversarial Networks (CycleGAN). By compared the quantitative and qualitative with existing state-of-the-art approaches, our method shows a pleasing performance on the underwater image dataset. <s> BIB006 </s> Diving Deeper into Underwater Image Enhancement: A Survey <s> Future and Emerging Directions <s> Underwater image enhancement has received much attention in underwater vision research. However, raw underwater images easily suffer from color distortion, underexposure, and fuzz caused by the underwater scene. To address the above-mentioned problems, we propose a new multiscale dense generative adversarial network (GAN) for enhancing underwater images. The residual multiscale dense block is presented in the generator, where the multiscale, dense concatenation, and residual learning can boost the performance, render more details, and utilize previous features, respectively. And the discriminator employs computationally light spectral normalization to stabilize the training of the discriminator. Meanwhile, nonsaturating GAN loss function combining $L_1$ loss and gradient loss is presented to focus on image features of ground truth. Final enhanced results on synthetic and real underwater images demonstrate the superiority of the proposed method, which outperforms nondeep and deep learning methods in both qualitative and quantitative evaluations. Furthermore, we perform an ablation study to show the contributions of each component and carry out application tests to further demonstrate the effectiveness of the proposed method. <s> BIB007
Underwater image enhancement is a classical research area and has improved a lot in recent years, mainly due to the rapid development of deep learning techniques. The performance is still lacking in many aspects when compared to other image enhancement techniques like image super-resolution, deblurring, and dehazing. There is ample room to advancement the underwater image enhancement direction. Here, in the following paragraphs, we present the list of some of the potential future directions. Fig. 7 Visual comparisons on Haze-line BIB003 : The Haze-line dataset provides an accurate distance based on the stereo. To be fair to the authors of Haze-line BIB003 , we have also included the results of the best performer (i.e., Haze-line BIB003 , a conventional method) on this dataset. • Datasets: Underwater image enhancement methods usually employ synthetic images for training due to lack of representative real-world underwater images and its corresponding ground-truth images. Although there are limited datasets available, which have underwater and their reference images; however, these datasets consist of a finite number of images and are typically used as test images rather than training the models. A true effort in this direction may improve the performance of underwater image enhancement models and also provide realistic feedback on the image quality of enhanced results by different methods. • Objective functions and evaluation metrics: Current algorithms predominantly employ objective functions common to image enhancement techniques. Although these functions produce some favorable results; however, none of them incorporate the underwater physical model properties. Likewise, the available evaluation metrics to underwater images are limited and have failure cases, which keeps the field of underwater image enhancement at a standstill. For example, the visual results shown in Figures 4-8 do not match the quantitative results in Table 2 . Therefore, more specialized objective functions and evaluation metrics are required to advance the underwater image enhancement research. • Prior knowledge: The human perception of the scene depends on the extensive domain or prior knowledge. When experts describe the image quality, they don't solely rely on the content of the visuals; instead, they also use their domain knowledge. An exciting venue to explore is to augment the current techniques with prior or domain knowledge BIB001 . * , Chongyi Li * In-air UWCNN (type-I) BIB004 DUIENet  Underwater MCycleGAN BIB006 UWGAN BIB005 DenseGAN BIB007 In-air UWCNN (type-I) BIB004 DUIENet  Underwater MCycleGAN BIB006 UWGAN BIB005 DenseGAN BIB007 This has shown an increase in the performance in areas like visual question answering and would likely help to improve underwater image enhancement. • Unsupervised learning: Due to the lack of dataset, which has underwater images and their ground-truth images, many methods generate synthetic data to train their models. Although these models exhibit promising results for synthetic underwater scenes; however, they fail on real-world underwater images. To deal with the lack of data, a possible research direction could be unsupervised learning, also known as zero-shot or few-shot learning. This capability may lead to promising results, but the zero-shot problem itself is not trivial. A more realistic scenario would be to employ the present limited datasets, few-shot learning, where the network learns from a few available images. The development of unsupervised learning is an open research problem. • Real vs. Synthetic: Existing algorithms use diverse physical (mathematical) models to generate underwater images. The distribution of the generated underwater scenes may not be conferred to the real-world scenes; therefore, the models trained on artificially produced datasets lack generalization capability. A more thorough and exhaustive effort is required to generate artificial datasets, and one solution may be to use GAN-based networks to transfer style from underwater images to the simulated scenes. Even though minimal work BIB002 has been done in this direction, still there is a lot of scope of improvement.