TECHNICAL FIELD

The invention belongs to the field of image processing and computer vision, and specifically relates to a monocular underwater image depth estimation and color correction method based on deep neural network.

BACKGROUND

Depth estimation and color correction of underwater images are important basics for underwater operations, e.g., underwater monitoring, robot operation, underwater visual navigation, and so on. However, underwater depth estimation and color correction have been challenged by poor visibility and geometric distortion in underwater environment, making them more difficult to handle compared with the land environment. Particles and bubbles in seawater cause visible light to scatter and weaken in the process of propagation, resulting in the color deviation of the collected images. For depth estimation, stereo matching technology and other professional depth sensing devices are mainly used to obtain depth information. For inferring the depth information, stereo matching technique uses two corresponding images collected by binocular camera for correlation matching and triangulation measurement. Other depth acquisition methods mostly use depth sensing devices, such as Time-of-Flight (TOF) cameras, Microsoft Kinect cameras, and so on. Although these methods have achieved good results in the land environment, the results are unsatisfactory in the underwater environment due to imaging limitations and optical distortion. With the development of deep learning, the use of deep convolution neural network can directly obtain the corresponding depth information from single color image on land, which solves the limitation of the imaging related in traditional methods, but there are still problems: compared to the land, datasets with valid labels of underwater images are very scarce, resulting in the unavailable of depth maps and color-corrected images from practical underwater scene, and yet most existing methods need effective supervision information, e.g., depth map, for training, and then build a more accurate depth estimation network.

Based on the above problems, the invention designs a monocular underwater image depth estimation and color correction framework based on deep neural network, which simultaneously completes two underwater related tasks. The framework consists of two parts: style transfer subnetwork and task subnetwork. The style transfer subnetwork is constructed based on Generative Adversarial Networks (I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets”, in NIPS, 2014, PP. 2672-2680). It is used to transfer the apparent information of underwater images to land images and obtain rich and effective synthetic labeled data. The task subnetwork combines depth estimation and color correction of underwater images to improve their respective accuracies through collaborative learning.

SUMMARY

The present invention aims to overcome the shortcomings of existing technologies and provides a high-quality depth estimation and color correction method based on image style transfer subnetwork and task subnetwork, then designs a framework for high quality depth estimation and color correction based on deep neural network. The framework comprises two parts: style transfer subnetwork and task subnetwork. The style transfer subnetwork is constructed based on generative adversarial network, which is used to transfer the apparent information of underwater images to land images and obtain abundant and effective synthetic labeled data, while the task subnetwork combines the underwater depth estimation and color correction tasks with the stack network structure, carries out collaborative learning to improve their respective accuracies, and reduces the gap between the synthetic underwater image and the real underwater image through the domain adaptation strategy, so as to improve the network's ability to process the real underwater image.

The specific technical solution of the invention is, a method for high-quality depth estimation and color correction based on style transfer network and task network, the method comprises the following steps:

(1) Preparing initial data: The initial data is the land labeled dataset, including the land color map and the corresponding depth map for training; In addition, a small number of real underwater color images are collected to assist the training and testing;

(2) The construction of the style transfer subnetwork;

(2-1) The style transfer subnetwork is constructed based on generative adversarial network model, in which the generator uses the U-Net structure (O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for Biomedical Image segmentation”, in MICCAI, 2015, PP. 234-241.), which is composed of an encoder and a decoder.

(2-2) The discriminator consists of three parts. The first part is a module composed of Cony and Leaky Rectified Linear unit (Leaky ReLU). The second part is three modules composed of Cony, BN and Leaky ReLU. The third part is a sigmoid function layer that is used to output the test results.

(2-3) The style loss function and the content loss function are used to preserve the content and transform the style, and the total loss function of the whole style transfer subnetwork is constructed.

(3) The construction of the task subnetwork;

(3-1) Depth estimation and color correction are separately realized by using two generative adversarial networks, in which the structure of generator and discriminator is the same as that of generator and discriminator in style transfer subnetwork. On this basis, the depth estimation generator and color correction generator are connected in series to form a stacked network structure.

(3-2) Two discriminators are used to realize the domain adaptation between the synthetic underwater image and the real underwater image, and to enhance the network's ability to process the real underwater image, so as to solve the domain adaptation problem at the feature level.

(3-3) Construct the total loss function of the entire task subnetwork.

(4) Training the whole network composed by (2) and (3).

(4-1) First of all, the land labeled data and underwater real data are used to train the style transfer subnetwork, and then a convergent training model is obtained, so as to obtain effective synthetic underwater labeled data.

(4-2) Then, the synthetic underwater labeled dataset obtained by style transfer subnetwork is used to train the task subnetwork. Real underwater images are simultaneously added to train together, so as to reduce the difference between real underwater domain and synthetic underwater domain and improve the network's ability to process real underwater images.

(4-3) The two networks are connected in series according to the order of style transfer subnetwork and task subnetwork, and the total loss function is used for unified training and fine-tuning the whole network framework. When the training is completed, the trained model can be used for testing on the test set to obtain the output result of the corresponding input image.

The present invention has the following beneficial effects:

The present invention is based on a deep neural network, which firstly builds a style transfer subnetwork based on generative adversarial network to obtain effective synthetic labeled data, and then builds a task subnetwork to realize depth estimation and color correction. It has the following characteristics:

1. The system is easy to build, and the deep neural network can be used to get the corresponding high-quality depth map and color-corrected underwater image from the single underwater color image in an end-to-end fashion.

2. The algorithm is simple and easy to be implemented.

3. This method makes the network have sufficient data to learn depth estimation and color correction through transferring the apparent information of underwater images to land images.

4. It adopts the feature domain adaptation method, which can effectively reduce the gap between the two domains of land images and underwater images.

