Novel Methods for Weather Distortions Mitigation in Images and Videos
Images and videos captured under adverse weather condition usually suffer from bad visual quality. The reduced visual quality can diminish the the human operator comprehension of the image or video content or even it can make the higher level computer vision applications such as object segmentation or detection, very challenging. This thesis focuses on recovering the images or videos affected by different adverse weather conditions.Airborne videos are extensively used for object detection and target tracking. However, under bad weather conditions, the presence of clouds and haze or even smoke coming from buildings can make the processing of these videos very challenging. Current cloud detection or classification methods only consider a single image. Moreover, the images they use are often captured by satellites or planes at high altitudes with very long ranges to clouds, which can help distinguish cloudy regions from non cloudy ones. We propose a new approach for cloud and haze detection by exploiting both spatial and temporal information in airborne videos. In this method, several consecutive frames are divided into patches. Then, consecutive patches are collected as patch sets and fed into a deep convolutional neural network. The network is trained to learn the appearance of clouds as well as their motion characteristics. Therefore, instead of relying on single frame patches, the decision on a patch in the current frame is made based on patches from previous and subsequent consecutive frames. This approach, avoids discarding the temporal information about clouds in videos, which v may contain important cues for discriminating between cloudy and non cloudy regions. Experimental results show that using temporal information besides the spatial characteristics of haze and clouds can greatly increase detection accuracy. The second problem that we address is the removal of haze problem in aerial images or airborne videos. Existing single image dehazing methods that work on ground to ground images, do not perform well on aerial images. Moreover, current dehazing methods are not capable for real time processing. We propose a new end to end aerial image dehazing method using a deep convolutional autoencoder. Using the convolutional autoencoder, the dehazing problem is divided into two parts, namely, encoder, which aims extract important features to dehaze hazy regions and decoder, which aims to reconstruct the dehazed image using the down sampled image received from the encoder. In this proposed method, we also exploit the superpixels in two different scales to generate synthetic thin cloud data to train our network. Since this network is trained in an end to end manner, in the test phase, for each input hazy aerial image, the proposed algorithm outputs a dehazed version without requiring any other information such as transmission map or atmospheric light value. The idea used for generating the synthetic hazy images is further extended for generation of the synthetic hazy frame sequences for airborne video dehazing. The thin clouds in these videos can be distinguished from the other objects on the ground by their two main characteristics: color and motion. Here, we propose a new method for airborne video dehazing by incorporating both the color and motion features of the thin clouds. Using the generated synthetic data, a new end to end convolutional autoencoder is trained for dehazing the hazy frames of airborne videos. This network is trained in a way to extract both the temporal and spatial information for the dehazing task. With the proposed dehazing methods, hazy regions are dehazed and objects within hazy regions become more visible while the contrast vi of non hazy regions is increased. The third challenging weather condition that we address is the atmospheric turbulence problem caused by heat. Atmospheric turbulence is one of the causes of quality degradation in long range imaging and the removal of its bad effects from degraded frame sequences is considered an ill posed problem. Although there have been numerous attempts to address this problem, the quality of the restored scenes is not promising. In contrast to the previous approaches to address this problem, in this thesis, we propose a data driven approach. First, an end to end deep convolutional autoencoder is trained using natural images and its encoder part is exploited to extract high level features from all the frames in a sequence that are distorted by atmospheric turbulence. Then, the Binary search algorithm and the unsupervised k means clustering method are jointly exploited to analyze these highlevel features to find the best set of frames that can help accurately reconstruct the original high quality image. After removing the geometric distortion from the selected frames, the saliency map of the average set of the selected frames is calculated and used with the original selected frames to train an end to end multi scale saliency guided deep convolutional autoencoder network to fuse the registered frames. This networks uses different scales of the input frames and their saliency map for better fusion performance. Specifically, the fusion network learns how to fuse these sets of frames and also exploit information from their saliency map to generate an image with more details of the scene. Finally, this fused image is post processed to boost the visual quality of the output fused image. The experimental results are conducted on both synthetically and naturally distorted sequences to show the performance of the proposed method compared to the other state of the art methods. The fourth and the last problem that is addressed in this thesis is the removal of rain streaks and snow particles from single images. Rainy or snowy weather conditions can vii severely impair the visual quality of single images. Moreover, due to lack of temporal information in single images, removal of these artifacts becomes more challenging. We address both the deraining and desnowing problem in a single framework using a data driven approach. In this method, the spatial characteristics of rain streaks and snow particles are well investigated and two maps, namely, direction map and intensity map are generated and exploited in the removal process. Using these two maps the type of the distortion is classified using a convolutional neural network (CNN) and this information is forwarded to the removal step. In the removal step, The input image with the two extracted maps and the information about the distortion type are used to train a deep fully convolutional rain/snow removal network (RSRNet) This network is trained in a way that separates the important background scene edges from the rain streaks or snow particles and use the extracted edge map to augment the quality of the output clear image. Moreover, single images usually suffer from atmospheric haze made in the case of heavy rain/snow. Therefore, a simple dehazing method based on the dark channel prior (DCP) algorithm is proposed which uses the edge map extracted in the RSRNet to build a transmission map for the haze removal task. The experimental results on both the real and synthetic single rainy/snowy images demonstrate the superiority of our method compared to the other rain/snow removal methods.