Unpaired Quad-Path Cycle Consistent Adversarial Networks for Single Image Defogging

Abstract: Adversarial learning-based image defogging methods have been extensively studied in computer vision due to their remarkable performance. However, most existing methods have limited defogging capabilities for real cases because they are trained on the paired clear and synthesized foggy images of the same scenes. In addition, they have limitations in preserving vivid color and rich textual details in defogging. To address these issues, we develop a novel generative adversarial network, called quad-path cycle consistent adversarial network (QPC-Net), for single image defogging. QPC-Net consists of a Fog2Fogfree block and a Fogfree2Fog block. In each block, there are three learning-based modules, namely, fog removal, color-texture recovery, and fog synthetic, which sequentially compose dual-path that constrain each other to generate high quality images. Specifically, the color-texture recovery model is designed to exploit the self-similarity of texture and structure information by learning the holistic channel-spatial feature correlations between the foggy image with its several derived images. Moreover, in the fog synthetic module, we utilize the atmospheric scattering model to guide it to improve the generative quality by focusing on an atmospheric light optimization with a novel sky segmentation network. Extensive experiments on both synthetic and real-world datasets show that QPC-Net outperforms state-of-the-art defogging methods in terms of quantitative accuracy and subjective visual quality.

\section{Introduction}
\label{introduction}
\IEEEPARstart{I}{n} outdoor natural scenes, the captured digital images often experience quality degradation due to bad weather conditions, such as fog, smoke and haze. The resulting foggy images suffer from low contrast, distorted colors, and severe texture information loss, which adversely impact most existing computer vision applications such as image alignment \cite{zhou2020alignment}, object tracking \cite{shen2019visual}, and remote sensing \cite{han2014object}. It is thus necessary to design effective single image defogging algorithms to restore the content, color, and texture details from the foggy images. Existing image defogging methods can be roughly divided into three classes: prior-based methods \cite{He2011PAMI}, \cite{hazingline2016CVPR}, \cite{zhu2015TIPCAP}, \cite{IDE2021ide}, \cite{huedisparity2010ACCV}, fusion-based methods \cite{fusiondehazing2013TIP}, \cite{galdran2018image}, \cite{gao2020fusion}, and learning-based methods \cite{dehazenet2016TIP}, \cite{GFN2018CVPR}, \cite{cycle-dehazing}, \cite{zhu2019tcyb}, \cite{zhang2021tcyb}, \cite{zhang2021dual}. Most prior-based methods remove the fog from an image by using or improving an atmospheric scattering model (ASM) \cite{ASM}. Although these methods work well for some scenes, they often fail to handle other scenes that do not meet the predetermined prior assumptions. For instance, a dark-channel prior does not work well for restoring the sky or white building since there is no dark channel in these areas. Traditional fusion-based defogging methods usually need to combine several derived inputs from the original foggy image to fuse a fine-weight map to remove the fog. The limitation of these methods is that the derived inputs fail to reflect the inherent correlation between the depth information and the foggy image, leading to poor defogging performance when the fog density is high.

In recent years, many learning-based defogging methods have been proposed to address the disadvantages of prior- and fusion-based methods by leveraging the powerful feature extraction ability and spatial mapping capacities of convolutional neural networks (CNN). These methods can be grouped into two categories: supervised and unsupervised defogging methods: for former requires pairs of fog and fog-free images in network training, whereas the latter does not. In addition, supervised defogging mainly includes ASM based defogging networks (ASMDN) \cite{dehazenet2016TIP} \cite{MSCNN2016ECCV} \cite{zhu2019tcyb} \cite{AODNET2017ICCV} and end-to-end defogging networks (EDN) \cite{GFN2018CVPR} \cite{wang2018aipnet}  \cite{pix2pixdehazing} \cite{zhang2021dual} \cite{zhang2021tcyb}. In ASM, transmission and atmospheric light are two most important parameters. The accuracy of their estimation greatly affects the quality of the recovered image, with estimation inaccuracy resulting in undesired artifacts such as color distortion, halo, and over enhancement. ASMDN exploits CNN to estimate these two parameters before using ASM for defogging. However, ASMDN still suffers if transmission approximations are inaccurate due to the colors of the objects in the scene are inherently similar to the atmospheric lights. Thus, EDN was proposed to directly remove the fog form foggy images by using CNN without ASM. EDN has gradually become the mainstream algorithm for defrogging. Although ASMDN and EDN have rectified some shortcomings of traditional methods (\textit{e.g.,} prior-based ones), training their networks require a large number of fog-fogfree image pairs, which are difficult to obtain in practice. Therefore, for a clear image, ASM is usually adopted to synthesize the corresponding foggy image. This synthesis method is simple and efficient, but the distribution of the generated fog is relatively uniform, whereas the atmosphere is heterogeneous. CNN based models trained with this synthesized dataset are not best suited for the foggy images in real-world scenes.  A suitable learning-based defogging method not only needs to have good performance, but also fine texture details and high fidelity.

Unsupervised defogging methods \cite{yang2018towards} \cite{cycle-dehazing} \cite{liu2020end} \cite{zhao2021refinednet} have been proposed to address the above issues because they are trained with unpaired real-world foggy and clear images. However, the defogged images generated through these methods usually have low contrast and quality. In this work, we propose a novel unsupervised defogging method, called holistic attention-fusion adversarial network (HAAN), for improved performance. HAAN shares a similar structure with CycleGAN in that it also includes two transformation blocks that we call Fog2Fogfree and Fogfree2Fog blocks. In each block, there are three modules: a Fog Removal Module (FRM), a Fog Synthetic Module (FSM), and a Color-Texture Recovery Module (CTRM), as shown in Fig. \ref{architctures}, which constitute a dual mapping path. Inside the Fog2Fogfree block, a defogging path is constructed by FRM and FSM (FRM$\rightarrow$FSM) to map the image from the fog to fogfree domain, and a color-texture recovery path for defogged result is constructed by CTRM and FSM (CTRM$\rightarrow$FSM) to improve image details. Similarly, inside the Fogfree2Fog block, FSM and FRM (FSM\textbf{$\rightarrow$}FRM) constitutes a synthesizing path to map the image from the fogfree to fog domain, FRM and CTRM (FRM$\rightarrow$CTRM) forms a color-texture recovery path for synthesized result to further enhance image details. Furthermore, in the color-texture recovery path, we devise a Holistic Attention-Fusion generator, which uses the derived feature maps of the inputs (see Fig. \ref{architctures}). Different from conventional attention networks, HAAN makes good use of the correlation between channels and positions of each channel based on input images. Specifically, HAAN concatenates the input and its several derived features to exploit their correlations through a Channel Attention and Spatial Attention network to retrieve more contextual information (as shown in Fig. \ref{attention-structure}). This allows us to better capture the global information to constrain the defog network to generate result with more texture details. A large number of qualitative and quantitative comparisons against state-of-the-art defogging approaches have been performed to show that HAAN improves the defogged image quality and gives more image details.

The remainder of this paper is organized as follows. Section II gives a literature review; Section III covers HAAN; Section IV provides implementation details, experiment results, and qualitative and quantitative comparisons; and Section V concludes the paper.

\begin{figure*}[t]\scriptsize
	\vspace{-4.0mm}
	\begin{center}
		\begin{tabular}{@{}c@{}}
			\includegraphics[width = 0.8\textwidth,height=0.34\textheight]{img1/discuss/1.png}\\
		\end{tabular}
	\end{center}
	\vspace{-0.3cm}
	\caption{
		The overall architecture of the proposed HAAN.  It consists of two main components: 1) a Fog2Fogfree Block; 2) a Fogfree2Fog Block. Each block has a dual-path mapping mechanism. Inside the Fog2Fogfree block, the bottom sequential units constitute a defogging path to map foggy images to fogfree images, and the top sequential units constitute a color-texture recovery path for defogged result by fusion several derived inputs. Similarly, inside the Fogfree2Fog block, the top sequential units constitute a synthesizing path to map fogfree images to foggy images, and the bottom sequential units constitute a color-texture recovery path for synthesized results.
	}
	\vspace{-5.0mm}
	\label{architctures}
\end{figure*}

\section{Related Work}

In this section, we only review those learning-based works, including supervised and unsupervised defogging methods and attention mechanism, that are closely related to ours.  Before introducing them, it helps to start with the ASM to better understand the formation of foggy images to catch on the design ideas of related defogging methods.

\subsection{Atmospheric Scattering Model}

Most existing prior- and learning-based defogging algorithms are based on the ASM \cite{McCartney}\cite{NarasimhanandNayar1}\cite{NarasimhanandNayar2}. It can be formulated as:
\begin{equation}
	I(x) = J(x)T(x)+A[1-T(x)]
	\label{adm}
\end{equation}
where  \textit{x} denotes the position of a pixel, \textit{I(x)} is the observed foggy image, \textit{J(x)} represents the fog-free image, \textit{T(x)} is the transmission map and \textit{A} is the atmospheric light. Unfortunately, in this ill-conditioned equation, both \textit{T(x)} and \textit{A} are unknown in practice. Thus, if we want to recover \textit{J} from \textit{I}, we need to first determine \textit{A} and \textit{T}. Most state-of-the-art methods either assume prior knowledge of them or use CNN to estimate them before attempting to recover the fog-free image \textit{J(x)} according (\ref{adm}). Moreover, in practice, \textit{T(x)} represents the medium transmission map, which is related to the depth of the scene via
\begin{equation}
	t(x) = e^{-\beta d(x)}
	\label{depth}
\end{equation}
where $\beta$ stands for the atmosphere scattering parameter and \textit{d} is the depth of the scene.

\subsection{Supervised methods}
\label{SM}

As mentioned before, learning-based supervised defogging methods can be mainly divided into two groups: ASMDN and EDN.

\textbf{Defogging via ASMDN.} These methods focus on estimating the transmission map \textit{T(x)} and atmospheric light \textit{A} through CNN before recovering the foggy image by using ASM. Cai \textit{et al.} \cite{dehazenet2016TIP} proposed a deep CNN based method to extract the multi-features from foggy image to optimize \textit{T(x)} with a nonlinear activation function. Similarly, a multiscale network was designed by Ren \textit{et al.} \cite{MSCNN2016ECCV}  to estimate \textit{T(x)} and \textit{A}. In \cite{zhu2019tcyb}, Zhu \textit{et al.} proposed a DehazeGAN, which combines a dense coarse-scale network with a fine-scale network to extract multiscale features from the foggy image, and they designed estimators for \textit{T(x)} and \textit{A} from these features. Different from most estimation methods, Li \textit{et al.} \cite{AODNET2017ICCV} reformulated (\ref{adm}) to combine \textit{T(x)} and \textit{A} as an integrated variable before proposing a light-weight CNN to generate the clean image with this new formulation. Compared with the traditional prior-based methods, ASMDN obtains better estimates of \textit{T(x)} and \textit{A}. However, artifacts will be present in defogged images when these two parameters are estimated incorrectly.

\textbf{Defogging via EDN.} These methods directly remove the fog from foggy images to generate fog-free images through CNNs without using ASM. Based on the derived features of an input foggy image, Ren \textit{et al.} \cite{GFN2018CVPR} employed an end-to-end trainable gated fusion network for single image defogging that can effectively learn the inherent correlation between fog-related features and the transmission maps. Wang \textit{et al.} \cite{wang2018aipnet} proposed an atmospheric illumination prior in which the luminance channel in the YCrCb color space is mainly disturbed by the atmospheric light in foggy weather. Based on this prior, they adopt an end-to-end multiscale trainable model to restore the Y channel of a foggy image. In \cite{zhang2021dual}, the authors presented a Dual-Path Recurrent network, which has two parallel branches to recover the image by simultaneously learning the characteristics of the basic content and details of foggy images. Zhang \textit{et al.} \cite{zhang2021tcyb} proposed a densely connected pyramid encoder to extract multiscale image features from foggy inputs before using a density-aware generator to obtain fog-free images; they also constructed a Laplacian pyramid decoder to refine the defogged images.  Different from the above methods, Qu \textit{et al.} \cite{pix2pixdehazing} treated image defogging as an image-to-image translation problem and proposed an enhanced pix2pix defogging network. This method constructs a multi-resolution generative adversarial network and an enhancer to translate a foggy image to a clear one.

\subsection{Unsupervised methods}

Although supervised learning-based defogging methods have achieved remarkable results, they require fog-fogfree image pair to train the network. In practice, it is difficult to synthesize fog around distant objects because the depth information is hard to estimate. Moreover, it is a time-consuming and labor-intensive task to synthesize a large number of foggy images with different fog concentrations. Therefore, unsupervised learning-based defogging methods are better suited for real-world defogging tasks. These methods not only learn the domain correlation between the foggy and clear images in the natural scenes, but also effectively reduce the cost of data preparation. Training on unpaired data, Yang \textit{et al.} \cite{yang2018towards} first proposed a disentangled dehazing network consists of three generators and a multi-scale discriminator to produce defogged results from the foggy images. By introducing a perceptual loss function, Engin \textit{et al.} \cite{cycle-dehazing} employed an enhanced CycleGAN \cite{zhu2017unpaired} to directly generate defogging results. Inspired by \cite{zhu2017unpaired}, we proposed in one of our previous works \cite{liu2020end} a network with a two-stage transformation path based on Cyclegan to map foggy images to the fog-free domain; the network in \cite{liu2020end} consists of two generators, two enhancers, and a discriminator. In \cite{zhao2021refinednet}, Zhao proposed a two-stage RefineDNet to combine the merits of prior- and learning-based approaches to achieve both visibility restoration and realness improvement. The methods above have a common characteristic: they all use an adversarial generator (AG) to recover foggy images. We thus refer to them as AG-based methods, which typically tend to generate low-quality defogged images with loss of texture detail and low contrast.

\subsection{Attention Mechanism}

Attention mechanism has become a high-profile technology in the field of computer vision since it can guide deep neural networks to focus on areas with most valuable information to advance the discriminative learning ability \cite{hu2018squeeze} \cite{jiang2020decomposition}. It has been successfully applied to the low-level tasks such as image super-resolution, deraining, and defogging. Niu \textit{et al.} \cite{niu2020single} proposed a holistic image super-resolution attention network consists of a layer attention module and a channel-spatial attention module to learn the correlation information among layers, channels, and positions. In \cite{jiang2020decomposition}, Jiang \textit{et al.} presented an improved attention-guided network to remove rain streaks from the input image by first decomposing rain streaks into multiple rain layers and then fusing them through a mixed attention block. To reduce artifacts such as over- and under-defogging introduced by retinex based networks, Li \textit{et al.} \cite{li2020deep} proposed a channel and spatial attention mechanism that can automatically adjust the channel- and pixel-wise attention weights. Liu \cite{liu2019griddehazenet} proposed GridDehazeNet to address the bottleneck effect of information flow in the hierarchical architecture while allowing efficient information exchange across different scales. Although the above methods can leverage the feature extraction ability of neural networks, they all train on paired image. There exist few studies on application of attention mechanisms to unpaired training due to diversity in training samples in unsupervised learning. However, attention mechanisms can be introduced into unsupervised learning if their strong feature extraction capability is able to effectively compensate for sample diversity in training.
\section{Proposed Model}

\subsection{Overview}
\label{overview}
As illustrated in Fig. \ref{architctures}, our proposed HAAN consists of a Fog2Fogfree block and a Fogfree2Fog block. In these two blocks,  there are three modules, including Fog Removal Module (FRM, $M_R$), Fog Synthetic Module (FSM, $M_S$) and Color-Texture Recovery Module (CTRM, $M_{CTR}$). As can be seen from Fig. \ref{architctures}, different modules combine to form four different mapping path, which are defogging path ( composed of $M_R$ and $M_S$), synthesizing path (composed of $M_S$ and $M_R$), color-texture recovery path for defogged result (composed of $M_{CTR}$ and $M_S$ ) and color-texture recovery path for synthesized result (composed of $M_R$ and $M_{CTR}$). Therefore, our method forms a game learning process of confrontation by these four paths to achieve the mapping transformation from the foggy images to the fog-free images domain. Moreover, the functions of $M_R$, $M_S$ and $M_{CTR}$ are mainly completed by our proposed Defogging generator, Synthesizing generator and Holistic Attention-Fusion generator respectively.

\textit{For Fog2Fogfree block}, its dual-path consists of a defogging path and a color-texture recovery path for defogged result. Given an input real-world foggy image $I_{rf}$, we first obtain the defogged image $I_{df}^{rf}$ by
\begin{equation}
	I_{df}^{rf}=M_{R}(I_{rf})
\end{equation}
Then, $I_{df}^{rf}$ is fed to fog synthetic module to reconstruct the first foggy image $I_{rcf}^{1}$,
\begin{equation}
	I_{rcf}^{1}=M_S(M_{R}(I_{rf}))=M_S(	I_{df}^{rf})
\end{equation}
The above process we call it defogging path. To further improve the performance of $M_R$, the color-texture recovery path for defogged result is proposed. This path first takes the defogged image $I_{df}^{rf}$ and several derived images of the $I_{rf}$ as the input, and generated a recovered result $I_{rr}^{rf}$,
\begin{equation}
	I_{rr}^{rf}=M_{CTR}(I_{df}^{rf},I_{d}^{rf})
\end{equation}
where $I_{d}^{rf}$ denotes the derived images of $I_{rf}$. In this paper, we select three derived images for $I_{rf}$. Then, we put this result again into fog synthetic module to reconstruct the second foggy image $I_{rcf}^{2}$ by
\begin{equation}
	I_{rcf}^{2}=M_S(M_{CTR}(I_{df}^{rf},I_{d}^{rf}))=M_S(I_{rr}^{rf})
\end{equation}
Finally, we can obtain two cycle consistency loss functions for foggy images among $I_{rf}$, $I_{rcf}^{1}$, and $I_{rcf}^{2}$.  Note that, the defogged result obtained by fog removal module is our goal. The color-texture recovery path for defogged result is to constrain the fog removal module to generate results with better color fidelity and texture details.

\textit{For Fogfree2Fog block}, there still has a dual-path, which consists of a synthesizing path and a color-texture recovery path for synthesized result. Given an input real-world fogfree image $I_{rff}$, the synthesizing path is first fed it to fog synthetic module to generate a synthesized result $I_{sf}$, and then use $I_{sf}$ as the input of the fog removal module to generate the reconstruct fogfree image $I_{rcff}$,
\begin{equation}
	\left\{
	\begin{aligned}
		I_{sf} =  M_S(I_{rff}) \\
	    I_{rcff} = M_R(I_{sf})
	\end{aligned}
	\right.
\end{equation}
For color-texture recovery path for synthesized result, we first use the fog removal module to remove the fog from $I_{sf}$ to generate the defogged result $I_{df}^{sf}$. Then we recovery this result by using color-texture recovery module.
\begin{equation}
	\left\{
	\begin{aligned}
		I_{df}^{sf} = & M_R(I_{sf}) \\
		I_{rr}^{sf} = & M_{CTR}(I_{df}^{sf},I_{d}^{sf})
	\end{aligned}
	\right.
\end{equation}
where $I_{rr}^{sf}$ is the recovered image for synthesized result $I_{sf}$, $I_{d}^{sf}$ denotes the derived images of $I_{sf}$. Similarly, two cycle consistency loss functions for fogfree images are constructed among $I_{rff}$, $I_{rcff}$, and $I_{rr}^{sf}$.

\subsection{Network Architecture}
\label{Network Architecture}

 As discussed in \ref{overview}, each module has a major generator to perform its corresponding functions (as shown in Fig. \ref{architctures}), which are Defogging generator for $M_R$, Synthesizing generator for $M_S$, and Holistic Attention-Fusion generator for $M_{CTR}$. Moreover, another major component of our HAAN is the discriminator.
\begin{figure}[H]\scriptsize
	\begin{center}
		\vspace{-0.2cm}
		\begin{tabular}{@{}c@{}}
			\includegraphics[width = 0.49\textwidth]{img1/discuss/2}\\
		\end{tabular}
	\end{center}
	\vspace{-0.3cm}
	\caption{The architecture of the defogging generator.}
	\vspace{-2.5mm}
	\label{defog-structure}
\end{figure}
 \textbf{\textit{Defogging generator:}} As can be seen from Fig. \ref{defog-structure}, we show the architecture details of the defogging generator. Inspired by \cite{dong2020fd}, we design a densely-residual connected encoder-decoder for remove the fog from the foggy image. Compared with the structure of densely connect alone, introducing the residually connect block in our architecture, the generator can not only better learn the correlation information from shallow layers to deep layers, but also can better preserve the image content and texture details for defogged results. As shown in Fig. \ref{defog-structure}, the encoder contains two convolution blocks(ConvBlocks) and two dense blocks (DenseBlocks), which including a series of convolutional, concatenation, batch normalization(BN), pooling layers and ReLU layers. After the encoder, six residual blocks (ResBlocks) are connected. For encoder, given a foggy image $I_{f}$, we first extract the shallow feature $\mathcal{F}_{C}^{1}\epsilon\mathbb{R}^{64\times H\times W}$ by the first convolution block:
\begin{equation}
		\left\{
		\begin{aligned}
			\mathcal{F}_{C}^{1} &= \mathcal{C}_{7\times 7}(I_{f}),\\
			\mathcal{C}_{k\times k}(.) &= \delta (\eta (C_{k\times k}(.)))
		\end{aligned}
	\right.
\end{equation}
where $\mathcal{C}_{k\times k}(.)$ denotes the ConvBlock operation, including convolution operation $C_{k\times k}$ where kernel size is $k\times k$, batch normalization function $\eta$, and rectified linear unit (ReLU) activation function $\delta$. Then, we fed $\mathcal{F}_{C}^{1}$ to the first DenseBlock to distill the first encoding feature $\mathcal{F}_{en}^{1}\epsilon\mathbb{R}^{128\times {H/2}\times {W/2}}$,
\begin{equation}
	\setlength\abovedisplayskip{6pt}
	\setlength\belowdisplayskip{6pt}
	\begin{split}
		\begin{matrix}
			\mathcal{F}_{en}^{1}=[\mathcal{P}(\mathcal{F}_{C}^{1}),\mathcal{P}(C_{1\times 1}(\eta(\mathcal{D}_{3\times 3}(\mathcal{F}_{C}^{1}))))]
		\end{matrix}
	\end{split}
\end{equation}
where $\mathcal{D}_{k\times k}(.)$ is the DenseBlock operation, which has three ConvBlocks (as shown in Fig. \ref{defog-structure}). $[.]$ represents concatenation operation, $\mathcal{P}$ is the pooling operation. Finally, $\mathcal{F}_{en}^{1}$ is fed to the second ConvBlock and DenseBlock to obtain the final encoding feature $\mathcal{F}_{en}^{2}\epsilon\mathbb{R}^{256\times {H/4}\times {W/4}}$,
\begin{equation}
	\setlength\abovedisplayskip{6pt}
	\setlength\belowdisplayskip{6pt}
	\begin{split}
		\begin{matrix}
			\mathcal{F}_{en}^{2}=[\mathcal{P}(\mathcal{F}_{C}^{2}),\mathcal{P}(C_{1\times 1}(\eta(\mathcal{D}_{3\times 3}(\mathcal{F}_{C}^{2}))))]
		\end{matrix}
	\end{split}
\end{equation}
where $	\mathcal{F}_{C}^{2}=\mathcal{C}_{3\times 3}(\mathcal{F}_{en}^{1})$. Moreover, we know that increasing the network depth can improve the representational ability of the network. However, to avoid the problem of gradient disappearance caused by increasing the network depth, we use the residual block to further refine the encoding feature $\mathcal{F}_{en}^{2}$. The residual features extracted from the ResBlock is expressed as follows:
\begin{equation}
	\begin{aligned}
			\mathcal{F}_{re}^{i}= &\mathcal{R}_{3\times 3}^{i}(\mathcal{F}_{re}^{i-1})=  \mathcal{F}_{re}^{i-1}+ \\ & \eta(C_{3\times 3}(\mathcal{C}_{3\times 3}(\mathcal{F}_{re}^{i-1}))), i \in [1,N]
		\end{aligned}
	    \label{res}
\end{equation}
where $ \mathcal{F}_{re}^{0} =  \mathcal{F}_{en}^{2}$, $\mathcal{R}_{k\times k}^{i}(.)$ represents the i-th ResBlock operation, N is the number of the ResBlock. In this work, N=6.

For decoder, as shown in Fig. \ref{defog-structure}, it contains two upsample blocks (UpsampleBlocks) and two convolution blocks(ConvBlocks). Inside decoder, to preserve the image content and recover the details from the input, we concate the shallow features. In our decoder, after each upsampling, its output is spliced with that of the convolution block in the encoding stage, and then the convolution block in the decoding stage is used to refine the splicing result to obtain the decoding feature. The first decoding feature $\mathcal{F}_{de}^{1}\epsilon\mathbb{R}^{128\times {H/2}\times {W/2}}$ is formed by
\begin{equation}
	\begin{split}
		\begin{matrix}
			\mathcal{F}_{de}^{1}= \delta(C_{1 \times 1}(\mathcal{C}_{3\times3}([\delta(\mathcal{U}_{3\times 3}(\mathcal{F}_{re}^{N})),\mathcal{F}_{C}^{2}])))
		\end{matrix}
	\end{split}
\end{equation}
where $\mathcal{U}_{k\times k}(.)$ is the upsampling operation where kernel size is  $k\times k$. Similarly, the second decoding feature $\mathcal{F}_{de}^{2}\epsilon\mathbb{R}^{64\times {H}\times {W}}$ is getted by
\begin{equation}
	\begin{split}
		\mathcal{F}_{de}^{2}= \delta(C_{1 \times 1}(\mathcal{C}_{3\times3}([\delta(\mathcal{U}_{3\times 3}(\mathcal{F}_{de}^{1})),\mathcal{F}_{C}^{1}])))
	\end{split}
\end{equation}
Then, through a convolution operation and a Tanh activation function, we obtain the defogged result $I_{df}\epsilon\mathbb{R}^{3\times {H}\times {W}}$.
\begin{equation}
	\begin{split}
		I_{df}= \tau(C_{7\times7}(\mathcal{F}_{de}^{2}))
	\end{split}
\end{equation}
where $\tau$ denotes the Tanh activation function.

 \textbf{\textit{Synthesizing generator:}} In practice, due to the diversity and complexity of fog distribution, it is very difficult to directly use CNN to synthesis the foggy image. Thus, as shown in Fig. \ref{synthetic-structure}, the ASM is embedded in our synthesizing generator. We design two CNN-based models to optimize the transmission $T$ and atmospheric light $A$ respectively. Especially for atmospheric light $A$, we propose a sky segmentation model (SSM, as shown in Fig. \ref{synthetic-structure}). To obtain finer  transmission $T$, we use 5 convolution layers. As shown in Fig. \ref{synthetic-structure}, we first take a clear image and the maximum pixel value in its channel as the input, and then extract the roughly feature by using a convolution operation.
 %
 \begin{figure}[H]\scriptsize
 	\begin{center}
 		\vspace{-0.2cm}
 		\begin{tabular}{@{}c@{}}
 			\includegraphics[width = 0.49\textwidth]{img1/discuss/4}\\
 		\end{tabular}
 	\end{center}
 	\vspace{-0.3cm}
 	\caption{The architecture of the synthesizing generator.}
 	\vspace{-2.5mm}
 	\label{synthetic-structure}
 \end{figure}
 %
 After this operation, we use three convolutional neural networks with ReLU function to refine the rough feature. Finally, the transmission $T$ is generated by a convolution network with a sigmoid function.
\begin{equation}
	\begin{split}
		T=\sigma(C_{3\times3}(\mathcal{M}(\mathcal{M}(\mathcal{M}(C_{9\times9}([J(x),Max(J(x))])))))
	\end{split}
\end{equation}
wehre $\mathcal{M}$ denotes a convolution neural network with a ReLU function, $\sigma$ denotes the sigmoid function. $Max(.)$ means to take the maximum pixel value for a image.
\begin{figure}[h]\footnotesize
	\centering
	\begin{center}
	\vspace{-0.2cm}
		\begin{tabular}{@{}ccccc@{}}
			\rotatebox{90}{  \quad Foggy} & \hspace{-0.5cm}
			\rotatebox{90}{ \quad images} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/486} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/168} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/363} \\

			\rotatebox{90}{Transmission} & \hspace{-0.5cm}
			\rotatebox{90}{ \quad maps} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/T/486_Our_T} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/T/168_Our_T} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/T/363_Our_T} \\

			\rotatebox{90}{ Defogged} &\hspace{-0.6cm}
			\rotatebox{90}{\quad results}& \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/486_Our} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/168_Our} & \hspace{-0.5cm}
			\includegraphics[width = 0.15\textwidth]{img1/transmission/363_Our} \\
		\end{tabular}
	\end{center}
	\vspace{-0.3cm}
	\caption{Defogged results and corresponding transmission maps through our HAAN on real-world outdoor foggy images.}
	\vspace{-0.25cm}
	\label{Transmission-map}
\end{figure}

Fig. \ref{Transmission-map} illustrates three transmission maps generated by our proposed network. It can be seen that each transmission map can effectively estimate the depth information of the scene and preserve richer texture details.

To optimize $A$, in our previous work \cite{liu2020end}, we propose a sky prior. For the foggy image with sky region, we assume that the depth of the sky is regarded as infinity, and define it as the intensity of pixels in the area of maximum fog density. Then, the average value of the sky region is described as $A$. For more details about this derivation process, please refer to our previous work  \cite{liu2020end}. Therefore, how to divide the sky region from the foggy image is crucial. Instead of using the existing technology \cite{he2010fast} \cite{closed2007TPAMI} \cite{otsu1979threshold} to segment the image to obtain the sky region, we propose a CNN-based sky segmentation model (SSM).

As shown in Fig. \ref{synthetic-structure}, SSM consists of an Ehnace Block and a Segmentation Block. Each block adopts the encoder-decoder network. For a foggy image, we observed that the boundary between sky area and non-sky area was often blurred due to the fog. Thus, we first enhance the foggy image to improve the edge information by using an enhance network, which can sharpness this boundary and label the sky region. Inside enhance block, we also use the dense connection to improve the texture details of the extracted features. For an input foggy image $I_{M}$, the output of each layer in encoding stage is $E_{e}^{i}$, and $D_{e}^{j}$ is the output of each layer in decoding stage. The encoding process is defined as:
\begin{equation}
	\begin{aligned}
		{E}_{e}^{i}= \delta(C_{4 \times 4}({E}_{e}^{i-1}))
	\end{aligned}
\end{equation}
where ${E}_{e}^{0}=I_{M}$. The decoding process is defined as:
\begin{equation}
		\left\{\begin{aligned}
			{D}_{e}^{1} = &\mathcal{U}_{4\times 4}({E}_{e}^{m}),\\
			{D}_{e}^{k+1} = &\mathcal{U}_{4\times 4}([{D}_{e}^{k},{E}_{e}^{m-k}]),k \in [1,n-1]\\
		\end{aligned}\right.
\end{equation}
where i and j are layer indices ($i=1,...,m$, $j=1,...,n$, $m$ and $n$ are the number of layers). In this work, $m=n=3$. Finally, the enhanced image $E_{M}$ is generated by using two convolution layers:
\begin{equation}
	\begin{split}
		E_{M}= \tau(C_{3\times3}(\delta(C_{1\times1}([\iota({E}_{e}^{m}),\iota([{D}_{e}^{1},{E}_{e}^{m-1}])\\
		,...,\iota([{D}_{e}^{n-1},{E}_{e}^{1}]),{D}_{e}^{n}]))))
	\end{split}
\end{equation}
where $\iota$ represents the nearest neighbor linear interpolation operation.
\begin{figure}[h]\footnotesize
	\centering
	\begin{center}
		\vspace{-0.2cm}
		\resizebox{\columnwidth}{!}{
		\begin{tabular}{@{}ccc@{}}

			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/img/city_0276} & \hspace{-0.4cm}
			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/img/scenery_0160} & \hspace{-0.4cm}
			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/img/water_21} \\

			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/label/city_0276} & \hspace{-0.4cm}
			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/label/scenery_0160} & \hspace{-0.4cm}
			\includegraphics[width = 0.15\textwidth,height=0.08\textheight]{img1/sky/label/water_21} \\
		\end{tabular}}
	\end{center}
	\vspace{-0.3cm}
	\caption{The samples from FSID. The images in the top row are
		foggy images in different scenes. The bottom row represents
		the labeled sky region for the above images.}
	\vspace{-0.25cm}
	\label{sky-map}
\end{figure}

In segmentation block, we add residual blocks in addition to encoding network and decoding network (as shown in Fig. \ref{synthetic-structure}). Similarly, we defined the outputs of each layer in encoding, decoding, and residual network as $E_{s}^{i1}$, $D_{s}^{j1}$, and $R_{s}^{k}$ respectively ($i1=1,...,m_{1}$; $j1=1...,n_{1}$; $k=1,...,K$; $m_{1}$, $n_{1}$ and $K$ are the number of layers). In this block, $m_{1}=3$, $n_{1}=2$, and $K=6$. Therefore, the encoding process in segmentation block is formulated as:
\begin{equation}
		\left\{\begin{aligned}
			{E}_{s}^{1} = & \delta(\eta(C_{7\times 7}(E_{M})),\\
			{E}_{s}^{2} = & \delta(C_{3 \times 3}({E}_{s}^{1})),\\
			{E}_{s}^{3} = & \delta(C_{3 \times 3}({E}_{s}^{2}))
		\end{aligned}\right.
\end{equation}
For $R_{s}^{k}$, we use the equation \ref{res} to obtain. Note that, $ \mathcal{F}_{re}^{0} =  {E}_{s}^{3}$. Then, the decoding process is formulated as:
\begin{equation}
		\left\{\begin{aligned}
			{D}_{s}^{1} =& \mathcal{U}_{3\times 3}({R}_{s}^{3}),\\
			{D}_{s}^{2} =& \mathcal{U}_{3\times 3}(\delta(C_{3\times 3}([{D}_{s}^{1},{E}_{s}^{2}])))
		\end{aligned}\right.
\end{equation}
After the above process, the final output segmentation image ($O_M$) is defined as:
\begin{equation}
	\begin{split}
		O_{M}= \tau(C_{7\times7}(\delta(C_{3\times3}([{D}_{s}^{2},{E}_{s}^{1}]))))
	\end{split}
\end{equation}
Thus, the atmospheric light $A$ is calculated as follows:
\begin{equation}
	\begin{split}
		A = mean(O_{M})
	\end{split}
\end{equation}
where $mean(.)$ denotes an average filter. However, due to the small capacity of our GPU, we used a pre-trained SSM to segment the sky region to improve the training efficiency of our  HAAN. Please refer to our online address (https://github.com/ChengChen-ai/Sky-Segmentation) for details about our SSM. Moreover, we proposed a foggy-sky image dataset (FSID) for sky segmentation research. It contains 900 natural outdoor scenes, including traffic scenes, landscape, water scenes, etc.. For each scene, as shown in Fig. \ref{sky-map}, there is one foggy image and corresponding labeled sky region image. In addition, similar to \cite{liu2020end}, if the image has no sky region, we followed by the dark channel \cite{He2011PAMI} that  picked the maximum of the top 0.1 percent brightest pixels in the image as A.
\begin{figure}[H]\scriptsize
	\begin{center}
		\vspace{-0.2cm}
		\begin{tabular}{@{}c@{}}
			\includegraphics[width = 0.49\textwidth]{img1/discuss/3}\\
		\end{tabular}
	\end{center}
	\vspace{-0.3cm}
	\caption{The architecture of Holistic Attention-Fusion generator.}
	\vspace{-2.5mm}
	\label{attention-structure}
\end{figure}
 \textbf{\textit{Holistic Attention-Fusion generator:}}  To further improve the ability of transformation to map fog images to a fogfree image domain through the defogging generator, the output of $M_R$ and three derived images of the foggy image are input into a color-texture recovery network to refine correlated information for generating a better-recovered image with a color coherence and exactness of detail. We refer to this network as Holistic Attention-Fusion generator, as shown in Fig. \ref{attention-structure}.
 Inspired by \cite{GFN2018CVPR}, we also use the gated fusion strategy to recover the color and visibility of the defogged image with three derived inputs of the original foggy image, such as white balanced image $I_{wb}$, contrast enhanced image $I_{ce}$, and gamma corrected image $I_{gc}$.

In past studies, white balance has been found to have the potential to restore color in scenes, which is intended to eliminate chromatic aberrations caused by atmospheric colors. In this paper, we use the technology based on the gray world hypothesis \cite{GFN2018CVPR} \cite{wb2001} to obtain the white balance feature map of foggy images, as shown in Fig. \ref{derived-map}(b). The formula is as follows:
\begin{equation}
	\begin{split}
		\left\{\begin{matrix} I(R^{'})=I(R)*k_r\\ I(G^{'})=I(G)*k_g\\ I(B^{'})=I(B)*k_b \end{matrix}\right.
	\end{split}
\end{equation}
\begin{equation}
	\begin{split}
		I_{wb}=[I(R^{'}),I(G^{'}),I(B^{'})]
	\end{split}
\end{equation}
where, $k_r$, $k_g$ and $k_b$ represent the gain coefficients of R, G, and B channels respectively. $I_{wb}$ represents the result of a white balance operation.
\begin{figure}[h]\footnotesize
	\centering
	\begin{center}
		\vspace{0.1cm}
		\resizebox{\columnwidth}{!}{
			\begin{tabular}{@{}cc@{}}
				\includegraphics[width = 0.15\textwidth]{img1/features/1} & \hspace{-0.4cm}
				\includegraphics[width = 0.15\textwidth]{img1/features/2}  \\
				(a)  & \hspace{-0.4cm} (b)\\
				\includegraphics[width = 0.15\textwidth]{img1/features/3} & \hspace{-0.4cm}
				\includegraphics[width = 0.15\textwidth]{img1/features/4} \\
				(c)  & \hspace{-0.4cm} (d)

		\end{tabular}}
	\end{center}
	\vspace{-0.3cm}
	\caption{ The derived inputs of foggy image. (a) Foggy input.
		(b) White balanced image. (c) Contrast enhanced image. (d)
		Gamma corrected image.}
	\vspace{-0.25cm}
	\label{derived-map}
\end{figure}
To further improve the contrast of the results, we introduce the following two derived inputs. For $I_{ce}$, we exploit the Ancuti \cite{fusiondehazing2013TIP} proposed method to improve the global visibility of haze areas in foggy images.
\begin{equation}
	\begin{split}
		I_{ce}=\mu (I-\overline{I})
	\end{split}
\label{ce}
\end{equation}
where $I$ is the foggy input, $\overline{I}$ is the average luminance value of $I$. $\mu$ is a linearly increase factor, $\mu=2(0.5 +\overline{I})$. However, it can be seen from (\ref{ce}) that as the value of $\overline{I}$ increases, $(I-\overline{I})$ may be negative, resulting in the dark area in foggy image tending to black after contrast enhancement (as shown in Fig. \ref{derived-map}(c)). To address this problem, we adopt gamma correction:
\begin{equation}
	\begin{split}
		I_{gc}=\alpha I^{\gamma }
	\end{split}
\end{equation}
As discussed in \cite{GFN2018CVPR}, we also set $\alpha=1$ ,and the decoding gamma correction $\gamma=2.5$. As shown in Fig. \ref{derived-map}(d), through gamma correction, we not only enhanced the visibility of the original foggy image, but also effectively avoided the appearance of the severe dark area in Fig. \ref{derived-map}(c).

After obtaining these derived inputs, how to fuse them together to optimize the defogged result is the key of this paper. In \cite{GFN2018CVPR}, however, Ren  \textit{et al.} ignored the influence of channel features and spatial features of each layer in the network on confidence maps respectively, resulting in the loss of partial texture details. Thus, we design a simple but effective method combining channel attention and spatial attention to fuse the confidence maps of each input to recover a higher quality image from foggy image, which can not only learns the hierarchical-specific knowledge from all preceding layers but also reduces the feature redundancy effectively. As shown in Fig. \ref{attention-structure}, we first concatenated each input image according to the channel direction to obtain the concatenation feature map $\mathcal{F}_{ct}$:
\begin{equation}
	\setlength\abovedisplayskip{6pt}
	\begin{split}
		\mathcal{F}_{ct}=[I_{df},I_{ec},I_{gm},I_{wb}]
	\end{split}
\end{equation}
where $I_{df}$ is the output of $M_R$. Then, a layer of convolutional network is used to initialize the global spatial information of the concatenation feature map to highlight the effective information of each channel. Thereafter, we use a channel attention network to assign weights to different channels of $\mathcal{F}_{ct})$. For channel attention network, we first use global average pooling and maximum pooling to compress the global spatial features of channels into two different spatial feature expressions: $g_{avg}^{c} \epsilon \mathbb{R}^{C\times 1\times 1}$ and $g_{max}^{c} \epsilon \mathbb{R}^{C\times 1\times 1}$. Then, we generate the channel-attention map $W_{c}\epsilon \mathbb{R}^{C\times 1\times 1}$ through a shared network:
\begin{equation}
	\begin{split}
		W_{c}&=\sigma (C_{3\times3}(\delta (C_{3\times3}(g_{avg}^{c})))\\
		&+C_{3\times3}(\delta (C_{3\times3}(g_{max}^{c})))), c \in \{df, wb, ec, gm\}
	\end{split}
\end{equation}
 %
where $W_{df}$ is the weight of $I_{df}$, $W_{wb}$ is the white balanced weight, $W_{ec}$ is the contrast enhanced weight, and $W_{gm}$ is the gamma correction weight. Finally, we get the preliminary fusion feature $\mathcal{F}_{ct}^{*}$ through the following formula:
\begin{equation}
	\begin{aligned}
		\mathcal{F}_{ct}^{*} &=W_{c}\cdot \mathcal{F}_{ct}=(W_{df},W_{ec},W_{gm},W_{wb})\cdot (I_{df},I_{ec},I_{gm},I_{wb})\\ &=W_{df}\odot I_{df} + W_{ec}\odot I_{ec} + W_{gm}\odot I_{gm} + W_{wb}\odot I_{wb}
	\end{aligned}
\end{equation}
where $\cdot$ denotes inner-vector product, $\odot$ denotes the element-wise product. To further uptake of channel dimension information, we propose a spatial attention network, which can contain responses from all dimensions of the feature
map $\mathcal{F}_{ct}^{*}$. It pays more attention to the location features information by assigning greater weight to high-frequency signals and dense fog areas. Along the channel direction, we calculate the mean value $g_{avg}^{s} \epsilon \mathbb{R}^{1\times H\times W}$ and maximum value $g_{max}^{s} \epsilon \mathbb{R}^{1\times H\times W}$ of $\mathcal{F}_{ct}^{*}$ in the spatial dimension to effectively highlight fog relevant feature to preserve more texture details for generated result. Then we concatenate these two spatial features and put them into a convolution layer and a sigmoid function to obtain the weight $W_{s}\epsilon \mathbb{R}^{1\times H\times W}$ of $\mathcal{F}_{ct}^{*}$:
\begin{equation}
	\begin{split}
		W_{s}=\sigma (C_{3\times3}([g_{avg}^{s},g_{max}^{s}]))
	\end{split}
\end{equation}
The final attention-fusion feature is calculated as follows:
\begin{equation}
	\begin{split}
		\mathcal{F}_{fusion}=W_{s} \odot \mathcal{F}_{ct}^{*}
	\end{split}
\end{equation}
\begin{figure}[H]\scriptsize
	\begin{center}
		\vspace{-0.2cm}
		\begin{tabular}{@{}cc@{}}
			\includegraphics[width = 0.5\textwidth]{img1/discuss/6}\\
		\end{tabular}
	\end{center}
	\vspace{-0.3cm}
	\caption{The architcture of Discriminator.}
	\vspace{-3.5mm}
	\label{Discriminate-structure}
\end{figure}

 \textbf{\textit{Discriminator:}} As shown in Fig. \ref{architctures}, we designed two discriminator networks and named them as $D_{ff}$ and $D_f$. For $D_{ff}$, it is used to distinguish the generated results of $M_{R}$ and $M_{CTR}$ from the input clear images. For $D_f$, it is used to distinguish between the generated results from $M_{S}$ and the input foggy images. As can be seen in Fig. \ref{Discriminate-structure}, our discriminator consists of 6 convolutional blocks. The LeakyReLU function and Batch Normalization (BN) are used alternately. From the beginning to the end, the number of convolutional filters at each layer is 64, 128, 256, 512, 512 and 1. The stride of the first four layers is 2, and the stride of the last two layers is 1. Moreover, all filters have the same convolution kernel size, which are 4 × 4.

\subsection{Loss Function}

To train our proposed holistic attention-fusion adversarial defogging model, our loss function consists of three terms: 1) adversarial loss $\mathcal{L} _{R_{adv}}$, 	$\mathcal{L} _{CTR_{adv}}$, $\mathcal{L} _{S_{adv}}$; 2) cycle-consistency loss $\mathcal{L}_{c1}$, $\mathcal{L}_{c2}$; 3) perceptual loss $\mathcal{L}_{Perceptual}$. The total loss of our network is presented as:
\begin{equation}
		\begin{aligned}
			\mathcal{L}_{Total} = &\lambda _1\mathcal{L} _{R_{adv}}+\lambda _2\mathcal{L} _{CTR_{adv}}+\lambda _3\mathcal{L} _{S_{adv}}+\\
		   &\lambda _4\mathcal{L}_{c1}+
			\lambda _5\mathcal{L}_{c2}+\lambda _6\mathcal{L}_{Perceptual}
		\end{aligned}
	\label{total}
\end{equation}
where $\lambda _1$, $\lambda _2$, $\lambda _3$, $\lambda _4$, $\lambda _5$ and $\lambda _6$ are the
positive weights, which are used to balance the importance of the corresponding loss.

\textbf{\textit{Adversarial Loss:}} We considered three losses of adversarial learning including $\mathcal{L} _{R_{adv}}$, $\mathcal{L} _{CTR_{adv}}$ and $\mathcal{L} _{S_{adv}}$, where $\mathcal{L} _{R_{adv}}$ and $\mathcal{L} _{CTR_{adv}}$ fooled the discriminator $D_{ff}$ by encouraging Defogging generator and Holistic Attention-Fusion generator to recover high-quality clean image, and $\mathcal{L} _{S_{adv}}$ aims to fool the $D_{f}$ using Synthesizing generator to synthesize a more realistic foggy image. These adversarial losses are defined as:
\begin{equation}
	\begin{aligned}
		\mathcal{L} _{R_{adv}}(M_{R},D_{ff})=&\underset{y_r\sim Y_R}{\mathbb{E}}\left[logD_{ff}(y_r)\right]+\\
		&\underset{x_r\sim X_R}{\mathbb{E}}\left[log(1-D_{ff}(M_{R}(x_r)))\right]
	\end{aligned}
\end{equation}
\begin{equation}
	\begin{aligned}
		\mathcal{L} _{CTR_{adv}}(M_{R},M_{CTR},D_{ff})=\underset{y_r\sim Y_R}{\mathbb{E}}\left[logD_{ff}(y_r)\right]+&\\
		\underset{x_r\sim X_R}{\mathbb{E}}\left[log(1-D_{ff}(M_{CTR}(M_{R}(x_r),I_{d}^{x_r})))\right]&
	\end{aligned}
\end{equation}
\begin{equation}
	\begin{aligned}
		\mathcal{L} _{S_{adv}}(M_{S},D_{f})=&\underset{x_r\sim X_R}{\mathbb{E}}\left[logD_{f}(x_r)\right]+\\
		&\underset{y_r\sim Y_R}{\mathbb{E}}\left[log(1-D_{f}(M_{S}(y_r)))\right]
	\end{aligned}
\end{equation}
where $X_R$ represents the real-world fog image domain and $Y_R$ refers to the real-world clear image domain; $x_r$ and $y_r$ represent the input image from $X_R$ and $Y_R$ respectively. $M_{R}(.)$, $M_{CTR}(.)$ and $M_{S}(.)$ represent the results generated by Fog Removal Module, Color-Texture Recovery Module and Fog Synthetic Module respectively; $I_{d}^{x_r}$ denotes the derived images of $x_r$.

\begin{figure*}[htbp]\footnotesize
	\begin{center}
		\vspace{-0.2cm}
		\resizebox{\textwidth}{!}{
			\begin{tabular}{@{}ccccccccc@{}}

				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/haze/fusion1} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/cycle/fusion1} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/cycle_dehaze/fusion1} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/without_VGG/fusion1} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/without_DC/fusion1}
				& \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/without_ASM/fusion1} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/without_HDA/fusion1}
				& \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth,height=0.165\textheight]{img1/ablation_study/our/fusion1} \\

				NIQE=3.3808
				&\hspace{-0.45cm} NIQE=3.1257
				&\hspace{-0.45cm} NIQE=2.7277
				&\hspace{-0.45cm} NIQE=2.1934
				&\hspace{-0.45cm} NIQE=2.4156
				&\hspace{-0.45cm} NIQE=2.2274
				&\hspace{-0.45cm} NIQE=2.4728
				&\hspace{-0.45cm} NIQE=2.1456 \\

				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/haze/fusion2} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/cycle/fusion2} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/cycle_dehaze/fusion2} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/without_VGG/fusion2} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/without_DC/fusion2}
				& \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/without_ASM/fusion2} & \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/without_HDA/fusion2}
				& \hspace{-0.45cm}
				\includegraphics[width = 0.19\textwidth]{img1/ablation_study/our/fusion2} \\

				NIQE=2.6351
				&\hspace{-0.45cm} NIQE=3.1334
				&\hspace{-0.45cm} NIQE=2.7820
				&\hspace{-0.45cm} NIQE=2.6722
				&\hspace{-0.45cm} NIQE=2.6277
				&\hspace{-0.45cm} NIQE=2.2740
				&\hspace{-0.45cm} NIQE=2.4565
				&\hspace{-0.45cm} NIQE=2.1156 \\

				(a) Foggy Image & \hspace{-0.45cm}
				(b) CycleGAN \cite{zhu2017unpaired} & \hspace{-0.45cm}
				(c) Cycle-Dehaze \cite{cycle-dehazing} & \hspace{-0.45cm}
				(d) w/o VGG & \hspace{-0.45cm}
				(e) w/o DC & \hspace{-0.45cm}
				(f) w/o ASM & \hspace{-0.45cm}
				(g) w/o HDA & \hspace{-0.45cm}
				(h) HAAN
		\end{tabular}}
	\end{center}
	\vspace{-0.3cm}
	\caption{Comparison results of different configurations in visual effects on real-world outdoor images.}
	\vspace{-0.45cm}
	\label{ablation_dehaze}
\end{figure*}

\textbf{\textit{Cycle-consistency Loss:}} We applied a mean-square-error loss to make the final output of each path in the both Fog2Fogfree and Fogfree2Fog block close to the corresponding initial input. For Fog2Fogfree block, we obtained cycle-consistency loss function for foggy images among $I_{rf}$, $I^1_{rcf}$, and $I^2_{rcf}$:
\begin{equation}
	\begin{aligned}
		\mathcal{L}_{c1}=\left \|I_{rf}-I^1_{rcf} \right \|_{2}^{2}
		+\left \|I_{rf}-I^2_{rcf} \right \|_{2}^{2}
	\end{aligned}
\end{equation}
where $I_{rf}$ is the input of real-world foggy image, $I^1_{rcf}$ and $I^2_{rcf}$ represent the output of the defogging path and the color-texture recovery path respectively. For Fogfree2Fog block, cycle-consistency loss function for fogfree images is
constructed among $I_{rff}$, $I_{rcff}$, and $I^{sf}_{rr}$:
\begin{equation}
	\begin{aligned}
		\mathcal{L}_{c2}=\left \|I_{rff}-I_{rcff} \right \|_{2}^{2}
		+\left \|I_{rff}-I^{sf}_{rr} \right \|_{2}^{2}
	\end{aligned}
\end{equation}
where $I_{rff}$ is the input of real-world clean image, $I_{rcff}$ and $I^{sf}_{rr}$ denote the output of the synthesizing path and the color-texture recovery path respectively.

\textbf{\textit{Perceptual Loss:}} To better extract more texture details from foggy images, we introduce a perceptual loss function, which is constructed with the pretrained VGG19 \cite{Simonyan2014} to compare the original input image with the reconstructed image in the feature space. This objective is defined as:
\begin{equation}
	\begin{aligned}
		\mathcal{L}_{Perceptual}=\left \|\phi _i(I_{rf})-\phi _i(I^1_{rcf}) \right \|_{2}^{2}
		+\left \|\phi _i(I_{rf})-\phi _i(I^2_{rcf}) \right \|_{2}^{2}&\\
		+\left \|\phi _i(I_{rff})-\phi _i(I_{rcff}) \right \|_{2}^{2}
		+\left \|\phi _i(I_{rff})-\phi _i(I^{sf}_{rr}) \right \|_{2}^{2}&
	\end{aligned}
    \label{perceptual}
\end{equation}
where $\phi _i$ denotes the feature map in $i^{th}$ layer of VGG19 network.
\section{Experiments and Discussion}
\label{experiment}
\subsection{Implementation Setting and Details}

1) \textbf{\textit{Evaluation methods}}

In this section, we compare the defogged performance of our proposed approach with that of seven state-of-the-art methods: the DehazeNet \cite{dehazenet2016TIP} (TIP 2016), a densely connected pyramid dehazing network (DCPDN) \cite{zhang2018densely} (CVPR 2018), an enhanced pix2pix dehazing network (EPDN) \cite{pix2pixdehazing} (CVPR 2019), a grid dehazing network (GridNet) \cite{liu2019griddehazenet} (ICCV 2019), a gated context aggregation network (GCANet) \cite{chen2019gated} (WACV 2019), a domain adaptation for image dehazing (DA-Dehaze) \cite{shao2020domain} (CVPR 2020) and an image dehazing and exposure (IDE) \cite{IDE2021ide} (TIP 2021).

2) \textbf{\textit{Implementation details}}

Due to our proposed HAAN uses unsupervised training, we randomly choose both outdoor synthetic and real-world unpaired hazy images from the RESIDE dataset \cite{RESIDE}. The dataset is widely used, which is divided into five subsets: Indoor Training Set (ITS), Outdoor Training Set (OTS), Synthetic Object Testing Set (SOTS), Unannotated real Hazy Images (URHI) and real Task-driven Testing Set (RTTS). We train the network by randomly selecting 10000 synthetic foggy images from the OTS and 8000 real foggy images from the URHI and RTTS.

Our HAAN is implemented by PyTorch 1.8.0 with one NVIDIA GeForce GTX 3060 GPU. During training, we resize all the images to $256 \times 256 $ and normalize the pixel values to [-1,1]. The models are trained using ADAM optimizer with exponential decay rates $\beta$ equal to 0.999, meanwhile the learning rate and batchsize are initialised to $1 \times 10^{-4}$ and 2, respectively. We set the total number of epoch to 15 for 270000 iterations. The perceptual loss in (\ref{perceptual}) is set using the latent features of the 2nd and 5th layers from the fixed pre-trained VGG19, and empirically initialise default values of $\lambda _1$, $\lambda _2$, $\lambda _3$, $\lambda _4$, $\lambda _5$ and $\lambda _6$ in (\ref{total}) are set to 10, 10, 10, 5, 5, 1 respectively.

3) \textbf{\textit{Evaluation datasets and metrics}}

We evaluate the proposed method on two synthetic datasets (SOTS, HazeRD) and two real-world datasets (O-HAZE, LIVE). For the synthetic datasets, SOTS comes from the RESIDE dataset, which contains 500 indoor scenes and 500 outdoor scenes. HazeRD synthesized by Zhang \textit{et al.} \cite{HAZERD2017ICIP} which contains 75 synthetic outdoor foggy images of different fog concentrations. The real-world foggy dataset O-HAZE \cite{O-HAZE} used in the NTIRE 2018 image dehazing challenge. It contains 45 outdoor real-world hazy images, and the LIVE dataset provided by Choi \textit{et al.}\cite{referencelessdefogging2015TIP} which contains 500 outdoor real-world foggy images.

To evaluate the performance of our method, we utilize reference indicators and non-reference indicators as the evaluation metrics. Like most defogging papers, the Structure Similarity (SSIM) \cite{wang2004image}, the Peak Signal to Noise Ratio (PSNR) \cite{huynh2008scope} and the Learned Perceptual Image Patch Similarity (LPIPS) \cite{zhang2018unreasonable} are usually used as criteria to evaluate the quantitative performance of defogging methods. Recently, the Realness Index (RI) was proposed by Zhao \textit{et al.} \cite{zhao2020dehazing} to evaluate the quality of the defogged image. The higher the value of the RI, the closer to the clear reference. In addition, as the corresponding ground truth of real-world foggy images is extremely difficult to available, the performance of defogged results on these real-world images was assessed by no-parameter models including Fog Aware Density Evaluator (FADE) \cite{referencelessdefogging2015TIP}, the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) \cite{MSCN2012TIP}, Natural Image Quality Evaluator (NIQE) \cite{mittal2012making} and Bilnd Assessment based on Visibility Enhancement (BAVE) \cite{hautiere2011blind}. Among them, FADE predicts the visibility of a single foggy image to perceive the fog density. The lower the value of the FADE, the stronger the defogged ability. BRISQUE uses natural scene statistics to quantify losses of “naturalness” in the image due to the distortions. The score of the BRISQUE ranges from 0 to 100, the closer to 0 the better the defogged quality. NIQE is a space domain natural scene statistics-based quality aware collection for predicting the quality of defogged image. The lower the value of the NIQE, the better the defogged effect. For BAVE, we consider the evaluation indicator $\overline{\gamma}$, which represents the mean ratio of the gradients at visible edges. The higher the value of $\overline{\gamma}$, the better the defogged performance.

\subsection{Ablation Studies}
 To demonstrate the effectiveness of our HAAN architecture, we perform a series of ablation analyses on the different modules of our proposed HAAN. We mainly consider the following four factors: 1) \textit{w/o VGG:} our full model without the VGG perceptual losses; 2) \textit{w/o DC:} our full model without the proposed densely-residual connected encoder-decoder; 3) \textit{w/o ASM:} our full model without the atmospheric scattering model; 4) \textit{w/o HDA:} our full model without the holistic attention-fusion generator. Moreover, we also compare our method to CycleGAN \cite{zhu2017unpaired} and Cycle-Dehaze \cite{cycle-dehazing}.

 \begin{table}[h]
 	\begin{center}
 		\captionsetup{justification=centering}
 		\caption{\\Q{\footnotesize UANTITATIVE} PSNR {\footnotesize AND} SSIM R{\footnotesize ESULTS} {\footnotesize ON} SOTS O{\footnotesize UTDOOR} D{\footnotesize ATASET} U{\footnotesize SING} D{\footnotesize IFFERENT} C{\footnotesize ONFIGURATIONS}}
 		\resizebox{\columnwidth}{!}{
 			\begin{tabular}{cccccccc}
 				\toprule
 				& CycleGAN & Cycle-Dehaze & w/o VGG  & w/o DC & w/o ASM & w/o HDA & HAAN \\
 				\midrule
 				PSNR & 14.7115 & 21.7522 & 21.2083 & 22.5896 & 23.2111 & 25.7661 & \textbf{29.3991} \\
 				SSIM & 0.5024 & 0.7930 & 0.9073 & 0.8795 & 0.8467 & 0.9331 & \textbf{0.9646}\\
 				\bottomrule
 		\end{tabular}}
 		\label{ablation-tb}
 		\vspace{-4.5mm}
 	\end{center}
 \end{table}

 \begin{figure*}[htbp]\footnotesize
 	\begin{center}
 		\vspace{-0.2cm}
 		\resizebox{\textwidth}{!}{
 			\begin{tabular}{@{}cccccccccc@{}}

 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/fog/1} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DehazeNet/1_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DCPDN/1_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/EPDN/1_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/GridDehazeNet/1-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/GCANet/1_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DA/1_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/IDE/1_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/Our/1_Our} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/fogfree/1} \\

 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/fog/201} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DehazeNet/201_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DCPDN/201_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/EPDN/201_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/GridDehazeNet/201-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/GCANet/201_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/DA/201_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/IDE/201_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/Our/201_12Our} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/synthesis_compare/fogfree/201} \\

 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/fog/450} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/DehazeNet/450_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/DCPDN/450_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/EPDN/450_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/GridDehazeNet/450-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/GCANet/450_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/DA/450_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/IDE/450_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/Our/450_Our} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.072\textheight]{img1/synthesis_compare/fogfree/450} \\

 				(a) & \hspace{-0.45cm}
 				(b) & \hspace{-0.45cm}
 				(c) & \hspace{-0.45cm}
 				(d) & \hspace{-0.45cm}
 				(e) & \hspace{-0.45cm}
 				(f) & \hspace{-0.45cm}
 				(g) & \hspace{-0.45cm}
 				(h) & \hspace{-0.45cm}
 				(i) & \hspace{-0.45cm}
 				(j)
 		\end{tabular}}
 	\end{center}
 	\vspace{-0.3cm}
 	\caption{Qualitative comparison between the proposed HAAN and the state-of-the-art methods on the SOTS dataset. (a) Foggy Image. (b) DehazeNet \cite{dehazenet2016TIP}. (c) DCPDN \cite{zhang2018densely}. (d) EPDN \cite{pix2pixdehazing}. (e) GridNet \cite{liu2019griddehazenet}. (f) GCANet \cite{chen2019gated}. (g) DA-Dehaze \cite{shao2020domain}. (h) IDE \cite{IDE2021ide}. (i) HAAN. (j) Ground truth.}
 	\vspace{-0.45cm}
 	\label{synthesis1_dehaze}
 \end{figure*}

 \begin{figure*}[htbp]\footnotesize
 	\begin{center}
 		\vspace{0.2cm}
 		\resizebox{\textwidth}{!}{
 			\begin{tabular}{@{}cccccccccc@{}}

 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fog/1-0} & \hspace{-0.45cm}
 				\includegraphics[width =0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DehazeNet/1-0_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DCPDN/1-0_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/EPDN/1-0_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GridDehazeNet/1-0-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GCANet/1-0_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DA/1-0_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/IDE/1-0_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/Our/1-0_Our} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fogfree/1_HazeRD} \\

 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fog/1-2} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DehazeNet/1-2_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DCPDN/1-2_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/EPDN/1-2_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GridDehazeNet/1-2-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GCANet/1-2_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DA/1-2_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/IDE/1-2_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/Our/1-2_Our} &
 				\hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fogfree/1_HazeRD} \\

 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fog/1-4} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DehazeNet/1-4_DehazeNet} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DCPDN/1-4_DCPCN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/EPDN/1-4_EPDN} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GridDehazeNet/1-4-Grid} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/GCANet/1-4_GCA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/DA/1-4_DA} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/IDE/1-4_IDE} & \hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/Our/1-4_Our} &
 				\hspace{-0.45cm}
 				\includegraphics[width = 0.09\textwidth,height=0.043\textheight]{img1/synthesis_compare/fogfree/1_HazeRD} \\

 				(a) & \hspace{-0.45cm}
 				(b) & \hspace{-0.45cm}
 				(c) & \hspace{-0.45cm}
 				(d) & \hspace{-0.45cm}
 				(e) & \hspace{-0.45cm}
 				(f) & \hspace{-0.45cm}
 				(g) & \hspace{-0.45cm}
 				(h) & \hspace{-0.45cm}
 				(i) & \hspace{-0.45cm}
 				(j)
 		\end{tabular}}
 	\end{center}
 	\vspace{-0.3cm}
 	\caption{Qualitative comparison between the proposed HAAN and the state-of-the-art methods on the HazeRD dataset. (a) Foggy Image. (b) DehazeNet \cite{dehazenet2016TIP}. (c) DCPDN \cite{zhang2018densely}. (d) EPDN \cite{pix2pixdehazing}. (e) GridNet \cite{liu2019griddehazenet}. (f) GCANet \cite{chen2019gated}. (g) DA-Dehaze \cite{shao2020domain}. (h) IDE \cite{IDE2021ide}. (i) HAAN. (j) Ground truth.}
 	\vspace{-0.45cm}
 	\label{synthesis2_dehaze}
 \end{figure*}

 \begin{table*}[ht]
 	\begin{center}
 		\captionsetup{justification=centering}
 		\caption{\\A{\footnotesize VERAGE} PSNR, SSIM, LPIPS, RI, FADE, BRISQUE, NIQE, $\overline{\gamma}$ {\footnotesize OF} D{\footnotesize EFOGGED} R{\footnotesize ESULTS} {\footnotesize ON} SOTS/HazeRD D{\footnotesize ATASETS}.\\T{\footnotesize HE} T{\footnotesize OP} T{\footnotesize WO} P{\footnotesize ERFORMANCE} V{\footnotesize ALUES} A{\footnotesize RE} H{\footnotesize IGHLIGHTED} {\footnotesize IN} R{\footnotesize ED} {\footnotesize AND} B{\footnotesize LUE}}
 		\resizebox{\textwidth}{!}{
 			\begin{tabular}{c|cccccccc|cccccccc}
 				\hline
 				Dataset & \multicolumn{8}{c|}{SOTS} & \multicolumn{8}{c}{HazeRD}\\

 				\hline
 				\diagbox{Method}{Metric} & PSNR & SSIM & LPIPS & RI & FADE & BRISQUE & NIQE & $\overline{\gamma}$ & PSNR & SSIM & LPIPS & RI & FADE & BRISQUE & NIQE & $\overline{\gamma}$ \\

 				\hline
 				DehazeNet &  23.6907 & 0.8698 & 0.0415 & 0.9742 & 0.6400 & 13.5029 & 3.3586 & 1.4635 &  15.3478 & 0.7956 & 0.1522 & 0.9473 & 1.3983 & 28.6569 & 4.4703 & 1.6570 \\

 				DCPDN &  21.0540 & 0.8716 & 0.0718 & 0.9713  & 0.9340 & 21.8789 & 3.8303 &  1.3136&  14.3647 & 0.7961 & 0.2094 & 0.9355 & 1.5799 & 25.9806 & 5.8394 & 2.3011 \\

 				EPDN &  22.0715 & 0.8376 & 0.0524 & 0.9712 & \textcolor[rgb]{1,0,0}{0.5374} & \textcolor[rgb]{0,0,1}{10.9584} & 3.6732 & 1.5170 &  15.6827 & 0.7925 & \textcolor[rgb]{0,0,1}{0.1335} & 0.9461 & 1.0842 & 19.1310 & 4.1340 & 2.0839\\

 				GridNet &  \textcolor[rgb]{0,0,1}{28.2466} & \textcolor[rgb]{1,0,0}{0.9774} & \textcolor[rgb]{1,0,0}{0.0113} & \textcolor[rgb]{1,0,0}{0.9773} & 0.8785 & 11.1510 & 3.4210 & 1.3322 &  14.2371 & 0.7761 & 0.1442 & 0.9462 & 1.3848 & 15.4152 & 4.0963 & 1.8436\\

 				GCANet &  24.1412 & 0.8926 & 0.0623 & 0.9709 & 0.6375 & 15.8146 & 3.5688 & 1.6159 &  15.5987 & \textcolor[rgb]{0,0,1}{0.8256} & \textcolor[rgb]{1,0,0}{0.1325} & 0.9460 & 0.8668 & \textcolor[rgb]{0,0,1}{14.5558} & \textcolor[rgb]{0,0,1}{3.7155} & 2.2867\\

 				DA-Dehaze &  27.4409 & 0.9557 & 0.0430 & 0.9761 & 0.6296 & 11.3146 & 3.3265 & 1.6264 &  \textcolor[rgb]{0,0,1}{15.7240} & 0.8234 & 0.1513 & \textcolor[rgb]{0,0,1}{0.9501} & 0.8533 & 24.6961 & 4.0871 & 2.3394\\

 				IDE &  17.4091 & 0.7853 & 0.0912 & 0.9635 & \textcolor[rgb]{0,0,1}{0.5900} & 13.4620 & \textcolor[rgb]{1,0,0}{3.0655} & \textcolor[rgb]{1,0,0}{2.3944} &  14.2400 & 0.8161 & 0.1445 & 0.9441 & \textcolor[rgb]{1,0,0}{0.5981} & 30.4148 & 4.3759 & \textcolor[rgb]{1,0,0}{3.2809} \\

 				HAAN &  \textcolor[rgb]{1,0,0}{29.3991} & \textcolor[rgb]{0,0,1}{0.9646} & \textcolor[rgb]{0,0,1}{0.0392} & \textcolor[rgb]{0,0,1}{0.9766} & 0.6179 & \textcolor[rgb]{1,0,0}{10.9396} & \textcolor[rgb]{0,0,1}{3.0948} & \textcolor[rgb]{0,0,1}{1.6809} &  \textcolor[rgb]{1,0,0}{16.0230} & \textcolor[rgb]{1,0,0}{0.8274} & 0.1413 & \textcolor[rgb]{1,0,0}{0.9529} & \textcolor[rgb]{0,0,1}{0.8215} & \textcolor[rgb]{1,0,0}{14.0798} & \textcolor[rgb]{1,0,0}{3.6820} & \textcolor[rgb]{0,0,1}{2.5805}\\

 				\hline
 		\end{tabular}}
 		\label{synthesis-tb}
 		\vspace{-0.65cm}
 	\end{center}
 \end{table*}

 The qualitative and quantitative results are shown in Fig. \ref{ablation_dehaze} and Table \ref{ablation-tb}, it demonstrates that HAAN achieves the best defogging performance in terms of  visual effect and quantitative accuracy. As can be seen in Fig. \ref{ablation_dehaze}(b), the CycleGAN's results have the worst defogging performance, which have the serious color distortion and the texture details are completely lost. Compared to CycleGAN, we can see from Fig. \ref{ablation_dehaze}(c) that the texture information of the Cycle-Dehaze's results are restored better due to the VGG perceptual loss can preserve more image content. However, as shown in the zoom-in regions in Fig. \ref{ablation_dehaze}(c), the color board (1st row) and the sign (2nd row) are blurred. Fig. \ref{ablation_dehaze}(d) shows the results obtained by using HAAN without VGG perceptual losses. We can see that there is a significant amount of fog in the defogged results and the sign is also blurred. In addition, without using the proposed DC module, there is a serious chessboard phenomenon in some regions (\textit{e.g.}, the color board and the sign in Fig. \ref{ablation_dehaze}(e)). It demonstrates that the proposed DC module could help the network preserve richer image content and texture details. As shown in Fig. \ref{ablation_dehaze}(f), the defogged results generated by our HAAN without ASM module have color distortion phenomenon. The reason is due to the atmospheric scattering model can guide the synthesizing network to generate a more natural foggy image to improve the mapping ability from foggy image domain to fogfree image domain.
 Finally, compared to the defogged results generated by our HAAN without HDA module (as shown in Fig. \ref{ablation_dehaze}(g)), our HAAN's results (as shown in Fig. \ref{ablation_dehaze}(h)) are clearer and more pleasing in visual effect. It demonstrates that HDA can fuse more contextual information from the layers to better recover the color-texture information of the defogged results. Moreover, the above analysis is also reflected in indicator NIQE. It can be seen from Fig. \ref{ablation_dehaze}, the NIQE values of our method are the smallest, followed by w/o VGG, w/o ASM, w/o DC, w/o HDA, Cycle-Dehaze \cite{cycle-dehazing} and  CycleGAN \cite{zhu2017unpaired}. This indicates that our method can achieve the best defogging performance. To further demonstrate the effectiveness of our HAAN, we quantitative compare on SOTS outdoor dataset for different configurations. As shown in Table \ref{ablation-tb}, our HAAN has the best values of PSNR and SSIM.

\subsection{Performance Comparison with State-of-the-arts}
 In this subsection, to verify the effectiveness of our proposed method, we qualitatively and quantitatively compared against the state-of-the-arts methods on both synthetic and real-world datasets.

\begin{figure*}[htbp]\footnotesize
	\begin{center}
		\vspace{-0.2cm}
		\resizebox{\textwidth}{!}{
			\begin{tabular}{@{}cccccccccc@{}}

				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/haze/12} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DehazeNet/12_DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DCPDN/12_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/EPDN/12_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GridDehazeNet/12-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GCANet/12_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DA/12_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/IDE/12_IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/Our/12_Our} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/gt/12} \\

				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/haze/24} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DehazeNet/24_DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DCPDN/24_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/EPDN/24_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GridDehazeNet/24-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GCANet/24_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DA/24_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/IDE/24_IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/Our/24_Our} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/gt/24} \\

				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/haze/35} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DehazeNet/35_DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DCPDN/35_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/EPDN/35_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GridDehazeNet/35-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GCANet/35_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DA/35_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/IDE/35_IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/Our/35_Our} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/gt/35} \\

				(a) & \hspace{-0.45cm}
				(b) & \hspace{-0.45cm}
				(c) & \hspace{-0.45cm}
				(d) & \hspace{-0.45cm}
				(e) & \hspace{-0.45cm}
				(f) & \hspace{-0.45cm}
				(g) & \hspace{-0.45cm}
				(h) & \hspace{-0.45cm}
				(i) & \hspace{-0.45cm}
				(j)
		\end{tabular}}
	\end{center}
	\vspace{-0.3cm}
	\caption{Qualitative comparison between the proposed HAAN and the state-of-the-art methods on O-HAZE dataset. (a) Foggy Image. (b) DehazeNet \cite{dehazenet2016TIP}. (c) DCPDN \cite{zhang2018densely}. (d) EPDN \cite{pix2pixdehazing}. (e) GridNet \cite{liu2019griddehazenet}. (f) GCANet \cite{chen2019gated}. (g) DA-Dehaze \cite{shao2020domain}. (h) IDE \cite{IDE2021ide}. (i) HAAN. (j) Ground truth.}
	\vspace{-0.45cm}
	\label{real-world1_dehaze}
\end{figure*}
\begin{figure*}[htbp]\footnotesize
	\begin{center}
		\vspace{0.1cm}
		\resizebox{\textwidth}{!}{
			\begin{tabular}{@{}ccccccccc@{}}

				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/haze/4} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DehazeNet/4-DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DCPDN/4_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/EPDN/4_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GridDehazeNet/4-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GCANet/4_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DA/4_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/IDE/4-IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/Our/4_Our} \\

				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/haze/13} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/DehazeNet/13-DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/DCPDN/13_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/EPDN/13_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/GridDehazeNet/13-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/GCANet/13_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/DA/13_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/IDE/13-IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.075\textheight]{img1/real_compare/Our/13_Our}\\

				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/haze/279} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DehazeNet/279_DehazeNet} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DCPDN/279_DCPCN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/EPDN/279_EPDN} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GridDehazeNet/279-Grid} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/GCANet/279_GCA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/DA/279_DA} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/IDE/279_IDE} & \hspace{-0.45cm}
				\includegraphics[width = 0.09\textwidth,height=0.05\textheight]{img1/real_compare/Our/279_Our}\\

				(a) & \hspace{-0.45cm}
				(b) & \hspace{-0.45cm}
				(c) & \hspace{-0.45cm}
				(d) & \hspace{-0.45cm}
				(e) & \hspace{-0.45cm}
				(f) & \hspace{-0.45cm}
				(g) & \hspace{-0.45cm}
				(h) & \hspace{-0.45cm}
				(i)
		\end{tabular}}
	\end{center}
	\vspace{-0.3cm}
	\caption{Qualitative comparison between the proposed HAAN and the state-of-the-art methods on real-world images. (a) Foggy Image. (b) DehazeNet \cite{dehazenet2016TIP}. (c) DCPDN \cite{zhang2018densely}. (d) EPDN \cite{pix2pixdehazing}. (e) GridNet \cite{liu2019griddehazenet}. (f) GCANet \cite{chen2019gated}. (g) DA-Dehaze \cite{shao2020domain}. (h) IDE \cite{IDE2021ide}. (i) HAAN.}
	\vspace{-0.45cm}
	\label{real-world2_dehaze}
\end{figure*}
\begin{table*}[ht]
	\begin{center}

		\captionsetup{justification=centering}
		\caption{\\A{\footnotesize VERAGE} PSNR, SSIM, LPIPS, RI, FADE, BRISQUE, NIQE, $\overline{\gamma}$ {\footnotesize OF} D{\footnotesize EFOGGED} R{\footnotesize ESULTS} {\footnotesize ON} O-HAZE/LIVE D{\footnotesize ATASETS}.\\T{\footnotesize HE} T{\footnotesize OP} T{\footnotesize WO} P{\footnotesize ERFORMANCE} V{\footnotesize ALUES} A{\footnotesize RE} H{\footnotesize IGHLIGHTED} {\footnotesize IN} R{\footnotesize ED} {\footnotesize AND} B{\footnotesize LUE}}
		\resizebox{\textwidth}{!}{
			\begin{tabular}{c|cccccccc|cccccccc}
				\hline
				Dataset & \multicolumn{8}{c|}{O-HAZE} & \multicolumn{8}{c}{LIVE}\\

				\hline
				\diagbox{Method}{Metric} & PSNR & SSIM & LPIPS & RI & FADE & BRISQUE & NIQE & $\overline{\gamma}$ & PSNR & SSIM & LPIPS & RI & FADE & BRISQUE & NIQE & $\overline{\gamma}$ \\

				\hline
				DehazeNet & 18.5655  & 0.7304 & 0.2566 & 0.9384 & 0.8119 & \textcolor[rgb]{1,0,0}{11.6304} & 2.6174 & 1.2240 &  - & - & - & - & 0.7587 & 16.8184 & 4.1288 & 1.0860\\

				DCPDN &  18.6372 & 0.7268 & 0.2754 & 0.9312 & 0.7051 & 21.1391 & 3.4793 & 1.4379 &  - & - & - & - & 1.1490 & 19.3902 & 4.3371 & 1.0150\\

				EPDN &  19.0154 & 0.7822 & \textcolor[rgb]{0,0,1}{0.1989} & \textcolor{blue}{0.9487} & 0.3505 & 18.1324 & 3.1017 &  1.7658  &  - & - & - & - & 0.6107 & \textcolor[rgb]{0,0,1}{9.7798} & 4.1584 & 1.4640\\

				GridNet &  \textcolor{blue}{19.6539} & 0.7348 & 0.2461 & 0.9361 & 0.7829 & 15.9378 & 2.8308 &  1.1424 &  - & - & - & - & 1.1124 & 12.7015 & 3.8707 & 1.1857\\

				GCANet &  17.9407 & 0.7443 & \textcolor[rgb]{1,0,0}{0.1917} & 0.9479 & \textcolor[rgb]{0,0,1}{0.2714}  & 21.7128 & 2.8813 & 1.9534 &  - & - & - & - & 0.6791 & 10.5648 & 3.8063 & 1.4816\\

				DA-Dehaze &  19.5841 & 0.7279 & 0.2505 & 0.9465 & 0.3527 & 15.9735 & 2.7854 & 2.1407 &  - & - & - & - & 0.6173 & 12.6701 & 3.8717 &  1.3834\\

				IDE &  17.2068 & \textcolor[rgb]{0,0,1}{0.7997} & 0.2446 & 0.9353 & \textcolor[rgb]{1,0,0}{0.2760} & 14.6594 & \textcolor[rgb]{1,0,0}{2.3629} & \textcolor[rgb]{1,0,0}{3.6548} &  - & - & - & - & \textcolor[rgb]{0,0,1}{0.5632} & 21.4205 & \textcolor[rgb]{1,0,0}{3.7041} & \textcolor[rgb]{1,0,0}{2.5919}\\

				HAAN &  \textcolor[rgb]{1,0,0}{20.2749} & \textcolor[rgb]{1,0,0}{0.8009} & 0.2241 & \textcolor{red}{0.9499} & 0.3270 & \textcolor[rgb]{0,0,1}{14.1695} & \textcolor[rgb]{0,0,1}{2.5366} & \textcolor[rgb]{0,0,1}{2.3397}  &  - & - & - & - & \textcolor[rgb]{1,0,0}{0.5156} & \textcolor[rgb]{1,0,0}{9.2760} & \textcolor[rgb]{0,0,1}{3.7943} & \textcolor[rgb]{0,0,1}{1.5796}\\

				\hline
		\end{tabular}}
		\label{real-tb}
		\vspace{-0.65cm}
	\end{center}
\end{table*}

1) \textbf{\textit{Comparison on synthesized foggy images}}

 Fig. \ref{synthesis1_dehaze} illustrates the defogged results on synthesized data by our method and other seven methods. As can be seen in Fig. \ref{synthesis1_dehaze} (c)-(d), although DCPDN \cite{zhang2018densely} and EPDN \cite{pix2pixdehazing} are able to remove fog from the input foggy images, some results appeared dark areas that make the texture information of the object invisible. as shown in the bottom row of Figs. \ref{synthesis1_dehaze}(f) and \ref{synthesis1_dehaze}(h), the defogged results of GCANet \cite{chen2019gated} and IDE \cite{IDE2021ide} have the color distorted phenomenon in the sky region. In contrast, our method does not have the above issues (as shown in Fig. \ref{synthesis1_dehaze}(i)). To further demonstrate the effectiveness of our HAAN, we compare our method with other methods on different fog concentration images from HazeRD dataset. We can observe that the defogged results of DCPDN \cite{zhang2018densely} and GridNet \cite{liu2019griddehazenet} (as shown in the first row of Figs. \ref{synthesis2_dehaze}(c) and \ref{synthesis2_dehaze}(e)) have completely lost most of the structural information when processing images with dense fog. For DehazeNet \cite{dehazenet2016TIP} as well as DA-Dehaze  \cite{shao2020domain}, attractive results can be produced when they recover the mist images. However, in the dense fog environment, these methods are fail that there is a significant amount of fog in the defogged result (as shown in Figs.\ref{synthesis2_dehaze}(b) and \ref{synthesis2_dehaze}(g)). The defogged results generated by IDE \cite{IDE2021ide} (as shown in \ref{synthesis2_dehaze}(h)) appear severely discolored in the sky region. In comparison, our model can remain sharper edge contours and the results (as shown in Fig. \ref{synthesis2_dehaze}(i)) are closer to the ground truth images visually.

 To quantitatively compare the recovery quality of the different techniques, Table \ref{synthesis-tb} summarizes the mean value of the eight metrics on the outdoor datasets of SOTS and HazeRD. Our method achieves excellent values for both reference and non-reference evaluations. Especially, on the HazeRD dataset, the proposed method outperforms other methods in terms of PSNR, SSIM, RI, BRISQUE and NIQE. It demonstrates that compared with other methods, our method performs better for dealing with dense fog images.

2)  \textbf{\textit{Comparison on real-world foggy images}}

 We evaluate our method on real foggy images, which are provided by the O-HAZE, LIVE and previous methods. Fig. \ref{real-world1_dehaze} and Fig. \ref{real-world2_dehaze} show the defogged results of qualitative comparison between our method and the other seven methods. As can be seen in Figs. \ref{real-world1_dehaze}(b)-\ref{real-world2_dehaze}(b), although DehazeNet \cite{dehazenet2016TIP} is able to remove fog from misty images, the defogged results still have some remaining fog artifacts. As shown in Figs. \ref{real-world1_dehaze}(c)-\ref{real-world2_dehaze}(c), most of the fog is unremoved in the DCPDN’s results, and some results appeared serious color distortion (such as the ground in the second row of Fig. \ref{real-world1_dehaze}(c) and the hills in the first row of Fig. \ref{real-world2_dehaze}(c)).
 The results of EPDN \cite{pix2pixdehazing} and GCANet \cite{chen2019gated} (as shown in Figs. \ref{real-world1_dehaze}(d)-\ref{real-world2_dehaze}(d) and Figs. \ref{real-world1_dehaze}(f)-\ref{real-world2_dehaze}(f)) have no remaining fog, however, some defogged images significantly suffer from over-enhancement. As can be seen in Figs. \ref{real-world1_dehaze}(e)-\ref{real-world2_dehaze}(e) and Figs. \ref{real-world1_dehaze}(g)-\ref{real-world2_dehaze}(g), there is some fog remain in the defogged results of GridNet \cite{liu2019griddehazenet} and DA-Dehaze \cite{shao2020domain}. Similar to the defogged results from synthetic images, as shown in the first row of Fig. \ref{real-world1_dehaze}(h),  the color distortion phenomenon still exists in the results by using IDE \cite{IDE2021ide} to process the real-world foggy images. In contrast, the results of our method HAAN (as shown in Figs. \ref{real-world1_dehaze}(i)-\ref{real-world2_dehaze}(i)) are more natural and clearer than other methods'. The reason is that our method uses unpaired real-world foggy dataset for training, and the proposed attention mechanism network can pay more attention to fog relevant features to preserve richer image content and texture details for defogged results. We performed the quantitative comparison of defogged results on the two real-world datasets of O-HAZE and LIVE. As shown in Table \ref{real-tb}, it is clear that HAAN outperforms state-of-the-art techniques in terms of PSNR, SSIM and RI, which proves our method has quite competitive in single image defogging. Our method has higher $\overline{\gamma}$ than others, which indicates that our method well restores texture details. Meanwhile, the BRISQUE scores and NIQE values of our results are lower than other methods, which means that our defogged results have higher fidelity. Moreover, our method has a smaller FADE than others, which demonstrates that our defogged results are clearer.

\section{Concluding Remarks}

In this work, we presented HAAN to exploit the self-similarity of multi-foggy relevant features and the complementary texture/structure information to improve the quality of defogged images based on a cyclic generative adversarial framework. We devised three basic modules (FRM, FSM, and CTRM) to constrain each other to generate high-quality images from the fog-fogfree to fogfree-fog domain by using four mapping paths. We also explored several derived images of foggy inputs to guide our channel-spatial attention to cooperatively retrieve more foggy related contextual information to enhance details in defogged images. Finally, we used a novel sky segmentation network to optimize the atmospheric light to guide the FSM to generate more natural foggy images to improve the transformation mapping ability from foggy to fog-free images by FRM. Extensive qualitative and quantitative comparisons demonstrate that HAAN delivers superior defogging performance over the state of the art.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

{\small
	\bibliographystyle{ieee}

Computer Vision and Pattern Recognition (cs.CV)