Dual Perceptual Loss for Single Image Super-Resolution Using ESRGAN

The proposal of perceptual loss solves the problem that per-pixel difference loss function causes the reconstructed image to be overly-smooth, which acquires a significant progress in the field of single image super-resolution reconstruction. Furthermore, the generative adversarial networks (GAN) is applied to the super-resolution field, which effectively improves the visual quality of the reconstructed image. However, under the condtion of high upscaling factors, the excessive abnormal reasoning of the network produces some distorted structures, so that there is a certain deviation between the reconstructed image and the ground-truth image. In order to fundamentally improve the quality of reconstructed images, this paper proposes a effective method called Dual Perceptual Loss (DP Loss), which is used to replace the original perceptual loss to solve the problem of single image super-resolution reconstruction. Due to the complementary property between the VGG features and the ResNet features, the proposed DP Loss considers the advantages of learning two features simultaneously, which significantly improves the reconstruction effect of images. The qualitative and quantitative analysis on benchmark datasets demonstrates the superiority of our proposed method over state-of-the-art super-resolution methods.


Introduction
A low-resolution (LR) image is reconstructed as a high-resolution (HR) image, which is called single image super-resolution (SISR) reconstruction. It is an extremely challenging task. Under the condition of high upscaling factors, the lack of LR information leads to a series of uncertain factors, which will make the reconstructed image have a certain deviation from the ground-truth (GT) image. The main challenge of this paper is to make the reconstructed image as close as possible to the GT image, which is of great significance to the fields such as medicine [1,2], remote sensing [3], monitoring [4] and so on.
As the field of super-resolution (SR) reconstruction attracted more and more attention, many excellent algorithms have been proposed. According to the existing studies, the algorithms can be divided into three categories: interpolation-based methods, model-based methods and learning-based methods. The interpolation-based methods (e.g., bicubic interpolation [5] and the nearest neighbor interpolation [6]) have low complexity and high efficiency. However, since the expanded pixels are calculated by the neighborhood pixel, these methods don't make any reasoning for the texture details, which makes the image smooth on the whole. The model-based method can generate high-quality images efficiently, but this method is dependent on prior images relatively. When there is a certain deviation between the image to be reconstructed and the prior image, the visual quality of the reconstructed image will decrease rapidly. However, most SR methods are based on the learning, including neighbor embedding [7], sparse coding [8][9][10][11] and random forests [12]. With the popularization and development of deep neural network, the field of SR reconstruction has made great progress. Chao et al. [13] used deep neural network for the first time to solve the problem of SR reconstruction and proposed SRCNN, which attracted extensive attention. Later, various SR reconstruction methods based on deep learning were proposed. At that time, the main optimization goal was to minimize the mean square error (MSE) of GT images and reconstructed images [13,14] or maximize the peak signal-to-noise ratio (PSNR), which were also two metrics commonly used to evaluate and compare SR methods [15]. However, the ability of MSE (or PSNR) to capture perceptual-related differences (e.g., texture details) is very limited. To solve this problem, Johnson et al. [16] proposed perceptual loss, which extracted image features through pre-trainied network to predict the uncertainty factors under LR. Subsequently, considering the great success of GAN [17] in image generation, Ledig et al. [18] combined it with the perceptual-driven method, and made a further breakthrough in the clarity of the reconstructed image.
Although perceptual loss has made great progress in improving visual quality, there are still some obvious defects. One of them is that only relying on a single pre-trained network cannot mine potential features of images throughly, which limits the reasoning ability of the network. In order to solve this problem, based on the ResNet network [19], this paper proposes a perceptual loss called ResNet loss different from the VGG loss [16,18] in extraction way. Compared with the network structure of VGG [20], the ResNet network is composed of a large number of residual blocks [19], and it can retain more feature information and will not be lost with the increase of network layers. Some features extracted by the two distinct networks will exist in a complementary form, so this paper adopts the way of combining VGG loss with ResNet loss to improve the information acquisition ability of the overall perceptual features.
In the process of optimizing VGG loss and ResNet loss simultaneously, we expect that them can both provide strong support for the network. However, the two losses are generated in different ways, so their magnitudes will be different. If they are added directly, the difference in magnitude will cause the network to be biased towards learning the perceptual features extracted by a single pre-trained network. As a result, the advantage of dual perceptual cannot be fully utilized. The use of static constants for weighting and combination to balance the magnitude also has some defects. It can only control the initial state of the two losses, but cannot ensure the two losses to continue to be at the same magnitude during the training process. To cope with the problem mentioned above, this paper proposes a dynamic weighting method. It takes the VGG loss as the reference target and weights the ResNet loss dynamically, which keeps the relative size of two losses stable. This method eliminates the influence of magnitude, and enables the network to learn both ResNet features and VGG features with a strong degree of attention. Therefore, we define the combination of the VGG loss and the ResNet loss with dynamically weighted as a dual perceptual loss (DP Loss). In addition, DP Loss has a certain degree of flexibility by using the dynamic weight, hence the DP Loss can be applied to different models conveniently.
The hyperparameter setting of DP Loss has a great influence on the performance of the model. Therefore, we first apply DP Loss to SRGAN [18] to get SRGAN with Dual Perceptual Loss (SRGAN-DP), and test the influence of different hyperparameter combinations on the model to obtain the optimal hyperparameter combination. According to the experimental results, the SRGAN-DP under the optimal condition is better than SRGAN in all the evaluation metrics. Then, we apply the DP Loss under the optimal hyperparameter combination to ESRGAN and get ESRGAN with Dual Perceptual Loss (ESRGAN-DP). Finally, the experimental results show that the ESRGAN-DP has better visual effect and evaluation metrics compared with SRGAN-DP.
The SR methods based on both perceptual-driven and GAN can acquire better visual effect. Fig. 1 shows the images reconstructed by ESRGAN-DP and those reconstructed by several SR methods mentioned above (SRGAN [18], EnhanceNet [21], ESRGAN [22], SFT-GAN [23], ESRGAN+ [24]) under the condition of a 4× upscaling factor. As shown in Fig.  1, it is obviously that the lines on the surface of buildings are clearer and the reconstructed images have more realistic details after the DP Loss is applied to ESRGAN. The experimental results show that the performance of proposed method outperforms to other SR methods. Fig. 2 shows the comparison of DP Loss and the state-of-the-art SR methods in terms of LPIPS [25] values on the Urban100. The smaller the value, the higher the perceptual similarity of the reconstructed image and the GT image. It can be seen from the line chart that the LPIPS value of ESRGAN-DP is the smallest among all SR methods. Therefore, it is concluded that the reconstruction effect of the image is effectively improved by adding DP Loss into the ESRGAN.
The main contributions of this paper are itemized as follows: 1. Following the VGG loss, we propose the ResNet loss that is generated by the perceptual features extracted by the pre-trained ResNet network. Compared with VGG, the ResNet network does not lose the original information with the increase of layers, thus, a deeper level of output can be used to obtain the higher level of perceptual features. 2. We propose the idea that the generator learns VGG features and ResNet features simultaneously. According to the differences of VGG and ResNet networks in forward calculation, it can be inferred that the features extracted by two networks are complementary, therefore, the recover ability of generator to texture details can be further improved. 3. In this paper, the ResNet loss is dynamically weighted to eliminate the interference caused by the magnitude difference, so that both VGG loss and ResNet loss can bring greater benefits to the model. 4. Considering the influence of the hyperparameter combination in DP Loss on the reconstruction results, we conduct experiments on different benchmark datasets. By comparing and analyzing the experimental results, the loss function under the optimal hyperparameter combination is selected and applied to ESRGAN. Experiments show that compared with the traditional ESRGAN, the visual effects and evaluation metrics have been significantly improved, and compared with other state-of-the-art methods are also outstanding.
In Section 2, we discuss the related network models and the research progress of loss functions. In Section 3, we focus on the theoretical ideas related to DP Loss. In Section 4, we integrate DP loss into SRGAN for hyperparameter analysis, and apply the loss function under the optimal hyperparameter combination to ESRGAN to obtain ESRGAN-DP, and then compare ESRGAN-DP with other state-of-the-art SR methods to verify the effectiveness of the proposed method; Section 5 concludes this paper.  This section first introduces different network structures in the SR field, and then focuses on the improvement of related loss functions.
Since the pioneering SRCNN [13] was proposed, the use of deep learning to solve the SISR problem has attracted more and more attention. Compared with traditional methods, deep learning has obvious advantages. Not only does it have a great improvement in visual quality, but also has more diversity in optimization and improvement. Furthermore, more and more network architectures are used to solve the problem of SR reconstruction. Kim et al. [26] proposed the VDSR, which increased the number of network layers to 20 and significantly improved the reconstruction effect. Lim et al. [27] constructed the EDSR by using residual blocks that deleted unnecessary Batch Normalization (BN) layers [28]. Inspired by DenseNet [29], Zhang et al. [30] proposed Residual Dense Block (RDB) and applied it to SR, and further discussed the deeper network structure with channel attention mechanism [31]. Wang et al. [22] proposed the Residual in Residual Dense Block (RRDB) without BN layer in the generative network. In order to restore more faithful textures, Wang et al. [23] proposed SFTGAN, which used the prior category information for targeted generation. They constructed the Spatial Feature Transform (SFT) layer, and adjusted the features of the middle layer in a single network by using semantic segmentation maps [32][33][34]. In order to fundamentally reduce the computational cost, Song et al. [35] applied the AdderNets [36] to SR, and developed a learnable energy activation to adjust the feature distribution and refine the relevant details. Aiming at solving the problem of the uncertainty of image downsampling methods in real scenes and the large solution space from LR to HR, Guo et al. [37] proposed a dual regression scheme that can not only learn the mapping relationship from LR to HR, but also can estimate the down-sampling kernel.
Johnson et al. [16] believed that only focusing on optimizing the MSE or PSNR of the pixel space ratio of the GT image and the reconstructed image would make the reconstructed image smooth. Therefore, they proposed the perceptual loss, which was to improve the reconstruction effect by minimizing the feature space error between the GT image and the reconstructed image. After that, Ledig et al. [18] applied the perceptual loss and the GAN [17] to SR, and constituted the total loss of the generator by using the weighted combination of several loss components, including the adversarial loss, the content loss and the perceptual loss. The loss function proposed by Sajjadi et al. [21] consisted of four parts, namely the pixel-wise loss, the perceptual loss, the texture matching loss and the adversarial loss. Wang et al. [22] proposed ESRGAN based on the idea of SRGAN, which used a variety of techniques to further improve the texture details of the reconstructed image. In terms of perceptual loss, they proposed to use the output before the activation of the convolutional layer to obtain more feature information, so that the object to be minimized was the error of the feature space before activation. As a result, a convincing effect was obtained on the texture details of the reconstructed image and they won the PIRM2018-SR Challenge [38] champion. Rad et al. [39] proposed a targeted perceptual loss on the basis of the labels of object, background and boundary, which made the network reconstruct the image from multiple perspectives and improved the overall effect of the image. Therefore, discussing the perceptual loss is crucial to the improvement of the reconstruction results, especially the facticity of texture details. In order to reduce the unnatural artifacts generated in the perceptual-driven method, this paper designs a novel perceptual loss function to achieve this goal.

Methods
We solve the SR problem by training deep neural network. According to the theoretical idea proposed by Chao et al. [13], the optimization objective is as follows: where I LR i ∈ R C×H×W and I HR i ∈ R C×H×W represent the i-th LR and HR sub-image pairs in the training set, respectively. G I LR i ; θ represents the up-sampling network. θ is the parameter to be optimized within the neural network. L is the loss function.
The definition of L is crucial, and it is closely related to the reconstruction effect. According to the theoretical idea Ledig et al. [18] propose, we define several losses by comparing the differences between the reconstructed image and the GT image from different perspectives, and constitute the total loss through the weighted sum of every loss component. Here, L can be represented as: where l content is the content loss of the pixel-wise 1-norm distance between images reconstructed by the generator and GT images, l adversarial is the loss caused by the application of Relativistic Average GAN (RaGAN) [40], l DP is the DP Loss we propose. λ , η and γ are the coefficients of balancing different loss terms, respectively. In Section 3.2, the DP Loss proposed in this paper will be introduced in detail.

Adversarial network structure
In order to enable a large number of features to be better learned, in the network structure, we select ESRGAN [22] with strong learning ability. The model is composed of a generator network and a discriminator network. The two networks conduct the adversarial training through the alternate optimization way. The generator network is a residual network composed of a large number of residual blocks. In order to retain more feature information, the network does not contain the BN layer. We progressively enlarge the image by continuously adding up-sampling blocks with a 2× upscaling factor (if a 3× upscaling image is required, only a up-sampling block with a 3× upscaling factor is used). Each up-sampling block consists of three steps. Firstly, the input data is amplified by using the nearest neighbor interpolation [6] with a 2× upscaling factor, and then it passes through a 3×3 filter, and finally the output data of the filter is activated by LeakyReLU.
The input image size of the discriminator network is fixed at 128×128. The discrimination method uses the theoretical idea of RaD proposed in [40], and the output form is: where avg[·] represents the operation of taking the average for the discriminant value of all fake data in the mini-batch. According to Eq. (3), D Ra is asymmetric, and the optimization direction will be substantially changed only by changing the parameter order and discriminant target value. Therefore, regardless of whether the optimization is the discrimination loss or the generation loss, the target network in this way can benefit from real data and fake data rather than the part of the data.

VGG loss and ResNet loss
Ledig et al. [18] proposed to define the VGG loss according to the ReLU activation layer of the pre-trained 19 layer VGG network. In order to obtain more feature information, Wang et al. [22] redefined the VGG loss after the convolutional layer and before the activation layer. This paper uses the VGG loss defined in [22], that is, the L1 norm loss function is used to define the Manhattan Distance between the reconstructed image features and the GT image features: where Φ i, j represents features obtained by the j-th convolution (before activation) before the i-th maxpooling layer in the VGG network. C i, j , W i, j and H i, j are the dimensions of their respective feature spaces in the VGG network. Based on the ideas of Bruna et al. [41], Gatys et al. [42], Johnson et al. [16] and Ledig et al. [18], we define the ResNet loss on the ReLU activation layer of the pre-trained 50 layer ResNet network described in He et al. [19]. Since the ResNet network is different from the VGG network in structure, we use a specific block way to specify each feature space. As shown in Fig. 3, we divide ResNet-50 into four stages, each of which contains several bottleneck layers. The extracted perceptual features use the output value of the bottleneck layer at each stage, and the ResNet loss can also be expressed as: where β m,n represents features obtained by the n-th bottleneck layer (after activation) at the m-th stage. C m,n , W m,n and H m,n are the dimensions of their respective feature spaces in the ResNet network. In this paper, we adopt features of the last bottleneck layer at each stage to calculate ResNet loss, so there are four places to be tried in the 50 layer ResNet we use. But it is not necessarily that the features in deeper layers can bring better overall effect. The optimal feature map under specific conditions needs to be determined through experimental analysis of different situations.
We can compare the differences and similarities of the perceptual features extracted by the pre-trained VGG network and the pre-trained ResNet network from a visual perspective. In order to make the features of two networks have certain comparability, we need to keep the length, width, and channel number of the two features as consistent as possible. The depth of the two networks that the images pass through is also as consistent as possible. In this paper, we choose β 1,2 and Φ 3,3 , which pass through two max-pooling layers and seven convolutions and are obtained after the ReLU activation layer. The length, width and the number of channels are exactly the same. The specific effect is shown in Fig. 4. By visualizing the feature map under a single channel, it can be clearly seen that the perceptual ways of the two pre-trained networks are different. The features under Φ 3,3 are more distinct, but some information is lost (e.g., the features under the 64th channel are very sparse). Benefiting from the excellent properties of the residual network, more original data is retained under β 1,2 , especially the features under the 1st channel basically retains all single-channel pixel information, and the features under the 256th channel focuses on perceiving the edge of the image. This huge difference of perceptual ways is exactly what we need.
In order to illustrate the source of the above-mentioned differences in a deeper level, we carry out the analysis from the forward calculation of the neural network, because the most essential difference between the two networks lies in the organization way of the network layer output. Fig. 5 shows the basic connection way of the VGG network and ResNet network. It can be seen that VGG uses multiple network layers to connect in sequence, while ResNet adopts a special jump connection way. The different connection ways make them have the distinct ways of forward calculation, which limits the back propagation, namely the optimization of network parameters. Furthermore, since the parameters of two networks are trained in the task of image recognition, some unimportant features will be discarded after the input features pass through the convolutional layer. Therefore, after the images pass through the VGG network, some features will become more obvious while other fea-tures will become weaker or even disappear with the number of layers increases. However, ResNet has the ability to retain the most original features while emphasizing important features due to its special forward calculation mechanism. Therefore, the features extracted by two networks have certain complementary properties. Fig. 4(g) and Fig. 4(d) can fully illustrate these two points, respectively. Fig. 4(g) directly discards unimportant features (e.g., background) and highlights the most important feature information, while Fig. 4(d)

DP Loss
Obtaining more feature information from the image can enable the network to have better recovery capabilities in terms of faithful texture details. Therefore, we adopt a joint optimization strategy for the two perceptual losses mentioned in Section 3.2.1. However, the differences in magnitude between the two losses bring some interference to the network training, which makes the advantages of dual perceptual not be exhibited. The details are as follows.
Firstly, the loss function under the several perceptual losses is expressed as: where PL represents a single perceptual loss function and n represents the number of loss functions. Then, according to the gradient descent method, the network weight under the loss is updated, which can be expressed as: where w is the network weight, γ is the learning rate. It can be seen from the Eq. (7) that if there is a numerical difference in magnitude between PLs, there will also be such a difference in range of changes. Therefore, the gradient value of w is limited by the magnitude, which leads to the fact that only a part of PLs occupy the dominant position. As a result, the advantages of dual perceptual cannot be exhibited. We can describe this problem through the extent to which the loss is sensitive to changes of network parameters. For example, we update the weight w: at a certain stage of the training process, PL 1 = 1, PL 2 = 0.02. Let ∆ w>0, when w + ∆ w, then PL 1 = 1.1, PL 2 = 0.01; when w − ∆ w, then PL 1 = 0.9, PL 2 = 0.03. From above, it can be seen as follows: The total gradient contains ∂ PL 1 ∂ w and ∂ PL 2 ∂ w , so it is dominated by the gradient under PL 1 and its value is positive. However, in terms of the sensitivity of the loss, PL 2 is more sensitive to ∆ w (comparing with itselves, PL 1 fluctuates by 0.1 times but PL 2 fluctuates by 0.5 times). Therefore, the total gradient is negative, that is, the gradient under PL 2 is dominant, which is more beneficial to the model. In summary, the difference of magnitude affects the correlation between the sensitivity of the loss and the total gradient, so that the model cannot have a strong attention to both perceptual features. Even in the later stages of the training process, PL 1 has converged, but the network still does not make too many adjustments for PL 2 .
In order to resolve the problems above, we eliminate the influence of the difference of magnitude by weighting PLs. In general, the static constant is used as the weight value, but the PLs in the training process is uncontrollable, which cannot ensure that the two losses are always at the same magnitude. In this paper, we use the dynamic weighting to solve this problem, and further define DP Loss l DP . The specific formula is expressed as: where the ResNet loss l RES is dynamically weighted and the weight value is 1 µ ζ l V GG ,l RES , µ is a nonzero constant. The ζ l V GG ,l RES can be expressed as: where value l V GG +c l RES +c means to take the value of l V GG +c l RES +c and disconnect the relationship between the functions associated with the value. c is a tiny positive constant, so it can be negligible, and its role is just to keep the denominator from being zero. Therefore, 1 µ ζ l V GG ,l RES is just a value that changes with the change of the ratio of l V GG to l RES . Thus, 1 µ ζ l V GG ,l RES is taken as the weight value under the ResNet loss, which can only change the update range of network parameters, not the update direction.
From Eq. (8) and Eq. (9), it can be seen that the weighted ResNet loss value always exists in DP Loss in the form of multiple of VGG loss value, and µ is used to determine the multiple. In the process of updating network, when the VGG loss becomes smaller, the ResNet loss will be forced to decrease, and vice versa. The purpose of this is to ensure that the relative size of the loss value between the VGG loss and the ResNet loss is in a fixed state, and the network is always trained in this state. Therefore, it is prevented the situation that the network focuses on learning the perceptual features extracted by a single network due to the difference of magnitude between the perceptual losses during the training process, so that the advantages of dual perceptual can be fully utilized.
The method proposed in this paper only performs dynamic weights for the perceptual loss. The purpose of this is to ensure that DP Loss has a certain degree of flexibility and is convenient for migration to other models. The specific process of obtaining DP Loss is described by algorithm 1. In addition, it is necessary to discuss the effect of the different combinations of µ, Φ and β in DP Loss on the model, this occurs because the effects under various conditions are very different in both the visual quality of the image and the evaluation metric. In Section 4.3, we compare and analyze the hyperparameter combinations under different conditions to determine the optimal hyperparameter of DP Loss in the relevant model.

Algorithm 1: Process of obtaining DP Loss
Input: The reconstructed image I SR , the high-resolution image I HR , the µ, the i and j of Φ i, j , and the m and n of β m,n . Output: DP Loss l DP 1 Initialization: 2 Load the pre-trained parameters for VGG19 and ResNet50 networks; 3 Intercept the VGG19 network to obtain TruncationVGG(·) according to i and j; 4 Intercept the ResNet50 network to obtain TruncationResNet(·) according to m and n; 5 1)Obtain VGG loss: 6 Extract I SR 's VGG features P SR V GG ← TruncationVGG(I SR ); 7 Extract I HR 's VGG features P HR V GG ← TruncationVGG(I HR ); 8 Calculate the Manhattan Distance between P SR V GG and P HR V GG to obtain VGG loss l V GG ; 9 2)Obtain ResNet loss: 10 Extract I SR 's ResNet features P SR RES ← TruncationResNet(I SR ); 11 Extract I HR 's ResNet features P HR RES ← TruncationResNet(I HR ); 12 Calculate the Manhattan Distance between P SR RES and P HR RES to obtain ResNet loss l RES ; 13 3)Obtain DP Loss: 14 Calculate the ratio R l V GG ,l RES of VGG loss to ResNet loss, where R l V GG ,l RES = l V GG l RES ; 15 Break off the relationship between the functions associated with the R l V GG ,l RES to obtain ζ l V GG ,l RES ; 16 Calculate DP Loss l DP ← l V GG + 1 µ ζ l V GG ,l RES l RES ;

Datasets and evaluation metrics
The training set includes 800 high-definition images from the public dataset DIV2K [43]. By using sliding window to crop the images, 32,592 non-overlapping sub-images with the size of 480×480 are obtained. And then we perform bicubic interpolation operation on these images to get the corresponding downsampling images. Following ESRGAN, only a 4× upscaling factor is considered in the experiment. We take the widely used benchmark datasets as our test datasets. The dataset contains Set5 [44], Set14 [11], BSD100 [45] and Urban100 [46], which have 5 images, 14 images, 100 images and 100 images, respectively. The peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [47] and learned perceptual image patch similarity (LPIPS) [25] are evaluation metrics in our experiment. The lower the LPIPS value, the higher the perceptual similarity, that is, the reconstructed images are closer to GT images in visual quality. In order to make a fair comparison, a 4pixel wide stripe is removed from each border of all the evaluated images, and the Y-channel of the images is used to calculate PSNR and SSIM.

Training details
We use a GTX 2080Ti GPU and apply the pytorch [48] framework to train and test all models. The setting of experimental parameters follows [22]. Herein, the mini-batch size is set to 16, the spatial size of the cropped HR patch is 128×128 and parameters of the generator loss function in Eq. (2) are λ = 1e − 2, η = 5e − 3 and γ = 1. The model training needs 400K iterations in total. The initial learning rate is 1e − 2, and the learning rate is halved when the iteration reaches [50k, 100k, 200k, 300k]. The Adam [49] with β 1 = 0.9 and β 2 = 0.99 is used to alternately optimize the generator and the discriminator.

Hyperparameter analysis
According to Eq. (4), (5) and (8), it can be seen that there are three important hyperparameters need to be specified, which are the VGG features Φ, the ResNet features β and the constant µ. For the Φ, we refer to [18,22] and designate it as Φ 5,4 (before activation), while β and µ need to be determined according to the experiment. In this paper, we set the optional hyperparameters of β as β 1,3 , β 2,4 , β 3,6 and β 4,3 (after activation), and the optional hyperparameters of µ as 0.2, 0.5, 1, 5, 10 and 20. Since SRGAN and ESRGAN are similar in implementation ideas, it can be proved that there is a certain positive correlation in the improvement of the image reconstruction effect of the two models using different hyperparameter combinations (the correlation will be proved in Section 4.3.4). Therefore, in order to reduce the training burden, we use SRGAN to conduct comparative experiments under each hyperparameter combination, and apply the obtained optimal hyperparameter combination to ESRGAN. In order to improve the efficiency of finding the optimal hyperparameters, we obtain the optimal hyperparameter combination by alternately fixing a certain hyperparameter. In order to ensure that the results are fair and effective, the number of iterations is 400K during training.

Determination of hyperparameter µ
We fix β and set it to β 1,3 firstly, and then compare the experimental results under the different values of µ. The experimental results on the visual effect is shown in Fig. 6. When the µ is 0.5 or 1, the lines become clearer and the textures are closer to the GT image. Table  1 gives the comparison of results caused by losses under the different values of µ, and all evaluation metrics have the better performance after adding DP Loss under the appropriate value of µ. When µ is 1, the values of SSIM and PSNR in each dataset are the best, followed by the values of LPIPS; when µ is 0.5, the values of LPIPS are the best, followed by most of the values of SSIM and PSNR. In this paper, we pay more attention to the LPIPS, namely the perceptual similarity. Compared with SSIM and PSNR, LPIPS is more in line with human perceptual habits. Finally, the value of µ is determined as 0.5.

Determination of hyperparameter β
We use the optimal µ obtained in Section 4.3.1 as a fixed value, and then observe the influence on the image reconstruction when β is under the different values. Comparisons of visual effects and objective evaluation metrics are shown in Fig. 7 and Table 2, respectively. It can be seen that the result of DP Loss under the condition of β 3,6 occupies more advantages. From visual effects, the images reconstructed by using β 3,6 have less unnatural artifacts, so that the reconstructed images are closer to the GT images in structure. From evaluation metrics, it can be found that both PSNR and SSIM occupy the first or second best, and the LPIPS is the best in BSDS100 and Urban100 datasets. To sum up, the β 3,6 will be the optimal features.    The best performance is highlighted in red (1st best) and blue (2nd best).

SRGAN and ESRGAN under the DP Loss with the optimal hyperparameter
We use the two optimal hyperparameters in DP Loss obtained from Section 4.3.1 and Section 4.3.2 and apply them to SRGAN and ESRGAN respectively to obtain SRGAN-DP and ESRGAN-DP. The specific effects are given in Table 3. Compared with SRGAN-DP, ESRGAN-DP has a more superior effect in terms of LPIPS.

The positive correlation between SRGAN-DP and ESRGAN-DP
In order to prove whether there is a certain positive correlation between the effects of SRGAN-DP and ESRGAN-DP under different hyperparameter combinations, we select four combinations of representative hyperparameters from Sections 4.3.1 and 4.3.2, which are µ = ∞, µ = 1 + β 1,3 , µ = 10 + β 1,3 and µ = 0.5 + β 3,6 , respectively. In SRGAN-DP, the performance effects of the four hyperparameter combinations are: when µ = ∞, all evaluation metrics are the worst; when µ = 1 + β 1,3 , PSNR and SSIM are the best; when µ = 10 + β 1,3 , each metric is neither the best nor the worst; when µ = 0.5 + β 3,6 , LPIPS is the best. It can be seen from Table 4 that the effects shown in SRGAN-DP are also shown in ESRGAN-DP. Therefore, it can be concluded that the effects of SRGAN-DP and ESRGAN-DP under different hyperparameter combinations have a certain positive correlation. The best performance is highlighted in bold and the worst performance is highlighted in underline.

Comparison with the state-of-the-art technologies
To demonstrate the effectiveness of ESRGAN-DP, we compare it with the state-of-the-art single image SR methods in terms of evaluation metrics and visual quality, including En-hanceNet [21], SRGAN [18], ESRGAN [22], SFTGAN [23] DRN [37] and ESRGAN+ [24]. As can be seen from Table 5, compared with ESRGAN, ESRGAN-DP shows a better performance on SSIM, PSNR and LPIPS. Especially in the four test datasets, LPIPS decreases by 0.01305 on average compared with the ESRGAN, and it is the best among all the models, but SSIM and PSNR are not the best. However, LPIPS is more convincing, because it is more in line with human perceptual habits from a visual perspective. For example, the PSNR value and SSIM value of DRN are mostly the best in all models, but it does not mean that it has the best visual quality. From Fig. 8 and Fig. 9, too little reasoning about texture details causes the images reconstructed by DRN to still have a certain sense of blur.
We can make a more detailed observation from the fingernails and the edge of sleeves in Fig. 8 and the shape and shadow of bricks in Fig. 9. These images contain a lot of interactive lines, which contribute to the comparison of the reconstruction ability of SR methods in complex scenes. It is observed that the images reconstructed by Bicubic are extremely blurry in vision. The images reconstructed by EnhanceNet and SRGAN make an improvement in visual quality, but have serious artifacts and distortions. The artifacts of the images reconstructed by ESRGAN are reduced, but too many unreal textures are generated, which results in a great difference between reconstructed images and GT images. SFTGAN performs better than the above methods, but the reconstructed images are poor in the clarity of texture. The images reconstructed by DRN retain more original information, but are still slightly smooth on the whole, which have a certain influence on the visual quality. In terms of clarity of textures, ESRGAN+ has a certain improvement compared with other models, but it also generates too many unrealistic structures to make the images lack facticity. Our proposed method enhances the ability to reason missing information in LR to restore the facticity of the image as much as possible, which makes the reconstructed image more clear and closer to GT image from visual perspective.
To sum up, in terms of both evaluation metrics and visual effect, compared with the original perceptual loss only using a single VGG loss, the perceptual loss using DP Loss has superior advantages in solving SR problems.

Ablation study
In order to demonstrate the effectiveness of the proposed DP Loss and dynamic weighting, ESRGAN is used to as the baseline model to conduct ablation study. It can be seen from Table 6 that whether ResNet loss is applied to the original ESRGAN or the dynamic weighting is applied on the basis of ResNet loss, all evaluation metrics have been improved to a certain extent. Furthermore, from the perspective of SSIM, applying ResNet Loss to ESRGAN can significantly improve the value of SSIM. It shows that the ability of the network to recover the realistic structures of images is greatly improved due to the complementary property of VGG features and ResNet features. From the perspective of LPIPS, both two perceptual losses and dynamic weighting make the images have a great positive impact, which is closely related to the improvement of the visual quality of images. It can be seen from the Fig. 10 that the application of ResNet loss can eliminate a large number of unrealistic artifacts, and then further application of dynamic weighting can make the texture clearer and the structure more in line with GT images.

Conclusions
In this paper, we propose a DP Loss to reslove the problem that there are always structural distortions in the results reconstructed by SR methods under the traditional perceptualdriven. Both the features extracted by pre-trained ResNet network and those extracted by pre-trained VGG network are applied to the perceptual loss, which improves the information acquisition ability of the perceptual features from the perspective of feature extraction ways to enhance the ability of network to reason texture details. The strategy of using the dynamic weighting method for ResNet loss is to eliminate the interference of the magnitude difference on the training, which makes the advantages of dual perceptual more obvious. In addition, we compare the different influences of different hyperparameter combinations in the DP Loss on the results, and the quantitative and qualitative assessments obtained from four popular benchmark datasets both demonstrate the effectiveness of the method proposed in this paper.