Recently, many deep-learning-based pan-sharpening methods have been proposed for generating high-quality pan-sharpened (PS) satellite images. These methods focused on various types of convolutional neural network (CNN) structures, which were trained by simply minimizing L1 or L2 losses between network outputs and the corresponding high-resolution multi-spectral (MS) target images. However, due to different sensor characteristics and acquisition times, high-resolution panchromatic (PAN) and low-resolution MS image pairs tend to have large pixel misalignments, especially for moving objects in the images. Conventional CNNs trained with L1 or L2 losses with these satellite image datasets often produce PS images of low visual quality including double-edge artifacts along strong edges and ghosting artifacts on moving objects. In this letter, we propose a novel loss function, called a spectral-spatial structure (S3) loss, based on the correlation maps between MS targets and PAN inputs. Our proposed S3 loss can be very effectively utilized for pan-sharpening with various types of CNN structures, resulting in significant visual improvements on PS images with suppressed artifacts.
Traditional pan-sharpening methods @cite_28 @cite_17 @cite_22 @cite_14 @cite_10 @cite_13 @cite_9 @cite_21 @cite_16 @cite_29 @cite_11 @cite_19 @cite_24 @cite_7 include component substitution @cite_28 @cite_17 @cite_22 @cite_14 @cite_10 , multiresolution analysis @cite_13 @cite_9 and machine-learning @cite_21 @cite_16 @cite_29 . Comparisons for component substitution and multiresolution analysis based approaches were presented thoroughly in @cite_11 . Component substitution based methods often incorporated the Brovey transform @cite_28 , the intensity-hue-saturation @cite_17 , principal component analysis (PCA) @cite_22 , or matting models @cite_14 for pan-sharpening. In multiresolution analysis based methods, the spatial structures of PAN images are decomposed using wavelet @cite_13 or undecimated wavelet @cite_9 decomposition techniques, and are fused with up-sampled MS images to produce PS images. These methods have relatively low computation complexity but tend to produce PS images with mismatched spectral information. Machine-learning based methods @cite_21 @cite_16 @cite_29 learn pan-sharpening models by optimizing a loss function of inputs and targets with some regularization terms.
@cite_13 @cite_14 @cite_11 @cite_22 @cite_7 @cite_28 @cite_9 @cite_21 @cite_29 @cite_24 @cite_19 @cite_16 @cite_10 @cite_17
2144436897 2394774286 2017121865 2717881181
High correlation among the neighboring pixels both spatially and spectrally in a multispectral image makes it necessary to use an efficient data transformation approach before performing pan-sharpening. Wavelets and principal component analysis (PCA) methods have been a popular choice for spatial and spectral transformations, respectively. Current PCA-based pan-sharpening methods make an assumption that the first principal component (PC) of high variance is an ideal choice for replacing or injecting it with high spatial details from the high-resolution histogram-matched panchromatic (PAN) image. This paper presents a combined adaptive PCA-contourlet approach for pan-sharpening, where the adaptive PCA is used to reduce the spectral distortion and the use of nonsubsampled contourlets for spatial transformation in pan-sharpening is incorporated to overcome the limitation of the wavelets in representing the directional information efficiently and capturing intrinsic geometrical structures of the objects. The efficiency of the presented method is tested by performing pan-sharpening of the high-resolution (IKONOS and QuickBird) and the medium-resolution (Landsat-7 Enhanced Thematic Mapper Plus) datasets. The evaluation of the pan-sharpened images using global validation indexes reveal that the adaptive PCA approach helps reducing the spectral distortion, and its merger with contourlets provides better fusion results. The development of multisensor systems in recent years has led to great increase in the amount of available remote sensing data. Image fusion techniques aim at inferring high quality images of a given area from degraded versions of the same area obtained by multiple sensors. This paper focuses on pansharpening, which is the inference of a high spatial resolution multispectral image from two degraded versions with complementary spectral and spatial resolution characteristics: 1) a low spatial resolution multispectral image and 2) a high spatial resolution panchromatic image. We introduce a new variational model based on spatial and spectral sparsity priors for the fusion. In the spectral domain, we encourage low-rank structure, whereas in the spatial domain, we promote sparsity on the local differences. Given the fact that both panchromatic and multispectral images are integrations of the underlying continuous spectra using different channel responses, we propose to exploit appropriate regularizations based on both spatial and spectral links between panchromatic and fused multispectral images. A weighted version of the vector total variation norm of the data matrix is employed to align the spatial information of the fused image with that of the panchromatic image. With regard to spectral information, two different types of regularization are proposed to promote a soft constraint on the linear dependence between the panchromatic and fused multispectral images. The first one estimates directly the linear coefficients from the observed panchromatic and low-resolution multispectral images by linear regression while the second one employs the principal component pursuit to obtain a robust recovery of the underlying low-rank structure. We also show that the two regularizers are strongly related. The basic idea of both regularizers is that the fused image should have low-rank and preserve edge locations. We use a variation of the recently proposed split augmented Lagrangian shrinkage algorithm to effectively solve the proposed variational formulations. Experimental results on simulated and real remote sensing images show the effectiveness of the proposed pansharpening method compared with the state-of-the-art. Pansharpening aims at creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multispectral (MS) image. Component substitution is the most widely used pansharpening method. However, most research in this field focuses on improving the existing component substitution-based pansharpening methods, e.g., principal component substitution and intensity hue saturation transform. The major contribution of this paper is a novel component substitution framework based on an image matting model. The matting model refers to an MS image that can be decomposed into three components, i.e., alpha channel, spectral foreground, and background. Through substituting the alpha channel of the MS image with the PAN image, the high-resolution MS image is able to be reconstructed perfectly. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pansharpening methods in terms of subjective and objective evaluation. Pan-sharpening has become an important tool in remote sensing, which normally aims at fusing a multi-spectral image with high spectral resolution and a panchromatic image with high spatial resolution. However, some problems, such as spectral distortion, are facing pan-sharpening methods. Inspired by the applications of convolutional neural network (CNN) in many areas, we adopt an effective CNN model to fulfill pan-sharpening. In our method, only the sparse residuals between the interpolated MS and the pan-sharpened image are learned, which achieves fast convergence and high pan-sharpening quality. The experimental results on real-world data validate the effectiveness of the method.