Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spectral mode of HrHS image into consideration. Then we design an iterative algorithm to solve the model by exploiting the proximal gradient method. And then, by unfolding the designed algorithm, we construct a deep network, called MS HS Fusion Net, with learning the proximal operators and model parameters by convolutional neural networks. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.
Recently, a number of DL-based pansharpening methods were proposed by exploiting different network structures @cite_10 @cite_16 @cite_45 @cite_11 @cite_49 @cite_8 @cite_29 . These methods can be easily adapted to MS HS fusion problem. For example, very recently, @cite_55 proposed a 3D-CNN based MS HS fusion method by using PCA to reduce the computational cost. This method is usually trained with prepared training data. The network inputs are set as the combination of HrMS panchromatic images and LrHS multispectral images (which is usually interpolated to the same spatial size as HrMS panchromatic images in advance), and the outputs are the corresponding HrHS images. The current DL-based methods have been verified to be able to attain good performance. They, however, just employ networks assembled with some off-the-shelf components in current deep learning toolkits, which are not specifically designed against the investigated problem. Thus the main drawback of this technique is the lack of interpretability to this particular MS HS fusion task. In specific, both the intrinsic observation model ), ) and the evident prior structures, like the spectral correlation property, possessed by HS images have been neglected by such kinds of black-box" deep model.
@cite_8 @cite_29 @cite_55 @cite_45 @cite_49 @cite_16 @cite_10 @cite_11
2792365373 2700882095 2717881181 2462592242 2154789478 2619662254
Remote sensing images with different spatial and spectral resolution, such as panchromatic (PAN) images and multispectral (MS) images, can be captured by many earth-observing satellites. Normally, PAN images possess high spatial resolution but low spectral resolution, while MS images have high spectral resolution with low spatial resolution. In order to integrate spatial and spectral information contained in the PAN and MS images, image fusion techniques are commonly adopted to generate remote sensing images at both high spatial and spectral resolution. In this study, based on the deep convolutional neural network, a remote sensing image fusion method that can adequately extract spectral and spatial features from source images is proposed. The major innovation of this study is that the proposed fusion method contains a two branches network with the deeper structure which can capture salient features of the MS and PAN images separately. Besides, the residual learning is adopted in our network to thoroughly study the relationship between the high- and low-resolution MS images. The proposed method mainly consists of two procedures. First, spatial and spectral features are respectively extracted from the MS and PAN images by convolutional layers with different depth. Second, the feature fusion procedure utilizes the extracted features from the former step to yield fused images. By evaluating the performance on the QuickBird and Gaofen-1 images, our proposed method provides better results compared with other classical methods. We proposed a deep convolutional network for multi-spectral image pan-sharpening to overcome the drawbacks of traditional methods and improve the fusion accuracy. To break the performance limitation of deep networks, residual learning with specific adaption to image fusion tasks is applied to optimize the architecture of proposed network. Results of adequate experiments support that our model can yield high resolution multi-spectral images with state-of-the-art qualities, as the information in both spatial and spectral domains has been accurately preserved. Pan-sharpening has become an important tool in remote sensing, which normally aims at fusing a multi-spectral image with high spectral resolution and a panchromatic image with high spatial resolution. However, some problems, such as spectral distortion, are facing pan-sharpening methods. Inspired by the applications of convolutional neural network (CNN) in many areas, we adopt an effective CNN model to fulfill pan-sharpening. In our method, only the sparse residuals between the interpolated MS and the pan-sharpened image are learned, which achieves fast convergence and high pan-sharpening quality. The experimental results on real-world data validate the effectiveness of the method. A new pansharpening method is proposed, based on convolutional neural networks. We adapt a simple and effective three-layer architecture recently proposed for super-resolution to the pansharpening problem. Moreover, to improve performance without increasing complexity, we augment the input by including several maps of nonlinear radiometric indices typical of remote sensing. Experiments on three representative datasets show the proposed method to provide very promising results, largely competitive with the current state of the art in terms of both full-reference and no-reference metrics, and also at a visual inspection. A deep neural network (DNN)-based new pan-sharpening method for the remote sensing image fusion problem is proposed in this letter. Research on representation learning suggests that the DNN can effectively model complex relationships between variables via the composition of several levels of nonlinearity. Inspired by this observation, a modified sparse denoising autoencoder (MSDA) algorithm is proposed to train the relationship between high-resolution (HR) and low-resolution (LR) image patches, which can be represented by the DNN. The HR LR image patches only sample from the HR LR panchromatic (PAN) images at hand, respectively, without requiring other training images. By connecting a series of MSDAs, we obtain a stacked MSDA (S-MSDA), which can effectively pretrain the DNN. Moreover, in order to better train the DNN, the entire DNN is again trained by a back-propagation algorithm after pretraining. Finally, assuming that the relationship between HR LR multispectral (MS) image patches is the same as that between HR LR PAN image patches, the HR MS image will be reconstructed from the observed LR MS image using the trained DNN. Comparative experimental results with several quality assessment indexes show that the proposed method outperforms other pan-sharpening methods in terms of visual perception and numerical measures. In the field of multispectral (MS) and panchromatic image fusion (pansharpening), the impressive effectiveness of deep neural networks has recently been employed to overcome the drawbacks of the traditional linear models and boost the fusion accuracy. However, the existing methods are mainly based on simple and flat networks with relatively shallow architectures, which severely limits their performance. In this letter, the concept of residual learning is introduced to form a very deep convolutional neural network to make the full use of the high nonlinearity of the deep learning models. Through both quantitative and visual assessments on a large number of high-quality MS images from various sources, it is confirmed that the proposed model is superior to all the mainstream algorithms included in the comparison, and achieves the highest spatial–spectral unified accuracy.