Recently, convolutional neural networks (CNN) have been successfully applied to many remote sensing problems. However, deep learning techniques for multi-image super-resolution from multitemporal unregistered imagery have received little attention so far. This work proposes a novel CNN-based technique that exploits both spatial and temporal correlations to combine multiple images. This novel framework integrates the spatial registration task directly inside the CNN, and allows to exploit the representation learning capabilities of the network to enhance registration accuracy. The entire super-resolution process relies on a single CNN with three main stages: shared 2D convolutions to extract high-dimensional features from the input images; a subnetwork proposing registration filters derived from the high-dimensional feature representations; 3D convolutions for slow fusion of the features from multiple images. The whole network can be trained end-to-end to recover a single high resolution image from multiple unregistered low resolution images. The method presented in this paper is the winner of the PROBA-V super-resolution challenge issued by the European Space Agency.
Concerning MISR, the first work was proposed by Tsai and Huang @cite_26 , who used a frequency-domain technique to combine multiple under-sampled images with sub-pixel displacements to improve the spatial resolution of Landsat TM acquisitions. Due to the drawbacks of the frequency-domain algorithms, like the difficulty to incorporate the prior information about HR images, many spatial-domain MISR techniques were proposed over the years @cite_38 . Typical spatial-domain methods include non-uniform interpolation @cite_29 , iterative back-projection (IBP) @cite_1 , projection onto convex sets (POCS) @cite_41 @cite_61 , regularized methods @cite_4 @cite_67 @cite_21 , and sparse coding @cite_27 @cite_17 .
@cite_61 @cite_38 @cite_67 @cite_26 @cite_4 @cite_41 @cite_29 @cite_21 @cite_1 @cite_27 @cite_17
2614746365 1959447026 2907551576 2097769097
Most multispectral remote sensors (e.g. QuickBird, IKONOS, and Landsat 7 ETM+) provide low-spatial high-spectral resolution multispectral (MS) or high-spatial low-spectral resolution panchromatic (PAN) images, separately. In order to reconstruct a high-spatial high-spectral resolution multispectral image volume, either the information in MS and PAN images are fused (i.e. pansharpening) or super-resolution reconstruction (SRR) is used with only MS images captured on different dates. Existing methods do not utilize temporal information of MS and high spatial resolution of PAN images together to improve the resolution. In this paper, we propose a multiframe SRR algorithm using pansharpened MS images, taking advantage of both temporal and spatial information available in multispectral imagery, in order to exceed spatial resolution of given PAN images. We first apply pansharpening to a set of multispectral images and their corresponding PAN images captured on different dates. Then, we use the pansharpened multispectral images as input to the proposed wavelet-based multiframe SRR method to yield full volumetric SRR. The proposed SRR method is obtained by deriving the subband relations between multitemporal MS volumes. We demonstrate the results on Landsat 7 ETM+ images comparing our method to conventional techniques. Purpose To present and validate a new method that formalizes a direct link between k-space and wavelet domains to apply separate undersampling and reconstruction for high- and low-spatial-frequency k-space data. Theory and Methods High- and low-spatial-frequency regions are defined in k-space based on the separation of wavelet subbands, and the conventional compressed sensing problem is transformed into one of localized k-space estimation. To better exploit wavelet-domain sparsity, compressed sensing can be used for high-spatial-frequency regions, whereas parallel imaging can be used for low-spatial-frequency regions. Fourier undersampling is also customized to better accommodate each reconstruction method: random undersampling for compressed sensing and regular undersampling for parallel imaging. Results Examples using the proposed method demonstrate successful reconstruction of both low-spatial-frequency content and fine structures in high-resolution three-dimensional breast imaging with a net acceleration of 11–12. Conclusion The proposed method improves the reconstruction accuracy of high-spatial-frequency signal content and avoids incoherent artifacts in low-spatial-frequency regions. This new formulation also reduces the reconstruction time due to the smaller problem size. Magn Reson Med 70:1306–1318, 2013. © 2012 Wiley Periodicals, Inc. Deep learning (DL) has been successfully applied to single image super-resolution (SISR), which aims at reconstructing a high-resolution (HR) image from its low-resolution (LR) counterpart. Different from most current DL-based methods, which perform reconstruction in the spatial domain, we use a scheme based in the frequency domain to reconstruct the HR image at various frequency bands. Further, we propose a method that incorporates the wavelet transform (WT) and the recursive Res-Net. The WT is applied to the LR image to divide it into various frequency components. Then, an elaborately designed network with recursive residual blocks is used to predict high-frequency components. Finally, the reconstructed image is obtained via the inverse WT. This paper has three main contributions: 1) an SISR scheme based on the frequency domain is proposed under a DL framework to fully exploit the potential to depict images at different frequency bands; 2) recursive block and residual learning in global and local manners are adopted to ease the training of the deep network, and the batch normalization layer is removed to increase the flexibility of the network, save memory, and promote speed; and 3) the low-frequency wavelet component is replaced by an LR image with more details to further improve performance. To validate the effectiveness of the proposed method, extensive experiments are performed using the NWPU-RESISC45 data set, and the results demonstrate that the proposed method outperforms several state-of-the-art methods in terms of both objective evaluation and subjective perspective. This paper addresses the problem of hallucinating the missing high-resolution (HR) details of a low-resolution (LR) video while maintaining the temporal coherence of the reconstructed HR details using dynamic texture synthesis (DTS). Most existing multiframe-based video superresolution (SR) methods suffer from the problem of limited reconstructed visual quality due to inaccurate subpixel motion estimation between frames in an LR video. To achieve high-quality reconstruction of HR details for an LR video, we propose a texture-synthesis (TS)-based video SR method, in which a novel DTS scheme is proposed to render the reconstructed HR details in a temporally coherent way, which effectively addresses the temporal incoherence problem caused by traditional TS-based image SR methods. To further reduce the complexity of the proposed method, our method only performs the TS-based SR on a set of key frames, while the HR details of the remaining nonkey frames are simply predicted using the bidirectional overlapped block motion compensation. After all frames are upscaled, the proposed DTS-SR is applied to maintain the temporal coherence in the HR video. Experimental results demonstrate that the proposed method achieves significant subjective and objective visual quality improvement over state-of-the-art video SR methods.