A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> Abstract To take advantage of the high spectral resolution of Landsat TM images and the high spatial resolution of SPOT panchromatic images (SPOT PAN), we present a wavelet transform method to merge the two data types. In a pyramidal fashion, each TM reflective band or SPOT PAN image was decomposed into an orthogonal wavelet representation at a given coarser resolution, which consisted of a low frequency approximation image and a set of high frequency, spatially-oriented detail images. Band-by-band, the merged images were derived by performing an inverse wavelet transform using the approximation image from each TM band and detail images from SPOT PAN. The spectral and spatial features of the merged results of the wavelet methods were compared quantitatively with those of intensity-hue-saturation (IHS), principal component analysis (PCA), and the Brovey transform. It was found that multisensor data merging is a trade-off between the spectral information from a low spatial-high spectral resolution sensor an... <s> BIB001 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> Usual image fusion methods inject features from a high spatial resolution panchromatic sensor into every low spatial resolution multispectral band trying to preserve spectral signatures and improve spatial resolution to that of the panchromatic sensor. The objective is to obtain the image that would be observed by a sensor with the same spectral response (i.e., spectral sensitivity and quantum efficiency) as the multispectral sensors and the spatial resolution of the panchromatic sensor. But in these methods, features from electromagnetic spectrum regions not covered by multispectral sensors are injected into them, and physical spectral responses of the sensors are not considered during this process. This produces some undesirable effects, such as resolution overinjection images and slightly modified spectral signatures in some features. The authors present a technique which takes into account the physical electromagnetic spectrum responses of sensors during the fusion process, which produces images closer to the image obtained by the ideal sensor than those obtained by usual wavelet-based image fusion methods. This technique is used to define a new wavelet-based image fusion method. <s> BIB002 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> A new methodology for fusing satellite sensor imagery, based on tailored filtering in the Fourier domain is proposed. Finite‐duration Impulse Response (FIR) filters have been designed through an objective criterion, which depends on source image characteristics only. The designed filters allow a weighted fusion of the information contained in a fine spatial resolution image (PAN) and in a multispectral image (MULTI), respectively, establishing a trade‐off between spatial and spectral quality of the resulting fused image. This new technique has been tested with Landsat Enhanced Thematic Mapper Plus (ETM+) imagery. Spatial and spectral quality of the fused images was compared with the results provided by Mallat's Wavelet algorithm. The images fused by the proposed method were characterized by a spatial resolution very close to the PAN image, and by the spectral resolution of the MULTI image. <s> BIB003 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> In January 2006, the Data Fusion Committee of the IEEE Geoscience and Remote Sensing Society launched a public contest for pansharpening algorithms, which aimed to identify the ones that perform best. Seven research groups worldwide participated in the contest, testing eight algorithms following different philosophies [component substitution, multiresolution analysis (MRA), detail injection, etc.]. Several complete data sets from two different sensors, namely, QuickBird and simulated Pleiades, were delivered to all participants. The fusion results were collected and evaluated, both visually and objectively. Quantitative results of pansharpening were possible owing to the availability of reference originals obtained either by simulating the data collected from the satellite sensor by means of higher resolution data from an airborne platform, in the case of the Pleiades data, or by first degrading all the available data to a coarser resolution and saving the original as the reference, in the case of the QuickBird data. The evaluation results were presented during the special session on data fusion at the 2006 international geoscience and remote sensing symposium in Denver, and these are discussed in further detail in this paper. Two algorithms outperform all the others, the visual analysis being confirmed by the quantitative evaluation. These two methods share the same philosophy: they basically rely on MRA and employ adaptive models for the injection of high-pass details. <s> BIB004 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> Our framework is the synthesis of multispectral images (MS) at higher spatial resolution, which should be as close as possible to those that would have been acquired by the corresponding sensors if they had this high resolution. This synthesis is performed with the help of a high spatial but low spectral resolution image: the panchromatic (Pan) image. The fusion of the Pan and MS images is classically referred as pan-sharpening. A fused product reaches good quality only if the characteristics and differences between input images are taken into account. Dissimilarities existing between these two data sets originate from two causes-different times and different spectral bands of acquisition. Remote sensing physics should be carefully considered while designing the fusion process. Because of the complexity of physics and the large number of unknowns, authors are led to make assumptions to drive their development. Weaknesses and strengths of each reported method are raised and confronted to these physical constraints. The conclusion of this critical survey of literature is that the choice in the assumptions for the development of a method is crucial, with the risk to drastically weaken fusion performance. It is also shown that the Amelioration de la Resolution Spatiale par Injection de Structures concept prevents from introducing spectral distortion into fused products and offers a reliable framework for further developments. <s> BIB005 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Spatial Quality Assessment <s> Pansharpening is a technique that fuses a low resolution multispectral image and a high resolution panchromatic image, to obtain a multispectral image with the spatial resolution and quality of the panchromatic image while preserving spectral information of the multispectral image. In this paper, we present a new pansharpening method based on contourlet transform and Bayesian inference that generalizes the classical contourlet based algorithms. The experimental results show that the proposed method not only enhances the spatial resolution of the pansharpened image, but also preserves the spectral information of the original multispectral image. <s> BIB006
To assess the spatial quality of a pansharpened image, its spatial detail information must be compared to that present in the reference HR MS image. Just a few quantitative measures have been found in the literature to evaluate the spatial quality of merged images. Zhou BIB001 proposed the following procedure to estimate the spatial quality of the merged images: to compare the spatial information present in each band of these images with the spatial information present in the PAN image. First, a Laplacian filter is applied to the images under comparison. Second, the correlation between these two filtered images is calculated, thus obtaining the spatial correlation coefficient (SCC). However, the use of the PAN image as a reference is incorrect as demonstrated in BIB005  , and the HR MS image has to be used, as done by Otazu et al.in BIB002 . A high SCC indicates that many of the spatial detail information of one of the images are present in the other one. The SCC ideal value of each band of the merged image is 1. Recently, a new spatial quality measure was suggested in  , related to quantitative edge analysis. The authors claim that a good pansharpening technique should retain all the edges present in the PAN image in the sharpened image  . Thus, a Sobel edge operator is applied on the image in order to detect its edges and then compared with the edges of the PAN image. However, the concept behind this index is false since the reference image is not the PAN but the HRMS  . Additionally, some spectral quality measures have been adapted to spatial quality assessment. Pradhan et al.  suggested the use of structural information in SSIM measure between panchromatic and pansharpened images as a spatial quality measure. Lillo-Saavedra et al. BIB003 proposed to use the spatial ERGAS index that includes in its definition the spatial RMSE calculated between each fused spectral band and the image obtained by adjusting the histogram of the original PAN image to the histogram of the corresponding band of the fused MS image. Although an exhaustive comparison of all the aforementioned pansharpening methods is out of the scope of this paper, for the sake of reference, we have included, in Table 1 , the figures of merit for some of the pansharpened images from the observed multispectral image shown in Figure 2a already presented in this paper. The two best values for each measure have been highlighted in Table 1 . From the obtained results, it is clear that BT, HPF and Price methods, depicted in Figures 2c, d and 3a , respectively, suffer the highest spectral distortions, with the lowest SSIM and MSSIM values and the highest ERGAS value. In this case, the IHS method ( Figure 2b ) and IHS-Wavelet (Figure 6c ) produce good numerical results, but note that these results have been obtained considering only the first three bands, the ones involved in the IHS transform. Methods based in multiresolution approaches, GLP (Figure 6a ), additive wavelets (Figure 6b ), and the method described in BIB006 (Figure 6f ) provide the best results with lower spectral distortion and higher SCC values. These results are consistent with the ones reported in BIB004 where the multiresolution methods perform better than other methods. Note that the lower values of SSIM for the GLP and AW methods are due to the ratio between band 4, which has a spatial resolution of 20 m per pixel, and the PAN image, which covers 5 m per pixel, while the three first bands have a spatial resolution of 10 m per pixel. For a comparison between some of the reported methods, the reader is referred, for instance, to the results of the 2006 GRS-S Data-Fusion Contest BIB004 where a set of eight methods, mainly CS and MRA based, were tested on a common set of images or BIB005 where the authors discuss from a theoretical point of view the strengths and weakness of CS, RSC and ARSIS concept implementations (which includes HFI and multiresolution families) and perform a comparison based on the fusion rule and the effect of this rule on the spectral and spatial distortion. Another comparative analysis was developed in  , where a general image fusion (GIF) framework for pansharpening IKONOS images is proposed and the performance of several image fusion method is analyzed based on the way they compute, from the LR MS image, an LR approximation of the PAN image, and how the modulation coefficients for the detail information are defined. This work has been extended in  to consider also context adaptive methods in the so-called extended GIF protocol. Recently, a comparison of pansharpening methods was carried out in  based on their performance on automatic classification and visual interpretation applications.