A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> A generalized Component Substitution (COS) technique is presented for enhancing spatial resolution of multispectral bands using a higher resolution data set. The process involves a simple one-step linear transformation of the combined data space, and the procedure can be used for the implementation of the popular Intensity Hue Saturation (IHS) transformation technique of data merging. In the generalized procedure the weights used in the linear transformation are scene dependent and are determined from multivariate statistical techniques. <s> BIB001 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> Abstract To take advantage of the high spectral resolution of Landsat TM images and the high spatial resolution of SPOT panchromatic images (SPOT PAN), we present a wavelet transform method to merge the two data types. In a pyramidal fashion, each TM reflective band or SPOT PAN image was decomposed into an orthogonal wavelet representation at a given coarser resolution, which consisted of a low frequency approximation image and a set of high frequency, spatially-oriented detail images. Band-by-band, the merged images were derived by performing an inverse wavelet transform using the approximation image from each TM band and detail images from SPOT PAN. The spectral and spatial features of the merged results of the wavelet methods were compared quantitatively with those of intensity-hue-saturation (IHS), principal component analysis (PCA), and the Brovey transform. It was found that multisensor data merging is a trade-off between the spectral information from a low spatial-high spectral resolution sensor an... <s> BIB002 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> Among various image fusion methods, intensity-hue-saturation (IHS) technique is capable of quickly merging the massive volumes of data. For IKONOS imagery, IHS can yield satisfactory "spatial" enhancement but may introduce "spectral" distortion, appearing as a change in colors between compositions of resampled and fused multispectral bands. To solve this problem, a fast IHS fusion technique with spectral adjustment is presented. The experimental results demonstrate that the proposed approach can provide better performance than the original IHS method, both in processing speed and image quality. <s> BIB003 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> In this work, a simple preprocessing patch is intro- duced before the Gram-Schmidt (GS) spectral sharpening method (as implemented in ENVI) such that the resulting fused multispec- tral (MS) data exhibit higher sharpness and spectral quality. This is achieved by defining a generalized intensity (GI) component as a weighted average of the MS bands, with weights taken either as percentages of overlap between the spectral responses of individual bands and the spectral response of panchromatic (Pan), or better as regression coefficients between the MS bands and the decimated Pan image. In the former case the weights are pre-calculated for each sensor. In the latter case, the weights are calculated by applying a multivariate regression to the data that are being fused. The above GI component is used as low- resolution approximation of the Pan image. Experimental results carried out on very-high resolution IKONOS data demonstrate that the proposed enhanced GS method visually outperforms both modes of the ENVI implementation of GS, especially in true color displays. Quantitative scores performed on spatially degraded data by means of such parameters as Wald's ERGAS and the novel Q4 score index based on quaternions theory, confirm the superiority of the enhanced GS method over its baseline. <s> BIB004 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> Many image fusion algorithms have been developed while more algorithms are being developed to improve the ability of preserving spectral information. Starting from the analysis of component substitution (COS) image fusion technique, a novel general component substitution (GCOS) image fusion framework is proposed, which could be used in three aspects: (1) comparative analysis to existing algorithms, (2) providing a fast technique for current COS fusion methods, and (3) guiding the development of the new COS algorithm. A demonstrative implementation of GCOS is provided, which can employ radiometric properties of sensors in the process of image fusion. An experiment based on degraded IKONOS images was carried out to demonstrate the effectiveness of the method, and the fusion image processed through the proposed method shows a higher correlation coefficient (CC) and a universal image quality index (UIQI), and a lower relative difference (RD) with the reference image in comparison to those yielded through Gram-Schmidt (GS) spectral sharpening, principal component analysis (PCA), smoothing filter-based intensity modulation (SFIM) and additive wavelet transform (AWT) methods, and provides more reasonable spatial details by visual validation. Validation of the proposed algorithm proved the ability of the proposed GCOS framework. <s> BIB005 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> In this paper, multivariate regression is adopted to improve spectral quality, without diminishing spatial quality, in image fusion methods based on the well-established component substitution (CS) approach. A general scheme that is capable of modeling any CS image fusion method is presented and discussed. According to this scheme, a generalized intensity component is defined as the weighted average of the multispectral (MS) bands. The weights are obtained as regression coefficients between the MS bands and the spatially degraded panchromatic (Pan) image, with the aim of capturing the spectral responses of the sensors. Once it has been integrated into the Gram-Schmidt spectral-sharpening method, which is implemented in environment for visualizing images (ENVI) program, and into the generalized intensity-hue-saturation fusion method, the proposed preprocessing module allows the production of fused images of the same spatial sharpness but of increased spectral quality with respect to the standard implementations. In addition, quantitative scores carried out on spatially degraded data clearly confirm the superiority of the enhanced methods over their baselines. <s> BIB006 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> Our framework is the synthesis of multispectral images (MS) at higher spatial resolution, which should be as close as possible to those that would have been acquired by the corresponding sensors if they had this high resolution. This synthesis is performed with the help of a high spatial but low spectral resolution image: the panchromatic (Pan) image. The fusion of the Pan and MS images is classically referred as pan-sharpening. A fused product reaches good quality only if the characteristics and differences between input images are taken into account. Dissimilarities existing between these two data sets originate from two causes-different times and different spectral bands of acquisition. Remote sensing physics should be carefully considered while designing the fusion process. Because of the complexity of physics and the large number of unknowns, authors are led to make assumptions to drive their development. Weaknesses and strengths of each reported method are raised and confronted to these physical constraints. The conclusion of this critical survey of literature is that the choice in the assumptions for the development of a method is crucial, with the risk to drastically weaken fusion performance. It is also shown that the Amelioration de la Resolution Spatiale par Injection de Structures concept prevents from introducing spectral distortion into fused products and offers a reliable framework for further developments. <s> BIB007 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Component substitution family <s> High correlation among the neighboring pixels both spatially and spectrally in a multispectral image makes it necessary to use an efficient data transformation approach before performing pan-sharpening. Wavelets and principal component analysis (PCA) methods have been a popular choice for spatial and spectral transformations, respectively. Current PCA-based pan-sharpening methods make an assumption that the first principal component (PC) of high variance is an ideal choice for replacing or injecting it with high spatial details from the high-resolution histogram-matched panchromatic (PAN) image. This paper presents a combined adaptive PCA-contourlet approach for pan-sharpening, where the adaptive PCA is used to reduce the spectral distortion and the use of nonsubsampled contourlets for spatial transformation in pan-sharpening is incorporated to overcome the limitation of the wavelets in representing the directional information efficiently and capturing intrinsic geometrical structures of the objects. The efficiency of the presented method is tested by performing pan-sharpening of the high-resolution (IKONOS and QuickBird) and the medium-resolution (Landsat-7 Enhanced Thematic Mapper Plus) datasets. The evaluation of the pan-sharpened images using global validation indexes reveal that the adaptive PCA approach helps reducing the spectral distortion, and its merger with contourlets provides better fusion results. <s> BIB008
The component substitution (CS) methods start by upsampling the low-resolution MS image to the size of the PAN image. Then, the MS image is transformed into a set of components, using usually a linear transform of the MS bands. The CS methods work by substituting a component of the (transformed) MS image, C l , with a component, C h , from the PAN image. These methods are physically meaningful only when these two components, C l and C h , contain almost the same spectral information. In other words, the C l component should contain all the redundant information of the MS and PAN images, but C h should contain more spatial information. An improper construction of the C l component tends to introduce high spectral distortion. The general algorithm for the CS sharpening techniques is summarized in Algorithm 1. This algorithm has been generalized by Tu et al.  , where the authors also prove that the forward and backward transforms are not needed and steps 2-5 of Algorithm 1 can be summarized as finding a new component C l and adding the difference between the PAN and this new component to each upsampled MS image band. This framework has been further extended by Wang et al.  and Aiazzi et al.  in the so-called general image fusion (GIF) and extended GIF (EGIF) protocol, respectively. Algorithm 1 The CS family includes many popular pansharpening methods, such as the IHS, PCS and Gram-Schmidt (GS) methods BIB001 BIB005 , each of them involving a different transformation of the MS image. CS techniques are attractive because they are fast and easy to implement and allow users' expectations to be fulfilled most of the time, since they provide pansharpened images with good visual/geometrical quality in most cases BIB006 . However, the results obtained by these methods highly depend on the correlation between the bands, and since the same transform is applied to the whole image, it does not take into account local dissimilarities between PAN and MS images BIB007  . A single type of transform does not always obtain the optimal component required for substitution, and it would be difficult to choose the appropriate spectral transformation method for diverse data sets. In order to alleviate this problem, recent methods incorporate statistical tests or weighted measures to adaptively select an optimal component for substitution and transformation. This results in a new approach known as adaptive component substitution  BIB008 BIB004 . The Intensity-Hue-Saturation (IHS) pansharpening method   is one of the classical techniques included in this family, and it uses the IHS color space, which is often chosen due to the tendency of the visual cognitive system of human beings to treat the intensity (I), hue (H) and saturation (S) components as roughly orthogonal perceptual axes. IHS transform originally was applied to RGB true color, but in the remote sensing applications and for display purposes only, arbitrary bands are assigned to RGB channel to produce false color composites  . The ability of IHS transform to separate effectively spatial information (band I) and spectral information (bands H and S)  makes it very applicable in pan-sharpening. There are different models of IHS transform, differing in the method used to compute the intensity value. Smith's hexacone and triangular models are two of the most widely used ones  . An example of pansharpened image using IHS method is shown in Figure 2b . The major limitation of this technique is that only three bands are involved. Tu et al.  proposed a generalized IHS transform that surpasses the dimensional limitation. In any case, since the spectral response of I, as synthesized from the MS bands, does not generally match the radiometry of the histogram-matched PAN BIB006 , when the fusion result is displayed in color composition, large spectral distortion may appear as color changes. In order to minimize the spectral distortion in IHS pansharpening, Tu et al. BIB003 proposed a new adaptive IHS method in which the intensity band approximates the PAN image for IKONOS images as closely as possible. This adaptive IHS has been extended by Rahmani et al.  to deal with any kind of image by determining the coefficients a i that best approximate subject to the physical constraint of nonnegativity of the coefficients a i . Note that, although this method reduces spectral distortion, local dissimilarities between MS and PAN images might remain BIB007 . Another method in the CS family is principal component substitution (PCS) that relies on the principal component analysis (PCA) mathematical transformation. The PCA, also known as the Karhunen-Loéve transform or the Hotelling transform, is widely used in signal processing, statistics and many other areas. This transformation generates a new set of rotated axes, in which the new image spectral components are not correlated. The largest amount of the variance is mapped to the first component, with decreasing variance going to each of the following ones. The sum of the variances in all the components is equal to the total variance present in the original input images. PCA and the calculation of the transformation matrices can be performed following the steps specified in  . Theoretically, the first principal component, PC1, collects the information that is common to all bands used as input data to the PCA, i.e., the spatial information, while the spectral information that is specific to each band is captured in the other principal components BIB002  . This makes PCS an adequate technique when merging MS and PAN images. PCS is similar to the IHS method, with the main advantage that an arbitrary number of bands can be considered. However, some spatial information may not be mapped to the first component, depending on the degree of correlation and spectral contrast existing among the MS bands  , resulting in the same problems that IHS had. To overcome this drawback, Shah et al. BIB008 proposed a new adaptive PCA-based pansharpening method that determines, using cross-correlation, the appropriate PC component to be substituted by the PAN image. By replacing this PC component by the high spatial resolution PAN component, adaptive PCA method will produce better results than traditional ones BIB008 . A widespread CS technique is the Gram-Schmidt (GS) spectral sharpening. This method was invented by Laben and Brover in 1998 and patented by Eastman Kodak  . The GS transformation, as described in  , is a common technique used in linear algebra and multivariate statistics. GS is used to orthogonalize matrix data or bands of a digital image removing redundant (i.e., correlated) information that is contained in multiple bands. If there were perfect correlation between input bands, the GS orthogonalization process would produce a final band with all its elements equal to zero. For its use in pansharpening, GS transformation had been modified  . In the modified process, the mean of each band is subtracted from each pixel in the band before the orthogonalization is performed to produce a more accurate outcome. In GS-based pansharpening, a lower-resolution PAN band needs to be simulated and used as the first band of the input to the GS transformation, together with the MS image. Two methods are used in  to simulate this band; in the first method, the LRMS bands are combined into a single lower-resolution PAN (LR PAN) as the weighted mean of MS image. These weights depend on the spectral response of the MS bands and high-resolution PAN (HR PAN) image and on the optical transmittance of the PAN band. The second method simulates the LR PAN image by blurring and subsampling the observed PAN image. The major difference in results, mostly noticeable in a true color display, is that the first method exhibits outstanding spatial quality, but spectral distortions may occur. This distortion is due to the fact that the average of the MS spectral bands is not likely to have the same radiometry as the PAN image. The second method is unaffected by spectral distortion but generally suffers from a lower sharpness and spatial enhancement. This is due to the injection mechanism of high-pass details taken from PAN, which is embedded into the inverse GS transformation, carried out by using the full-resolution PAN, while the forward transformation uses the low-resolution approximation of PAN obtained by resampling the decimated PAN image provided by the user. In order to avoid this drawback, Aiazzi et al. BIB004 proposed an Enhanced GS method, where the LR PAN is generated by a weighted average of the MS bands and the weights are estimated to minimize the MMSE with the downsampled PAN. GS is more general than PCA, which can be understood as a particular case of GS in which LR PAN is the first principal component  .