A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> The merging of multisensor image data is becoming a widely used procedure because of the complementary nature of various data sets. Ideally, the method used to merge data sets with high-spatial and high-spectral resolution should not distort the spectral characteristics of the high-spectral resolution data. This paper compares the results of three different methods used to merge the information contents of the Landsat Thermatic Mapper (TM) and Satellite Pour l'Observation de la Terre (SPOT) panchromatic data. The comparison is based on spectral characteristics and is made using statistical, visual, and graphical analyses of the results <s> BIB001 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> The Nature of Remote Sensing: Introduction. Remote Sensing. Information Extraction from Remote-Sensing Images. Spectral Factors in Remote Sensing. Spectral Signatures. Remote-Sensing Systems. Optical Sensors. Temporal Characteristics. Image Display Systems. Data Systems. Summary. Exercises. References. Optical Radiation Models: Introduction. Visible to Short Wave Infrared Region. Solar Radiation. Radiation Components. Surface-Reflected. Unscattered Component. Surface-Reflected. Atmosphere-Scattered Component. Path-Scattered Component. Total At-Sensor. Solar Radiance. Image Examples in the Solar Region. Terrain Shading. Shadowing. Atmospheric Correction. Midwave to Thermal Infrared Region. Thermal Radiation. Radiation Components. Surface-Emitted Component. Surface-Reflected. Atmosphere-Emitted Component. Path-Emitted Component. Total At-Sensor. Emitted Radiance. Total Solar and Thermal Upwelling Radiance. Image Examples in the Thermal Region. Summary. Exercises. References. Sensor Models: Introduction. Overall Sensor Model. Resolution. The Instrument Response. Spatial Resolution. Spectral Resolution. Spectral Response. Spatial Response. Optical PSFopt. Image Motion PSFIM. Detector PSFdet. Electronics PSFel. Net PSFnet. Comparison of Sensor PSFs. PSF Summary for TM. Imaging System Simulation. Amplification. Sampling and Quantization. Simplified Sensor Model. Geometric Distortion. Orbit Models. Platform Attitude Models. Scanner Models. Earth Model. Line and Whiskbroom ScanGeometry. Pushbroom Scan Geometry. Topographic Distortion. Summary. Exercises. References. Data Models: Introduction. A Word on Notation. Univariate Image Statistics. Histogram. Normal Distribution. Cumulative Histogram. Statistical Parameters. Multivariate Image Statistics. Reduction to Univariate Statistics. Noise Models. Statistical Measures of Image Quality. Contrast. Modulation. Signal-to-Noise Ratio (SNR). Noise Equivalent Signal. Spatial Statistics. Visualization of Spatial Covariance. Covariance with Semivariogram. Separability and Anisotropy. Power Spectral Density. Co-occurrence Matrix. Fractal Geometry. Topographic and Sensor Effects. Topography and Spectral Statistics. Sensor Characteristics and Spectral Stastistics. Sensor Characteristics and Spectral Scattergrams. Summary. Exercises. References. Spectral Transforms: Introduction. Feature Space. Multispectral Ratios. Vegetation Indexes. Image Examples. Principal Components. Standardized Principal Components (SPC) Transform. Maximum Noise Fraction (MNF) Transform. Tasseled Cap Tranformation. Contrast Enhancement. Transformations Based on Global Statistics. Linear Transformations. Nonlinear Transformations. Normalization Stretch. Reference Stretch. Thresholding. Adaptive Transformation. Color Image Contrast Enhancement. Min-max Stretch. Normalization Stretch. Decorrelation Stretch. Color Spacer Transformations. Summary. Exercises. References. Spatial Transforms: Introduction. An Image Model for Spatial Filtering. Convolution Filters. Low Pass and High Pass Filters. High Boost Filters. Directional Filters. The Border Region. Characterization of Filtered Images. The Box Filter Algorithm. Cascaded Linear Filters. Statistical Filters. Gradient Filters. Fourier Synthesis. Discrete Fourier Transforms in 2-D. The Fourier Components. Filtering with the Fourier Transform. Transfer Functions. The Power Spectrum. Scale Space Transforms. Image Resolution Pyramids. Zero-Crossing Filters. Laplacian-of-Gaussian (LoG) Filters. Difference-of-Gaussians (DoG) Filters.Wavelet Transforms. Summary. Exercises. References. Correction and Calibration: Introduction. Noise Correction. Global Noise. Sigma Filter. Nagao-Matsuyama Filter. Local Noise. Periodic Noise. Distriping 359. Global,Linear Detector Matching. Nonlinear Detector Matching. Statistical Modification to Linear and Nonlinear Detector. Matching. Spatial Filtering Approaches. Radiometric Calibration. Sensor Calibration. Atmospheric Correction. Solar and Topographic Correction. Image Examples. Calibration and Normalization of Hyperspectral Imagery. AVIRIS Examples. Distortion Correction. Polynomial Distortion Models. Ground Control Points (GCPs). Coordinate Transformation. Map Projections. Resampling. Summary. Exercises References. Registration and Image Fusion: Introduction. What is Registration? Automated GCP Location. Area Correlation. Other Spatial Features. Orthrectification. Low-Resolution DEM. High-Resolution DEM. Hierarchical Warp Stereo. Multi-Image Fusion. Spatial Domain Fusion. High Frequency Modulation. Spectral Domain Fusion. Fusion Image Examples. Summary. Exercises. References. Thematic Classification: Introduction. The Importance of Image Scale. The Notion of Similarity. Hard Versus Soft Classification. Training the Classifier. Supervised Training. Unsupervised Training. K-Means Clustering Algorithm. Clustering Examples. Hybrid Supervised/Unsupervised Training. Non-Parametric Classification Algorithms. Level-Slice. Nearest-Mean. Artificial Neural Networks (ANNs). Back-Propagation Algorithm. Nonparametric Classification Examples. Parametric Classification Algorithms. Estimation of Model-Parameters. Discriminant Functions. The Normal Distribution Model. Relation to the Nearest-Mean Classifier. Supervised Classification Examples and Comparison to Nonparametric Classifiers. Segmentation. Region Growing. Region Labeling. Sub-Pixel Classification. The Linear Mixing Model. Unmixing Model. Hyperspectral Image Analysis. Visualization of the Image Cube. Feature Extraction. Image Residuals. Pre-Classification Processing and Feature Extraction. Classification Algorithms. Exercises. Error Analysis. Multitemporal Images. Summary. References. Index. <s> BIB002 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> This paper gives brief overviews on common image fusion methods in both spatial and spectral domains, and proposes a frequency-based image fusion approach for preserving the spatial and spectral information in merging multiresolution images. Procedures of these image fusion approaches and resulted images from merging QuickBird panchromatic and multispectral images are presented. In comparison with the other fusion methods tested, the suggested frequency-based fusion approach is flexible and suitable for enhancing the information content of the multiresolution images. <s> BIB003 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> Since Chavez proposed the highpass filtering procedure to fuse multispectral and panchromatic images, several fusion methods have been developed based on the same principle: to extract from the panchromatic image spatial detail information to later inject it into the multispectral one. In this paper, we present new fusion alternatives based on the same concept, using the multiresolution wavelet decomposition to execute the detail extraction phase and the intensity-hue-saturation (IHS) and principal component analysis (PCA) procedures to inject the spatial detail of the panchromatic image into the multispectral one. The multiresolution wavelet decomposition has been performed using both decimated and undecimated algorithms and the resulting merged images compared both spectral and spatially. These fusion methods, as well as standard IHS-, PCA-, and wavelet-based methods have been used to merge Systeme Pour l'Observation de la Terre (SPOT) 4 XI and SPOT 4 M images with a ratio 4:1. We have estimated the validity of each fusion method by analyzing, visually and quantitatively, the quality of the resulting fused images. The methodological approaches proposed in this paper result in merged images with improved quality with respect to those obtained by standard IHS, PCA, and standard wavelet-based fusion methods. For both proposed fusion methods, better results are obtained when an undecimated algorithm is used to perform the multiresolution wavelet decomposition. <s> BIB004 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> This paper deals with the assessment of the quality of the products resulting from the application of fusion methods, exploiting the synergy of multimodal images at a low spatial resolution and images at a higher spatial resolution but with a lower spectral content. It concentrates on the assessment of the geometrical quality through the analysis of the quality of selected contours. The first objective of our paper is to present a method for the estimation of the MTF. This method applies to excerpts of images containing long well contrasted linear features. The performance of the method is assessed by applying it to several Ikonos images and comparing the results to published works. The second objective is to demonstrate that the method may be exploited to assess the geometrical quality of the fused product. <s> BIB005 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> High-frequency injection family <s> Quality assessment of pansharpening methods is not an easy task. Quality-assessment indexes, like Q4, spectral angle mapper, and relative global synthesis error, require a reference image at the same resolution as the fused image. In the absence of such a reference image, the quality of pansharpening is assessed at a degraded resolution only. The recently proposed index of Quality Not requiring a Reference (QNR) is one among very few tools available for assessing the quality of pansharpened images at the desired high resolution. However, it would be desirable to cross the outcomes of several independent quality-assessment indexes, in order to better determine the quality of pansharpened images. In this paper, we propose a method to assess fusion quality at the highest resolution, without requiring a high-resolution reference image. The novel method makes use of digital filters matching the modulation transfer functions (MTFs) of the imaging-instrument channels. Spectral quality is evaluated according to Wald's spectral consistency property. Spatial quality measures interscale changes by matching spatial details, extracted from the multispectral bands and from the panchromatic image by means of the high-pass complement of MTF filters. Eventually, we highlight the necessary and sufficient condition criteria for quality-assessment indexes by developing a pansharpening method optimizing the QNR spatial index and assessing the quality of fused images by using the proposed protocol. <s> BIB006
The high-frequency injection family methods were first proposed by Schowengerdt  , working on full-resolution and spatially compressed Landsat MSS data. He demonstrated the use of a high-resolution band to "sharpen" or edge-enhance lower-resolution bands having the same approximate wavelength characteristics. Some years later, Chavez  proposed a project whose primary objective was to extract the spectral information from the Landsat TM and combine (inject) it with the spatial information from a data set having much higher spatial resolution. To extract the details from the highresolution data set, he used a high-pass filter in order to "enhance the high-frequency/spatial information but, more important, suppress the low frequency/spectral information in the higher-resolution image"  . This was necessary so that simple addition of the images did not distort the spectral balance of the combined product. A useful concept for understanding spatial filtering is that any image is made of spatial components at different kernel sizes. Suppose we process an image in such a way that the value at each output pixel is the average of a small neighborhood of input pixels, a box filter. The result is a low-pass (LP) blurred version of the original image that will be noted as LP. Subtracting this image from the original one produces high-pass (HP) image that represents the difference between each original pixel and the average of its neighborhood. This relation can be written as the following equation: which is valid for any neighborhood size (scale). As the neighborhood size is increased, the LP image hides successively larger and larger structures, while the HP image picks up the smaller structures lost in the LP image (see Equation 4) BIB002 . The idea behind this type of spatial domain fusion is to transfer the high-frequency content of the PAN image to the MS images by applying spatial filtering techniques BIB003 . However, the size of the filter kernels cannot be arbitrary because it has to reflect the radiometric normalization between the two images. Chavez et al. BIB001 suggested that the best kernel size is approximately twice the size of the ratio of the spatial resolutions of the sensors, which produce edge-enhanced synthetic images with the least spectral distortion and edge noises. According to  , pansharpening methods based on injecting high-frequency components into resampled versions of the MS data have demonstrated a superior performance and compared with many other pansharpening methods such as the methods in the CS family. Several variations of high-frequency injection pansharpening methods have been proposed as HighPass Filtering Pansharpening and High Pass Modulation. As we have already mentioned, the main idea of the high-pass filtering (HPF) pansharpening method is to extract from the PAN image the high-frequency information, to later add or inject it into the MS image previously expanded to match the PAN pixel size. This spatial information extraction is performed by applying a low-pass spatial filter to the PAN image, where h 0 is a low-pass filter and * the convolution operator. The spatial information injection is performed adding, pixel by pixel, the filtered image that results from subtracting filtered PAN from the original PAN image, to the MS one  BIB004 . There are many different filters that can be used: Box filter, Gaussian, Laplacian, and so on. Recently, the use of the modulation transfer function (MTF) of the sensor as the low-pass filter has been proposed in BIB006 . The MTF is the amplitude spectrum of the system point spread function (PSF) BIB005 . In BIB006 , the HP image is also multiplied by a weight selected to maximize the Quality Not requiring a Reference (QNR) criterion proposed in the paper. As expected, HPF images present low spectral distortion. However, the ripple in the frequency response will have some negative impact  . The HPF method could be considered the predecessor of an extended group of image pansharpening procedures based on the same principle: to extract spatial detail information from the PAN image not present in the MS image and inject it into the latter in a multiresolution framework. This principle is known as the ARSIS concept  . In the High Pass Modulation (HPM), also known as High Frequency Modulation (HFM) algorithm BIB002 , the PAN image is multiplied by each band of the LRMS image and normalized by a low-pass filtered version of the PAN image to estimate the enhanced MS image bands. The principle of HPM is to transfer the high-frequency information of the PAN image to the LRMS band b (LRMS b ) with a modulation coefficient k b which equals the ratio between the LRMS and the low-pass filtered version of the PAN image  . Thus, the algorithm assumes that each pixel of the enhanced (sharpened) MS image in band b is simply proportional to the corresponding higher-resolution image at each pixel. This constant of proportionality is a spatially variable gain factor, calculated by, where filtered PAN is a low-pass filtered version of PAN image (see Equation 5) BIB002 . According to  (where HFI has also been formulated into the GIF framework and relations with CS, RSC and some multiresolution family methods are explored) when the low-pass filter is chosen as in the HPF method, the HPM method will give slightly better performance than HPF because the color of the pixels is not biased toward gray. The process flow diagram of the HFI sharpening techniques is shown in Algorithm 3. Also, a pansharpened image using the HPM method is shown in Figure 2d . Note that the HFI methods are closely related, as we will see later, to the multiresolution family. The main differences are the types of filter used, that a single level of decomposition is applied to the images and the different origins of the approaches. Algorithm 3 High-frequency injection pansharpening