A survey of classical methods and new trends in pansharpening of multispectral images <s> Introduction <s> The Nature of Remote Sensing: Introduction. Remote Sensing. Information Extraction from Remote-Sensing Images. Spectral Factors in Remote Sensing. Spectral Signatures. Remote-Sensing Systems. Optical Sensors. Temporal Characteristics. Image Display Systems. Data Systems. Summary. Exercises. References. Optical Radiation Models: Introduction. Visible to Short Wave Infrared Region. Solar Radiation. Radiation Components. Surface-Reflected. Unscattered Component. Surface-Reflected. Atmosphere-Scattered Component. Path-Scattered Component. Total At-Sensor. Solar Radiance. Image Examples in the Solar Region. Terrain Shading. Shadowing. Atmospheric Correction. Midwave to Thermal Infrared Region. Thermal Radiation. Radiation Components. Surface-Emitted Component. Surface-Reflected. Atmosphere-Emitted Component. Path-Emitted Component. Total At-Sensor. Emitted Radiance. Total Solar and Thermal Upwelling Radiance. Image Examples in the Thermal Region. Summary. Exercises. References. Sensor Models: Introduction. Overall Sensor Model. Resolution. The Instrument Response. Spatial Resolution. Spectral Resolution. Spectral Response. Spatial Response. Optical PSFopt. Image Motion PSFIM. Detector PSFdet. Electronics PSFel. Net PSFnet. Comparison of Sensor PSFs. PSF Summary for TM. Imaging System Simulation. Amplification. Sampling and Quantization. Simplified Sensor Model. Geometric Distortion. Orbit Models. Platform Attitude Models. Scanner Models. Earth Model. Line and Whiskbroom ScanGeometry. Pushbroom Scan Geometry. Topographic Distortion. Summary. Exercises. References. Data Models: Introduction. A Word on Notation. Univariate Image Statistics. Histogram. Normal Distribution. Cumulative Histogram. Statistical Parameters. Multivariate Image Statistics. Reduction to Univariate Statistics. Noise Models. Statistical Measures of Image Quality. Contrast. Modulation. Signal-to-Noise Ratio (SNR). Noise Equivalent Signal. Spatial Statistics. Visualization of Spatial Covariance. Covariance with Semivariogram. Separability and Anisotropy. Power Spectral Density. Co-occurrence Matrix. Fractal Geometry. Topographic and Sensor Effects. Topography and Spectral Statistics. Sensor Characteristics and Spectral Stastistics. Sensor Characteristics and Spectral Scattergrams. Summary. Exercises. References. Spectral Transforms: Introduction. Feature Space. Multispectral Ratios. Vegetation Indexes. Image Examples. Principal Components. Standardized Principal Components (SPC) Transform. Maximum Noise Fraction (MNF) Transform. Tasseled Cap Tranformation. Contrast Enhancement. Transformations Based on Global Statistics. Linear Transformations. Nonlinear Transformations. Normalization Stretch. Reference Stretch. Thresholding. Adaptive Transformation. Color Image Contrast Enhancement. Min-max Stretch. Normalization Stretch. Decorrelation Stretch. Color Spacer Transformations. Summary. Exercises. References. Spatial Transforms: Introduction. An Image Model for Spatial Filtering. Convolution Filters. Low Pass and High Pass Filters. High Boost Filters. Directional Filters. The Border Region. Characterization of Filtered Images. The Box Filter Algorithm. Cascaded Linear Filters. Statistical Filters. Gradient Filters. Fourier Synthesis. Discrete Fourier Transforms in 2-D. The Fourier Components. Filtering with the Fourier Transform. Transfer Functions. The Power Spectrum. Scale Space Transforms. Image Resolution Pyramids. Zero-Crossing Filters. Laplacian-of-Gaussian (LoG) Filters. Difference-of-Gaussians (DoG) Filters.Wavelet Transforms. Summary. Exercises. References. Correction and Calibration: Introduction. Noise Correction. Global Noise. Sigma Filter. Nagao-Matsuyama Filter. Local Noise. Periodic Noise. Distriping 359. Global,Linear Detector Matching. Nonlinear Detector Matching. Statistical Modification to Linear and Nonlinear Detector. Matching. Spatial Filtering Approaches. Radiometric Calibration. Sensor Calibration. Atmospheric Correction. Solar and Topographic Correction. Image Examples. Calibration and Normalization of Hyperspectral Imagery. AVIRIS Examples. Distortion Correction. Polynomial Distortion Models. Ground Control Points (GCPs). Coordinate Transformation. Map Projections. Resampling. Summary. Exercises References. Registration and Image Fusion: Introduction. What is Registration? Automated GCP Location. Area Correlation. Other Spatial Features. Orthrectification. Low-Resolution DEM. High-Resolution DEM. Hierarchical Warp Stereo. Multi-Image Fusion. Spatial Domain Fusion. High Frequency Modulation. Spectral Domain Fusion. Fusion Image Examples. Summary. Exercises. References. Thematic Classification: Introduction. The Importance of Image Scale. The Notion of Similarity. Hard Versus Soft Classification. Training the Classifier. Supervised Training. Unsupervised Training. K-Means Clustering Algorithm. Clustering Examples. Hybrid Supervised/Unsupervised Training. Non-Parametric Classification Algorithms. Level-Slice. Nearest-Mean. Artificial Neural Networks (ANNs). Back-Propagation Algorithm. Nonparametric Classification Examples. Parametric Classification Algorithms. Estimation of Model-Parameters. Discriminant Functions. The Normal Distribution Model. Relation to the Nearest-Mean Classifier. Supervised Classification Examples and Comparison to Nonparametric Classifiers. Segmentation. Region Growing. Region Labeling. Sub-Pixel Classification. The Linear Mixing Model. Unmixing Model. Hyperspectral Image Analysis. Visualization of the Image Cube. Feature Extraction. Image Residuals. Pre-Classification Processing and Feature Extraction. Classification Algorithms. Exercises. Error Analysis. Multitemporal Images. Summary. References. Index. <s> BIB001 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Introduction <s> Our framework is the synthesis of multispectral images (MS) at higher spatial resolution, which should be as close as possible to those that would have been acquired by the corresponding sensors if they had this high resolution. This synthesis is performed with the help of a high spatial but low spectral resolution image: the panchromatic (Pan) image. The fusion of the Pan and MS images is classically referred as pan-sharpening. A fused product reaches good quality only if the characteristics and differences between input images are taken into account. Dissimilarities existing between these two data sets originate from two causes-different times and different spectral bands of acquisition. Remote sensing physics should be carefully considered while designing the fusion process. Because of the complexity of physics and the large number of unknowns, authors are led to make assumptions to drive their development. Weaknesses and strengths of each reported method are raised and confronted to these physical constraints. The conclusion of this critical survey of literature is that the choice in the assumptions for the development of a method is crucial, with the risk to drastically weaken fusion performance. It is also shown that the Amelioration de la Resolution Spatiale par Injection de Structures concept prevents from introducing spectral distortion into fused products and offers a reliable framework for further developments. <s> BIB002 </s> A survey of classical methods and new trends in pansharpening of multispectral images <s> Introduction <s> This article aims at explaining the ARSIS concept. By fusing two sets of images A and B, one with a high spatial resolution, the other with a low spatial resolution and different spectral bands, the ARSIS concept permits to synthesise the dataset B at the resolution of A that is as close as possible to reality. It is based on the assumption that the missing information is linked to the high frequencies in the sets A and B. It searches a relationship between the high frequencies in the multispectral set B and the set A and models this relationship. The general problem for the synthesis is presented first. The general properties of the fused product are given. Then, the ARSIS concept is discussed. The general scheme for the implementation of a method belonging to this concept is presented. Then, this article intends to help practitioners and researchers to better understand this concept through practical details about implementations. Two Multiscale Models are described as well as two Inter-Band Structure Models (IBSM). They are applied to an Ikonos image as an illustration case. The fused products are assessed by means of a known protocol comprising a series of qualitative and quantitative tests. The products are found of satisfactory quality. This case illustrates the differences existing between the various models, their advantages and limits. Tracks for future improvements are discussed. <s> BIB003
Nowadays, huge quantities of satellite images are available from many earth observation platforms, such as SPOT [1], Landsat 7 [2], IKONOS [3] , QuickBird [4] and OrbView [5] . Moreover, due to the growing number of satellite sensors, the acquisition frequency of the same scene is continuously increasing. Remote sensing images are recorded in digital form and then processed by computers to produce image products useful for a wide range of applications. The spatial resolution of a remote sensing imaging system is expressed as the area of the ground captured by one pixel and affects the reproduction of details within the scene. As the pixel size is reduced, more scene details are preserved in the digital representation  . The instantaneous field of view (IFOV) is the ground area sensed at a given instant of time. The spatial resolution depends on the IFOV. For a given number of pixels, the finer the IFOV is, the higher the spatial resolution. Spatial resolution is also viewed as the clarity of the high-frequency detail information available in an image. Spatial resolution in remote sensing is usually expressed in meters or feet, which represents the length of the side of the area covered by a pixel. Figure  1 shows three images of the same ground area but with different spatial resolutions. The image at 5 m depicted in Figure 1a was captured by the SPOT 5 satellite, while the other two images, at 10 m and 20 m, are simulated from the first image. As can be observed in these images, the detail information becomes clearer as the spatial resolution increases from 20 m to 5 m. Spectral resolution is the electromagnetic bandwidth of the signals captured by the sensor producing a given image. The narrower the spectral bandwidth is, the higher the spectral resolution. If the platform captures images with a few spectral bands, typically 4-7, they are referred to as multispectral (MS) data, while if the number of spectral bands is measured in hundreds or thousands, they are referred to as hyperspectral (HS) data  . Together with the MS or HS image, satellites usually provide a panchromatic (PAN) image. This is an image that contains reflectance data representative of a wide range of wavelengths from the visible to the thermal infrared, that is, it integrates the chromatic information; therefore, the name is "pan" chromatic. A PAN image of the visible bands captures a combination of red, green and blue data into a single measure of reflectance. Remote sensing systems are designed within often competing constraints, among the most important ones being the trade-off between IFOV and signal-to-noise ratio (SNR). Since MS, and to a greater extent HS, sensors have reduced spectral bandwidths compared to PAN sensors, they typically have for a given IFOV a reduced spatial resolution in order to collect more photons and preserve the image SNR. Many sensors such as SPOT, ETM+, IKONOS, OrbView and QuickBird have a set of MS bands and a co-registered higher spatial resolution PAN band. With appropriate algorithms, it is possible to combine these data and produce MS imagery with higher spatial resolution. This concept is known as multispectral or multisensor merging, fusion or pansharpening (of the lower-resolution image) BIB001 . Pansharpening can consequently be defined as a pixellevel fusion technique used to increase the spatial resolution of the MS image  . Pansharpening is shorthand for panchromatic sharpening, meaning the use of a PAN (single band) image to sharpen an MS image. In this sense, to sharpen means to increase the spatial resolution of an MS image. Thus, pansharpening techniques increase the spatial resolution while simultaneously preserving the spectral information in the MS image, giving the best of the two worlds: high spectral resolution and high spatial resolution  . Some of the applications of pansharpening include improving geometric correction, enhancing certain features not visible in either of the single data alone, changing detection using temporal data sets and enhancing classification BIB002 . During the past years, an enormous amount of pansharpening techniques have been developed, and in order to choose the one that better serves to the user needs, there are some points, mentioned by Pohl  , that have to be considered. In the first place, the objective or application of the pansharpened image can help in defining the necessary spectral and spatial resolution. For instance, some users may require frequent, repetitive coverage, with relatively low spatial resolution (i.e., meteorology applications), others may desire the highest possible spatial resolution (i.e., mapping), while other users may need both high spatial resolution and frequent coverage, plus rapid image delivery (i.e., military surveillance). Then, the data that are more useful to meet the needs of the pansharpening applications, like the sensor, the satellite coverage and atmospheric constraints such as cloud cover and sun angle have to be selected. We are mostly interested in sensors that can capture simultaneously a PAN channel with high spatial resolution and some MS channels with high spectral resolution like SPOT 5, Landsat 7 and QuickBird satellites. In some cases, PAN and MS images captured by different satellite sensors at different dates for the same scene can be used for some applications BIB002 , like in the case of fusing different MS SPOT 5 images captured at different times with one PAN IKONOS image  , which can be considered as a multisensor, multitemporal and multiresolution pansharpening case. We also have to take into account the need for data pre-processing, like registration, upsampling and histogram matching, as well as the selection of a pansharpening technique that makes the combination of the data most successful. Finally, evaluation criteria are needed to specify which is the most successful pansharpening approach. In this paper, we examine the classical and state-ofthe-art pansharpening methods described in the literature giving a clear classification of the methods and a description of their main characteristics. To the best of our knowledge, there is no recent paper providing a complete overview of the different pansharpening methods. However, some papers partially address the classification of pansharpening methods, see  for instance, or relate already proposed techniques of more global paradigms BIB003   . This paper is organized as follows. In Section 2 data pre-processing techniques are described. In Section 3 a classification of the pansharpening methods is presented, with a description of the methods related to each category and some examples. In this section, we also point out open research problems in each category. In Section 4 we analyze how the quality of the pansharpened images can be assessed both visually and quantitatively and examine the different quality measures proposed for that purpose, and finally, Section 5 concludes the paper.