Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> A new multiway regression method called N-way partial least squares (N-PLS) is presented. The emphasis is on the three-way PLS version (tri-PLS), but it is shown how to extend the algorithm to higher orders. The developed algorithm is superior to unfolding methods, primarily owing to a stabilization of the decomposition. This stabilization potentially gives increased interpretability and better predictions. The algorithm is fast compared with e.g. PARAFAC, because it consists of solving eigenvalue problems. ::: ::: An example of the developed algorithm taken from the sugar industry is shown and compared with unfold-PLS. Fluorescence excitation—emission matrices (EEMs) are measured on white sugar solutions and used to predict the ash content of the sugar. The predictions are comparable by the two methods, but there is a clear difference in the interpretability of the two solutions. Also shown is a simulated example of EEMs with very noisy measurements and a low relative signal from the analyte of interest. The predictions from unfold-PLS are almost twice as bad as from tri-PLS despite the large number of samples (125) used in the calibration. ::: ::: The algorithms are available from World Wide Web: hhtp:\\newton.foodsci.kvl.dk\foodtech. <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> We propose conditional mutual information (cMI) as a new similarity measure for nonrigid image registration. We start from a 3D joint histogram incorporating, besides the reference and floating intensity dimensions, also a spatial dimension expressing the location of the joint intensity pair in the reference image. cMI is calculated as the expectation value of the conditional mutual information between the reference and floating intensities given the spatial distribution. ::: ::: Validation experiments were performed comparing cMI and global MI on artificial CT/MR registrations and registrations complicated with a strong bias field; both a Parzen window and generalised partial volume kernel were used for histogram construction. In both experiments, cMI significantly outperforms global MI. Moreover, cMI is compared to global MI for the registration of three patient CT/MR datasets, using overlap and centroid distance as validation measure. The best results are obtained using cMI. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> The application of sensor technology has brought considerable interest in the area of image fusion. Written by leading experts in the field, this book brings together in one volume the most recent algorithms, design techniques and applications in the topical field of image fusion. The applications are drawn from military, medical and civilian areas and give practical advice and pointers to the development of future applications in a variety of fields.This book will be an invaluable resource to R&D engineers, academic researchers and system developers requiring the most up-to-date and complete information on image fusion algorithms, design architectures and applications.*Brings together the latest algorithms, design techniques and applications in the hot area of Image Fusion*A large number of applications from military, medical and civilian fields give practical advice on how to develop future applications.* Combines theory and practice to create a unique point of reference <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> A novel model is presented to learn bimodally informative structures from audio-visual signals. The signal is represented as a sparse sum of audio-visual kernels. Each kernel is a bimodal function consisting of synchronous snippets of an audio waveform and a spatio-temporal visual basis function. To represent an audio-visual signal, the kernels can be positioned independently and arbitrarily in space and time. The proposed algorithm uses unsupervised learning to form dictionaries of bimodal kernels from audio-visual material. The basis functions that emerge during learning capture salient audio-visual data structures. In addition, it is demonstrated that the learned dictionary can be used to locate sources of sound in the movie frame. Specifically, in sequences containing two speakers, the algorithm can robustly localize a speaker even in the presence of severe acoustic and visual distracters. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> Abstract Some fields where calibration of multi-way data is required, such as hyphenated chromatography, can suffer of high inaccuracy when traditional N-PLS is used, due to the presence of shifts or peak shape changes in one of the modes. To overcome this problem, a new regression method for multi-way data called SCREAM (Shifted Covariates REgression Analysis for Multi-way data), which is based on a combination of PARAFAC2 and principal covariates regression (PCovR), is proposed. In particular, the algorithm combines a PARAFAC2 decomposition of the X array and a PCovR-like way of computing the regression coefficients, analogously to what has been described by Smilde and Kiers (A.K. Smilde and H.A.L. Kiers, 1999) in the case of other multi-way PCovR models. The method is tested on real and simulated datasets providing good results and performing as well or better than other available regression approaches for multi-way data. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> Abstract. One of the main challenges for meteorological and hydrological modelling is accurate rainfall measurement and mapping across time and space. To date, the most effective methods for large-scale rainfall estimates are radar, satellites, and, more recently, received signal level (RSL) measurements derived from commercial microwave networks (CMNs). While these methods provide improved spatial resolution over traditional rain gauges, they have their limitations as well. For example, wireless CMNs, which are comprised of microwave links (ML), are dependant upon existing infrastructure and the ML' arbitrary distribution in space. Radar, on the other hand, is known in its limitation for accurately estimating rainfall in urban regions, clutter areas and distant locations. In this paper the pros and cons of the radar and ML methods are considered in order to develop a new algorithm for improving rainfall measurement and mapping, which is based on data fusion of the different sources. The integration is based on an optimal weighted average of the two data sets, taking into account location, number of links, rainfall intensity and time step. Our results indicate that, by using the proposed new method, we not only generate more accurate 2-D rainfall reconstructions, compared with actual rain intensities in space, but also the reconstructed maps are extended to the maximum coverage area. By inspecting three significant rain events, we show that our method outperforms CMNs or the radar alone in rain rate estimation, almost uniformly, both for instantaneous spatial measurements, as well as in calculating total accumulated rainfall. These new improved 2-D rainfall maps, as well as the accurate rainfall measurements over large areas at sub-hourly timescales, will allow for improved understanding, initialization, and calibration of hydrological and meteorological models mainly necessary for water resource management and planning. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> Smart rooms are used for a growing number of practical applications. They are often equipped with microphones and cameras allowing acoustic and visual tracking of persons. For that, the geometry of the sensors has to be calibrated. In this paper, a method is introduced that calibrates the microphone arrays by using the visual localization of a speaker at a small number of fixed positions. By matching the positions to the direction of arrival (DoA) estimates of the microphone arrays, their absolute position and orientation are derived. Data from a reverberant smart room is used to show that the proposed method can estimate the absolute geometry with about 0.1 m and 2 precision. The calibration is good enough for acoustic and multi modal tracking applications and eliminates the need for dedicated calibration measures by using the tracking data itself. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> Pansharpening aims at fusing a multispectral and a panchromatic image, featuring the result of the processing with the spectral resolution of the former and the spatial resolution of the latter. In the last decades, many algorithms addressing this task have been presented in the literature. However, the lack of universally recognized evaluation criteria, available image data sets for benchmarking, and standardized implementations of the algorithms makes a thorough evaluation and comparison of the different pansharpening techniques difficult to achieve. In this paper, the authors attempt to fill this gap by providing a critical description and extensive comparisons of some of the main state-of-the-art pansharpening methods. In greater details, several pansharpening algorithms belonging to the component substitution or multiresolution analysis families are considered. Such techniques are evaluated through the two main protocols for the assessment of pansharpening results, i.e., based on the full- and reduced-resolution validations. Five data sets acquired by different satellites allow for a detailed comparison of the algorithms, characterization of their performances with respect to the different instruments, and consistency of the two validation procedures. In addition, the implementation of all the pansharpening techniques considered in this paper and the framework used for running the simulations, comprising the two validation procedures and the main assessment indexes, are collected in a MATLAB toolbox that is made available to the community. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> A Bayesian framework is proposed to define flexible coupling models for joint decompositions of data sets. Under this framework, a solution to the joint decomposition can be cast in terms of a maximum a posteriori estimator. Examples of joint posterior distributions are provided, including general Gaussian priors and non Gaussian coupling priors. Then simulations are reported and show the effectiveness of this approach to fuse information from data sets, which are inherently of different size due to different time resolution of the measurement devices. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Challenges at the Acquisition and Observation Level <s> We focus on the problem of analyzing corpora composed of irregularly sampled multi-rate heterogeneous temporal data. We propose a novel convolutive multi-rate factorization model for extracting multi-modal patterns from such multi-rate data. Our model builds up on previously proposed multi-view coupled nonnegative matrix factorization techniques, and extends them by accounting for heterogeneous sample rates and enabling the patterns to have a duration. We illustrate the proposed methodology on the joint study of audiovisual data for speech analysis. <s> BIB011
Challenge IV-A.1: Noncommensurability: As explained in Sections I and II, a key motivation for multimodality is that different instruments are sensitive to different physical phenomena, and consequently, report on different aspects of the underlying processes. A natural outcome is that the raw measurements may be represented by different types of physical units that do not commute. This situation is known as noncommensurability. Numerous examples of noncommensurable data fusion scenarios were given in Section II. Allowing noncommensurable data sets to inform each other and interact is probably the first and foremost task that one encounters in a large number of multimodal data fusion scenarios BIB005 . Challenge IV-A.2: Different Resolutions: It is most natural that different types of acquisition methods and observation setups provide data at different sampling points, and often at very disparate resolutions. The specific type of challenge that is associated with this property varies according to the task. Consequently, solutions are diverse. In some cases, different resolutions may be associated with various types of uncertainty, as explained in Section IV-B. Below, we list some scenarios in which challenges related to different resolutions occur. Data with different resolutions is a prevalent challenge in multimodal image fusion BIB003 , as well as many other imaging techniques. For example, EEG has an excellent temporal but low spatial resolution, whereas fMRI has a fine spatial resolution but a very large integration time (Example II-B.1). In remote sensing (Example II-C.1), a common task is ''pan-sharpening'' [13, Ch. 9], BIB009 : merging a high-spatial, low-spectral (single band) resolution panchromatic image with a lower spatial, higher spectral (several bands) resolution multispectral image, in order to generate a new synthetic image that has both the higher spectral and spatial resolution of the two. In audiovisual applications, the temporal resolution of the signals differs by orders of magnitude. An audio signal is usually sampled at several kilohertz whereas the video signal is typically sampled at 15-60 Hz BIB004 (Example II-A.1). In meteorological monitoring (Example II-C.2), each modality has very distinct spatial and temporal resolutions; this is probably the reason why solutions based on data integration BIB007 (see Section V-A) are preferred. Different sampling schemes in coupled matrix and tensor decompositions are discussed, e.g., in BIB010 and BIB011 . Challenge IV-A.3: Incompatible Size: In practical situations, it is quite rare that different data sets contain exactly the same number of data samples. As explained in Section IV-B, this incompatibility may be associated with various types of uncertainty. Data size incompatibility may be due to a different number of samples at each observational mode, as explained in Challenge IV-A.2. Among possible causes are different acquisition techniques and experimental setups. The difference in size becomes even more acute if data sets are arrays of different orders BIB005 ,  , as is often encountered in chemometrics, metabolomics [e.g., Fig. 2(a) ], and psychometrics, among others. Challenge IV-A.4: Alignment and Registration: Registration is the task of aligning several data sets, usually images, on the same coordinate system. Registration is particularly challenging when 3-D biomedical imaging techniques are involved (Example II-B.1). In the first scenario, images of the same subject are taken at different times using the same imaging technique. The difficulty arises from the fact that each image has some bias and spatial distortion w.r.t. the others since the patient is never precisely in the same position. In this case, image registration usually relies on the basic assumption that image intensities are linearly correlated  . This assumption, however, is much less likely in the second scenario, for multimodal images. Consider, for example, registration of modalities that convey anatomical information with others that report on functional and metabolic activity. Naturally, the information conveyed by each modality is inherently of different physical nature. Other complicating factors include different types of noise, spatial distortions, varying contrasts, and different positions of the imaging instruments. One approach uses information theory and maximizes mutual information  , BIB002 . In remote sensing (Example II-C.1), images of the same area are taken by different types of instruments, e.g., airborne SAR and satellite-borne LiDAR, and possibly at different times and conditions, e.g., before and after landscape-changing events such as natural disasters. In principle, one can use the global positioning system (GPS) for aligning the images. However, even the GPS signal has a finite spatial precision. In biomedical imaging, the BOLD signal, to which fMRI is sensitive, has a large integration time and thus a delay w.r.t. EEG. This leads to noninstantaneous coupling, even if the measurements themselves are perfectly synchronized. Calibration can be interpreted as a special case of alignment and registration using two sets of measurements, and thus it can be considered as a form of data fusion. Calibration is a major task in chemometrics, where it is often achieved via regression methods. Frequently used models such as multiway partial least squares (PLS) BIB001 and PARAFAC  are less adequate when the underlying profiles change shape from sample to sample. A regression method that can accommodate such variability in multiway arrays is proposed in BIB006 , and a multimodal audiovisual calibration technique is given in BIB008 . The advantage of the proposed solution is that it is based on direction of arrival estimation, an easier task than microphone-based time-difference-of-arrival estimation, which requires strict synchronization between microphones. The challenge of automatic calibration of audiovisual sensors (and others) in the context of HMI (Example II-A.2) is discussed in [10, Sec. V.C].