Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> In most data fusion systems, the information extracted from each image or sensor (either numerical or symbolic) is represented as a degree of belief in an event with real values, generally in [0,1], taking in this way into account the imprecise, uncertain and incomplete nature of information. The combination of such degrees of belief is performed through numerical fusion operators. A very large variety of such operators has been proposed in the literature. We propose a classification of these operators issued from different data fusion theories (probabilities, fuzzy sets, possibilities, Dempster- Shafer, etc.) w.r.t. their behaviour. Three classes are thus defined: context independent constant behaviour operators, which have the same behaviour (compromise, disjunction or conjunction) whatever the values to be combined, context independent variable behaviour operators (whose behaviour depends on the combined values), and context dependent operators (where the result depends also on some global knowledge like conflict, reliability of sources). This classification provides a guide for choosing an operator in a given problem. This choice can then be refined from the desired properties of the operators, from their decisiveness, and by examining how they deal with conflictual situations. These aspects are illustrated on simple examples in multispectral satellite imaging. In particular, we stress the interest of the third class for classification problems. <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> A two-scale theoretical description outlines relationships between brain current sources and the resulting extracranial electric field, recorded as EEG. Finding unknown sources of EEG, the so-calledg “inverse problem”, is discussed in general terms, with emphasis on the fundamental non-uniqueness of inverse solutions. Hemodynamic signatures, measured with fMRI, are expressed as voxel integrals to facilitate comparisons with EEG. Two generally distinct cell groups (1 and 2), generating EEG and fMRI signals respectively, are embedded within the much broader class of synaptic action fields. Cell groups 1 and 2 may or may not overlap in specific experiments. Implications of this incomplete overlap for co-registration studies are considered. Each experimental measure of brain function is generally sensitive to a different kind of source activity and to different spatial and temporal scales. Failure to appreciate such distinctions can exacerbate conflicting views of brain function that emphasize either global integration or functional localization. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> In recent years, the principles of network science have increasingly been applied to the study of the brain's structural and functional organization. Bullmore and Sporns review this growing field of research and discuss its contributions to our understanding of brain function. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> Microphones and cameras have been extensively used to observe and detect human activity and to facilitate natural modes of interaction between humans and intelligent systems. Human brain processes the audio and video modal- ities, extracting complementary and robust information from them. Intelligent systems with audiovisual sensors should be capable of achieving similar goals. The audiovisual information fusion strategy is a key component in designing such systems. In this paper, we exclusively survey the fusion techniques used in various audiovisual information fusion tasks. The fusion strategy used tends to depend mainly on the model, probabi- listic or otherwise, used in the particular task to process sensory information to obtain higher level semantic informa- tion. The models themselves are task oriented. In this paper, we describe the fusion strategies and the corresponding models used in audiovisual tasks such as speech recognition, tracking, biometrics, affective state recognition, and meeting scene analysis. We also review the challenges and existing solutions and also unresolved or partially resolved issues in these fields. Specifically, we discuss established and upcoming work in hierarchical fusion strategies and cross-modal learning tech- niques, identifying these as critical areas of research in the future development of intelligent systems. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> Functional magnetic resonance imaging (fMRI) data and electroencephalography (EEG) data provide complementary spatio-temporal information about brain function. Methods to couple the relative strengths of these modalities usually involve two stages: first forming a feature set from each dataset based on one criterion followed by exploration of connections among the features using a second criterion. We propose a data fusion method for simultaneously acquired fMRI and EEG data that combines these steps using a single criterion for finding the cross-modality associations and performing source separation. Using multi-set canonical correlation analysis (M-CCA), we obtain a decomposition of the two modalities, into spatial maps for fMRI data and a corresponding temporal evolution for EEG data, based on trial-to-trial covariation across the two modalities. Additionally, the analysis is performed on data from a group of subjects in order to make group inferences about the covariation across modalities. Being multivariate, the proposed method facilitates the study of brain connectivity along with localization of brain function. M-CCA can be easily extended to incorporate different data types and additional modalities. We demonstrate the promise of the proposed method in finding covarying trial-to-trial amplitude modulations (AMs) in an auditory task involving implicit pattern learning. The results show approximately linear decreasing trends in AMs for both modalities and the corresponding spatial activations occur mainly in motor, frontal, temporal, inferior parietal, and orbito-frontal areas that are linked both to sensory function as well as learning and expectation—all of which match activations related to the presented paradigm. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> Simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) provide complementary noninvasive information of brain activity, and EEG/fMRI fusion can achieve higher spatiotemporal resolution than each modality separately. This focuses on independent component analysis (ICA)-based EEG/fMRI fusion. In order to appreciate the issues, we first describe the potential and limitations of the developed fusion approaches: fMRI-constrained EEG imaging, EEG-informed fMRI analysis, and symmetric fusion. We then outline some newly developed hybrid fusion techniques using ICA and the combination of data-/model-driven methods, with special mention of the spatiotemporal EEG/fMRI fusion (STEFF). Finally, we discuss the current trend in methodological development and the existing limitations for extrapolating neural dynamics. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> There has been an ever-increasing interest in multi-disciplinary research on multisensor data fusion technology, driven by its versatility and diverse areas of application. Therefore, there seems to be a real need for an analytical review of recent developments in the data fusion domain. This paper proposes a comprehensive review of the data fusion state of the art, exploring its conceptualizations, benefits, and challenging aspects, as well as existing methodologies. In addition, several future directions of research in the data fusion community are highlighted and described. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> Abstract. One of the main challenges for meteorological and hydrological modelling is accurate rainfall measurement and mapping across time and space. To date, the most effective methods for large-scale rainfall estimates are radar, satellites, and, more recently, received signal level (RSL) measurements derived from commercial microwave networks (CMNs). While these methods provide improved spatial resolution over traditional rain gauges, they have their limitations as well. For example, wireless CMNs, which are comprised of microwave links (ML), are dependant upon existing infrastructure and the ML' arbitrary distribution in space. Radar, on the other hand, is known in its limitation for accurately estimating rainfall in urban regions, clutter areas and distant locations. In this paper the pros and cons of the radar and ML methods are considered in order to develop a new algorithm for improving rainfall measurement and mapping, which is based on data fusion of the different sources. The integration is based on an optimal weighted average of the two data sets, taking into account location, number of links, rainfall intensity and time step. Our results indicate that, by using the proposed new method, we not only generate more accurate 2-D rainfall reconstructions, compared with actual rain intensities in space, but also the reconstructed maps are extended to the maximum coverage area. By inspecting three significant rain events, we show that our method outperforms CMNs or the radar alone in rain rate estimation, almost uniformly, both for instantaneous spatial measurements, as well as in calculating total accumulated rainfall. These new improved 2-D rainfall maps, as well as the accurate rainfall measurements over large areas at sub-hourly timescales, will allow for improved understanding, initialization, and calibration of hydrological and meteorological models mainly necessary for water resource management and planning. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> The separation of speech signals measured at multiple microphones in noisy and reverberant environments using only the audio modality has limitations because there is generally insufficient information to fully discriminate the different sound sources. Humans mitigate this problem by exploiting the visual modality, which is insensitive to background noise and can provide contextual information about the audio scene. This advantage has inspired the creation of the new field of audiovisual (AV) speech source separation that targets exploiting visual modality alongside the microphone measurements in a machine. Success in this emerging field will expand the application of voice-based machine interfaces, such as Siri, the intelligent personal assistant on the iPhone and iPad, to much more realistic settings and thereby provide more natural human?machine interfaces. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Level of Data Fusion <s> The 2013 Data Fusion Contest organized by the Data Fusion Technical Committee (DFTC) of the IEEE Geoscience and Remote Sensing Society aimed at investigating the synergistic use of hyperspectral and Light Detection And Ranging (LiDAR) data. The data sets distributed to the participants during the Contest, a hyperspectral imagery and the corresponding LiDAR-derived digital surface model (DSM), were acquired by the NSF-funded Center for Airborne Laser Mapping over the University of Houston campus and its neighboring area in the summer of 2012. This paper highlights the two awarded research contributions, which investigated different approaches for the fusion of hyperspectral and LiDAR data, including a combined unsupervised and supervised classification scheme, and a graph-based method for the fusion of spectral, spatial, and elevation information. <s> BIB011
At first thought, it may seem that fusing multiple data sets at the raw-data level should always yield the best inference, since there would be no loss of information. In practice, however, due to the complex and largely unknown nature of the underlying phenomena (Section II), various complicating factors (Section IV), as well as the specific research question (Sections I and II), it may turn out to be more useful to fuse the data sets at a higher level of abstraction BIB008 , and after certain simplification and reduction steps. The procedures listed below precede the actual fusion of the data. Therefore, they are related to the preprocessing stage. Naturally, the choice of analytical model is influenced by decisions taken at this point. The first strategy that we mention is data integration. It implies parallel processing pipelines for each modality, followed by a decision-making step. Integration is a common approach to deal with heterogeneous data. When modalities are completely noncommensurable (Challenge IV-A.1), as with remote sensing techniques that report on material content versus others that report on 3-D structures (Example II-C.1), integration becomes a natural choice, and is often related to classification tasks. Integration can be done via soft decision, using optimal weights, as in the fusion of data from wireless microwave sensor networks and radar, for rainfall measurement and mapping BIB009 (Example II-C.2). Bullmore and Sporns BIB003 study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities, and fuse them using a graph-theoretical framework. Data integration may be preferred when modality-specific information carries more weight compared with the shared information, as argued for the joint analysis of EEG-fMRI in  (Example II-B.1). A framework to choose between alternative soft decision strategies in the presence of multiple sensor outputs, given various assumptions on uncertainty or partial knowledge, confidence levels, reliability, and conflicts, in a data fusion context, is given in BIB001 . Due to its simplicity, and relative stability since it allows to rely on well-established methods from single-modal data analysis, a large number of existing data fusion approaches are still based on decision-level fusion. Pros and cons to data integration are further discussed in BIB004 . A second type of data fusion strategy is processing modalities sequentially, where one (or more) modality(ies) is used to constrain another. Mathematically, this amounts to using one modality to restrict the number of degrees of freedom, and thus the set of possible solutions, in another. A sequential approach makes sense when one modality has better quality in terms of the information that it conveys, in a certain respect, as in certain audiovisual scenarios BIB005 , BIB010 (Example II-A.1), as well as in the fMRI-constrained solution for the otherwiseunderdetermined, ill-posed EEG inverse problem  , BIB002 (Example II-B.1) . In this paper, we focus on a third strategy, true fusion, that lets modalities fully interact and inform each other as claimed in Section I. True fusion is also characterized by assigning a symmetric role to all modalities, i.e., not sequential. The data fusion models mentioned in Section III fall into this category, as well as most of the models that we mention in the rest of this section. Within ''true fusion,'' there are varying degrees. True fusion using high-level features. In this approach, the dimensionality is significantly reduced by associating each modality with a small number of variables. High-level features are often univariate. Examples include standard variation, skewness, ratio of active voxels, other variables which concisely summarize statistics, or geometrical and other properties. In this case, inference is typically of classification type. Examples include multisensor BIB008 , HMI BIB005 , and remote sensing BIB011 applications. True fusion using multivariate features. Unlike highlevel features, this approach leaves the data sufficiently multivariate within each modality (which now is in feature form) such that data in each modality can fully interact BIB004 ,  . In neuroimaging, common features are taskrelated spatial maps from fMRI, gray matter images from sMRI, and event-related potential (ERP) from EEG, extracted for each subject BIB004 ,  ,  . In audiovisual applications, features often correspond to speech spectral coefficients and visual cues such as lip contours or speaker's presence in the scene BIB010 . True fusion using the data as is, or with minimal reduction. In fact, working with features implies a twostep approach: in the first step, features are computed using a certain criterion; in the second step, features are fused using a different, second criterion. An approach that merges the two, and thus expected to better exploit the whole raw data, is proposed in BIB006 for the fusion of fMRI and EEG. A remote sensing application in which it is natural to work with raw data is pan-sharpening (explained in Challenge IV-A.2). Here, acquisition conditions are favorable since the two sensors (multispectral and pan) acquire data over the same area, with the same angle of view and simultaneously, and the modalities are commensurable. Features, at different levels, may accommodate heterogeneities across modalities, such as different types of uncertainty and noncommensurability (Section IV). Features may significantly reduce the number of samples involved, i.e., allow compression. Example V-A.1 illustrates this point. It also serves as a conclusion to the discussion on the strategy for data fusion by showing how different levels of features can be used for varying data fusion purposes. For further discussion on features and choosing the right level of data fusion, see, e.g., BIB004 , BIB007 ,  (biomedical imaging) and BIB005 ,  