Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> The authors present general descriptive models for spatiotemporal MEG (magnetoencephalogram) data and show the separability of the linear moment parameters and nonlinear location parameters in the MEG problem. A forward model with current dipoles in a spherically symmetric conductor is used as an example: however, other more advanced MEG models, as well as many EEG (electroencephalogram) models, can also be formulated in a similar linear algebra framework. A subspace methodology and computational approach to solving the conventional least-squares problem is presented. A new scanning approach, equivalent to the statistical MUSIC method, is also developed. This subspace method scans three-dimensional space with a one-dipole model, making it computationally feasible to scan the complete head volume. > <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> Magnetoencephalography (MEG) is a noninvasive technique for investigating neuronal activity in the living human brain. The time resolution of the method is better than 1 ms and the spatial discrimination is, under favorable circumstances, 2--3 mm for sources in the cerebral cortex. In MEG studies, the weak 10 fT--1 pT magnetic fields produced by electric currents flowing in neurons are measured with multichannel SQUID (superconducting quantum interference device) gradiometers. The sites in the cerebral cortex that are activated by a stimulus can be found from the detected magnetic-field distribution, provided that appropriate assumptions about the source render the solution of the inverse problem unique. Many interesting properties of the working human brain can be studied, including spontaneous activity and signal processing following external stimuli. For clinical purposes, determination of the locations of epileptic foci is of interest. The authors begin with a general introduction and a short discussion of the neural basis of MEG. The mathematical theory of the method is then explained in detail, followed by a thorough description of MEG instrumentation, data analysis, and practical construction of multi-SQUID devices. Finally, several MEG experiments performed in the authors' laboratory are described, covering studies of evoked responses and of spontaneousmore » activity in both healthy and diseased brains. Many MEG studies by other groups are discussed briefly as well.« less <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> Abstract This paper introduces a new tool for functional neuroimage analysis: partial least squares (PLS). It is unique as a multivariate method in its choice of emphasis for analysis, that being the covariance between brain images and exogenous blocks representing either the experiment design or some behavioral measure. What emerges are spatial patterns of brain activity that represent the optimal association between the images and either of the blocks. This process differs substantially from other multivariate methods in that rather than attempting to predict the individual values of the image pixels, PLS attempts to explain the relation between image pixels and task or behavior. Data from a face encoding and recognition PET rCBF study are used to illustrate two types of PLS analysis: an activation analysis of task with images and a brain–behavior analysis. The commonalities across the two analyses are suggestive of a general face memory network differentially engaged during encoding and recognition. PLS thus serves as an important extension by extracting new information from imaging data that is not accessible through other currently used univariate and multivariate image analysis tools. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> In the last few years, the functional brain imaging community has witnessed numerous efforts (and perhaps even more discussion) directed at multimodality data fusion: combining high-quality localization information provided by the hemodynamic-based brain imaging methods such as PET and fMRI with highquality temporal data generated by the electromagnetic-based techniques such as EEG and MEG [Dale and Halgren, 2001]. The article in this issue by Vitacco et al. [2002] provides an interesting research effort aimed at this problem, one that forces us to confront a number of critical questions about the entire data fusion enterprise. Almost every neuroscientist, and certainly every functional neuroimager, tries in one way or another to combine data from multiple methods. Three distinct approaches are used. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> A two-scale theoretical description outlines relationships between brain current sources and the resulting extracranial electric field, recorded as EEG. Finding unknown sources of EEG, the so-calledg “inverse problem”, is discussed in general terms, with emphasis on the fundamental non-uniqueness of inverse solutions. Hemodynamic signatures, measured with fMRI, are expressed as voxel integrals to facilitate comparisons with EEG. Two generally distinct cell groups (1 and 2), generating EEG and fMRI signals respectively, are embedded within the much broader class of synaptic action fields. Cell groups 1 and 2 may or may not overlap in specific experiments. Implications of this incomplete overlap for co-registration studies are considered. Each experimental measure of brain function is generally sensitive to a different kind of source activity and to different spatial and temporal scales. Failure to appreciate such distinctions can exacerbate conflicting views of brain function that emphasize either global integration or functional localization. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> Abstract Data may now be recorded concurrently from EEG and functional MRI, using the Simultaneous Imaging for Tomographic Electrophysiology (SITE) method. As yet, there is no established means to integrate the analysis of the combined data set. Recognizing that the hemodynamically convolved time-varying EEG spectrum, S, is intrinsically multidimensional in space, frequency, and time motivated us to use multiway Partial Least-Squares (N-PLS) analysis to decompose EEG (independent variable) and fMRI (dependent variable) data uniquely as a sum of “atoms”. Each EEG atom is the outer product of spatial, spectral, and temporal signatures and each fMRI atom the product of spatial and temporal signatures. The decomposition was constrained to maximize the covariance between corresponding temporal signatures of the EEG and fMRI. On all data sets, three components whose spectral peaks were in the theta, alpha, and gamma bands appeared; only the alpha atom had a significant temporal correlation with the fMRI signal. The spatial distribution of the alpha-band atom included parieto-occipital cortex, thalamus, and insula, and corresponded closely to that reported by Goldman et al. [NeuroReport 13(18) (2002) 2487] using a more conventional analysis. The source reconstruction from EEG spatial signature showed only the parieto-occipital sources. We interpret these results to indicate that some electrical sources may be intrinsically invisible to scalp EEG, yet may be revealed through conjoint analysis of EEG and fMRI data. These results may also expose brain regions that participate in the control of brain rhythms but may not themselves be generators. As of yet, no single neuroimaging method offers the optimal combination of spatial and temporal resolution; fusing fMRI and EEG meaningfully extends the spatio-temporal resolution and sensitivity of each method. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> This review presents the current state of the art regarding multiparametric magnetic resonance (MR) imaging of prostate cancer. Technical requirements and clinical indications for the use of multiparametric MR imaging in detection, localization, characterization, staging, biopsy guidance, and active surveillance of prostate cancer are discussed. Although reported accuracies of the separate and combined multiparametric MR imaging techniques vary for diverse clinical prostate cancer indications, multiparametric MR imaging of the prostate has shown promising results and may be of additional value in prostate cancer localization and local staging. Consensus on which technical approaches (fi eld strengths, sequences, use of an endorectal coil) and combination of multiparametric MR imaging techniques should be used for specifi c clinical indications remains a challenge. Because guidelines are currently lacking, suggestions for a general minimal protocol for multiparametric MR imaging of the prostate based on the literature and the authors’ experience are presented. Computer programs that allow evaluation of the various components of a multiparametric MR imaging examination in one view should be developed. In this way, an integrated interpretation of anatomic and functional MR imaging techniques in a multiparametric MR imaging examination is possible. Education and experience of specialist radiologists are essential for correct interpretation of multiparametric prostate MR imaging fi ndings. Supportive techniques, such as computer-aided diagnosis are needed to obtain a fast, cost-effective, easy, and more reproducible prostate cancer diagnosis out of more and more complex multiparametric MR imaging data. q RSNA, 2011 <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> In this study we classify short echo-time brain magnetic resonance spectroscopic imaging (MRSI) data by applying a model-based canonical correlation analyses algorithm and by using, as prior knowledge, multimodal sources of information coming from high-resolution magic angle spinning (HR-MAS), MRSI and magnetic resonance imaging. The potential and limitations of fusing in vivo and ex vivo nuclear magnetic resonance sources to detect brain tumors is investigated. We present various modalities for multimodal data fusion, study the effect and the impact of using multimodal information for classifying MRSI brain glial tumors data and analyze which parameters influence the classification results by means of extensive simulation and in vivo studies. Special attention is drawn to the possibility of considering HR-MAS data as a complementary dataset when dealing with a lack of MRSI data needed to build a classifier. Results show that HR-MAS information can have added value in the process of classifying MRSI data. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> Simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) provide complementary noninvasive information of brain activity, and EEG/fMRI fusion can achieve higher spatiotemporal resolution than each modality separately. This focuses on independent component analysis (ICA)-based EEG/fMRI fusion. In order to appreciate the issues, we first describe the potential and limitations of the developed fusion approaches: fMRI-constrained EEG imaging, EEG-informed fMRI analysis, and symmetric fusion. We then outline some newly developed hybrid fusion techniques using ICA and the combination of data-/model-driven methods, with special mention of the spatiotemporal EEG/fMRI fusion (STEFF). Finally, we discuss the current trend in methodological development and the existing limitations for extrapolating neural dynamics. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> During atrial fibrillation (AF), atrial activity (AA) on the surface ECG consists of a pattern of quasi-periodic oscillations (f-waves), which are related to the electrical activation of the atrial substrate. However, to date no direct comparison between the extracted f-wave pattern in surface recordings and specific activation sites within the atria has been carried out. In the present study, one reference intracardiac modality consisting of a bipolar electrogram (EGM) recorded from the left atrial appendage (LAA) is exploited for the first time to guide the extraction of LAA electrical activity from standard 12-lead ECG recordings. A periodic component analysis (πCA) technique is employed for this task. The performance of the proposed multimodal extraction technique is compared to that obtained employing a noninvasive, fully blind approach, namely, independent component analysis (ICA). On a database of 31 AF patients, results suggest that the estimation of LAA activity is indeed possible, even though its contribution to the ECG total power is relatively low. Interestingly, ICA seems to provide a slightly better estimation of LAA activation rate, expressed in terms of dominant frequency (DF). On the other hand, the multimodal invasive approach performs better QRST complex suppression and provides AA waveforms with narrower spectra. <s> BIB011 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> Abstract Purpose There is a need for a seizure-detection system that can be used long-term and in home situations for early intervention and prevention of seizure related side effects including SUDEP (sudden unexpected death in epileptic patients). The gold standard for monitoring epileptic seizures involves video/EEG (electro-encephalography), which is uncomfortable for the patient, as EEG electrodes are attached to the scalp. EEG analysis is also labour-intensive and has yet to be automated and adapted for real-time monitoring. It is therefore usually performed in a hospital setting, for a few days at the most. The goal of this article is to provide an overview of body signals that can be measured, along with corresponding methods, state-of-art research, and commercially available systems, as well as to stress the importance of a good detection system. Method Narrative literature review. Results A range of body signals can be monitored for the purpose of seizure detection. It is particularly interesting to include monitoring of autonomic dysfunction, as this may be an important patho-physiological mechanism of SUDEP, and of movement, as many seizures have a motor component. Conclusion The most effective seizure detection systems are multimodal. Such systems should also be comfortable and low-power. The body signals and modalities on which a system is based should take account of the user's seizure types and personal preferences. <s> BIB012 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Biomedical, Health <s> The objective of brain source imaging consists in reconstructing the cerebral activity everywhere within the brain based on EEG or MEG measurements recorded on the scalp. This requires solving an ill-posed linear inverse problem. In order to restore identifiability, additional hypotheses need to be imposed on the source distribution, giving rise to an impressive number of brain source imaging algorithms. However, a thorough comparison of different methodologies is still missing in the literature. In this paper, we provide an overview of priors that have been used for brain source imaging and conduct a comparative simulation study with seven representative algorithms corresponding to the classes of minimum norm, sparse, tensor-based, subspace-based, and Bayesian approaches. This permits us to identify new benchmark algorithms and promising directions for future research. <s> BIB013
Example II-B.1: Understanding Brain Functionality: Functional brain study deals with understanding how the different elements of the brain take part in various perceptual and cognitive activities. Functional brain study largely relies on noninvasive imaging techniques, whose purpose is to reconstruct a high-resolution spatiotemporal image of the neuronal activity within the brain. The neuronal activity within the brain generates ionic currents that are often modeled as dipoles. These dipoles induce electric and magnetic fields that can be directly recorded by electroencephalography (EEG) and magnetoencephalography (MEG), respectively. In addition, neuronal activity induces changes in magnetization between oxygen-rich and oxygen-poor blood, known as the haemodynamic response. This effect, also called blood-oxygen-leveldependent (BOLD) changes, can be detected by functional magnetic resonance imaging (fMRI). Therefore, fMRI is an indirect measure of neuronal activity. These three modalities register data at regular time intervals and thus reflect temporal dynamics. However, these techniques vary greatly in their spatiotemporal resolutions: EEG and MEG data provide high temporal (in milliseconds) resolution, whereas fMRI images have low temporal (in seconds) resolution. fMRI data are a set of high-resolution 3-D images, taken at regular time intervals, representing the whole volume of the brain of a patient lying in an fMRI scanner. EEG and MEG data are a set of time-series signals reflecting voltage or neuromagnetic field changes recorded at each of the (usually a few dozen of) electrodes attached to the scalp (EEG) or fixed within an MEG scanner helmet. The sensitivity of EEG and MEG to deep-brain signals is limited. In addition, they have different selectivity to signals as a function of brain morphology. Therefore, they provide data at much poorer spatial resolution and do not have access to the full brain volume. Consequently, the spatiotemporal information provided by EEG, MEG, and fMRI is highly complementary. Functional imaging techniques can be complemented by other modalities that convey structural information. For example, structural magnetic resonance imaging (sMRI) and diffusion tensor imaging (DTI) report on the structure of the brain in terms of gray matter, white matter, and cerebrospinal fluid. sMRI is based on nuclear magnetic resonance of water protons. DTI measures the diffusion process of molecules, mainly water, and thus reports also on brain connectivity. Each of these methods is based on different physical principles and is thus sensitive to different types of properties within the brain. In addition, each method has different pros and cons in terms of safety, cost, accuracy, and other parameters. Recent technological advances allow recording data from several functional brain imaging techniques simultaneously BIB005 , BIB010 , thus further motivating advanced data fusion. It is a well-accepted paradigm in neuroscience that EEG and fMRI carry complementary information about brain function BIB005 , BIB004 . However, their very heterogeneous nature and the fact that brain processes are very complicated systems that depend on numerous latent phenomena imply that simultaneously extracting useful information from them is not an evident task. The fact that there is no ground truth is reflected in the very broad range of methods and approaches that are being proposed  ,  , BIB003 , BIB007 , BIB004 -  . Works on biomedical brain imaging often emphasize the exploratory nature of this task. Despite decades of study, the underlying relationship between EEG and fMRI is far from being understood BIB003 ,  , BIB006 ,  . A well-known challenge in brain imaging is the EEG inverse problem. A prevalent assumption is that the measured EEG signal is generated by numerous current dipoles within the brain, and the goal is to localise the origins of this neuronal activity. Often formulated as a linear inverse problem, it is ill-posed: many different spatial current patterns within the skull can give rise to identical measurements BIB001 . In order to make the problem well-conditioned, additional hypotheses are required. A large number of solutions are based on adding various priors to the EEG data BIB013 . Alternatively, an identifiable and unique solution can be obtained using spatial constraints from fMRI  , BIB002 , BIB006 . Example II-B.2: Medical Diagnosis: Various medical conditions such as potentially malignant tumors cannot be diagnosed by a single type of measurement due to many factors such as low sensitivity, low positive predictive values, low specificity (high false-positive), a limited number of spatial samples (as in biopsy), and other limitations of the various assessment techniques. In order to improve the performance of the diagnosis, risk assessment, and therapy options, it is necessary to perform numerous medical assessments based on a broad range of medical diagnostic techniques BIB008 , BIB009 . For example, one can augment physical examination, blood tests, biopsies, static and functional magnetic resonance imaging, with other parameters such as genetic, environmental, and personal risk factors. The question of how to analyze all these simultaneously available resources is largely open. Currently, this task relies mostly on human medical experts. One of the main challenges is the automation of such decision procedures, in order to improve correct interpretation, as well as save costs and time BIB008 . Example II-B.3: Developing Noninvasive Medical Diagnosis Techniques: In some cases, the use of multimodal data fusion is only a first step in the design of a single-modal system. In BIB011 , the challenge is understanding the link between surface and intracardiac electrodes measuring the same atrial fibrillation event and the goal is eventually extracting relevant atrial fibrillation activity using only the noninvasive modality. For this aim, the intracardiac modality is exploited as a reference to guide the extraction of an atrial electrical signal of interest from noninvasive electrocardiography (ECG) recordings. The difficulty lies in the fact that the intracardiac modality provides a rather pure signal whereas the ECG signal is a mixture of the desired signal with other sources, and the mixing model is unknown. Example II-B.4: Smart Patient Monitoring: Health monitoring using multiple types of sensors is drawing increasing attention from modern health services. The goal is to provide a set of noninvasive, nonintrusive, reasonable-cost sensors that allow the patient to run a normal life while providing reliable warnings in real time. Here, we focus on monitoring, predicting, and warning epileptic patients from potentially dangerous seizures BIB012 . The gold standard in monitoring epileptic seizures is combining EEG and video, where EEG is manually analyzed by experts and the whole diagnostic procedure requires a stay of up to several days in a hospital setting. This procedure is expensive, time consuming, and physically inconvenient for the patient. Obviously, it is not practical for daily life. While much effort has already been dedicated to the prediction of epileptic seizures from EEG, with no clear-cut results so far, a considerable proportion of potentially lethal seizures are hardly detectable by EEG at all. Therefore, a primary challenge is to understand the link between epileptic seizures and additional body parameters: movement, breathing, heart rate, and others. Due to the fact that epileptic seizures vary within and across patients, and due to the complex relations between different body systems, it is likely that any such system should rely on more than one modality BIB012 .