Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Concepts of correlation and regression may be applied not only to ordinary one-dimensional variates but also to variates of two or more dimensions. Marksmen side by side firing simultaneous shots at targets, so that the deviations are in part due to independent individual errors and in part to common causes such as wind, provide a familiar introduction to the theory of correlation; but only the correlation of the horizontal components is ordinarily discussed, whereas the complex consisting of horizontal and vertical deviations may be even more interesting. The wind at two places may be compared, using both components of the velocity in each place. A fluctuating vector is thus matched at each moment with another fluctuating vector. The study of individual differences in mental and physical traits calls for a detailed study of the relations between sets of correlated variates. For example the scores on a number of mental tests may be compared with physical measurements on the same persons. The questions then arise of determining the number and nature of the independent relations of mind and body shown by these data to exist, and of extracting from the multiplicity of correlations in the system suitable characterizations of these independent relations. As another example, the inheritance of intelligence in rats might be studied by applying not one but s different mental tests to N mothers and to a daughter of each <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> This paper is concerned with the study of similarities and differences in factor structures between different groups. A common situation occurs when a battery of tests has been administered to samples of examinees from several populations.A very general model is presented, in which any parameter in the factor analysis models (factor loadings, factor variances, factor covariances, and unique variances) for the different groups may be assigned an arbitrary value or constrained to be equal to some other parameter. Given such a specification, the model is estimated by the maximum likelihood method yielding a large samplex2 of goodness of fit. By computing several solutions under different specifications one can test various hypotheses.The method is capable of dealing with any degree of invariance, from the one extreme, where nothing is invariant, to the other extreme, where everything is invariant. Neither the number of tests nor the number of common factors need to be the same for all groups, but to be at all interesting, it is assumed that there is a common core of tests in each battery that is the same or at least content-wise comparable. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> The present paper discusses several methods for (simultaneous) component analysis of scores of two or more groups of individuals on the same variables. Some existing methods are discussed, and a new method (SCA-S) is developed for simultaneous component analysis in such a way that for each set essentially the same component structure is found (SCA-S). This method is compared to alternative methods for analysing such data which employ the same component weights matrix (SCA-W) or the same pattern matrix (SCA-P) across data sets. Among these methods, SCA-W always explains the highest amount of variance, SCA-S the lowest, and SCA-P takes the position in between. These explained variances can be compared to the amount of variance explained by separate PCAs. Implications of such fit differences are discussed. In addition, it is shown how, for cases where SCA-S does not fit well, one can use SCA-W (and SCA-P) to find out if and how correlational structures differ. Finally, some attention is paid to facilitating the interpretation of an SCA-S solution. Like the other SCA methods, SCA-S has rotational freedom. This rotational freedom is exploited in a specially designed simple structure rotation technique for SCA-S. This technique is illustrated on an empirical data set. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> A new multiway regression method called N-way partial least squares (N-PLS) is presented. The emphasis is on the three-way PLS version (tri-PLS), but it is shown how to extend the algorithm to higher orders. The developed algorithm is superior to unfolding methods, primarily owing to a stabilization of the decomposition. This stabilization potentially gives increased interpretability and better predictions. The algorithm is fast compared with e.g. PARAFAC, because it consists of solving eigenvalue problems. ::: ::: An example of the developed algorithm taken from the sugar industry is shown and compared with unfold-PLS. Fluorescence excitation—emission matrices (EEMs) are measured on white sugar solutions and used to predict the ash content of the sugar. The predictions are comparable by the two methods, but there is a clear difference in the interpretability of the two solutions. Also shown is a simulated example of EEMs with very noisy measurements and a low relative signal from the analyte of interest. The predictions from unfold-PLS are almost twice as bad as from tri-PLS despite the large number of samples (125) used in the calibration. ::: ::: The algorithms are available from World Wide Web: hhtp:\\newton.foodsci.kvl.dk\foodtech. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We consider the problem of using a large unlabeled sample to boost performance of a learning algorit,hrn when only a small set of labeled examples is available. In particular, we consider a problem setting motivated by the task of learning to classify web pages, in which the description of each example can be partitioned into two distinct views. For example, the description of a web page can be partitioned into the words occurring on that page, and the words occurring in hyperlinks t,hat point to that page. We assume that either view of the example would be sufficient for learning if we had enough labeled data, but our goal is to use both views together to allow inexpensive unlabeled data to augment, a much smaller set of labeled examples. Specifically, the presence of two distinct views of each example suggests strategies in which two learning algorithms are trained separately on each view, and then each algorithm’s predictions on new unlabeled examples are used to enlarge the training set of the other. Our goal in this paper is to provide a PAC-style analysis for this setting, and, more broadly, a PAC-style framework for the general problem of learning from both labeled and unlabeled data. We also provide empirical results on real web-page data indicating that this use of unlabeled examples can lead to significant improvement of hypotheses in practice. *This research was supported in part by the DARPA HPKB program under contract F30602-97-1-0215 and by NSF National Young investigator grant CCR-9357793. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. TO copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. COLT 98 Madison WI USA Copyright ACM 1998 l-58113-057--0/98/ 7...%5.00 92 Tom Mitchell School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3891 mitchell+@cs.cmu.edu <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We describe a comparative mathematical framework for two genome-scale expression data sets. This framework formulates expression as superposition of the effects of regulatory programs, biological processes, and experimental artifacts common to both data sets, as well as those that are exclusive to one data set or the other, by using generalized singular value decomposition. This framework enables comparative reconstruction and classification of the genes and arrays of both data sets. We illustrate this framework with a comparison of yeast and human cell-cycle expression data sets. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Abstract Event-related potential (ERP) studies of the brain's response to infrequent, target (oddball) stimuli elicit a sequence of physiological events, the most prominent and well studied being a complex, the P300 (or P3) peaking approximately 300 ms post-stimulus for simple stimuli and slightly later for more complex stimuli. Localization of the neural generators of the human oddball response remains challenging due to the lack of a single imaging technique with good spatial and temporal resolution. Here, we use independent component analyses to fuse ERP and fMRI modalities in order to examine the dynamics of the auditory oddball response with high spatiotemporal resolution across the entire brain. Initial activations in auditory and motor planning regions are followed by auditory association cortex and motor execution regions. The P3 response is associated with brainstem, temporal lobe, and medial frontal activity and finally a late temporal lobe “evaluative” response. We show that fusing imaging modalities with different advantages can provide new information about the brain. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Abstract We discuss model-free analysis of multisubject or multisession FMRI data by extending the single-session probabilistic independent component analysis model (PICA; Beckmann and Smith, 2004. IEEE Trans. on Medical Imaging, 23 (2) 137–152) to higher dimensions. This results in a three-way decomposition that represents the different signals and artefacts present in the data in terms of their temporal, spatial, and subject-dependent variations. The technique is derived from and compared with parallel factor analysis (PARAFAC; Harshman and Lundy, 1984. In Research methods for multimode data analysis, chapter 5, pages 122–215. Praeger, New York). Using simulated data as well as data from multisession and multisubject FMRI studies we demonstrate that the tensor PICA approach is able to efficiently and accurately extract signals of interest in the spatial, temporal, and subject/session domain. The final decompositions improve upon PARAFAC results in terms of greater accuracy, reduced interference between the different estimated sources (reduced cross-talk), robustness (against deviations of the data from modeling assumptions and against overfitting), and computational speed. On real FMRI ‘activation’ data, the tensor PICA approach is able to extract plausible activation maps, time courses, and session/subject modes as well as provide a rich description of additional processes of interest such as image artefacts or secondary activation patterns. The resulting data decomposition gives simple and useful representations of multisubject/multisession FMRI data that can aid the interpretation and optimization of group FMRI studies beyond what can be achieved using model-based analysis techniques. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We present a new approach to independent component analysis (ICA) by extending the formulation of univariate source signals to multivariate source signals. The new approach is termed independent vector analysis (IVA). In the model, we assume that linear mixing model exists in each dimension separately, and the latent sources are independent of the others. In contrast to ICA, the sources are random vectors, not just single variables, which means the elements of a random vector are closely related to the others. Thus, we assume the dependency between the elements of a source vector. In this manner, we define dependence between vectors as Kullback-Leibler divergence between the total joint probability of vectors and the product of marginal probabilities of vectors. Then, the model allows independence between multivariate source signals represented as random vectors, and dependence between the source signals within the vector representation. The proposed vector density model can for example capture variance dependencies within a vector source signal. There are several applications of this new formulation. In the separation of acoustic sources, the algorithm mitigates the permutation problem, i.e. the usual ICA algorithms applied in to the frequency domain mixture data suffer from the unknown permutation of the output signals. Although there are several engineering solutions to fix this problem after the ICA stage, the proposed method provides a natural solution to the problem by capturing the inherent dependencies of the acoustic signals. It therefore avoids the permutation problem and allows the separation of sources in very challenging environments for many sound sources. <s> BIB011 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We describe the use of a higher-order singular value decomposition (HOSVD) in transforming a data tensor of genes × “x-settings,” that is, different settings of the experimental variable x × “y-settings,” which tabulates DNA microarray data from different studies, to a “core tensor” of “eigenarrays” × “x-eigengenes” × “y-eigengenes.” Reformulating this multilinear HOSVD such that it decomposes the data tensor into a linear superposition of all outer products of an eigenarray, an x- and a y-eigengene, that is, rank-1 “subtensors,” we define the significance of each subtensor in terms of the fraction of the overall information in the data tensor that it captures. We illustrate this HOSVD with an integration of genome-scale mRNA expression data from three yeast cell cycle time courses, two of which are under exposure to either hydrogen peroxide or menadione. We find that significant subtensors represent independent biological programs or experimental phenomena. The picture that emerges suggests that the conserved genes YKU70, MRE11, AIF1, and ZWF1, and the processes of retrotransposition, apoptosis, and the oxidative pentose phosphate pathway that these genes are involved in, may play significant, yet previously unrecognized, roles in the differential effects of hydrogen peroxide and menadione on cell cycle progression. A genome-scale correlation between DNA replication initiation and RNA transcription, which is equivalent to a recently discovered correlation and might be due to a previously unknown mechanism of regulation, is independently uncovered. <s> BIB012 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Abstract Independent component analysis (ICA) is becoming increasingly popular for analyzing functional magnetic resonance imaging (fMRI) data. While ICA has been successfully applied to single-subject analysis, the extension of ICA to group inferences is not straightforward and remains an active topic of research. Current group ICA models, such as the GIFT [Calhoun, V.D., Adali, T., Pearlson, G.D., Pekar, J.J., 2001. A method for making group inferences from functional MRI data using independent component analysis. Hum. Brain Mapp. 14, 140–151.] and tensor PICA [Beckmann, C.F., Smith, S.M., 2005. Tensorial extensions of independent component analysis for multisubject FMRI analysis. Neuroimage 25, 294–311.], make different assumptions about the underlying structure of the group spatio-temporal processes and are thus estimated using algorithms tailored for the assumed structure, potentially leading to diverging results. To our knowledge, there are currently no methods for assessing the validity of different model structures in real fMRI data and selecting the most appropriate one among various choices. In this paper, we propose a unified framework for estimating and comparing group ICA models with varying spatio-temporal structures. We consider a class of group ICA models that can accommodate different group structures and include existing models, such as the GIFT and tensor PICA, as special cases. We propose a maximum likelihood (ML) approach with a modified Expectation–Maximization (EM) algorithm for the estimation of the proposed class of models. Likelihood ratio tests (LRT) are presented to compare between different group ICA models. The LRT can be used to perform model comparison and selection, to assess the goodness-of-fit of a model in a particular data set, and to test group differences in the fMRI signal time courses between subject subgroups. Simulation studies are conducted to evaluate the performance of the proposed method under varying structures of group spatio-temporal processes. We illustrate our group ICA method using data from an fMRI study that investigates changes in neural processing associated with the regular practice of Zen meditation. <s> BIB013 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In this work, we introduce a simple and effective scheme to achieve joint blind source separation (BSS) of multiple datasets using multi-set canonical correlation analysis (M-CCA) [1]. We first propose a generative model of joint BSS based on the correlation of latent sources within and between datasets. We specify source separability conditions, and show that, when the conditions are satisfied, the group of corresponding sources from each dataset can be jointly extracted by M-CCA through maximization of correlation among the extracted sources. We compare source separation performance of the M-CCA scheme with other joint BSS methods and demonstrate the superior performance of the M-CCA scheme in achieving joint BSS for a large number of datasets, group of corresponding sources with heterogeneous correlation values, and complex-valued sources with circular and non-circular distributions. We apply M-CCA to analysis of functional magnetic resonance imaging (fMRI) data from multiple subjects and show its utility in estimating meaningful brain activations from a visuomotor task. <s> BIB014 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> A novel model is presented to learn bimodally informative structures from audio-visual signals. The signal is represented as a sparse sum of audio-visual kernels. Each kernel is a bimodal function consisting of synchronous snippets of an audio waveform and a spatio-temporal visual basis function. To represent an audio-visual signal, the kernels can be positioned independently and arbitrarily in space and time. The proposed algorithm uses unsupervised learning to form dictionaries of bimodal kernels from audio-visual material. The basis functions that emerge during learning capture salient audio-visual data structures. In addition, it is demonstrated that the learned dictionary can be used to locate sources of sound in the movie frame. Specifically, in sequences containing two speakers, the algorithm can robustly localize a speaker even in the presence of severe acoustic and visual distracters. <s> BIB015 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB016 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> One of the simplest (and earliest) approaches to blind source separation is to estimate the mixing matrix from the generalized eigenvalue decomposition (GEVD), or Exact Joint Diagonalization, of two “target-matrices.” In a second-order statistics (SOS) framework, these target-matrices are two different correlation matrices (e.g., at different lags, taken over different time-intervals, etc.), attempting to capture the diversity of the sources (e.g., diverse spectra, different nonstationarity profiles, etc.). More generally, such matrix pairs can be constructed as “generalized correlation” matrices, whose structure is prescribed by two selected “association-matrices.” In this paper, we provide a small-errors performance analysis of GEVD-based separation in such SOS frameworks. We derive explicit expressions for the resulting interference-to-source ratio (ISR) matrix in terms of the association-matrices and of the sources' temporal covariance matrices. The validity of our analysis is illustrated in simulation. <s> BIB017 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In this paper, we show that the joint blind source separation (JBSS) problem can be solved by jointly diagonalizing cumulant matrices of any order higher than one, including the correlation matrices and the fourth-order cumulant matrices. We introduce an efficient iterative generalized joint diagonalization algorithm such that a series of orthogonal procrustes problems are solved. We present simulation results to show that the new algorithms can reliably solve the permutation ambiguity in JBSS and that they offer superior performance compared with existing multiset canonical correlation analysis (MCCA) and independent vector analysis (IVA) approaches. Experiment on real-world data for separation of fetal heartbeat in electrocardiogram (ECG) data demonstrates a new application of JBSS, and the success of the new algorithms for a real-world problem. <s> BIB018 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In many areas of science, research questions imply the analysis of a set of coupled data blocks, with, for instance, each block being an experimental unit by variable matrix, and the variables being the same in all matrices. To obtain an overall picture of the mechanisms that play a role in the different data matrices, the information in these matrices needs to be integrated. This may be achieved by applying a data-analytic strategy in which a global model is fitted to all data matrices simultaneously, as in some forms of simultaneous component analysis (SCA). Since such a strategy implies that all data entries, regardless the matrix they belong to, contribute equally to the analysis, it may obfuscate the overall picture of the mechanisms underlying the data when the different data matrices are subject to different amounts of noise. One way out is to downweight entries from noisy data matrices in favour of entries from less noisy matrices. Information regarding the amount of noise that is present in each matrix, however, is, in most cases, not available. To deal with these problems, in this paper a novel maximum-likelihood-based simultaneous component analysis method, referred to as MxLSCA, is proposed. Being a stochastic extension of SCA, in MxLSCA the amount of noise in each data matrix is estimated and entries from noisy data matrices are downweighted. Both in an extensive simulation study and in an application to data stemming from cross-cultural emotion psychology, it is shown that the novel MxLSCA strategy outperforms the SCA strategy with respect to disclosing the mechanisms underlying the coupled data. <s> BIB019 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We introduce a factor analysis model that summarizes the dependencies between observed variable groups, instead of dependencies between individual variables as standard factor analysis does. A group may correspond to one view of the same set of objects, one of many data sets tied by co-occurrence, or a set of alternative variables collected from statistics tables to measure one property of interest. We show that by assuming groupwise sparse factors, active in a subset of the sets, the variation can be decomposed into factors explaining relationships between the sets and factors explaining away set-specific variation. We formulate the assumptions in a Bayesian model providing the factors, and apply the model to two data analysis tasks, in neuroimaging and chemical systems biology. <s> BIB020 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In recent years, neuroimaging studies have increasingly been acquiring multiple modalities of data and searching for task- or disease-related changes in each modality separately. A major challenge in analysis is to find systematic approaches for fusing these differing data types together to automatically find patterns of related changes across multiple modalities, when they exist. Independent Component Analysis (ICA) is a popular unsupervised learning method that can be used to find the modes of variation in neuroimaging data across a group of subjects. When multimodal data is acquired for the subjects, ICA is typically performed separately on each modality, leading to incompatible decompositions across modalities. Using a modular Bayesian framework, we develop a novel "Linked ICA" model for simultaneously modelling and discovering common features across multiple modalities, which can potentially have completely different units, signal- and contrast-to-noise ratios, voxel counts, spatial smoothnesses and intensity distributions. Furthermore, this general model can be configured to allow tensor ICA or spatially-concatenated ICA decompositions, or a combination of both at the same time. Linked ICA automatically determines the optimal weighting of each modality, and also can detect single-modality structured components when present. This is a fully probabilistic approach, implemented using Variational Bayes. We evaluate the method on simulated multimodal data sets, as well as on a real data set of Alzheimer's patients and age-matched controls that combines two very different types of structural MRI data: morphological data (grey matter density) and diffusion data (fractional anisotropy, mean diffusivity, and tensor mode). <s> BIB021 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Coupled nonnegative matrix factorization (CNMF) unmixing is proposed for the fusion of low-spatial-resolution hyperspectral and high-spatial-resolution multispectral data to produce fused data with high spatial and spectral resolutions. Both hyperspectral and multispectral data are alternately unmixed into end member and abundance matrices by the CNMF algorithm based on a linear spectral mixture model. Sensor observation models that relate the two data are built into the initialization matrix of each NMF unmixing procedure. This algorithm is physically straightforward and easy to implement owing to its simple update rules. Simulations with various image data sets demonstrate that the CNMF algorithm can produce high-quality fused data both in terms of spatial and spectral domains, which contributes to the accurate identification and classification of materials observed at a high spatial resolution. <s> BIB022 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In many research domains different pieces of information are collected regarding the same set of objects. Each piece of information constitutes a data block, and all these (coupled) blocks have the object mode in common. When analyzing such data, an important aim is to obtain an overall picture of the structure underlying the whole set of coupled data blocks. A further challenge consists of accounting for the differences in information value that exist between and within (i.e., between the objects of a single block) data blocks. To tackle these issues, analysis techniques may be useful in which all available pieces of information are integrated and in which at the same time noise heterogeneity is taken into account. For the case of binary coupled data, however, only methods exist that go for a simultaneous analysis of all data blocks but that do not account for noise heterogeneity. Therefore, in this paper, the SIMCLAS model, being a Hierarchical Classes model for the simultaneous analysis of coupled binary two-way matrices, is presented. In this model, noise heterogeneity between and within the data blocks is accounted for by downweighting entries from noisy blocks/objects within a block. In a simulation study it is shown that (1) the SIMCLAS technique recovers the underlying structure of coupled data to a very large extent, and (2) the SIMCLAS technique outperforms a Hierarchical Classes technique in which all entries contribute equally to the analysis (i.e., noise homogeneity within and between blocks). The latter is also demonstrated in an application of both techniques to empirical data on categorization of semantic concepts. <s> BIB023 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> In this paper, we consider the joint blind source separation (JBSS) problem and introduce a number of algorithms to solve the JBSS problem using the independent vector analysis (IVA) framework. Source separation of multiple datasets simultaneously is possible when the sources within each and every dataset are independent of one another and each source is dependent on at most one source within each of the other datasets. In addition to source separation, the IVA framework solves an essential problem of JBSS, namely the identification of the dependent sources across the datasets. We propose to use the multivariate Gaussian source prior to achieve JBSS of sources that are linearly dependent across datasets. Analysis within the paper yields the local stability conditions, nonidentifiability conditions, and induced Cramer-Rao lower bound on the achievable interference to source ratio for IVA with multivariate Gaussian source priors. Additionally, by exploiting a novel nonorthogonal decoupling of the IVA cost function we introduce both Newton and quasi-Newton optimization algorithms for the general IVA framework. <s> BIB024 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Canonical polyadic decomposition (CPD) of a third-order tensor is decomposition in a minimal number of rank-1 tensors. We call an algorithm algebraic if it is guaranteed to find the decomposition when it is exact and if it relies only on standard linear algebra (essentially sets of linear equations and matrix factorizations). The known algebraic algorithms for the computation of the CPD are limited to cases where at least one of the factor matrices has full column rank. In this paper we present an algebraic algorithm for the computation of the CPD in cases where none of the factor matrices has full column rank. In particular, we show that if the famous Kruskal condition holds, then the CPD can be found algebraically. <s> BIB025 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Given multivariate multiblock data (e.g., subjects nested in groups are measured on multiple variables), one may be interested in the nature and number of dimensions that underlie the variables, and in differences in dimensional structure across data blocks. To this end, clusterwise simultaneous component analysis (SCA) was proposed which simultaneously clusters blocks with a similar structure and performs an SCA per cluster. However, the number of components was restricted to be the same across clusters, which is often unrealistic. In this paper, this restriction is removed. The resulting challenges with respect to model estimation and selection are resolved. <s> BIB026 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Abstract Three-way three-mode data are collected regularly in scientific research and yield information on the relation between three sets of entities. To summarize the information in such data, three-way component methods like CANDECOMP/PARAFAC (CP) and Tucker3 are often used. When applying CP and Tucker3 in empirical practice, one should be cautious, however, because they rely on very strict assumptions. We argue that imposing these assumptions may obscure interesting structural information included in the data and may lead to substantive conclusions that are appropriate for some part of the data only. As a way out, this paper demonstrates that this structural information may be elegantly captured by means of component methods for multiset data, that is to say, simultaneous component analysis (SCA) and its clusterwise extension (clusterwise SCA). <s> BIB027 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Abstract Recent technological advances enable us to collect huge amounts of data from multiple sources. Jointly analyzing such multi-relational data from different sources, i.e., data fusion (also called multi-block, multi-view or multi-set data analysis), often enhances knowledge discovery. For instance, in metabolomics, biological fluids are measured using a variety of analytical techniques such as Liquid Chromatography–Mass Spectrometry and Nuclear Magnetic Resonance Spectroscopy. Data measured using different analytical methods may be complementary and their fusion may help in the identification of chemicals related to certain diseases. Data fusion has proved useful in many fields including social network analysis, collaborative filtering, neuroscience and bioinformatics. In this paper, unlike many studies demonstrating the success of data fusion, we explore the limitations as well as the advantages of data fusion. We formulate data fusion as a coupled matrix and tensor factorization (CMTF) problem, which jointly factorizes multiple data sets in the form of higher-order tensors and matrices by extracting a common latent structure from the shared mode. Using numerical experiments on simulated and real data sets, we assess the performance of coupled analysis compared to the analysis of a single data set in terms of missing data estimation and demonstrate cases where coupled analysis outperforms analysis of a single data set and vice versa. <s> BIB028 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Incorporating domain-specific side information via coupled factorization models is useful in source separation applications. Coupled models can easily incorporate information from source modalities with different statistical properties by estimating shared factors via divergence minimization. Here, it is useful to use mixed divergences, a specific divergence for each modality. However, this extra freedom requires choosing the correct divergence as well as an optimal weighting mechanism to select the relative ‘importance’. In this paper, we present an approach for determining the relative weights, framed as dispersion parameter estimation, based on an inference framework introduced previously as Generalized Coupled Tensor Factorization (GCTF). The dispersion parameters play a key role on inference as they form a balance between the information obtained from multimodal observations. We tackle the problem of optimal weighting by maximum likelihood exploiting the relation between β-divergences and the family of Tweedie distributions. We demonstrate the usefulness of our approach on a drum source separation application. <s> BIB029 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> This study deals with missing link prediction, the problem of predicting the existence of missing connections between entities of interest. We approach the problem as filling in missing entries in a relational dataset represented by several matrices and multiway arrays, that will be simply called tensors. Consequently, we address the link prediction problem by data fusion formulated as simultaneous factorization of several observation tensors where latent factors are shared among each observation. Previous studies on joint factorization of such heterogeneous datasets have focused on a single loss function (mainly squared Euclidean distance or Kullback---Leibler-divergence) and specific tensor factorization models (CANDECOMP/PARAFAC and/or Tucker). However, in this paper, we study various alternative tensor models as well as loss functions including the ones already studied in the literature using the generalized coupled tensor factorization framework. Through extensive experiments on two real-world datasets, we demonstrate that (i) joint analysis of data from multiple sources via coupled factorization significantly improves the link prediction performance, (ii) selection of a suitable loss function and a tensor factorization model is crucial for accurate missing link prediction and loss functions that have not been studied for link prediction before may outperform the commonly-used loss functions, (iii) joint factorization of datasets can handle difficult cases, such as the cold start problem that arises when a new entity enters the dataset, and (iv) our approach is scalable to large-scale data. <s> BIB030 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> A new generalized multilinear regression model, termed the higher order partial least squares (HOPLS), is introduced with the aim to predict a tensor (multiway array) Y from a tensor X through projecting the data onto the latent space and performing regression on the corresponding latent variables. HOPLS differs substantially from other regression models in that it explains the data by a sum of orthogonal Tucker tensors, while the number of orthogonal loadings serves as a parameter to control model complexity and prevent overfitting. The low-dimensional latent space is optimized sequentially via a deflation operation, yielding the best joint subspace approximation for both X and Y. Instead of decomposing X and Y individually, higher order singular value decomposition on a newly defined generalized cross-covariance tensor is employed to optimize the orthogonal loadings. A systematic comparison on both synthetic data and real-world decoding of 3D movement trajectories from electrocorticogram signals demonstrate the advantages of HOPLS over the existing methods in terms of better predictive ability, suitability to handle small sample sizes, and robustness to noise. <s> BIB031 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> This paper deals with joint blind source separation (JBSS) of multidimensional components. JBSS extends classical BSS to simultaneously resolve several BSS problems by assuming statistical dependence between latent sources across mixtures. JBSS offers some significant advantages over BSS, such as identifying more than one Gaussian white stationary source within a mixture. Multidimensional BSS extends classical BSS to deal with a more general and more flexible model within each mixture: the sources can be partitioned into groups exhibiting dependence within a given group but independence between two different groups. Motivated by various applications, we present a model that is inspired by both extensions. We derive an algorithm that achieves asymptotically the minimal mean square error (MMSE) in the estimation of Gaussian multidimensional components. We demonstrate the superior performance of this model over a two-step approach, in which JBSS, which ignores the multidimensional structure, is followed by a clustering step. <s> BIB032 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We introduce a Bayesian extension of the tensor factorization problem to multiple coupled tensors. For a single tensor it reduces to standard PARAFAC-type Bayesian factorization, and for two tensors it is the first Bayesian Tensor Canonical Correlation Analysis method. It can also be seen to solve a tensorial extension of the recent Group Factor Analysis problem. The method decomposes the set of tensors to factors shared by subsets of the tensors, and factors private to individual tensors, and does not assume orthogonality. For a single tensor, the method empirically outperforms existing methods, and we demonstrate its performance on multiple tensor factorization tasks in toxicogenomics and functional neuroimaging. <s> BIB033 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> This paper presents an algorithm based on sparse representation for fusing hyperspectral and multispectral images. The observed images are assumed to be obtained by spectral or spatial degradations of the high resolution hyperspectral image to be recovered. Based on this forward model, the fusion process is formulated as an inverse problem whose solution is determined by optimizing an appropriate criterion. To incorporate additional spatial information within the objective criterion, a regularization term is carefully designed,relying on a sparse decomposition of the scene on a set of dictionaryies. The dictionaries and the corresponding supports of active coding coef�cients are learned from the observed images. Then, conditionally on these dictionaries and supports, the fusion problem is solved by iteratively optimizing with respect to the target image (using the alternating direction method of multipliers) and the coding coefcients. Simulation results demonstrate the ef�ciency of the proposed fusion method when compared with the state-of-the-art. <s> BIB034 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> Data often consists of multiple diverse modalities. For example, images are tagged with textual information and videos are accompanied by audio. Each modality is characterized by having distinct statistical properties. We propose a Deep Boltzmann Machine for learning a generative model of such multimodal data. We show that the model can be used to create fused representations by combining features across modalities. These learned representations are useful for classification and information retrieval. By sampling from the conditional distributions over each data modality, it is possible to create these representations even when some data modalities are missing. We conduct experiments on bimodal image-text and audio-video data. The fused representation achieves good classification results on the MIR-Flickr data set matching or outperforming other deep models as well as SVM based models that use Multiple Kernel Learning. We further demonstrate that this multimodal model helps classification and retrieval even when only unimodal data is available at test time. <s> BIB035 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods. <s> BIB036 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB037 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> We introduce Bayesian multi-tensor factorization, a model that is the first Bayesian formulation for joint factorization of multiple matrices and tensors. The research problem generalizes the joint matrix-tensor factorization problem to arbitrary sets of tensors of any depth, including matrices, can be interpreted as unsupervised multi-view learning from multiple tensors, and can be generalized to relax the usual trilinear tensor factorization assumptions. The result is a factorization of the set of tensors into factors shared by any subsets of the tensors, and factors private to individual tensors. We demonstrate the performance against existing baselines in multiple tensor factorization tasks in structural toxicogenomics and functional neuroimaging. <s> BIB038 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> A Bayesian framework is proposed to define flexible coupling models for joint decompositions of data sets. Under this framework, a solution to the joint decomposition can be cast in terms of a maximum a posteriori estimator. Examples of joint posterior distributions are provided, including general Gaussian priors and non Gaussian coupling priors. Then simulations are reported and show the effectiveness of this approach to fuse information from data sets, which are inherently of different size due to different time resolution of the measurement devices. <s> BIB039 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> C. Analytical Framework <s> This paper deals with a novel generalization of classical blind source separation (BSS) in two directions. First, relaxing the constraint that the latent sources must be statistically independent. This generalization is well-known and sometimes termed independent subspace analysis (ISA). Second, jointly analyzing several ISA problems, where the link is due to statistical dependence among corresponding sources in different mixtures. When the data are one-dimensional, i.e., multiple classical BSS problems, this model, known as independent vector analysis (IVA), has already been studied. In this paper, we combine IVA with ISA and term this new model joint independent subspace analysis (JISA). We provide full performance analysis of JISA, including closed-form expressions for minimal mean square error (MSE), Fisher information and Cramer–Rao lower bound, in the separation of Gaussian data. The derived MSE applies also for non-Gaussian data, when only second-order statistics are used. We generalize previously known results on IVA, including its ability to uniquely resolve instantaneous mixtures of real Gaussian stationary data, and having the same arbitrary permutation at all mixtures. Numerical experiments validate our theoretical results and show the gain with respect to two competing approaches that either use a finer block partition or a different norm. <s> BIB040
Certain data fusion approaches rely on existing theoretical analytical frameworks that have originally been devised for nonfusion applications, at least not explicitly. Such are ICA and algebraic-based methods such as PARAFAC, generalized eigenvalue decomposition (GEVD), GSVD, and HOSVD, as will be elaborated below. These methods have been around for a while and there is a large body of works that has been dedicated to their computation. Data fusion approaches that rely on these well-established, widely known methods are often more easily accepted and integrated within the research communities. However, these approaches may not be able to exploit the full range of diversity in the data, and thus, more advanced data fusion methods may be preferred. Below, we briefly review some of the analytical approaches that have been proposed for data fusion. Well-known matrix and tensor factorizations can be used for data fusion. In BIB007 , Alter et al. use GSVD for the comparison of genetic data from two different organisms. In BIB012 , HOSVD is proposed for the analysis of data from different studies. In the presence of two data sets, or in a noise-free scenario, many matrix-and tensorbased methods can be reformulated as GEVD BIB025 . This holds for various BSS closed-form solutions BIB017 , CCA  Ch. 12] and its multiset extension  , BIB014 , joint BSS BIB018 , and coupled tensor decompositions  . As explained in Section V-B1, algebraic (possibly approximate) solutions to models with ''soft'' links often exist. Certain data fusion methods concatenate or reorganize data such that it can be analyzed by a classical ICA algorithm. Such is the case in joint ICA BIB009 and group ICA  ,  . These models can thus be solved using any existing ICA approach  . Guo et al. BIB013 propose a tensor extension to group ICA  ,  and to tensor ICA BIB010 that can accommodate different group structures. Parallel ICA  and IVA BIB011 ,  - BIB018 (Example III-D.1) jointly solve several separate ICA problems by exploiting covariations or statistical dependence, respectively. CCA BIB001 , its extension to multiple data sets  , as well as one of the approaches to LMWCA BIB036 , search for maximal correlation, or other second-order-based relationships, between variables. Certain methods minimize the Frobenius norm between model and data. In the presence of additive white Gaussian noise, this amounts to maximum likelihood (ML). Further considerations associated with this choice of norm are given in BIB037 Sec. II] . This type of optimization is used in group NMF  , coupled NMF BIB022 , certain SCA-based methods BIB026 , BIB003 - BIB027 , and numerous coupled tensor decompositions; see, e.g., BIB037 , BIB028 ,  , and references therein. In some cases, it may be better to tailor loss functions individually to each data set, and use norms other than Frobenius BIB016 , BIB019 . Such is the case in the GCTF framework BIB029 , BIB030 , for example. An ML framework can accommodate data sets with different noise patterns. Such are flexible simultaneous FA BIB002 and ML-based SCA BIB019 , BIB023 . ML underlies certain noiseless stochastic models, e.g., SOS-IVA BIB024 and JISA BIB032 - BIB040 . Regression provides another solution to data fusion. Regression searches for latent factors that best explain the covariance between two sets of observations. We mention PLS  ,  and its multilinear extensions N-way PLS  , BIB004 and higher order PLS (HOPLS) BIB031 . Bayesian group FA BIB020 , its tensor extension BIB038 , BIB033 , coupled matrix and tensor decompositions with flexible coupling BIB039 , and certain methods for the fusion of hyperspectral and multispectral images  ,  , rely on a Bayesian framework for the decomposition. Certain tensor extensions of ICA rely on a probabilistic Bayesian framework BIB021 . In BIB034 , fusion of hyperspectral and multispectral images is achieved via dictionary learning and sparse coding. This is also the underlying technique of BIB015 for learning bimodal structure in audiovisual data. Beal et al. BIB008 , BIB006 use probabilistic generative models, also termed graphical models, in order to fuse audio and video models into a single probabilistic graphical model. Lederman and Talmon  use an alternating-diffusion method for manifold learning that extracts a common source of variability from measurements in multiple sensors, where all sensors observe the same physical phenomenon but have different sensorspecific effects. Combining labeled and unlabeled data via cotraining is described in BIB005 . A survey of techniques for multiview machine learning can be found in  . A multimodal deep-learning method for information retrieval from bimodal data consisting of images and text is described in BIB035 .