Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Concepts of correlation and regression may be applied not only to ordinary one-dimensional variates but also to variates of two or more dimensions. Marksmen side by side firing simultaneous shots at targets, so that the deviations are in part due to independent individual errors and in part to common causes such as wind, provide a familiar introduction to the theory of correlation; but only the correlation of the horizontal components is ordinarily discussed, whereas the complex consisting of horizontal and vertical deviations may be even more interesting. The wind at two places may be compared, using both components of the velocity in each place. A fluctuating vector is thus matched at each moment with another fluctuating vector. The study of individual differences in mental and physical traits calls for a detailed study of the relations between sets of correlated variates. For example the scores on a number of mental tests may be compared with physical measurements on the same persons. The questions then arise of determining the number and nature of the independent relations of mind and body shown by these data to exist, and of extracting from the multiplicity of correlations in the system suitable characterizations of these independent relations. As another example, the inheritance of intelligence in rats might be studied by applying not one but s different mental tests to N mothers and to a daughter of each <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> This paper is concerned with the study of similarities and differences in factor structures between different groups. A common situation occurs when a battery of tests has been administered to samples of examinees from several populations.A very general model is presented, in which any parameter in the factor analysis models (factor loadings, factor variances, factor covariances, and unique variances) for the different groups may be assigned an arbitrary value or constrained to be equal to some other parameter. Given such a specification, the model is estimated by the maximum likelihood method yielding a large samplex2 of goodness of fit. By computing several solutions under different specifications one can test various hypotheses.The method is capable of dealing with any degree of invariance, from the one extreme, where nothing is invariant, to the other extreme, where everything is invariant. Neither the number of tests nor the number of common factors need to be the same for all groups, but to be at all interesting, it is assumed that there is a common core of tests in each battery that is the same or at least content-wise comparable. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> The present paper discusses several methods for (simultaneous) component analysis of scores of two or more groups of individuals on the same variables. Some existing methods are discussed, and a new method (SCA-S) is developed for simultaneous component analysis in such a way that for each set essentially the same component structure is found (SCA-S). This method is compared to alternative methods for analysing such data which employ the same component weights matrix (SCA-W) or the same pattern matrix (SCA-P) across data sets. Among these methods, SCA-W always explains the highest amount of variance, SCA-S the lowest, and SCA-P takes the position in between. These explained variances can be compared to the amount of variance explained by separate PCAs. Implications of such fit differences are discussed. In addition, it is shown how, for cases where SCA-S does not fit well, one can use SCA-W (and SCA-P) to find out if and how correlational structures differ. Finally, some attention is paid to facilitating the interpretation of an SCA-S solution. Like the other SCA methods, SCA-S has rotational freedom. This rotational freedom is exploited in a specially designed simple structure rotation technique for SCA-S. This technique is illustrated on an empirical data set. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We discuss a multilinear generalization of the singular value decomposition. There is a strong analogy between several properties of the matrix and the higher-order tensor decomposition; uniqueness, link with the matrix eigenvalue decomposition, first-order perturbation effects, etc., are analyzed. We investigate how tensor symmetries affect the decomposition and propose a multilinear generalization of the symmetric eigenvalue decomposition for pair-wise symmetric tensors. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> This paper describes two- and multiset canonical correlations analysis (CCA) for data fusion, multisource, multiset, or multitemporal exploratory data analysis. These techniques transform multivariate multiset data into new orthogonal variables called canonical variates (CVs) which, when applied in remote sensing, exhibit ever-decreasing similarity (as expressed by correlation measures) over sets consisting of 1) spectral variables at fixed points in time (R-mode analysis), or 2) temporal variables with fixed wavelengths (T-mode analysis). The CVs are invariant to linear and affine transformations of the original variables within sets which means, for example, that the R-mode CVs are insensitive to changes over time in offset and gain in a measuring device. In a case study, CVs are calculated from Landsat Thematic Mapper (TM) data with six spectral bands over six consecutive years. Both Rand T-mode CVs clearly exhibit the desired characteristic: they show maximum similarity for the low-order canonical variates and minimum similarity for the high-order canonical variates. These characteristics are seen both visually and in objective measures. The results from the multiset CCA R- and T-mode analyses are very different. This difference is ascribed to the noise structure in the data. The CCA methods are related to partial least squares (PLS) methods. This paper very briefly describes multiset CCA-based multiset PLS. Also, the CCA methods can be applied as multivariate extensions to empirical orthogonal functions (EOF) techniques. Multiset CCA is well-suited for inclusion in geographical information systems (GIS). <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We describe a comparative mathematical framework for two genome-scale expression data sets. This framework formulates expression as superposition of the effects of regulatory programs, biological processes, and experimental artifacts common to both data sets, as well as those that are exclusive to one data set or the other, by using generalized singular value decomposition. This framework enables comparative reconstruction and classification of the genes and arrays of both data sets. We illustrate this framework with a comparison of yeast and human cell-cycle expression data sets. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Abstract Event-related potential (ERP) studies of the brain's response to infrequent, target (oddball) stimuli elicit a sequence of physiological events, the most prominent and well studied being a complex, the P300 (or P3) peaking approximately 300 ms post-stimulus for simple stimuli and slightly later for more complex stimuli. Localization of the neural generators of the human oddball response remains challenging due to the lack of a single imaging technique with good spatial and temporal resolution. Here, we use independent component analyses to fuse ERP and fMRI modalities in order to examine the dynamics of the auditory oddball response with high spatiotemporal resolution across the entire brain. Initial activations in auditory and motor planning regions are followed by auditory association cortex and motor execution regions. The P3 response is associated with brainstem, temporal lobe, and medial frontal activity and finally a late temporal lobe “evaluative” response. We show that fusing imaging modalities with different advantages can provide new information about the brain. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> An unsupervised learning algorithm for the separation of sound sources in one-channel music signals is presented. The algorithm is based on factorizing the magnitude spectrogram of an input signal into a sum of components, each of which has a fixed magnitude spectrum and a time-varying gain. Each sound source, in turn, is modeled as a sum of one or more components. The parameters of the components are estimated by minimizing the reconstruction error between the input spectrogram and the model, while restricting the component spectrograms to be nonnegative and favoring components whose gains are slowly varying and sparse. Temporal continuity is favored by using a cost term which is the sum of squared differences between the gains in adjacent frames, and sparseness is favored by penalizing nonzero gains. The proposed iterative estimation algorithm is initialized with random values, and the gains and the spectra are then alternatively updated using multiplicative update rules until the values converge. Simulation experiments were carried out using generated mixtures of pitched musical instrument samples and drum sounds. The performance of the proposed method was compared with independent subspace analysis and basic nonnegative matrix factorization, which are based on the same linear model. According to these simulations, the proposed method enables a better separation quality than the previous algorithms. Especially, the temporal continuity criterion improved the detection of pitched musical sounds. The sparseness criterion did not produce significant improvements <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We describe the use of a higher-order singular value decomposition (HOSVD) in transforming a data tensor of genes × “x-settings,” that is, different settings of the experimental variable x × “y-settings,” which tabulates DNA microarray data from different studies, to a “core tensor” of “eigenarrays” × “x-eigengenes” × “y-eigengenes.” Reformulating this multilinear HOSVD such that it decomposes the data tensor into a linear superposition of all outer products of an eigenarray, an x- and a y-eigengene, that is, rank-1 “subtensors,” we define the significance of each subtensor in terms of the fraction of the overall information in the data tensor that it captures. We illustrate this HOSVD with an integration of genome-scale mRNA expression data from three yeast cell cycle time courses, two of which are under exposure to either hydrogen peroxide or menadione. We find that significant subtensors represent independent biological programs or experimental phenomena. The picture that emerges suggests that the conserved genes YKU70, MRE11, AIF1, and ZWF1, and the processes of retrotransposition, apoptosis, and the oxidative pentose phosphate pathway that these genes are involved in, may play significant, yet previously unrecognized, roles in the differential effects of hydrogen peroxide and menadione on cell cycle progression. A genome-scale correlation between DNA replication initiation and RNA transcription, which is equivalent to a recently discovered correlation and might be due to a previously unknown mechanism of regulation, is independently uncovered. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Relational learning is concerned with predicting unknown values of a relation, given a database of entities and observed relations among entities. An example of relational learning is movie rating prediction, where entities could include users, movies, genres, and actors. Relations encode users' ratings of movies, movies' genres, and actors' roles in movies. A common prediction technique given one pairwise relation, for example a #users x #movies ratings matrix, is low-rank matrix factorization. In domains with multiple relations, represented as multiple matrices, we may improve predictive accuracy by exploiting information from one relation while predicting another. To this end, we propose a collective matrix factorization model: we simultaneously factor several matrices, sharing parameters among factors when an entity participates in multiple relations. Each relation can have a different value type and error distribution; so, we allow nonlinear relationships between the parameters and outputs, using Bregman divergences to measure error. We extend standard alternating projection algorithms to our model, and derive an efficient Newton update for the projection. Furthermore, we propose stochastic optimization methods to deal with large, sparse matrices. Our model generalizes several existing matrix factorization methods, and therefore yields new large-scale optimization algorithms for these problems. Our model can handle any pairwise relational schema and a wide variety of error models. We demonstrate its efficiency, as well as the benefit of sharing parameters among relations. <s> BIB011 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In this work, we introduce a simple and effective scheme to achieve joint blind source separation (BSS) of multiple datasets using multi-set canonical correlation analysis (M-CCA) [1]. We first propose a generative model of joint BSS based on the correlation of latent sources within and between datasets. We specify source separability conditions, and show that, when the conditions are satisfied, the group of corresponding sources from each dataset can be jointly extracted by M-CCA through maximization of correlation among the extracted sources. We compare source separation performance of the M-CCA scheme with other joint BSS methods and demonstrate the superior performance of the M-CCA scheme in achieving joint BSS for a large number of datasets, group of corresponding sources with heterogeneous correlation values, and complex-valued sources with circular and non-circular distributions. We apply M-CCA to analysis of functional magnetic resonance imaging (fMRI) data from multiple subjects and show its utility in estimating meaningful brain activations from a visuomotor task. <s> BIB012 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> A novel model is presented to learn bimodally informative structures from audio-visual signals. The signal is represented as a sparse sum of audio-visual kernels. Each kernel is a bimodal function consisting of synchronous snippets of an audio waveform and a spatio-temporal visual basis function. To represent an audio-visual signal, the kernels can be positioned independently and arbitrarily in space and time. The proposed algorithm uses unsupervised learning to form dictionaries of bimodal kernels from audio-visual material. The basis functions that emerge during learning capture salient audio-visual data structures. In addition, it is demonstrated that the learned dictionary can be used to locate sources of sound in the movie frame. Specifically, in sequences containing two speakers, the algorithm can robustly localize a speaker even in the presence of severe acoustic and visual distracters. <s> BIB013 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB014 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> The number of high-dimensional datasets recording multiple aspects of a single phenomenon is increasing in many areas of science, accompanied by a need for mathematical frameworks that can compare multiple large-scale matrices with different row dimensions. The only such framework to date, the generalized singular value decomposition (GSVD), is limited to two matrices. We mathematically define a higher-order GSVD (HO GSVD) for N$2 matrices Di[R m i|n , each with full column rank. Each matrix is exactly factored as Di=UiSiV T , where V, identical in all factorizations, is obtained from the eigensystem SV=VL of the arithmetic mean S of all pairwise quotients AiA {1 j of the matrices Ai~D T i Di, i?j. We prove that this decomposition extends to higher orders almost all of the mathematical properties of the GSVD. The matrix S is nondefective with V and L real. Its eigenvalues satisfy lk$1. Equality holds if and only if the corresponding eigenvector vk is a right basis vector of equal significance in all matrices Di and Dj, that is si,k/sj,k=1 for all i and j, and the corresponding left basis vector ui,k is orthogonal to all other vectors in Ui for all i. The eigenvalues lk=1, therefore, define the ‘‘common HO GSVD subspace.’’ We illustrate the HO GSVD with a comparison of genome-scale cell-cycle mRNA expression from S. pombe, S. cerevisiae and human. Unlike existing algorithms, a mapping among the genes of these disparate organisms is not required. We find that the approximately common HO GSVD subspace represents the cell-cycle mRNA expression oscillations, which are similar among the datasets. Simultaneous reconstruction in the common subspace, therefore, removes the experimental artifacts, which are dissimilar, from the datasets. In the simultaneous sequence-independent classification of the genes of the three organisms in this common subspace, genes of highly conserved sequences but significantly different cell-cycle peak times are correctly classified. <s> BIB015 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In recent years, neuroimaging studies have increasingly been acquiring multiple modalities of data and searching for task- or disease-related changes in each modality separately. A major challenge in analysis is to find systematic approaches for fusing these differing data types together to automatically find patterns of related changes across multiple modalities, when they exist. Independent Component Analysis (ICA) is a popular unsupervised learning method that can be used to find the modes of variation in neuroimaging data across a group of subjects. When multimodal data is acquired for the subjects, ICA is typically performed separately on each modality, leading to incompatible decompositions across modalities. Using a modular Bayesian framework, we develop a novel "Linked ICA" model for simultaneously modelling and discovering common features across multiple modalities, which can potentially have completely different units, signal- and contrast-to-noise ratios, voxel counts, spatial smoothnesses and intensity distributions. Furthermore, this general model can be configured to allow tensor ICA or spatially-concatenated ICA decompositions, or a combination of both at the same time. Linked ICA automatically determines the optimal weighting of each modality, and also can detect single-modality structured components when present. This is a fully probabilistic approach, implemented using Variational Bayes. We evaluate the method on simulated multimodal data sets, as well as on a real data set of Alzheimer's patients and age-matched controls that combines two very different types of structural MRI data: morphological data (grey matter density) and diffusion data (fractional anisotropy, mean diffusivity, and tensor mode). <s> BIB016 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We introduce a factor analysis model that summarizes the dependencies between observed variable groups, instead of dependencies between individual variables as standard factor analysis does. A group may correspond to one view of the same set of objects, one of many data sets tied by co-occurrence, or a set of alternative variables collected from statistics tables to measure one property of interest. We show that by assuming groupwise sparse factors, active in a subset of the sets, the variation can be decomposed into factors explaining relationships between the sets and factors explaining away set-specific variation. We formulate the assumptions in a Bayesian model providing the factors, and apply the model to two data analysis tasks, in neuroimaging and chemical systems biology. <s> BIB017 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In this paper, we show that the joint blind source separation (JBSS) problem can be solved by jointly diagonalizing cumulant matrices of any order higher than one, including the correlation matrices and the fourth-order cumulant matrices. We introduce an efficient iterative generalized joint diagonalization algorithm such that a series of orthogonal procrustes problems are solved. We present simulation results to show that the new algorithms can reliably solve the permutation ambiguity in JBSS and that they offer superior performance compared with existing multiset canonical correlation analysis (MCCA) and independent vector analysis (IVA) approaches. Experiment on real-world data for separation of fetal heartbeat in electrocardiogram (ECG) data demonstrates a new application of JBSS, and the success of the new algorithms for a real-world problem. <s> BIB018 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In many research domains different pieces of information are collected regarding the same set of objects. Each piece of information constitutes a data block, and all these (coupled) blocks have the object mode in common. When analyzing such data, an important aim is to obtain an overall picture of the structure underlying the whole set of coupled data blocks. A further challenge consists of accounting for the differences in information value that exist between and within (i.e., between the objects of a single block) data blocks. To tackle these issues, analysis techniques may be useful in which all available pieces of information are integrated and in which at the same time noise heterogeneity is taken into account. For the case of binary coupled data, however, only methods exist that go for a simultaneous analysis of all data blocks but that do not account for noise heterogeneity. Therefore, in this paper, the SIMCLAS model, being a Hierarchical Classes model for the simultaneous analysis of coupled binary two-way matrices, is presented. In this model, noise heterogeneity between and within the data blocks is accounted for by downweighting entries from noisy blocks/objects within a block. In a simulation study it is shown that (1) the SIMCLAS technique recovers the underlying structure of coupled data to a very large extent, and (2) the SIMCLAS technique outperforms a Hierarchical Classes technique in which all entries contribute equally to the analysis (i.e., noise homogeneity within and between blocks). The latter is also demonstrated in an application of both techniques to empirical data on categorization of semantic concepts. <s> BIB019 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Coupled nonnegative matrix factorization (CNMF) unmixing is proposed for the fusion of low-spatial-resolution hyperspectral and high-spatial-resolution multispectral data to produce fused data with high spatial and spectral resolutions. Both hyperspectral and multispectral data are alternately unmixed into end member and abundance matrices by the CNMF algorithm based on a linear spectral mixture model. Sensor observation models that relate the two data are built into the initialization matrix of each NMF unmixing procedure. This algorithm is physically straightforward and easy to implement owing to its simple update rules. Simulations with various image data sets demonstrate that the CNMF algorithm can produce high-quality fused data both in terms of spatial and spectral domains, which contributes to the accurate identification and classification of materials observed at a high spatial resolution. <s> BIB020 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Background: In systems biology it is common to obtain for the same set of biological entities information from multiple sources. Examples include expression data for the same set of orthologous genes screened in different organisms and data on the same set of culture samples obtained with different high-throughput techniques. A major challenge is to find the important biological processes underlying the data and to disentangle therein processes common to all data sources and processes distinctive for a specific source. Recently, two promising simultaneous data integration methods have been proposed to attain this goal, namely generalized singular value decomposition (GSVD) and simultaneous component analysis with rotation to common and distinctive components (DISCO-SCA). Results: Both theoretical analyses and applications to biologically relevant data show that: (1) straightforward applications of GSVD yield unsatisfactory results, (2) DISCO-SCA performs well, (3) provided proper pre-processing and algorithmic adaptations, GSVD reaches a performance level similar to that of DISCO-SCA, and (4) DISCO-SCA is directly generalizable to more than two data sources. The biological relevance of DISCO-SCA is illustrated with two applications. First, in a setting of comparative genomics, it is shown that DISCO-SCA recovers a common theme of cell cycle progression and a yeast-specific response to pheromones. The biological annotation was obtained by applying Gene Set Enrichment Analysis in an appropriate way. Second, in an application of DISCO-SCA to metabolomics data for Escherichia coli obtained with two different chemical analysis platforms, it is illustrated that the metabolites involved in some of the biological processes underlying the data are detected by one of the two platforms only; therefore, platforms for microbial metabolomics should be tailored to the biological question. Conclusions: Both DISCO-SCA and properly applied GSVD are promising integrative methods for finding common and distinctive processes in multisource data. Open source code for both methods is provided. © 2012 Van Deun et al. <s> BIB021 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Given multivariate multiblock data (e.g., subjects nested in groups are measured on multiple variables), one may be interested in the nature and number of dimensions that underlie the variables, and in differences in dimensional structure across data blocks. To this end, clusterwise simultaneous component analysis (SCA) was proposed which simultaneously clusters blocks with a similar structure and performs an SCA per cluster. However, the number of components was restricted to be the same across clusters, which is often unrealistic. In this paper, this restriction is removed. The resulting challenges with respect to model estimation and selection are resolved. <s> BIB022 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In many research domains it has become a common practice to rely on multiple sources of data to study the same object of interest. Examples include a systems biology approach to immunology with collection of both gene expression data and immunological readouts for the same set of subjects, and the use of several high-throughput techniques for the same set of fermentation batches. A major challenge is to find the processes underlying such multiset data and to disentangle therein the common processes from those that are distinctive for a specific source. Several integrative methods have been proposed to address this challenge including canonical correlation analysis, simultaneous component analysis, OnPLS, generalized singular value decomposition, DISCO-SCA, and ECO-POWER. To get a better understanding 1) of the methods with respect to finding common and distinctive components and 2) of the relations between these methods, this paper brings the methods together and compares them both on a theoretical level and in terms of analyses of high-dimensional micro-array gene expression data obtained from subjects vaccinated against influenza. <s> BIB023 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Abstract Recent technological advances enable us to collect huge amounts of data from multiple sources. Jointly analyzing such multi-relational data from different sources, i.e., data fusion (also called multi-block, multi-view or multi-set data analysis), often enhances knowledge discovery. For instance, in metabolomics, biological fluids are measured using a variety of analytical techniques such as Liquid Chromatography–Mass Spectrometry and Nuclear Magnetic Resonance Spectroscopy. Data measured using different analytical methods may be complementary and their fusion may help in the identification of chemicals related to certain diseases. Data fusion has proved useful in many fields including social network analysis, collaborative filtering, neuroscience and bioinformatics. In this paper, unlike many studies demonstrating the success of data fusion, we explore the limitations as well as the advantages of data fusion. We formulate data fusion as a coupled matrix and tensor factorization (CMTF) problem, which jointly factorizes multiple data sets in the form of higher-order tensors and matrices by extracting a common latent structure from the shared mode. Using numerical experiments on simulated and real data sets, we assess the performance of coupled analysis compared to the analysis of a single data set in terms of missing data estimation and demonstrate cases where coupled analysis outperforms analysis of a single data set and vice versa. <s> BIB024 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> This study deals with missing link prediction, the problem of predicting the existence of missing connections between entities of interest. We approach the problem as filling in missing entries in a relational dataset represented by several matrices and multiway arrays, that will be simply called tensors. Consequently, we address the link prediction problem by data fusion formulated as simultaneous factorization of several observation tensors where latent factors are shared among each observation. Previous studies on joint factorization of such heterogeneous datasets have focused on a single loss function (mainly squared Euclidean distance or Kullback---Leibler-divergence) and specific tensor factorization models (CANDECOMP/PARAFAC and/or Tucker). However, in this paper, we study various alternative tensor models as well as loss functions including the ones already studied in the literature using the generalized coupled tensor factorization framework. Through extensive experiments on two real-world datasets, we demonstrate that (i) joint analysis of data from multiple sources via coupled factorization significantly improves the link prediction performance, (ii) selection of a suitable loss function and a tensor factorization model is crucial for accurate missing link prediction and loss functions that have not been studied for link prediction before may outperform the commonly-used loss functions, (iii) joint factorization of datasets can handle difficult cases, such as the cold start problem that arises when a new entity enters the dataset, and (iv) our approach is scalable to large-scale data. <s> BIB025 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Matrix factorizations and their extensions to tensor factorizations and decompositions have become prominent techniques for linear and multilinear blind source separation (BSS), especially multiway Independent Component Analysis (ICA), NonnegativeMatrix and Tensor Factorization (NMF/NTF), Smooth Component Analysis (SmoCA) and Sparse Component Analysis (SCA). Moreover, tensor decompositions have many other potential applications beyond multilinear BSS, especially feature extraction, classification, dimensionality reduction and multiway clustering. In this paper, we briefly overview new and emerging models and approaches for tensor decompositions in applications to group and linked multiway BSS/ICA, feature extraction, classification andMultiway Partial Least Squares (MPLS) regression problems. Keywords: Multilinear BSS, linked multiway BSS/ICA, tensor factorizations and decompositions, constrained Tucker and CP models, Penalized Tensor Decompositions (PTD), feature extraction, classification, multiway PLS and CCA. <s> BIB026 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Often data are collected that consist of different blocks that all contain information about the same entities (e.g., items, persons, or situations). In order to unveil both information that is common to all data blocks and information that is distinctive for one or a few of them, an integrated analysis of the whole of all data blocks may be most useful. Interesting classes of methods for such an approach are simultaneous-component and multigroup factor analysis methods. These methods yield dimensions underlying the data at hand. Unfortunately, however, in the results from such analyses, common and distinctive types of information are mixed up. This article proposes a novel method to disentangle the two kinds of information, by making use of the rotational freedom of component and factor models. We illustrate this method with data from a cross-cultural study of emotions. <s> BIB027 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> In many disciplines, data from multiple sources are acquired and jointly analyzed for enhanced knowledge discovery. For instance, in metabolomics, different analytical techniques are used to measure biological fluids in order to identify the chemicals related to certain diseases. It is widely-known that, some of these analytical methods, e.g., LC-MS (Liquid Chromatography - Mass Spectrometry) and NMR (Nuclear Magnetic Resonance) spectroscopy, provide complementary data sets and their joint analysis may enable us to capture a larger proportion of the complete metabolome belonging to a specific biological system. Fusing data from multiple sources has proved useful in many fields including bioinformatics, signal processing and social network analysis. However, identification of common (shared) and individual (unshared) structures across multiple data sets remains a major challenge in data fusion studies. With a goal of addressing this challenge, we propose a novel unsupervised data fusion model. Our contributions are two-fold: (i) We formulate a data fusion model based on joint factorization of matrices and higher-order tensors, which can automatically reveal common and individual components. (ii) We demonstrate that the proposed approach provides promising results in joint analysis of metabolomics data sets consisting of fluorescence and NMR measurements of plasma samples in terms of separation of colorectal cancer patients from controls. <s> BIB028 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We introduce a Bayesian extension of the tensor factorization problem to multiple coupled tensors. For a single tensor it reduces to standard PARAFAC-type Bayesian factorization, and for two tensors it is the first Bayesian Tensor Canonical Correlation Analysis method. It can also be seen to solve a tensorial extension of the recent Group Factor Analysis problem. The method decomposes the set of tensors to factors shared by subsets of the tensors, and factors private to individual tensors, and does not assume orthogonality. For a single tensor, the method empirically outperforms existing methods, and we demonstrate its performance on multiple tensor factorization tasks in toxicogenomics and functional neuroimaging. <s> BIB029 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> This paper presents an algorithm based on sparse representation for fusing hyperspectral and multispectral images. The observed images are assumed to be obtained by spectral or spatial degradations of the high resolution hyperspectral image to be recovered. Based on this forward model, the fusion process is formulated as an inverse problem whose solution is determined by optimizing an appropriate criterion. To incorporate additional spatial information within the objective criterion, a regularization term is carefully designed,relying on a sparse decomposition of the scene on a set of dictionaryies. The dictionaries and the corresponding supports of active coding coef�cients are learned from the observed images. Then, conditionally on these dictionaries and supports, the fusion problem is solved by iteratively optimizing with respect to the target image (using the alternating direction method of multipliers) and the coding coefcients. Simulation results demonstrate the ef�ciency of the proposed fusion method when compared with the state-of-the-art. <s> BIB030 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> This work introduces a new framework for nonnegative matrix factorization (NMF) in multisensor or multimodal data configurations, where taking into account the mutual dependence that exists between the related parallel streams of data is expected to improve performance. In contrast with previous works that focused on co-factorization methods -where some factors are shared by the different modalities-we propose a soft co-factorization scheme which accounts for possible local discrepancies across modalities or channels. This objective is formalized as an optimization problem where concurrent factorizations are jointly performed while being tied by a coupling term that penalizes differences between the related factor matrices associated with different modalities. We provide majorization-minimization (MM) algorithms for three common measures of fit-the squared Euclidean norm, the Kullback-Leibler divergence and the Itakura-Saito divergence-and two possible coupling variants, using either the l1 or the squared Euclidean norm of differences. The approach is shown to achieve promising performance in two audio-related tasks: multimodal speaker diarization using audiovisual data and audio source separation using stereo data. <s> BIB031 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Matrix decompositions such as the eigenvalue decomposition (EVD) or the singular value decomposition (SVD) have a long history in ?signal processing. They have been used in spectral analysis, signal/noise subspace estimation, principal component analysis (PCA), dimensionality reduction, and whitening in independent component analysis (ICA). Very often, the matrix under consideration is the covariance matrix of some observation signals. However, many other kinds of matrices can be encountered in signal processing problems, such as time-lagged covariance matrices, quadratic spatial time-frequency matrices [21], and matrices of higher-order statistics. <s> BIB032 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> Behavioral researchers often obtain information about the same set of entities from different sources. A main challenge in the analysis of such data is to reveal, on the one hand, the mechanisms underlying all of the data blocks under study and, on the other hand, the mechanisms underlying a single data block or a few such blocks only (i.e., common and distinctive mechanisms, respectively). A method called DISCO-SCA has been proposed by which such mechanisms can be found. The goal of this article is to make the DISCO-SCA method more accessible, in particular for applied researchers. To this end, first we will illustrate the different steps in a DISCO-SCA analysis, with data stemming from the domain of psychiatric diagnosis. Second, we will present in this article the DISCO-SCA graphical user interface (GUI). The main benefits of the DISCO-SCA GUI are that it is easy to use, strongly facilitates the choice of model selection parameters (such as the number of mechanisms and their status as being common or distinctive), and is freely available. <s> BIB033 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> A Bayesian framework is proposed to define flexible coupling models for joint decompositions of data sets. Under this framework, a solution to the joint decomposition can be cast in terms of a maximum a posteriori estimator. Examples of joint posterior distributions are provided, including general Gaussian priors and non Gaussian coupling priors. Then simulations are reported and show the effectiveness of this approach to fuse information from data sets, which are inherently of different size due to different time resolution of the measurement devices. <s> BIB034 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We introduce Bayesian multi-tensor factorization, a model that is the first Bayesian formulation for joint factorization of multiple matrices and tensors. The research problem generalizes the joint matrix-tensor factorization problem to arbitrary sets of tensors of any depth, including matrices, can be interpreted as unsupervised multi-view learning from multiple tensors, and can be generalized to relax the usual trilinear tensor factorization assumptions. The result is a factorization of the set of tensors into factors shared by any subsets of the tensors, and factors private to individual tensors. We demonstrate the performance against existing baselines in multiple tensor factorization tasks in structural toxicogenomics and functional neuroimaging. <s> BIB035 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods. <s> BIB036 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> B. Link Between Data Sets <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB037
Data fusion is all about enabling modalities to fully interact and inform each other. Hence, a key point is choosing an analytical model that faithfully represents the relationship between modalities and yields a meaningful combination thereof, without imposing phantom connections or suppressing existing ones. The underlying idea of data fusion is that an ensemble of data sets is ''more than the sum of its parts'' in the sense that it contains precious information that is lost if these relations are ignored. The purpose of properly defined links is to support this goal, as motivated by the discussion in Section III. In order to maximize diversity, we would like links to be able to exploit the heterogeneity among data sets. Properly defined links provide a clear picture of the underlying structure of the ensemble of the related data sets  . Consider, for example, two data sets, patients Â symptoms and patients Â diagnosis; we would like the data fusion model to allow us to uncover the medical conditions that underlie both symptoms and diagnoses BIB019 . Properly defined links explain similarities and differences among data sets and allow better interpretability. As explained in Section III, one of the first motivations for linking data sets in joint matrix decomposition scenarios is resolving the arbitrary ordering of the latent components in each individual data set. It is interesting to note that all types of links eventually alleviate this problem since they provide a single frame of reference. Since data fusion generally deals with heterogeneous data sets, we would like links to be flexible enough to allow each data set to remain in its most explanatory form, as further discussed in Section V-B1. In various scenarios, certain elements may be present only in a specific data set whereas others are shared by two or more. We would like the model not only to properly express these elaborate interactions but also to have the capacity to inform us about (non)existence of links when this information is not available in advance, a topic further elaborated on in Section V-B2. As stated in Section II, the raison d'être of multimodal data fusion is the paradigm that certain natural processes and phenomena express themselves under completely different physical guises. Due to the often complex nature of the driving phenomena, it is likely that data sets will be related via more than one type of diversity; e.g., time, space, and frequency. Therefore, links should be designed such that they support a relationship via several types of diversity simultaneously, whenever applicable. Models based on multilinear relationships, as well as those that admit multiple types of links simultaneously, seem to better support this aim. 1) ''Soft'' and ''Hard'' Links Between Data Sets: One type of decision that has to be made is whether each data set will have its own set of individual parameters, disjoint of the others. In the first case, none of the parameters that define each data set's model are shared by any other data set. As a result, additional information is required to define the link. In such cases, the link is often defined as some correspondence between data sets that can be interpreted as similarity, smoothness, or continuity BIB009 . Therefore, we call such links ''soft.'' In the second case, data sets explicitly share certain factor matrices or latent variables. For the sake of our discussion, we call such links ''hard'' BIB034 . ''Hard'' links between data sets. We have already seen shared factor matrices in numerous examples in Section III. Naturally, data fusion methods that are based on stacking data in a single tensor fall within this category. Such are PARAFAC (Section III-C-2), generalized singular value decomposition (GSVD) BIB006 and its higher order generalization BIB015 , the higher order SVD (HOSVD) BIB010 , and more. In joint ICA BIB008 and group ICA  ,  , several ICA problems share a mixing matrix or source subspace, respectively, by concatenating the observation matrices in rows or columns. Simultaneous factor analysis (FA) BIB002 and simultaneous component analysis (SCA)-based methods BIB022 , BIB003 - BIB023 deal with multiway data that have at least one shared mode, but do not stack it in a single tensor (Section III-C2) due to various complicating factors, such as those mentioned in Section IV. Linked tensor ICA BIB016 has one factor matrix shared by all decompositions. In Bayesian group FA BIB017 and its tensor generalization BIB035 , BIB029 , as well as in collective matrix factorization (CMF) BIB011 , several matrices or tensors share all but one factor matrix. In fusion of hyperspectral and multispectral images (Example II-C.1), the joint factor is a matrix that reflects the (desired, unknown) high-resolution image before spatial and spectral degradation BIB020 - BIB030 . In group nonnegative matrix factorization (NMF), shared columns of the feature matrix reflect task-related variations  . The generalized linked-mode framework for multiway data BIB014 allows flexible links across data sets by shared (sub)factors, as do other flexible tensor-based data fusion models such as coupled matrix and tensor factorization (CMTF) BIB024 and its probabilistic extension generalized coupled tensor factorization (GCTF) BIB025 , linked multiway component analysis (LMWCA) BIB036 , and structured data fusion (SDF) BIB037 . In the fusion of astrophysical observations of the CMB from different experiments (Example II-C.3), the link may be established by a joint distribution of the ensemble of samples from all data sets. In this case, the fusion is based on the assumption that the random processes from which all samples are generated are controlled by the same underlying cosmological parameters  . A shared random variable is used also in  to extract a common source of variability from measurements in multiple sensors using diffusion operators. ''Soft'' links between data sets. Prevalent types of ''soft'' links are statistical dependence, as in IVA (Example III-D.1); covariations, as in CCA BIB001 and its extension to more than two matrices, multiset CCA (MCCA)  ,  , BIB012 , BIB005 , and parallel ICA  ; and ''similarity,'' in the sense of minimizing some distance measure between corresponding elements, as in soft nonnegative matrix cofactorization BIB031 and joint matrix and tensor decompositions with flexible coupling BIB034 . For audiovisual data fusion, a dictionary learning model where each atom consists of an audio and video component has been proposed in BIB013 . A graphical model in which audio and video shifts are linearly related in far-field conditions is proposed in BIB007 . Generalized linked-mode for multiway data BIB014 and LMWCA BIB036 mention explicitly that they can be defined both with ''soft'' or ''hard'' links. Although the partition into ''soft'' and ''hard'' links is conceptually appealing and simplifies our presentation, the following reservation is in order. In practice, when it comes to writing the optimization problem, models with ''soft'' links are often reformulated using shared variables. In the models that we have just mentioned, shared variables are, for example, cross correlation or cross cumulants when statistical (in)dependence and covariation are concerned, or a shared latent variable in regression models. The reformulated models often take the form of (approximate) (coupled) matrix or tensor factorizations. We mention BIB018 ,  , and BIB032 as just a few examples; further discussion is beyond the scope of this paper. The bottom line is that the distinction between ''soft'' and ''hard'' links is often immaterial. The implication is that models with ''soft'' links can sometimes neatly fit within optimization frameworks that assume shared variables, for example, SDF BIB037 . 2) Shared Versus Unshared Elements: The idea that data sets have both shared (common) and unshared (individual, modality-specific) elements w.r.t. the others can be found in numerous models. It can be formulated mathematically by defining certain columns of a factor matrix or subelements of a latent variable as shared, while others are unshared. Models that admit this formulation include incomplete mode PARAFAC [69, Sec. 5.1.2] [ Fig. 2(b) ], group NMF  , LMWCA BIB026 , and SDF BIB037 . In the extraction of a common source of variability from heterogeneous sensors  , it is hidden random variables that are either shared or unshared. Another example is Bayesian group FA BIB017 and its tensor extensions BIB035 , BIB029 , where a dedicated factor matrix determines which of the factors in a common pool are active within each data set. The more fundamental challenge, however, is to identify the shared and unshared elements from the data itself, without a priori assignment of individual and shared variables. Bayesian linked tensor ICA BIB016 holds a modality-specific factor matrix of optimally determined weights that can eliminate a source from some modalities while keeping it in others. In BIB006 , Alter et al. propose GSVD to infer, from two genome-scale expression data sets, shared and individual processes. Ponnapalli et al. BIB015 extend this GSVD-based approach to more than two data sets. Shared and unshared processes in genomic and metabolomic data may also be revealed by a proper rotation of the components resulting from SCA. The proposed approach, called distinctive and common components with simultaneous-component analysis (DIS-CO-SCA) BIB021 , BIB033 , BIB027 , may outperform GSVD in certain scenarios and can be straightforwardly generalized to more than two data sets. A comparative study of GSVD, DISCO-SCA, and other methods that can identify shared and unshared processes underlying multiset data can be found in BIB023 . HOSVD BIB004 can differentiate between shared and unshared phenomena in DNA analysis from multiple experiments BIB010 . As a last example, in CMTF BIB028 , model constraints may be defined in the form of sparse weights such that unshared components have norms equal or close to zero in one of the data sets.