Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Concepts of correlation and regression may be applied not only to ordinary one-dimensional variates but also to variates of two or more dimensions. Marksmen side by side firing simultaneous shots at targets, so that the deviations are in part due to independent individual errors and in part to common causes such as wind, provide a familiar introduction to the theory of correlation; but only the correlation of the horizontal components is ordinarily discussed, whereas the complex consisting of horizontal and vertical deviations may be even more interesting. The wind at two places may be compared, using both components of the velocity in each place. A fluctuating vector is thus matched at each moment with another fluctuating vector. The study of individual differences in mental and physical traits calls for a detailed study of the relations between sets of correlated variates. For example the scores on a number of mental tests may be compared with physical measurements on the same persons. The questions then arise of determining the number and nature of the independent relations of mind and body shown by these data to exist, and of extracting from the multiplicity of correlations in the system suitable characterizations of these independent relations. As another example, the inheritance of intelligence in rats might be studied by applying not one but s different mental tests to N mothers and to a daughter of each <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> The model for three-mode factor analysis is discussed in terms of newer applications of mathematical processes including a type of matrix process termed the Kronecker product and the definition of combination variables. Three methods of analysis to a type of extension of principal components analysis are discussed. Methods II and III are applicable to analysis of data collected for a large sample of individuals. An extension of the model is described in which allowance is made for unique variance for each combination variable when the data are collected for a large sample of individuals. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Abstract A three-way array X (or three-dimensional matrix) is an array of numbers xijk subscripted by three indices. A triad is a multiplicative array, xijk = aibjck. Analogous to the rank and the row rank of a matrix, we define rank (X) to be the minimum number of triads whose sum is X, and dim1(X) to be the dimensionality of the space of matrices generated by the 1-slabs of X. (Rank and dim1 may not be equal.) We prove several lower bounds on rank. For example, a special case of Theorem 1 is that rank(X)⩾dim 1 (UX) + rank(XW) − dim 1 (UXW) , where U and W are matrices; this generalizes a matrix theorem of Frobenius. We define the triple product [A, B, C] of three matrices to be the three-way array whose (i, j, k) element is given by ⩞rairbjrckr; in other words, the triple product is the sum of triads formed from the columns of A, B, and C. We prove several sufficient conditions for the factors of a triple product to be essentially unique. For example (see Theorem 4a), suppose [A, B, C] = [ A , B , C ] , and each of the matrices has R columns. Suppose every set of rank (A) columns of A are independent, and similar conditions hold for B and C. Suppose rank (A) + rank (B) + rank (C) ⩾ 2R + 2. Then there exist diagonal matrices Λ, M, N and a permutation matrix P such that A = APΛ, B = BP M , C = CP N . Our results have applications to arithmetic complexity theory and to statistical models used in three-way multidimensional scaling. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> It is noted that the problem of source separation has no solution without a priori information when only the spectral matrix is used. The authors have developed an algorithm using the fourth-order cumulants that solves the problem of the two-source separation without the necessity of a priori information. They show that, in the case of two sources and two sensors, the model is described by two parameters. They then present the identification of these two parameters with the cumulants up to order 4. The potentialities of this source-identification algorithm are illustrated using simulated data. > <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Blind separation of communication signals invariably relies on some form(s) of diversity to overdetermine the problem and thereby recover the signals of interest. More often than not, linear (e.g., spreading) diversity is employed, i.e., each diversity branch provides a linear combination of the unknown signals, albeit with possibly unknown weights. If multiple forms of linear diversity are simultaneously available, then the resulting data exhibit multilinear structure, and the blind recovery problem can be shown to be tantamount to low-rank decomposition of the multi-dimensional received data array. This paper generalizes Kruskal's fundamental result on the uniqueness of low-rank decomposition of 3-way arrays to the case of multilinear decomposition of 4- and higher-way arrays. The result characterizes diversity combining for blind identifiability when N forms of linear diversity are available; that is the balance between different forms of diversity that guarantees blind recovery of all signals involved. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> We generalize Kruskal's fundamental result on the uniqueness of trilinear decomposition of three-way arrays to the case of multilinear decomposition of four- and higher-way arrays. The result is surprisingly general and simple and has several interesting ramifications. Copyright © 2000 John Wiley & Sons, Ltd. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> This paper describes two- and multiset canonical correlations analysis (CCA) for data fusion, multisource, multiset, or multitemporal exploratory data analysis. These techniques transform multivariate multiset data into new orthogonal variables called canonical variates (CVs) which, when applied in remote sensing, exhibit ever-decreasing similarity (as expressed by correlation measures) over sets consisting of 1) spectral variables at fixed points in time (R-mode analysis), or 2) temporal variables with fixed wavelengths (T-mode analysis). The CVs are invariant to linear and affine transformations of the original variables within sets which means, for example, that the R-mode CVs are insensitive to changes over time in offset and gain in a measuring device. In a case study, CVs are calculated from Landsat Thematic Mapper (TM) data with six spectral bands over six consecutive years. Both Rand T-mode CVs clearly exhibit the desired characteristic: they show maximum similarity for the low-order canonical variates and minimum similarity for the high-order canonical variates. These characteristics are seen both visually and in objective measures. The results from the multiset CCA R- and T-mode analyses are very different. This difference is ascribed to the noise structure in the data. The CCA methods are related to partial least squares (PLS) methods. This paper very briefly describes multiset CCA-based multiset PLS. Also, the CCA methods can be applied as multivariate extensions to empirical orthogonal functions (EOF) techniques. Multiset CCA is well-suited for inclusion in geographical information systems (GIS). <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Blind signal separation (BSS) and independent component analysis (ICA) are emerging techniques of array processing and data analysis that aim to recover unobserved signals or "sources" from observed mixtures (typically, the output of an array of sensors), exploiting only the assumption of mutual independence between the signals. The weakness of the assumptions makes it a powerful approach, but it requires us to venture beyond familiar second order statistics, The objectives of this paper are to review some of the approaches that have been developed to address this problem, to illustrate how they stem from basic principles, and to show how they relate to each other. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Let X be a real-valued three-way array. The Candecomp/Parafac (CP) decomposition is written as (X) under bar = (Y) under bar ((1)) + ... + (Y) under bar ((R)) + E, where Y-(r) are rank-1 arrays and (E) under bar is a rest term. Each rank-1 array is defined by the outer product of three vectors a((r)). b((r)) and c((r)), i.e. y(ijk)((r)) =a(i)((r)) b(j)((r)) c(k)((t)). These vectors make up the R columns of the component matrices A, B and C. If 2R + 2 is less than or equal to the sum of the k-ranks of A, B and C, then the fitted part of the decomposition is unique up to a change in the order of the rank-1 arrays and rescaling/counterscaling of each triplet of vectors (a((r)), b((r)), c((r))) forming a rank-1 array. This classical result was shown by Kruskal. His proof is, however, rather inaccessible and does not seem intuitive. In order to contribute to a better understanding of CP uniqueness, this paper provides an accessible and intuitive proof of Kruskal's condition. The proof is both self-contained and compact and can easily be adapted for the complex-valued CP decomposition. (c) 2006 Elsevier Inc. All rights reserved. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Abstract We discuss model-free analysis of multisubject or multisession FMRI data by extending the single-session probabilistic independent component analysis model (PICA; Beckmann and Smith, 2004. IEEE Trans. on Medical Imaging, 23 (2) 137–152) to higher dimensions. This results in a three-way decomposition that represents the different signals and artefacts present in the data in terms of their temporal, spatial, and subject-dependent variations. The technique is derived from and compared with parallel factor analysis (PARAFAC; Harshman and Lundy, 1984. In Research methods for multimode data analysis, chapter 5, pages 122–215. Praeger, New York). Using simulated data as well as data from multisession and multisubject FMRI studies we demonstrate that the tensor PICA approach is able to efficiently and accurately extract signals of interest in the spatial, temporal, and subject/session domain. The final decompositions improve upon PARAFAC results in terms of greater accuracy, reduced interference between the different estimated sources (reduced cross-talk), robustness (against deviations of the data from modeling assumptions and against overfitting), and computational speed. On real FMRI ‘activation’ data, the tensor PICA approach is able to extract plausible activation maps, time courses, and session/subject modes as well as provide a rich description of additional processes of interest such as image artefacts or secondary activation patterns. The resulting data decomposition gives simple and useful representations of multisubject/multisession FMRI data that can aid the interpretation and optimization of group FMRI studies beyond what can be achieved using model-based analysis techniques. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> We present a new approach to independent component analysis (ICA) by extending the formulation of univariate source signals to multivariate source signals. The new approach is termed independent vector analysis (IVA). In the model, we assume that linear mixing model exists in each dimension separately, and the latent sources are independent of the others. In contrast to ICA, the sources are random vectors, not just single variables, which means the elements of a random vector are closely related to the others. Thus, we assume the dependency between the elements of a source vector. In this manner, we define dependence between vectors as Kullback-Leibler divergence between the total joint probability of vectors and the product of marginal probabilities of vectors. Then, the model allows independence between multivariate source signals represented as random vectors, and dependence between the source signals within the vector representation. The proposed vector density model can for example capture variance dependencies within a vector source signal. There are several applications of this new formulation. In the separation of acoustic sources, the algorithm mitigates the permutation problem, i.e. the usual ICA algorithms applied in to the frequency domain mixture data suffer from the unknown permutation of the output signals. Although there are several engineering solutions to fix this problem after the ICA stage, the proposed method provides a natural solution to the problem by capturing the inherent dependencies of the acoustic signals. It therefore avoids the permutation problem and allows the separation of sources in very challenging environments for many sound sources. <s> BIB011 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Abstract Independent component analysis (ICA) of fMRI data generates session/individual specific brain activation maps without a priori assumptions regarding the timing or pattern of the blood-oxygenation-level-dependent (BOLD) signal responses. However, because of a random permutation among output components, ICA does not offer a straightforward solution for the inference of group-level activation. In this study, we present an independent vector analysis (IVA) method to address the permutation problem during fMRI group data analysis. In comparison to ICA, IVA offers an analysis of additional dependent components, which were assigned for use in the automated grouping of dependent activation patterns across subjects. Upon testing using simulated trial-based fMRI data, our proposed method was applied to real fMRI data employing both a single-trial task-paradigm (right hand motor clenching and internal speech generation tasks) and a three-trial task-paradigm (right hand motor imagery task). A generalized linear model (GLM) and the group ICA of the fMRI toolbox (GIFT) were also applied to the same data set for comparison to IVA. Compared to GLM, IVA successfully captured activation patterns even when the functional areas showed variable hemodynamic responses that deviated from a hypothesized response. We also showed that IVA effectively inferred group-activation patterns of unknown origins without the requirement for a pre-processing stage (such as data concatenation in ICA-based GIFT). IVA can be used as a potential alternative or an adjunct to current ICA-based fMRI group processing methods. <s> BIB012 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this paper we introduce a new class of tensor decompositions. Intuitively, we decompose a given tensor block into blocks of smaller size, where the size is characterized by a set of mode-$n$ ranks. We study different types of such decompositions. For each type we derive conditions under which essential uniqueness is guaranteed. The parallel factor decomposition and Tucker's decomposition can be considered as special cases in the new framework. The paper sheds new light on fundamental aspects of tensor algebra. <s> BIB013 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this paper, we present a powerful technique for the blind extraction of direct-sequence code-division multiple access (DS-CDMA) signals from convolutive mixtures received by an antenna array. The technique is based on a generalization of the canonical or parallel factor decomposition (CANDECOMP/PARAFAC) in multilinear algebra. We present a bound on the number of users under which blind separation and deconvolution is guaranteed. The solution is computed by means of an alternating least squares (ALS) algorithm. The excellent performance is illustrated by means of a number of simulations. We include an explicit expression of the Cramer-Rao bound (CRB) of the transmitted symbols. <s> BIB014 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this work, we introduce a simple and effective scheme to achieve joint blind source separation (BSS) of multiple datasets using multi-set canonical correlation analysis (M-CCA) [1]. We first propose a generative model of joint BSS based on the correlation of latent sources within and between datasets. We specify source separability conditions, and show that, when the conditions are satisfied, the group of corresponding sources from each dataset can be jointly extracted by M-CCA through maximization of correlation among the extracted sources. We compare source separation performance of the M-CCA scheme with other joint BSS methods and demonstrate the superior performance of the M-CCA scheme in achieving joint BSS for a large number of datasets, group of corresponding sources with heterogeneous correlation values, and complex-valued sources with circular and non-circular distributions. We apply M-CCA to analysis of functional magnetic resonance imaging (fMRI) data from multiple subjects and show its utility in estimating meaningful brain activations from a visuomotor task. <s> BIB015 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB016 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Feature extraction and selection are key factors in model reduction, classification and pattern recognition problems. This is especially important for input data with large dimensions such as brain recording or multiview images, where appropriate feature extraction is a prerequisite to classification. To ensure that the reduced dataset contains maximum information about input data we propose algorithms for feature extraction and classification. This is achieved based on orthogonal or nonnegative tensor (multi-array) decompositions, and higher order (multilinear) discriminant analysis (HODA), whereby input data are considered as tensors instead of more conventional vector or matrix representations. The developed algorithms are verified on benchmark datasets, using constraints imposed on tensors and/or factor matrices such as orthogonality and nonnegativity. <s> BIB017 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this paper, we show that the joint blind source separation (JBSS) problem can be solved by jointly diagonalizing cumulant matrices of any order higher than one, including the correlation matrices and the fourth-order cumulant matrices. We introduce an efficient iterative generalized joint diagonalization algorithm such that a series of orthogonal procrustes problems are solved. We present simulation results to show that the new algorithms can reliably solve the permutation ambiguity in JBSS and that they offer superior performance compared with existing multiset canonical correlation analysis (MCCA) and independent vector analysis (IVA) approaches. Experiment on real-world data for separation of fetal heartbeat in electrocardiogram (ECG) data demonstrates a new application of JBSS, and the success of the new algorithms for a real-world problem. <s> BIB018 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this study we classify short echo-time brain magnetic resonance spectroscopic imaging (MRSI) data by applying a model-based canonical correlation analyses algorithm and by using, as prior knowledge, multimodal sources of information coming from high-resolution magic angle spinning (HR-MAS), MRSI and magnetic resonance imaging. The potential and limitations of fusing in vivo and ex vivo nuclear magnetic resonance sources to detect brain tumors is investigated. We present various modalities for multimodal data fusion, study the effect and the impact of using multimodal information for classifying MRSI brain glial tumors data and analyze which parameters influence the classification results by means of extensive simulation and in vivo studies. Special attention is drawn to the possibility of considering HR-MAS data as a complementary dataset when dealing with a lack of MRSI data needed to build a classifier. Results show that HR-MAS information can have added value in the process of classifying MRSI data. <s> BIB019 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this work, we apply a novel statistical method, multiset canonical correlation analysis (M-CCA), to study a group of functional magnetic resonance imaging (fMRI) datasets acquired during simulated driving task. The M-CCA method jointly decomposes fMRI datasets from different subjects/sessions into brain activation maps and their associated time courses, such that the correlation in each group of estimated activation maps across datasets is maximized. Therefore, the functional activations across all datasets are extracted in the order of consistency across different dataset. On the other hand, M-CCA preserves the uniqueness of the functional maps estimated from each dataset by avoiding concatenation of different datasets in the analysis. Hence, the cross-dataset variation of the functional activations can be used to test the hypothesis of functional-behavioral association. In this work, we study 120 simulated driving fMRI datasets and identify parietal-occipital regions and frontal lobe as the most consistently engaged areas across all the subjects and sessions during simulated driving. The functional-behavioral association study indicates that all the estimated brain activations are significantly correlated with the steering operation during the driving task. M-CCA thus provides a new approach to investigate the complex relationship between the brain functions and multiple behavioral variables, especially in naturalistic tasks as demonstrated by the simulated driving study. <s> BIB020 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Independent component analysis (ICA) and blind source separation (BSS) deal with extracting a number of mutually independent elements from a set of observed linear mixtures. Motivated by various applications, this paper considers a more general and more flexible model: the sources can be partitioned into groups exhibiting dependence within a given group but independence between two different groups. We argue that this is tantamount to considering multidimensional components as opposed to the standard ICA case which is restricted to one-dimensional components. The core of the paper is devoted to the statistical analysis of the blind separation of multidimensional components based on second-order statistics, in a piecewise-stationary model. We develop the likelihood and the associated estimating equations for the Gaussian case. We obtain closed-form expressions for the Fisher information matrix and the Cramer-Rao bound of the de-mixing parameters, as well as the mean-square error (MSE) of the component estimates. The derived MSE is valid also for non-Gaussian data. Our analysis is verified through numerical experiments, and its performance is compared to classical ICA in various dependence scenarios, quantifying the gain in the accuracy of component recovery in presence of multidimensional components. <s> BIB021 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this paper we propose a new flexible group tensor analysis model called the linked CP tensor decomposition (LCPTD). The LCPTD method can decompose given multiple tensors into common factor matrices, individual factor matrices, and core tensors, simultaneously. We applied the Hierarchical Alternating Least Squares (HALS) algorithm to the LCPTD model; besides we impose additional constraints to obtain sparse and nonnegative factors. Furthermore, we conducted some experiments of this model to demonstrate its advantages over existing models. <s> BIB022 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Recent advances in Joint Blind Source Separation (JBSS) extend the BSS framework to the simultaneous source separation of multiple datasets. In this paper we provide a comparative study of four such JBSS algorithms on human dual-electroencephalographic (dual-EEG) data. Appropriateness of second order JBSS is demonstrated for concurrent estimation of correlated sources in a multi-subject synchronous steady-state visually evoked potentials experiment. This approach gives a new starting point for the exploration of brain activities in a hyperscanning framework. <s> BIB023 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Recently, an extension of independent component analysis (ICA) from one to multiple datasets, termed independent vector analysis (IVA), has been a subject of significant research interest. IVA has also been shown to be a generalization of Hotelling's canonical correlation analysis. In this paper, we provide the identification conditions for a general IVA formulation, which accounts for linear, nonlinear, and sample-to-sample dependencies. The identification conditions are a generalization of previous results for ICA and for IVA when samples are independently and identically distributed. Furthermore, a principal aim of IVA is identification of dependent sources between datasets. Thus, we provide additional conditions for when the arbitrary ordering of the estimated sources can be common across datasets. Performance bounds in terms of the Cramer-Rao lower bound are also provided for demixing matrices and interference to source ratio. The performance of two IVA algorithms are compared to the theoretical bounds. <s> BIB024 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> There has been an ever-increasing interest in multi-disciplinary research on multisensor data fusion technology, driven by its versatility and diverse areas of application. Therefore, there seems to be a real need for an analytical review of recent developments in the data fusion domain. This paper proposes a comprehensive review of the data fusion state of the art, exploring its conceptualizations, benefits, and challenging aspects, as well as existing methodologies. In addition, several future directions of research in the data fusion community are highlighted and described. <s> BIB025 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> For the case of a single colocated receive antenna array and additional linear diversity (e.g. oversampling or polarization), tensor decomposition based signal separation is now well-established. For increasing the spatial diversity of communication systems, the use of several widely separated colocated antenna arrays has been suggested. However, for such problems no algebraic framework has been proposed. We explain that recently developed coupled tensor decompositions provide such a framework. In particular, we explain that the use of several widely separated colocated antenna arrays may lead to improved identifiability results. <s> BIB026 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Abstract Recent technological advances enable us to collect huge amounts of data from multiple sources. Jointly analyzing such multi-relational data from different sources, i.e., data fusion (also called multi-block, multi-view or multi-set data analysis), often enhances knowledge discovery. For instance, in metabolomics, biological fluids are measured using a variety of analytical techniques such as Liquid Chromatography–Mass Spectrometry and Nuclear Magnetic Resonance Spectroscopy. Data measured using different analytical methods may be complementary and their fusion may help in the identification of chemicals related to certain diseases. Data fusion has proved useful in many fields including social network analysis, collaborative filtering, neuroscience and bioinformatics. In this paper, unlike many studies demonstrating the success of data fusion, we explore the limitations as well as the advantages of data fusion. We formulate data fusion as a coupled matrix and tensor factorization (CMTF) problem, which jointly factorizes multiple data sets in the form of higher-order tensors and matrices by extracting a common latent structure from the shared mode. Using numerical experiments on simulated and real data sets, we assess the performance of coupled analysis compared to the analysis of a single data set in terms of missing data estimation and demonstrate cases where coupled analysis outperforms analysis of a single data set and vice versa. <s> BIB027 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Several problems in signal processing have been formulated in terms of the Canonical Polyadic Decomposition of a higher-order tensor with one or more Vandermonde constrained factor matrices. We first propose new, relaxed uniqueness conditions. We explain that, under these conditions, the number of components may simply be estimated as the rank of a matrix. We propose an efficient algorithm for the computation of the factors that only resorts to basic linear algebra. We demonstrate the use of the results for various applications in wireless communication and array processing. <s> BIB028 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Independent component analysis (ICA) is a widely applied technique to derive functionally connected brain networks from fMRI data. Group ICA (GICA) and Independent Vector Analysis (IVA) are extensions of ICA that enable users to perform group fMRI analyses; however a full comparison of the performance limits of GICA and IVA has not been investigated. Recent interest in resting state fMRI data with potentially higher degree of subject variability makes the evaluation of the above techniques important. In this paper we compare component estimation accuracies of GICA and an improved version of IVA using simulated fMRI datasets. We systematically change the degree of component spatial variability and evaluate estimation accuracy over all spatial maps (SMs) and time courses (TCs) of the decomposition. Our results indicate the following: (1) at low levels of SM variability or when just one SM is varied, both GICA and IVA perform well, (2) at higher levels of SM variability or when more than one SMs are varied, IVA continues to perform well but GICA yields SM estimates that are composites of other SMs with errors in TCs, (3) both GICA and IVA remove spatial correlations of overlapping SMs and introduce artificial correlations in their TCs, (4) if number of SMs is over estimated, IVA continues to perform well but GICA introduces artifacts in the varying and extra SMs with artificial correlations in the TCs of extra components, and (5) in the absence or presence of SMs unique to one subject, GICA produces errors in TCs and IVA estimates are accurate. In summary, our simulation experiments (both simplistic and realistic) and our holistic analyses approach indicate that IVA produces results that are closer to ground truth and thereby better preserves subject variability. The improved version of IVA is now packaged into the GIFT toolbox (http://mialab.mrn.org/software/gift). <s> BIB029 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> It has become quite common for multiple brain imaging types to be collected for a particular study. This raises the issue of how to combine these imaging types to gain the most useful information for inference. One can perform data integration, where one modality is used to improve the results of another, or true data fusion, where multiple modalities are used to inform one another. We propose two new methods of data fusion, entropy bound minimization (EBM) for joint independent component analysis (jICA) and independent vector analysis with a Gaussian prior (IVA-G), and compare them to the established data fusion techniques of multiset canonical correlation analysis (MCCA) and jICA using Infomax. Additionally, we propose a simulation model and use it to probe the limitations of these methods. Results show that EBM with jICA outperforms the other selected methods but is highly sensitive to the availability of joint information provided by these modalities. <s> BIB030 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Abstract Recent work on both task-induced and resting-state functional magnetic resonance imaging (fMRI) data suggests that functional connectivity may fluctuate, rather than being stationary during an entire scan. Most dynamic studies are based on second-order statistics between fMRI time series or time courses derived from blind source separation, e.g., independent component analysis (ICA), to investigate changes of temporal interactions among brain regions. However, fluctuations related to spatial components over time are of interest as well. In this paper, we examine higher-order statistical dependence between pairs of spatial components, which we define as spatial functional network connectivity (sFNC), and changes of sFNC across a resting-state scan. We extract time-varying components from healthy controls and patients with schizophrenia to represent brain networks using independent vector analysis (IVA), which is an extension of ICA to multiple data sets and enables one to capture spatial variations. Based on mutual information among IVA components, we perform statistical analysis and Markov modeling to quantify the changes in spatial connectivity. Our experimental results suggest significantly more fluctuations in patient group and show that patients with schizophrenia have more variable patterns of spatial concordance primarily between the frontoparietal, cerebellar and temporal lobe regions. This study extends upon earlier studies showing temporal connectivity differences in similar areas on average by providing evidence that the dynamic spatial interplay between these regions is also impacted by schizophrenia. <s> BIB031 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> This paper deals with joint blind source separation (JBSS) of multidimensional components. JBSS extends classical BSS to simultaneously resolve several BSS problems by assuming statistical dependence between latent sources across mixtures. JBSS offers some significant advantages over BSS, such as identifying more than one Gaussian white stationary source within a mixture. Multidimensional BSS extends classical BSS to deal with a more general and more flexible model within each mixture: the sources can be partitioned into groups exhibiting dependence within a given group but independence between two different groups. Motivated by various applications, we present a model that is inspired by both extensions. We derive an algorithm that achieves asymptotically the minimal mean square error (MMSE) in the estimation of Gaussian multidimensional components. We demonstrate the superior performance of this model over a two-step approach, in which JBSS, which ignores the multidimensional structure, is followed by a clustering step. <s> BIB032 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> This paper deals with the separation problem of dependent sources. The separation is made possible thanks to side information on the dependence nature of the considered sources. In this work, we ﬁrst show how this side information can be used to achieve desired source separation using joint matrix decomposition techniques. Indeed, in the case of statistically independent sources, many BSS methods are based on joint matrix diagonalization. In our case, we replace the target diagonal structure by appropriate non diagonal one which reﬂects the dependence nature of the sources. This new concept is illustrated with two simple 2 2 source separation examples where second-order-statistics and high order-statistics are used respectively. <s> BIB033 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> In this paper, we present an overview of constrained PARAFAC models where the constraints model linear dependencies among columns of the factor matrices of the tensor decomposition, or alternatively, the pattern of interactions between different modes of the tensor which are captured by the equivalent core tensor. Some tensor prerequisites with a particular emphasis on mode combination using Kronecker products of canonical vectors that makes easier matricization operations, are first introduced. This Kronecker product based approach is also formulated in terms of the index notation, which provides an original and concise formalism for both matricizing tensors and writing tensor models. Then, after a brief reminder of PARAFAC and Tucker models, two families of constrained tensor models, the co-called PARALIND/CONFAC and PARATUCK models, are described in a unified framework, for $N^{th}$ order tensors. New tensor models, called nested Tucker models and block PARALIND/CONFAC models, are also introduced. A link between PARATUCK models and constrained PARAFAC models is then established. Finally, new uniqueness properties of PARATUCK models are deduced from sufficient conditions for essential uniqueness of their associated constrained PARAFAC models. <s> BIB034 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Tensor decompositions are at the core of many blind source separation (BSS) algorithms, either explicitly or implicitly. In particular, the canonical polyadic (CP) tensor decomposition plays a central role in the identification of underdetermined mixtures. Despite some similarities, CP and singular value decomposition (SVD) are quite different. More generally, tensors and matrices enjoy different properties, as pointed out in this brief introduction. <s> BIB035 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> Canonical Polyadic Decomposition (CPD) of a third-order tensor is a minimal decomposition into a sum of rank-$1$ tensors. We find new mild deterministic conditions for the uniqueness of individual rank-$1$ tensors in CPD and present an algorithm to recover them. We call the algorithm"algebraic"because it relies only on standard linear algebra. It does not involve more advanced procedures than the computation of the null space of a matrix and eigen/singular value decomposition. Simulations indicate that the new conditions for uniqueness and the working assumptions for the algorithm hold for a randomly generated $I\times J\times K$ tensor of rank $R\geq K\geq J\geq I\geq 2$ if $R$ is bounded as $R\leq (I+J+K-2)/2 + (K-\sqrt{(I-J)^2+4K})/2$ at least for the dimensions that we have tested. This improves upon the famous Kruskal bound for uniqueness $R\leq (I+J+K-2)/2$ as soon as $I\geq 3$. In the particular case $R=K$, the new bound above is equivalent to the bound $R\leq(I-1)(J-1)$ which is known to be necessary and sufficient for the generic uniqueness of the CPD. An existing algebraic algorithm (based on simultaneous diagonalization of a set of matrices) computes the CPD under the more restrictive constraint $R(R-1)\leq I(I-1)J(J-1)/2$ (implying that $R<(J-\frac{1}{2})(I-\frac{1}{2})/\sqrt{2}+1$). On the other hand, optimization-based algorithms fail to compute the CPD in a reasonable amount of time even in the low-dimensional case $I=3$, $J=7$, $K=R=12$. By comparison, in our approach the computation takes less than $1$ sec. We demonstrate that, at least for $R\leq 24$, our algorithm can recover the rank-$1$ tensors in the CPD up to $R\leq(I-1)(J-1)$. <s> BIB036 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB037 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods. <s> BIB038 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> We introduce Bayesian multi-tensor factorization, a model that is the first Bayesian formulation for joint factorization of multiple matrices and tensors. The research problem generalizes the joint matrix-tensor factorization problem to arbitrary sets of tensors of any depth, including matrices, can be interpreted as unsupervised multi-view learning from multiple tensors, and can be generalized to relax the usual trilinear tensor factorization assumptions. The result is a factorization of the set of tensors into factors shared by any subsets of the tensors, and factors private to individual tensors. We demonstrate the performance against existing baselines in multiple tensor factorization tasks in structural toxicogenomics and functional neuroimaging. <s> BIB039 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. A Link Between Data Sets as a New Form of Diversity <s> This paper deals with a novel generalization of classical blind source separation (BSS) in two directions. First, relaxing the constraint that the latent sources must be statistically independent. This generalization is well-known and sometimes termed independent subspace analysis (ISA). Second, jointly analyzing several ISA problems, where the link is due to statistical dependence among corresponding sources in different mixtures. When the data are one-dimensional, i.e., multiple classical BSS problems, this model, known as independent vector analysis (IVA), has already been studied. In this paper, we combine IVA with ISA and term this new model joint independent subspace analysis (JISA). We provide full performance analysis of JISA, including closed-form expressions for minimal mean square error (MSE), Fisher information and Cramer–Rao lower bound, in the separation of Gaussian data. The derived MSE applies also for non-Gaussian data, when only second-order statistics are used. We generalize previously known results on IVA, including its ability to uniquely resolve instantaneous mixtures of real Gaussian stationary data, and having the same arbitrary permutation at all mixtures. Numerical experiments validate our theoretical results and show the gain with respect to two competing approaches that either use a finer block partition or a different norm. <s> BIB040
As explained in Section III-C, if all data sets share the same underlying factorization model, and in addition, admit a (multi)linear relationship, then it may be possible to use a single matrix or tensor decomposition in order to perform data fusion. This assumption may be challenged in various scenarios. An obvious conflict arises when data sets are given in different types of physical units. A technical difficulty is when data sets are stacked in arrays of different orders, such as matrices versus higher order arrays. Further examples are data sets with different latent models, different types of uncertainty, or when not all factors or latent variables are shared by all data sets. In such cases, we say that data sets are heterogeneous BIB016 . While each of these complicating factors may be accommodated by preprocessing the data sets such that they all comply, e.g., by normalizing, realigning, interpolating, upsampling or downsampling, using features, or reducing dimensions, these procedures have the risk of being lossy in various respects (for further discussion on complicating factors in data fusion, see Section IV). For these reasons, more elaborate models that allow heterogeneous data sets to remain in their most explanatory form and still perform true data fusion, i.e., in the sense of Definition I.2 and Section V-A, have been devised. In the following, we discuss data fusion approaches that go beyond single matrix or tensor factorization. Our emphasis is on demonstrating how the concepts of true data fusion allow pushing even further the limits of extracting knowledge from data that were summarized in Section III-C2. We show how these properties are carried over to more elaborate data fusion models and how they can be reinforced into stronger properties that cannot be achieved using single-set single-modal data. In particular: 1) allowing more relaxed uniqueness conditions that admit more challenging scenarios: for example, more relaxed assumptions on the underlying factors, and the ability to resolve more latent variables (low-rank terms) in each data set; and 2) terms that are shared across data sets enjoy the same permutation at all data sets. This obviates the need for an additional step of identifying the arbitrarily ordered outputs of each individual decomposition and matching them, a task that generally cannot be accomplished without additional information, in a blind or data-driven context. Fixing the permutation reduces the number of degrees of freedom and thus enhances performance and interpretability. The following examples illustrate these points. Example III-D.1: Coupled Independent Component Analysis: Consider the ICA problem (6). It is well known that statistically independent real-valued Gaussian processes with independent and identically distributed (i.i.d.) samples, mixed by an invertible A, cannot be blindly separated based on their observations xðtÞ alone BIB008 , BIB004 . If several such data sets are considered simultaneously, however, without changing the model within each mixture, but allowing statistical dependence across data sets, then a unique and identifiable solution to all these mixtures, up to unavoidable scale and permutation ambiguities, exists  . This model, when not restricted to Gaussian i.i.d. samples, is known as independent vector analysis (IVA)  , BIB011 ,  and can be solved using second-order statistics (SOS) alone BIB015 , BIB018 . IVA was originally proposed to separate convolutive mixtures of audio signals BIB011 ,  (11) is solved separately, it is associated with an individual permutation matrix P ðmÞ . It is clear that proper separation and reconstruction of the I audio signals cannot be achieved if the elements of the same source at different frequency bins are not properly matched. The key point in IVA w.r.t. a collection of ICA is that it exploits statistical dependence among latent sources that belong to different mixtures, as illustrated in Fig. 1 . Under certain conditions, the IVA framework provides a single R Â R permutation matrix P ðmÞ ¼ P that applies to all the involved mixtures  ,  . The ability of IVA to obviate the need to match the outputs of M separate ICA soon turned out useful far beyond convolutive mixtures: it has since been applied to fMRI group data analysis BIB012 , BIB029 , multimodal fusion of several brain-imaging modalities BIB030 , and the analysis of temporal dynamic changes BIB031 . IVA extends CCA BIB001 and its multiset extension (MCCA)  , which have both been widely used for fusion  , BIB019 ,  , BIB020 - BIB007 , to the case where not only second-order statistics but all-order statistics are taken into account  . Recently, a generalization of IVA that allows decomposition into terms of rank larger than one has been proposed BIB032 - BIB040 . In addition, since IVA is a generalization of ICA, it readily accommodates additional types of diversity such as coloured (i.e., not spectrally flat) or nonstationary sources BIB018 , BIB023 (recall Section III-C1). Identifiability analysis of the multiple types of diversity in IVA is given in  and BIB024 . It should be noted that the uniqueness results for coupled CPD  (Example III-D.2) require at least one tensor of order larger than two in the coupled set and thus they cannot be applied to IVA. Example III-D.2: Coupled Tensor Decompositions: In multilinear algebra, an ongoing endeavour is to obtain uniqueness conditions on a tensor decomposition BIB003 ,  , BIB009 - BIB036 . The goal is to derive bounds that are as relaxed as possible on the largest R that still satisfies essential uniqueness BIB025 . As an example, two necessary conditions for the essential uniqueness of the CPD of a third-order tensor BIB016 are that ðA BÞ; ðC AÞ and ðB CÞ have full column rank; and minðk A ; k B ; k C Þ ! 2 ( 1 2 ) (e.g.,  and BIB009 ) where denotes the columnwise Khatri-Rao product and k A is the Kruskal-rank of matrix A, equal to the largest integer k A such that every subset of k A columns of A is linearly independent BIB003 . Consider now M third-order tensors X ðmÞ 2 C I m ÂJ m ÂK , m ¼ 1; . . . ; M, with the same factorization as BIB016 , that are coupled by sharing one factor  where the factor matrices of the mth tensor are A ðmÞ ¼ ½a This fundamental result extends to more elaborate scenarios. Uniqueness can be further improved if the order of (at least one of) the involved tensors increases  . This is analogous to the previously mentioned result (Section III-C2) for a single tensor, that increasing its order N relaxes the bound on R BIB005 , BIB006 . Adding assumptions such as individual uniqueness of one of the involved CPDs, full column rank of the shared factor C, or a specific structure such as a Vandermonde matrix, also reinforces the uniqueness of the whole decomposition  ,  . Finally, all these results can be extended to more elaborate tensor decompositions that are not limited to rank-1 terms  . Another benefit from coupling is that it helps relax the permutation ambiguity. Coupled tensor decompositions have a unique arbitrary permutation matrix in a manner that extends single-tensor results (9) BIB026 , multidimensional harmonic retrieval (MHR)  , chemometrics and psychometrics BIB016 , BIB027 , and more. See Fig. 2(a) for an example in metabolomics. Linked-mode analysis has also been proposed as a means to represent missing values: each tensor is a data set that by itself is complete, but as a whole, each data set has only partial information w.r.t. a larger array in which all these data sets are enclosed. This idea is accompanied by a more flexible coupling design where more than one mode may be shared between two tensors and the coupling may even involve only parts of modes, i.e., shared (sub) factors [69, Sec. 5.1.2]. Fig. 2(b) illustrates this idea. Missing values are further discussed in Section IV-B4. We now summarize Examples III-D.1 and III-D.2. In Section III-C, we have shown that both ICA and PARAFAC can provide sufficient diversity to overcome the indeterminacy problem inherent to FA. We then extended our discussion to jointly analyzing M such problems: We have shown that by properly defining a link between data sets, we can extend and reinforce uniqueness and identifiability beyond those obtained by individual analysis, up to the point of establishing uniqueness of otherwise nonunique scenarios. In PARAFAC, mixtures share certain factors, whereas in IVA, each mixture has its own individual parameters and the link is via statistical dependence between certain variables. Next, we have shown that all these models are flexible in the sense that they can easily be fine-tuned and modified in multiple ways, in order to better fit various real-life data. More specifically, they readily admit various types of diversity. A first generalization of these basic models is by relaxing the assumptions within each decomposition: allowing statistical dependence between latent sources of the same mixture in ICA (respectively, IVA) leads to independent subspace analysis (ISA)  - BIB021 (respectively, joint independent subspace analysis (JISA) BIB032 - BIB040 ) as well as other BSS models  , BIB033 . Relaxing the sum-of-rank-1-terms constraint in PARAFAC leads to more flexible tensor decompositions such as Tucker  , BIB002 , block term decomposition (BTD) BIB013 , three-way decomposition into directional components (DEDICOM)  , and others BIB034 . A second generalization is by combining several types of constraints and assumptions: for example, PARAFAC may be combined with statistical independence BIB010 ,  , nonnegativity, sparsity, as well as structure of the latent factors: Vandermonde  - BIB028 , Toeplitz BIB014 , among others BIB037 , BIB038 , BIB035 , BIB022 . A third generalization is increasing the number of types of observational diversity by increasing the tensor order BIB005 , BIB006 . A fourth is by linking data sets, leading to various coupled models, as explained in this section. When all these types of generalizations are taken into account, one obtains very general data fusion frameworks such as structured data fusion (SDF) BIB037 , coupled matrix and tensor factorization (CMTF) BIB027 , linked multiway component analysis (LMWCA) BIB038 , and others BIB017 - BIB039 . These generalizations, and many more, are further discussed in Section V. In all cases, the link between underlying factors at different modalities helps not only to enhance uniqueness but also to enable the same ordering for all decompositions, thus further enhancing performance, identifiability, and interpretability.