Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> E. Validation <s> Abstract Recent technological advances enable us to collect huge amounts of data from multiple sources. Jointly analyzing such multi-relational data from different sources, i.e., data fusion (also called multi-block, multi-view or multi-set data analysis), often enhances knowledge discovery. For instance, in metabolomics, biological fluids are measured using a variety of analytical techniques such as Liquid Chromatography–Mass Spectrometry and Nuclear Magnetic Resonance Spectroscopy. Data measured using different analytical methods may be complementary and their fusion may help in the identification of chemicals related to certain diseases. Data fusion has proved useful in many fields including social network analysis, collaborative filtering, neuroscience and bioinformatics. In this paper, unlike many studies demonstrating the success of data fusion, we explore the limitations as well as the advantages of data fusion. We formulate data fusion as a coupled matrix and tensor factorization (CMTF) problem, which jointly factorizes multiple data sets in the form of higher-order tensors and matrices by extracting a common latent structure from the shared mode. Using numerical experiments on simulated and real data sets, we assess the performance of coupled analysis compared to the analysis of a single data set in terms of missing data estimation and demonstrate cases where coupled analysis outperforms analysis of a single data set and vice versa. <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> E. Validation <s> There has been an ever-increasing interest in multi-disciplinary research on multisensor data fusion technology, driven by its versatility and diverse areas of application. Therefore, there seems to be a real need for an analytical review of recent developments in the data fusion domain. This paper proposes a comprehensive review of the data fusion state of the art, exploring its conceptualizations, benefits, and challenging aspects, as well as existing methodologies. In addition, several future directions of research in the data fusion community are highlighted and described. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> E. Validation <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB003
Despite accumulating empirical evidence of the benefits of data fusion, there is still very little theoretical validation and quantitative measure of its gain  , BIB001 . Choosing an appropriate model is a widely open question, and approximate and highly simplified models are often preferred. Therefore, a validation step is indispensable. The following points are of particular interest. 1) Lower bounds on the best achievable error: How far are we from the best possible result (for a given data set, task, goal, and model)? 2) Theoretical results on the reliability and practical usefulness of the method: Can we prove that the model is identifiable? Is the solution unique? Is the output physically meaningful? Are the results sufficiently interpretable? IVA (Example III-D.1) and coupled tensor decompositions (Example III-D.2) are two of the models for which there now exists a comprehensive theoretical analysis that answers this type of questions. Although these questions are not specific to multimodal data fusion, they take special interpretation in the presence of multiple data sets. Some of the new questions that arise are as follows. 1) What is the mathematical formulation of ''success,'' ''optimality,'' and ''error,'' when heterogeneous modalities and types of uncertainty are involved? What is the most appropriate target function and criterion of success? 2) How do we evaluate performance of exploratory tasks? 3) How do we design a figure of merit that can inform us how to exploit the advantages of each modality without suffering from its drawbacks w.r.t. the other modalities? 4) How do we identify and process information that is shared by several modalities, and how do we identify and exploit modality-specific information? 5) How do we compare alternative design choices such as of level of data fusion, order selection, and analytical model within and across modalities? As an example, theoretical figures of merit such as the Cramér-Rao lower bound may help answer some of these questions. However, calculating theoretical error bounds for all possible alternatives (especially in view of the modular approach of Section V-D) is a prohibitive task, both due to the very large number of options, and also since many models are not mathematically tractable. Rescue may come from the computational front. As an example, Tensorlab [208] , a Matlab toolbox that follows the modular principles of SDF BIB003 (Section V-D), enables the user to switch between the numerous combinations arising from multiple choices in the model design. As such, it allows the user to rapidly iterate toward a plausible solution for the problem at hand. Therefore, Tensorlab [208] (or any other computational tool following the modularity principles) may serve as a verification and validation tool, at least in the preliminary stages of the design. A class on their own are questions regarding the choice of modalities and the added value from using multiple modalities in general. 1) Should all available modalities be used, and/or given equal importance? 2) How much (information, diversity, redundancy) does each modality bring in to the total equations? How do we quantify this ''extra contribution''? Some of these questions (and examples of possible answers) have been brought up within the challenges in Section IV; others are related to the design of a data fusion model (Section V). Information theory seems like a natural framework to evaluate the contribution of various types of diversity, as discussed, e.g., in  . Uniqueness analysis of (coupled) tensor decompositions, as well as other forms of error analysis, such as those mentioned in Section -III, quantify the added value of diversity in terms of the admissible number of uniquely identifiable components or factors. Attention should be paid, for example, when modalities are too close to each other: in this case, they may not really convey new information; in addition, they may be exposed to similar noise, and thus bias results BIB002 . Due to the heterogeneous characteristics of the data, and particularly in exploratory tasks, the interpretability of the output should be given special care. Questions related to the representation of the output of multimodal data analysis are discussed, e.g., in  Sec. 8] .