Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> Concepts of correlation and regression may be applied not only to ordinary one-dimensional variates but also to variates of two or more dimensions. Marksmen side by side firing simultaneous shots at targets, so that the deviations are in part due to independent individual errors and in part to common causes such as wind, provide a familiar introduction to the theory of correlation; but only the correlation of the horizontal components is ordinarily discussed, whereas the complex consisting of horizontal and vertical deviations may be even more interesting. The wind at two places may be compared, using both components of the velocity in each place. A fluctuating vector is thus matched at each moment with another fluctuating vector. The study of individual differences in mental and physical traits calls for a detailed study of the relations between sets of correlated variates. For example the scores on a number of mental tests may be compared with physical measurements on the same persons. The questions then arise of determining the number and nature of the independent relations of mind and body shown by these data to exist, and of extracting from the multiplicity of correlations in the system suitable characterizations of these independent relations. As another example, the inheritance of intelligence in rats might be studied by applying not one but s different mental tests to N mothers and to a daughter of each <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> The choosing of a set of factors likely to correspond to the real psychological unitary traits in a situation usually reduces to finding a satisfactory rotation in a Thurstone centroid analysis. Seven principles, three of which are new, are described whereby rotation may be determined and/or judged. It is argued that the most fundamental is the principle of “parallel proportional profiles” or “simultaneous simple structure.” A mathematical proof of the uniqueness of determination by this means is attempted and equations are suggested for discovering the unique position. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> The model for three-mode factor analysis is discussed in terms of newer applications of mathematical processes including a type of matrix process termed the Kronecker product and the definition of combination variables. Three methods of analysis to a type of extension of principal components analysis are discussed. Methods II and III are applicable to analysis of data collected for a large sample of individuals. An extension of the model is described in which allowance is made for unique variance for each combination variable when the data are collected for a large sample of individuals. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> An individual differences model for multidimensional scaling is outlined in which individuals are assumed differentially to weight the several dimensions of a common “psychological space”. A corresponding method of analyzing similarities data is proposed, involving a generalization of “Eckart-Young analysis” to decomposition of three-way (or higher-way) tables. In the present case this decomposition is applied to a derived three-way table of scalar products between stimuli for individuals. This analysis yields a stimulus by dimensions coordinate matrix and a subjects by dimensions matrix of weights. This method is illustrated with data on auditory stimuli and on perception of nations. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> Simple structure and other common principles of factor rotation do not in general provide strong grounds for attributing explanatory significance to the factors which they select. In contrast, it is shown that an extension of Cattell's principle of rotation to Proportional Profiles (PP) offers a basis for determining explanatory factors for three-way or higher order multi-mode data. Conceptual models are developed for two basic patterns of multi-mode data variation, systemand object-variation, and PP analysis is found to apply in the system-variation case. Although PP was originally formulated as a principle of rotation to be used with classic two-way factor analysis, it is shown to embody a latent three-mode factor model, which is here made explicit and generalized frown two to N "parallel occasions". As originally formulated, PP rotation was restricted to orthogonal factors. The generalized PP model is demonstrated to give unique "correct" solutions with oblique, non-simple structure, and even non-linear factor structures. A series of tests, conducted with synthetic data of known factor composition, demonstrate the capabilities of linear and non-linear versions of the model, provide data on the minimal necessary conditions of uniqueness, and reveal the properties of the analysis procedures when these minimal conditions are not fulfilled. In addition, a mathematical proof is presented for the uniqueness of the solution given certain conditions on the data. Three-mode PP factor analysis is applied to a three-way set of real data consisting of the fundamental and first three formant frequencies of 11 persons saying 8 vowels. A unique solution is extracted, consisting of three factors which are highly meaningful and consistent with prior knowledge and theory concerning vowel quality. The relationships between the three-mode PP model and Tucker's multi-modal model, McDonald's non-linear model and Carroll and Chang's multi-dimensional scaling model are explored. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> Abstract This paper introduces a new tool for functional neuroimage analysis: partial least squares (PLS). It is unique as a multivariate method in its choice of emphasis for analysis, that being the covariance between brain images and exogenous blocks representing either the experiment design or some behavioral measure. What emerges are spatial patterns of brain activity that represent the optimal association between the images and either of the blocks. This process differs substantially from other multivariate methods in that rather than attempting to predict the individual values of the image pixels, PLS attempts to explain the relation between image pixels and task or behavior. Data from a face encoding and recognition PET rCBF study are used to illustrate two types of PLS analysis: an activation analysis of task with images and a brain–behavior analysis. The commonalities across the two analyses are suggestive of a general face memory network differentially engaged during encoding and recognition. PLS thus serves as an important extension by extracting new information from imaging data that is not accessible through other currently used univariate and multivariate image analysis tools. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> There has been an ever-increasing interest in multi-disciplinary research on multisensor data fusion technology, driven by its versatility and diverse areas of application. Therefore, there seems to be a real need for an analytical review of recent developments in the data fusion domain. This paper proposes a comprehensive review of the data fusion state of the art, exploring its conceptualizations, benefits, and challenging aspects, as well as existing methodologies. In addition, several future directions of research in the data fusion community are highlighted and described. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> I. INTRODUCTION <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB010
Information about a phenomenon or a system of interest can be obtained from different types of instruments, measurement techniques, experimental setups, and other types of sources. Due to the rich characteristics of natural processes and environments, it is rare that a single acquisition method provides complete understanding thereof. The increasing availability of multiple data sets that contain information, obtained using different acquisition methods, about the same system, introduces new degrees of freedom that raise questions beyond those related to analyzing each data set separately. The foundations of modern data fusion have been laid in the first half of the 20th century BIB001 , BIB002 . Joint analysis of multiple data sets has since been the topic of extensive research, and earned a significant leap forward in the late 1960s/early 1970s with the formulation of concepts and techniques such as multiset canonical correlation analysis (CCA)  , parallel factor analysis (PARAFAC) BIB004 , BIB005 , and other tensor decompositions  , BIB003 . However, until rather recently, in most cases, these data fusion methodologies were confined within the limits of psychometrics and chemometrics, the communities in which they evolved. With recent technological advances, in a growing number of domains, the availability of data sets that correspond to the same phenomenon has increased, leading to increased interest in exploiting them efficiently. Many of the providers of multiview, multirelational, and multimodal data are associated with high-impact commercial, social, biomedical, environmental, and military applications, and thus the drive to develop new and efficient analytical methodologies is high and reaches far beyond pure academic interest. Motivations for data fusion are numerous. They include obtaining a more unified picture and global view of the system at hand; improving decision making; exploratory research; answering specific questions about the system, such as identifying common versus distinctive elements across modalities or time; and in general, extracting knowledge from data for various purposes. However, despite the evident potential benefit, and massive work that has already been done in the field (see, for example, BIB008 - BIB010 and references therein), the knowledge of how to actually exploit the additional diversity that multiple data sets offer is still at its very preliminary stages. Data fusion is a challenging task for several reasons  , BIB006 -  . First, the data are generated by very complex systems: biological, environmental, sociological, and psychological, to name a few, driven by numerous underlying processes that depend on a large number of variables to which we have no access. Second, due to the augmented diversity, the number, type, and scope of new research questions that can be posed is potentially very large. Third, working with heterogeneous data sets such that the respective advantages of each data set are maximally exploited, and drawbacks suppressed, is not an evident task. We elaborate on these matters in Sections II-IV. Most of these questions have been devised only in the very recent years, and, as we show in this paper, only a fraction of their potential has already been exploited. Hence, we refer to them as ''challenges.' ' A rather wide perspective on challenges in data fusion is presented in BIB008 , where Van Mechelen and Smilde discuss linked-mode decomposition models within the framework of chemometrics and psychometrics, and BIB009 , where Khaleghi et al. focus on ''automated decision making'' with special attention to multisensor information fusion. In practice, however, challenges in data fusion are most often brought up within a framework dedicated to a specific application, model, and data set; examples will be given in the sections that follow. In this paper, we bring together a comprehensive (but definitely not exhaustive) list of challenges in data fusion. Following from BIB008 , BIB009 , BIB010 , and  (and others), and further emphasized by our discussion in this paper, it is clear that at the appropriate level of abstraction, the same challenge in data fusion can be relevant to completely different and diverse applications, goals, and data types. Consequently, a solution to a challenge that is based on a sufficiently data-driven, model-free approach may turn out to be useful in very different domains. Therefore, there is an obvious interest in opening up the discussion of data fusion challenges to include and involve disparate communities, so that each community could inform the others. Our goal is to stimulate and emphasize the relevance and importance of a perspective based on challenges to advanced data fusion. More specifically, we would like to promote data-driven approaches, that is, approaches with minimal and weak priors and constraints, such as sparsity, nonnegativity, low-rank, and independence, among others, that can be useful to more than one specific application or data set. Hence, we present these challenges in quite a general framework that is not specific to an application, goal, or data type. We also give examples and motivations from different domains. In order to contain our discussion, we focus on setups in which a phenomenon or a system is observed using multiple instruments, measurement devices, or acquisition techniques. In this case, each acquisition framework is denoted as a modality and is associated with one data set. The whole setup, in which one has access to data obtained from multiple modalities, is known as multimodal. A key property of multimodality is complementarity, in the sense that each modality brings to the whole some type of added value that cannot be deduced or obtained from any of the other modalities in the setup. In mathematical terms, this added value is known as diversity. Diversity allows to reduce the number of degrees of freedom in the system by providing constraints that enhance uniqueness, interpretability, robustness, performance, and other desired properties, as will be illustrated in the rest of this paper. Diversity can be found in a broad range of scenarios, and plays a key role in a wide scope of mathematical and engineering studies. Accordingly, we suggest the following operative definition for the special type of diversity that is associated with multimodality. Definition I.1: Diversity (due to multimodality) is the property that allows to enhance the uses, benefits and insights (such as those discussed in Section II), in a way that cannot be achieved with a single modality. Diversity is the key to data fusion, as will be explained in Section III. Furthermore, in Section III, we demonstrate how a diversity approach to data fusion can provide a fresh new look on previously well-known and well-founded data and signal processing techniques. As already noted, ''data fusion'' is quite a diffuse concept that takes different interpretations with applications and goals BIB008 , BIB009 ,  . Therefore, within the context of this paper, and in accordance with the types of problems on which we focus, our emphasis is on the following tighter interpretation BIB007 : Definition I.2: Data fusion is the analysis of several data sets such that different data sets can interact and inform each other. This concept will be given a more concrete meaning in Sections III and V. The goal of this paper is to provide some ideas, perspectives, and guidelines as to how to approach data fusion. This paper is not a review, not a literature survey, not a tutorial, nor a cookbook. As such, it does not propose or promote any specific solution or method. On the contrary, our message is that whatever specific method or approach is considered, it should be kept in mind that it is just one among a very large set, and should be critically judged as such. In the same vein, any example in this paper should only be regarded as a concretization of a much broader idea. How to read this paper? In order to make this paper accessible for readers with various interests and backgrounds, it is organized in two types of cross sections. The first part (Sections II and III) deals with the question ''why?'', i.e., why we need data fusion. The second part (Sections IV and V) deals with the question ''how?'', i.e., how we perform data fusion. Each question is treated on two levels: data (Sections II and IV) and theory (Sections III and V). More specifically, Section II presents the concepts of multimodality and data fusion and motivates them using examples from various applications. In Section III, we introduce the concept of diversity as a key to data fusion and give it a concrete mathematical formulation. Section IV discusses complicating factors that should be addressed in the actual processing of heterogeneous data. Section V gives some guidelines as to how to actually approach a data fusion problem from a model design perspective. Section VI concludes our work.