Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Concepts of correlation and regression may be applied not only to ordinary one-dimensional variates but also to variates of two or more dimensions. Marksmen side by side firing simultaneous shots at targets, so that the deviations are in part due to independent individual errors and in part to common causes such as wind, provide a familiar introduction to the theory of correlation; but only the correlation of the horizontal components is ordinarily discussed, whereas the complex consisting of horizontal and vertical deviations may be even more interesting. The wind at two places may be compared, using both components of the velocity in each place. A fluctuating vector is thus matched at each moment with another fluctuating vector. The study of individual differences in mental and physical traits calls for a detailed study of the relations between sets of correlated variates. For example the scores on a number of mental tests may be compared with physical measurements on the same persons. The questions then arise of determining the number and nature of the independent relations of mind and body shown by these data to exist, and of extracting from the multiplicity of correlations in the system suitable characterizations of these independent relations. As another example, the inheritance of intelligence in rats might be studied by applying not one but s different mental tests to N mothers and to a daughter of each <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> This paper is concerned with the study of similarities and differences in factor structures between different groups. A common situation occurs when a battery of tests has been administered to samples of examinees from several populations.A very general model is presented, in which any parameter in the factor analysis models (factor loadings, factor variances, factor covariances, and unique variances) for the different groups may be assigned an arbitrary value or constrained to be equal to some other parameter. Given such a specification, the model is estimated by the maximum likelihood method yielding a large samplex2 of goodness of fit. By computing several solutions under different specifications one can test various hypotheses.The method is capable of dealing with any degree of invariance, from the one extreme, where nothing is invariant, to the other extreme, where everything is invariant. Neither the number of tests nor the number of common factors need to be the same for all groups, but to be at all interesting, it is assumed that there is a common core of tests in each battery that is the same or at least content-wise comparable. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Abstract Finding the means to efficiently summarize electroencephalographic data has been a long-standing problem in electrophysiology. A popular approach is identification of component modes on the basis of the time-varying spectrum of multichannel EEG recordings—in other words, a space/frequency/time atomic decomposition of the time-varying EEG spectrum. Previous work has been limited to only two of these dimensions. Principal Component Analysis (PCA) and Independent Component Analysis (ICA) have been used to create space/time decompositions; suffering an inherent lack of uniqueness that is overcome only by imposing constraints of orthogonality or independence of atoms. Conventional frequency/time decompositions ignore the spatial aspects of the EEG. Framing of the data being as a three-way array indexed by channel, frequency, and time allows the application of a unique decomposition that is known as Parallel Factor Analysis (PARAFAC). Each atom is the tri-linear decomposition into a spatial, spectral, and temporal signature. We applied this decomposition to the EEG recordings of five subjects during the resting state and during mental arithmetic. Common to all subjects were two atoms with spectral signatures whose peaks were in the theta and alpha range. These signatures were modulated by physiological state, increasing during the resting stage for alpha and during mental arithmetic for theta. Furthermore, we describe a new method (Source Spectra Imaging or SSI) to estimate the location of electric current sources from the EEG spectrum. The topography of the theta atom is frontal and the maximum of the corresponding SSI solution is in the anterior frontal cortex. The topography of the alpha atom is occipital with maximum of the SSI solution in the visual cortex. We show that the proposed decomposition can be used to search for activity with a given spectral and topographic profile in new recordings, and that the method may be useful for artifact recognition and removal. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> We present a unified view of matrix factorization that frames the differences among popular methods, such as NMF, Weighted SVD, E-PCA, MMMF, pLSI, pLSI-pHITS, Bregman co-clustering, and many others, in terms of a small number of modeling choices. Many of these approaches can be viewed as minimizing a generalized Bregman divergence, and we show that (i) a straightforward alternating projection algorithm can be applied to almost any model in our unified view; (ii) the Hessian for each projection has special structure that makes a Newton projection feasible, even when there are equality constraints on the factors, which allows for matrix co-clustering; and (iii) alternating projections can be generalized to simultaneously factor a set of matrices that share dimensions. These observations immediately yield new optimization algorithms for the above factorization methods, and suggest novel generalizations of these methods such as incorporating row and column biases, and adding or relaxing clustering constraints. <s> BIB004 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Relational learning is concerned with predicting unknown values of a relation, given a database of entities and observed relations among entities. An example of relational learning is movie rating prediction, where entities could include users, movies, genres, and actors. Relations encode users' ratings of movies, movies' genres, and actors' roles in movies. A common prediction technique given one pairwise relation, for example a #users x #movies ratings matrix, is low-rank matrix factorization. In domains with multiple relations, represented as multiple matrices, we may improve predictive accuracy by exploiting information from one relation while predicting another. To this end, we propose a collective matrix factorization model: we simultaneously factor several matrices, sharing parameters among factors when an entity participates in multiple relations. Each relation can have a different value type and error distribution; so, we allow nonlinear relationships between the parameters and outputs, using Bregman divergences to measure error. We extend standard alternating projection algorithms to our model, and derive an efficient Newton update for the projection. Furthermore, we propose stochastic optimization methods to deal with large, sparse matrices. Our model generalizes several existing matrix factorization methods, and therefore yields new large-scale optimization algorithms for these problems. Our model can handle any pairwise relational schema and a wide variety of error models. We demonstrate its efficiency, as well as the benefit of sharing parameters among relations. <s> BIB005 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors. <s> BIB006 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> As a consequence of our information society, not only more and larger data sets become available, but also data sets that include multiple sorts of information regarding the same system. Such data sets can be denoted by the terms coupled, linked, or multiset data, and the associated data analysis can be denoted by the term data fusion. In this paper, we first give a formal description of coupled data, which allows the data-analyst to typify the structure of a coupled data set at hand. Second, we list two meta-questions and a series of complicating factors that may be useful to focus the initial content-driven research questions that go with coupled data, and to choose a suitable data-analytic method. Third, we propose a generic framework for a family of decomposition-based models pertaining to an important subset of data fusion problems. This framework is intended to constitute both a means to arrive at a better understanding of the features and the interrelations of the specific models subsumed by it, and as a powerful device for the development of novel, custom-made data fusion models. We conclude the paper by showing how the proposed formal data description, meta-questions, and generic model may assist the data-analyst in choosing and developing suitable strategies for the treatment of coupled data in practice. Throughout the paper we illustrate with examples from the domain of systems biology. <s> BIB007 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Abstract The problem of incomplete data – i.e., data with missing or unknown values – in multi-way arrays is ubiquitous in biomedical signal processing, network traffic analysis, bibliometrics, social network analysis, chemometrics, computer vision, communication networks, etc. We consider the problem of how to factorize data sets with missing values with the goal of capturing the underlying latent structure of the data and possibly reconstructing missing values (i.e., tensor completion). We focus on one of the most well-known tensor factorizations that captures multi-linear structure, CANDECOMP/PARAFAC (CP). In the presence of missing data, CP can be formulated as a weighted least squares problem that models only the known entries. We develop an algorithm called CP-WOPT (CP Weighted OPTimization) that uses a first-order optimization approach to solve the weighted least squares problem. Based on extensive numerical experiments, our algorithm is shown to successfully factorize tensors with noise and up to 99% missing data. A unique aspect of our approach is that it scales to sparse large-scale data, e.g., 1000 × 1000 × 1000 with five million known entries (0.5% dense). We further demonstrate the usefulness of CP-WOPT on two real-world applications: a novel EEG (electroencephalogram) application where missing data is frequently encountered due to disconnections of electrodes and the problem of modeling computer network traffic where data may be absent due to the expense of the data collection process. <s> BIB008 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> In many areas of science, research questions imply the analysis of a set of coupled data blocks, with, for instance, each block being an experimental unit by variable matrix, and the variables being the same in all matrices. To obtain an overall picture of the mechanisms that play a role in the different data matrices, the information in these matrices needs to be integrated. This may be achieved by applying a data-analytic strategy in which a global model is fitted to all data matrices simultaneously, as in some forms of simultaneous component analysis (SCA). Since such a strategy implies that all data entries, regardless the matrix they belong to, contribute equally to the analysis, it may obfuscate the overall picture of the mechanisms underlying the data when the different data matrices are subject to different amounts of noise. One way out is to downweight entries from noisy data matrices in favour of entries from less noisy matrices. Information regarding the amount of noise that is present in each matrix, however, is, in most cases, not available. To deal with these problems, in this paper a novel maximum-likelihood-based simultaneous component analysis method, referred to as MxLSCA, is proposed. Being a stochastic extension of SCA, in MxLSCA the amount of noise in each data matrix is estimated and entries from noisy data matrices are downweighted. Both in an extensive simulation study and in an application to data stemming from cross-cultural emotion psychology, it is shown that the novel MxLSCA strategy outperforms the SCA strategy with respect to disclosing the mechanisms underlying the coupled data. <s> BIB009 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> In many research domains different pieces of information are collected regarding the same set of objects. Each piece of information constitutes a data block, and all these (coupled) blocks have the object mode in common. When analyzing such data, an important aim is to obtain an overall picture of the structure underlying the whole set of coupled data blocks. A further challenge consists of accounting for the differences in information value that exist between and within (i.e., between the objects of a single block) data blocks. To tackle these issues, analysis techniques may be useful in which all available pieces of information are integrated and in which at the same time noise heterogeneity is taken into account. For the case of binary coupled data, however, only methods exist that go for a simultaneous analysis of all data blocks but that do not account for noise heterogeneity. Therefore, in this paper, the SIMCLAS model, being a Hierarchical Classes model for the simultaneous analysis of coupled binary two-way matrices, is presented. In this model, noise heterogeneity between and within the data blocks is accounted for by downweighting entries from noisy blocks/objects within a block. In a simulation study it is shown that (1) the SIMCLAS technique recovers the underlying structure of coupled data to a very large extent, and (2) the SIMCLAS technique outperforms a Hierarchical Classes technique in which all entries contribute equally to the analysis (i.e., noise homogeneity within and between blocks). The latter is also demonstrated in an application of both techniques to empirical data on categorization of semantic concepts. <s> BIB010 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Incorporating domain-specific side information via coupled factorization models is useful in source separation applications. Coupled models can easily incorporate information from source modalities with different statistical properties by estimating shared factors via divergence minimization. Here, it is useful to use mixed divergences, a specific divergence for each modality. However, this extra freedom requires choosing the correct divergence as well as an optimal weighting mechanism to select the relative ‘importance’. In this paper, we present an approach for determining the relative weights, framed as dispersion parameter estimation, based on an inference framework introduced previously as Generalized Coupled Tensor Factorization (GCTF). The dispersion parameters play a key role on inference as they form a balance between the information obtained from multimodal observations. We tackle the problem of optimal weighting by maximum likelihood exploiting the relation between β-divergences and the family of Tweedie distributions. We demonstrate the usefulness of our approach on a drum source separation application. <s> BIB011 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> This study deals with missing link prediction, the problem of predicting the existence of missing connections between entities of interest. We approach the problem as filling in missing entries in a relational dataset represented by several matrices and multiway arrays, that will be simply called tensors. Consequently, we address the link prediction problem by data fusion formulated as simultaneous factorization of several observation tensors where latent factors are shared among each observation. Previous studies on joint factorization of such heterogeneous datasets have focused on a single loss function (mainly squared Euclidean distance or Kullback---Leibler-divergence) and specific tensor factorization models (CANDECOMP/PARAFAC and/or Tucker). However, in this paper, we study various alternative tensor models as well as loss functions including the ones already studied in the literature using the generalized coupled tensor factorization framework. Through extensive experiments on two real-world datasets, we demonstrate that (i) joint analysis of data from multiple sources via coupled factorization significantly improves the link prediction performance, (ii) selection of a suitable loss function and a tensor factorization model is crucial for accurate missing link prediction and loss functions that have not been studied for link prediction before may outperform the commonly-used loss functions, (iii) joint factorization of datasets can handle difficult cases, such as the cold start problem that arises when a new entity enters the dataset, and (iv) our approach is scalable to large-scale data. <s> BIB012 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> In many disciplines, data from multiple sources are acquired and jointly analyzed for enhanced knowledge discovery. For instance, in metabolomics, different analytical techniques are used to measure biological fluids in order to identify the chemicals related to certain diseases. It is widely-known that, some of these analytical methods, e.g., LC-MS (Liquid Chromatography - Mass Spectrometry) and NMR (Nuclear Magnetic Resonance) spectroscopy, provide complementary data sets and their joint analysis may enable us to capture a larger proportion of the complete metabolome belonging to a specific biological system. Fusing data from multiple sources has proved useful in many fields including bioinformatics, signal processing and social network analysis. However, identification of common (shared) and individual (unshared) structures across multiple data sets remains a major challenge in data fusion studies. With a goal of addressing this challenge, we propose a novel unsupervised data fusion model. Our contributions are two-fold: (i) We formulate a data fusion model based on joint factorization of matrices and higher-order tensors, which can automatically reveal common and individual components. (ii) We demonstrate that the proposed approach provides promising results in joint analysis of metabolomics data sets consisting of fluorescence and NMR measurements of plasma samples in terms of separation of colorectal cancer patients from controls. <s> BIB013 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Tensor decompositions are at the core of many blind source separation (BSS) algorithms, either explicitly or implicitly. In particular, the canonical polyadic (CP) tensor decomposition plays a central role in the identification of underdetermined mixtures. Despite some similarities, CP and singular value decomposition (SVD) are quite different. More generally, tensors and matrices enjoy different properties, as pointed out in this brief introduction. <s> BIB014 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> This work introduces a new framework for nonnegative matrix factorization (NMF) in multisensor or multimodal data configurations, where taking into account the mutual dependence that exists between the related parallel streams of data is expected to improve performance. In contrast with previous works that focused on co-factorization methods -where some factors are shared by the different modalities-we propose a soft co-factorization scheme which accounts for possible local discrepancies across modalities or channels. This objective is formalized as an optimization problem where concurrent factorizations are jointly performed while being tied by a coupling term that penalizes differences between the related factor matrices associated with different modalities. We provide majorization-minimization (MM) algorithms for three common measures of fit-the squared Euclidean norm, the Kullback-Leibler divergence and the Itakura-Saito divergence-and two possible coupling variants, using either the l1 or the squared Euclidean norm of differences. The approach is shown to achieve promising performance in two audio-related tasks: multimodal speaker diarization using audiovisual data and audio source separation using stereo data. <s> BIB015 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Higher-order tensors and their decompositions are abundantly present in domains such as signal processing (e.g., higher-order statistics [1] and sensor array processing [2]), scientific computing (e.g., discretized multivariate functions [3]?[6]), and quantum information theory (e.g., representation of quantum many-body states [7]). In many applications, the possibly huge tensors can be approximated well by compact multilinear models or decompositions. Tensor decompositions are more versatile tools than the linear models resulting from traditional matrix approaches. Compared to matrices, tensors have at least one extra dimension. The number of elements in a tensor increases exponentially with the number of dimensions, and so do the computational and memory requirements. The exponential dependency (and the problems that are caused by it) is called the curse of dimensionality. The curse limits the order of the tensors that can be handled. Even for a modest order, tensor problems are often large scale. Large tensors can be handled, and the curse can be alleviated or even removed by using a decomposition that represents the tensor instead of using the tensor itself. However, most decomposition algorithms require full tensors, which renders these algorithms infeasible for many data sets. If a tensor can be represented by a decomposition, this hypothesized structure can be exploited by using compressed sensing (CS) methods working on incomplete tensors, i.e., tensors with only a few known elements. <s> BIB016 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> This article combines a tutorial on state-of-the-art tensor decomposition as it relates to big data analytics, with original research on parallel and distributed computation of low-rank decomposition for big tensors, and a concise primer on Hadoop?MapReduce. A novel architecture for parallel and distributed computation of low-rank tensor decomposition that is especially well suited for big tensors is proposed. The new architecture is based on parallel processing of a set of randomly compressed, reduced-size replicas of the big tensor. Each replica is independently decomposed, and the results are joined via a master linear equation per tensor mode. The approach enables massive parallelism with guaranteed identifiability properties: if the big tensor is of low rank and the system parameters are appropriately chosen, then the rank-one factors of the big tensor will indeed be recovered from the analysis of the reduced-size replicas. Furthermore, the architecture affords memory/storage and complexity gains of order for a big tensor of size of rank F with No sparsity is required in the tensor or the underlying latent factors, although such sparsity can be exploited to improve memory, storage, and computational savings. <s> BIB017 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> Many problems in computational neuroscience, neuroinformatics, pattern/image recognition, signal processing and machine learning generate massive amounts of multidimensional data with multiple aspects and high dimensionality. Tensors (i.e., multi-way arrays) provide often a natural and compact representation for such massive multidimensional data via suitable low-rank approximations. Big data analytics require novel technologies to efficiently process huge datasets within tolerable elapsed times. Such a new emerging technology for multidimensional big data is a multiway analysis via tensor networks (TNs) and tensor decompositions (TDs) which represent tensors by sets of factor (component) matrices and lower-order (core) tensors. Dynamic tensor analysis allows us to discover meaningful hidden structures of complex data and to perform generalizations by capturing multi-linear and multi-aspect relationships. We will discuss some fundamental TN models, their mathematical and graphical descriptions and associated learning algorithms for large-scale TDs and TNs, with many potential applications including: Anomaly detection, feature extraction, classification, cluster analysis, data fusion and integration, pattern recognition, predictive modeling, regression, time series analysis and multiway component analysis. ::: Keywords: Large-scale HOSVD, Tensor decompositions, CPD, Tucker models, Hierarchical Tucker (HT) decomposition, low-rank tensor approximations (LRA), Tensorization/Quantization, tensor train (TT/QTT) - Matrix Product States (MPS), Matrix Product Operator (MPO), DMRG, Strong Kronecker Product (SKP). <s> BIB018 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> We present structured data fusion (SDF) as a framework for the rapid prototyping of knowledge discovery in one or more possibly incomplete data sets. In SDF, each data set—stored as a dense, sparse, or incomplete tensor—is factorized with a matrix or tensor decomposition. Factorizations can be coupled, or fused, with each other by indicating which factors should be shared between data sets. At the same time, factors may be imposed to have any type of structure that can be constructed as an explicit function of some underlying variables. With the right choice of decomposition type and factor structure, even well-known matrix factorizations such as the eigenvalue decomposition, singular value decomposition and QR factorization can be computed with SDF. A domain specific language (DSL) for SDF is implemented as part of the software package Tensorlab, with which we offer a library of tensor decompositions and factor structures to choose from. The versatility of the SDF framework is demonstrated by means of four diverse applications, which are all solved entirely within Tensorlab’s DSL. <s> BIB019 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods. <s> BIB020 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> A Bayesian framework is proposed to define flexible coupling models for joint decompositions of data sets. Under this framework, a solution to the joint decomposition can be cast in terms of a maximum a posteriori estimator. Examples of joint posterior distributions are provided, including general Gaussian priors and non Gaussian coupling priors. Then simulations are reported and show the effectiveness of this approach to fuse information from data sets, which are inherently of different size due to different time resolution of the measurement devices. <s> BIB021 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> D. Structured Data Fusion: A General Mathematical Framework <s> The estimation of directions of arrival is formulated as the decomposition of a 3-way array into a sum of rank-one terms, which is possible when the receive array enjoys some geometrical structure. The main advantage is that this decomposition is essentially unique under mild assumptions, if computed exactly. The drawback is that a low-rank approximation does not always exist. Therefore, a coherence constraint is introduced that ensures the existence of the latter best approximate, which allows to localize and estimate closely located or highly correlated sources. Then Cramer-Rao bounds are derived for localization parameters and source signals, assuming the others are nuisance parameters; some inaccuracies found in the literature are pointed out. Performances are eventually compared with unconstrained reference algorithms such as ESPRIT, in the presence of additive complex Gaussian noise, with possibly noncircular distribution. <s> BIB022
In Sections III, IV and V.A-V.C, we mentioned a large number of data fusion models. However, it is clear that no list of existing solutions, comprehensive as it might be, can cover the practically endless number of current, future and potential data sets, problems and tasks. Indeed, the purpose of this paper is not in promoting specific models or methods. Instead, and building upon BIB007 -  , BIB019 , BIB002 , BIB004 , BIB005 , and others, we wish to provide a deeper and broader understanding of the concepts and ideas that underlie data fusion. As such, in the model design front, our goal is providing guidelines and insights that may apply also to data sets, problems, and tasks that do not necessarily conform to any of the specific examples, solutions, and mathematical frameworks that we mention. The concept of diversity, presented in Section III, is one such example. In the same vein, we now present a general mathematical framework that will allow us to give a more concrete meaning to some of the model design concepts that have been discussed. Although this formulation is given in terms of matrices and higher order arrays, also known as tensors, the underlying ideas behind ''structured data fusion'' BIB019 , such as flexibility and modularity, are not limited to these. The mathematical formulation that we use is only a concretization of a more general idea, applied to data sets that admit certain types of decompositions. We now present a formulation proposed by Sorber et al. BIB019 , followed by a few examples for motivation and clarification. Model of an individual data set: Consider an ensemble of M data sets, collected in M arrays (tensors) T ðmÞ 2 C I 1 ÂÁÁÁÂI Nm , m ¼ 1; . . . ; M, where N m ¼ 1; 2; 3; . . . implies a vector, a matrix or a higher order tensor, respectively. In order to allow maximal flexibility in the model associated with each of these data sets, Sorber et al. BIB019 define several layers of underlying structures. The first layer is an ordered set of V variables z ¼ fz 1 ; . . . ; z V g, where each variable may be anything from a scalar to a higher order tensor, real or complex [recall BIB001 ]. The second layer is an ordered set of F factors XðzÞ ¼ fx 1 ðz i 1 Þ; . . . ; x F ðz i F Þg that are driven by the V variables z. Each factor x f ðz i f Þ is a mapping of the i f th variable to a tensor. In the third layer, each data set T ðmÞ is associated with a decomposition model M ðmÞ that approximates it. Function M ðmÞ ðX ðzÞÞ maps a subgroup of the factors XðzÞ to a tensor. Fig. 3 illustrates these layers. For simplicity, each data set T ðmÞ is associated with a tensor model M ðmÞ of the same order and size. This is not evident: the order N m and dimensions I 1 Â Á Á Á Â I N m of the model tensor, as used in the analysis, may differ from those that most naturally represent the acquired data, as well as from the natural way to visualize the samples. As a first example, raw EEG data are a time series in several electrodes, i.e., electrode Â time. However, it has been proposed to augment these data using a third type of diversity, electrode Â time Â frequency  - BIB003 . In this case, the EEG data will be stacked in a third-order tensor. On the other hand, data that are naturally visualized in 2-D or 3-D arrays such as images do not necessarily admit any useful (multi)linear relationships among their pixels or voxels. Hence, in the analysis, image data are often vectorized into 1-D arrays. Further discussion of how to choose the right array structure for data analysis can be found, e.g., in BIB007 , BIB006 , BIB020 , BIB014 , and references therein. Link between data sets: In the SDF formulation, a link between data sets can be established if their models share at least one factor or variable. This corresponds to the ''hard'' links, mentioned in Section V-B1. However, this does not exclude other types of interaction between data sets: ''soft'' links may be established by reformulating ''soft'' links using shared parameters, as explained in Section V-B1, and possibly via regularization terms. The following examples provide a more concrete meaning to the mathematical formulation that we have just laid out. Consider two matrix data sets, patients Â diagnosis and patients Â symptoms. A latent variable may be the syndrome that underlies both diagnoses and symptoms factors. The link is established via the shared ''patients'' mode BIB010 . As a second example, certain properties of a communication system in a coupled CPD framework may be expressed using a factor with a Vandermonde structure  . This can be implemented in SDF with z f a vector of p scalars and x f ðz f Þ a p Â q Vandermonde matrix constructed thereof. Further examples for factor structures that can be reformulated in SDF include orthogonality, Toeplitz structure, nonnegativity, as well as fixed and known entries. Constraints such as sparsity and smoothness within factors may be implemented using regularization terms. Tensor decompositions that can be reformulated as M ðmÞ ðX ðzÞÞ include rank-1 decompositions (CPD, PARAFAC), decompositions into rank ! 1 terms, Tucker, BTD, PARAFAC2  , and many others. These options, as well as other alternatives for latent variables, factors, and models for each data set, are mentioned in Section III. For further explanations about implementation, see BIB019 and [208] . Loss/objective function, regularization, and penalty terms: The next step is to fit the model to the data. Depending on the analytical framework that we choose (Section V-C), each data set or the whole ensemble is attributed with a loss/objective function D ðmÞ ðÁ; ÁÞ, between observed and modeled data BIB009 , BIB011 , BIB012 . An individual loss/objective function allows flexibility both in the analytical framework applied to each data set, and in the individual types of uncertainty (Challenge IV-B.1). The loss/objective function may be complemented by various penalty or regularization terms, in order to impose constraints that are not expressed by the other optimization functionals. Regularization terms may impose certain types of sparsity, nonnegativity BIB013 , similarity BIB021 , BIB015 , or coherence BIB022 , to name a few. Missing values: In order not to take account of unknown data entries in the optimization procedure, these values are masked. This is done via an entrywise (Hadamard) product (denoted as Ã )) of the data tensor T ðmÞ with a binary tensor B ðmÞ of the same size; see, e.g., BIB019 , BIB008 , and references therein. Missing values are discussed in Challenge IV-B.4. The whole optimization problem: Given these elements, SDF may be written as the optimization problem The optimization problem  implements the overall analytical framework associated with the model. Equation (14) Fig. 3 . As noted in BIB019 , a large number of the existing data fusion models can be reformulated in terms of SDF, or some variation thereof. However, an even more interesting insight is that each step in the design of (14) is independent of the others: to a large extent, the choice of constraints, assumptions, types of links, loss/objective functions, and other parameters can be done disjointly. In other words, SDF is a modular approach to data fusion. These insights have led to the key observation that in fact, a large number of existing data fusion models can be regarded as composed of a rather small number of building blocks BIB019 . In other words, if all admissible combinations are considered, then the number of potential analytical data fusion models is significantly larger than what is currently available in the literature BIB007 . The modular perspective on data fusion offers several benefits. First, a major challenge in data fusion is its augmented complexity due to the increased number of degrees of freedom. The modular approach to data fusion answers this challenge by reformulating the problem in a small set of disjoint simpler components that can be separately analyzed, optimized, and coded. Second, the modular approach, in which the problem is factorized into smaller standalone elements, allows a broader view that makes it easier to come up with new combinations of the basic building blocks, thus leading to new mathematical models, algorithms, and concepts BIB007 , BIB019 , BIB004 , BIB005 . Third, modularity of the formulation makes it easier to adapt it to computational challenges such as large-scale data BIB019 , BIB020 , BIB016 , BIB017 , BIB018 . Fourth, in Sections I and II, we have emphasized the importance of exploratory research in data fusion. The modularity of the design is particularly helpful in that, making it straightforward to come up with new exploratory variations, to test and compare alternatives with minimal effort BIB019 . Modularity allows to easily diagnose which elements in the model are particularly useful, need to be modified, replaced or finetuned, without having to undo the whole derivation, coding or analysis. The latter also facilitates the validation stage; see Section V-E for further discussion.