A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Is perception of the whole based on perception of its parts? There is psychological and physiological evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign. <s> BIB001 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> This chapter describes gene expression analysis by Singular Value Decomposition (SVD), emphasizing initial characterization of the data. We describe SVD methods for visualization of gene expression data, representation of the data using a smaller number of variables, and detection of patterns in noisy gene expression data. In addition, we describe the precise relation between SVD analysis and Principal Component Analysis (PCA) when PCA is calculated using the covariance matrix, enabling our descriptions to apply equally well to either method. Our aim is to provide definitions, interpretations, examples, and references that will serve as resources for understanding and extending the application of SVD and PCA to gene expression analysis. <s> BIB002 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> A generic preprocessing method for multivariate data, called orthogonal projections to latent structures (O-PLS), is described. O-PLS removes variation from X (descriptor variables) that is not correlated to Y (property variables, e.g. yield, cost or toxicity). In mathematical terms this is equivalent to removing systematic variation in X that is orthogonal to Y. In an earlier paper, Wold et al. (Chemometrics Intell. Lab. Syst. 1998; 44: 175-185) described orthogonal signal correction (OSC). In this paper a method with the same objective but with different means is described. The proposed O-PLS method analyzes the variation explained in each PLS component. The non-correlated systematic variation in X is removed, making interpretation of the resulting PLS model easier and with the additional benefit that the non-correlated variation itself can be analyzed further. As an example, near-infrared (NIR) reflectance spectra of wood chips were analyzed. Applying O-PLS resulted in reduced model complexity with preserved prediction ability, effective removal of non-correlated variation in X and, not least, improved interpretational ability of both correlated and non-correlated variation in the NIR spectra. <s> BIB003 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> In this paper the O-PLS method [1] has been modified to further improve its interpretational functionality to give (a) estimates of the pure constituent profiles in X as well as model (b) the Y-orthogonal variation in X, (c) the X-orthogonal variation in Y and (d) the joint X-Y covariation. It is also predictive in both ways, X Y. We call this the O2-PLS approach. In earlier papers we discussed the improved interpretation using O-PLS compared to the partial least squares projections to latent structures (PLS) when systematic Y-orthogonal variation in X exists, i.e. when a PLS model has more components than the number of Y variables. In this paper we show how the parameters in the PLS model are affected and to what degree the interpretational ability of the PLS components changes with the amount of Y-orthogonal variation. In both real and synthetic examples, the O2-PLS method provided improved interpretation of the model and gave a good estimate of the pure constituent profiles, and the prediction ability was similar to the standard PLS model. The method is discussed from geometric and algebraic points of view, and a detailed description of this modified O2-PLS method is given and reviewed. <s> BIB004 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> The O2-PLS method is derived from the basic partial least squares projections to latent structures (PLS) prediction approach. The importance of the covariation matrix (YX) is pointed out in relation to both the prediction model and the structured noise in both X and Y. Structured noise in X (or Y) is defined as the systematic variation of X (or Y) not linearly correlated with Y (or X). Examples in spectroscopy include baseline, drift and scatter effects. If structured noise is present in X, the existing latent variable regression (LVR) methods, e.g. PLS, will have weakened score-loading correspondence beyond the first component. This negatively affects the interpretation of model parameters such as scores and loadings. The O2-PLS method models and predicts both X and Y and has an integral orthogonal signal correction (OSC) filter that separates the structured noise in X and Y from their joint X-Y covariation used in the prediction model. This leads to a minimal number of predictive components with full score-loading correspondence and also an opportunity to interpret the structured noise. In both a real and a simulated example, O2-PLS and PLS gave very similar predictions of Y. However, the interpretation of the prediction models was clearly improved with O2-PLS, because structured noise was present. In the NIR example, O2-PLS revealed a strong water peak and baseline offset in the structured noise components. In the simulated example the O2-PLS plot of observed versus predicted Y-scores (u vs U) showed good predictions. The corresponding loading vectors provided good interpretation of the covarying analytes in X and Y. <s> BIB005 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Partial least squares regression (PLSR) is a linear regression technique developed to deal with high-dimensional regressors and one or several response variables. In this paper we introduce robustified versions of the SIMPLS algorithm, this being the leading PLSR algorithm because of its speed and efficiency. Because SIMPLS is based on the empirical cross-covariance matrix between the response variables and the regressors and on linear least squares regression, the results are affected by abnormal observations in the data set. Two robust methods, RSIMCD and RSIMPLS, are constructed from a robust covariance matrix for high-dimensional data and robust linear regression. We introduce robust RMSECV and RMSEP values for model calibration and model validation. Diagnostic plots are constructed to visualize and classify the outliers. Several simulation results and the analysis of real data sets show the effectiveness and robustness of the new approaches. Because RSIMPLS is roughly twice as fast as RSIMCD, it stands out as the overall best method. Copyright © 2003 John Wiley & Sons, Ltd. <s> BIB006 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> When large multivariate datasets are analyzed, it is often desirable to reduce their dimensionality. Principal component analysis is one technique for doing this. It replaces the p original variables by a smaller number, q, of derived variables, the principal components, which are linear combinations of the original variables. Often, it is possible to retain most of the variability in the original variables with q very much smaller than p. Despite its apparent simplicity, principal component analysis has a number of subtleties, and it has many uses and extensions. A number of choices associated with the technique are briefly discussed, namely, covariance or correlation, how many components, and different normalization constraints, as well as confusion with factor analysis. Various uses and extensions are outlined. ::: ::: ::: Keywords: ::: ::: dimension reduction; ::: factor analysis; ::: multivariate analysis; ::: variance maximization <s> BIB007 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data. <s> BIB008 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> The technological advances in the instrumentation employed in life sciences have enabled the collection of a virtually unlimited quantity of data from multiple sources. By gathering data from several analytical platforms, with the aim of parallel monitoring of, e.g. transcriptomic, metabolomic or proteomic events, one hopes to answer and understand biological questions and observations. This 'systems biology' approach typically involves advanced statistics to facilitate the interpretation of the data. In the present study, we demonstrate that the O2PLS multivariate regression method can be used for combining 'omics' types of data. With this methodology, systematic variation that overlaps across analytical platforms can be separated from platform-specific systematic variation. A study of Populus tremula x Populus tremuloides, investigating short-day-induced effects at transcript and metabolite levels, is employed to demonstrate the benefits of the methodology. We show how the models can be validated and interpreted to identify biologically relevant events, and discuss the results in relation to a pairwise univariate correlation approach and principal component analysis. <s> BIB009 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> The orthogonal projections to latent structures (OPLS) method has been successfully applied in various chemical and biological systems for modeling and interpretation of linear relationships betwee ... <s> BIB010 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> We describe the use of a higher-order singular value decomposition (HOSVD) in transforming a data tensor of genes × “x-settings,” that is, different settings of the experimental variable x × “y-settings,” which tabulates DNA microarray data from different studies, to a “core tensor” of “eigenarrays” × “x-eigengenes” × “y-eigengenes.” Reformulating this multilinear HOSVD such that it decomposes the data tensor into a linear superposition of all outer products of an eigenarray, an x- and a y-eigengene, that is, rank-1 “subtensors,” we define the significance of each subtensor in terms of the fraction of the overall information in the data tensor that it captures. We illustrate this HOSVD with an integration of genome-scale mRNA expression data from three yeast cell cycle time courses, two of which are under exposure to either hydrogen peroxide or menadione. We find that significant subtensors represent independent biological programs or experimental phenomena. The picture that emerges suggests that the conserved genes YKU70, MRE11, AIF1, and ZWF1, and the processes of retrotransposition, apoptosis, and the oxidative pentose phosphate pathway that these genes are involved in, may play significant, yet previously unrecognized, roles in the differential effects of hydrogen peroxide and menadione on cell cycle progression. A genome-scale correlation between DNA replication initiation and RNA transcription, which is equivalent to a recently discovered correlation and might be due to a previously unknown mechanism of regulation, is independently uncovered. <s> BIB011 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Over the years, many tensor based algorithms, e.g. two dimensional principle component analysis (2DPCA), two dimensional singular value decomposition (2DSVD), high order SVD, have been proposed for the study of high dimensional data in a large variety of computer vision applications. An intrinsic limitation of previous tensor reduction methods is the sensitivity to the presence of outliers, because they minimize the sum of squares errors (L2 norm). In this paper, we propose a novel robust tensor factorization method using R1 norm for error accumulation function using robust covariance matrices, allowing the method to be efficiently implemented instead of resorting to quadratic programming software packages as in other L1 norm approaches. Experimental results on face representation and reconstruction show that our new robust tensor factorization method can effectively handle outliers compared to previous tensor based PCA methods. <s> BIB012 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors. <s> BIB013 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Summary: Coordinated Gene Activity in Pattern Sets (CoGAPS) provides an integrated package for isolating gene expression driven by a biological process, enhancing inference of biological processes from transcriptomic data. CoGAPS improves on other enrichment measurement methods by combining a Markov chain Monte Carlo (MCMC) matrix factorization algorithm (GAPS) with a threshold-independent statistic inferring activity on gene sets. The software is provided as open source C++ code built on top of JAGS software with an R interface. ::: ::: Availability: The R package CoGAPS and the C++ package GAPS-JAGS are provided open source under the GNU Lesser Public License (GLPL) with a users manual containing installation and operating instructions. CoGAPS is available through Bioconductor and depends on the rjags package available through CRAN to interface CoGAPS with GAPS-JAGS. ::: ::: URL: http://www.cancerbiostats.onc.jhmi.edu/cogaps.cfm ::: ::: Contact: ude.imhj@gitrefje; ude.uhj@ofm ::: ::: Supplementary Information: Supplementary data is available at Bioinformatics online. <s> BIB014 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Non-negative information can benefit the analysis of microarray data. This paper investigates the classification performance of non-negative matrix factorization (NMF) over gene-sample data. We also extends it to higher-order version for classification of clinical time-series data represented by tensor. Experiments show that NMF and the higher-order NMF can achieve at least comparable prediction performance. <s> BIB015 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Funding: Janssen Pharmaceutica N.V. and Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT project 80536). <s> BIB016 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Motivation: The molecular complexity of a tumor manifests itself at the genomic, epigenomic, transcriptomic and proteomic levels. Genomic profiling at these multiple levels should allow an integrated characterization of tumor etiology. However, there is a shortage of effective statistical and bioinformatic tools for truly integrative data analysis. The standard approach to integrative clustering is separate clustering followed by manual integration. A more statistically powerful approach would incorporate all data types simultaneously and generate a single integrated cluster assignment. ::: ::: Methods: We developed a joint latent variable model for integrative clustering. We call the resulting methodology iCluster. iCluster incorporates flexible modeling of the associations between different data types and the variance–covariance structure within data types in a single framework, while simultaneously reducing the dimensionality of the datasets. Likelihood-based inference is obtained through the Expectation–Maximization algorithm. ::: ::: Results: We demonstrate the iCluster algorithm using two examples of joint analysis of copy number and gene expression data, one from breast cancer and one from lung cancer. In both cases, we identified subtypes characterized by concordant DNA copy number changes and gene expression as well as unique profiles specific to one or the other in a completely automated fashion. In addition, the algorithm discovers potentially novel subtypes by combining weak yet consistent alteration patterns across data types. ::: ::: Availability: R code to implement iCluster can be downloaded at http://www.mskcc.org/mskcc/html/85130.cfm ::: ::: Contact: [email protected] ::: ::: Supplementary information:Supplementary data are available at Bioinformatics online. <s> BIB017 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> We propose simple transformation of the hidden states in variational Bayesian factor analysis models to speed up the learning procedure. The speed-up is achieved by using proper parameterization of the posterior approximation which allows joint optimization of its individual factors, thus the transformation is theoretically justified. We derive the transformation formulae for variational Bayesian factor analysis and show experimentally that it can significantly improve the rate of convergence. The proposed transformation basically performs centering and whitening of the hidden factors taking into account the posterior uncertainties. Similar transformations can be applied to other variational Bayesian factor analysis models as well. <s> BIB018 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> We introduce a factor analysis model that summarizes the dependencies between observed variable groups, instead of dependencies between individual variables as standard factor analysis does. A group may correspond to one view of the same set of objects, one of many data sets tied by co-occurrence, or a set of alternative variables collected from statistics tables to measure one property of interest. We show that by assuming groupwise sparse factors, active in a subset of the sets, the variation can be decomposed into factors explaining relationships between the sets and factors explaining away set-specific variation. We formulate the assumptions in a Bayesian model providing the factors, and apply the model to two data analysis tasks, in neuroimaging and chemical systems biology. <s> BIB019 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Research in several fields now requires the analysis of datasets in which multiple high-dimensional types of data are available for a common set of objects. In particular, The Cancer Genome Atlas (TCGA) includes data from several diverse genomic technologies on the same cancerous tumor samples. In this paper we introduce Joint and Individual Variation Explained (JIVE), a general decomposition of variation for the integrated analysis of such datasets. The decomposition consists of three terms: a low-rank approximation capturing joint variation across data types, low-rank approximations for structured variation individual to each data type, and residual noise. JIVE quantifies the amount of joint variation between data types, reduces the dimensionality of the data, and provides new directions for the visual exploration of joint and individual structure. The proposed method represents an extension of Principal Component Analysis and has clear advantages over popular two-block methods such as Canonical Correlation Analysis and Partial Least Squares. A JIVE analysis of gene expression and miRNA data on Glioblastoma Multiforme tumor samples reveals gene-miRNA associations and provides better characterization of tumor types. <s> BIB020 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> This paper presents a new multiblock analysis method called OnPLS, a general extension of O2PLS to the multiblock case. The proposed method is equivalent to O2PLS in cases involving only two matric ... <s> BIB021 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> BackgroundNon-negative matrix factorization (NMF) has been introduced as an important method for mining biological data. Though there currently exists packages implemented in R and other programming languages, they either provide only a few optimization algorithms or focus on a specific application field. There does not exist a complete NMF package for the bioinformatics community, and in order to perform various data mining tasks on biological data.ResultsWe provide a convenient MATLAB toolbox containing both the implementations of various NMF techniques and a variety of NMF-based data mining approaches for analyzing biological data. Data mining approaches implemented within the toolbox include data clustering and bi-clustering, feature extraction and selection, sample classification, missing values imputation, data visualization, and statistical comparison.ConclusionsA series of analysis such as molecular pattern discovery, biological process identification, dimension reduction, disease prediction, visualization, and statistical comparison can be performed using this toolbox. <s> BIB022 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Recent technology has made it possible to simultaneously perform multi-platform genomic profiling (e.g. DNA methylation (DM) and gene expression (GE)) of biological samples, resulting in so-called ‘multi-dimensional genomic data’. Such data provide unique opportunities to study the coordination between regulatory mechanisms on multiple levels. However, integrative analysis of multi-dimensional genomics data for the discovery of combinatorial patterns is currently lacking. Here, we adopt a joint matrix factorization technique to address this challenge. This method projects multiple types of genomic data onto a common coordinate system, in which heterogeneous variables weighted highly in the same projected direction form a multi-dimensional module (md-module). Genomic variables in such modules are characterized by significant correlations and likely functional associations. We applied this method to the DM, GE, and microRNA expression data of 385 ovarian cancer samples from the The Cancer Genome Atlas project. These md-modules revealed perturbed pathways that would have been overlooked with only a single type of data, uncovered associations between different layers of cellular activities and allowed the identification of clinically distinct patient subgroups. Our study provides an useful protocol for uncovering hidden patterns and their biological implications in multi-dimensional ‘omic’ data. <s> BIB023 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> OnPLS was recently presented as a general extension of O2PLS to the multiblock case. OnPLS is equivalent to O2PLS in the case of two matrices, but generalises symmetrically to cases with more than ... <s> BIB024 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> BackgroundHigh-throughput genomic and proteomic data have important applications in medicine including prevention, diagnosis, treatment, and prognosis of diseases, and molecular biology, for example pathway identification. Many of such applications can be formulated to classification and dimension reduction problems in machine learning. There are computationally challenging issues with regards to accurately classifying such data, and which due to dimensionality, noise and redundancy, to name a few. The principle of sparse representation has been applied to analyzing high-dimensional biological data within the frameworks of clustering, classification, and dimension reduction approaches. However, the existing sparse representation methods are inefficient. The kernel extensions are not well addressed either. Moreover, the sparse representation techniques have not been comprehensively studied yet in bioinformatics.ResultsIn this paper, a Bayesian treatment is presented on sparse representations. Various sparse coding and dictionary learning models are discussed. We propose fast parallel active-set optimization algorithm for each model. Kernel versions are devised based on their dimension-free property. These models are applied for classifying high-dimensional biological data.ConclusionsIn our experiment, we compared our models with other methods on both accuracy and computing time. It is shown that our models can achieve satisfactory accuracy, and their performance are very efficient. <s> BIB025 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Many real-world datasets are comprised of different representations or views which often provide information complementary to each other. To integrate information from multiple views in the unsupervised setting, multiview clustering algorithms have been developed to cluster multiple views simultaneously to derive a solution which uncovers the common latent structure shared by multiple views. In this paper, we propose a novel NMFbased multi-view clustering algorithm by searching for a factorization that gives compatible clustering solutions across multiple views. The key idea is to formulate a joint matrix factorization process with the constraint that pushes clustering solution of each view towards a common consensus instead of fixing it directly. The main challenge is how to keep clustering solutions across different views meaningful and comparable. To tackle this challenge, we design a novel and effective normalization strategy inspired by the connection between NMF and PLSA. Experimental results on synthetic and several real datasets demonstrate the effectiveness of our approach. <s> BIB026 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> BackgroundReactive oxygen species (ROS) are involved in the regulation of diverse physiological processes in plants, including various biotic and abiotic stress responses. Thus, oxidative stress tolerance mechanisms in plants are complex, and diverse responses at multiple levels need to be characterized in order to understand them. Here we present system responses to oxidative stress in Populus by integrating data from analyses of the cambial region of wild-type controls and plants expressing high-isoelectric-point superoxide dismutase (hipI-SOD) transcripts in antisense orientation showing a higher production of superoxide. The cambium, a thin cell layer, generates cells that differentiate to form either phloem or xylem and is hypothesized to be a major reason for phenotypic perturbations in the transgenic plants. Data from multiple platforms including transcriptomics (microarray analysis), proteomics (UPLC/QTOF-MS), and metabolomics (GC-TOF/MS, UPLC/MS, and UHPLC-LTQ/MS) were integrated using the most recent development of orthogonal projections to latent structures called OnPLS. OnPLS is a symmetrical multi-block method that does not depend on the order of analysis when more than two blocks are analysed. Significantly affected genes, proteins and metabolites were then visualized in painted pathway diagrams.ResultsThe main categories that appear to be significantly influenced in the transgenic plants were pathways related to redox regulation, carbon metabolism and protein degradation, e.g. the glycolysis and pentose phosphate pathways (PPP). The results provide system-level information on ROS metabolism and responses to oxidative stress, and indicate that some initial responses to oxidative stress may share common pathways.ConclusionThe proposed data evaluation strategy shows an efficient way of compiling complex, multi-platform datasets to obtain significant biological information. <s> BIB027 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Abstract In the recent years, non-negative matrix factorization and sparse representation models have been successfully applied in high-throughput biological data analysis due to its interpretability and robustness to noise. In this paper, we propose a unified matrix factorization model, coined versatile sparse matrix factorization (VSMF) model, for biological data analysis. We discuss the modelling, optimization, and applications of VSMF. We show that many well-known sparse matrix factorization models are specific cases of our VSMF. Through tuning parameters, sparsity, smoothness, and non-negativity can be easily controlled in VSMF. Our computational experiments for feature extraction, feature selection, and clustering corroborate the advantages of VSMF. <s> BIB028 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Nonnegative Matrix Factorization (NMF) has been one of the most widely used clustering techniques for exploratory data analysis. However, since each data point enters the objective function with squared residue error, a few outliers with large errors easily dominate the objective function. In this article, we propose a Robust Manifold Nonnegative Matrix Factorization (RMNMF) method using e2,1-norm and integrating NMF and spectral clustering under the same clustering framework. We also point out the solution uniqueness issue for the existing NMF methods and propose an additional orthonormal constraint to address this problem. With the new constraint, the conventional auxiliary function approach no longer works. We tackle this difficult optimization problem via a novel Augmented Lagrangian Method (ALM)--based algorithm and convert the original constrained optimization problem on one variable into a multivariate constrained problem. The new objective function then can be decomposed into several subproblems that each has a closed-form solution. More importantly, we reveal the connection of our method with robust K-means and spectral clustering, and we demonstrate its theoretical significance. Extensive experiments have been conducted on nine benchmark datasets, and all empirical results show the effectiveness of our method. <s> BIB029 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Factor analysis provides linear factors that describe relationships between individual variables of a data set. We extend this classical formulation into linear factors that describe relationships between groups of variables, where each group represents either a set of related variables or a data set. The model also naturally extends canonical correlation analysis to more than two sets, in a way that is more flexible than previous extensions. Our solution is formulated as variational inference of a latent variable model with structural sparsity, and it consists of two hierarchical levels: The higher level models the relationships between the groups, whereas the lower models the observed variables given the higher level. We show that the resulting solution solves the group factor analysis problem accurately, outperforming alternative factor analysis based solutions as well as more straightforward implementations of group factor analysis. The method is demonstrated on two life science data sets, one on brain activation and the other on systems biology, illustrating its applicability to the analysis of different types of high-dimensional data sources. <s> BIB030 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Motivation: Modelling methods that find structure in data are necessary with the current large volumes of genomic data, and there have been various efforts to find subsets of genes exhibiting consistent patterns over subsets of treatments. These biclustering techniques have focused on one data source, often gene expression data. We present a Bayesian approach for joint biclustering of multiple data sources, extending a recent method Group Factor Analysis to have a biclustering interpretation with additional sparsity assumptions. The resulting method enables data-driven detection of linear structure present in parts of the data sources. Results: Our simulation studies show that the proposed method reliably infers biclusters from heterogeneous data sources. We tested the method on data from the NCI-DREAM drug sensitivity prediction challenge, resulting in an excellent prediction accuracy. Moreover, the predictions are based on several biclusters which provide insight into the data sources, in this case on gene expression, DNA methylation, protein abundance, exome sequence, functional connectivity fingerprints and drug sensitivity. Availability and Implementation: http://research.cs.aalto.fi/pml/software/GFAsparse/ Contacts: kerstin.bunte@googlemail.com or samuel.kaski@aalto.fi <s> BIB031 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> MOTIVATION ::: Within any given tissue, gene expression levels can vary extensively among individuals. Such heterogeneity can be caused by genetic and epigenetic variability and may contribute to disease. The abundance of experimental data now enables the identification of features of gene expression profiles that are shared across tissues and those that are tissue-specific. While most current research is concerned with characterizing differential expression by comparing mean expression profiles across tissues, it is believed that a significant difference in a gene expression's variance across tissues may also be associated with molecular mechanisms that are important for tissue development and function. ::: ::: ::: RESULTS ::: We propose a sparse multi-view matrix factorization (sMVMF) algorithm to jointly analyse gene expression measurements in multiple tissues, where each tissue provides a different 'view' of the underlying organism. The proposed methodology can be interpreted as an extension of principal component analysis in that it provides the means to decompose the total sample variance in each tissue into the sum of two components: one capturing the variance that is shared across tissues and one isolating the tissue-specific variances. sMVMF has been used to jointly model mRNA expression profiles in three tissues obtained from a large and well-phenotyped twins cohort, TwinsUK. Using sMVMF, we are able to prioritize genes based on whether their variation patterns are specific to each tissue. Furthermore, using DNA methylation profiles available, we provide supporting evidence that adipose-specific gene expression patterns may be driven by epigenetic effects. ::: ::: ::: AVAILABILITY AND IMPLEMENTATION ::: Python code is available at http://wwwf.imperial.ac.uk/~gmontana/. ::: ::: ::: CONTACT ::: giovanni.montana@kcl.ac.uk ::: ::: ::: SUPPLEMENTARY INFORMATION ::: Supplementary data are available at Bioinformatics online. <s> BIB032 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Recent advances in high-throughput omics technologies have enabled biomedical researchers to collect large-scale genomic data. As a consequence, there has been growing interest in developing methods to integrate such data to obtain deeper insights regarding the underlying biological system. A key challenge for integrative studies is the heterogeneity present in the different omics data sources, which makes it difficult to discern the coordinated signal of interest from source-specific noise or extraneous effects.We introduce a novel method of multi-modal data analysis that is designed for heterogeneous data based on non-negative matrix factorization. We provide an algorithm for jointly decomposing the data matrices involved that also includes a sparsity option for high-dimensional settings. The performance of the proposed method is evaluated on synthetic data and on real DNA methylation, gene expression and miRNA expression data from ovarian cancer samples obtained from The Cancer Genome Atlas. The results show the presence of common modules across patient samples linked to cancer-related pathways, as well as previously established ovarian cancer subtypes.The source code repository is publicly available at https://github.com/yangzi4/iNMF.gmichail@umich.eduSupplementary data are available at Bioinformatics online. <s> BIB033 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> Abstract The focus of the present paper is to propose and discuss different procedures for performing variable selection in a multi-block regression context. In particular, the focus is on two multi-block regression methods: Multi-Block Partial Least Squares (MB-PLS) and Sequential and Orthogonalized Partial Least Squares (SO-PLS) regression. A small simulation study for regular PLS regression was conducted in order to select the most promising methods to investigate further in the multi-block context. The combinations of three variable selection methods with MB-PLS and SO-PLS are examined in detail. These methods are Variable Importance in Projection (VIP) Selectivity Ratio (SR) and forward selection. In this paper we focus on both prediction ability and interpretation. The different approaches are tested on three types of data: one sensory data set, one spectroscopic (Raman) data set and a number of simulated multi-block data sets. <s> BIB034 </s> A review on machine learning principles for multi-view biological data integration <s> Feature extractions and matrix factorizations for detecting shared and view-specific components <s> In many areas of science multiple sets of data are collected pertaining to the same system. Examples are food products which are characterized by different sets of variables, bio-processes which are on-line sampled with different instruments, or biological systems of which different genomics measurements are obtained. Data fusion is concerned with analyzing such sets of data simultaneously to arrive at a global view of the system under study. One of the upcoming areas of data fusion is exploring whether the data sets have something in common or not. This gives insight into common and distinct variation in each data set, thereby facilitating understanding the relationships between the data sets. Unfortunately, research on methods to distinguish common and distinct components is fragmented, both in terminology as well as in methods: there is no common ground which hampers comparing methods and understanding their relative merits. This paper provides a unifying framework for this subfield of data fusion by using rigorous arguments from linear algebra. The most frequently used methods for distinguishing common and distinct components are explained in this framework and some practical examples are given of these methods in the areas of (medical) biology and food science. <s> BIB035
While it is often challenging to combine features of multiple views in the original input spaces, new features generated by feature extraction methods can be easily combined. As illustrated in Figure 5 , an idea is to extract new features from each data view first, and then incorporate these new features together. Finally, a classification or clustering algorithm can be applied on the combined features. Depending on the nature of an individual data view, a feature extraction method learns the representations of samples in a new feature space. Matrix factorization methods, such as principal component analysis BIB007 BIB002 , factor analysis (FA)   , NMF BIB001 BIB014 BIB022  , SR BIB025 and tensor decomposition methods BIB013 BIB015 , are commonly used feature extraction models. Other dimensionality reduction methods than matrix decompositions, such as autoencoder BIB008 and restricted Boltzmann machine (RBM)  , can be applied as well. There are several benefits of using feature extraction in data integration. First, the natures of heterogeneous data from multiple omics data can be separately well counted. Despite the original data types, the new features in the ations are yet to be inferred (marked in dashed lines). There might exist multiple paths for indirect associations, for example in RR-disease associations, we can have RR-gene-pathway-drug-disease and RR-gene-drug-disease. corresponding feature spaces are usually numeric, implying an easy concatenation. Second, the high-dimensionality is dramatically reduced so that the downstream analysis will be more efficient. Third, extracting new features separately for each data view implements the principle of divide and conquer, thus computational complexity can be significantly reduced. Fourth, relational data can be well incorporated by kernel feature extraction methods BIB028 . However, one pitfall of the feature-extractionbased integrative principle is that the interactions (correlation, dependency or association) between input features from different views cannot be taken into account in the separated feature extraction procedures. To consider the interactions between features from different views, (Bayesian) multi-view matrix factorization methods can be applied to extract new features on the feature-wise concatenated matrix. By inducing group-wise (that is view-wise) sparsity on the basis matrix (that is factor loading matrix), Bayesian group factor analysis (GFA) BIB019 BIB030 detects ubiquitous and view-specific factors (see Figure 6 ), which is informative to discover features from multiple views involved in a potential pathway. As a special case of GFA, Bayesian canonical correlation analysis (CCA)  only detects correlated factors between two views. Sparse GFA has been developed for biclustering multi-view data with co-occurring samples BIB031 . Simulation study revealed that sparse GFA could more accurately recover predefined biclusters in terms of F 1 score compared with factor analysis for bicluster acquisition BIB016 . Sparse GFA was applied to predict drug response of cancer cell lines supported by multi-omics data, where the prediction was modeled as inferring missing values (drug response), crossvalidation performance was measured by correlation coefficients and real predictions were validated by enrichment analysis. NMF has been extended for multi-view data for clustering and latent FA. Similar to iCluster BIB017 , MultiNMF BIB026 and EquiNMF  allow all views to share a unique coefficient matrix for collective clustering, while Joint-NMF BIB023 restricts all views to share the same basis matrix for finding common factors among multi-omics data. Using ubiquitous basis matrix and view-specific basis matrices as well as view-specific coefficient matrices, integrative-NMF BIB033 is able to detect homogeneous and heterogeneous factors from multi-omics data. Comparative studies showed that integrative-NMF significantly outperforms joint-NMF on both a simulated data set with heterogeneous noise in terms of a module-detection score and a real ovarian cancer multi-omics data set in terms of purity indices. The discovered modules were validated using pathway enrichment analysis. Although not in a non-negative setting but closely relevant, joint and individual clustering (JIC)  and joint and individual variation explained (JIVE) BIB020 use both common and view-specific coefficient matrices. Comparison showed that JIC is advantageous over iCluster on both simulated data in terms of precision and multi-omics (RNA-seq and miRNA-seq) breast cancer data in terms of validation using clinical information. This idea can be easily generalized to any matrix factorizations, including NMF. Because high-throughput sequencing platforms generate read-count data, which are naturally non-negative, multi-view NMF models have a great potential for various analyses such as tumor subtyping, pathway analysis and biomarker selection. Partial least squares (PLS, more precisely it should stand for projection to latent structures) is a long-standing model that integrates two-view data together  . Given centered predictor data X 2 R M ðXÞ ÂN and the corresponding response data Y 2 R M ðYÞ ÂN where M ðXÞ is the number of predictor variables, M ðYÞ the number of response variables and N the number of samples, it projects X and Y into two new K-dimensional spaces, spanned by columns of W ðXÞ 2 R M ðXÞ ÂK and W ðYÞ 2 R M ðYÞ ÂK , respectively, where the covariance between their representations H ðXÞ and H ðYÞ are maximized. A typical PLS can be formulated as where W ðXÞ and W ðYÞ denote orthonormal basis matrices with K columns, H ðXÞ and H ðYÞ symbolize coefficient matrices, as well as E ðXÞ and E ðYÞ are error terms. By making W ðXÞ and W ðYÞ sparse, PLS can be applied to select key predictor variables relevant to the response variables. For example, the predictor variables corresponding to non-zero elements in the k-th column of W ðXÞ can be treated as important features to explain the response variables corresponding to non-zero elements in the k-th column of W ðYÞ . The attraction of PLS-based integrative models is their capability of selecting co-varying features among multiple feature sets BIB034 . Orthogonal projections to latent structures is an extension of PLS that improves interpretability by removing from X the systematic variation that is not correlated to Y BIB003 . O2-PLS further generalizes this idea in a predictive model in both ways, namely, X $ Y, by modeling the Y-orthogonal variation in X, the X-orthogonal variation in Y and the X-Y joint covariation BIB004 BIB005 . O2-PLS has been applied to integrate transcript and metabolite data in plant BIB009 . OnPLS extends O2-PLS for more than two views by using a global one-versus-one strategy BIB021 BIB024 (other PLS-related models are also discussed in BIB021 ). It extracts shared components maximally co-varying across all views and local components specific to combinations of views or individual views. It has been applied to integrate transcriptomic, proteomic and metabolomic data for discovering shared pathways BIB027 . Indeed, pathway analysis corroborated that the computationally identified pathways are biologically meaningful. In case of strongly nonlinear associations between X and Y, kernel-based orthogonal projections to latent structures (K-OPLS) was proposed to carry out PLS on a high-dimensional feature space BIB010 . In addition to matrix factorization models, their extensions-tensor decompositions BIB011 BIB013  should be also considered for dimensionality reduction (in classification or clustering) and FA on multi-view data naturally represented by a tensor BIB035 . Many matrix decomposition models are non-convex from the optimization perspective. Thus, their performances are affected by initial values. Moreover, feature selection based on l 1norm regularized sparse matrix factorizations may suffer from inconsistency. Applying the strategy of stability selection on matrix factorizations can make the procedure consistent BIB032 . Matrix or tensor factorization models assuming Gaussian or Poisson distributions to the data are not robust to outlying data points. Robust factorizations, such as robust PLS BIB006 , robust NMF BIB029 and tensor factorization using l 1 -norm BIB012 , are developed to address this issue. In the near future, more work needs to be done on robust multi-view matrix factorizations. Missing values can be nicely handled by weighted matrix factorizations BIB022 and Bayesian matrix factorizations BIB018  , which can ignore missing entries in their learning. Future developments of new multi-view matrix factorization tools should consider the functionality of dealing with missing values. Overfitting is not an issue for penalized and Bayesian multiview matrix factorizations.