A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described. <s> BIB001 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition. <s> BIB002 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> Summary. We consider the problem of selecting grouped variables (factors) for accurate prediction in regression. Such a problem arises naturally in many practical situations with the multifactor analysis-of-variance problem as the most important and well-known example. Instead of selecting factors by stepwise backward elimination, we focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection. The lasso, the LARS algorithm and the non-negative garrotte are recently proposed regression methods that can be used to select individual variables. We study and propose efficient algorithms for the extensions of these methods for factor selection and show that these extensions give superior performance to the traditional stepwise backward elimination method in factor selection problems. We study the similarities and the differences between these methods. Simulations and real examples are used to illustrate the methods. <s> BIB003 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> We missed an important reference in Section 3.4. In page 309 we stated that ‘. . . which is based on the recently proposed algorithm LARS of Efron et al. (2004). They proved that, starting from zero, the lasso solution paths grow piecewise linearly in a predictable way. They proposed a new algorithm called LARS to solve the entire lasso solution path efficiently by using the same order of computations as a single OLS fit. . . .’ The following sentence should have been included. The piecewise linearity of the lasso solution path was first proved by Osborne et al. (2000), who also described an efficient algorithm for calculating the complete lasso solution path. Reference Osborne, M. R., Presnell, B. and Turlach, B. A. (2000) A new approach to variable selection in least squares problems. IMA J. Numer. Anal., 20, 389–403. <s> BIB004 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> SUMMARY We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm—the graphical lasso—that is remarkably fast: It solves a 1000-node problem (∼500 000 parameters) in at most a minute and is 30–4000 times faster than competing methods. It also provides a conceptual link between the exact problem and the approximation suggested by Meinshausen and B¨ uhlmann (2006). We illustrate the method on some cell-signaling data from proteomics. <s> BIB005 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> We propose a new penalty function which, when used as regularization for empirical risk minimization procedures, leads to sparse estimators. The support of the sparse vector is typically a union of potentially overlapping groups of co-variates defined a priori, or a set of covariates which tend to be connected to each other when a graph of covariates is given. We study theoretical properties of the estimator, and illustrate its behavior on simulated and breast cancer gene expression data. <s> BIB006 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> We consider the group lasso penalty for the linear model. We note that the standard algorithm for solving the problem assumes that the model matrices in each group are orthonormal. Here we consider a more general penalty that blends the lasso (L1) with the group lasso ("two-norm"). This penalty yields solutions that are sparse at both the group and individual feature levels. We derive an efficient algorithm for the resulting convex problem based on coordinate descent. This algorithm can also be used to solve the general form of the group lasso, with non-orthonormal model matrices. <s> BIB007 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> In literature, hundreds of features have been proposed to assess the quality of tandem mass spectra. However, many of these features are irrelevant in describing the spectrum quality and they can degenerate the spectrum quality assessment performance. We propose a two-stage Recursive Feature Elimination based on Support Vector Machine (SVM-RFE) method to select the highly relevant features from those collected in literature. Classifiers are trained to verify the relevance of selected features. The results demonstrate that these selected features can better describe the quality of tandem mass spectra and hence improve the performance of tandem mass spectrum quality assessment. <s> BIB008 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> As an increasing number of genome-wide association studies reveal the limitations of attempting to explain phenotypic heritability by single genetic loci, there is growing interest for associating complex phenotypes with sets of genetic loci. While several methods for multi-locus mapping have been proposed, it is often unclear how to relate the detected loci to the growing knowledge about gene pathways and networks. The few methods that take biological pathways or networks into account are either restricted to investigating a limited number of predetermined sets of loci, or do not scale to genome-wide settings. ::: We present SConES, a new efficient method to discover sets of genetic loci that are maximally associated with a phenotype, while being connected in an underlying network. Our approach is based on a minimum cut reformulation of the problem of selecting features under sparsity and connectivity constraints that can be solved exactly and rapidly. ::: SConES outperforms state-of-the-art competitors in terms of runtime, scales to hundreds of thousands of genetic loci, and exhibits higher power in detecting causal SNPs in simulation studies than existing methods. On flowering time phenotypes and genotypes from Arabidopsis thaliana, SConES detects loci that enable accurate phenotype prediction and that are supported by the literature. ::: Matlab code for SConES is available at this http URL <s> BIB009 </s> A review on machine learning principles for multi-view biological data integration <s> Feature concatenation <s> A drastic amount of data have been and are being generated in bioinformatics studies. In the analysis of such data, the standard modeling approaches can be challenged by the heavy-tailed errors and outliers in response variables, the contamination in predictors (which may be caused by, for instance, technical problems in microarray gene expression studies), model mis-specification and others. Robust methods are needed to tackle these challenges. When there are a large number of predictors, variable selection can be as important as estimation. As a generic variable selection and regularization tool, penalization has been extensively adopted. In this article, we provide a selective review of robust penalized variable selection approaches especially designed for high-dimensional data from bioinformatics and biomedical studies. We discuss the robust loss functions, penalty functions and computational algorithms. The theoretical properties and implementation are also briefly examined. Application examples of the robust penalization approaches in representative bioinformatics and biomedical studies are also illustrated. <s> BIB010
Concatenating all features into a single vector seems to be the 'simplest' principle that brings all features together. Heterogeneous data contain much richer information than homogeneous data. But it is challenging to combine heterogeneous features. Applying feature selection techniques to the concatenated data may improve performance BIB008 . Sparse linear discriminative models, such as support vector machines (SVMs) BIB002 and the LASSO family BIB001 , play an important role in the classification, regression and feature selection of multi-class data with combined homogeneous or heterogeneous features. A general linear model can be formulated as y ¼ gðw T x þ bÞ where x is a vector of predictor variables; y is a binary discrete response variable (classification problem) or real-valued variable (regression problem); w is the weight vector; b is a scalar bias term; and gðÁÞ is an inverse link function. From an optimization perspective, linear models can be generally written as min w;b X N n¼1 lðy n ; w T x n þ bÞ þ krðwÞ; where lðÁÞ is a loss function, rðÁÞ is a regularization term and k balances the trade-off between fitting error and model complexity. By making w sparse, using, for example, l 1 -regularization, predictor variables corresponding to non-zero weights are taken as important features for the response variable. Thus, sparse linear models can be used for simultaneous prediction and variable selection. The l 1 -norm-based shrinkage is known as LASSO. Important generalizations of LASSO include (1) elastic net BIB004 that uses a weighted combination of l 1 -and l 2 -norms to select highly correlated predictor variables together, (2) group LASSO BIB003 that selects among predefined groups of predictor variables and (3) graphical LASSO BIB005 that applies l 1 -regularization to produce sparse precision matrix for estimating sparse graphs. Among them, (sparse) group LASSO BIB007 is useful to incorporate prior knowledge in the grouping of features BIB006 . It can also be applied to concatenated multi-omics data with features from the same view hierarchically grouped. Graphical LASSO is useful to incorporate structured information. For example, a graph regularization is used in BIB009 to select a connected component from SNP network associated to a phenotype. LASSO is variable selection consistent if and only if the training data X satisfies the irrepresentable condition  . But its generalizations-adaptive LASSO  and randomized LASSOusing the stability selection  are consistent variable selection procedures. Sparse linear models using a robust loss function, such as hinge loss, is robust to outlier samples BIB010 . The sparse regularization techniques reduce model complexity, thus prevent model selection from overfitting. The concatenated features require additional downstream processing that may lead to loss of key information. First, because multi-view data are observed in different forms that may include continuous features, discrete features, characters and even graphic data types, converting them into acceptable types (e.g. continuous to discrete, categorical to binary coding) is necessary for certain models such as hidden Markov models. Second, features from multiple views usually have different scales. Particularly for discriminative models, it is sometimes Table 1 . An overview of integrative analyses that can be conducted by machine learning methods on four types of multi-view data. Details regarding these methods and applications are described in separate sections