The inference of gene regulatory networks gained within recent years a considerable interest in the biology and biomedical community. The purpose of this paper is to investigate the influence that environmental conditions can exhibit on the inference performance of network inference algorithms. Specifically, we study five network inference methods, Aracne, BC3NET, CLR, C3NET and MRNET, and compare the results for three different conditions: (I) observational gene expression data: normal environmental condition, (II) interventional gene expression data: growth in rich media, (III) interventional gene expression data: normal environmental condition interrupted by a positive spike-in stimulation. Overall, we find that different statistical inference methods lead to comparable, but condition-specific results. Further, our results suggest that non-steady-state data enhance the inferability of regulatory networks. Human Genome Project  observational gene expression data: normal environmental condition (II) interventional gene expression data: growth in rich media (III) interventional gene expression data: normal environmental condition interrupted by a brief, positive stimulation (spike-in stimulation). We study these data for five network inference methods observational gene expression data: normal environmental condition(II)interventional gene expression data: growth in rich media(III)interventional gene expression data: normal environmental condition interrupted by a positive spike-in stimulation That means, we are generating gene expression data that correspond to observational (I) and interventional data (II and III). However, we are not generating data by gene knockout or silencing  but add at a certain time point, ts, a constant but random stimulation Es×r for each gene. Here Es is a constant factor and r is a random variable uniformly sampled from . This stimulation lasts a short period of time Δt = 0.2. After this period, the gene expression is again governed by the same conditions as in (I). Biologically, this corresponds to a normal condition that is interrupted by a short positive stimulation, e.g., the administration of a drug.The first type of data we are generating corresponds to cells in normal environmental conditions meaning that for these simulations we do not use an external stimulation of the gene expression. The second type of data can be seen as a media rich environment which has a favorable effect on the proliferation of cells. For this condition each gene receives an external positive stimulus facilitating its expression. For your simulations this is accomplished by using a constant stimulation of a fixed positive constant S. cerevisiae  or negative (repressor) and, hence, lead to the enhancement or repression of a transcription regulation.In netsim to generate time series data that are measured at T different time points, i.e, {t1,…,tT}. We are not using the time series data themselves to estimate the underlying network, given by Gtrue, but, instead, we generate an ensemble of T×E×S different data sets. We organize these data sets according to the observation time points, i.e., 𝒟i = {D1(ti),D2(ti),…,DE(ti)} with i∈{1,…,T}. This gives us T different sets of data sets, 𝒟i, each consisting of E different data sets De(ti) with e∈{1,…,E} and i∈{1,…,T} with S samples. That means, each data set De(ti) contains measurements that correspond to one particular time point ti only. See We use i, allow us to assess the inference characteristics of statistical network inference methods on the population level, because when the value of E is large enough chosen it allow us to draw conclusions with respect to the behavior of the population. Specifically, we use each of the E data sets De(ti) in 𝒟i to infer E networks, {G1(ti),…,GE(ti)}. By using knowledge about the true underlying network structure among the genes, given by Gtrue, we obtain E different F-scores that quantify the inference performance of the used network estimation algorithm, i.e., {F1(ti),…,FE(ti)}. Now the ensemble of F-scores allows us to estimate the mean inference performance and its variability. It is important to emphasize that information about the variability of the inference performance is necessary in order to obtain a robust evaluation. If only one or a few data sets would be used, the obtained results could be spurious. To avoid this, we use for our following numerical analysis E = 100, T = 11 and a sample size of S = 300. This results in a total of T×E = 1100 different data sets for each network Gtrue and each condition. Application of 5 different inference methods results in the inference of 5500 networks for each network Gtrue and each condition. In total, we infer for the two different networks we are studying  and the five different inference methods  33,000 different networks.These sets of data sets, 𝒟P corresponds to the precision and R to the recall, i.e.,  Gest, with the true network, Gtrue. More precisely, for an estimated network, Gest, the true network, Gtrue, and their corresponding adjacency matrices, Aest, and, Atrue, we obtain In order to evaluate the performance of a network inference algorithm we are using the F-score. The F-score is defined by I corresponds to the indicator function that is 1 if its argument is true and 0 otherwise.Here For our numerical analysis to infer gene regulatory networks, we use 5 different network inference methods, BC3NET, C3NET, CLR, MRNET and Aracne. In P(X) and P(Y) and the joint probability P of two random variables X and Y , between two random variables has the property to be always ≥ 0. I is equal to zero if the two random variables are  independent from each other, because in this case P = P(y)P(x).All 5 methods are information theory based utilizing estimates of mutual information coefficients . Mutual  X and Y : (7)\docPractically, the marginal and joint probability distributions are not available and, hence, mutual information values need to be estimated by means of statistical methods from the data. In X is defined by: H is given by Hemp can be estimated from the observed frequency distribution with nk number of samples in bin k, the total number of samples N and the total number of bins b. For example, for the entropy in H, due to an undersampling of the cell frequencies when the number of bins increases. A variety of approaches have been developed to account for this bias that range from correcting the estimate by a constant factor or using a multinomial distribution to model the extend of missing information.The Miller–Madow estimator utilizes the fact that the mutual information can also be written in terms of entropies , (8)\doceach bin . Specifib is the number of bins and N is the number of samples.The Miller–Madow estimator  accountsX,Y) is the Pearson correlation coefficient. For normal distributed random variables X and Y this expression is exact.A practical problem when applying the Miller–Madow estimator is that it is computationally demanding, .e.g., compared to the Pearson estimator for mutual information . The PeaFrom a numerical comparison of both estimators we find that the application of the Miller–Madow estimator takes about two orders of magnitude longer than the application of the Pearson estimator for mutual information. Further, from comparing different network inference methods we find that the performance for all methods is similarly effected by the estimators. For reasons of computational ease, we use for our following simulations the Pearson estimator, because our principle results are independent of the selected estimator and do not depend on the selection of the best estimator leading to the highest F-scores.t), E = 100 different data sets for an Erdös–Réyni network ,…,DE(tj)} different data sets and each of these data sets contains S = 300 samples (as described in section ‘Simulation design of our study’). The inference performance of each algorithm is estimated by F-scores that are presented in dependence on t.We begin our analysis by studying data that correspond to normal environmental conditions (I).  network  and a surevisiae . Each oft1 = 0.0 the shown F-scores correspond to F-scores assumed by chance, because the data for t1 = 0.0 correspond to the random initial values of the underlying dynamical system used to simulate the gene expression data. As one can see, for all methods these F-scores are close to zero without being identically zero.From our results in t larger than t9 = 10.0. This behavior indicates that the dynamical systems reach steady-state values and simulating for longer times does not lead to further changes. From our results we see that t = t11 = 50.0 can be safely assumed to lead to steady-state values for all five method.The long term behavior of the F-scores for all five methods, for both regulatory networks, converge to nearly constant F-scores for values of t2 = 0.5 and t3 = 1.0, depending on the method and the underlying regulatory network. However, in either case t2,t3≪t11, which means that the most informative time step is far from the steady-state of the dynamical system. In order to quantify the gain in the inference performance for each observational time step, we relate all median F-scores to the steady-state values, i.e., F(tj)/F(t11). Due to the fact that the F-scores do no longer change beyond t11, the value of F(t11) is equivalent to the asymptotic value of the dynamical system, i.e., Interestingly, the highest F-scores are observed for Ec, for each gene. The results from this analysis are shown in t≈2.5). We repeated this analysis for different values of Ec and found that the larger this constant stimulation is the further one can delay the time to reach optimal F-scores. However, for too large values of Ec the transcription regulation is essentially driven by the external stimulation which does not lead to meaningful results.The next experimental condition we are investigating corresponds to the growth of cells in a rich media (II), as modeled by a constant and positive external stimulation, S. cerevisiae  (III). These results are shown in S. cerevisiae  observational gene expression data: normal environmental condition, (II) interventional gene expression data: growth in rich media, (III) interventional gene expression data: normal environmental condition interrupted by a positive spike-in stimulation. We found that different statistical inference methods lead to comparable but condition-specific results. That means, qualitatively, the five network inference methods  The problem to infer gene regulatory networks from expression data is very challenging and depends on (A) the time point when data are measured, (B) the kind of the external stimulation and (C) the interconnectedness of the genes respectively their molecular interactions. Regarding the experimental design of future experiments our results suggest that it is not necessary to ensure that the gene expression data reached a stead-state value, and it could actually be detrimental for the inference of networks. Instead, usually, expression data far from the steady-state of the dynamical system contain more exploitable information that translates into increased F-scores. This finding is consistent among all five network inference methods. This makes actually the design of an experiment easier because it is practically not straight forward to control if the expression of genes reached their steady-state values. Further, for samples from human patients such a control is usually not possible for medical and ethical reasons. Hence, our findings relieve the experimenter from the need to ensure steady-state conditions in microarray experiments.A potential explanation for this effect could be that the noise-level in the system is for the optimal time points large enough to change occasionally the expression of a gene but not too strong to shatter the concerted interaction among groups of genes. This may be comparable to the functioning of the optimization method simulated annealing . For thiglobal perturbation on the expression of the genes this effect did not translate beneficially into an increase in the observed F-scores. This suggests that local perturbations or interventions need to be applied to a cellular systems in order to obtain data containing more information. For example, the knockout of genes or silencing techniques may be beneficial in this respect ((2) Another important finding is that the presence of an external stimulation (as studied in this paper) did not lead to an increase in the observed F-scores. Also this finding is consistent among all five network inference methods. That means that despite the presence of a  respect . HoweverS. cerevisiae (see (3) Our results for (III) interventional gene expression data  and the subnetwork of the transcriptional regulatory network of siae see  hint to siae see  with 4B siae see , and not


Large-scale molecular profiling technologies have assisted the identification of disease biomarkers and facilitated the basic understanding of cellular processes. However, samples collected from human subjects in clinical trials possess a level of complexity, arising from multiple cell types, that can obfuscate the analysis of data derived from them. Failure to identify, quantify, and incorporate sources of heterogeneity into an analysis can have widespread and detrimental effects on subsequent statistical studies.We describe an approach that builds upon a linear latent variable model, in which expression levels from mixed cell populations are modeled as the weighted average of expression from different cell types. We solve these equations using quadratic programming, which efficiently identifies the globally optimal solution while preserving non-negativity of the fraction of the cells. We applied our method to various existing platforms to estimate proportions of different pure cell or tissue types and gene expression profilings of distinct phenotypes, with a focus on complex samples collected in clinical trials.We tested our methods on several well controlled benchmark data sets with known mixing fractions of pure cell or tissue types and mRNA expression profiling data from samples collected in a clinical trial. Accurate agreement between predicted and actual mixing fractions was observed. In addition, our method was able to predict mixing fractions for more than ten species of circulating cells and to provide accurate estimates for relatively rare cell types . Furthermore, accurate changes in leukocyte trafficking associated with Fingolomid (FTY720) treatment were identified that were consistent with previous results generated by both cell counts and flow cytometry. These data suggest that our method can solve one of the open questions regarding the analysis of complex transcriptional data: namely, how to identify the optimal mixing fractions in a given experiment. Molecular analysis of cells in their native tissue environment provides the most accurate picture of the There are several approaches used to identify changes in gene expression that occur in different cellular compartments within tissues or tumors comprised of multiple cell types. Microdissection techniques that might allow a purer sampling of cells from fresh tumor specimens is time-consuming and requires an amplification of the sample that could distort transcriptional profiles et al. pioneered the application of a simulated annealing-based algorithm to identify the proportions of cells et al. et al. described cell-type-specific significance analysis of microarrays (csSAM) for analyzing differential gene expression for each cell type in a biological sample by incorporating heterogeneity in gene expression ad hoc post-processing to eliminate non-physical results such as negative mixing fractions The inversion of sample heterogeneity can be facilitated by providing accurate estimates of the mixing percentages of different cell types through computational deconvolution. Since computational dissection does not require microdissection of all samples or change of routine biological protocols, several authors have tried to answer whether it is possible to decompose the DNA microarray data from a cell population to survey the proportions of different cell types, by treating specific transcriptional patterns in DNA microarray data as cell-type-specific markers through computational methods in silico approach to deconvolute gene expression profiles obtained from heterogeneous clnicial samples into cell-type-specific patterns when the mixing matrix is unknown. We developed an approach built upon linear latent variable models that efficiently identifies the globally optimal solution in the least squares sense. Moreover, our approach explicitly incorporated physical constraints, specifically the mixing weights were required to be non-negative and sum to one, and therefore generated results that can be directly interpreted as mRNA mixing fractions. Technically, we employed a supervised selection of cell-type-specific genes to provide a basis that described the transcriptional state of “pure” cell populations. These cell-type-specific transcripts were then used to deconvolute the samples of interest using a quadratic programming technique that was highly efficient, providing directly interpretable results , and guaranteed to find the globally optimal solution. The results demonstrated that our method was able to accurately predict mixing fractions for more than ten species of circulating cells, and was even able to provide accurate estimates for relatively rare cell types.What we sought to demonstrate here was an We implemented our procedure for estimating fractions of different cell types in multiple gene expression data sets. First we assessed the utility of our method by applying it to three well controlled benchmark data sets with known mixing fractions. Satisfied that our approach worked, we then applied it to more challenging mRNA expression profiling data from human blood samples collected as part of a clinical trial.We used three benchmark datasets as proof of concept experiments. In the first experiment, tissues used for microarray analyses included independent, triplicate pools of blood and breast tissue samples from female adults. Double standed cDNA synthesis and labeling was carried out with 5 µg of total RNA, each sample was hybridized to Human Genome 133 Plus 2.0 GeneChips as specified by the manufacturer and the resulting CEL files were processed by Robust Multiarray Average (RMA) normalization In the second experiment, we employed the MAQC Rat Toxicogenomics Dataset For the last benchmark dataset, we used rat liver and brain as described in p<0.05 to identify statistically significant associations.Microarray expression data were used to generate cell-type-specific gene lists through pairwise comparisons of expression between all pure samples as described in vs. breast cell line data confirmed this approach returned known cell specific transcripts, pathways, and biological processes. The gene signature included genes whose expression is specific for blood specific genes , breast specific genes   . And allFirst, we measured the accuracy of our method with three benchmark experiments where known proportions of different tissues or cells are mixed, assayed on expression microarrays, and computationally separated.vs. breast are as depicted to the In each case, we generated gene signatures by analyzing the data from the “pure” samples (Training Data) and then applied these signatures into our approach to estimate the mixing fractions for the complex samples (Test Data). The results of the first mixtures - blood To test the utility of our algorithm to track clinically relevant changes in blood populations, we applied our method to expression profiling data generated for whole-blood samples collected from Multiple Sclerosis patients (MS) treated with Fingolimod (FTY720), a novel immunomodulator. Fingolomid is a structural analog of sphingosine that, in its phosphorylated form (FTY720-P), antagonizes S1P1 receptors expressed on the surface of lymphocytes. This in turn prevents the egress of lymphocytes from the lymph nodes, thereby impacting the trafficking of lymphocytes in the circulation Blood is a particularly complex tissue type, with over a dozen distinct cell types that can vary in frequency up to 10∼20-fold between healthy individuals p<0.01, Wilcoxon ranked sum test). Several other populations had increases in their relative predicted proportions in the treated subjects. Specifically, the predicted relative abundance of monocytes, NK cells and dendritic cells increased following Fingolomid treatment. These populations are not sequestered in the lymph nodes following treatment by Fingolomid, so their absolute numbers in circulation remain unchanged. Because microarray data is inherently semi-quantitative, we are only able to determine the relative abundance of each cell type in a sample. Consequently, the relative abundance of the cells appears to increase concomitant with the Fingolomid-induced reduction in other lymphocytes. Notably, none of the populations demonstrated changes in the placebo-treated subjects (p>0.4), which suggested that our approach was capable of a high degree of specificity even in complex, highly variable data sets.Inspection of the predicted mRNA fractions revealed that our method was able to dissect the lymphocyte population and track changes in specific populations induced by Fingolomid. X = AS. They differ substantially, however in how they dissect this system of equations, their optimization methods, and other important details. A summary of the methods surveyed here, and their main characteristics can be found in All surveyed microarray deconvolution methods make use of a system similar to that described in equation (1), X and mixing fractions A, estimate the basis matrix S. Shen-Orr et al.X and basis matrix S, estimate the mixing fractions A. Our approach falls into this category. (C) Given microarray data X, simultaneously estimate the mixing fractions A and basis matrix S. This approach is unsupervised. Consequently, such methods require prior information to initialize the optimization These methods approach equation (1) in one of three ways: (A) Given microarray data et al.et al. 's et al. 's approach et al. 's performed similarly for neutrophils, whereas ours performed substantially better for lymphocytes and monocyte. Erkkila et al. 's approach performed better than the other methods when seeded with the actual CBC values with mild noise (20 dB); however its performance degraded for more realistic tests with added noise. At 10 dB noise its performance drops but is similar to the results of our method; in the absence of prior information (i.e. seeded with random estimates) it could not find any solutions. These benchmarks indicate that deconvolution performance varies by the complexity of the experimental system, and the availability of prior biological knowledge. Our method, when fed with cell-type-specific transcriptional signatures, appears to perform well across a number of different biological systems of varying complexity. For the more general case in which both cell-type-specific signatures and cell fractions are not known, the performance of available methods varies substantially. For simple systems with few cell types, all tested methods perform well. For more complex systems, the use of accurate prior knowledge in the form of signatures (our method) or accurate cell fraction estimates  to estimate a small number of parameters in the linear system (n<20), so errors in any one measurement should have only a minimal effect on the final estimations. We performed several simulations using the blood/breast data to verify the robustness of our approach to fluctuations in the construction of the basis matrix.The foundation for this approach is the identification of a set of signatures that are generally representative of the cell types of interest. Any errors or uncertainities introduced in the design of this basis matrix could propogate through the analysis and impact the final results. Our approach leverages the thousands of expression level measurements made on each microarray to define a system of linear equations that are overdetermined, and can be optimally solved globally via quadratic programming. Taken together, this strategy should be robust to small deviations in the basis matrix; it uses many measurements of probes on the chip  promises the delivery of increasingly higher resolution views of the transcriptome. Data from such RNA-Seq experiments is already providing more exquisite views of low-abundance transcripts and alternative splicing In general, the application of highly sensitive, high throughput experimental technologies to complex biological samples will require increasing sophistication in the way that we think about and analyze our data. In some cases, this complexity has the potential to obfuscate relevant phenomenon if not addressed. In others, accurately estimating the complexity itself can be a useful endpoint. The approach we introduced here represents one specific application of a general framework for explicitly handling such complexities. The mathematical underpinnings and optimization algorithm are agnostic to the details of the biological system, and are generalizable to other data types that can be described via a system of overdetermined linear equations.Whole blood transcriptional analysis was performed as part of a clinical trial X is the microarray data from complex biological samples, A is the set of unknown proportions of the cellular constituents of X, and S is the known matrix of expression levels of the genes in all the cellular constituents of X.Estimating the proportions of different cell types is based upon a latent variable model framework Based on this model, we will first describe how we modeled the total expression signal of each microarray probe as the sum of the expression signals of its constituent parts in each mixture sample and solve it in constrained linear least-squares problems. We will then describe the identification of expression signatures using Limma (Linear Models for Microarray Data) jkx of gene j in a sample k is the average of cell type expectations, ijs, weighted by cell type fractions kia:kia) in the sample than known expression level measured on the chip (jkx), so the system was underdetermined.Expression deconvolution, which takes advantage of the linear latent model to represent the original expression signals as a mixture of each compartment signal, was performed on linear, untransformed data as follows. Starting from Eq. (1), the expression level j>n (more probesets than cell types), this system of equations is over determined. Physical constraints could be explicitly added to this system. Microarray data sets are inherently closed. Consequently, we were limited to estimating the proportions of mRNA present from each cell type, and these proportions must sum to one: For multiple probesets, we could extend this to a system of linear equations:kia) that satisfy: kia) that minimizes the residuals for kia is a scalar parameter between 0 and 1 to represent the fraction of cell subtype. When linear non-negative inequalities and equalities were given, Eq. (4) could be solved with quadratic programming Ideally, we would like to find mRNA fractions <1e-5 were retained for further evaluation. In the next step, we wanted to adjust the number of genes included in the signature to derive a high performance basis matrix that would be attributable to the estimated proportions. Following The probesets comprising the basis for deconvolution were determined as follows. First, the differential expression of each gene for different tissues or phenotypes was assessed by linear modeling and empirical Bayes methods using Limma  Function of the condition number vs. the number of probesets from the gene signature was characterized in blood and breast mixture cell lines. The local minima of condition number is shown in green line and the corresponding number of genes was selected as the ‘optimal’ number of expression signature; (b) Root mean square deviation (RMSD) between the estimated fractions and the actual fractions showed clear patterns to support the ‘optimal’ number of expression signature selected in (a). To the right of the green line, the RMSD almost formed a horizontal line with minor oscillations, suggesting that increasing the number of genes would not increase the accuracy of the deconvolution estimates. (c) The RMSD of the fitting residual also had high correlation with the condition number. This correlation is weaker when selecting more than the ‘optimal’ number of genes (shown in green line here).(TIF)Click here for additional data file.Figure S2Condition number varies with the number of probesets included in liver/kidney signatures. (a) Function of the condition number vs. the number of probesets from the gene signature was characterized in rat liver and kidney mixture cell lines. The local minima of condition number is shown in green line and the corresponding number of genes was selected as the ‘optimal’ number of expression signature; (b) shows the relationship between the RMSD of the estimated fractions and the number of genes in basis matrix; (c) is the plot of the RMSD of the fitting residual vs. the number of genes in basis matrix.(TIF)Click here for additional data file.Figure S3Condition number varies with the number of probesets included in liver/brain signatures. (a) Function of the condition number vs. the number of probesets from the gene signature was characterized in rat liver and brain mixture cell lines. The local minima of condition number is shown in green line and the corresponding number of genes was selected as the ‘optimal’ number of expression signature; (b) shows the relationship between the RMSD of the estimated fractions and the number of genes in basis matrix; (c) is the plot of the RMSD of the fitting residual vs. the number of genes in basis matrix.(TIF)Click here for additional data file.Figure S4Stability of chosen signature matrix. (a) Boxplot displaying the stability of chosen signature matrix. The chosen signatures are distorted by randomly selecting 5, 10, or 15 percent of its genes and randomly modulating their values with 5 fold changes. The distribution of correlations between actual mixing fractions and fractions estimated using these signatures is depicted. (b) Condition number of the basis matrix with respect to the percentage of simulated differentially expressed genes in the basis matrix.(TIF)Click here for additional data file.Table S1Experimental design for rat liver vs. kidney microarray experiment.(DOC)Click here for additional data file.Table S2Experimental design for rat brain vs. liver microarray experiment.(DOC)Click here for additional data file.Table S3Gene Annotation for blood vs. breast, liver vs. kidney and liver vs. brain experiments.(XLS)Click here for additional data file.Table S4Leukocyte types used as the basis for whole blood deconvolution.(DOC)Click here for additional data file.


Ensemble predictors such as the random forest are known to have superior accuracy but their black-box predictions are difficult to interpret. In contrast, a generalized linear model (GLM) is very interpretable especially when forward feature selection is used to construct the model. However, forward feature selection tends to overfit the data and leads to low predictive accuracy. Therefore, it remains an important research goal to combine the advantages of ensemble predictors (high accuracy) with the advantages of forward regression modeling (interpretability). To address this goal several articles have explored GLM based ensemble predictors. Since limited evaluations suggested that these ensemble predictors were less accurate than alternative predictors, they have found little attention in the literature.Comprehensive evaluations involving hundreds of genomic data sets, the UCI machine learning benchmark data, and simulations are used to give GLM based ensemble predictors a new and careful look. A novel bootstrap aggregated (bagged) GLM predictor that incorporates several elements of randomness and instability  often outperforms a host of alternative prediction methods including random forests and penalized regression models . This random generalized linear model (RGLM) predictor provides variable importance measures that can be used to define a “thinned” ensemble predictor (involving few features) that retains excellent predictive accuracy.randomGLM.RGLM is a state of the art predictor that shares the advantages of a random forest  with those of a forward selected generalized linear model (interpretability). These methods are implemented in the freely available R software package  Prediction methods  are widely used in biomedical research. For example, reliable prediction methods are essential for accurate disease classification, diagnosis and prognosis. Since prediction methods based on multiple features  can greatly outperform predictors based on a single feature , it is iEnsemble predictors are particularly attractive since they are known to lead to highly accurate predictions. An ensemble predictor generates and integrates multiple versions of a single predictor (often referred to as base learner), and arrives at a final prediction by aggregating the predictions of multiple base learners, e.g. via plurality voting across the ensemble. One particular approach for constructing an ensemble predictor is bootstrap aggregation (bagging) . Here mumtry, is the only parameter of the random forest predictor. The random forest predictor has deservedly received a lot of attention for the following reasons: First, the bootstrap aggregation step allows one to use out-of-bag (OOB) samples to estimate the predictive accuracy. The resulting OOB estimate of the accuracy often obviates the need for cross-validation and other resampling techniques. Second, the RF predictor provides several measures of feature (variable) importance. Several articles explore the use of these importance measures to select genes [generalized linear model indicates, it can be used for a general outcome such as a binary outcome, a multi-class outcome, a count outcome, and a quantitative outcome. We show that several incremental (but important) changes to the original bagged GLM predictor by Breiman add to up to a qualitatively new predictor (referred to as random GLM predictor) that performs at least as well as the RF predictor on the UCI benchmark data sets. While the UCI data are the benchmark data for evaluating predictors, only a dozen such data sets are available for binary outcome prediction. To provide a more comprehensive empirical comparison of the different prediction methods, we also consider over 700 comparisons involving gene expression data. In these genomic studies, the RGLM method turns out to be slightly more accurate than the considered alternatives. While the improvements in accuracy afforded by the RGLM are relatively small they are statistically significant.Breiman (1996) showed that bagging weak predictors (e.g. tree predictors or forward selected linear models) often yields substantial gains in predictive accuracy . But it ct genes ,13,14. Tct genes . Fourth,ct genes  but it rThis article is organized as follows. First, we present a motivating example that illustrates the high prediction accuracy of the RGLM. Second, we compare the RGLM with other state of the art predictors when it comes to binary outcome prediction. Toward this end, we use the UCI machine learning benchmark data, over 700 empirical gene expression comparisons, and extensive simulations. Third, we compare the RGLM with other predictors for quantitative (continuous) outcome prediction. Fourth, we describe several variable importance measures and show how they can be used to define a thinned version of the RGLM that only uses few important features. Even for data sets comprised of thousands of gene features, the thinned RGLM often involves fewer than 20 features and is thus more interpretable than most ensemble predictors.nFeaturesInBag) is selected for each bootstrap sample. This amounts to a random sub-space method [RGLM is an ensemble predictor based on bootstrap aggregation (bagging) of generalized linear models whose features (covariates) are selected using forward regression according to AIC criterion. GLMs comprise a large class of regression models, e.g. linear regression for a normally distributed outcome, logistic regression for binary outcome, multi-nomial regression for multi-class outcome and Poisson regression for count outcome . Thus, Re method  applied nBags  determines how many of such bootstrap data sets (referred to as bags) are being generated. Second, a random set of features (determined by the parameter nFeaturesInBag) is randomly chosen for each bag. Thus, the GLM predictor for bag 1 will typically involve a different set of features than that for bag 2. Third, the nFeaturesInBag of randomly selected features per bag are rank-ordered according to their individual association with the outcome variable y in each bag. For a quantitative outcome y, one can simply use the absolute value of the correlation coefficient between the outcome and each feature to rank the features. More generally, one can fit a univariate GLM model to each feature to arrive at an association measure . Only the top ranking features (i.e. features with the most significant univariate significance levels) will become candidate covariates for forward selection in a multivariate regression model. The top number of candidate features is determined by the input parameter nCandidateCovariates . Fourth, forward variable (feature) selection is applied to the nCandidateCovariates of each bag to arrive at a multivariable generalized linear model per bag. The forward selection procedure used by RGLM is based on the stepAIC R function in the MASS R library where method is set to “forward”. Fifth, the predictions of each forward selected multivariate model (one per bag) are aggregated across bags to arrive at a final ensemble prediction. The aggregation method depends on the type of outcome. For a continuous outcome, predicted values are simply averaged across bags. For a binary outcome, the adjusted Majority Vote (aMV) strategy [The steps of the RGLM construction are presented in Figure strategy  is used maxInteractionOrder  for creating interactions up to a given order among features in the model construction. For example, RGLM.inter2 results from setting maxInteractionOrder=2, i.e. considering pairwise  interaction terms. As example, consider the case when only pairwise interaction terms are used. For each bag a random set of features is selected  from the original covariates, i.e. covariates without interaction terms. Next, all pairwise interactions among the nFeaturesInBag randomly selected features are generated. Next, the usual RGLM candidate feature selection steps will be applied to the combined set of pairwise interaction terms and the nFeaturesInBag randomly selected features per bag resulting in nCandidateCovariates top ranking features per bag, which are subsequently subjected to forward feature selection.Importantly, RGLM also has a parameter randomGLM which allows the user to input a training set and optionally a test set. It automatically outputs out-of-bag estimates of the accuracy and variable importance measures.These methods are implemented in our R software package nBags=100 bootstrap data sets. The default value for nFeaturesInBag depends on the total number of features. It is easier to explain it in terms of the proportion of features randomly selected per bag, nFeaturesInBag/N, where N is the total number of features of the training set. Apart from N it is also valuable to consider the effective number of features which equals the number of features N plus the number of interaction terms, e.g. N∗=N+N(N−1)/2 in case of pairwise interactions. Using this notation, the default value of nFeaturesInBag can be arrived at by solving equations presented in Table nFeaturesInBag values  is often a good choice, whereas if N∗>300 then setting nFeaturesInBag/N=0.2 works well. The default value nFeaturesInBag/N=1.0276−0.00276N∗ in the intermediate case (10<N∗<=300) results from fitting an interpolation line through the two points  and . We find that RGLM is quite robust with respect to the parameter nFeaturesInBag. To limit the number of covariates considered in forward selection , the default value of nCandidateCovariates is set to 50. Overall, the default values perform well in our simulations, empirical gene expression and machine learning benchmark studies. But we recommend to use the OOB estimate of predictive accuracy to inform the choice of the parameter values.As discussed below, we find that it is usually sufficient to consider only As discussed below, RGLM can be interpreted as a variant of a bagged predictor . In part1. RGLM allows for interaction terms between features which greatly improve the performance on some data sets (in particular the UCI benchmark data sets). We refer to RGLM involving two-way or three way interactions as RGLM.inter2 and RGLM.inter3, respectively.nFeaturesInBag that allows one to restrict the number of features used in each bootstrap sample. This parameter is conceptually related to the mtry parameter of the Random Forest predictor. In essence, this parameter allows one to use a random subspace method  in eachnCandidateCovariates that allows one to restrict the number of features in forward regression, which not only has computational advantages but also introduces additional instability into the individual predictors, which is a desirable characteristic of an ensemble predictor.3. RGLM has a parameter 4. RGLM optimizes the AIC criterion during forward selection.5. RGLM has a “thinning threshold” parameter which allows one to reduce the number of features involved in prediction while maintaining good prediction accuracy. Since a thinned RGLM involves far fewer features, it facilitates the understanding how the ensemble arrives at its predictions.RGLM is not only related to bagging but also to the random subspace method (RSM) proposed by . In the Another prediction method, random multinomial logit model (RMNL), also shares a similar idea with RGLM. It was recently proposed for multi-class outcome prediction . RMNL barandomGLM. The R function randomGLM allows the user to output training set predictions, out-of-bag predictions, test set predictions, coefficient values, and variable importance measures. The predict function can be used arrive at test set predictions. Tutorials can be found at the following webpage: http://labs.genetics.ucla.edu/horvath/RGLM.The RGLM method is implemented in the freely available R package Forward selected generalized linear model predictor (forwardGLM) We denote by forwardGLM the (single) generalized linear model predictor whose covariates were selected using forward feature selection (according to the AIC criterion). Thus, forwardGLM does not involve bagging, random feature selection, and is not an ensemble predictor.Random forest (RF) RF is an ensemble predictor that consists of a collection of decision trees which vote for the class of observations [randomForest R package in our studies. We considered two choices for the RF parameter mtry: i) the default RF predictor where mtry equals the square root of the number of features and ii) RFbigmtry where mtry equals the total number of features. We always generated at least 500 trees per forest but used 1000 trees when calculating variable importance measures.rvations . The RF Recursive partitioning and regression trees (Rpart) Classification and regression trees were generated using the default settings rpart R package. Tree methods are described in [ribed in .Linear discriminant analysis (LDA) LDA aims to find a linear combination of features (referred to as discriminant variables) to predict a binary outcome . We Diagonal linear discriminant analysis (DLDA) DLDA is similar to LDA but it ignores the correlation patterns between features. While this is often an unrealistic assumption, DLDA  has been found to work well in in gene expression applications [supclust R package [ications . Here we package .K nearest neighbor (KNN) We used the knn R function in the class R package [k of nearest neighbors using 3-fold cross validation (CV). package ,23, whicSupport vector machines (SVM) We used the default parameters from the e1071 R package to fit SVMs [fit SVMs . Additiofit SVMs .Shrunken centroids (SC) The SC predictor is known to work well in the context of gene expression data [pamr R package [ion data . Here we package  which chPenalized regression models Various convex penalties can be applied to generalized linear models. We considered ridge regression [ℓ2 penalty, the lasso corresponding to an ℓ1 penalty [ℓ1 and ℓ2 penalties [glmnet R function from the glmnet R package [glmnet also involves another parameter (lambda) which was chosen as the median of the lambda sequence output resulting from glmnet. For UCI benchmark data sets, pairwise interaction between features were considered.gression  correspo penalty , and elaenalties . We used package ,31 with http://ligarto.org/rdiaz/Papers/rfVS/randomForestVarSel.html. The BrainTumor2 and DLBCL data sets were downloaded from http://www.gems-system.org/. The remaining 8 data sets (lung1 – MSdiagnosis2) were downloaded from either the Gene Expression Omnibus (GEO) database or the ArrayExpress data base in raw form and subsequently preprocessed using MAS5 normalization and quantile normalization. Only the top 10000 probes (features) with highest mean expression were considered for outcome prediction. We briefly point out that Diaz et al (2006) report prediction error rates estimated using a bootstrap method. In contrast, we report 3-fold cross validation estimates (averaged over 100 random partitions of the data into 3 folds), which may explain minor numerical differences between our study and that of Diaz et al (2006).We use 20 disease related gene expression data sets involving cancer and other human diseases  to generate binary outcomes. For all data sets except “Brain cancer”, Brain cancer data sets These two related data sets contain 55 and 65 microarray samples of glioblastoma (brain cancer) patients, respectively. Gene expression profiles were measured using Affymetrix U133 microarrays. A detailed description can be found in [found in . The firSAFHS blood lymphocyte data set This data set [data set  was deriWB whole blood gene expression data set This is the whole blood gene expression data from healthy controls. Peripheral blood samples from healthy individuals were analyzed using Illumina Human HT-12 microarrays. After pre-processing, 380 samples remained in the data set.Mouse tissue gene expression data sets The 4 tissue specific gene expression data sets were generated by the lab of Jake Lusis at UCLA. These data sets measure gene expression levels (Agilent array platform) from adipose (239 samples), brain (221 samples), liver (272 samples) and muscle (252 samples) tissue of mice from the B ×H F2 mouse intercross described in [ribed in ,54. In aThe 12 machine learning benchmark data sets used in this article are listed in Table y and gene expression data that contained 5 modules (clusters). Only 2 of the modules were comprised of genes that correlated with the outcome y. 45% of the genes were background genes, i.e. these genes were outside of any module. The simulation scheme is detailed in Additional file simulateDatExpr5Modules from the WGCNA R package [y was thresholded according to its median value.We simulated an outcome variable  package . This R We considered 180 different simulation scenarios involving varying sizes of the training data  and varying numbers of genes  that served as features. Test sets contained the same number of genes as in the corresponding training set and 1000 samples. For each simulation scenario, we simulate 5 replicates resulting from different choices of the random seed.We compare the prediction accuracy of RGLM with that of other widely used methods on 20 gene expression data sets involving human disease related outcomes. Many of the 20 data sets (Table mtry parameter), random forest , tree predictor , linear discriminant analysis (LDA), diagonal linear discriminant analysis (DLDA), k nearest neighbor (KNN), support vector machine (SVM) and shrunken centroid (SC). A short description of these prediction methods is provided in Methods.To arrive at an unbiased estimate of prediction accuracy, we used 3-fold cross validation (averaged over 100 random partitions of the data into 3 folds). Note that the accuracy equals 1 minus the median misclassification error rate. Table As seen from Table Our evaluations focused on the accuracy (and misclassification error). However, a host of other accuracy measures could be considered. Additional file A strength of this empirical comparison is that it involves clinically or biologically interesting data sets but a severe limitation is that it only involves 20 comparisons. Therefore, we now turn to more comprehensive empirical comparisons.y. The goal of each prediction analysis was to predict the dichotomized gene trait y based on the other genes. At first sight, this artificial outcome is clinically uninteresting but it is worth emphasizing that clinicians often deal with dichotomized measures of gene products, e.g. high serum creatinine levels may indicate kidney damage, high PSA levels may indicate prostate cancer, and high HDL levels may indicate hypercholesterolemia. To arrive at unbiased estimates of prediction accuracy, we split each data set into a training and test set. Figure p=2.1×10−51), RFbigmtry , LDA , SVM  and SC . Other predictors perform even worse, and the corresponding p-values are not shown.Many previous empirical comparisons of gene expression data considered fewer than 20 data sets. To arrive at 700 comparisons, we use the following approach: We started out with 7 human and mouse gene expression data sets. For each data set, we randomly chose 100 genes as gene traits (outcomes) resulting in 7×100 possible outcomes. We removed the gene corresponding to the gene trait from the feature set. Next, each gene trait was dichotomized by its median value to arrive at a binary outcome The fact that RFbigmtry is more accurate in this situation than the default version of RF probably indicates that relatively few genes are informative for predicting a dichotomized gene trait. Also note that RGLM is much more accurate than the unbagged forward selected GLM which reflects that forward selection greatly overfits the training data. In conclusion, these comprehensive gene expression studies show that RGLM has outstanding prediction accuracy.mlbench R package: 9 UCI data sets and 3 synthetic data sets  and RF  for the first place in the benchmark data. As can be seen from Additional file Overall, we find that RGLM.inter2 ties with SVM  ranged from 60 to 10000. The sample sizes (number of observations) of the training data ranged from 50 to 2000. To robustly estimate the test set accuracy we chose a large size for the corresponding test set data, In the following, we show that RGLM also performs exceptionally well when dealing with continuous quantitative outcomes. We not only compare RGLM to a standard forward selected linear model predictor (forwardGLM) but also a random forest predictor (for a continuous outcome). We do not report the findings for the k-nearest neighbor predictor of a continuous outcome since it performed much worse than the above mentioned approaches in our gene expression applications (the accuracy of a KNN predictor was decreased by about 30 percent). We again split the data into training and test sets. We use the correlation between test set predictions and truly observed test set outcomes as measure of predictive accuracy. Note that this correlation coefficient can take on negative values (in case of a poorly performing prediction method).mtry parameter and both methods perform significantly worse than the RGLM predictor.Here we used the same 700 gene expression comparisons as described above (100 randomly chosen gene traits from each of 7 gene expression data sets) but did not dichotomize the gene traits. Incidentally, prediction methods for gene traits are often used for imputing missing gene expression values. Our results presented in Figure Here we used the mouse liver and adipose tissue gene expression data sets to predict 21 clinical outcomes (detailed in Methods). Again, RGLM achieved significantly higher median prediction accuracy compared to the other predictors Figure .y was not dichotomized. As shown in Figure 180 gene expression data sets are simulated in the same way as described previously  but here the outcome diff =0.025,p=2×10−52) and the lasso  on the 700 dichotomized gene expression trait data. Also, RGLM is significantly better than elastic net  and lasso  in simulations with binary outcomes. Figure diff =0.035,p=2×10−86) in the 700 continuous gene expression traits data and outperforms elastic net  and lasso  in simulations with continuous outcomes.In our previous comparisons, we found that RGLM greatly outperforms forward selected GLM methods based on the AIC criterion. Many powerful alternatives to forward variable selection have been developed in the literature, in particular penalized regression models. Here, we compare RGLM to 3 major types of penalized regression models: ridge regression , elasticAs a caveat, we mention that cross validation methods were not used to inform the parameter choices of the penalized regression models since the RGLM predictor was also not allowed to fine tune its parameters. By only using default parameter choices we ensure a fair comparison. In a secondary analysis, however, we allowed penalized regression models to use cross validation for informing the choice of the parameters. While this slightly improved the performance of the penalized regression models (data not shown), it did not affect our main conclusion. RGLM outperforms penalized regression models in these comparisons.Here we briefly describe how RGLM naturally gives rise to variable (feature) importance measures. We compare the variable importance measures of RGLM with alternative approaches and show how variable importance measures can be used for defining a thinned RGLM predictor with few features.timesSelectedByForwardRegression, is a natural measure of variable importance . AnotheTo reveal relationships between different types of variable importance measures, we present a hierarchical cluster tree of RGLM measures, RF measures and standard marginal analysis based on correlations in Figure timesSelectedByForwardRegression is expected to lean towards selecting genes that are highly associated with the outcome, it comes as no surprise that only a few genes selected by the procedure of Díaz-Uriarte et al. turn out to have a top ranking in terms of the RGLM measure timesSelectedByForwardRegression  in the empirical studies. In contrast, RGLM typically only involves a few hundred genes in these studies. There are several reasons why RGLM uses far fewer features in its construction. First, and foremost, it uses forward selection (coupled with the AIC criterion) to select features in each bag. Second, the number of candidate covariates considered for forward regression is chosen to be low, i.e. nCandidateCovariates=50.Both RGLM and random forest have superior prediction accuracy but they differ with respect to how many features are being used. Recall that the random forest is composed of individual trees. Each tree is constructed by repeated node splits. The number of features considered at each node split is determined by the RF parameter timesSelectedByForwardRegression, follows a highly skewed distribution. Only few features are repeatedly selected into the model while most features are selected only once . It stands to reason that an even sparser, highly accurate predictor can be defined by refitting the GLM on each bag without considering these rarely selected features. We refer to this feature removal process as RGLM predictor thinning. Thus, features whose value of timesSelectedByForwardRegression lies below a pre-specified thinning threshold will be removed from the model fit a posteriori.In RGLM, the number of times a feature is selected by forward regression models among all bags, nFeaturesInBag is chosen to equal the total number of features. To ensure a fair comparison, we constructed and thinned the resulting RGLM in the training set only. Next, we evaluated the accuracy of the resulting thinned predictor in a test data set. Results were averaged across the 700 studies used in Figure nBags, is chosen to be 100, timesSelectedByForwardRegression takes on a value ≤100. Note that for a thinning threshold of 70 or larger, the median accuracy is constant at 0.5 which indicates that for at least 50% of comparisons the prediction is no longer informative. This reflects the fact that for large thinning thresholds, no covariates remain in the GLM models and the resulting predictor reduces to the “naive predictor” which assigns a constant outcome to all observations.Figure Interestingly, the accuracy diminishes very slowly for initial, low threshold values. But even low threshold values lead to a markedly sparser ensemble predictor  and proportion of features left in the thinned RGLM predictor:We have found that the following empirical function accurately describes the relationship between thinning threshold  (Equation 1) is a monotonically decreasing function which accurately describes the proportion of remaining features as can be seen from Figure nBags, we also study how these results depend on the choice of nBags. Toward this end, we varied nBags from 20 to 500 for predicting the 100 dichotomized gene traits in the mouse adipose data set. Additional file nBags, which indicates that Equation 1 accurately estimates the proportion of remaining features for range of different values of nBags.where Our results demonstrate that the number of required features decreases rapidly even for low values of the thinning threshold without compromising the prediction accuracy of the thinned predictor. Figure The idea behind RGLM thinning is to remove features with low values of the variable importance measure. Of course, a similar idea can be applied to other predictors. Here we briefly evaluate the performance of a thinned random forest predictor which removed variables based on a low value of its importance measure (“mean decreased accuracy”). To arrive at an unbiased comparison, both RGLM and RF are thinned based on results obtained in the training data. Next, accuracies of the thinned predictors are evaluated in the test set data. Figure random GLM. Our results show that the proposed changes  results in a more accurate predictor that involves surprisingly few features .After Breiman proposed the idea of bagged linear regression models in 1996 , many auAdditional reasons why the merits of RGLM have not been recognized earlier may be the following. First, it may be a historical accident. Bagging was quickly over-shadowed by other seemingly more accurate ways of constructing ensemble predictors, such as boosting  and the Second, previous comparisons of bagged predictors in the context of genomic data were based on limited empirical evaluations. Many comparisons involved fewer than 20 microarray data sets when comparing predictors ,5. WhileThird, previous studies probably did not consider enough bootstrap samples (bags). While previous studies used 10 to 50 bags, we always used 100 bags when constructing the RGLM. To illustrate how prediction accuracy depends on the number of bags, we evaluate the brain cancer data with 1 to 500 bags using 5 gene traits randomly selected from those used in our binary and continuous outcome prediction, respectively. The results are shown in Additional file RGLM shares many advantages of bagged predictors including a nearly unbiased estimate of the prediction accuracy (the out-of-bag estimate) and several variable importance measures. While our empirical studies focus on binary and continuous outcomes, it is straightforward to define RGLM for count outcomes (resulting in a random Poisson regression model) and for multi-class outcomes .stepAIC[MASS R package) is particularly time-consuming. The total time depends on the number of candidate features, the order of interaction terms, and the number of bags. Our R implementation allows the user to use parallel processing for speeding up the calculations.A noteworthy limitation of RGLM is computational complexity since the forward selection process  are used to inform parameter choices. Our Since individual forward selected GLMs are highly interpretable, the resulting ensemble predictor is more interpretable than an RF predictor. Our empirical studies  clearly highlight the outstanding prediction accuracy afforded by the RGLM. High accuracies are achieved not only in genomic data sets  but also in the UCI benchmark data .RGLM: Random generalized linear model; RGLM: Inter2 - RGLM considering pairwise interactions between features; RGLM: Inter3 - RGLM considering two-way and three-way interactions between features; forwardGLM: Forward selected generalized linear model; RF: Random forest with default mtry; RFbigmtry: Random forest with mtry equal to the total number of features; GLM: Generalized linear model; Rpart: Recursive partitioning; LDA: Linear discriminant analysis; DLDA: Diagonal linear discriminant analysis; KNN: K nearest neighbor; SVM: Support vector machine; SC: Shrunken centroids; RSM: Random subspace method; RMNL: Random multinomial logit model; RKNN: Random K nearest neighbor; E-RFE: Entropy-based recursive feature elimination; AIC: Akaike information criteria; aMV: Adjusted majority vote.The authors declare that they have no competing interest.LS carried out all analyses. PL helped with the R implementation and analysis. LS and SH developed the method and wrote the article. SH conceived of the study. All authors read and approved the final manuscript.Simulation study design. This file describes the simulation studies and presents R code used for simulating the data set.Click here for fileSensitivity and specificity of predictors in the 20 disease gene expression data sets. For each data set and prediction method, the table reports the sensitivity and specificity estimated using 3-fold cross validation. More precisely, the table reports the average 3-fold CV estimate over 100 random partitions of the data into 3 folds. Median sensitivity and specificity across data sets are summarized at the bottom.Click here for fileSensitivity and specificity of predictors in the UCI machine learning benchmark data. For each data set and prediction method, the table reports the sensitivity and specificity estimated using 3-fold cross validation. More precisely, the table reports the average 3-fold CV estimate over 100 random partitions of the data into 3 folds. Median sensitivity and specificity across data sets are summarized at the bottom.Click here for filePrediction accuracy when including pairwise interactions between features in the UCI machine learning benchmark data. This table is an extension to Table Click here for fileComparison of RGLM based feature selection method with the RF based method of Díaz-Uriarte et al. For each data set in the 20 disease gene expression data, the RF based variable selection method by Díaz-Uriarte et al selects a small set of genes. For each of the selected genes, the file reports the ranking in terms of the RGLM variable importance measure timesSelectedByForwardRegression. As expected, only a few of the selected genes have a high rank in terms of timesSelectedByForwardRegression illustrating that these variable selection methods are different.Click here for fileEffect of the number of bags on RGLM predictor thinning. s This figure reports how prediction accuracy changes as variable thinning is applied to the RGLM. Results are averaged over the 100 dichotomized gene traits in the mouse adipose data set. The five rows correspond to nBags values of 20, 50, 100, 200, 500 respectively. Within each row, the two panels have the same meaning as in Figure Click here for filePrediction accuracy versus number of bags used for RGLM. This figure presents the results for predicting 5 gene traits in the brain cancer data set when different numbers of bags (bootstrap samples) are used for constructing the RGLM. Each color represents one gene trait. (A) Binary outcome prediction. The 5 gene traits were randomly selected from all 100 gene traits used in the binary outcome prediction section. (B) Continuous outcome prediction. The 5 gene traits were randomly selected from all 100 gene traits used in the continuous outcome prediction.Click here for file


We use several approaches to compare their performance on two real gene expression data sets. For this purpose we apply four evaluation measures in our analysis: (1) we examine how well the considered (bi)clustering methods differentiate various sample types; (2) we evaluate how well the groups of genes discovered by the (bi)clustering methods are annotated with similar Gene Ontology categories; (3) we evaluate the capability of the methods to differentiate genes that are known to be specific to the particular sample types we study and (4) we compare the running time of the algorithms. In the end, we conclude that as long as the samples are well defined and annotated, the contamination of the samples is limited, and the samples are well replicated, biclustering methods such as Plaid and SAMBA are useful for discovering relevant subsets of genes and samples.DNA microarray technologies are used extensively to profile the expression levels of thousands of genes under various conditions, yielding extremely large data-matrices. Thus, analyzing this information and extracting biologically relevant knowledge becomes a considerable challenge. A classical approach for tackling this challenge is to use clustering  methods where genes (or respectively samples) are grouped together based on the similarity of their expression profiles across the set of all samples (or respectively genes). An alternative approach is to develop biclustering methods to identify local patterns in the data. These methods extract subgroups of genes that are co-expressed across only a subset of samples and may feature important biological or medical implications. In this study we evaluate 13 biclustering and 2 clustering ( One of the approaches for analysis and interpretation of large quantities of high-throughput data is clustering , where genes, samples, or both, are grouped together based on their gene expression profiles k-means clustering and self-organizing maps vice versa). Thus, groups of genes that are co-expressed only in a subset of samples may be left undetected. A promising solution to identify subgroups of genes and samples is the so called biclustering approach k-means, is that the clustering is done simultaneously for genes and samples. Wang et al. Hierarchical clustering with heatmap visualization k-means and hierarchical) methods. The first data comprises five different types of tissues consisting of expression data with heterogeneous samples that resides bicluster structures with small overlaps on their genes and samples. For the second data set we chose two clinically well-defined subgroups of breast tumor (ER+/PR+/HER2+ and ER−/PR−/HER2−) and reference breast samples. Due to the homogeneity of the samples and the common active biological pathways in different tumor subtypes, the breast cancer data is expected to reside bicluster structures with overlapping genes and samples. For our comparison analysis, we applied four benchmarks: Sample differentiation, Gene Ontology-based significance, Tissue specificity of the genes, and Running time.Given that the concept behind the biclustering approach is appealing in biosciences, a number of biclustering methods have been developed Biclustering methods can be categorized based on the type of the searched biclusters as well as the mathematical formulation used to discover them. Using these two criteria, we have categorized biclustering techniques into four classes: Correlation maximization biclustering methods, Variance minimization biclustering methods, Two-way clustering methods, and Probabilistic and generative methods.Correlation maximization biclustering methods (CMB) seek for subsets of genes and samples where the expression values of the genes (or respectively samples) correlate highly among the samples (or respectively genes). et al.Variance minimization biclustering methods (VMB) search for biclusters in which the expression values have low variance throughout the selected genes, conditions or the whole submatrix. For instance, XMOTIF Two-way clustering methods (TWC) discover the homogeneous subsets of genes and samples, i.e. biclusters, by iteratively performing one-way clustering on the genes and samples. For instance, the algorithm proposed by Getz et al.vice versa. Another example is an algorithm proposed by Chun Tang et al.Probabilistic and generative methods (PGM) employ probabilistic techniques to discover genes (or respectively samples) that are similarly expressed across a subset of samples (or respectively genes) in the data-matrix et al., called cMonkey Detailed information regarding the biclustering methods used in our study, including their class, parameters and characteristics, are listed in The multi-tissue data we use within our study consists of 228 samples from 5 distinct healthy human tissues from the GeneSapiens database http://cancergenome.nih.gov/. The TSP study accession number of the raw data in the database of Genotype and Phenotype (dbGaP) is phs000569.v1.p7. Two clinically well-defined subgroups of breast tumor (ER+/PR+/HER2+ and ER−/PR−/HER2−) and healthy breast samples were chosen for our analysis. All genes with a variance less than one across the samples were also discarded.To create the breast tumor data gene expression microarrays were downloaded from The Cancer Genome Atlas for primary breast carcinoma tumors and controls. First, probes matching either multiple or no genes were removed. Then, data were normalized to a mean of 0. The original data can be obtained from TCGA web site Recently, K. Eren and colleagues studied a collection of biclustering methods on several synthetic data matrices that housed various types of bicluster structures and estimated how well each method discovers them a priori knowledge. These benchmarks can be classified as external benchmarks as described by Santamaria et al.In addition to running time analysis, they reported results of Gene Ontology based enrichment analysis in order to evaluate the gene-sets of biclusters discovered in a gene-expression data of Rat peripheral and brain regions. Here we, however, focus on the biological relevance of the biclusters discovered by the 13 biclustering and 2 one-way clustering methods. In particular, we focused on the ability of these methods to distinguish various sample types rather than their performance in discovering various bicluster patterns in the data. In this regard, we consider four kinds of benchmarks: one sample-based, two gene-based and the running time. All the applied benchmarks measure how much the generated clusters succeed in incorporating Sample-based benchmarks evaluate the (bi)clusters generated by a given method by assessing the set of samples included in them. These benchmarks answer the question of how well a method can distinguish different types of samples. If we denote by sample differentiation benchmark SampleDif. Note that contamination of the samples with other tissue types or miss-annotation of the samples can affect the sample differentiation.This category refers to those benchmarks that estimate the quality of the (bi)clusters by assessing the genes included in them. Here we consider two such benchmarks.Gene Ontology-based significance (denoted by GO-Sig) is one of the widest used gene-based benchmarks for biclustering methods et al.TiGER-based significance (denoted by TiGER-Sig) indicates the percentage of the biclusters generated by each method that include genes specific to the studied sample-types. For the multi-tissue type gene expression data we employ the Tissue-specific Gene Expression and Regulation (TiGER) database In addition to the quality of the extracted (bi)clusters, it is also important that the analysis is done in a reasonable amount of time. Thus, we compared the running time of the studied algorithms.k-means Using these four benchmark measures we evaluated 13 biclustering methods: SAMBA k-means method and the Pearson distance for the hierarchical method. The cluster number threshold for both was also set to 10 when clustering the genes. For clustering of the samples the threshold was set to 5 for the multi-tissue type gene-expression data and 3 for the breast tumor. Moreover, within the hierarchical clustering we used the complete linkage method. In SAMBA the overlap prior factor was set to 0.1. The responding probes to hash was set to 100 and the hash kernel size  was set to 4. The hash-tables are the data structures used by the algorithm to store the converging biclusters (i.e. weighs of the edges of a bipartite graph in which the nodes represent a selection of genes and samples)The Euclidean distance metric was used for the In the additive and constant MSBE biclustering methods the The results for the sample-based and gene-based evaluations of the 13 biclustering and 2 clustering methods on the multi-tissue type data and the breast tumor data are illustrated in k-means (59.5%), as well as those given by constant MSBE and FABIA biclustering methods, were also relatively high (∼60%).For the heterogeneous data, three biclustering methods feature sample differentiation values larger than 80%: Plaid (96.2%), SAMBA (89.5%), and CTWC (86.5%), as shown in k-means (100%), hierarchical (100%), and SAMBA (97.5%) generated a high percentage of gene sets that were significantly annotated. Additionally, a relatively high proportion of the OPSM (80%) and Plaid (85%) biclustering results were also enriched. The TiGER-Sig analysis also showed that Plaid (65%), QUBIC (50%), additive MSBE (46.1%), hierarchical clustering (40%), and k-means (40%) algorithms discover gene-sets significantly enriched with genes specific to the studied samples.The GO enrichment analysis indicated that k-means and heirarchical) were decreased. However, as opposed to the Hierarchical clustering the k-means differentiated the two breast tumor subtypes (ER+/PR+/HER2+ and ER−/PR−/HER2−) and the healthy breast samples accurately (99%). FABIA (93%), CTWC (91%), and Plaid (89%) biclustering methods also differentiated the various cancer sub-types very well. The 5 methods that discovered gene-lists in which the highest percentage feature significant common GO annotations are FLOC (85%), SAMBA (73%), Hierarchical (70%), ISA (63%), and FABIA (55%). Moreover, more than half of biclusters discovered by SAMBA (67%) and Plaid (55%) significantly overlapped genes that were reported by the CancerGenes to be related to cancer. When we executed Bimax on the breast tumor data the algorithm did not converge in a reasonable time (720 hours) hence we could not extract any biclusters from the data. In addition to estimating the fraction of the (bi)clusters that featured similar GO annotations, we also studied the significantly common GO categories that were extracted by GO analysis of the bicluster results. We found that gene-lists discovered by the CTWC, FABIA, ISA, Plaid, SAMBA, and hierarchical clustering were significantly enriched with GO terms: cell cycle, M phase of the cell cycle, mitosis, cell division, proliferation, and response to stress. Moreover, gene-lists discovered by CTWC, FABIA, ISA, and SAMBA were annotated with immune response. A subset of these GO categories  were also reported by Wang et al.The sample differentiation and the GO-Sig measurements of the biclustering methods ISA, FABIA, FLOC, CC, and BiMine were clearly improved for the more homogenous Breast tumor data comparing to their performance for the heterogeneous multi-tissue type data, see k-means clustering method , MSBE-Additive , MSBE-Constant , QUBIC , ISA , SAMBA , Plaid .The running time of the 13 biclustering and 2 clustering algorithms that worked successfully on our two micro-array data are illustrated in Clustering is a powerful approach to extract biologically relevant information from the high-throughput data. While clustering techniques, such as k-means or hierarchical clustering, are able to find similarities of genes over all conditions , biclustering methods search for local patterns that may feature important biological or medical implications. Here we have compared 15 (bi)clustering methods by analyzing different aspects, such as their approach and parameter settings. Moreover, we have introduced several evaluation measures for comparing the performance and application of these biclustering methods.k-means, SAMBA, hierarchical clustering, OPSM, and Plaid methods were significantly annotated with similar Gene Ontology categories when they were applied on the multi-tissue type data. However, OPSM discovered biclusters with relatively high mean overlap on their genes (55%). This can bias the GO-Sig results in favoring OPSM algorithm. The TiGER-Sig analysis on the multi-tissue data also confirmed that the Plaid, QUBIC, additive MSBE, hierarchical clustering, and k-means discovered gene-sets significantly enriched with genes that are specific to our studied samples. The high performance of the one-way clustering methods on the multi-tissue data was expected since the heterogeneity of the samples can favor methods that extract non-overlapping sets of genes or samples from the data . On breast cancer data, k-means best differentiated the two breast tumor subtypes (ER+/PR+/HER2+ and ER−/PR−/HER2−) and the healthy breast samples. FABIA, CTWC, and Plaid differentiated the samples almost as good as the k-means. The gene-sets generated by FLOC method were also most frequently enriched with similar GO categories in the breast tumor data analysis. However, similar to OPSM in multi-tissue type data analysis, we believe that the high GO-Sig of this method is biased by the high mean overlap (55%) of the genes discovered by FLOC. A considerable fraction of the results generated by SAMBA, hierarchical and ISA were also significantly annotated with similar Gene Ontology categories. Taken together, we found that no single method performs the best in all measurements and on both data.Our results show that Plaid, SAMBA, CTWC, hierarchical clustering, constant MSBE, and FABIA methods best distinguished the various sample-types in the multi-tissue type gene expression matrix. Moreover, the GO enrichment analysis indicated that the gene-sets generated by the When comparing the performance of the (bi)clustering methods on the two data sets of our study we realized that in the more homogeneous breast tumor data the GO-sig of the two conventional clustering methods  have decreased. Also, when applied on the breast tumor data set (with more homogeneous samples comparing to the multi-tissue type data) all benchmarks for CC, FLOC, and BiMine (and sample-differentiation and GO-Sig of FABIA) increased while all benchmarks for the Qubic, MSBE-A, MSBE-C and Plaid decreased. It is worth mentioning that, except FABIA, all the biclusters with improved performance  were members of the CMB (Correlation Maximization Biclusters) class. FABIA seeks for Variance Minimized Biclusters although classified as PGM because of its use of probabilistic and generative models. The methods with declined performances were of different classes: Qubic, MSBE-A and MSBE-C methods are VMB (Variance Minimization biclusters); The Plaid model is PGM (Probabilistic and Generative Methods); and OPSM is CMB.et al.et al.Our results are in line with other biclustering comparison studies. For instance, Hochreiter To conclude, taking into consideration our analysis and the results reported by K. Eren et al. and S. Hochreiter et al. as well as the limitations and pitfalls of the evaluation methods, biclustering methods such as Plaid and SAMBA are useful for extracting relevant subsets of genes and samples from microarray experiments as long as the samples are well defined and annotated, the contamination of the samples is limited, and the samples are well replicated. Moreover, our results indicate that biclustering algorithms such as Plaid and SAMBA find more relevant gene-sets comparing to the clustering algorithms when the samples are not highly heterogeneous. This suggests that in studies where different samples feature common active biological processes and genes are also active in several biological processes (e.g. cancer studies), biclustering algorithms could discover more relevant genes comparing to one-way clustering methods.