With the large amount of biological data that is currently publicly available, many investigators combine multiple data sets to increase the sample size and potentially also the power of their analyses. However, technical differences (“batch effects”) as well as differences in sample composition between the data sets may significantly affect the ability to draw generalizable conclusions from such studies.The current study focuses on the construction of classifiers, and the use of cross-validation to estimate their performance. In particular, we investigate the impact of batch effects and differences in sample composition between batches on the accuracy of the classification performance estimate obtained via cross-validation. The focus on estimation bias is a main difference compared to previous studies, which have mostly focused on the predictive performance and how it relates to the presence of batch effects.We work on simulated data sets. To have realistic intensity distributions, we use real gene expression data as the basis for our simulation. Random samples from this expression matrix are selected and assigned to group 1  or group 2 . We introduce batch effects and select some features to be differentially expressed between the two groups. We consider several scenarios for our study, most importantly different levels of confounding between groups and batch effects.We focus on well-known classifiers: logistic regression, Support Vector Machines (SVM), k-nearest neighbors (kNN) and Random Forests (RF). Feature selection is performed with the Wilcoxon test or the lasso. Parameter tuning and feature selection, as well as the estimation of the prediction performance of each classifier, is performed within a nested cross-validation scheme. The estimated classification performance is then compared to what is obtained when applying the classifier to independent data. Largely facilitated by online repositories such as Gene Expression Omnibus  confounded with the class variable. As an example, consider a situation where we are interested in building a classifier to distinguish cancer patients (group 1) from healthy volunteers (group 2), and we combine one public data set consisting of only healthy volunteers with our own data set consisting of only patients. In such a situation, the fact that the patients and the healthy volunteers come from different data sets may introduce apparent differences between them that are not truly related to the disease, and that thus may fail to generalize or be replicated in other studies. The main goal of this study is to evaluate whether the presence of a confounding factor introduces a bias in “internal” classifier performance estimates obtained via cross-validation compared to the actual performance of the classifier on external data. Subsequently, we are also interested in whether, in the presence of a confounding factor, a commonly used batch effect removal method is able to eliminate the potential bias it introduces. The importance of the study follows from the observation that internal cross-validation estimates are often used in practice as proxys for the true  performance (that we expect if the classifier is applied to an independent data set), since researchers typically want to use as many samples as possible to construct the classifier. Simulation studies like this one are therefore important to estimate the accuracy of the internal measure. We demonstrate that running a standard pipeline of statistical tools in cases where there is strong inherent bias in the input data can give very misleading classification performance estimates, and that not all experimental design problems can be corrected retrospectively. This stresses the need for careful planning before an experiment is performed, in order to avoid batch effect confounding with the endpoint of interest as much as possible.Our input data is simulated, which means that we have access to the ground truth to estimate the classifiers' performance. We restrict ourselves to situations with two groups (binary classification). In this setting, we perform all steps from the simulated data  gene expression matrix . Furthermore, we simulate both training data sets where the two batches behave the same  and data sets where the two batches behave differently . For training data sets with batch effect, we simulated We simulated validation data sets are balanced and are not affected by batch effects, which is motivated by a desire to generate validation data sets that are representative of a general population. The influence of batch effects between the training and validation sets was examined by R code All generated To eliminate the effect of the confounding factor on the training data set, we use ComBat classifiers. A classifier consists of a collection of predictor variables and a classification rule such that, given observed values of the predictor variables for a sample, we can plug them into the classification rule and based on the result assign a group label to the sample. To build such a classifier in a supervised setting, we need a training data set, consisting of samples for which we are given the observed values of a number of variables as well as the true group labels. This data is used to select appropriate predictor variables from the variable pool and to construct the classification rule. The goal, however, is to obtain a classifier that will work well for predicting the group labels of new samples, that are not part of the training set. In fact, we can even claim that predicting the group labels of the samples in the training set correctly is meaningless since the true group labels are already known. To measure the performance of a classifier constructed from a training set, we thus need to apply it to an independent test set that is representative of the data sets for which we want to apply our classifier, but for which the true group labels are known. In this way, we can record the agreement between the labels predicted by the classifier and the true labels. This will tell us how well the classifier can be expected to work in general on independent test data sets (where we do not know the true group labels).The main theme in this paper is the construction of fold), and using one of the parts as the test set and the remaining In practical situations, we typically want to use all the available data to construct the classifier. Various methods have been proposed to artificially generate training and test sets from a single data set. Cross-validation is one of the most widely used such approaches, and involves randomly splitting the data set into Cross-validation is used for many different purposes, arguably the most common ones being to determine the optimal value of hyperparameters for a classifier and to estimate the performance we can expect from a given classification procedure if applied to independent data.Since the estimates obtained through cross-validation are derived from subsets of the original data set, with different numbers of samples than in the whole set, there will be an inherent bias in these estimates that depends on the value of stratified cross-validation, meaning that the fraction of samples from each class is kept as constant as possible across the different cross-validation folds. This ensures for example that each class is present in all training sets.We use In this study we focus on the additional biases that may result if the data set used to build the classifier  is not representative of the collection of new data sets to which the classifier will eventually be applied. In this case, as we will see, the performance estimate obtained from the cross-validation can be far from the actual performance of the classifier on a new independent data set.outer cross-validation loop is to provide an estimate of the classification performance of a constructed classifier. The inner cross-validation loop, in contrast, is used to build the classifier, that is, to select the optimal combination of hyperparameters and the subset of predictor variables for the classifier. To distinguish the training and test sets from the two levels, we denote the data sets generated in the outer cross-validation loop the outer training and test sets, respectively. Given that the fold of the outer cross-validation is inner training and test sets. Note that since the cross-validation procedure is nested, each pair of inner training and test sets will be generated from, and thus be subsets of, one of the outer training sets. The fold of the inner cross-validation is denoted In the present study, we apply the cross-validation procedure in a nested, or two-level, fashion as illustrated in the top panel of In each round of the outer cross-validation loop, we thus create an outer training set and an outer test set. On the outer training set, we then apply the inner cross-validation procedure. More precisely, for each inner training set we build several classifiers . Each of the classifiers is applied to the corresponding inner test set, and the classification performance is recorded. This is repeated for all inner training sets, and the observed performances are averaged across the inner test sets. The combination of hyperparameter value and number of predictor variables giving the best averaged predictive performance is returned from the inner cross-validation loop. Bearing in mind that the combination was chosen since it performed best in the inner cross-validation loop, the performance estimate obtained from the inner cross-validation is biased. It is an overoptimistic estimate of the actual performance of the classifier. Thus, to obtain a better estimate of the performance, a new classifier is built on the outer training set using the selected optimal hyperparameter value and number of variables. This new classifier is then applied to the corresponding outer test set. The predictive performance obtained for this test set is recorded. The final cross-validation estimate of the classification performance is obtained by averaging the estimates obtained for each of the not an estimate of the performance of a specific classifier , since different classifiers are built in each round of the outer cross-validation loop. Rather, the value obtained from the cross-validation provides an estimate of the performance of a classifier generated through a specified workflow . We compare the performance estimate obtained by the outer cross-validation procedure (perfCV) to what we consider the “true” performance (perftrue). We obtain perftrue by building a classifier on the whole original training data set (following the same procedure as above), and applying it to an independent validation data set. This workflow is depicted in the lower panel of perfCV and perftrue are similar, we conclude that the cross-validation based estimate is unbiased, and that perfCV provides a useful estimate of the real performance of our classifier. Conversely, if the two values are far from each other, the cross-validation estimate does not say much about the actual performance of the classifier, and consequently can be quite misleading in practical situations. The whole procedure used to build and evaluate a classifier is outlined in It is important to note that this value is misclassification rate, which is defined as the fraction of the samples that are assigned the wrong label by the classifier. This often works well but can be misleading when the groups are of very different size.Many measures have been proposed for quantifying the performance of a classifier. The simplest measure is the Since all our examples are on balanced data, we present the results in the form of the misclassification rate. Results for other performance measures can be found in As noted above, a classifier consists of a collection of predictor variables and a classification rule to combine the observed predictor values and yield a predicted group label. In this section we briefly describe the four approaches we use to build prediction rules, and the next section outlines the methods for selecting predictor variables. We consider four classification approaches in this study; random forests, support vector machines, k-nearest neighbor classifiers and penalized logistic regression.rfCMA function in the CMA package RrandomForest R package The random forest classifier was proposed by Breiman svmCMA function in the CMA R package. This implicitly calls functions from the e1071 R package Support vector machines (SVMs) loop see . We builknnCMA function in the CMA R package. This implicitly calls functions from the class R package The k-nearest neighbor  that are confounded with the group label in the training data and that affect the values of the variables. For example, imagine a situation where most of the patients treated with drug A  come from one data set, and most of the samples treated with drug B (sample group 2) come from another data set. In another situation, the patients treated with drug A could be significantly older than those treated with drug B. Confounding factors like these may affect certain genes in such a way that we observe a difference between the two sample groups in the training data, which are not truly related to the factor we are interested in . This means that it may very well be possible to build a classifier that works well on this specific data set, but since these variables are not truly linked to the differences between the drugs, but rather to technical effects, they are unlikely to hold up as good discriminators in another data set. Moreover, the estimate of the predictive performance obtained through the cross-validation procedure on the training set may be far from accurate. Note that in this context, we consider also batch effects that are not necessarily known to the investigator.Under the internal measure) as well as the “true” performance obtained by applying the final classifier to an independent data set . We are interested in knowing whether the internal measure is an accurate reflection of the external measure. In the figures, we have combined the results for all four evaluated classification algorithms , since they behave similarly. In The training set is simulated to contain a batch effect, which is not present in the validation data. Moreover, we assume that we are unaware of this batch effect, and thus do not make any attempt to eliminate it at this stage. The four columns of As expected, in all cases, the performance of the classifiers when applied to the external validation set is not better than chance, with an average misclassification rate close to 50%. However, the cross-validation estimate  depends strongly on the level of confounding between the batch and the group labels. When there is no confounding , the cross-validation estimate is almost unbiased, although the variance is larger than for the external measure. In other words, the performance estimate provided by the cross-validation is a useful proxy for the true performance. As the degree of confounding increases (moving towards the right in the figure), the cross-validation performance estimate becomes increasingly over-optimistic, and with full confounding the cross-validation procedure estimates the misclassification rate to 0 . This is not surprising since in the case of full confounding, there is no way to discriminate the batch effect from a true group difference in the training data set. The results observed in the presence of confounding thus imply that the performance estimate obtained by the cross-validation is in fact far from the performance we can expect if we apply the classifier to an external data set, and thus rather misleading.number of variables that are selected as being optimal for each degree of confounding. When there is no confounding between the group and the batch, almost no variables that are associated with the batch effect will be selected with any of the variable selection methods. As the level of confounding increases, the fraction of batch-related genes that are included in the final classifier increases, most rapidly for the Wilcoxon variable selection. Recall that these genes are not truly associated with the group discrimination, and that in fact they do not hold up as good discrimination rules when applied to a data set without this specific batch effect , and we apply ComBat Since the batch effect is the only non-random effect in this data, eliminating it makes the previously batch-affected variables indistinguishable from the rest of the variables. Thus, it is not surprising that around half of the selected variables are “batch-related”, and the other half are “non-batch-related” .An interesting effect is seen in the right-most panel of the figure. When the batch is completely confounded with the group labels, and the batch effect is removed using ComBat, the cross-validation often gravely underestimates the classification performance. This can be explained by an overcompensation mechanism. In this case, since the batch and the group factor are in fact identical, we can not provide the group factor as a covariate to ComBat in order to retain the information related to it. Instead, we just attempt to eliminate the effect of the batch on the expression data. In principle, one goal of the batch effect removal is to make the average expression of each gene equal in the two batches. Since the batch effect removal was performed on the entire training data set this means that if we, in the classifier construction, find a gene that is higher in group 2 than in group 1 in an outer training set, it necessarily has the opposite pattern in the corresponding outer test set. Hence, when we apply the trained classifier to this test set to estimate the performance, many of the samples will be assigned to the wrong group and the estimate of the misclassification rate will be high. However, for the external data set used to validate the classifier, these genes are equally distributed in the two groups and thus the classifier performs no better (or worse) than chance.Here, we consider the case where, in contrast to the example above, there actually are some genes that are differentially expressed between the two groups that we wish to distinguish. As for the null case above, we examine the effect of a confounding variable (the batch), with varying degree of confounding between the batch and the group labels. In this case, since there are genes that are truly differentially expressed between the two groups of interest, we expect that it should be possible to obtain a classifier with good classification performance. However, if the batch effect is strong and confounded with the factor of interest, we anticipate that the variable selection may be guided towards these variables, which are not truly differentially expressed between the two populations and thus do not generalize well to independent data sets.As the degree of confounding increases, the classifier tends to a higher extent to select genes that are associated with the batch effect, which provides a classifier that works well for the training data but that does not generalize well, especially for the Wilcoxon variable selection.Finally, we investigate the results obtained after eliminating the batch effect using ComBat . CompariFor extensive confounding we make the same observation as for the null case: the elimination is not fully efficient and the performance estimates from the cross-validation are heavily biased. Moreover, for the full confounding case we again overcompensate in the batch effect elimination, and thus the cross-validation underestimates the classification performance. The results can be compared to those given in The increasing amount of publicly available gene expression data provides researchers with the potential of combining them in order to create large collections of samples, which may provide higher power in addressing research hypotheses. Many researchers have already taken advantage of the large collection of public data, either by considering exclusively already published data sets, or by combining the public data with their own generated data There are several, qualitatively different, approaches to combining a collection of data sets, depending on the research question of interest. One widely used approach is meta-analysis, which aggregates the results obtained by performing the analysis of interest separately on each data set. Meta-analysis is appropriate for example to examine whether a certain gene is consistently found to be associated with a given phenotype across several studies. However, meta-analysis is not always a feasible alternative. For example, to perform unsupervised analysis like clustering, or to construct a classifier, the different data matrices typically have to be merged in order for the analysis to profit from the increased sample size. In these situations, unaddressed batch effects may severely compromise the results of the analysis by concealing the real signal or introducing an artificial one Unless the batch effect is heavily confounded with the outcome of interest, eliminating the batch effect typically improves the performance of the resulting classifier. From this point of view, hence, elimination of batch effects before constructing a classifier is beneficial. However, the bias in the cross-validation performance estimates is not eliminated by the batch effect removal, and consequently the cross-validation performance estimates obtained after batch effect elimination are not more reliable measures of the true performance than those obtained without batch effect elimination. This apparent insufficiency of the batch effect removal stresses the importance of careful experimental design where confounding between potential batch effects and the signal of interest are avoided as much as possible. In other words, batch effect removal methods should not be trusted blindly as a ‘post-experimental’ way of rescuing a badly designed experiment.The presented results have implications not only for binary classifier evaluation, but also for evaluation of multi-class classifiers and predictive models with other endpoints, such as survival time. Moreover, the results presented here are important for differential expression analysis and other approaches for ranking genes in terms of significance. We have shown that in the presence of batch effects that are confounded with the signal of interest, many of the highly ranked variables are associated only with the batch effect and not truly differentially expressed between the interesting groups. Consequently, they are unlikely to hold up as statistically, as well as biologically, significant discriminators in any other study. For the two variable selection methods we evaluated, the Wilcoxon test seemed to be more sensitive to the confounding, and included more batch effect related genes in the final classifier than the lasso variable selection. We hypothesize that one reason for this is that the lasso considers all variables simultaneously when doing the selection, and tends to include relatively uncorrelated variables in the final selection While the present study is focused on cross-validation as the method for estimating classification performance, other methods have been suggested for the same purpose . It has been shown that in general, the bias of these methods can be quite different Supporting Information S1Additional results and simulation details. Detailed description of the data simulation procedure, as well as further details and alternative representations of the reported results.(PDF)Click here for additional data file.Supporting Information S2R code used for data simulation and analysis. Documented R code for the simulation procedure as well as the analysis and result generation.(HTML)Click here for additional data file.


PPAR−γ (NR1C3) and C-MYC gene hubs interacting through the AP-1 (c-Jun) transcription factor. PPAR−γ (NR1C3) is a drug target for pioglitazone, which has been shown to reduce depression symptoms in patients with MDD. Using a data-driven approach we were able to overcome challenges of non-i.i.d. data when combining epigenetic studies from MZ twins discordant for MDD. Individually, the studies yielded negative results but when combined classification of the disease state from blood epigenome alone was possible. Network analysis revealed genes and gene networks that support the inflammation hypothesis of MDD.Although monozygotic (MZ) twins share the majority of their genetic makeup, they can be phenotypically discordant on several traits and diseases. DNA methylation is an epigenetic mechanism that can be influenced by genetic, environmental and stochastic events and may have an important impact on individual variability. In this study we explored epigenetic differences in peripheral blood samples in three MZ twin studies on major depressive disorder (MDD). Epigenetic data for twin pairs were collected as part of a previous study using 8.1-K-CpG microarrays tagging DNA modification in white blood cells from MZ twins discordant for MDD. Data originated from three geographical regions: UK, Australia and the Netherlands. Ninety-seven MZ pairs  discordant for MDD were included. Different methods to address non independently-and-identically distributed (non-i.i.d.) data were evaluated. Machine-learning methods with feature selection centered on support vector machine and random forest were used to build a classifier to predict cases and controls based on epivariations. The most informative variants were mapped to genes and carried forward for network analysis. A mixture approach using principal component analysis (PCA) and Bayes methods allowed to combine the three studies and to leverage the increased predictive power provided by the larger sample. A machine-learning algorithm with feature reduction classified affected from non-affected twins above chance levels in an independent training-testing design. Network analysis revealed gene networks centered on the  This can be defined simply as the variation in phenotypic expression produced by the up- and downregulation of genes using DNA methylation and histone modification, rather than changes in the DNA sequence. It has been suggested that through these processes environmental factors can cause lasting changes in gene expression.12 For example, early-life stress leads to epigenetic changes in neural tissue involved in the stress response in both animal and human studies.14Studying depression from the epigenetic perspective sheds light on its etiology by uncovering how environmental factors modulate gene expression.15Quantitative genetics research typically compares phenotypic concordance rates between monozygotic (MZ) and dizygotic twins. However, discordant MZ twins (who are assumed to share 100% of their genome) can also be used to identify epigenetic factors, such as differing levels of DNA methylation at specific loci that may contribute to explain part of their phenotypic discordance.et al.16 successfully replicated multiple epigenetic features of depression between brain and non-brain tissues, suggesting that peripheral blood cells are adequate for investigating the epigenetics of depression.16 Studies using peripheral blood cells may also have an advantage in that they may show fewer changes associated with disease-related external factors, including exposures to stress and pharmacological treatments.17 On the other hand, it is likely that disease-relevant epigenetic variations may be considerably weaker in peripheral blood cells than in the neural tissue of interest or even be undetectable.The genome (the DNA sequence) is consistent throughout all cell types in the human body: hence, buccal epithelial cells are largely used in genetic research because of their convenience. However, epigenetic changes are more tissue-specific. In the field of psychiatry, this presents a problem as, although neural tissues of interest can be obtained post-mortem, in live patients, only cells in peripheral tissues can generally be obtained. A recent study by Oh et al.16 analyzed epigenetic variations across different tissues in three data sets of discordant MZ twin pairs from three centers in different countries in order to identify epigenetic markers associated with depression.16 However, when they combined the data sets collected from blood, they were not able to discriminate affected siblings. This study used the same three previously published peripheral blood data sets from MZ twins discordant for MDD, aiming to address some of the statistical challenges of this data set to produce a global classification model. This required several data consolidation issues to be addressed. First, batch effects may lead to the data not being independently and identically distributed (non-i.i.d.); this refers to error introduced by systemic differences in the ways samples are taken and analyzed between different laboratories. The second refers to population stratification and noise variance. This study aims to explore methods to combine these data sets, adjusting for sources of error and then use the increased power of the combined data to classify affected and non-affected individuals according to epigenetic markers from peripheral blood samples of MZ twins discordant for MDD.The study by Oh Merging raw data often violates independent-and-identically distributed (i.i.d.) assumptions and thus leads to biased model parameter estimations, higher generalization error and lower cross-validation model performance. The study first used an atheoretical, data-driven approach to evaluated methods for batch effect removal and non-i.i.d. data consolidation. In the second part of the study, we used two different machine-learning approaches (linear support vector machines (SVMs) and random forest (RF)) in order to build a model that could classify affected siblings from non-affected ones using the merged data set from the three, independent, cross-country, epigenetic studies described below. Levels of methylation were used as features and affected and non-affected status within each discordant twin pair as outcome labels. Training and testing of the algorithms was performed on independent samples (80:20% split). Feature selection methods for each algorithm were chosen based on cross-validation. Finally, several different thresholds for the number of selected features were evaluated to extract epigenetics markers likely to contribute to variation in the pathology. These features were then carried forward for gene-network analysis.et al., and are available from the Gene Expression Omnibus (accession number GSE37579). Briefly, methylation profiles were obtained using Illumina 8.1-K CpG island microarrays from white blood cells sampled from MZ twins discordant for MDD. The samples originated from three different regions: the United Kingdom (28 samples), Australia (80 samples) and the Netherlands (86 samples). Levels of methylation were measured for either coding or noncoding DNA regions common across the samples. Following quality control procedures, a total of 8448 features with log-normalized levels of methylation for each sample were obtained that were consistent across the three studies. Depending on the source of information, depressive symptoms were evaluated by structured clinical interview. A diagnosis of MDD was determined by the results of these questionnaires according to the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition criteria. Australian participants were drawn from the Australian Twin Registry. Forty pairs were selected for this study, including 31 female pairs. The average age of the twins was 41.2 years, with a s.d. of 11.5 years. Dutch participants were drawn from the Netherlands Twin Register. Forty-three twin pairs were selected for this study, with an average age of 38.4 years and a s.d. of 12.7 years. UK participants were drawn from the St Thomas's Hospital Twin Registry, the Maudsley Hospital (London) Twin Register and from ongoing studies of volunteer twins. Fourteen pairs including two males were selected for this study. The average age of participants was 53.7 years, with a range from 21 to 65 years. This data set has been chosen for several key reasons. First, it is based on MZ discordant twins, which are considered matched for: genetics, age, sex, cohort effects, maternal influences and common environments. By design, MZ studies have more power to detect disease-related epigenetic differences than studies on unrelated individuals. Second, the same Illumina array has been used across the three studies avoiding potential confounds resulting from matching markers on different arrays and artefacts injected by imputation methods. Lastly, Oh et al. reported that, using all the twin samples , they were unable to discriminate affected individuals from control subjects from blood. Given their promising results from other tissue, there was good reason to believe that important information could still be gained from this published data set. Further information on the samples, ethical statements and approval data collection and data pre-processing is described elsewhere.16The data used for this study have been obtained from a recently published study by Oh Empirical non-i.i.d. data consolidation methods, including principal component analysis (PCA) and non-parametric Bayes methods (ComBat), have been evaluated in isolation and in combination to batch effect removal, population stratification and other noise variance. Machine-learning methods and feature selection using linear SVMs and RF were used to build a classifier to predict cases and controls for MDD from epigenetic markers and extract features with higher probability of explaining variations in pathology.Two main approaches of batch effect removal were evaluated in isolation and in combination. The PCA method relies on the idea that the direction with higher variance might relate to noise or population stratification rather than disease. The non-parametric ComBat approach is an empirical Bayes method that aims to adjust for unknown, unmodeled or latent sources of noise and systematic bias. MZ twin studies discordant on any phenotype are intrinsically balanced by nature: in each twin pair, we have an affected and non-affected twin. This balance was preserved in the training and test data sets during the resampling procedure via randomly sampling from twin pairs rather than from the whole data set. Although receiver operating characteristic (ROC) returns generally higher values, in this study accuracy was preferred as a more representative, conservative and honest measure of model performance.et al.,18 and to control for population stratification as described by Price et al.19 This study considered the approaches above, and unwanted variance was removed from each data set before merging data into one larger set as follows.PCA, ComBat and mixture approaches were evaluated to address issues of non-i.i.d. data and control for potential confounding effects. The PCA method relies on the notion that eigenvectors with higher variance relate to subgroup phenotypes as opposed to disease groups. This approach removes unwanted variance by subtracting a matrix achieved via eigenvector decomposition. Removal of unwanted variance could relate to the removal of batch effect as described by the paper by Nielsen i, data were adjusted as follows: U and V contain the top k eigenvectors corresponding to the top k eigenvalues given in W. Y represents gene expression and Wik is a diagonal matrix corresponding to the top k eigenvalues. For the purpose of evaluation we subtracted the matrix related to the most informative principal components ranked by eigenvalue. Therefore, PCA.1 subtracts the matrix related to the first principal component where PCA.2, PCA.3, PCA.4 and PCA.5 sequentially subtracted the matrix related to principle components 2–5.For each region 20 This is an empirical Bayes method aimed to adjust for unknown, unmodeled or latent sources of noise. ComBat adjusts for systematic batch bias common across genes, assuming that batch effect factors often affect many genes in similar ways, similar to increased expression or higher variability. The other benefit of adjusting for systematic bias with ComBat is that it robustly adjusts batch bias for even small batch sizes.21 ComBat is a three-step empirical Bayes method: (1) standardization of the data is achieved using the formula:The second approach was based on the non-parametric ComBat method implemented in the Surrogate Variable Analysis 'sva' R-package available from Bioconductor.αg, βg, γig in a modelwhere N is the total number of samples, m is number of batches, ni is number of samples within a batch i for i=1,…,m, for genes g=1,…,G. Yijg represents the expression value for gene g for sample j from batch i, αg is the overall gene expression, X is a design matrix for sample conditions and βg is the vector of regression coefficients corresponding to X. The error terms, γig and δig represent the additive and multiplicative batch effects of batch i for gene g, respectively.where Zijg~N are normally distributed, the non-parametric estimates are:(2) Batch effect parameters are estimate batch using empirical priors. Assuming that standardized data where (3) The method adjusts the data for batch effectsusing estimated batch effects Lastly, we evaluated mixture approaches by combining PCA with empirical Bayes method: first, we removed unwanted variance from irrelevant factors and then tried to identify remaining noise and systematic bias using non-parametric Bayes method. Different classification algorithms were subsequently used on the different combinations.T-statistic: The absolute value of Welsh's t-test statistic value. In case of a two-class classification problem, t-statistic is equivalent to the difference between two means for each class adjusted for s.d. for both classes.Mean difference: The absolute value of the difference between average values in each class.P-value for Welsh's t-test with degrees of freedom.Correlation: Absolute value of Pearson correlation between a variable and the predicting factor.The package ‘gene-filter' for R, available from Bioconductor, has been used to apply the above filtering criteria.P-value Limma: The P-value calculated for the single-variable regression model using Limma. Bioconductor limma R package has been used for computing this score.RankProd: RankProd modifies and extends the rank product method proposed by Breitling et al. to integrate multiple studies from different laboratories and/or platforms. In case of pairwise ratios, for each gene it calculates feature rank for each comparison and returns—in this case, a log ratio of methylation levels in twins pair. Rank product is a geometric mean of ranks under different comparisons. The method has been implemented using the 'RankProd' R package.Several filtering scores were compared in order to select the best-performing method for each classification algorithm. Average classification accuracy and its s.d. were obtained from 200 random, independent resamplings. For each resampling step, the data set was split 80/20 into training and test sets, respectively, and the features with the highest scores on the training set were selected. The model was then built on the training set and performance was evaluated on the test set. The following filtering scores have been compared and the best-performing methods for support vector machine and RF classification algorithms were carried forward for the purposes of analysis:27 SVM and RF have parameters that need to be tuned. The linear SVM was implemented using a soft margin parameter C. RF has a number of variables randomly sampled as candidates at each split and used as parameter. The tuning and selection of parameters is required for both classifiers.Two machine-learning classification approaches differing in several key aspects, namely linear SVMs and RF, were used to evaluate the effectiveness of the methods chosen to remove batch and population stratification effects and to classify affected from non-affected siblings based on methylation variation.ℓ2 norm classification methods, SVM is sensitive to noise coming from all variables used. Selecting relevant features and decreasing the number of features aids noise reduction. The linear SVM model used has a soft margin parameter C, which is responsible for misclassification penalties. Higher values of parameter C forces the SVM to a higher penalty of misclassification on a training set that could lead to a higher generalization error if the final performance is only marginally above chance. Lower values of C could lead to lower generalization error. A soft margin parameter of C=0.05 was chosen following cross-validation.Linear SVM is a powerful method with an efficient computation time for a large number of variables and a small number of samples. As many other P-value) were evaluated and the ratio=0.1 was chosen as it returned higher average accuracy over 200 resampling.RF has an embedded control for sensitivity to noise, but it is computationally inefficient for a large number of variables. Feature selection aids computation efficiency with additional noise reduction achieved by removal of irrelevant features, and therefore it is particularly important for large data sets using RF. RF has a tuning parameter, which is effectively the ratio of the number of variables randomly sampled as candidates at each split over the total number of variables. Control over the number of variables at each split aims to avoid correlation between trees in a forest. Prediction of a single tree is highly sensitive to noise in a training set, whereas average prediction across many trees is less sensitive if trees are not correlated. Different ratios for the RF model .NormScale—the mean and s.d. normalized raw data before merging .Average classification accuracy as a measure of classification performance and its s.d. was obtained from 200 random independent resamples. At each resampling step, the data set was split into a training and a test set using a 80/20 ratio; features with the highest scores on the training set were selected. A model was built on the training set and performance evaluated on the independent test set. Consolidated data set were compared against the following unadjusted data sets:The mean normalization and scaling aims to remove inequality between data sets in cases where for each epigenetic marker relative values between samples remain similar across different data sources, whereas the absolute level could vary together with variance. The above methods provide a baseline to evaluate the increase in performance following correction methods for violation of non-i.i.d. assumptions.https://portal.genego.com/) was used to explore the intersection between the list of genes mapped to variants uncovered from this study and known pathway maps and networks. Features most likely to be selected following feature reduction methods and that yielded the highest classification accuracy were mapped to known genes using the UHN Microarray Centre's CpG Island instrument (https://www.pmgenomics.ca/cpg). MetaCore scores and prioritizes networks and pathway based on the relevance of the genes uploaded. The gene list returned by the analysis presented may point at genes that show potential interaction, differential expression and may be involved in the pathology. The gene list could also potentially point at possible targets for therapeutic drug discovery. MetaCore evaluates the magnitude of the intersection between the reference gene list and the set of genes corresponding to a network module and returns different metrics including P-values and G-scores. P-values are calculated based on hypergeometric distributions, and these are used to establish whether saturation with the genes of interest is higher than random. When exploring signaling cascades, it is possible to evaluate whether a network contains any fragments of well-understood  signaling pathways. The G-score is another metric used by the software, which effectively modifies the Z-score based on the number of the linear canonical pathway fragments contained within the network. A high G-score therefore points at a network highly saturated with reference genes and containing several canonical pathways. In this study we have explored the top-ranking networks by P-value and by G-score.In order to gain further biological insight into the most predictive features, MetaCore (http://www.adamlab.org/epigenetics-of-depression/).All code including R scripts used to generate this analysis are available from the corresponding author on request or can be downloaded online , with and without feature selection, returned overall poor classification accuracy. It was not possible to classify cases above chance level across any of the studies. The worse model performance was obtained in the UK study likely because of the smaller sample size compared with Dutch and Australian studies. The Dutch and Australian studies performed better in comparison with the UK study but were also underpowered and returned accuracy scores around chance levels . We thenPCA.3 and mixture models using PCA3.ComBat showed a steady improvement of performance over raw data set for both linear SVM and RF models with different numbers of selected features. Methods with one more or one less number of removed principal components such as PCA.4 and PCA.2 with or without ComBat methods returned better accuracy for some classifiers, but were not steady across different numbers of selected features and different models using independent training and testing sets  was centered on the c-MYC gene hub  includes 10 reference genes and is centered on the PPARGC1A (human accelerated region 20 (HAR20)) gene hub (C-JUN). c-Jun, in combination with c-Fos, forms the AP-1 early-response transcription factor. Activation is dependent on double phosphorylation by the c-Jun N-terminal kinase pathway, which has an important role in initiating inflammatory cellular responses.28A network analysis using MetaCore was peformed in order to gain additional insight into the potential relationship between the feature-set uncovered from the above analysis. The top-ranking genes that could be mapped to probes on the 8.1-K array with higher probability of being selected across the different resampling were uploaded as reference molecule to MetaCore's database. We first explored the top two ranking networks returned by gene hub . C-MYC igene hub . The gengene hub . Interesc-Jun (AP-1), with interactions between c-Jun N-terminal kinase (MAPK) and the X-box-binding protein 1 (XPB-1) gene. The XBP1 gene encodes a key transcription factor in the unfolded protein stress response and has been implicated in the pathophysiology of MDD.29Lastly, we explored the top-ranking network by G-score . The netLandscapes of DNA methylation can show great variation between different tissues, but accessibility to brain tissues in human studies is limited to post-mortem and surgical resection. Therefore, peripheral tissues, including blood, can be an important source of information for the identification of biomarkers and mechanisms associated with the pathology that are assumed to also be manifested in the brain. However, disease-associated epivariations in blood may be less pronounced, more sparsely distributed and complicated to detect because of noise variance compared to disease-affected brain tissues. In this study we combined three non-i.i.d. data sets using a data-driven approach to uncover a suitable method to control for both batch effect and noise variance in order to build a global classifier that could classify affected siblings in MZ pairs discordant for MDD from epivariation in peripheral blood.Of the range of methods evaluated for non-i.i.d. data consolidation, PCA with the top three eigenvector methods and mixture approaches using PCA and ComBat showed steady positive results on our data sets. The resulting model could classify the disease state above chance levels in an independent testing set using differences in methylation levels. Out of several methods considered for classification, the linear support vector machine with the top 100 selected features returned the highest classification accuracy. The accuracy prediction above chance level can be considered important, given the complex molecular architecture underpinning the pathology. Accuracy measures were reported instead of receiver operating characteristic scores, even if these can often be higher. Indeed, high sensitivity and specificity can be achieved even in the absence of high accuracy. Being prudent in quantifying the ability to discriminate discordant twins for MDD from epigenetic markers in peripheral blood alone was preferred to reporting higher ROC scores. This is particularly important in the context of disease classification where it is likely that accumulation of epigenetic changes may explain only a small fraction of individual differences in the pathology. Enrichment analysis of the top-ranking features points at an epigenetic signature of MDD that can be detected in peripheral blood and may be used to inform candidate gene selection in future molecular studies of MDD.PPAR-γ (peroxisome proliferator-activated receptor-gamma), AP-1, XPB-1 and NF-κβ. One of the gene maps uncovered from this study focused around PPAR-γ as the center gene hub  and nuclear factor kappa-light-chain-enhancer of activated B cells (NF-κβ).30AP-1 is a gene that codes for a transcription factor that regulates gene expression in response to cytokines as well as environmental stress and bacterial and viral infections. AP-1 downregulation has been implicated as part of the mechanism by which administration of IFN-alpha therapy induces depression symptoms.31NF-κβ is a transcription regulator that has a role in peripheral inflammation with both pro- and anti-inflammatory effects.33The results from this study support previous human and animal studies on MDD that have uncovered genes centered around a stress-response cascade involving the activator protein 1 PPAR-γ gene that codes for the glitazone receptor (NR1C3). Activation of the PPAR-γ system in the central nervous system is thought to decrease parainflammation, endoplasmic reticulum (ER) stress, formation of reactive oxygen species and glutamate toxicity while increasing neurogenesis and neuroplasticity.34 It has been shown that activation of PPAR-γ leads to an increased neurogenesis as well as antidepressant effects in rodent models.36 While the mechanisms by which this leads to an antidepressant effect are unclear, it has been shown that exposure to stress is associated with decreased hippocampal neurogenesis. Induction of neurogenesis has been shown to be a mechanism of action of several antidepressant medications in animal models.39One of the loci uncovered by the network pathway analysis is the PPAR-γ agonists thiazolidinediones  for the treatment of patients with concomitant MDD and metabolic syndrome or diabetes, as adjunctive therapy in patients with moderate-to-severe MDD in the absence of other metabolic disorders and as monotherapy.43 In addition to their action as insulin-sensitizing agents, these drugs also have anti-inflammatory, neuroprotective and anti-excitotoxic properties.44 Activation of PPAR-γ receptors by their natural (15d-PGJ2) or synthetic ligands has been shown to support neuronal glucose and glutamate metabolism following exposure to stress and to increased levels of neurotropic factors.46 These processes are believed to be dysregulated in MDD, thus making PPAR-γ activation a drug target of interest.Interestingly, several clinical cross-over and randomized control trials have evaluated the efficacy of the insulin-sensitizing XPB-1 as a principal hub  release from the ER, leading to cell apoptosis. XPB-1 activates unfolded protein degradation as well as production of chaperones for protein secretion.34 A polymorphism leading to decreased XPB-1 expression has also been implicated in bipolar disorder.48 Sodium valproate, an anticonvulsant with mood-stabilizing properties, affects the ER-stress response, reducing the likelihood of cell apoptosis. One mechanism of action for this is the activation of ATF6, the gene that activates XPB-1 in the hippocampus and the cerebral cortex.50The top-ranking network by G-score revealed a network with the ipal hub . XPB-1 iDNA methylation is known to exhibit tissue specificity, but human studies are limited to either post-mortem, surgical resection or often surrogate tissues, which include blood cells. The extent to which epivariations in surrogate tissues resemble those in central tissues of interest is unknown. Given the complexity of the pathology, it is likely that many epigenetic variants involved in a number of mechanisms underpinning MDD cannot be detected in blood. On the other hand, blood cells may show fewer epigenetic changes associated with factors associated with the disorder  but not involved in etiology. According to the peripheral inflammation hypothesis of depression, it is systemic inflammation that could potentially explain part of the molecular etiology of the disorder. Many of the inflammatory molecules implicated by this hypothesis are detectable in peripheral blood cells. The most predictive epigenetic markers uncovered by this study were mapped to genes and gene networks with mechanisms associated to inflammation, which is reasonable to expect from blood. It is, however, unclear whether peripheral inflammation is involved in the etiology of depression or is purely driven in response to it.This study was able to control for the non-i.i.d. nature of the data, allowing the integration of three separate data sets, each from a different center and country, overcoming one of the limitations from a previous published study using the same three data sets. In isolation, the studies were too underpowered to allow classification of affected and non-affected siblings, but leveraging the increased power of the larger sample allowed detection of a clear signal. Independent testing and training and careful optimization of parameters to avoid model over-fitting increased the probability of true-positive findings. Although classification was possible above chance levels from peripheral blood, classification accuracy remains generally low. On the other hand it is likely that only a small portion of the phenotypic variance is explained by epigenetic variations. However, this approach allowed us to explore the etiology of MDD beyond gene + environment and gene × environment etiological paradigms.This method of modeling allows the identification of epigenetic loci that account for a portion of the variance in phenotype between MZ twin pairs discordant for depression. Whereas these loci can be identified, this methodology does not give any information as to their role. The construction of networks of epigenetic loci identified from our analysis revealed which epigenes are linked, suggesting pathways and relationships between the proteins these genes code for. The genes implicated by the model can then be corroborated with the literature, shedding light on their role in MDD pathophysiology, although this is inevitably subjected to bias. Furthermore, the majority of the literature does not look at the role of specific genes from the epigenetic perspective; therefore, there is potential for further research involving genes both implicated in the pathophysiology of depression, such as the peripheral inflammation hypothesis, and the epigenetics of depression.C-Myc is a good example of this (Many of the genes identified in the networks of this analysis are yet to be included in any literature regarding the pathophysiology of depression.  of this . It is aThis study employs a cross-sectional design, meaning that it is not possible to infer causality. Accessibility of peripheral blood samples from the same MZ twin pairs over time could allow to identify changes in DNA methylation in relation to time of onset of depression and potentially identify a causal link.


Different versions of sparse LDA have been proposed to address this significant challenge. One implicit assumption of various LDA-based methods is that the covariance matrices are the same across different classes. However, rewiring of genetic networks (therefore different covariance matrices) across different diseases has been observed in many genomics studies, which suggests that LDA and its variations may be suboptimal for disease classifications. However, it is not clear whether considering differing genetic networks across diseases can improve classification in genomics studies.Although Linear Discriminant Analysis (LDA) is commonly used for classification, it may not be directly applied in genomics studies due to the large We propose a sparse version of Quadratic Discriminant Analysis (SQDA) to explicitly consider the differences of the genetic networks across diseases. Both simulation and real data analysis are performed to compare the performance of SQDA with six commonly used classification methods.SQDA provides more accurate classification results than other methods for both simulated and real data. Our method should prove useful for classification in genomics studies and other research settings, where covariances differ among classes. Although Linear Discriminant Analysis (LDA) is commonly used for classification, it cannot be directly applied in the large p, small n setting, where the number of samples is far less than the number of features . This isOne straightforward solution for the matrix singularity problem is to use the generalized inverse, e.g. the Moore-Penrose pseudo-inverse, as mentioned in -4 in derQuadratic discriminant analysis (QDA) is closely related to LDA, where each class is assumed to have its own covariance matrix. This is in contrast to LDA where the covariance matrices for different classes are assumed to be the same. For many practical problems, the assumption of common covariance matrix across different classes is inappropriate. For example, in the case of using gene expression data to distinguish tumor samples from normal controls, or to distinguish tissue types, many studies have shown that the correlation patterns among genes do differ between cancer and normal samples and among tissues. In fact, such rewiring of genetic networks in patients and across tissues are common and sometimes offer valuable information for classifications -19. ThesAlthough many efforts have been made to improve LDA in high-dimensional settings, as far as we are aware, relatively little has been done to improve QDA for the large p, small n problem. This is partly due to the fact that it is already challenging to estimate the covariance or precision matrix in the high-dimensional setting when all the classes are assumed to have the same form for LDA, it will be more difficult to estimate covariances in QDA, since more parameters need to be estimated where each class is allowed to have a different covariance matrix. In this paper, we propose a novel QDA based classification method that estimates separate sparse covariance matrices for different classes, called SQDA, and compare its performance with existing methods, including diagonal linear discriminant analysis (DLDA), diagonal quadratic discriminant analysis (DQDA), regularized discriminant analysis with shrunken centroids (SCRDA), nearest neighbor (NN) , supportFor our proposed method, the covariance matrices for different classes are assumed to be block-diagonal with each block having the same size and variable selection by blocks is performed. For block-diagonal matrices, the block size has to be determined. For variable selection by blocks, the blocks with cross validation errors exceeding the smallest error among all blocks by predefined error margin will be excluded.In this section, we study the effects of block size, error margin, and sample size on our method. We also compare the performance of our method with other classification methods under different simulation settings. The detailed simulations can be found in the Materials and Methods section.N and those in the normal samples are drawn from N, where Σ1=Σ2=I are the identity matrix. We assume that the first 400 elements in μ1 are 0.5 and the rest are 0 while all elements in μ2 are 0. The testing dataset has 500 tumor samples and 500 normal samples.In the first simulation setting, Independent Structure Same Covariance (ISSC), the covariance matrices are the same and diagonal for all classes. We let the number of genes be 10,000 and generate 100 training samples consisting of 50 “tumor” and 50 “normal” samples. The gene expression values in the tumor samples are drawn from Σ1 is changed from the identity matrix with the first 400 diagonal elements replaced by 1.5.In the second simulation setting, Independent Structure Different Covariance (ISDC), the set-up is the same as above except that Σ1 and Σ2 have the same blockwise autoregression correlation structure ΣThe third simulation setting, Dependent Structure Same Covariance (DSSC), puts some dependences on the genes. More specifically, andρ=0.95 and Σρ− has the same structure as Σρ, but with -0.95 for ρ.where Σ1 and Σ2 are the blockwise autoregression correlation structure as in Equation , For each simulation setting, we generate 10 datasets and the misclassification rates are reported as the average of the errors on the 10 testing datasets. The standard deviations of the misclassification rates are also reported together with the number of features selected for classification.We study the effects of two important parameters, block size and error margin, on the performance of SQDA.For the fourth simulation setting DSDC, we compare the performance of SQDA under different block sizes and error margins. The block sizes are varied from 100, 150, 200, 250, 300, 350, to 400, and the error margins are varied from 0.05, 0.10, to 0.15. The average misclassification rates under different combinations of block sizes and error margins are shown in Figure The performance of any classification depends on sample size, which may be especially so for our method, since the number of parameters to be estimated is large and low sample size may lead to unstable results. To study the effect of sample size on our method and other methods, all the parameters are kept the same except that the sample size is varied among 20, 25, 30, 35, 40, 45 and 50 under the fourth simulation setting.The average misclassification rates are shown in Figure In this part, we compare the performance of different methods on the simulated data, where we consider sample size of 50, block size of 100, and error margin of 0.05.It is clear from Table When the true covariance matrix is diagonal and the same for different classes (ISSC), DLDA2 performs the best and is much better than DLDA.When the true covariance matrix is diagonal and different for different classes (ISDC), DQDA2 performs the best and is much better than DQDA.Also, comparing the number of predictors selected by DLDA and DLDA2 or DQDA and DQDA2 suggests variable selection by the blocks method is favorable over variable selection by cross-validation. The number of predictors selected by cross-validation is either too small (50 or 100 for simple underlying covariance structure) or too large (1000+ for complex underlying covariance structure) while the number of predictors selected by variable selection by blocks is closer to the true number of predictors, which is 400 for all scenarios.When the covariance structure is complex (DSSC and DSDC), the performance of SQDA is much better than all the other methods. The relative performance of SQDA with that of DLDA2 and DQDA2 suggests that the sparse estimation of different covariance matrix for different classes can indeed improve sample classification.https://genome-cancer.ucsc.edu), with the level 3 RNAseq data for all cancers log2-transformed and mean normalized across all cancer cohorts. We consider all cancer cohorts with at least 40 normal tissue samples, including Liver Cancer , Colon Cancer , Head and Neck Cancer , Breast Cancer , Lung Adenocarcinoma , Prostate Cancer , Kidney Clear Cell Carcinoma, Lung Squamous Cell Carcinoma  and Thyroid Cancer .In this section, we compare the performance of different classification methods on the TCGA dataset, which we downloaded from the UCSC cancer browser  selected.For real data analysis, we choose the block size to be 100 and error margin to be 0.05 for all datasets, as suggested by simulations. The results are shown in Table In this paper, we have proposed a sparse version of QDA to take into account differing genetic networks across different classes in sample classifications. When the proposed method, SQDA, was compared with six commonly used classification methods on both simulated data and real data, we found that SQDA has good performance, especially when different classes have different covariance matrices.In order to alleviate the intensive computation burden, we have imposed the block-diagonal structure assumption on the covariance matrix. We further assumed that the block size is the same for all blocks. Although this assumption seems too simple to characterize the complex relationships among genes, it does offer a good compromise between the diagonal covariance matrix assumption made in previous methods and the more complex covariance matrix structures that may require much more data to model. The better performance of this approach has been demonstrated in both simulations and, more importantly, the TCGA datasets.In our comparison of SQDA with six other classification methods on simulated and real data, we consider block size of 100 and error margin of 0.05 based on the simulation. In practice, the block size and error margin can be selected using cross-validation. However, due to the small sample size in real data, tuning too many parameters using cross-validation may lead to more variations in the results. As sample size increases, cross validation-based or other methods may be used to select block size and error margin to improve the performance of our method.t statistics and using the same size for each block, a data-driven way of determining the blocks might be better. For example, as suggested in [Instead of determining the blocks based on two sample ested in , hierarcWe have considered binary classification for both simulations and real data analysis. We note that SQDA can be easily extended to multi-class classification problems.http://cran.r-project.org).In summary, we have proposed a sparse version of QDA, which has better or similar performance than commonly used classification methods based on both simulated data and real data. We believe SQDA will prove useful for classification in genomics studies and other research settings, where covariances differ among classes. A R package, SQDA, can be used to perform sparse quadratic discriminant data analysis and is freely available on CRAN  is estimated by the diagonal common sample covariance matrix, that is In DLDA, the covariance matrix (Σk) is estimated by the diagonal sample covariance matrix based on the samples belonging to class k.In DQDA, the covariance matrix for each class  need to be tuned based on cross-validation. If there are several pairs of  having the same cross-validation error, we use the “MIN-MIN” rule mentioned in [rda.In practice,  or the F statistics (for multi-class classification problem).We use multiple gene expression data to compare the performance of different classification methods. For gene expression datasets, the number of genes is usually on the scale of thousands while the number of samples is on the scale of tens. Thus, it may be better to perform gene pre-screening to improve classification performance. In this paper, we use the R package For DLDA, DQDA, NN, and SVM, we use five-fold cross validation based on the training sample to pick the number of top genes used for prediction ranging from 50 to 2000 with step size 50.For SCRDA and RF, since they can perform variable selection and classification simultaneously, no variable selection step is performed.λ to be 0.2 and calculate cross-validation error for each block and then we choose the blocks with cross validation errors less than error margin + the smallest cross-validation error, where error margin can be any number from 0 to 0.5 and usually 0.05, 0.10 and 0.15 are used.For our method, SQDA, we do variable selection by blocks. We set the sparsity parameter For DLDA and DQDA, we also perform variable selection by blocks, leading to DLDA2 and DQDA2, that is to use the same procedure as our method to do variable selection except that the sparse estimation of covariance matrix is replaced by diagonalized estimators for covariance matrix.


Genomic alterations affecting drug target proteins occur in several tumor types and are prime candidates for patient-specific tailored treatments. Increasingly, patients likely to benefit from targeted cancer therapy are selected based on molecular alterations. The selection of a precision therapy benefiting most patients is challenging but can be enhanced with integration of multiple types of molecular data. Data integration approaches for drug prioritization have successfully integrated diverse molecular data but do not take full advantage of existing data and literature.We have built a knowledge-base which connects data from public databases with molecular results from over 2200 tumors, signaling pathways and drug-target databases. Moreover, we have developed a data mining algorithm to effectively utilize this heterogeneous knowledge-base. Our algorithm is designed to facilitate retargeting of existing drugs by stratifying samples and prioritizing drug targets. We analyzed 797 primary tumors from The Cancer Genome Atlas breast and ovarian cancer cohorts using our framework. FGFR, CDK and HER2 inhibitors were prioritized in breast and ovarian data sets. Estrogen receptor positive breast tumors showed potential sensitivity to targeted inhibitors of FGFR due to activation of FGFR3.http://csblcanges.fimm.fi/GOPredict/.Our results suggest that computational sample stratification selects potentially sensitive samples for targeted therapies and can aid in precision medicine drug repositioning. Source code is available from The online version of this article (doi:10.1186/s13040-016-0097-1) contains supplementary material, which is available to authorized users. Finding the right drug for the right patient is an integral part of precision medicine and computational methods to facilitate matching patients to drugs are urgently needed . PatientThe molecular landscape of a tumor affects the efficacy of several drugs and is central for clinical trial design for targeted therapies . In partThe large variety of molecular level alterations in cancers calls for computational data integration methods to enable precision medicine via improved patient stratification and drug repositioning –12. MostWe introduce here a computational algorithm called GOPredict that allows patient stratification and drug repositioning via comprehensive integration of genomics data, signaling pathway information, drug target databases and curated knowledge in databases. We demonstrate the utility of GOPredict by stratifying Cancer Genome Atlas (TCGA) breast and ovarian cancer samples and prioritizing drugs in these two cohorts –19.Our data integration approach consists of two major steps. First, we have developed a knowledge-base that contains molecular, drug information and analysis results from multiple public databases and private sources. Second, we have developed an algorithm (GOPredict) to mine the knowledge-base. An overall schematic of the approach is given in Fig. The major design principles of the knowledge-base are as follows. First, the knowledge-base is gene-centric (meaning that the information in the knowledge-base is associated to a gene identifier) because this allows taking into account published results that are mostly gene-centric. This design also allows automated analysis of drug-gene target-pairs. Second, results in the knowledge-base are stored as ranks. The use of ranks enables comparing and combining data over multiple data sets, data types, and measurement technologies as well as between numerical, ordinal and categorical data . In addiGOPredict uses signaling pathway information defined with Gene Ontology (GO) biological processes . The GenThe underlying modeling question for GOPredict is “what is the best drug to target proteins affected by genomic aberrations and driving tumorigenic signaling in tumors”. The GOPredict algorithm is described in detail in Additional file in-house data and curated data. In-house data comprise raw data that we downloaded and analyzed. Curated data comprise analyzed gene-level result data that we downloaded from the source databases and did not process further. A short description of studies is in Table We gathered results of genomic, transcriptomic, and epigenomic (DNA methylation), and descriptive (gene-phenotype connections) analysis data from nine public cancer data sets. Each data set consisted of one or more of these data levels. There are two sources of data which we call The in-house data comprise four Cancer Genome Atlas (TCGA) primary tumor data sets totaling approximately 2,200 samples of breast, ovarian, colorectal and glioblastoma brain cancer –19, 25. studies in the knowledge-base. A study is a ranked list of genes that are ranked based on a statistical analysis of a molecular data type in a specific cancer or literature source. All studies are stored gene-wise and for each gene the database contains its rank order in each study . Full dactivating , inactivating , or survival-associated . A gene may belong to one or several studies and study sets. A full list of studies in each default study set is in Additional file Studies can be combined into study sets. Users can tailor and modify study sets flexibly to suit different research questions. We provide three default study sets: http://geneontology.org/, downloaded August 2013). This compound design enables rapid integration, combination and comparison of data over multiple data sets, data types, and measurement technologies. KEGGDrug, DrugBank and Gene Ontology are stored in the database as described in Additional file In addition to gene ranks in studies, the knowledge-base contains drug gene-target information from KEGGDrug  and DrugThe overall goals of the GOPredict algorithm are to prioritize drugs with known protein targets, characterize genes, and stratify samples. GOPredict works in four steps  has a rank in two studies which are used to calculate its K-rank. The two studies for FGFR3 are a curated study (activating mutations in the Cancer Gene Census) and an in-house study . Each K-rank quantifies the cancer-essentiality of a gene.First, gene ranks are extracted from the database for each study and used to calculate normalized gene ranks called the K-ranks are used to calculate GOPredict cancer-essentiality scores for GO processes . The K-ranks of these genes from step one are summed up to produce the cancer-essentiality score for ‘cell development’ and statistical significance is assessed with a permutation test  and 390 ovarian adenocarcinoma (OVCA) tumors from the Cancer Genome Atlas , 19. We A byproduct of the knowledge-base design is that it allows defining hypothesis-driven selection of study sets and calculating cancer-essentiality in the study sets. A full list of studies in each study set is in Additional file K-rank. The genes, which GOPredict characterized to be cancer-essential, include known cancer genes such as EGFR, ERBB2 and FGFR3, tumor suppressors such as RB1, TP53 and PTEN as well as genes not previously associated with cancer  are ranked the highest in the entire sample set , all FGFR family members (FGFR1-4) and possible confounders . In addition, the immunohistochemical staining status of estrogen receptor, progesterone receptor and HER2 receptor are shown. Samples sensitive to the three drugs were assigned almost exclusively according to FGFR3 activation status . The proportion of sensitive samples in these three subtypes differed significantly from luminal A (Fisher’s exact test P=0.0006) and luminal B proportions (Fisher’s exact test P=0.00001). In addition, FGFR3 inhibitor sensitive samples were enriched in luminal B samples when compared directly with luminal A (Fisher’s exact test P=0.004). In summary, luminal subtypes in general and preferentially luminal B breast cancer samples were significantly enriched for FGFR inhibitor sensitive samples according to activity patterns of FGFR inhibitor targets.To further characterize the sensitive samples, we compared GOPredict’s strata to the PAM50 subtypes. PAM50 is a gene expression based molecular subtyping method for breast cancer and is well established . FGFR inIn addition to breast cancer, we tested GOPredict in an independent set of ovarian adenocarcinoma primary tumors. In OVCA . Many of our predicted sensitive samples harbored genomic alterations in FGFRs. One of the first clinical breast cancer studies where the selection of patients was based on FGFR1 amplification status, found dovitinib to reduce tumor size more in FGFR1 amplified than non-amplified patients [Five of the top ten prioritized drugs for breast cancer were FGFR inhibitors. All five are in Phase 2 or 3 trials for multiple cancers  and pazopatients .FGFR3 activated and were enriched for PAM50 luminal A and B breast cancer subtypes. Luminal breast cancers are characterized by estrogen receptor (ER) positivity [FGFR4 predicts poor response to tamoxifen therapy in primary tumors [The samples predicted to be FGFR inhibitor sensitive were almost exclusively sitivity . Tamoxifsitivity . Interessitivity  and highy tumors . FurtherTamoxifen resistant breast tumors have been found to be sensitive to vismodegib (Smoothened antagonist) in xenograft mice . In our FGFR3. According to current guidelines, HER2 positive patients are pharmacologically treated with HER2 inhibitors [In our breast cancer data, roughly 20 % of HER2 positive tumors had potentially activating alterations in hibitors . Our resAmiloride was the highest ranked drug in HER2 positive tumors. Amiloride is a pyrazine compound used to treat hypertension and heart failure. Interestingly, amiloride and its derivatives have been recently suggested to have anti-cancer effects in breast cancer cells independent of subtype , 54. An CDK4 in ovarian cancer [https://clinicaltrials.gov/, Accessed 25 Jan 2015). Moreover, dinaciclib, the highest scoring CDK inhibitor by GOPredict, has been shown to sensitize ovarian cancer cell lines to platinum drugs via downregulation of BRCA1 [In ovarian cancer, seven out of ten top priority drugs are cyclin-dependent kinase (CDK) inhibitors. CDKs are a family of protein kinases that participate in the cell cycle and are targeted by several inhibitors . Severaln cancer . To daten cancer  and of tof BRCA1 , 60. TheOur knowledge-base contains both in-house and curated microarray data sets from multiple microarray platforms and sources. Since GOPredict is designed to be extendable and to contain data sets from multiple sources, preprocessing steps such as normalization cannot be fully standardized and can therefore induce some noise. Nonetheless, our results with varying study sets indicate that GOPredict scoring is robust to noise in the study data.GOPredict is dependent on centralized and frequently maintained databases such as GO and Ensembl. Many databases, however, undergo changes in both data content and database interfaces. These changes increase the maintenance burden of tools such as GOPredict. GOPredict could be improved in the future in three ways: (1) the addition of binding-affinity of drug-target pairs to weight each drug target gene; (2) inclusion of data on drug combinations and synthetic lethal interactions and; (3) addition of more result databases. Data on the first two points are currently scattered and automated retrieval is infeasible. GOPredict utilizes many result databases but this list is incomplete. For example, the Comparative Toxicogenomics Database (CTD) contains special disease related GO annotations as well as breast and ovarian cancer marker gene studies which could be incorporated into GOPredict .GOPredict results can in future work guide experimental design. For example, top scoring drugs from our GOPredict analysis could be administered to breast cells with suitable genetic profiles to test their efficacy in vitro.SLC25A32 received high cancer-essentiality scores through alterations in both ovarian and breast cancer study sets which could indicate a role for SLC25A32 in these cancers. Accordingly, we built a multivariate Cox survival model in TCGA OVCA data and found that the overexpression of SLC25A32 (ANOVA P=0.003) and lack of residual tumor (ANOVA P=0.02) were significant independent predictors of poor survival .GOPredict produces several by-product results when prioritizing drugs. For example, Here we present GOPredict, which is a novel approach to integrate information from multiple public sources, signaling pathway and drug target data with local genomic data in cancer. Our results suggest that GOPredict can augment current pathology-based definitions of patient groups for targeted drug therapies which can potentially benefit many cancer patients. A practical application for GOPredict is to screen genomic measurements of cancer model systems for previously overlooked druggable genomic alterations and simultaneously prioritize which drugs to screen.Our approach is able to infer kinase inhibitors as highly relevant drugs in breast and ovarian cancer based solely on signaling pathway information, pre-existing genomic result data and molecular measurement data. These inhibitors are prime candidates for further testing in drug repositioning experiments. Furthermore, our results highlight none-cancer drugs such as amiloride which have only recently been tested for anti-cancer efficacy with promising results. Our primary results indicate that FGFR inhibitors in breast cancer and CDK inhibitors in ovarian cancer as well as pazopanib in both cancers are predicted to have the largest proportion of putatively sensitive samples.


Although numerous investigations have compared gene expression microarray platforms, preprocessing methods and batch correction algorithms using constructed spike-in or dilution datasets, there remains a paucity of studies examining the properties of microarray data using diverse biological samples. Most microarray experiments seek to identify subtle differences between samples with variable background noise, a scenario poorly represented by constructed datasets. Thus, microarray users lack important information regarding the complexities introduced in real-world experimental settings. The recent development of a multiplexed, digital technology for nucleic acid measurement enables counting of individual RNA molecules without amplification and, for the first time, permits such a study.Using a set of human leukocyte subset RNA samples, we compared previously acquired microarray expression values with RNA molecule counts determined by the nCounter Analysis System (NanoString Technologies) in selected genes. We found that gene measurements across samples correlated well between the two platforms, particularly for high-variance genes, while genes deemed unexpressed by the nCounter generally had both low expression and low variance on the microarray. Confirming previous findings from spike-in and dilution datasets, this “gold-standard” comparison demonstrated signal compression that varied dramatically by expression level and, to a lesser extent, by dataset. Most importantly, examination of three different cell types revealed that noise levels differed across tissues.Microarray measurements generally correlate with relative RNA molecule counts within optimal ranges but suffer from expression-dependent accuracy bias and precision that varies across datasets. We urge microarray users to consider expression-level effects in signal interpretation and to evaluate noise properties in each dataset independently.The online version of this article (doi:10.1186/1471-2164-15-649) contains supplementary material, which is available to authorized users. Over the last decade, gene expression microarrays have become a common tool for examining gene transcript levels in hypothesis-free investigations. Microarray data is used for a wide variety of analyses, such as unsupervised clustering, classification, differential expression analysis, and expression quantitative trait loci mapping (as reviewed in ). These Microarray data preprocessing and batch correction are important strategies for minimizing such confounding. Preprocessing consists of three steps: background correction, normalization, and summarization; and the choice of methods for these steps can dramatically change experiment results . PreprocThese data processing methods enable comparisons of individual genes between samples, but they leave direct interpretation of the normalized values somewhat ambiguous. This is particularly problematic for genes with low log-transformed expression values, where it can be difficult to differentiate true expression from background noise. After quantile normalization there is no direct calculation of the real expression difference indicated by a log-fold-change in microarray expression values. Common validation technologies, such as quantitative polymerase chain reaction  and mulPrevious studies with spike-in and dilution datasets , 17, 18 Here we have used a count-based transcript detection technology to address some of the questions raised above, comparing measurements from the Affymetrix Human Gene 1.1 ST microarray with those from the NanoString Technologies nCounter Analysis System . In contThis study was approved by the Cambridgeshire 3 Research Ethics Committee (08/H0306/21) and all individuals provided written informed consent. Samples were from healthy controls and individuals with active, untreated disease with the following diagnoses: IBD, specifically Crohn’s disease (CD) and ulcerative colitis (UC); and AAV, specifically granulomatosis with polyangiitis (GPA) and microscopic polyangiitis (MPA). Whole blood was collected and separated into peripheral blood leukocyte subsets as previously described –29. BrieAliquots of 200 ng total RNA were amplified and labelled for Human Gene 1.1 ST 96-Array (Affymetrix) using the Ambion WT Expression Kit and GeneChip WT Terminal Labeling and Controls Kit (Affymetrix), according to the manufacturer’s protocols. Samples were run on a GeneTitan Multi-Channel (MC) Instrument (Affymetrix) as part of a larger dataset acquired over multiple years. For comparison with nCounter data, we selected only batches with at least 10 samples from the desired cohorts covering at least two different diagnoses to ensure batch correction was resistant to outliers and confounding structure. Microarray data for samples used in the nCounter comparison have been deposited in ArrayExpress with Accession Number E-MTAB-2452.Gene expression microarrays were filtered for sex discordance and global dimness before data processing. Because the robustness of microarray normalization improves with the number of samples included, arrays were normalized in large, cell-type-specific batches, including all available samples from the selected batches with diagnoses tested in this study , according to the manufacturer’s protocol in a total of 6 runs over 8 days. To test for RNA degradation, several sentinel samples from each cell-type-specific dataset were examined by 2100 Bioanalyzer  according to the manufacturer’s protocol. All sentinel samples were confirmed to be of good quality with RIN ≥ 9.5. One sample was duplicated in a separate nCounter run to evaluate reproducibility and run-specific effects  for quality control. nCounter data was normalized for hybridization and counting efficiency in cell-type specific groups. Each sample was multiplicatively normalized to have the same geometric mean of nCounter-provided positive control probe counts. No normalization factors were outside the NanoString-recommended range of 0.3-3. Thresholds to identify expressed genes from background noise were then calculated as the median of the maxima of the negative control probe measurements for each cell type.Microarray probesets mapping to genes measured by the nCounter Analysis System were identified by Affymetrix GeneChip Human Gene ST Arrays Probeset Annotations Release 33.2. Where multiple probesets were available for a given gene, the probeset with the best target region overlap with the nCounter target region was chosen. Where all probesets were equivalent in overlap, one was chosen at random  all genes called unexpressed by the nCounter; and 2) all genes called invariant across samples by the nCounter  but strictly unsaturated on the microarray Additional file 1::Mean v variance. Binned mean and variance characteristics of microarray datasets: For each gene, a microarray expression value mean and mean within-batch variance was calculated. Genes were then binned by expression value means, and statistics were averaged to achieve an average mean expression value and average mean within-batch-variance for each bin. These two values are plotted. (PDF 64 KB)Additional file 2::Control genes. Microarray properties of nCounter control genes: Details of each cell-type-specific control gene used. (PDF 40 KB)Additional file 3::nCounter run comparison. Examination of inter-run technical effects of nCounter data: A) Log-transformed raw nCounter counts for technical replicates of the same sample are plotted with Pearson correlation indicated. All genes determined to be globally expressed in CD4 samples are included. B) Boxplots depict Pearson correlations between log-transformed raw nCounter counts for samples of the same and different diagnoses in the same and in different nCounter runs. All genes determined to be globally expressed in the designated cell types are included. In the bottom panel, the outlying CD14 sample has been removed. C) Inter-sample Pearson correlation coefficients of log-transformed raw nCounter counts between all CD14 samples. Red star indicates outlier. (PDF 131 KB)Additional file 4::nCounter probes. nCounter probe details and mapped Affymetrix Hugene 1.1 ST array probesets: nCounter probe design schemes, isoform coverage, and microarray probeset mappings are tabulated. (PDF 96 KB)Additional file 5::nCounter samples. Sample composition: Details are provided for samples run on the nCounter analysis system. (PDF 91 KB)Additional file 6::Correlation comparison. Effects of microarray and nCounter processing on inter-platform correlation: Cell-type-specific nCounter datasets were normalized to the indicated control genes and log-transformed. Microarray data were preprocessed by RMA and then batch normalized through ComBat and/or normalized to control genes where indicated. Boxplots show Pearson correlation. (PDF 76 KB)Additional file 7::Correlation comparison table. Effects of microarray and nCounter processing on inter-platform correlation: Table summarizes inter-platform correlation of datasets using different processing and normalization procedures. (PDF 66 KB)Additional file 8::Microarray batch effects. Batch effects in microarray datasets: A) Samples from full CD4 and CD14 microarray datasets are plotted by first and second principle components before and after ComBat batch correction. Color indicates batch membership. B) Pearson correlation of expressed genes across samples in nCounter versus RMA-preprocessed microarray datasets was subtracted from the same correlation in nCounter versus RMA-preprocessed and ComBat-corrected microarray datasets. Boxplots depict these differences in CD4 and CD14 datasets to indicate the effect of batch correction on gene-based platform correlation. (PDF 77 KB)Additional file 9::Batch correction and accuracy. Effect of batch correction on signal detection accuracy: A) Signal detection slope is plotted versus inter-platform correlation as in Figure Additional file 10::Noise v expression value. Comparison of noise versus microarray expression value. A) For each unexpressed gene, the standard deviation of log-ratios of all pairs of samples from RMA + ComBat- (CD4 and CD14) or RMA-preprocessed (CD16) microarray data is plotted versus the gene’s median microarray expression value. B) As (A) for invariant genes. (PDF 81 KB)Additional file 11::Mean expression histograms. Mean expression profiles: Histograms depict mean RMA + ComBat- (CD4 and CD14) or RMA-preprocessed (CD16) microarray expression values from full microarray datasets. (PDF 65 KB)Additional file 12::