There is an ever-expanding range of technologies that generate very large numbers of biomarkers for research and clinical applications. Choosing the most informative biomarkers from a high-dimensional data set, combined with identifying the most reliable and accurate classification algorithms to use with that biomarker set, can be a daunting task. Existing surveys of feature selection and classification algorithms typically focus on a single data type, such as gene expression microarrays, and rarely explore the model's performance across multiple biological data types.This paper presents the results of a large scale empirical study whereby a large number of popular feature selection and classification algorithms are used to identify the tissue of origin for the NCI-60 cancer cell lines. A computational pipeline was implemented to maximize predictive accuracy of all models at all parameters on five different data types available for the NCI-60 cell lines. A validation experiment was conducted using external data in order to demonstrate robustness.As expected, the data type and number of biomarkers have a significant effect on the performance of the predictive models. Although no model or data type uniformly outperforms the others across the entire range of tested numbers of markers, several clear trends are visible. At low numbers of biomarkers gene and protein expression data types are able to differentiate between cancer cell lines significantly better than the other three data types, namely SNP, array comparative genome hybridization (aCGH), and microRNA data.Interestingly, as the number of selected biomarkers increases best performing classifiers based on SNP data match or slightly outperform those based on gene and protein expression, while those based on aCGH and microRNA data continue to perform the worst. It is observed that one class of feature selection and classifier are consistently top performers across data types and number of markers, suggesting that well performing feature-selection/classifier pairings are likely to be robust in biological classification problems regardless of the data type used in the analysis. Due to the recent rise of big-data in biology, predictive models based on small panels of biomarkers are becoming increasingly important in clinical, translational and basic biomedical research. In clinical applications such predictive models are increasingly being used for diagnosis , patientMany types of biological data can be used to identify informative biomarker panels. Common ones include microarray based gene expression, microRNA, genomic copy number, and SNP data, but the rise of new technologies including high-throughput transcriptome sequencing (RNA-Seq) and mass spectrometry will continue to increase the diversity of biomarker types readily available for biomarker mining.feature selection, to identify a short list of informative biomarkers, and classification, used to make predictions for new samples based on patterns extracted from the training data. Both of these steps have been explored extensively in the statistics and machine learning literature, and many alternative algorithms are available for each. Due to the sheer number of available choices and the lack of predictable interactions between feature selection method, classification algorithm, and data type, assembling the most robust biomarker assay for a given biomedical application is rarely undertaken systematically. Rather, it is more often driven by the intuition and a priori preferences of the statistician.Useful predictive models are typically restricted to use a small number of biomarkers that can be cost-effectively assayed in the lab . The useAvailable feature selection methods can be grouped into three broad categories: filter, wrapper and embedded. Filtering approaches use an easy to calculate metric which allows quick ranking of the features, with top ranking features being selected. Wrapper methods use a classification algorithm to interrogate the effect of various biomarker subsets. Embedded approaches are classification algorithms which eliminate features as part of the training process. Recent studies -8 investThere are hundreds of publications describing classification algorithms and their applications to genetic research and medicine. Many publications advocating a new method employ a limited comparison between similar approaches. However non-uniform validation strategies make it difficult to assess performance of a wide variety of approaches. A previous study compared both classification and feature selection approaches in a unified framework , howeverThis work presents a large scale empirical comparison of the effects of the interaction between the main components of the predictive model , the number of features utilized, and the underlying data type on the performance of the overall model. This study also implements exhaustive parametrization of all models to ensure a fair comparison between models.In order to test the performance of the large number of models tested in this study, and in order to be able to run direct comparisons of the models on different biological data types, we took advantage of the publicly available NCI-60 cancer cell line data set . The NCICancers are widely believed to derive from a single event in which one cell escapes the many surveillance mechanisms in place to prevent uncontrolled proliferation. Once this has occurred, the cancer often evolves quickly, rapidly acquiring large numbers of mutations, ranging from small point mutations to very large chromosomal aberrations and regional amplifications (DNA duplications). The original identity of the cancer cell (its cell type or tissue type) appears to exert a very strong influence on the course of evolution of the cancer. For this reason, characteristic mutations will often be found in cancers derived from the same tissue, even in different patients. In addition, because identical cell types from different patients will share very similar gene expression signatures, cancers derived from these tissues will often do the same. In the present study we take advantage of these two features of cancer to test the ability of various statistical models to correctly infer the cell type (or "tissue-of-origin") of each cancer cell line. The ability to make this inference correctly not only represents an excellent test of these models on real biological data, it is a good example of the type of classification ability required for targeted cancer therapeutics.In order to test the predictive models in this study we use publicly available data from the NCI-60 cancer cell lines as provided by CellMiner . For theThe area of feature selection in machine learning has recently been quite robust. There are numerous specialized feature selection algorithms which identify the most informative biomarkers from high-dimensional data. This study utilized at least one approach from each of the three broad categories identified above . Every approach utilized allowed for a specific number of features to be chosen. No requirement was established that induced a relationship between feature sets from the same algorithm. So the 16 features chosen by one approach are not required to be a subset of the 32 features chosen by the same. For all algorithms we used the implementations in the Scikit-learn  Python pThe fastest and most simplistic selection method is univariate filtering. These approaches rank features according to some score, and the user selects the best k features accordingly. Here the F-statistic (Anova), a generalization of the t-test, is used as a filter, as suggested in  and 6]..6]. TherWrapper approaches typically use some type of greedy strategy to select influential features using a black box classifier. They are more computationally intensive, however SVM recursive feature elimination (SVM-RFE) is extensively used in medical applications . The parThe final class of feature selection algorithms is embedded approaches where the features are chosen while building the classifier. To represent this class two tree-based methods were adapted; random forest (RF)  and extrA summary of parameters of all considered feature selection methods along with the range of values searched for each parameter are given in Supplemental Table S3.An exhaustive comparison of all classification algorithms would be quite challenging. Therefore only a small number of approaches was explored, chosen to represent most common machine learning approaches used in bioinformatics. Identifying the cancer type from the NCI-60 dataset is inherently a multi-category classification problem. Therefore each considered approach must accommodate this setting or be adaptable by one-vs-one  or equivLinear classifiers use a linear function to score classes by taking the dot product of feature values and feature weights computed during training. One of the most powerful, flexible and ubiquitous linear classifier is the support vector machine (SVM) with linear kernel . SVM hasClassification trees are a machine learning tool which has found extensive use in the biological and medical communities. This is partially due to both their resilience to over-fitting and ease of interpretation. This work looks at three related approaches; vanilla decision trees (DT) , random k  closest matches.Distance based methods surveyed are k-nearest neighbors (KNN), cosine (Cos) and correlation (Corr). Cosine and correlation are simple classifiers which calculate the distance to all training samples from the test sample and assigns the label based on the closest match. KNN is a slightly more advanced version of the same concept in which class membership is assigned by majority voting among the A summary of parameters of all considered classification algorithms along with the range of values searched for each parameter are given in Supplemental Table S4.k-fold cross-validation [k equal size subsets with each set used in turn for testing while the other k − 1 subsets are used as training data. Care must be taken taken to avoid substantial biases [k-fold cross-validation is used to tune parameter values. This requires an additional cross-validation experiment on each training dataset, where a grid-search over the considered parameter range is performed. The inner phase identifies the best parameter values which are then used exclusively in the outer cross-validation. In order to build stronger evidence for the models' performance, the outer cross-validation phase was repeated 100 times, however the parameterization was only performed in the first iteration. Biases towards selecting more complex models with more parameters or overly fine grid-steps are still a possibility, however nested cross-validation should largely mitigate them. More advanced techniques presented in [A common validation strategy used in evaluating machine-learning methods is lidation ,8. Here l biases  by ensurented in  could bek-fold cross-validation strategy is computationally very intensive. With 4 × 9 = 36 models (combinations of feature selection and classifier) to evaluate, dozens of parameter values and different number of selected markers there can be upwards of 1,000,000 individual classifier runs per data type. The majority of the jobs occur in the inner cross-validation loop, and fortunately can all be run in parallel on a cluster or multi-core server. Also, a pre-filtering heuristic was applied to speed up the feature selection process. For all datasets with more than 1,000 features we retained only the top 1,000 features as ranked by the F-statistic prior to any additional feature selection.The nested To further validate the results on external datasets, eight primary tumor cohorts from The Cancer Genome Atlas (TCGA) were identified to match five NCI-60 tissue-of-origin cell lines; central nervous system, colon, non-small cell lung, ovarian, and renal. The mapping of the TCGA cohorts to the NCI-60 cell lines can be found in Supplemental Table 7. The TCGA derived gene expression microarray data was obtained from the Broad Institute's GDAC Firehose utility -34. The area under the receiver operating characteristic (ROC) curve. The ROC curve is a plot of the true positive rate against the false positive rate. The AUC is then the area under this curve and is used as a single measurement of classifier performance. This definition is typically for binary classification tasks, however there are several extensions to multiclass classification problems [AUC(ic) is the typical binary classification AUC for class i cand p(ic) is the prevalence in the data of class ic.There are numerous metrics used in evaluating the accuracy of a predictive model. One common metric is AUC, or problems . Since tThis study is evaluating the effect of three parameters simultaneously: the model, the data type and the number of markers. Therefore conclusions about the best predictive model are presented from the perspective of each parameter individually. In Figure The accuracy of the predictive models varies greatly, with the various combinations of feature selection and classification algorithms. If the feature selection and classification algorithms are grouped by class, a high-level ranking becomes much clearer. In Figure If the data type and number of features are fixed the effects of the models can be explored further. As seen in Figure This conclusion is contrary to that of , where iThe relatively small effect of classifier choice is interesting and unexpected. This indicates that much more care should be given to choosing the right features, as this has the biggest effect on model performance.The rich selection of data types available for these cell lines provides the opportunity to compare the ability of many types of biological data to classify the tissue of origin of a tumor cell line. Some of these data types fundamentally reflect gene expression levels: mRNA, protein and microRNA. The other two: CNV and SNP, are generally assumed to reflect genomic changes at large (CNV) and small (SNP) scales. Comparisons of data type effects at all marker sizes are best seen in Figure ground truth and measure the ability of each data type to correctly infer the tissue of origin of a sample based upon each data type.The transition from normal tissue to cancerous tissue is generally associated with changes at the level of both gene expression and the genome. Frequent mutations, genomic rearrangements and large scale changes in gene expression are all characteristic of oncogenic transformation. However, cancer cells also retain many, if not most, of the essential hallmarks of the tissue of origin of the cancer. In this study, we use the tissue of origin as the A priori, we expect some of these data types to be better at this task than others. For instance, mRNA profiles are highly distinct between different tissue types. For this reason, even after oncogenic transformation, an mRNA transcriptional profile characteristic of the tissue of origin is expected to resemble that of the normal tissue, more than it would the transcriptional profile of tumors derived from other tissues. For this reason, we expect (and find) that mRNA transcriptional profiles reliably and accurately infer the tissue of origin of tumor cell lines. Similarly, protein expression profiles are also very reliable indicators of the tissue of origin of a tumor. microRNA profiles are less powerful than either mRNA or protein expression profiles, but still fairly powerful indicators of tissue of origin. The relative weakness of microRNA profiles compared to mRNA and protein expression profiles may in part result from lower tissue specificity of microRNA expression relative to mRNA and protein.The ability of genomic data to infer the tissue of origin of the tumor is subject to a very different set of biological constraints than expression data. While expression data is expected to be approximately identical across tissues regardless of patient identity, and thus similar between tumors derived from the same tissue but from different individuals; genomic data is identical across normal tissues in an individual, and differs between individuals. Thus, at first glance, genomic data would be expected to track with the individual, and be a very poor predictor of the tissue of origin of a cancer. However, dramatic genomic alterations are a hallmark of cancer progression, and distinct genomic alterations are often found in distinct cancer types. Accordingly, we find that copy number variation is about as powerful as microRNA profiles at inferring the tissue of origin of a cancer cell. This is likely due to the preferential occurrence of specific DNA rearrangements in cancers derived from specific cell types . The SNPAs one uses more biomarkers to classify samples, one expects increased performance, the possibility of overfitting, and the appearance of a plateau beyond which additional markers do not increase the power of classification. However, the rate at which these changes occur as more markers are used to classify a sample can be very different for various types of data.∼0.97).Our analysis shows that mRNA, protein, and SNP data all plateau at about the same AUC  or coverage (protein), and thus power of discrimination. The mRNA arrays used to characterize the NCI-60 cell lines provide direct assessment of the activity of thousands of protein-coding genes, while the protein arrays measure only somewhat more than 300 proteins. With thousands of potential markers to choose from, the mRNA-based models can select informative markers from a larger marker pool, and thus maximize the performance of a gene expression-based model more quickly than the protein arrays, which are restricted to a small subset of the protein coding genes represented on the mRNA arrays. The more direct nature of the protein measurement  does not appear to outweigh the disadvantage of the lower coverage in the starting set of protein markers. As discussed in the preceding section, the SNP array appears to be measuring, in part, gene expression levels resulting from the amplification of specific regions of the genome in specific cancer types. However, there is likely to be a complex and possibly heterogeneous and non-linear relationship between signal intensity on the SNP array, and gene expression levels. Thus, despite the very large number of markers to choose from on the SNP array, highly informative markers are not as abundant in this data as they appear to be in the mRNA data. As a result, many more SNP markers are required to achieve the same level of performance as mRNA-based markers. It is hard to predict how the power of SNPs to infer cancer type might change when newer arrays, that control for copy number changes, are used to characterize these cell lines.Similarly, CNV and microRNA markers approach the same level of performance as one another, but do so at different rates. While microRNA markers plateau quickly (at about 16 markers) CNV markers require 64-96 markers to reach the same level of performance. The quick plateau of microRNA-based markers is likely due to the highly tissue-specific expression of a minority of microRNA's, and the more global expression of the remaining majority. Once the few highly informative microRNA's have been selected and used, adding more provides little additional classification power. In the case of CNV's, like SNP's these markers reflect changes in the cancer cells genome that can lead to changes in gene expression that are distinctive features of cancer subtypes. However, not only do the CNV markers suffer from the indirect relationship between the marker and gene expression expected for SNP's, they are also a much lower resolution marker than SNP's (megabases vs single bases), and far fewer CNV's were measured on the arrays, thus limiting the likelihood that the most informative CNV's were available for selection. Thus, the power of the CNV biomarker panel climbs slowly.Taken together, these observations suggest that the absolute performance of a given biomarker data type to classify a cancer can be understood in the context of: the number of available markers for the model to choose from, the power of the most informative markers in the set, and the directness with which the data type reflects an informative aspect of the sample biology. Data types with a large number of markers to choose from, that are closely related to the biology of the sample, are most likely to yield highly effective small biomarker panels. On another hand, data types with lower saturation (fewer markers measured), and/or a less direct relationship to the biological differences between samples, will require more markers to reach maximal performance.Ultimately all parameters should be considered simultaneously when attempting to build the best targeted predictive model. In order to do this it is necessary to build a validation framework to explore all parameters fairly and efficiently. Although it is a difficult task it is not impractical and interesting nuances can be extracted.In this study it was observed that at the lowest number of markers (8) mRNA and protein were the best data types for cancer identification. For mRNA, SNP and protein the SVM-RFE was the best feature selection choice and ET was the best classifier. For CNV and microRNA the best classifier was LR and ET respectively. Interestingly for all data types at 8 markers except CNV a tree based classifier performed the best as seen in Table At the highest number of markers tests (96) both RFE and ET perform strongly on all data types, however LR is the best classifier for all types except SNP where KNN is the best. Both of these classification tools are technically simple, yet they perform best which lends credence to the Occam's razor principle which when applied to machine learning places preference on simpler explanations.The amount of over-fitting when building a predictive model is always a concern. This effect was measured in an external validation experiment utilizing analogous gene expression microarray data obtained from several studies which are part of the TCGA project -34. The The NCI-60 data is derived from decades old cell lines, while the TCGA data was derived from recently sampled primary solid tumors. Additionally the matched cancer types did not have comparable histological classification. Finally, there were three aditional cancer types  which were present in NCI-60 but not included in the external validation set. These classes were included in the training. Despite these differences the presented method was able to perform biomarker selection and build accurate predictive models for this challenging external validation experiment. A complete breakdown of the per-class prediction rate by cancer marker set size is provided in Supplemental Table 9.The initial hypothesis motivating this research was that certain predictive models will perform better on different data types at different dimensionalities. While this hypothesis holds, the difference in accuracy between models is often small and allows for several generalizations. Namely that RFE is clearly the best feature selection algorithm and both SVM and LR are the best classifiers as seen in Figures NCI-60. National Cancer Institutes tumor cell line project; TCGA. The Cancer Genome Atlas; SVM. Support vector machine; DT. Decision tree; RF. Random forest; ET. Extra tree; GB. Gradient boosting; KNN. K-nearest neighbor; RFE. Recursive feature elimination; LR. Logistic regression; COS. Cosine; COR. Correlation; BR. Breast; CNS. Central nervous system; CO. Colon; LC. Lung cancer; RE. Renal; OV. Ovarian; ME. Melanoma; SNP. Single nucleotide polymorphismIIM, CEN, EH and JL are co-founders and own shares of Smpl Bio LLC a for-profit business created to commercialize the application of the work presented here. Smpl Bio LLC has not contributed financially to this publication. The University of Connecticut has filed a patent application on work related to this publication.IIM and CEN conceived the study. EH, JL, and CL conducted the experiments. All the authors contributed valuable ideas to the study, drafted the manuscript, and were involved in manuscript revision. All authors have read and approved the final manuscript.Supplementary figures, methods, and tables are supplied in PDF format.Click here for file


Classification methods used in microarray studies for gene expression are diverse in the way they deal with the underlying complexity of the data, as well as in the technique used to build the classification model. The MAQC II study on cancer classification problems has found that performance was affected by factors such as the classification algorithm, cross validation method, number of genes, and gene selection method. In this paper, we study the hypothesis that the disease under study significantly determines which method is optimal, and that additionally sample size, class imbalance, type of medical question , and microarray platform are potentially influential. A systematic literature review was used to extract the information from 48 published articles on non-cancer microarray classification studies. The impact of the various factors on the reported classification accuracy was analyzed through random-intercept logistic regression. The type of medical question and method of cross validation dominated the explained variation in accuracy among studies, followed by disease category and microarray platform. In total, 42% of the between study variation was explained by all the study specific and problem specific factors that we studied together. Microarray gene expression technology continues to be used to obtain more understanding of the mechanisms of human diseases. The statistical analysis of microarray data may be challenging, with the inherent risk of finding a false positive result due to the high dimensional nature of the data. Common flaws in the three distinctive goals for the statistical analysis of microarray data , and class prediction (supervised)) have been found Most of the studies evaluating classification performance have concentrated on classification of cancer patients. In general, non-cancer diseases have received low attention in the gene expression literature, maybe because they have more varying levels of complexity than cancers. However, to test the hypothesis that disease complexity influences classification performance, it may be beneficial to use a variety of studies on non-cancerous diseases, instead of cancer studies.This study focuses on the factors that might be associated with the accuracy of classification models on gene expression datasets, namely the type of disease, the medical question, sample size, the number of genes, the gene selection method, the classification method and cross validation techniques, using published studies outside of the field of cancer. Although it was evaluated differentially, we noticed that there is an overlap of the aforementioned study factors with the observed factors by the MAQC II study We searched microarray gene expression studies through PubMed  for relevant papers. Applied studies in which the investigator aimed to build supervised models based on microarray gene expression experimental data were primarily of interest. The studies that 1) were published in methodological journals 2) focused on cancer 3) were published before 2005 4) had non-human species as experimental objects 5) were not written in English or 6) were categorized as review papers, were not included. For the details of the search strategy and keywords see The search strategy and selection of studies satisfied the general methods for Cochrane reviews. The following details were recorded from each selected study: classification performance, disease type, medical question , microarray platform (one- or two-color system), total sample size, sample size per group, cross validation technique , gene selection technique , classification method(s), and the number of genes. The selected studies had evaluated the classification models in diverse ways, e.g. accuracy or misclassification error, sensitivity and specificity, positive and negative predictive values, as well as AUC. The accuracy was then used to represent the classification performance, since it is the most commonly used measure by the selected studies and feasible to be produced in some studies when the information about the accuracy lacked.Sample size was recorded as the sample size in the training set, used to build the model. The degree of class imbalance was measured by dividing the number of samples in the majority class with the total sample size in the training set. Due to the diversity of cross validation methods used, we grouped the cross validation techniques into single and nested cross validation. The studies that used cross validation for both model assessment and model selection were grouped into nested cross validation. Otherwise, it was regarded as single cross validation.k-Nearest Neighbours (kNN), artificial neural networks (ANN), and weighting voting methods. Meanwhile, discriminant analysis, prediction analysis of microarray (PAM), compound covariate predictor, nearest centroid, and LASSO were classified into the group of methods that could not automatically detect interactions .Some classification methods have the ability to automatically handle the curse of dimensionality (p>>n), but others need a gene selection step to reach a lower dimension before applying the classification method. Some of the studies selected genes univariately based on a statistic passing a threshold for selection or the top-K genes to feed the classifier. In other studies, the gene selection method was aimed at finding an optimal set of genes by stepwise iterating between selection and classifier building. Thus, we grouped the gene selection technique based on their interaction with the classifier, namely filter (e.g. univariate selection), wrapper (e.g. stepwise optimization of the selected gene set), and embedded . Grouping was also done on the classification method into two categories, depending on their ability to detect interactions between genes. Genes can be activated independently but also be activated through the activation of other genes. Due to this phenomenon, the classification methods that can automatically model interactions are expected to have better performance than those who cannot, at least in some studies. The methods that could detect the interaction (referred to as “interaction classifiers”) in our review were tree-based methods, logistic regression, support vector machines (SVM), http://eagl.unige.ch/SNOCat/) producing 16 categories. Further re-categorization was done to establish etiology-based disease groups. As a result, we obtained 6 disease types: inflammatory disorder, immune disease, degenerative disease, infection, mental disorder and other (i.e. obesity and acute lung injury). See Among the selected studies, we found 34 different disease types. The diseases were categorized according to SNOMED . The PRISMA workflow diagram of the systematic literature review For further statistical analyses, we selected the 48 studies Within the search period, the number of classification studies that had used microarray technology outside the field of cancer tended to increase with calendar time, and the one color system array (35/48) was mostly used, compare to the two-color system (13/48). We found 34 different diseases, predominantly the inflammatory disorder and infection disease groups, i.e. 17 (35%) and 12 (32%) studies, respectively. The diagnostic problem is the most common medical question addressed by microarray gene expression supervised learning. Classification models were built by either the single (19/48) or the nested (29/48) cross validation technique. The search result shows that there is no clear preference in dimensionality reduction technique among the selected studies. With regards to the classification methods, we notice that SVM (24%) and PAM (21%) are the most commonly used methods. However, when we grouped the classification methods based on their ability to detect interaction between genes, there appeared to be no clear preference for no-interaction or interaction classifiers.The relative contribution of each study factor to the explained-variation of the full model is shown in This study was conducted on 48 selected papers that were published between 2005 and 2013 and identified through the PubMed repository. Targeted keywords efficiently selected the relevant papers, among thousands of published microarray classification studies. We aimed to assess the influence of study and method specific determinants of classification model accuracy outside the field of cancer, by analysing eight factors through random effects meta-regression. The accuracy is used as a representation of classification model performance due to the availability of the information in the majority of the selected study. The accuracy is a well-known rough measurement for the performance of a classification model. Especially in highly imbalanced datasets, accuracy may yield overoptimistic results, because a classification model might easily send all samples to the majority class. The class imbalance should therefore be taken into account when interpreting the prediction accuracy The main finding of this study is that four factors were associated with the classification accuracy. The clinical problem  had the highest relative contribution to the explained-variation of the full model, which in other terms also had been experienced by the MAQC II consortium study The other important factor that should be considered in building a classification model is the cross-validation technique. It has the second largest of individual explained variability. The variability of classification performance could be due to the diversity of cross validation techniques that were used by investigators. It suggests that more attention shall be put to this factor when we build microarray classification models. Overoptimistic assessment of model performance is the most common flaw in class prediction studies, which causes an upward bias in estimates of prediction accuracy Another interesting finding in our study is the significant effect of microarray color system on model performance. The one-color microarray platform tends to yield higher accuracy than two-color systems . This reIndividual evaluation yielded no significant study factors that influenced the class prediction accuracy. A study factor may not be significant in a univariable model, but it may show its effect in a multivariable evaluation, due to the presence of other study factors in the meta-regression model, a mechanism well known as confounding. Hence, we also evaluated the study factors simultaneously by using backward selection. High associations were shown by the medical question and cross validation method . The proStatistical power might be an issue in our study, due to relatively small number of selected studies used for analysis. However, it is also important to note that there are not much published gene expression classification studies outside the cancer field. We found forty eight studies that fulfilled all our requirements . Given the relatively small population of non-cancer published studies, our search yielded a considerable number of studies when it is compared to the literature review studies in the cancer field conducted by This study evaluated eight factors that represent the characteristics of the experiments as well as the gene expression data. The multivariable analysis shows that 42% of the between study variation was explained by these factors, while the other 58% might be explained by un-observed factors, which may include the preprocessing procedures , the microarray type, and the number of features after the preprocessing steps. We did not include factors such as normalization and batch effect removal due lacking information in the majority of the selected studies. This also rises a recommendation for each published gene expression study to report all steps both in the experiment and in the data analyses, as mentioned by the MIAME (Minimum Information About a Microarray Experiment) guideline The accuracy of classification models based on gene expression microarray data depends on study specific and problem specific factors. Investigators should pay more attention to these factors when building microarray classification models. The cross validation technique has an important impact in explaining the variability across the studies. Nested cross validation is suggested to be used in any microarray classification study.Figure S1Plot of proportional sample size  and the classification model accuracy. The class imbalance level was calculated by dividing the sample size in the majority class by the total sample size in the training data. The diagonal line represents a minimum accuracy that should be achieved by a classification model, based on assigning all subjects to the majority class.(JPG)Click here for additional data file.Figure S2Boxplot of Cross Validation Technique against model accuracy.(JPG)Click here for additional data file.Figure S3Boxplot of microarray platform (color system) against model accuracy.(JPG)Click here for additional data file.Table S1Details of study factors in the selected studies.(XLSX)Click here for additional data file.Table S2The variability explained by each modeling factor.(DOCX)Click here for additional data file.Table S3Study factors that were included in the multivariable random effect logistic regression models via Jackknife resampling.(DOCX)Click here for additional data file.Checklist S1PRISMA checklist.(DOC)Click here for additional data file.Material S1Literature search strategy.(DOCX)Click here for additional data file.Material S2Disease classification.(DOCX)Click here for additional data file.Material S3R script.(DOCX)Click here for additional data file.


The purpose of the study was to apply a supervised data analysis strategy in a context where prior knowledge is available, We developed a structured strategy with innovative applications of existing bioinformatics methods for supervised analyses including: 1) the combination of two statistical (t- and Anderson-Darling) test results to detect miRNAs with significant fold change or general distributional differences in class comparison, which could reveal hidden differential biological processes worth to be considered for building predictive tools; 2) a bootstrap selection procedure together with machine learning techniques in class prediction to guarantee the transferability of results and explore the interconnections among the selected miRNAs, which is important for highlighting their inherent biological dependences. The strategy was applied to develop a classifier for discriminating between hemolyzed and not hemolyzed plasma samples, defined according to a recently published hemolysis score. We identified five miRNAs with increased expression in hemolyzed plasma samples .We identified four miRNAs previously reported in the literature as hemolysis related together with a new one (miR-22).which needs further investigations. Our findings confirm the validity of the proposed strategy and, in parallel, the hemolysis score capability to be used as pre-analytic hemolysis detector. R codes for implementing the approaches are provided.The online version of this article (doi:10.1186/s12859-015-0820-9) contains supplementary material, which is available to authorized users. This class of small RNAs is being widely studied in oncology and a functional implication in cancer development and progression has been demonstrated [MicroRNAs (miRNAs) are highly conserved single-stranded small RNA molecules (~19–22 nucleotides long) that play a key role in post-transcriptional gene regulation. To date, more than 2600 human miRNAs have been identified  test [In the attempt to develop a multimarker classifier using plasma miRNA data, some issues arising during the discovery process challenge the researchers. Moreover, so far there are no consolidated guidelines for data analysis in this context. This prompted us to develop a structured strategy for supervised analyses with the aim of: (1) in class comparison analysis, detecting differences of miRNA distributions between the two compared classes(2) in class prediction analysis, discovering the top discriminating features, study their associations and interconnections, and developing a ‘robust’ cross validated classifier. In the class comparison step we proposed the combined use of two tests: the AD) test . The forThe above approaches were applied to plasma miRNAs determined on a subset of patient samples from a clinical trial series . RNA exthttp://www.bioconductor.org). The details are reported below.The R codes implementing the proposed approach are provided as Additional files i.e., disease status, age at drawing and drawing year), we matched each case with two controls with the closest PS within the specified range . We used the recommended caliper width, which is equal to the 20 % of the standard deviation of the PS logit [Plasma samples included in the present study come from patients entering a randomized breast cancer prevention Trial . In detaPS logit . After mPlasma isolation and RNA extraction were carried out as previously described . Briefly2 transformed. Only the 1205 human (‘hsa’) miRNAs were considered in subsequent analyses. Microarray data are MIAME compliant and were deposited into the NCBI’s Gene Expression Omnibus (GEO) database with accession number ‘GSE59993’ (http://www.ncbi.nlm.nih.gov/geo/). MiRNAs were filtered at 90 %, i.e., we retained only miRNAs detected in at least 90 % of all samples. By applying a less stringent filtering , no additional differentially expressed (DE) miRNAs could be identified (data not shown), as compared with those obtained with the 90 % filtering.Raw data were summarized as previously described . Brieflyi.e., each miRNA pair only appeared once.As regard to the normalization step, we applied the ratio-based approach  that is t-test considers only location differences, the AD test is an ‘omnibus test’ [i.e., it considers the whole feature distribution, granting more importance to the observations in the tails. The latter characteristic becomes valuable when one is interested in finding signals that are only present in patient subsets diverging from the center of the distribution.We implemented supervised approaches for class comparison and class prediction on the training set samples using both raw  and ratio-normalized data. Class comparison analysis, aimed at identifying features (miRNAs or miRNA ratios) DE between cases and controls, was based on the combined use of the t- and the non parametric AD  tests. Wus test’ , i.e., iMoreover, plasma miRNA data, like other ‘omics’ data, have often not normal distributions and the sample sizes are often small. In presence of distributions with asymmetries, multimodality or heavy tails, the AD test reveals useful for the identification of interesting features. We considered the asymptotic version of the AD test, with correction for the presence of ties. The Benjamini-Hochberg method was used to distinctly adjust t- and AD p-values in order to control for the False Discovery Rate (FDR) . In parti.e., Prediction Analysis for Microarrays (PAM) [i.e., more features than subjects) and because they are conceptually different algorithms that we considered as ‘representative’ of methodological categories using different decision rules for classification . PAM, being characterized by a minor complexity respect to the other two algorithms, may be insufficient to appreciate complex classification patterns. Among the other two more sophisticated methods, RF overcome the main disadvantage of decision trees methods, which is their tendency to data overfitting and, like PAM, are fast and nonparametric, so one has not to worry about outliers. On the other hand, RF only output measures of feature importance, the interpretation of which is controversial with correlated features [i.e., the cost, which controls model complexity and the class weights, indicating the influence assigned to the two classes. Different models were fitted by varying the number of included features, forwardly selected according to the bootstrap generated list. The models were then cross validated with a leave-one-out cross validation procedure [ i.e., sensitivity, specificity and Youden index [For class prediction analysis, aimed at developing a classifier able to accurately discriminate between hemolyzed and not-hemolyzed samples, a two-step procedure was set up: firstly, with the purpose of obtaining a robust ranking of features with distributional differences between the two classes, a ‘bootstrap selection’ was performed, according to the strategy proposed by Austin and Tu . We extrys (PAM) , Random ys (PAM)  and Elasys (PAM) , while mfeatures . The inhfeatures , using trocedure  to adjusen index . The finAfter case–control matching, 78 samples were selected, 26 hemolyzed and 52 not-hemolyzed; 39 samples (13 hemolyzed vs 26 not-hemolyzed) were included in the training and validation set, respectively. The summary distribution of matching variables before and after matching is included in the Additional file t-test  were significant at the t- or AD test. We detected 104 ratios as significantly up-regulated and 94 ratios as significantly down-regulated with the t-test, for a total of 198 ratios, which involved 80 miRNAs , while miR-22 was selected in a high number of bootstrap samples and linked to the top four miRNAs. Ratio data generally led to smaller bootstrap occurrences, since each miRNA appeared in several ratios. However, miR-486-5p, miR-92a, miR-451 and miR-16 were included in the top eight ratios, with occurrences equal to 357 (1st position), 304 (2nd position), 270 (4th position) and 214 (8th position), respectively. MiR-22 appeared at the 31st position, with 121 occurrences. The ‘autoselected’ specific normalizers were miR-4257 for miR-486-5p and miR-92a, and miR-4286 for miR-451 and miR-16. The top co-occurrence involved miR-92a/miR-4257 and miR-486-5p/miR-4257, with a frequency equal to 200.The complete lists of the bootstrap-ranked features for both raw and ratio data are reported in the Additional file e.g., ID = 1 indicates the group of models with the highest Youden index). Considering raw data , the ‘ROC space’ plot in Fig. , the ‘ROata Fig. , we idenata Fig.  only onei.e., those ID numbered in the Fig. i.e., miR-451, miR-16 and miR-486-5p; such a model achieved the highest classification performance in the validation set (Youden index = 0.73).The above results are numerically shown in Table Regarding ratio data, the Youden index ranged from 0.61 to 0.73 in the training set and from 0.58 to 0.77 in the validation set. The chosen model included 500 ratios (Youden index = 0.73), corresponding to 88 features. Alternative choices could be the model with ID = 8, including two ratios (miR-486-5p/miR-4257 and miR-92a/miR-4257) or that with ID = 5, including 4 ratios , the latter presenting a slightly better classification performance in the training set (Youden index = 0.65 vs 0.61); also in this case, the two parsimonious models had the best performance in the validation set (Youden index = 0.77). It is worth to notice that with ratio data the miR-16 would not have been selected, since the top ratios contained more than once the other hemolysis-related miRNAs, producing redundancy in the results.Globally, we noticed that the SVM cost parameters, which control model complexity, were smaller with the ratio data and that, regardless the type of data, it was more difficult to validate a model containing a large number of miRNAs. Moreover, in the validation set the Youden index showed wide bootstrap confidence intervals (CI), due to the small sample size.e.g., possible availability of housekeeping features). While in our investigation we adopted a joint analysis of raw and ratio-normalized data, other methods might be suitable, like for instance the quantile method, previously shown to work best in reducing differences in miRNA expression values for tissue samples [i.e., cancer cell death) or active  release in bloodstream. To establish which miRNA in a ratio has relevant discriminating role and which act as normalizer  the results of raw and ratio data analyses should be interpreted together. An advantage of the ratio method is that, in the absence of known housekeeping miRNAs, it allows identification and automatic handling of a specific normalizer for each DE miRNA.In the present work we developed a general analysis strategy in order to deal with some issues arising in the supervised analyses of plasma miRNA from hybridisation array data. In the data pre-processing step, any normalization method can be applied and does not preclude the subsequent conduction of supervised analyses, although contributing to the final results. The normalization method should be chosen in relation to the type of features, their precision level and to the domain knowledge , more robust and possibly generalizable results can be obtained. Together with the bootstrap selection, we want to point out the egg-shaped plot, which can be used as a tool for giving an insight of interconnections among the selected features, becoming useful for highlighting their inherent biological dependences.In class comparison analysis, the search for DE features is usually intended for detecting significantly different means in the two groups, and location tests, such as the In the second step of the class prediction analysis, the classifiers are obtained by using statistical models including subgroups of selected features, and this categorizes class prediction in the domain of multivariable statistical analyses. The joint use of bootstrap selection and classifier cross validation should ensure the robustness of the class prediction results. A limitation of the procedure is that we could identify several best models in terms of classification performance. In some cases  the best models included a large number of features, thus being more prone to overfitting. However, we observed that the use of a small number of strongly predictive features resulted in a non significant decay of the cross validated classification performance measures in the testing set. Therefore, our strategy was to choose more parsimonious models, since is likely that the features included in such models will not be filtered out during the data pre-processing step. However, our results have to be taken with caution due to the small sample size, as it emerged from the large bootstrap intervals of the classification performance measures. By using our strategy we identified four top miRNAs  that have been reported in the literature as related to the presence of hemolysis, together with another one (miR-22), which is worth to further investigate. Even though miR-22 was not directly described as hemolyis-susceptible miRNA, it was identified as a signature miRNA for erythrocyte maturation . In addii.e., data from Agilent miRNA hybridization array coupled with hemolysis score evaluation) were still unavailable.Classifier development should rely on availability of three distinct datasets for training, validation, and testing. We are aware that a limitation of the present study is the lack of availability of a testing set on which an unbiased assessment of classifier performance could be obtained. Unfortunately, threefold splitting was not applicable in our case study, because was hampered by the small number of hemolyzed samples, and suitable public datasets .Our strategy may be extended to other kinds of ‘omics’ studies by introducing proper methodological adjustments. For instance, with non-coding RNA Sequencing data, which are count variables, the Anderson-Darling test could be used for class comparison analysis; in class prediction analysis, models suitable for analyzing count data should be used  that approved the use of the samples for this specific study and the relative data publication.


Studies show that thousands of genes are associated with prognosis of breast cancer. Towards utilizing available genetic data, efforts have been made to predict outcomes using gene expression data, and a number of commercial products have been developed. These products have the following shortcomings: 1) They use the Cox model for prediction. However, the RSF model has been shown to significantly outperform the Cox model. 2) Testing was not done to see if a complete set of clinical predictors could predict as well as the gene expression signatures.We address these shortcomings. The METABRIC data set concerns 1981 breast cancer tumors. Features include 21 clinical features, expression levels for 16,384 genes, and survival. We compare the survival prediction performance of the Cox model and the RSF model using the clinical data and the gene expression data to their performance using only the clinical data. We obtain significantly better results when we used both clinical data and gene expression data for 5 year, 10 year, and 15 year survival prediction. When we replace the gene expression data by PAM50 subtype, our results are significant only for 5 year and 15 year prediction. We obtain significantly better results using the RSF model over the Cox model. Finally, our results indicate that gene expression data alone may predict long-term survival.Our results indicate that we can obtain improved survival prediction using clinical data and gene expression data compared to prediction using only clinical data. We further conclude that we can obtain improved survival prediction using the RSF model instead of the Cox model. These results are significant because by incorporating more gene expression data with clinical features and using the RSF model, we could develop decision support systems that better utilize heterogeneous information to improve outcome prediction and decision making. If D and H have different values for predictor F, then F separates two close data items with the same value of the target, which indicates F is not a good predictor; so we decrease the weight. On the other hand, if D and M have different values for predictor F, then F separates two close data items with different values of the target, which indicates F is a good predictor; so we increase the weight.By “nearest hit Method 1 combined the clinical features and the gene expression data, and then used ReliefF to extract features. Method 2 extracted gene expression features from the gene expression data, and then combined those features with all 21 clinical features. When using this latter method, we only extracted 9, 29, 79, and 129 features so that the total number of features was still 30, 50, 100, and 150.When ReliefF is employed, different sized sets of top predictive features can be used to perform prediction, and the set of features that yields the best performance is the one that is used in the prediction system. We used sets of 30, 50, 100, and 150 top predictive features. We did this in two ways: We developed 5 year, 10 year, and 15 year survival functions using each of the methods. The 5 year survivor functions were learned only from survival information obtained from the first 5 years of post-initial visit information, the 10 year survivor functions were learned only from the first 10 years of information, and the 15 year survivor functions were learned only from the first 15 years of information. When using ReliefF, we used as our target survivorship at 5 years to extract features for the 5 year survival functions, at 10 years to extract features for the 10 year survival functions, and at 15 years to extract features the 15 year survival functions. Note that we did not utilize any assumed biological knowledge concerning which genes might be good predictors. We relied only on machine learning to find good predictors. In this way our results do not depend on any previous analyses.k-fold cross validation, we divide the data into k partitions of the same size. For each partition j we learn a model using the data in the remaining k-1 partitions, and we then apply the model to partition j. For each survival timeframe  we compared methods using 5-fold cross validation. Specifically, for each method, survival timeframe, and partition, we learned a survival function from the individuals in the remaining 4 partitions. We then used that function to develop a survival curve for each individual in the given partition. Next, we computed the concordance for the individuals. The concordance index is the probability that, given two randomly drawn individuals, the individual who has the event first has a worse survival curve. By “worse” we mean the area under the curve is smaller. To estimate the concordance index, we pair each individual j known to be dead, with each individual k known to be alive later than j or known to die later than j. Let areaj be the area under the survival curve for individual j, and areak be the area under the survival curve for individual k. If areaj < areak, we add 1 to a variable Total. Finally, we divide Total by the number of pairs to obtain the concordance index. We statistically compared the concordance indexes for two methods using the chi-square test.In Clinical_Gene. For both the Cox model and the RSF model these best results were obtained using Method 2. We see that the RSF model performs significantly better (p < 2.2 × 10–16) than the Cox model in every case except for Clinical_Only and 15 years, and this latter result is not significant. The most improvement for the RSF model was realized with Clinical_Gene. The Cox model is based on linear regression and does not do shrinkage to handle a large number of variables such as that done by a regularized technique such as Lasso [as Lasso . This woClinical_Gene performs significantly better than Clinical_Only for all values of Year (p < 2.2 × 10–16). Clinical_Pam performs about as well as Clinical_Gene for 5 years and 15 years, but barely performs better than Clinical_Only for 10 years.We are most interested in comparing the best results obtained for each data set over both models (Cox and RSF) and all values of the number of features provided to ReliefF because we would use the best results in an actual prediction method. Our chief purpose was to investigate whether we would obtain better performance by including gene expression data and by using the RSF model instead of using only clinical data and using the Cox model. http://www.partek.com/), which allows one to perform either hierarchical clustering or partition clustering. Hierarchical clustering break up the data in to a hierarchy of clusters. Partitional clustering divided the data into mutually disjoint partitions. We did partition clustering for the individuals on the y-axis and hierarchical clustering for the genes on the x-axis. In Figs. Clinical_Gene exhibits the least improvement relative to Clinical_Only in the case of 5 year survival prediction. Looking at the 5 year survival heat map in Our studies were done using 5-fold cross validation. In general, if we were to then use the method in an actual production system, we would learn from the entire data set. MND1 gene has been shown to be essential for meiotic recombination between homologous chromosomes [It is interesting to investigate which genes were found to be predictive for two or more time frames. omosomes . Its disomosomes . The faclymph_nodes_positive and group are the top two features. However, in 15 year survival, lymph_nodes_positive does not appear at all, and there is only one clinical feature in the top 50 features. That feature is the Nottingham prognostic index (NPI), which is calculated using the size of the lesion, the number of lymph nodes involved, and the grade of the tumor; and which therefore includes the information in the feature lymph_nodes_positive. A similar situation holds for the 15 year survival training sets (not shown). Namely, NPI is the top clinical feature in all of them, and its average location is 25th. Only one of them has another clinical feature (“group” which appears 11th) in the top 50 features, and one has no clinical features in the top 50 features. Yet, as we can see from It is also revealing to investigate the clinical features chosen by Method 1. We compared survival prediction using both clinical data and gene expression data to survival prediction using only clinical data. We obtained significantly better results when we used both clinical data and gene expression data relative to when we used only clinical data. When we included Pam50_subtype, which is a composite of the expression data for 50 genes, with the clinical features, our results were significantly better only for 5 year and 15 year prediction. When we used ReliefF to choose good gene expression predictors, the results were significant for 5 years, 10 year, and 15 year prediction. This information is meaningful because often 10 year survival is the most utilized survival criterion.We also compared survival prediction using both the Cox model and the RSF model, and we obtained significantly better performance using the RSF model. This result is also notable because the Cox model is often used to do survival prediction in current systems (as noted in the Introduction Section). Furthermore, our results indicate that the Cox model cannot handle a large number of covariates, and might explain why existing methods (as discussed in the Background Section) limit the number of genes or group them into subtypes. As noted in the Introduction Section, studies show that thousands of genes are associated with subtype and prognosis of breast cancer . As alsocopy number variations (CNV) and copy number alterations (CNA). We can utilize both machine learning knowledge and biological knowledge in our choice of genetic features. Finally, since we have learned that the prediction method which is used can make a substantial difference, we can investigate using other survival prediction methods such as the recently developed Bayesian network based method S_EBMC [Our results indicate that we can improve prediction performance by using gene expression date beyond the subtype composites in PAM50. Future research can investigate feature selection further. First, we can investigate whether we get similar, or even improved, results using strategies other than ReliefF to locate good predictors. For example, we could use the Bayesian network-based methods that look for interacting causes of a target . We can d S_EBMC .We obtained an interesting result when we applied ReliefF in Method 1 to simultaneously learn good clinical and gene expression predictors. That is, our results show that very few clinical features are needed for 15 year prediction. This would indicate that much of long-term survival can be explained by gene expression signatures. If someone survives long-term, it is likely they died of something other than breast cancer . So, perhaps long-term survival in general can be predicted by gene expression signatures, which opens the possibility of increasing longevity by targeting gene expression.S1 Table(DOC)Click here for additional data file.S2 Table(DOC)Click here for additional data file.S3 Table(DOC)Click here for additional data file.S4 Table(DOC)Click here for additional data file.S5 Table(XLSX)Click here for additional data file.


A response to ‘Comprehensive evaluation of differential gene expression analysis methods for RNA-seq data’ by Rapaport F, Khanin R, Liang Y, Pirun M, Krek A, Zumbo P, Mason CE, Socci ND and Betel D in Genome Biology, 2013, 14:R95The online version of this article (doi:10.1186/s13059-015-0781-3) contains supplementary material, which is available to authorized users. Statistical methods for determining transcriptional changes between (replicated) groups of cell populations using RNA sequencing (RNA-seq) data are now quite mature. Several themes that emerged from the past decade of modeling microarray data apply analogously to RNA-seq data: parameter moderation is critical, multiple testing corrections are necessary and flexible frameworks  to account for the effect of covariates are essential. For RNA-seq data, popular packages such as edgeR, DESeq and DESeq2 –3 perforA recent report in Genome Biology from Rapaport and co-authors claimed that some methods, namely PoissonSeq  and limmThe Rapaport manuscript is an excellent model of modern bioinformatics research, in terms of making processed data and code available that reproduce figures from their manuscript. In many cases, the small details can be important and this open-source model facilitates quick access in understanding precisely what settings were used. We fully support this model and by default, also make our code available. In this correspondence, we investigate the genesis of differences in method performance that Rapaport and co-authors observed and provide our view of how performance results can be sensitive to decisions made.We first briefly summarize the analysis that Rapaport and colleagues reported, with respect to the all-zero-in-one-condition case.P values for several methods; (ii) calculate S/N using ‘normalized’ data; (iii) plot negative log P value versus S/N, where they expect a monotonic positive dependency (correlation); and, (iv) generate receiver operating characteristic (ROC) curves with thresholds on the S/N to illustrate the ability to separate low S/N (<3) from high S/N (>3).Using gene-level read counts, they isolated genes that exhibit zero-counts across all replicates of a single condition; in general, the number of such genes is related to the depth of sequencing dedicated to each sample, with deeper sequencing resulting in fewer such cases. The dataset in question, comparing GM12892 cells to H1-hESC cells [They highlighted that count-based methods such as DESeq and edgeR, which infer changes in expression via the negative binomial (NB) model, do not perform very well in this case. It is worth noting that this is a non-standard use of ROC curves: here, all genes are strictly DE, but they vary in their magnitude of change. So, the ROC curve represents the ability to separate low S/N from high S/N. Rapaport and colleagues postulated that the NB model reduces to Poisson (dispersion ≈ 0) and lacks the ability to handle the ‘wide variations’ in gene counts among replicate libraries. Our aim with this report is to understand the origins of this result, whether it is a shortcoming of the dispersion estimation strategy or in the inference machinery, since parameter estimates are on the boundary of the parameter space.P values for the expressed-in-one-condition versus zero differences. In practice, however, there are some potential difficulties: the sample sizes are small and therefore, the S/N itself is subject to considerable estimation uncertainty; it is well known that for count data the variance is intimately tied to the mean, so it is not clear whether S/N should be calculated on a linear scale. In addition, a notable aspect of the Rapaport ROC comparison is that while the same S/N cutoff (= 3) is used across all methods, different sets of true and false DE labels are used; this makes the curves difficult to compare, since both the truth and score change by method. We explore these issues here.We became interested in the suitability and robustness of the S/N metric itself, since it forms the basis for the ‘truth’ in Rapaport’s ROC result. In theory, the S/N of the non-zero observations should accurately reflect the significance of model-based P values seem to reflect appropriately the magnitude of evidence for DE, although they are on drastically different scales between edgeR and voom , the theoretical S/N is then: ϕ−1/2 with sufficiently large μ. Thus, depending on the mean, the S/N calculation is capturing the (inverse square root of) dispersion. For the ENCODE data, this relationship is shown in Additional file Another aspect to understand is the scale on which the S/N is calculated. As is well known with count data, the variance is related to the mean. In particular, using the NB parameterization with mean entclass1pt{minimaentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaGiven recent efforts in simulating RNA-seq count tables –11, we tBased on the results of this simulation Fig. , ROC curAs developers and users of bioinformatics strategies, we are particularly interested in the metrics and methods that differentiate performance between the available tools. In this paper, we claim that count-based methods perform well when genes are only expressed in one condition, in contrast to an earlier report. We showed that a code error and the chosen scale of S/N resulted in the earlier conclusion that count-based methods suffer performance in this situation. By calculating the S/N on a different scale and using the same set of labels across methods, a reversal of method performance was observed. This highlights a sensitivity to decisions made in constructing the benchmark.Using a customized simulation that introduces zero-counts in one experimental condition, we demonstrated that the performance of the count-based method is actually on a par with or better than other methods. We also debunked the postulation that poor performance is related to dispersion estimation in count models.P values is drastically different between edgeR and voom. Although this observation appears rather unrelated to the ability to separate true from false DE genes, it is an indication that the scale of observations modeled affects the magnitude of statistical evidence derived. Not surprisingly, method performance is ultimately dependent on the scales, parameters and datasets used for the evaluation.In the process of seeking the origins of this statistical performance difference, we discovered another potentially interesting phenomenon that may affect the interpretation of results. Looking at Table R code and data that can be used to reproduce the figures in the main manuscript and in the supplement are available online .


RNA-sequencing (RNA-seq) has rapidly become the method of choice in many genome-wide transcriptomic studies. To meet the high expectations posed by this technology, powerful computational techniques are needed to translate the measurements into biological and biomedical understanding. A number of statistical procedures have already been developed to identify differentially expressed genes between distinct sample groups. With these methods statistical testing is typically performed after the data has been summarized at the gene level. As an alternative strategy, developed with the aim to improve the results, we demonstrate a method in which statistical testing at the exon level is performed prior to the summary of the results at the gene level. Using publicly available RNA-seq datasets as case studies, we illustrate how this exon-based strategy can improve the performance of the widely used differential expression software packages as compared to the conventional gene-based strategy. In particular, we show how it enables robust detection of moderate but systematic changes that are missed when relying on single gene-level summary counts only. Deep sequencing of RNA (RNA-seq) has rapidly become a widely used technique to characterize transcriptomes. Whilst facilitating detailed mapping of the transcriptome over different cell types, perturbations and states, and providing superior sensitivity over expression microarrays, the method has generated high expectations A fundamental research aim in RNA-seq studies is the identification of differentially expressed genes between distinct sample groups . Accordingly, a number of statistical tools have already been developed for this task, including methods based on negative binomial models Currently, the most common strategy for statistical analysis of RNA-seq data is based on the use of gene-level read counts To systematically investigate the benefits of the proposed exon-based strategy in detecting differentially expressed genes, we consider two widely-used software packages that are conventionally applied to gene-level read counts, edgeR p-values, taking into account the directions of the changes. More specifically, if we denote by ix the estimated log2 fold change of an exon i and by ip the corresponding p-value obtained from the statistical testing, we determined the median over the signed log-transformed p-values n is the number of exons in the particular gene and sgn is the sign function. The log transformation forces the least significant p-value to zero. Alternatively to median, the method can, in principle, be used with any mean descriptor preferred.A schematic illustration of the exon-based strategy is shown in p-values, we considered two popular R/Bioconductor packages for detecting differential expression in RNA-seq data, limma p-value score was assessed by comparing the observed value to the null distribution obtained under the assumption that the exon-level p-values were uniformly distributed, taking into account the number of exons per gene. False discovery rates (FDR) were determined using the Benjamini-Hochberg multiple testing adjustment method p-value was below the corresponding FDR level by considering the maximum of these two values when ranking the genes or calling them differentially expressed.For determining the exon-level Prior to the analysis, we filtered out very lowly expressed exons on the basis of their overall average count across the biological conditions as recommended for example in The MAQC RNA-seq data were downloaded from the Sequence Read Archive (SRA accession SRA010153). The data contain two types of biological samples: human brain reference (brain) and human universal reference RNA (uhr), both of which have been assayed using seven lanes on the Illumina Genome Analyzer II sequencing platform The RNA-seq data were downloaded from the European Nucleotide Archive under the accession number SRP001540. The samples have been assayed using the Illumina Genome Analyzer II platform at two different centres using different read lengths (46 bp or 35 bp). The reads were aligned to the human genome (hg19) using Tophat (version 1.4.0) and default settings. Gene- and exon-level read counts were determined based on RefSeq annotations using the python scripts incorporated in the DEXSeq R/Bioconductor package.We first assessed the performance of the gene- and exon-based strategies in the MAQC RNA-seq data, generated for benchmarking purposes Using the qRT-PCR data as a gold standard, we constructed receiver operating characteristic (ROC) curves for the different gene- and exon-based differential expression statistics. To summarize the performance in single values, we determined the partial area under the curves (pAUC) at a specificity of 0.8, standardized to have a maximal value of 1.0 . The pAUThe qRT-PCR validations also enabled us to calculate the empirical false discovery rates of the detections . Across In order to take yet a closer look at the differences between the gene- and exon-based strategies, we examined the largest discrepancies between the detections in the MAQC data . This suTo assess the performance of the exon-based strategy in more complex real datasets involving substantial biological variation between replicates, we considered RNA-seq data on lymphoblastoid cell lines derived from unrelated Nigerian individuals as part of the International HapMap project p-values 3.29e–7 and 0.002, respectively; David tool p-values 0.004 and 0.002 respectively, David tool Focusing only on the most promising genes identified at FDR<0.05 and showing at least a 1.5-fold change between the male and female groups across at least five exons, we identified 16 and 67 genes using limma with the gene- and exon-based strategies, respectively . All of Taken together, we demonstrated in this study how an exon-based strategy can significantly increase the sensitivity and specificity of the widely used differential expression methods for RNA-seq data over the conventional gene-based strategy. In particular, we observed that taking advantage of the exon-level signals enabled detection of such moderate but systematic gene expression changes that were missed by the gene-based strategy relying on single gene-level summary counts only. Additionally, our results showed how the gene-based approaches are prone to effects of single exons, while the exon-based strategy is robust against them.Although we focused here on the most basic design of comparing two groups of samples, the exon-based strategy extends naturally to more complex study designs and to basically any current or future test statistic for detecting differential expression. It is not limited to a specific gene or transcript model either but can be applied to any user-defined feature model, such as windows across de novo assembled gene contigs.


Accurately quantifying gene expression levels is a key goal of experiments using RNA-sequencing to assay the transcriptome. This typically requires aligning the short reads generated to the genome or transcriptome before quantifying expression of pre-defined sets of genes. Differences in the alignment/quantification tools can have a major effect upon the expression levels found with important consequences for biological interpretation. Here we address two main issues: do different analysis pipelines affect the gene expression levels inferred from RNA-seq data? And, how close are the expression levels inferred to the “true” expression levels? We evaluate fifty gene profiling pipelines in experimental and simulated data sets with different characteristics . In the absence of knowledge of the ‘ground truth’ in real RNAseq data sets, we used simulated data to assess the differences between the “true” expression and those reconstructed by the analysis pipelines. Even though this approach does not take into account all known biases present in RNAseq data, it still allows to estimate the accuracy of the gene expression values inferred by different analysis pipelines. The results show that i) overall there is a high correlation between the expression levels inferred by the best pipelines and the true quantification values; ii) the error in the estimated gene expression values can vary considerably across genes; and iii) a small set of genes have expression estimates with consistently high error (across data sets and methods). Finally, although the mapping software is important, the quantification method makes a greater difference to the results. Over the past five years, RNA-sequencing (RNA-seq) has been gradually overtaking microarrays to become the tool of choice for genome-wide analysis of the transcriptome 1. Filtering out reads that fall below some quality threshold2. Aligning the reads to the genome or transcriptome3. Quantifying gene expression levelsThe first step involves the identification of low quality reads  that can be filtered out prior to mapping, thus avoiding potential mismatches and downstream errors. To perform the alignment, the user typically applies one of the many high-throughput sequencing mapping tools available Clearly, at each step of this process, the practitioner must select one of many possible options. A specific analysis pipeline is often selected based on subjective factors, such as popularity, ease of use, or past experience with some of the methods (possibly in a different context). It is unclear whether this choice, which is fundamental in many RNA-seq experiments, affects the measures of gene expression and hence downstream interpretation.Previous studies have essentially compared set of alignment methods, typically used one quantification method, and have focused their results on assessing whether these approaches result in differences in the sets of differentially expressed genes identified. For example To address these questions we applied fifty RNA-sequencing pipelines  to both real and simulated data and investigated whether this affected the estimates of gene expression levels. The alignment methods considered include both splice-aware  and splice-unaware  aligners in order to assess the effect of splice-aware aligners upon the inferred expression estimates. The quantification methods used were different versions or modes of HT-Seq, Cufflinks and Flux-capacitor.in silico with different characteristics  to assess the error of each pipeline , the effect of sequencing depth on the results and whether longer reads lead to reduced errors.A limitation of using real data to compare different pipelines for processing RNA-sequencing data is that no ground truth is known, and thus it is impossible to state that one method is more accurate than another. To address this limitation forty data sets were generated We first investigated how the choice of processing pipeline affected gene expression measurement levels obtained from four independent RNA-sequencing experiments (See Methods). For each of the 50 pipelines we obtained measures of gene expression levels and compared them across pipelines. We observed that, in general, the measurements of gene expression obtained from different pipelines were highly correlated, with a median Spearman correlation of 0.99 . Neverthin silico data sets with different read lengths , overall coverage  and with both single- and paired-end reads . Note that each data set contains eight libraries. These data allowed us to assess how near the output of each pipeline was to the true expression values, the effect of sequencing depth on the results and whether longer reads lead to reduced errors. Obviously an ideal simulated data set should have properties similar to those observed in real data sets. However, to be able to compare all genes across different data sets thirty two data sets were generated in silico where all transcripts had approximately the same level of expression (referred as STE). In this setting the genes in a data set still have different raw expression levels that result from the stochastic nature of the simulator, different transcript lengths and number of transcripts, and sequencing depth. However, whilst having distinct advantages, these data sets (STE) do not recapitulate the expression patterns tipically observed in real data where a subset of all genes is expressed at varying levels. To address this potential limitation and validate any findings found using the STE data sets we generated in silico eight data sets (designated as MTE) in a way that resembles the expression patterns often observed in real data.A limitation of using “real” data is that no ground truth exists, hence we next explored this more rigorously by performing a simulation study. We generated Overall, 82% of all pipeline runs  to analyze the STE data sets succeeded . When we segment the error by the quantification method  we obserTo obtain an overall summary , we rankImportantly, the error observed is independent of gene length. When the error is compared to the coverage of a gene the regression line of the error remains at the same level for different gene lengths  aligner used. It is also striking that one of the simplest quantification methods, HTSeq, when used in conjunction with different aligners shows up frequently towards the top of the rankings. Note that HTSeq discards all multi-mapping reads  while Cufflinks and Flux-Capacitor attempt to allocate these reads to the most “probable” transcript/gene.The rankings of the pipelines evaluated using paired-end data (a.k.a. pair-end tags) or single-end data are similar, having a Spearman correlation of 0.88 . This was confirmed in the additional eight simulated MTE data sets where there was no observed overlap between the genes with low error and the set of genes with consistent low error.Overall the measurements of gene expression obtained from different pipelines are often highly correlated in the experimental data sets. However, between any two pairs of pipelines, many genes had significantly different expression estimates, thus different pipelines yield very different results for a subset of genes.When considering the simulated data sets, the read counts inferred by the pipelines are highly correlated with the true values see . Since tPrevious studies mentioned the critical role of the aligner Overall, increasing the sequencing depth reduced the quantification errors of the pipelines see , but onlsine qua non condition to discriminate them  was a ate them . We nextate them .The median error observed across all data sets and pipelines is around 20%. The error does not affect all genes in the same way. The trend observed is that the error tends to get smaller with higher expression values (read counts). However, this does not mean that lower expressed genes will always have high errors. In fact we observed genes  across all data sets and pipelines with an error close to 0% with either low or high expression values. Furthermore, genes with very high error but with either low or high expression values were also found, suggesting that other factors besides the expression values may affect the accuracy of the gene profiling methods.in silico simulated data sets. The main findings can be resumed as follows:All in all, we have evaluated and compared gene profiling computational pipelines as a whole, from the initial sequencing reads to gene expression values. Fifty gene profiling pipelines were evaluated on thirty two The overall correlation between the quantification values inferred by the different pipelines and true values is above 0.90 and the error is around 20%. The error does not affect all genes in the same way and can vary considerably. A set of genes was found to have consistent high error (across data sets and pipelines) suggesting that other factors besides the expression values may affect the accuracy of the gene profiling methods.The rankings indicate that quantification method used by a pipeline makes a greater difference in the results than the (spliced) aligner selected.The read length effect on the performance of the pipelines varies depending if a spliced or unspliced aligner is used. The difference between using spliced and unspliced aligners is important - the error increases with the length of the reads with unspliced aligners.The sequencing depth has an effect upon the errors of the pipelines. The overall error tends to stabilize around 30x/60x, hence deeper sequencing depths (more than 60x) will not yield more accurate results in the context of gene expression quantification.in silico which may introduce some bias. Thirdly, since we simulated RNA-seq from a single species (Human) it is possible that the pipelines evaluated may behave differently on data from evolutionary distant species. Lastly, this study does include several mappers and quantification methods but it is certainly not exaustive since many more mappers and quantification methods exist. In spite of these limitations, the results and conclusions presented have practical implications for RNA-seq data analysis.There are several limitations with our approach. First, part of the results have been obtained using synthetic data and it is unclear to what extent these results are applicable to other experimental data sets. Although this question can arguably be raised for any kind of data used, simulated or experimental, we attempted to minimize this issue by using a large number of data sets. Second, we employed a single method to generate RNA-Seq data http://www.ebi.ac.uk/arrayexpress) and three other experimental data sets. More details are provided in Table S3 in We analyzed the Illumina Human Body Map RNA-seq set  in silico reproduction of the experimental pipelines for RNA-Seq by using models based on analyzed RNA-Seq experiments from different cell types, sample preparation protocols and sequencing platforms.Synthetic RNA-seq reads were generated Each simulated data set represents two conditions containing four replicates, where all transcripts have approximately the same level of expression. To achieve a sequencing depth of The Human genome (GRCh37.66) and respective annotation were obtained from Ensembl. The analysis focused on features annotated as protein coding genes and reads were simulated independently from all transcripts of each gene. Table S3 summarizes the experimental data used and Table S4 in in silico using Flux-simulator as described above with the following differences: i) the range of expression values was automatically determined by Flux-simulator to mimic the range of expression values observed in experimental data; ii) the read lengths considered were 50 bp and 100 bp; iii) all libraries were single-end.An additional set of eight data sets was generated http://www.ebi.ac.uk/~nf/gpeval/.The data sets are publicly available at A RNA-seq pipeline for estimating gene expression is here considered as being composed by two steps: i) one tool for mapping the reads and generating the alignments; and ii) one quantification tool that takes the alignments and other information to infer the expression values of the genes.http://www.sanger.ac.uk/resources/software/smalt/), BWA1 There is a considerable number of tools for mapping high throughput sequencing reads and RNA-seq data in particular The quantification tools considered include HT-Seq The pipelines considered in the comparison include all combinations of selected aligners and quantification methods, which gives a total of 50 pipelines. All pipelines considered were implemented and executed using the iRAP RNA-seq pipeline The following metrics were used to compare the expression estimates inferred by different pipelines given the same data set: Spearman correlation between the true and inferred read counts per gene; Relative error.error. For a given data set, let The error metric measures how different the inferred quantification values are from the true values. The Spearman correlation allows an investigation of whether the ranking of gene expression levels in the inferred and true data sets are consistent. Arguably, the most appropriate metric to evaluate the accuracy of the pipelines is their The quantification error for a gene The formulation does not acknowledge that some reads may be discarded in some stage of the analysis pipeline and, therefore, not included in the quantification. To tackle this issue we propose the relative  quantification error for a gene The relative quantification error is often referred to simply as error. In practice, to handle the cases where the true quantification of a gene is zero (reads), the error is computed by adding a 1 to the denominator and numerator.The overall quantification error of a pipeline Finally, a note about the read counts for Cufflinks. Cufflinks does not output read count; instead it produces, amongst many values, the read coverage by transcript. Therefore, pseudo-counts were produced based on read coverage for each transcript and then aggregated to obtain the pseudo-counts per gene.The pipelines were ranked using the two metrics mentioned above, separately for each data set. The average rank of a pipeline, using each of the two metrics, was computed as the mean rank of the pipeline across all data sets. The sum of the rankings of each metric was used to create an overall rank. The mean overall rank was determined when aggregating the results across data sets.A gene's sequence “uniqueness” is calculated based on how many regions in the genome are similar to the genes's exons. The similarity of each exon was computed by aligning each exon to the genome using BLAST (version 2.2.12). Only high-scoring segment pairs (HSP) with an e-value below 0.01 and more that 90% identity, were considered. The alignments were later filtered by length (greater than 30 or 150 nucleotides). The uniqueness of a gene File S1Supplementary tables and figures.(PDF)Click here for additional data file.