Besides the selection of the clustering method itself, determining which distance is going to be employed between data objects is probably one of the most difficult decisions.Clustering is crucial for gene expression data analysis. As an unsupervised exploratory procedure its results can help researchers to gain insights and formulate new hypothesis about biological data from microarrays. Given different settings of microarray experiments, clustering proves itself as a versatile exploratory tool. It can help to unveil new cancer subtypes or to identify groups of genes that respond similarly to a specific experimental condition. In order to obtain useful clustering results, however, different We analyze how different distances and clustering methods interact regarding their ability to cluster gene expression, i.e., microarray data. We study 15 distances along with four common clustering methods from the literature on a total of 52 gene expression microarray datasets. Distances are evaluated on a number of different scenarios including clustering of cancer tissues and genes from short time-series expression data, the two main clustering applications in gene expression. Our results support that the selection of an appropriate distance depends on the scenario in hand. Moreover, in each scenario, given the very same clustering method, significant differences in quality may arise from the selection of distinct distance measures. In fact, the selection of an appropriate distance measure can make the difference between meaningful and poor clustering outcomes, even for a suitable clustering method. Microarray development has enabled researchers to gather huge amounts of data from the most diverse biological phenomena. A single microarray is capable of determining expression levels for virtually all the genes of a particular biological sample of interest. Once combined, related microarray experiments give rise to what is usually referred to as gene expression data, a highly dimensional dataset with measurements over thousands of genes and few biological samples (microarrays). Obtaining the data is, however, only the first step towards the laborious path that comprehends its analysis.To transform gene expression data into knowledge, efficient and effective computational methods are required. Methods from Data Mining, Machine Learning, and Statistics have been applied since the birth of the gene expression data analysis field -3. A freThe second clustering application concerning gene expression data is found when genes that show similar expression patterns are clustered together ,10-12. ITaking into account the peculiarities of each one of the aforementioned scenarios, several clustering methods have been proposed for the problem of tissue clustering, e.g., -19, and parameters that comprise the clustering procedure. Provided that most clustering methods are based on distance calculations, i.e., clusters are determined based on distances between objects, selecting the distance between pairs of objects to be employed by the clustering method is at least as important as selecting the clustering method itself 38]. AlthOne of the first empirical studies concerned with the comparison of distances for gene expression data was conducted by . The autConsidering the clustering of cancer samples, different distances were evaluated by , 30], a, a30], aDistance measures are also compared by  and 43]43]. It iseparately for analysis, given their different characteristics. The authors also reviewed and evaluated, for the first time, distances that were explicitly introduced for short gene time-series clustering.The first large study analyzing different distances regarding gene expression microarray data was performed by . This wawithout applying a clustering method. This was possible due to the concept of intrinsic separation ability, which compares directly a distance measure against a desired ground truth solution, i.e., a reference partition. Therefore, there is no guarantee that the distance measures that provided good performance in ,,4], respa priori, as well as the cluster memberships for objects in these datasets, i.e., we have a ground truth. In such a case, one can employ measures such as the Adjusted Rand Index (ARI) 41], the We applied statistical tests (see Methods Section for description) in separate for each clustering method to detect which distances provided statistically superior results regarding their ARI values. For both cDNA and Affymetrix, considering AL, CL, and KM clustering methods, PE, JK, and RM provide better results than SUP in virtually all cases. For Single-Linkage no statistical differences are suggested.o stands for the number of objects. Note that partitions with number of clusters different from those found in the reference partition may, in certain cases, contain more natural clusters than those found in a partition with the "right" number of clusters, see, e.g., 51]. In bp-value ≤ 0.05 within each cluster. The enrichment test is based on the Fisher Exact Test, which indicates if the overlap between genes in a cluster and in a GO term is higher than expected by change  interval. Since the sign of the correlation is important for gene expression data, one minus the value of the correlation provides the distance we use for clustering in our experiments . In the following, both x and y are sequences of real numbers in the form x =  and y = .Correlation coefficients are popular choices for clustering microarray data, with values in the .To estimate the number of clusters in our third evaluation scenario, a relative index of comparison between partitions is also employed. The Silhouette index is defined by Eq. (23), considering a partitioning of We choose the Silhouette based on its superior results in comparison to other relative criteria, as demonstrated by ,67,68. WFinally, we would like to note, that by using the Silhouette index we simulate a real application in which the user does not have any a priori information regarding the number of clusters present in the data. It is important to make clear, that the use of relative indexes (such as the Silhouette) is just part of the more general procedure that comprehends the whole clustering analysis, i.e., (i) pre-processing, (ii) clustering and, (iii) validation . To thisStatistical tests were employed to assess the significance of the results obtained during our experimental evaluation. Based on the work of  we use FThe authors declare that they have no competing interests.PAJ implemented the methods and performed the experiments. PAJ, RJGBC and IGC designed the study, evaluated the results and wrote the manuscript. All authors read and approved the final manuscript.


The prediction of breast cancer intrinsic subtypes has been introduced as a valuable strategy to determine patient diagnosis and prognosis, and therapy response. The PAM50 method, based on the expression levels of 50 genes, uses a single sample predictor model to assign subtype labels to samples. Intrinsic errors reported within this assay demonstrate the challenge of identifying and understanding the breast cancer groups. In this study, we aim to: a) identify novel biomarkers for subtype individuation by exploring the competence of a newly proposed method named CM1 score, and b) apply an ensemble learning, as opposed to the use of a single classifier, for sample subtype assignment. The overarching objective is to improve class prediction.The microarray transcriptome data sets used in this study are: the METABRIC breast cancer data recorded for over 2000 patients, and the public integrated source from ROCK database with 1570 samples. We first computed the CM1 score to identify the probes with highly discriminative patterns of expression across samples of each intrinsic subtype. We further assessed the ability of 42 selected probes on assigning correct subtype labels using 24 different classifiers from the Weka software suite. For comparison, the same method was applied on the list of 50 genes from the PAM50 method.The CM1 score portrayed 30 novel biomarkers for predicting breast cancer subtypes, with the confirmation of the role of 12 well-established genes. Intrinsic subtypes assigned using the CM1 list and the ensemble of classifiers are more consistent and homogeneous than the original PAM50 labels. The new subtypes show accurate distributions of current clinical markers ER, PR and HER2, and survival curves in the METABRIC and ROCK data sets. Remarkably, the paradoxical attribution of the original labels reinforces the limitations of employing a single sample classifiers to predict breast cancer intrinsic subtypes. The cla tumours , 8. Foll tumours –11.The transcriptomic patterns observed across subtypes has given us insight into the molecular complexity and inherent alterations in tumour cells modelling the breast cancer heterogeneity and unpredicted outcome , 13. StrAlthough independent cohorts attempted to identify molecular subtypes, the chosen microarray-based Single Sample Predictor (SSP) model revealed unreliable assignments and modest agreement between studies , 22. In In this report, we focus on the use of a ranking feature method based on the newly proposed CM1 score  to identhttp://www.ebi.ac.uk/ega/, under accession number EGAS00000000083. It consists of transcriptomic information (cDNA microarrays profiling) processed on the Illumina HT-12 v3 platform (Illumina_Human_WG-v3), as described in [2-normalised gene expression values of primary tumours were divided into two subsets by METABRIC: discovery (997 samples) and validation (989 samples), which were respectively used as training and test sets in our experiments. The original study collected and analysed data under the approval of the ethics Institutional Review Board (details in [The METABRIC microarray data set used in this study is hosted by the European Bioinformatics Institute (EBI) and deposited in the European Genome-Phenome Archive (EGA) at ribed in . The logtails in ). The ushttp://rock.icr.ac.uk/, under data source access GSE47561. This source integrates ten data studies  performed on the Affymetrix Human Genome U133A Array (HG-U133A) platform. The matrix contains log2 RMA re-normalised gene expression data in a unique comprehensive report of 1570 samples. Thus, the GSE47561 data set was used as a second validation set to test our method.The second data set is publicly available in ROCK online portal  at http:In brief, both METABRIC and ROCK data sets have information on patients’ long-term clinical and pathological outcomes, including the sample assignment into intrinsic subtypes  according to the PAM50 method . The METIn this study, we propose a systematic approach that aims at improving breast cancer subtype prediction. The systematic approach is built based on feature selection and data mining concepts. We first compute the CM1 score—using the microarray mRNA expression values—to rank the whole set of probes based on their discriminative power across breast cancer subtypes. We then select the top 10 probes that best represent each intrinsic subtype. The quality of this selection is assessed using a set of classifiers from the Weka software suite with the METABRIC and ROCK data sets, followed by the statistical analysis. The process flow is depicted in X and Y be a partition of a set of samples into two classes, with X the class of interest and Y the remaining classes. A sample either belongs to X or to Y. For each probe i we compute the CM1 score as:i for samples in class X, i for samples in class Y; max{yi} and min{yi} are the minimum and maximum expression values of the probe i for samples in the class Y. X and Y. The normalisation is proportional to the range of values in Y.The CM1 score is a supervised univariate method used to measure the difference in expression levels of samples in two different classes . In thisTo define the most discriminative probes for each breast cancer subtype , we computed the CM1 score for each of 48803 probes taking the subtype of interest and the remaining ones. This results in 5 lists of 48803 CM1 scores.balanced top ten in this paper. Collecting the balanced top ten lists of all subtypes leads to a new set of 42 unique Illumina probes, meaning that 8 probes appear in multiple subtypes. This list is hereafter called the CM1 list.Considering the fact that Parker et al. (2009)  were abldiscovery set. The new labels were attributed based on the consensus of the majority of the classifiers (i.e. more than 50% percent), and whenever such condition was not achieved samples were marked as inconsistent (INC).The quality of the CM1 list for distinguishing subtypes was assessed using a list of well-known classifiers available in the Weka data mining software suite . It usesA similar approach was performed with the PAM50 list to serve as baseline for comparing the results obtained with the 42 probes from the CM1 list. The 50 genes identified by Parker et al. (2009)  were mapr × c contingency table describing the association between the original labels and those predicted by the majority of classifiers, Cramer’s V measures the level of association between those two nominal variables. The statistic ranges from 0, representing no association between the two variables, to 1, representing complete association. Cramer’s V is computed using N is the number of samples in the data set, and χ2 is Pearson’s chi-squared value.Given a r × c contingency table associating initial and predicted labels, the average sensitivity of a classifier is given by r is the number of classes (subtypes), nii is the number of samples of class i correctly predicted as i, and ni• is the number of samples of class i .The average sensitivity (AS)  was alsoThe consensus of the different classification methods concerning the samples’ labels was measured by the popular interrater reliability metric Fleiss’ kappa , 36. Thes × c contingency table informing how many times each of the c classes were assigned to each of the s samples in the k different sample labellings, the Fleiss’ kappa statistic is computed as defined by nij contains the number of times sample i was assigned label j, ∑jnij = k, and pj = (∑inij)/sk is the probability with which the label j is assigned to a sample.Assuming a poor agreement; (2) values between zero and 0.2 are considered slight agreement; (3) 0.21 ≤ κ ≤ 0.40 is fair agreement; (4) 0.41 ≤ κ ≤ 0.60 moderate agreement; (5) 0.61 ≤ κ ≤ 0.80 substantial agreement; and (6) 0.81 ≤ κ ≤ 1 is regarded as an almost perfect agreement.Kappa values range from division , can be Adjusted Rand Index is a version of Rand index corrected for chance when the partitions are picked at random [r × c contingency table between two labelling R and C, it can be measured by:i ≤ r, 1 ≤ j ≤ c, and nij is an entry of the contingency table representing the number of samples that are in class Ri in the partition R and Cj in the partition C, ni• and nj• are the table’s marginals.The agreement between pairs of sample labellings was also quantified using this metric. It ranges between 0 to 1, where 1 indicates an almost perfect concordance between the two compared bipartitions, and 0 a complete discordance between them. The t random , 39. Givsurvival in the R software [overall survival and disease free survival is limited to only 405, with no specification on the cause of death (i.e. if due to disease or not).The survival analysis for each breast cancer subtype is performed using Cox proportional hazards model from the package software , 41. Onlsoftware : age at CM1 score and ensemble learning. First, we detail the selection of discriminative probes ranked according to the CM1 score; calculated for each of the five breast cancer subtypes. Second, we show the quality of our probes by using 24 classification models based on a 10-fold cross-validation and training-test setting in the METABRIC and ROCK data sets. The same approach is also performed with the list of 50 genes used in the PAM50 method. In addition, statistical analysis are reported to determine the power of both lists on predicting breast cancer subtypes. Finally, we demonstrate the consistency between the new labels assigned with current clinical markers ER, PR and HER2, and survival curves. The step-by-step approach is detailed in the To understand the results described in this section, we introduce the sequence of our approach which combines the CM1 list. Our selection includes 30 novel biomarkers, while the remaining 12 genes are common with the PAM50 list.The CM1 score was applied to rank the set of 48803 probes for each of the five subtypes in the METABRIC discovery data set  and PAM50 (κ = 0.84) lists in the discovery set. In the validation and ROCK sets, on the other hand, labels showed only a moderate agreement for both lists (κ ≃ 0.6). Strikingly, the Fleiss’ kappa between subtypes predicted using the CM1 and PAM50 lists  revealed an almost perfect agreement. This statistical measure confirm our visual analysis of the contingency tables as they find strong relationship across the subtype labels in each data set. A detail of the agreement among classifiers by intrinsic subtype is shown in ; luminal B ; HER2-enriched ; and basal-like  [Given the heterogeneity among breast cancer patients and the intricate assignment of PAM50 labels in the original METABRIC data set, we further investigated whether significant differences exist in the analysis of current clinical markers . Figs , HER2−) .Subsequently, we illustrate the survival curves for all breast cancer subtypes using Cox proportional hazards model, as described in AGR3, HPN, ANKRD30A, AURKB, PROM1, VTCN1, CRYAB, CDK1, CDKN3, SERPINA3, SOX11, TRPV6, CLCA2, MUCL1, COL11A1, DARC, TFF3, IGF2BP3, IL33, SUSD3, PSAT1, and GABRP are reported in different studies associated with breast cancer; however not in the context of subtype differentiation. Noteworthy, the CM1 list revealed a set of probes for which little literature exists in relation to breast cancer subtypes: CDCA5, CCL15, COL17A1, GLYATL2, ROPN1, LINC00993 and C6orf211. Their expression levels vary across different subtypes, and are yet a new avenue to be explored. We also emphasize the 12 common genes  due to their important role for breast cancer disease and intrinsic subtyping.In this study, we exposed the power of the CM1 list for improving the breast cancer subtype prediction in the METABRIC and ROCK data sets. The CM1 score portrayed 30 novel genes as potential biomarkers, along with 12 well-established markers shared between CM1 and PAM50 lists. The 42 biomarkers have a great potential to differentiate breast cancer intrinsic subtypes. Among them, almost perfect agreement between the labellings obtained with the majority of classifiers using both lists; however different from the original labels. In this study, we want to highlight the weakness of relying in a single method to assign subtypes labels, as opposed to the power and robustness of an ensemble learning. We therefore discourage label assignments based on a single classifier and also suggests a thorough review of those intrinsic subtypes given the importance of such data sets to breast cancer research. The results indicate that there is an issue to be considered by researchers when using the original PAM50 labels for analysing data. The use of incorrect labels would lead to a plethora of misguided and misleading results by other investigators that use METABRIC or ROCK data sets.Within the application of an ensemble of classifiers, CM1 and PAM50 lists showed concordant predictive power for disease subtyping. In fact, there was an In spite of luminals sharing the same origin and large molecular commonalities , 228, thESR1 (ER) among the 42 probes, but brings other independent genes that are also relevant for overall predictions. Robust data sets like METABRIC have contributed to the understanding of breast cancer disease in terms of its molecular complexity and intrinsic alterations. The main limitations of the research in the field, nevertheless, is the uncertainty in the exact classification of intrinsic subtypes; over and above the discovery of molecular signatures and standard clinical biomarkers. Under consideration, a consistent taxonomy needs yet to be established prior to implementation in clinical practice. Additional research involving the genome, transcriptome, proteome, and epigenome, will lastly portray a true landscape of subtypes and contribute to breast cancer management.Overall, the new intrinsic subtype labels based on the CM1 list and ensemble learning revealed more accurate distributions of clinical markers  and survival curves, when compared to the original PAM50 labels in the METABRIC cohort and ROCK test set. Interestingly, the CM1 list shows S1 TextThe document shows the CM1 probe list along with an extensive literature review. The 42 CM1 biomarkers revealed a great potential to differentiate breast cancer intrinsic subtypes in the METABRIC and ROCK data sets. The 30 novel markers and 12 well-established genes vary the expression levels across different subtypes. The vast majority has been associated with breast cancer disease, either included or not in the subtyping context.(PDF)Click here for additional data file.S1 FigBox plots illustrating the expression levels for all selected transcripts in the CM1 list in the METABRIC discovery and validation sets, and ROCK test set. The figure shows the probes differential behaviour across breast cancer intrinsic subtypes.(TIFF)Click here for additional data file.S1 TableTable listing the CM1 score used to rank the set of 48803 probes for each of the five breast cancer subtypes in the METABRIC discovery data set. In each case, we selected the top 10 highly discriminative probes .(XLSX)Click here for additional data file.S2 TableTable describing the performance of each classifier on the METABRIC discovery and validation sets, and ROCK test set using the CM1 list. It shows the percentage of correctly, incorrectly and not classified samples, Fleiss Kappa index, Cramer’s V, Average Sensitivity, and other values for classification. The 24 classifiers from the Weka software suite are also listed. The labels predicted by each classifier for all samples using CM1 list are defined as: 1—luminal A, 2—luminal B, 3—HER2-enriched, 4—normal-like, 5—basal-like. Count of predicted labels was obtained with the consensus of the majority of classifiers.(XLSX)Click here for additional data file.S3 TableTable describing the performance of each classifier on the METABRIC discovery and validation sets, and ROCK test set using the PAM50 list. It shows the percentage of correctly, incorrectly and not classified samples, Fleiss Kappa index, Cramer’s V, Average Sensitivity, and other values for classification. The 24 classifiers from the Weka software suite are also listed. The labels predicted by each classifier for all samples using CM1 list are defined as: 1—luminal A, 2—luminal B, 3—HER2-enriched, 4—normal-like, 5—basal-like. Count of predicted labels was obtained with the consensus of the majority of classifiers.(XLSX)Click here for additional data file.S4 TableAmong classifiers using CM1 and PAM50 lists, as well as the agreement for each subtype. The predicted—original are described in the table and contain the agreement between the mostly predicted and initial labels of samples; whereas the CM1—PAM50 show agreement between labels assigned by the majority of classifiers using CM1 and PAM50 lists. The file also has the Jensen-Shannon divergence between two probability distributions. Numbers represent the similarity between subtypes distribution for METABRIC discovery and validation sets, and ROCK test set. The similarity is measured using the square root of the Jensen-Shannon divergence.Table containing the Fleiss’ Kappa agreement of labels for the METABRIC discovery and validation sets, and ROCK test set. It shows the overall agreement (XLSX)Click here for additional data file.


A large number of computational methods have been developed for analyzing differential gene expression in RNA-seq data. We describe a comprehensive evaluation of common methods using the SEQC benchmark dataset and ENCODE data. We consider a number of key features, including normalization, accuracy of differential expression detection and differential expression analysis when one condition has no detectable expression. We find significant differences among the methods, but note that array-based methods adapted to RNA-seq data perform comparably to methods designed for RNA-seq. Our results demonstrate that increasing the number of replicate samples significantly improves detection power over increased sequencing depth. P values generated by statistical modeling.High-throughput sequencing technology is rapidly becoming the standard method for measuring RNA expression levels (aka RNA-seq) . The advThe expression level of each RNA unit is measured by the number of sequenced fragments that map to the transcript, which is expected to correlate directly with its abundance level. This measure is fundamentally different from gene probe-based methods, such as microarrays. In RNA-seq the expression signal of a transcript is limited by the sequencing depth and is dependent on the expression levels of other transcripts, whereas in array-based methods probe intensities are independent of each other. This, as well as other technical differences, has motivated the development of a growing number of statistical algorithms that implement a variety of approaches for normalization and differential expression (DE) detection. Typical approaches use Poisson or negative binomial distributions to model the gene count data and a variety of normalization procedures . Such matrices can be produced from alignment data using tools such as HTSeq /Var[log(y)], where y is the ratio of the normalized counts between two conditions, and this ratio approximately follows a normal distribution; hence, a t-test is used to calculate the P value for DE. limma uses a moderated t-statistic to compute P values in which both the standard error and the degrees of freedom are modified [a priori number of degrees of freedom for the model. The baySeq approach estimates two models for every gene, one assuming no differential expression and a second assuming differential expression using the two sample groups. The posterior likelihood of the model of DE, given the observed data, is used to identify differentially expressed genes. In the PoissonSeq method the test for differential expression is simply a test for the significance of the i γterm , which is evaluated by score statistics. By simulation experiments it was shown that these score statistics follow a chi-squared distribution, which is used to derive P values for DE. All methods use standard approaches for multiple hypothesis correction  with the exception of PoissonSeq, which implemented a novel estimation of false discovery rate (FDR) for count data that is based on permutation.The estimation of the parameters for the respective statistical model is followed by the test for differential expression, the calculation of the significance of change in expression of gene modified . The sta2 transformation of the normalized count values. We expect that normalization will remove variations that are not due to biological differences and hence the resulting clusters will coincide with biological sources. Indeed, all methods achieved perfect separation between sample types for both the SEQC and the ENCODE datasets suggesting that all normalization methods are able to correct for variable sequencing depths  as the true negative set and all others as true positives, we performed a ROC analysis to compare the performance of the various methods in detecting differentially mixed spike-in controls. Overall, all methods performed reasonably well in detecting the truly differentiated spike-in sequences with an average area under the curve (AUC) of 0.78  as the threshold for true differentiation. The AUC values at this cutoff indicate comparable performance among all methods with a slight advantage for DESeq and edgeR . To test the number of false positive predictions from the null models we performed a series of intra-condition comparisons using the SEQC technical replicate samples from each condition . No genes are expected to be differentially expressed in these comparisons and the distribution of P values is expected to be uniform since they are derived from the null model. We note that baySeq was excluded from this analysis since it reports posterior probabilities of a model and not P values, which does not allow us to control it with the same stringency as other methods. We indeed found that the P values for all methods were largely uniform although less so for the lower 25% expressed genes where experimental noise is larger than the expression signal  for the Cuffdiff distribution indicating a large number of false positives. A similar observation was noted by Anders et al.: Cuffdiff had an inflated number of false positive predictions in their null model comparison [A 1 vs Sample A 2, Figure S6 in Additional file A primary goal for any differential expression algorithm is to minimize type I errors, which are incorrect rejections of the null hypothesis l Figure . A noticmparison . This trP value cutoff (or FDR) of ≤0.05. Null model P values were computed from three intra-condition comparisons between replicated samples from the same biological condition . In total, 16,287, 16,286, 1,620 and 12,139 P values were calculated for genes in the 100%, 75%, 50% and 25% read count quartiles, respectively. Hence, every gene has three reported P values from every method representing the three null model comparisons. Note that at the bottom 25% quantile, genes with zero counts were excluded. Although the number of false predictions is below the 5% false discovery rates, the reduced specificity points to inflation of differential expression detection by Cuffdiff. When the comparison was performed with no replicated samples, Cuffdiff's false discovery exceeded 5% where all other methods remained well below this limit.Table P values that are inconsistent with the expression levels. Ideally, for this subset of genes the P values for differential expression should be monotonically correlated with the signal-to-noise ratios in the expressed condition  such that higher ratios will be assigned more significant P values to reflect the confidence in the expression measurement.Almost all RNA-seq experiments include a subset of genes that have no detectable read counts in one of the tested conditions due to very low or lack of expression. In those cases the assessment of differential expression is confounded by the lack of expression signal in one of the tested conditions, which can lead to reduced sensitivity (type II error), or more commonly to P value) variables with the added constraint of maintaining a monotonic dependency  ≤ f(jx)). The results clearly show that the limma and baySeq approaches (and to some extent PoissonSeq) exhibit the desired monotonic behavior between the signal-to-noise and confidence in differential expression as measured by adjusted P values whereas DESeq, edgeR and Cuffdiff have poor correlation between these measures  but there is no correction for wide variations in gene counts among replicate libraries.We evaluated this correlation using pair-wise comparisons among the three ENCODE datasets. We performed an isotonic regression that models the relation between predictor  and response . We defined the true set of DE genes as the intersection of the DE genes identified by DESeq, edgeR, limmaVoom and baySeq using the full-size libraries and all five replicates. We then evaluated DESeq, edgeR, limma and PoissonSeq using a decreasing number of replicates and sequence depth, by calculating their: i) sensitivity rates, measured as the fraction of the true set, and ii) false positive (FP) rates, defined as the number of genes identified only by the evaluated algorithm. This analysis was performed on both the SEQC technical replicate samples and the ENCODE biological replicate samples.As expected, all methods had a smaller number of FPs with increasing number of replications and increased sequencing depths although there are noticeable differences between the methods. limmaQN and edgeR had the lowest rates of FPs whereas DESeq had the highest Figure . InteresSensitivity rates also improve significantly with increased sequencing depth and number of replicates although, here as well, significant variability exists between methods and between expression levels Figure . SurprisTaken together these results lead to two conclusions. First, the number of replicate libraries has a greater effect on DE detection accuracy than sequencing depth. This is true for both technical and biological replicates. Second, DE detection of lowly expressed genes is most sensitive to the number of reads and replication whereas there is little benefit to increasing sequencing depths for detecting DE in highly expressed genes.In this study we performed a detailed comparative analysis of a number of methods for differential expression analysis from RNA-seq data. For the various methods, our comparison focused on the performance of the normalization, control of false positives, effect of sequencing depth and replication, and on the subset of gene expressed exclusively in one condition. In contrast to other approaches, which rely on simulated data generated by specific statistical distribution or limited experimental datasets ,33,34, wSurprisingly, the limma package, which was developed and optimized for expression array analysis, had comparable, and by some measures improved, performance for both normalization versions tested relative to the other models, which were tailored for RNA-seq analysis. Furthermore, the difference between quantile normalization or the RNA-seq specific voom function in limma was evident in the number of false DE genes in the null model and in the sensitivity to the sequencing depth and number of replicated samples. limma models the data as a normal distribution, which is a reasonable assumption for array intensities but perhaps counterintuitive for count data since it models discrete data with a continuous distribution. However, it is plausible that in the limit of large counts it is more important to model the variance accurately than the discreteness. This study demonstrates that for datasets with a large number of genes (or tags), the limma package is well suited for detecting DE genes and that modeling gene count data as a log normal distribution, with the appropriate pseudo counts, is a reasonable approximation.The results from sequencing depth and replication analysis demonstrate conclusively that the number of sample replicates is the most significant factor in accurate identification of DE genes . This isThe emergence of RNA-seq as the method of choice for transcriptional profiling has motivated the development of a growing number of algorithms for normalization and analysis. This comparative study is the first exhaustive comparison of the widely used DE methods on experimental data. It provides important guidelines for evaluating RNA-seq analysis methods and points the direction for future improvements.A contain the Strategene Universal Human Reference RNA (UHRR), which is composed of total RNA from ten human cell lines, with 2% by volume of ERCC mix 1. The second group of samples B contains Ambion's Human Brain Reference RNA (HBRR) with 2% by volume of ERCC mix 2. The ERCC spike-in control is a mixture of 92 synthetic polyadenylated oligonucleotides, 250 to 2,000 nucleotides long, which are meant to resemble human transcripts. The two ERCC mixtures in groups A and B contain different concentrations of four subgroups of the synthetic spike-ins such that the log expression change is predefined and can be used to benchmark DE performance (see the Methods section in main SEQC publication). Four replicate libraries from groups A and B were prepared by a single technician and a fifth sample was prepared by Illumina for a total of ten libraries. All libraries were sequenced as paired-end 100 bases in the Epigenomics Core facility at Weill Cornell Medical College with a full block design on two flow cells on a single HiSeq2000 instrument (GEO accession GSE49712). We note that these samples are considered technical replicates and therefore represent an idealized scenario of minimal variation.In this study, we used samples from two sources that were part of the SEQC study, each generated from a mixture of biological sources and a set of synthetic RNAs from the External RNA Control Consortium (ERCC) at known concentrations. The samples from group ENCODE Biological replicate datasets were generated by the ENCODE project  and the All sequenced libraries were mapped to the human genome (hg19) using TopHat(v.2.0.3)  with theP values for multiple hypothesis corrections were used as calculated by the methods. The following are the details for each package used in this study:With the exception of Cuffdiff, all differential expression analysis was performed using the same gene count matrix output from HTSeq. Analysis followed the procedures and steps described in the package documentation and unless stated otherwise default parameters were used in all function calls. Adjusted • DESeq (v.1.10.1): The dispersion estimate call to estimateDispersions had parameters: 'method="per-condition"' and 'fitType="local"' and for null model evaluation with no replicates 'method="blind"', 'fitType="local"' and 'sharingMode="fit-only"'.• edgeR (v.3.0.2): In the null model comparison with no replicates the common.dispersion value was set to 0.4 as suggested by the documentation.• PoissonSeq (v.1.1.2): No minimum expression mean was applied and the number of permutations was 500.P values.• baySeq (v.1.12.0): Sequence length correction was added to the normalization as suggested in the documentation. Negative binomial parameter estimation was performed using getPriors.NB using quasi-likelihood estimation. Note that baySeq reports posterior probabilities for differences between two models and not 2 transformed gene counts (with the addition of 1 to avoid a log of 0) by normalizeBetweenArrays function (known as limmaQN). In the second mode, counts were normalized using the voom function where library sizes were scaled by edgeR normalization factors and the mean-variance trend was calculated using LOWESS regression (known as limmaVoom). Note that limma does not allow contrasting libraries with no replication and therefore limma was excluded from the single library comparisons.• limma(v.3.14.1) Analysis was performed in two modes, which have different normalization procedures. Quantile normalization was performed on the log• cuffdiff (v.2.0.0 (3365)) with the options: '--no-update-check --emit-count-tables' and GTF file $SEQCLIB/hg19_150_ERCC.gtf.A with the five replicates from type B. In the null model comparison two models were tested, with replication and without replication. In the replication model, replicates from the same samples were contrasted: {A1, A2} vs {A3, A4}, {A1, A2} vs {A3, A4, A5} and {B1, B2} vs {B3, B4}. Comparisons without replication were performed between the following samples: A1 vs A2, A3 vs A4, B1 vs B2 and B3 vs B4.For each method, comparisons were performed between the five replicates from sample type 2 transformed after addition of pseudo counts. For counts produced by HTSeq the pseudo counts were set to the smallest non-zero gene count in each library and for FPKM data the pseudo count was set to 0.001. Clustering was performed using the R hclust function with the Euclidean distance measure.Normalized counts were logp1 = 0.5 for retaining each read. We then subsampled the resulting files with a probability p2 = 0.8. Therefore, we generated a set that subsampled the original files with a probability p1 × p2 = 0.4 representing 40% sequencing depth. We continued this subsampling cascade, ultimately generating six sets of files with 0.5, 0.4, 0.3, 0.2, 0.1 and 0.05 of the reads sampled from the original files. We then repeated the operation five times, generating five random datasets for each fraction value.To assess the effect of a reduced sequencing depth, we used DownsampleSam, a function from Picard  that ranA compared to all groups of two samples from condition B). We evaluated the DE using DESeq, edgeR, PoissonSeq and limma using the two described modes.For each subsampled fraction, we used the five independent samplings to compute differential expression between every combination of subsets of samples (for example, all groups of two samples from condition The source code and data files are available online .AUC: area under the curve; bp: base pair; CLIP: cross-linking and immunoprecipitation; DE: differential expression; ERCC: External RNA Control Consortium; FDR: false discovery rate; FP: false positive; FPKM: fragments per kilobase of exon per million mapped reads; HBRR: Human Brain Reference RNA; NB: negative binomial; RMSD: root-mean-square deviation; RPKM: reads per kilobase per million reads; SEQC: Sequencing Quality Control; TMM: trimmed means of M values; UHRR: Universal Human Reference RNA.The authors declare that they have no competing interests.DB, FR, RK, YL, MP and AK performed the analysis. CEM and PZ performed the sequencing experiments. DB, NDS and RK led the study and DB, FR, NDS and RK wrote the manuscript.Supplementary figures. All the supplementary figures referenced in the main text. 1 Hierarchical clustering of the SEQC libraries from sample A and B . . . . . . . . . . 3. 2 Hierarchical clustering of the ENCODE samples .................... 4. 3 Dunn clustering validity index............................... 5. 4 Normalized read counts .................................. 6. 5 ROC analysis of ERCC spike-in controls.........................7. 6 Null model p-values distribution without replicate samples . . . . . . . . . . . . . . . 8. 7 Evaluating monotonic correlation between signal-to-noise and p-values in genes expressed in only one condition ............................... 9. 8 Correlation of signal-to-noise ratio and DE p-values from SEQC data set . . . . . . . 10. 9 Methods performances with reduced sequencing depth and number of replicates for detecting DE between GM12892 and H1-hESC............... 11. 10 Methods performances with reduced sequencing depth and number of replicates for detecting DE between H1-hESC and MCF-7.................. 12. 11 Impact of sequencing depth and number of replicate samples on DE detection by DESeq using SEQC data.................................. 13. 12 Impact of sequencing depth and number of replicate samples on DE detection by edger using SEQC data.................................. 14. 13 Impact of sequencing depth and number of replicate samples on DE detection by limmaQN using SEQC data................................ 15. 14 Impact of sequencing depth and number of replicate samples on DE detection by limmaVoom using SEQC data............................. 16. 15 Impact of sequencing depth and number of replicate samples on DE detection by PoissonSeq using SEQC data ............................. 17. 16 Over-dispersion of the ENCODE dataset ........................ 18}.Click here for file


Andropogon gerardii) is a wide-ranging dominant prairie grass of ecological and agricultural importance to the US Midwest while edaphic subspecies sand bluestem  grows exclusively on sand dunes. Relative to big bluestem, sand bluestem exhibits qualitative phenotypic divergence consistent with enhanced drought tolerance, plausibly associated with transcripts of low expression levels. Our dataset consists of RNA-seq read counts for 25,582 transcripts (60 % of which are classified as low-count) collected from leaf tissue of individual plants of big bluestem (n = 4) and sand bluestem (n = 4). Focused on low-count transcripts, we compare alternative ad-hoc data filtering techniques commonly used in RNA-seq pipelines and assess the inferential performance of recently developed statistical methods for DE analysis, namely DESeq2 and edgeR robust. These methods attempt to overcome the inherently noisy behavior of low-count transcripts by either shrinkage or differential weighting of observations, respectively.Differential expression (DE) analysis of RNA-seq data still poses inferential challenges, such as handling of transcripts characterized by low expression levels. In this study, we use a plasmode-based approach to assess the relative performance of alternative inferential strategies on RNA-seq transcripts, with special emphasis on transcripts characterized by a small number of read counts, so-called low-count transcripts, as motivated by an ecological application in prairie grasses. Big bluestem  contains supplementary material, which is available to authorized users. RNA sequencing (RNA-seq) technology has rapidly become the preferred choice for gene expression analysis as it allows for high throughput over a wide range of expression levels . Yet, soData filtering prior to DE analyses was originally implemented in an attempt to control noise and reduce the impact of multiple testing adjustments on power for DE detection by removal of uninformative or weakly expressed transcripts , 6. NeveAndropogon gerardii) and its edaphic subspecies sand bluestem . Big bluestem (BB) is a widely-distributed dominant grass of North American grasslands [th percentile of least relative abundance, accounting for approximately 3 % of total read counts  were randomly partitioned into 2 groups of 2 samples each, yielding a total of 3 null plasmodes. In turn, performance under the alternative hypothesis was evaluated using DE plasmodes, that is, modified null plasmodes for which one of the groups had a known proportion of transcripts “spiked” with estimates of effect sizes of transcripts called DE from a preliminary analysis [Given the benchmark status of BB as a widely distributed dominant prairie grass, we generated all plasmodes using BB samples only. To evaluate inferential performance of statistical methods under the null hypothesis, samples of individual BB plants /(323 + 132)) of such transcripts as DE   and of rCriteria for data filtering varies across strategies. Specifically, the RP filtering approach indicates removal of a transcript if the overall number of samples with mapped reads present for that transcript (i.e. samples with read counts greater than zero) is smaller than the number of samples per treatment group (i.e. 4 in this case) . In turnNext, filtered datasets were subjected to DE analysis using edgeR robust and DESeq2, as described in the previous section. Tables Figure The most recent release of the DESeq package, namely DESeq2 , implemeMotivated by our interest in low-count transcripts, we further compared the relative performance of DESeq2-based LRT and Wald tests for DE inference on individual transcripts. Figure In this study, we used plasmodes generated from RNA-seq data to compare inferential performance of statistical methods for differential expression, with a special focus on low-count transcripts, as motivated by an application on bluestem prairie grasses. More specifically, we evaluated the inferential performance of the recently developed statistical methods DESeq2, edgeR and edgeR robust, the latter specified over a range of the DF parameter. We also considered data filtering strategies that, while pervasively implemented in RNA-seq data pipelines, impose arbitrary criteria for data exclusion, the impact of which on DE inference was shown to be substantial.This study is one of few to use a plasmode-based approach to compare statistical methods on a specific RNA-seq dataset and more specifically, on a subset of transcripts characterized by low expression levels . We recoOur results on inferential performance indicated adequate control of Type I error within nominal levels using either DESeq2 or edgeR robust both for all transcripts and for low-count transcripts. Still, false positive rates increased with greater degrees of freedom under edgeR robust, indicating the need for careful consideration of how this parameter is specified. In turn, edgeR robust showed greater power than DESeq2, which is consistent with results from previous simulation studies  and is hOur results from the plasmode-based approach to assess inferential performance suggest that the specification of DF for edgeR robust can impact DE inference of RNA-seq data, particularly that of low-count transcripts and thus, should be considered carefully. Most relevant to our dataset, the default DF specification (i.e. DF = 10) was not optimal and led to a decrease in inferential precision and accuracy relative to using an estimated DF value. The default value of the shrinkage parameter for edgeR robust (i.e. DF = 10) seems to be based on an assortment of simulation studies . HoweverData filtering is a common processing step in the RNA-seq data management pipeline , though Overall, the rationale for arbitrary filtering RNA-seq data based on either a RP criterion or a CPM criterion seems poorly justified, either biologically or otherwise, particularly given the availability of powerful state-of-the-art statistical methodology developed to deal with the associated challenges in RNA-seq data. Instead, researchers may consider using the complete unfiltered RNA-seq data for DE analyses, ensuring use of modern statistical methods to properly borrow information across transcripts and moderate (i.e. shrink or weigh) DE inference based on expression levels. In particular, DESeq2 and edgeR robust have shown promising inferential performance in handling low-count transcripts with minimal effect on the DE analysis for the remaining transcripts. Further, forgoing the use of data filtering at arbitrary thresholds in favor of more elegant approaches to deal with the inherent challenges of RNA-seq data may be particularly relevant for research questions focused on biologically relevant transcripts characterized by low expression levels, such as transcription factors.Finally, it is of concern that differential expression assessments for low-count transcripts based on the Wald test implemented by default in DESeq2 yielded more liberal results relative to those based on a likelihood ratio test. Both tests assume that certain regularity conditions hold, though such conditions are rarely verified in practice . It is fWe implemented a recently adapted plasmode-based approach to compare inferential performance of modern statistical methods, namely DESeq2 and edgeR robust, on RNA-seq data. Motivated by interest on a transcriptomic comparison of bluestem grass species, we pay special attention to transcripts of low expression levels, defined here as low-count transcripts. We emphasize that implications of these results may be relevant to other biological applications that involve transcripts of high biological importance but low expression levels, such as transcription factors. Both DESeq2 and edgeR robust seemed to properly control family-wise type 1 error on all transcripts as well as on low-count transcripts. For low-count transcripts, edgeR robust showed greater power whereas DESeq2 showed greater precision and accuracy. Overall, both methods showed promising inferential performance on low-count transcripts and yielded a substantial amount of overlap in DE calling, thus supporting their combined use for fine-tuned DE inference. Still, a note of caution is in order regarding the approximate nature of DE tests on individual transcripts, particularly low-count ones.Regarding edgeR robust, the specification of a degree of freedom parameter was found to be non-trivial. This is to be expected as degrees of freedom determine the amount of shrinkage and borrowing of information across transcripts, thereby impacting precision and accuracy of DE inference. Our results raise legitimate questions about the use of a default value for the degree of freedom hyperparameter, recognizing that a default value may not be appropriate for all datasets, as it was certainly not optimal in our case study. The edgeR robust degrees of freedom parameter should thus be given careful consideration in any data application and, whenever possible, it should be estimated from the data.Finally, our results support that filtering of RNA-seq data can have serious implications for inference as mostly low-count transcripts are removed from the data and excluded from DE analyses. Standard RNA-seq data management pipelines that call for filtering transcripts out at arbitrary thresholds should be reconsidered. Instead, researchers may implement modern state-of-the-art statistical methodologies specifically developed to deal directly with the inherent challenges of RNA-seq data, including transcripts of low expression levels.Andropogon gerardii, Saline population) and sand bluestem . Phenotypic divergence was established using a species specific, established hybrid index [de-novo reference transcriptome assembly (described next) [RNA was extracted from leaf tissue of 4 individual plants of each of two phenotypically divergent bluestem subspecies, namely big bluestem (id index . All plaed next)  and numbk (23-61) using Velvet v1.2.08 and Oases v0.2.08 [k-mer Illumina assemblies were merged with a k-mer value of 27. The resultant Illumina and 454 assemblies were merged with miraEST v3.4.11 [Tagcleaner v. 0.12  was used v0.2.08 . The sin v3.4.11  to produAndropogon contigs for putative homologs to S. bicolor Phytozome v9.0 proteins and the NCBI nr protein database. The custom scripts Blastx.pl v 1.0 and FindFailed.pl were used to run BLASTX against the NCBI nr protein database. Significant BLASTX hits  were retained. Ortholog Hit Ratio (OHR) [S. bicolor database. Therefore, OHR is an estimate of the percent of the full length protein sequence represented in the assembly. An OHR of 1 indicates a potential full length transcript. In an effort to reduce the influence of redundant or fragmented contigs, the script UniqueBlast.pl [S. bicolor database. Only these contigs were used to calculate OHR.Assemblies were evaluated on the basis of N25, N50, N75, cumulative length of contigs, and number of contigs. BLASTX, NCBI BLAST+ v2.2.28 , was runio (OHR)  was calcBlast.pl  was usedk-mer assemblies and the 454 assembly. Additionally, N-values  are higher for single-k-mer Illumina assemblies  as shown in Additional file de novo grass transcriptomes  and the MIRA assembly of 454 reads . The final merged assembly had the smallest number of sequences  and the shortest contiguous length (~65 Mb). Taken with the increased N-values  in big bluestem. In turn, BB-only transcripts were defined as transcripts with expression levels present in big bluestem and absent  in sand bluestem.th percentile of least relative abundance, which accounted for approximately 3 % of total read counts  with mean μij = sjqij and link function log (qij) = xjβi, where sj is the normalized library size for sample j as previously defined [αi is the variability between samples for transcript i, xj contains the elements of the known design matrix for sample j, and βi describes the corresponding coefficient for transcript i. Estimation of the dispersion parameter is conducted in 3 steps [μij and high dispersion αi in small datasets are pulled more strongly toward zero. Default DE testing on the shrunken LFCs is based on a Wald test, whereas a LRT alternative is also available [The R package DESeq2  for whic defined . In turnYij for transcript i in sample j is described with a generalized linear model of the Negative Binomial family with logarithmic link, such that Yij ~ NB with link function log (μij) = Xβi + log (Nj), where X is the design matrix containing the covariates, βi is a vector of regression parameters, Nj is the library size for sample j, and ϕi is the square of the biological coefficient of variation for transcript i. Dispersion parameters are estimated as follows. First, initial gene-wise dispersion is estimated using adjusted penalized likelihood. These estimates are then moderated by shrinkage towards a common dispersion estimate obtained by maximizing a common likelihood function. Shrinkage is determined by a prior degree of freedom parameter afforded to the shared likelihood and specified arbitrarily by the researcher [βi are estimated using maximum likelihood that incorporates working weights attached to each observation. Weights are attached to each observation so observations that deviate strongly from model fit are given a lower weight. Observations weights are defined as functions of a Pearson residual [R package edgeR robust  for whicsearcher . Unless searcher . In turnresidual  that areAs previously indicated, edgeR robust uses DF = 10 as a default to specify the amount of shrinkage applied to dispersion parameters . While tWe compared the performance of edgeR robust at varying DF specifications. In particular, we considered DF = 4, 10 and 50, to indicate a range of shrinkage around the default specification. Further, we considered using the classical edgeR software  to estimFollowing DE analyses based on either DESeq2 or EdgeR robust, transcripts were called DE based on a FDR = 0.05 using the Benjamini-Hochberg procedure .Table Performance metrics were computed on each plasmode dataset fitted with each statistical method for DE analyses. Each metric was then fitted with a general linear mixed model to compare methods for DE analysis accounting for plasmode dataset as a random blocking factor. Models were fitted using the GLIMMIX procedure of SAS . Residual assumptions were evaluated using studentized residuals. Pairwise comparisons in performance metrics between analyses methods were conducted using a Tukey-Kramer adjustment to prevent the inflation of type 1 error rate.Filtering criteria are often applied to RNA-set datasets prior to DE analyses. A relatively common filtering criterion removes transcripts from a dataset if the total number of samples with mapped reads present for that transcript is smaller than the number of samples per treatment . For our


RNA-Seq is increasingly being used for gene expression profiling. In this approach, next-generation sequencing (NGS) platforms are used for sequencing. Due to highly parallel nature, millions of reads are generated in a short time and at low cost. Therefore analysis of the data is a major challenge and development of statistical and computational methods is essential for drawing meaningful conclusions from this huge data. In here, we assessed three different types of normalization  and evaluated if normalized data reduces technical variability across replicates. In addition, we also proposed two novel methods for detecting differentially expressed genes between two biological conditions: (i) likelihood ratio method, and (ii) Bayesian method. Our proposed methods for finding differentially expressed genes were tested on three real datasets. Our methods performed at least as well as, and often better than, the existing methods for analysis of differential expression. One of the recent methods for gene expression profiling is RNA-Seq. An advantage of RNA-Seq over other gene expression profiling technologies is that it allows a comprehensive assay that does not require probes for targets to be specified in advance. It has particularly been used for de novo detection of splice junctions and allows genome wide expression profiling of organisms with unknown genome sequence .By obtaining millions of short reads from the population of interest and by mapping these reads to the reference genome, RNA-Seq produces read count data. With enough reads from a sample, it has the potential to detect and quantify biologically significant RNAs with low and moderate abundances. Before detecting biologically significant RNAs, systematic technical variations due to experimental variability need to be removed retaining effects resulting from the biological process of interest. This process is also known as normalization. Various procedures for normalization of RNA-Seq have been proposed in literature, such as transcripts parts per million , trimmed t-test and alternatively also applied a Bayesian model of Vêncio et al. . We extended the above results when the condition is replicated thrice andλ, we getλ. This givesm times, we consider the following probability. a, b such thatxm of the mth replicate lies in the interval  then we conclude with probability (1 − 2α) that there are no systematic differences between the replicates. Similarly, the results can be derived for n replicates of a gene in condition-2 . For Identifying differential expression across two conditions, define x1 = ∑j=1mxj, y1 = ∑j=1nyj to be independent Poisson random variables with parameters mλ and nμ, respectively, and use (m = n).  Back in 1997, the method of Audic and Claverie was used to establish the probability distribution governing the occurrence of the same rare event in repeated experiments and was applied for the analysis of digital gene expression profiles. It was then described for only 2 replicates which we have attempted to extend to 3 or more replicates and apply to RNA-Seq data. As defined before, uch that∑xp=0ap~ genes: Likelihood ratio method and Bayesian method on Datasets For comparison between any two biological conditions, the read count values from the conditions can be categorized under three categories. (1) When both conditions have zero count. In this situation, nothing can be said about differential expression between the two conditions. (2) When one sample has zero or low counts and a reasonable count in the other. This is an interesting biological phenomena where a gene is not expressed in one of the conditions. (3) When both the conditions have reasonable count. We shall evaluate the performance of our methods based on second and third category.P value 0.01 and EdgeR identified 4 miRNAs as DE with P value 0.01. Similar analysis was performed for Normal versus K562 and globin reduced versus nonreduced samples. See Additional file 1, 2, and 3 in supplementary material available online at doi:10.5402/2012/817508 for detailed analysis and For the quantile normalized Normal versus HL60 data , 19 miRNFrom Additional file 1 in supplementary material, it is clear that likelihood ratio method and Bayesian method give very similar results for Normal versus HL60 and Normal versus K562 datasets . Both meP value 0.01 and EdgeR reports 2987 genes with P value 0.01. From these numbers alone, it is difficult to comment on the performance of any method. For globin reduced versus non-reduced data , likelih We assessed three different types of normalizations and showed that though Illumina data is highly replicable before normalization, normalization further reduces the technical variability, likelihood ratio method was used to statistically evaluate variation across replicates. We also presented two methods for finding differentially expressed genes for RNA-Seq data with or without replicates, likelihood ratio method is a general method that does not impose any restriction on the equality of the number of replicates across the two conditions. Bayesian method on the other hand can only be applied if there is equality on the number of replicates for the two conditions being compared. The performance of both the methods was compared to DESeq, EdgeR. For small RNA dataset, likelihood ratio method and Bayesian method perform similarly but better than EdgeR and DESeq. For For both likelihood ratio method and Bayesian method, we assume that the underlying distribution for observed number of reads to be Poisson. Poisson distribution is intuitively appealing and mathematically easy to handle but with a limitation that the mean and variance of Poisson random variable are the same. To avoid this, authors generally assume negative binomial distribution instead of Poisson. However, the efficiency of the proposed methods in identifying differentially expressed genes, their mathematical convenience, and simplicity should make these methods extremely useful.Supplementary Table 1 presents the differential expression analysis for Normal vs HL60 data using Likelihood ratio method, Bayesian method, DESeq and EdgeR.Supplementary Table 2 presents the differential expression analysis for Normal vs K562 data using Likelihood ratio method, Bayesian method, DESeq and EdgeR.Supplementary Table 3 presents the differential expression analysis for six globin reduced vs six non reduced samples using Likelihood ratio method, Bayesian method, DESeq and EdgeR.


Sequencing-based molecular characterization of tumors provides information required for individualized cancer treatment. There are well-defined molecular subtypes of breast cancer that provide improved prognostication compared to routine biomarkers. However, molecular subtyping is not yet implemented in routine breast cancer care. Clinical translation is dependent on subtype prediction models providing high sensitivity and specificity. In this study we evaluate sample size and RNA-sequencing read requirements for breast cancer subtyping to facilitate rational design of translational studies. We applied subsampling to ascertain the effect of training sample size and the number of RNA sequencing reads on classification accuracy of molecular subtype and routine biomarker prediction models (unsupervised and supervised). Subtype classification accuracy improved with increasing sample size up to N = 750 (accuracy = 0.93), although with a modest improvement beyond N = 350 (accuracy = 0.92). Prediction of routine biomarkers achieved accuracy of 0.94 (ER) and 0.92 (Her2) at N = 200. Subtype classification improved with RNA-sequencing library size up to 5 million reads. Development of molecular subtyping models for cancer diagnostics requires well-designed studies. Sample size and the number of RNA sequencing reads directly influence accuracy of molecular subtyping. Results in this study provide key information for rational design of translational studies aiming to bring sequencing-based diagnostics to the clinic. Over the past decade, genetic analysis has revealed a detailed molecular portrait of breast cancer and established it as a heterogeneous disease2et al.68910111314151718Breast cancer is the second most common cancer worldwide and a leading cause of cancer-related mortality in women. It has been estimated that in 2012 alone there were 1.67 million new cases and 522,000 deaths attributed to breast cancerGene expression-based molecular subtyping has been applied to characterise tumors in multitude of cancer types, including prostate cancer202342829In this study we investigate the dependency of prediction accuracy on the training sample size (learning curve) using The Cancer Genome Atlas (TCGA) Data setTo assess the effect of sample size on subtype classification, we examined how the performance of three different classification models  varies as a function of the training sample size. From the TCGA Breast cancer data we created two datasets: PAM50 dataset, consist of only PAM50 features and Top25 dataset consist of top 25% features of highest variation. Sample size effect was evaluated for both datasets. We observed a gain in performance of classifiers as the training sample size increases. However, the improvement in performance depends on the chosen classification model and also the dataset. The performance of Elastic net and Random forest classifiers improved with increasing training sample size, which is typically expected in classification problems with noisy data and not perfect class separation . HoweverWhen we investigated the performance gain across both datasets (PAM50 and Top25), we observed that the dataset with a large number of features (Top25) made a relatively larger increase in classification performance as a function of increasing sample size compared to the small PAM50 dataset. A likely explanation of this difference is the fact that the PAM50 feature is a carefully selected set of genes that are informative for breast cancer subtype classification. We also note that the subtype labels used for benchmarking are based on the PAM50 gene panel.For both regularisation and ensemble based classification, classification accuracy increases more rapidly as a function of the training sample size up to around N = 350. Beyond N = 350 observations, the increase in classification performance is more modest as learning curves start achieving a plateau.To assess the effect of sample size on receptor status prediction from RNASeq data we examined ER, PR and Her2 receptor status classification performance. 6 reads per sample, the increase in classification accuracy is not improving, suggesting that subtype classification can be accomplished at this read count. It is important to note that the read counts referred to here are read counts that are mapped correctly and annotated as protein coding. For the Top25 dataset, the increase in classification accuracy starts to stagnate at 2 × 106 reads per sample  in the molecular profiling data. In the case of “easy” classification problems, where there are large molecular differences between the groups, smaller training datasets are required to estimate a good model, while in more challenging classification problems, i.e. with smaller effect sizes and more subtle differences between groups, larger training data sets are required. In biomedical applications the study sample size is usually limited by either economical considerations or by the availability of biological material, e.g. tumour samples, biopsies or other biological samples, or by both. Study design is therefore a key consideration to ensure studies with sufficient sample sizes to produce the required results, which in this case would be defined by models that can be used in the clinical context.Previous studies of sample size effects on classification models with molecular predictors were based on microarray-based gene expression profiling data and smaller sample sizesIn the case of breast cancer, we found that the prediction accuracy is improving as a function of sample size for both molecular subtyping as well as for prediction of routine markers throughout the sample size range we have investigated (N = 100 to N = 750). However, the improvement is more dramatic up to around N = 350 in the case of subtyping, after which the rate of improvement in accuracy is slowing down. i.e. the cost of improving e.g. 1% in accuracy is gradually increasing as the sample size gets larger.The unsupervised molecular subtype problem is a harder one than supervised learning. This was indicated by the substantially lower classification performance in the unsupervised case. A large part of the misclassification can, however, be attributed to samples within the luminal group that gets misclassified between the LumA and LumB groups.It is important to realise that the quality of the clinical data also plays an important role for the classification accuracy that can be achieved. In retrospective material and biobanks, there is almost always a degree of mislabelling, errors in clinical meta data and other issues that will put an upper bound on how good models can be developed.We also investigated the impact of RNAseq read depth, or rather the number of RNAseq reads, on prediction accuracy. For subtype prediction we found that the prediction performance did not improve as the number of mapped reads of protein coding transcripts reached 5 million. However in the case of the Top25 dataset prediction performance stagnates at about 2 million reads. It is important to realise that this figure is only valid for the particular challenge of predicting the intrinsic subtype, and not for other scientific questions that might be put forward. We also expect that the number of reads that are required will increase as the number of subtypes gets larger in alternative subtype modelsSequencing-based molecular characterisation of breast cancer tumours is expected to provide direct patient benefits due to a more precise characterisation of the molecular aetiology of the tumour and will lead to improved prognostication, patient stratification and ultimately a more personalised treatment. The results presented here provide guidance for study design of translational breast cancer genomics studies aiming to implement models for prediction of routine markers and for molecular subtyping.https://tcga-data.nci.nih.gov/tcga/) on 11th of December 2013 and included data for 1148 cases. Unaligned RNAseq data from the TCGA dataset was downloaded (June 2014) after approval from the TCGA data access committee . 1073 cases were available with both unaligned paired-end RNAseq data and clinical data. Out of these, 35 observations were excluded as potential outliers based on inspection of Principal Component Analysis scores and residuals. In the remaining data, 885 samples had molecular subtype (PAM50) assignments available and each has 20477 gene expression features. The dataset has five different subtypes: Basal (=132), Her2 (=65), LumA (=393), LumB (=190), Normal (=105). We calculated the variance for each feature and selected the top 25% features. This dataset, containing 885 samples and 5120 features, was called Top25 dataset. We created another dataset, including PAM50 features only and named it PAM50 datasetIn this study we utilize publicly available TCGA RNAseq and clinical data, in accordance with the TCGA data access and publication guidelines. For all samples materials and clinical data, informed consent and ethics approval have obtained by TCGA, based on guidelines laid out by the TCGA Ethics, Law and Policy Group. Clinical data from the TCGA invasive breast carcinoma dataset  was downloaded from the TCGA data portal (Standard Illumina adapters (AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC and AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA) were trimmed using skewer version-0.1.117We examined three different multivariate prediction models in combination with the previously mentioned datasets: Elastic net43The Random forest method is based on an ensemble of decision trees. It constructs a large set of decision trees using bootstrap sampling and random subset of the variables. The outcome is predicted by averaging over the trees. During the training, trees are not pruned which creates trees with low-bias. The decision tree method is known to over-fit, while Random forest overcomes this problem by baggingThe Nearest Shrunken Centroid method is a modification of the standard nearest centroid approach. NSC has been extensively used for microarray data analysis43N, for each prediction model the following parameters were optimised: for Elastic net alpha and lambda were optimised; for Random forest models the number of trees and the number of variables per level were optimised and for Nearest shrunken centroid models the parameter shrinkage threshold was optimised.Given a training dataset of size To estimate prediction performance we apply cross-validation. Since we have to optimise model parameters empirically while also estimating prediction performance, we applied a nested cross-validation approach. For a cross-validation cycle the procedure is as follows: In the first step, the test dataset which contains 10% original data is selected using a stratified, i.e. class-balanced, random sampling procedure. Thus, in the test dataset, subtype classes are in the same proportion as in the original dataset. This dataset is not used during any training and optimisation process and thus can be considered as an external test dataset in each “outer” cross-validation round. From the remaining dataset, we selected the training sets of different sample sizes ranging from 100 to 750 samples. This selection of different sample sizes training dataset is also carried out using stratified random sampling procedure. The training dataset is used in parameter optimisation for the prediction models, while the prediction performance is evaluated based on the “outer” test dataset and the optimised prediction model.To assess effects of RNA read count on classification performance, reads were sub-sampled at different read count per sample and subsequently normalised. Performance of optimised prediction models was assessed using overall classification accuracy and class balanced classification accuracy in the multi-class case. Accuracy is defined as number of correctly predicted cases across all classes, divided by the number of total cases. Given the high degree of class imbalance in the dataset, the overall accuracy may lead to erroneous interpretationTo assess the effect of training sample size on receptor status prediction, we analyse three clinicopathological relevant receptors, estrogen receptor (ER), progesterone receptor (PR) and Her2 receptor (Her2). The predictions were made using a logistic regression model with the clinical receptor status (positive/negative) as the response variable and using the expression level of ESR1 as the predictor in the ER model, expression of PGR as the predictor in the PR model and expression level of ERBB2 as the predictor in the Her2 model. For each cross-validation round 10% of the dataset is selected as test dataset using a stratified random sampling procedure. From the remaining data, training sets of different sample sizes were selected. The class probabilities were estimated for the training sets by applying logistic regression model and were used for calculating receiver operating characteristic (ROC) curve. An optimal cut-off point for receptor status was calculated from the ROC curve and Youden’s index method. Using the optimal cut-off point, labels for the test dataset were predicted and accuracy was calculated.et al.SigClust methodGenefu packageIn the unsupervised setting we started with molecular profiling data with the aim of inferring distinct subtypes of cases through clustering. Here, we applied the approach originally proposed by Parker How to cite this article: Mer, A. S. et al. Study design requirements for RNA sequencing-based breast cancer diagnostics. Sci. Rep.6, 20200; doi: 10.1038/srep20200 (2016).


A software implementation is available from http://www.bioconductor.org/packages/release/bioc/html/quantro.html.Normalization is an essential step in the analysis of high-throughput data. Multi-sample global normalization methods, such as quantile normalization, have been successfully used to remove technical variation. However, these methods rely on the assumption that observed global changes across samples are due to unwanted technical variability. Applying global normalization methods has the potential to remove biologically driven variation. Currently, it is up to the subject matter experts to determine if the stated assumptions are appropriate. Here, we propose a data-driven alternative. We demonstrate the utility of our method (The online version of this article (doi:10.1186/s13059-015-0679-0) contains supplementary material, which is available to authorized users. Technical variation can cause perceived differences between samples processed on high-throughput technologies, irrespective of the biological variation. These differences are typically due to changes in experimental conditions that are hard or impossible to control [Multi-sample normalization techniques such as quantile normalization , 2 have  control  and conf control , 5.global adjustment methods [targeted set of genes) are expected to be differentially expressed. However, if, for example, a substantially higher percentage of genes are expected to be expressed in only one group of samples, it may not be appropriate to use global adjustment methods.Some of the first attempts at normalizing microarray data mimicked the use of so-called house-keeping genes  as was d methods . ExampleQuantile normalization was originally developed for gene expression microarrays , 2 but tquantro, for the assumptions of global adjustment methods, such as quantile normalization, that tests for global differences in distributions between groups of samples. Our test uses the raw unprocessed high-throughput data as input to calculate a test statistic comparing the variability of distributions within groups relative to between groups. If the variability between groups is sufficiently larger than the variability within groups, then this suggests there may be global differences in distributions between groups of samples and global adjustment methods may not be appropriate. We demonstrate the advantages of our method by applying it to several gene expression and DNA methylation datasets with targeted and global changes in distributions  samples in each of the k ∈  groups  from a gene expression or DNA methylation experiment. We assume Xik has some common distribution (Xik ∼ ℱk) where ℱk is the theoretical distribution for the kth group. We define ith sample in the kth group. As a first step, we use an ANOVA to test if the average of the medians of the distributions are different across groups and median normalize the samples accordingly. Let kth group and let Consider a set of raw high-throughput data total variance of the distributions as the sum of squared differences between Fik− 1 and p = 2) as:To quantify the differences between two distributions, we use Mallow’s distance , which itotal variance can be decomposed  and the variance within groups (SSwithin):The Fquantro, to test for global differences in the distributions between the K groups. The null hypothesis is that there are no global differences in the distributions between the groups and the alternative hypothesis is that at least one group is different from the rest.We propose using a data-driven test statistic, referred to as If there are no global differences in the distributions between the groups , we can apply a global adjustment method, such as quantile normalization, to remove any unwanted technical variation. If there are global differences in the distributions between the groups, quantile normalization may not be an appropriate normalization technique depending on the source of variation .Fquantro test statistic  to the mean squared error within groups (MSwithin):The Fquantro and reject the null hypothesis if the p value  population stratified by genotype based on an expression quantitative trait loci (eQTL) , prostate (p < 0.001), thyroid (p < 0.001), stomach (p < 0.001) and liver tissues (p = 0.044) , quantro and quantile normalization are comparable in the number of false discoveries , but if there are global differences in the distributions between groups , quantro is able to detect those global differences and reduce the number of false discoveries compared with quantile normalization   in detecting true differences between groups of samples because it removes unwanted technical variation  and biological variation. Data alone cannot determine if global changes are caused by technical variation or biological variation  or to platform-dependent technical variation. Because global changes in distributions between groups can be caused by both technical variation and biological variation, it is important to note that our test statistic ion Fig. , but quaquantro assess the assumptions of global normalization methods, it is important to note other global normalization methods such as VSN [quantro, is not a normalization method, but rather it tests the assumptions of global normalization methods that assume there are no global differences in the distributions to guide the choice or whether or not global normalization methods are appropriate.As h as VSN , Loess [h as VSN  and the h as VSN  make difh as VSN . For exaglobal adjustments may not be appropriate depending on the type and source of variation. If global adjustment methods are not appropriate, other methods, such as application-specific methods [Here, we have shown if there are global changes in the distributions across a set of groups, normalization methods with  methods , can be  methods  and PEER methods , are exa methods , are simPrevious studies have evaluated and discussed normalization methods with and without global adjustments , 32, 36,global adjustments are widely used for data analysis in genomics, but rely on assumptions about the data generation process that are not appropriate in certain contexts. To the best of our knowledge, there is no quantitative method available to assess if the stated assumptions are appropriate or not, leaving the decision up to subject matter experts. Our method is the first to provide a data-driven solution to test the assumptions of global normalization methods. We have demonstrated the utility of our method by applying it to several gene expression and DNA methylation datasets, revealing examples of both targeted and global changes in distributions across groups, such as the global changes in distributions detected between the gene expression of brain and liver tissues. We demonstrated that global normalization methods can lead to increased bias and MSE in downstream analyses when there are global differences in distributions and quantro can detect when global normalization methods are not appropriate, which can prevent removing potentially interesting biological variation. We have implemented our method into the quantro R-package providing researchers a tool to test the assumptions of global normalization methods in the analysis of their own data.Normalization methods with quantro R-package available on Bioconductor. We used permutation testing to assess the statistical significance of the test statistic and distributed the computations across multiple cores to increase the speed. To test for global differences in distributions between groups of samples from high-throughput data sets, we applied quantro to several publicly available gene expression and DNA methylation data sets. Table S1 in Additional file global changes in the distributions across groups.The method introduced here has been implemented into the affy R/Bioconductor package [2 scale. For the gene expression samples using RNA-Seq, we used the rlogTransformation provided in the DESeq2 R/Bioconductor package [2 scale, which reduces the variability in the low counts, but other transformations can be used such as the Variance Stabilizing Transformation (VST) in the DESeq2 package. The RNA-Seq data were obtained from ReCount [minfi R/Bioconductor package [offset parameter equal to 100.To compare the gene expression on microarrays of cancer samples and brain and liver tissues, we considered multiple studies from GEO  to repre package . To visu package , we divi package  to trans ReCount , which p package . We extrquantroSim  and relative MSE while varying the cutoff threshold from quantro and for a fixed threshold at α = 0.05. We simulated DNA methylation samples with a varying proportion of differentially methylated CpGs between the two groups and a varying level of technical variation  (softwan GitHub .Additional file 1:Supplementary materials are available in a single pdf. All scripts containing the code for these analyses are available on GitHub [n GitHub .