New normal linear modeling strategies are presented for analyzing read counts from RNA-seq experiments. The voom method estimates the mean-variance relationship of the log-counts, generates a precision weight for each observation and enters these into the limma empirical Bayes analysis pipeline. This opens access for RNA-seq analysts to a large body of methodology developed for microarrays. Simulation studies show that voom performs as well or better than count-based RNA-seq methods even when the data are generated according to the assumptions of the earlier methods. Two case studies illustrate the use of linear modeling and gene set testing methods. Gene expression profiling is one of the most commonly used genomic techniques in biological research. For most of the past 16 years or more, DNA microarrays have been the premier technology for genome-wide gene expression experiments, and a large body of mature statistical methods and tools has been developed to analyze intensity data from microarrays. This includes methods for differential expression analysis , random Borrowing information between genes is a crucial feature of the genome-wide statistical methods, as it allows for gene-specific variation while still providing reliable inference with small sample sizes. The normal-based empirical Bayes statistical procedures can adapt to different types of datasets and can provide exact type I error rate control even for experiments with a small number of replicate samples .In the past few years, RNA-seq has emerged as a revolutionary new technology for expression profiling . One comet al. [Later statistical publications argued that RNA-seq data should be analyzed by statistical methods designed specifically for counts. Much interest has focused on the negative binomial (NB) distribution as a model for read counts, and especially on the problem of estimating biological variability for experiments with small numbers of replicates. One approach is to fit a global value or global trend to the NB dispersions ,18,19, aet al.  proposedet al. -26.F-test in place of the usual likelihood ratio test, but this relies on other approximations, in particular that the residual deviances are analogous to residual sums of squares from a normal analysis of variance.Unfortunately, the mathematical theory of count distributions is less tractable than that of the normal distribution, and this tends to limit both the performance and the usefulness of the RNA-seq analysis methods. One problem relates to error rate control with small sample sizes. Despite the use of probabilistic distributions, all the statistical methods developed for RNA-seq counts rely on approximations of various kinds. Many rely on the statistical tests that are only asymptotically valid or are theoretically accurate only when the dispersion is small. All the differential expression methods currently available based on the NB distribution treat the estimated dispersions as if they were known parameters, without allowing for the uncertainty of estimation, and this leads to statistical tests that are overly liberal in some situations ,28. ThisA related issue is the ability to adapt to different types of data with high or low dispersion heterogeneity. None of the empirical Bayes methods based on the NB distribution achieve the same adaptability, robustness or small sample properties as the corresponding methods for microarrays, due to the mathematical intractability of count distributions compared to the normal distribution.The most serious limitation though is the reduced range of statistical tools associated with count distributions compared to the normal distribution. This is more fundamental than the other problems because it limits the types of analyses that can be done. Much of the statistical methodology that has been developed for microarray data relies on use of the normal distribution. For example, we often find it useful in our own microarray gene expression studies to estimate empirical quality weights to downweight poor quality RNA samples , to use For these reasons, the purpose of this article is to revisit the idea of applying normal-based microarray-like statistical methods to RNA-seq read counts. An obstacle to applying normal-based statistical methods to read counts is that the counts have markedly unequal variabilities, even after log-transformation. Large counts have much larger standard deviations than small counts. While a logarithmic transformation counteracts this, it overdoes the adjustment somewhat so that large log-counts now have smaller standard deviations than small log-counts. We explore the idea that it is more important to model the mean-variance relationship correctly than it is to specify the exact probabilistic distribution of the counts. There is a body of theory in the statistical literature showing that correct modeling of the mean-variance relationship inherent in a data generating process is the key to designing statistically powerful methods of analysis . Such vaOur approach is to estimate the mean-variance relationship robustly and non-parametrically from the data. We work with log-counts normalized for sequence depth, specifically with log-counts per million (log-cpm). The mean-variance is fitted to the gene-wise standard deviations of the log-cpm as a function of average log-count. We explore two ways to incorporate the mean-variance relationship into the differential expression analysis. The first is to modify limma’s empirical Bayes procedure to incorporate a mean-variance trend. The second method incorporates the mean-variance trend into a precision weight for each individual normalized observation. The normalized log-counts and associated precision weights can then be entered into the limma analysis pipeline, or indeed into any statistical pipeline for microarray data that is precision weight aware. We call the first method limma-trend and the second method voom, an acronym for ‘variance modeling at the observational level’. limma-trend applies the mean-variance relationship at the gene level whereas voom applies it at the level of individual observations.This article compares the performance of the limma-based pipelines to edgeR ,34, DESeEither voom or limma-trend give RNA-seq analysts immediate access to many techniques developed for microarrays that are not otherwise available for RNA-seq, including all the quality weighting, random effects and gene set testing techniques mentioned above. This article presents two case studies that demonstrate how voom can handle heterogeneous data and complex experiments as well as facilitating pathway analysis and gene set testing.libraries) are available for a set of n RNA samples. Each profile records the number of sequence reads from that sample that have been mapped to each one of G genomic features. A genomic feature can be any predefined subset of the transcriptome, for example a transcript, an exon or a gene. For simplicity, we will assume throughout this article that reads have been summarized by gene, so that the RNA-seq profiles give the number of reads from each sample that have been mapped to each gene. Typically G is large, in the tens of thousands or more, whereas n can be as low as three. The total number of mapped reads (library size) for each sample might vary from a few hundred thousand to hundreds of millions. This is the same context as assumed by a number of previous articles [We suppose that RNA-seq profiles  yields counts per million (cpm), a simple measure of read abundance that can be compared across libraries of different sizes. Standardizing further by transcript length (in kilobases) gives rise to reads per kilobase per million (rpkm), a well-accepted measure of gene expression . In thisThis article treats log-cpm as analogous to log-intensity values from a microarray experiment, with the difference that log-cpm values cannot be treated as having constant variances. Differences in log-cpm between samples can be interpreted as log-fold-changes of expression. The counts are augmented by a small positive value  to avoid taking the logarithm of zero. This ensures no missing log-cpm values and reduces the variability at low count values.Probability distributions for counts are naturally heteroscedastic, with larger variances for larger counts. It has previously been argued that the mean-variance relationship for RNA-seq counts should be approximately quadratic . This leλ is the expected size of the count and ϕ is a measure of biological variation [ where ariation . The firA simple linearization calculation suggests that the standard deviation of the log-cpm should be approximately equal to the CV of the counts . Examination of a wide range of real datasets confirms these expectations. For studies where the replicates are entirely technical in nature, the standard deviation of the log-cpm decreases steadily as a function of the mean Figure a. For stWe conclude that log-cpm values generally show a smoothly decreasing mean-variance trend with count size, and that the log-cpm transformation roughly de-trends the variance of the RNA-seq counts as a function of count size for genes with larger counts.et al. [limma-trend, whereby the log-cpm values are analyzed as for microarray data but with a trended prior variance. For comparison, the more naive approach without the mean-variance trend will be called limma-notrend.A simple approach to analyzing RNA-seq data would be to input the log-cpm values into a well-established microarray analysis pipeline such as that provided by the limma software package ,9. This et al.  for micrThe limma-trend pipeline models the variance at the gene level. However, in RNA-seq applications, the count sizes may vary considerably from sample to sample for the same gene. Different samples may be sequenced to different depths, so different count sizes may be quite different even if the cpm values are the same. For this reason, we wish to model the mean-variance trend of the log-cpm values at the individual observation level, instead of applying a gene-level variability estimate to all observations from the same gene.Our strategy is to estimate non-parametrically the mean-variance trend of the logged read counts and to use this mean-variance relationship to predict the variance of each log-cpm value. The predicted variance is then encapsulated as an inverse weight for the log-cpm value. When the weights are incorporated into a linear modeling procedure, the mean-variance relationship in the log-cpm values is effectively eliminated.A technical difficulty is that we want to predict the variances of individual observations although there is, by definition, no replication at the observational level from which variances could be estimated. We work around this inconvenience by estimating the mean-variance trend at the gene level, then interpolating this trend to predict the variances of individual observations Figure .The algorithm proceeds as follows. First, gene-wise linear models are fitted to the normalized log-cpm values, taking into account the experimental design, treatment conditions, replicates and so on. This generates a residual standard deviation for each gene Figure a. A robuAlso available from the linear models is a fitted value for each log-cpm observation. Taking the library sizes into account, the fitted log-cpm for each observation is converted into a predicted count. The standard deviation trend is then interpolated to predict the standard deviation of each individual observation based on its predicted count size Figure c. FinallThe log-cpm values and associated weights are then input into limma’s standard differential expression pipeline. Most limma functions are designed to accept quantitative weights, providing the ability to perform microarray-like analyses while taking account of the mean-variance relationship of the log-cpm values at the observation level.P values that control error rates correctly over a wide range of simulation scenarios. For illustration, we present results from simulations in which read counts were generated under the same NB model as assumed by a number of existing RNA-seq analysis methods. These simulations should represent the ideal for the NB-based methods. If the normal-based methods can give performance comparable to or better than count-based methods in these simulations, then this is strong evidence that they will be competitive across a wide range of data types.We have found that voom and limma-trend, especially voom, perform well and produce ϕ was set to decrease on average with expected count size, asymptoting to 0.2 squared for large counts. This degree of biological variation is representative of what we observe for real RNA-seq data, being larger than we typically observe between genetically identical laboratory mice but less than we typically see between unrelated human subjects   Figure .To check voom’s conservativeness on real data, we used a set of four replicate libraries from the SEQC Project . All fouNext we examined the power to detect true differential expression. For the following simulations, 100 randomly selected genes were twofold upregulated in the first group and another 100 were twofold upregulated in the second group. This represents a typical scenario for a functional genomics experiment in which the differential expression effects are large enough to be biologically important but nevertheless sufficiently subtle as to challenge many analysis methods. Figure P value for the other methods. The results show that voom has the lowest FDR at any cutoff . A CAMERA gene set test was even more convincing, confirming that these genes are significantly more upregulated in males than are other genes in the genome (P=2×10-28).We examined 12 particular genes that are known to belong to the male-specific region of chromosome Y ,43. A ROP=0.0001, CAMERA P=10-10). The log-fold-changes for the X and Y chromosome genes involved in the gene set tests are highlighted on an MA plot  value for each count as: rgi+0.5)/(Ri+1) is strictly less than 1 as well as strictly greater than zero.The counts are offset away from zero by 0.5 to avoid taking the log of zero, and to reduce the variability of log-cpm for low expression genes. The library size is offset by 1 to ensure that (λ=E(r) for the expected value of a read count given the experimental conditions, and suppose that: Write ϕ is a dispersion parameter. If r is large, then the log-cpm value of the observation is:  where R is the library size. Note that the analysis is conditional on R, so R is treated as a constant. It follows that var(y)≈var(log2(r). If λ also is large, then:  where  by Taylor’s theorem , so: This article develops differential expression methods for RNA-seq experiments of arbitrary complexity, for example experiments with multiple treatment factors, batch effects or numerical covariates. As has been done previously ,7,8,34, xi is a vector of covariates and βg is a vector of unknown coefficients representing log2-fold-changes between experimental conditions. In matrix terms:  where yg is the vector of log-cpm values for gene g and X is the design matrix with the xi as rows. Interest centers on testing whether one or more of the βgj are equal to zero, where ygi for each gene. This yields regression coefficient estimates sg.The above linear model is fitted, by ordinary least squares, to the log-cpm values Also computed is the average log-cpm  where To obtain a smooth mean-variance trend, a LOWESS curve is fitted to square-root standard deviations Next the fitted log-cpm values ygi.The function value ygi and associated weights wgj are then input into limma’s standard linear modeling and empirical Bayes differential expression analysis pipeline.Finally, the voom precision weights are the inverse variances 4 Figure c. The loROAST  and CAMERi can be scale normalized to adjust for compositional differences between the RNA-seq libraries [Ri in the voom pipeline. Alternatively, between-array normalization methods developed for single channel microarray data, such as quantile or cyclic LOESS, can be are applied to the log-cpm values.The log-cpm values are by definition normalized for sequencing depth. Other normalization steps can optionally be done. The library sizes ibraries . This prThe simulations were designed to generate data with characteristics similar to real data that we analyze in our own practice. First a set of baseline expression values was generated representing the relative proportion of counts expected to arise from each gene. These proportions were translated into expected count sizes by multiplying by library size, and then multiplied by true fold-changes as appropriate. Counts were then generated following a NB distribution with the specified mean and dispersion for each observation.The distribution of baseline values was chosen to match that from RNA-seq experiments conducted at our institution. Specifically we used the goodTuringProportions function of the edgeR package , which iψgi with: The NB dispersions were generated as follows. The trend in the dispersions was set to be λgi is the expected count size. A modest amount of gene-wise biological variation was generated from an inverse chi-square distribution with 40 degrees of freedom. The individual dispersions were set to be ϕgi=ψgiδg where  where δg were simulated as log-normal with mean 0 and standard deviation 0.25 on the log-scale. This produces a distribution with a similar CV as for the inverse chi-square simulation.In an alternative simulation, to investigate sensitivity to the distribution of gene-wise dispersions, the For each simulated dataset, genes with less than ten reads across all samples were filtered from the analysis. PoissonSeq resets the seed of the random number generator in R, so it was necessary to save and restore the state of the random number generator before and after each call of the main PoissonSeq function.Complete runnable code that reproduces all the simulations is provided as Additional file The SEQC project, also known as MAQC-III, aims to provide a comprehensive study of next-generation sequencing technologies . We analThe groups are labeled A–D and are closely analogous to the similarly labeled RNA samples used in the earlier microarray quality control study . LibrariThe voom mean-variance trend shown in Figure The comparison between technical replicates to check the type I error rate control used only the four group B libraries. Genes were filtered out if they failed to achieve a cpm >1 in at least two libraries and the log-cpm values for the 16,745 remaining genes were quantile normalized. Samples were separated into all possible two-versus-two and three-versus-one combinations and a limma analysis using voom weights was carried out for each partition.The false discovery rate analysis was conducted on the spike-in transcripts only. ERCC Mixes 1 and 2 contain 92 transcripts spiked in at different concentrations. For this analysis, fragments were mapped to the known sequences of the spiked-in transcripts using Subread. The experiment is designed so that 23 transcripts have the same concentration in Mix 1 and Mix 2. The remaining transcripts were spiked-in in such a way that 23 transcripts are fourfold more abundant in Mix 1, 23 are 1.5 higher in Mix 2 and 23 are twofold higher in Mix 2. A majority of the spike-in transcripts data are DE. We replicated the counts for each of the 23 non-DE transcripts three times, so that each non-DE transcript was treated as three different transcripts. This resulted in a dataset of 138 transcripts, half DE and half non-DE. Our analysis used read counts for the spike-in transcripts only. TMM-scale normalization  was usedpickrell1, annotEnsembl63 and genderGenes. Genes were filtered if they failed to achieve a cpm value of 1 in at least 20 libraries. Library sizes were scale-normalized by the TMM method [As part of the International HapMap Project, RNA samples were obtained from lymphoblastoid cell lines derived from 69 unrelated Nigerian individuals including 29 males and 40 females , specifiM method  using edM method  prior toD. melanogaster[RNA-seq was used to explore the developmental transcriptome of anogaster. Mapped anogaster. Specifianogaster providesanogaster using edanogaster prior toP values less than 10-10.Gene ontology analysis used the GOstats software package  and versAn RNA-seq experiment was carried out to detect differential striatal gene expression between the C57BL/6J (B6) and DBA/2J (D2) inbred mouse strains . ProfileThe results presented in this article were obtained using R version 3.0.0 and the software packages limma 3.16.2, edgeR 3.2.3, baySeq 1.14.1, DESeq 1.12.0, DSS 1.4.0, PoissonSeq 1.1.2 and tweeDEseqCountData 1.0.8. All of these packages are part of the Bioconductor project ,68, excetrend=TRUE for the eBayes function. Hence the limma-trend pipeline was the same as that for voom except that weights were not used in the linear modeling step and the trend option was turned on for the empirical Bayes step. The limma package can be installed from the Bioconductor project repository [The voom methodology proposed in the article is implemented in the voom function of the limma package. The limma-trend method was implemented by inputting the log-cpm values from voom into limma’s standard pipeline, with pository .sharingMode=~maximum~ and fitType=~local~ and conducted tests using nbinomTest.All the count-based packages were used with the default differential expression pipelines as recommended in the software for each package. For edgeR 3.2.3 the default prior degrees of freedom for squeezing the gene-wise dispersions is 10. Note that this is a change from versions 3.0.X and earlier for which the default had been 20. For DSS the Wald test was used as recommended in the documentation. The DESeq defaults have changed considerably since the original publication. We used the DESeq function estimateDispersions with The different count-based packages implement different methods of compositional normalization . For ourCV: coefficient of variation; DE: differentially expressed; FDR: false discovery rate; HBRR: Ambion’s Human Brain Reference RNA; log-cpm: log-counts per million; LOWESS: locally weighted regression; NB: negative binomial; rpkm: reads per kilobase per million; UHRR: Stratagene’s Universal Human Reference RNA.The authors declare that they do not have any competing interests.CL and GS developed the method and wrote the manuscript. GS implemented the method and CL performed the analyses. YC contributed to the comparative simulation studies and carried out the gene ontology analysis. WS performed read alignment and summarized the SEQC and mouse RNA-seq datasets. All authors read and approved the final manuscript.Simulation code. R code to reproduce the simulations presented in Figures Click here for file


MicroRNAs (miRNAs) are important key regulators in multiple cellular functions, due to their a crucial role in different physiological processes. MiRNAs are differentially expressed in specific tissues, during specific cell status, or in different diseases as tumours. RNA sequencing (RNA-seq) is a Next Generation Sequencing (NGS) method for the analysis of differential gene expression. Using machine learning algorithms, it is possible to improve the functional significance interpretation of miRNA in the analysis and interpretation of data from RNA-seq. Furthermore, we tried to identify some patterns of deregulated miRNA in human breast cancer (BC), in order to give a contribution in the understanding of this type of cancer at the molecular level.We adopted a biclustering approach, using the Iterative Signature Algorithm (ISA) algorithm, in order to evaluate miRNA deregulation in the context of miRNA abundance and tissue heterogeneity. These are important elements to identify miRNAs that would be useful as prognostic and diagnostic markers. Considering a real word breast cancer dataset, the evaluation of miRNA differential expressions in tumours versus healthy tissues evidenced 12 different miRNA clusters, associated to specific groups of patients. The identified miRNAs were deregulated in breast tumours compared to healthy controls. Our approach has shown the association between specific sub-class of tumour samples having the same immuno-histo-chemical and/or histological features. Biclusters have been validated by means of two online repositories, MetaMirClust database and UCSC Genome Browser, and using another biclustering algorithm.The obtained results with biclustering algorithm aimed first of all to give a contribute in the differential expression analysis in a cohort of BC patients and secondly to support the potential role that these non-coding RNA molecules could play in the clinical practice, in terms of prognosis, evolution of tumour and treatment response. Moreoveet al , reporteet al . Unfortuet al ,21. The et al ,23.In the past decade, the microarray technique was successfully used for the analysis of gene expression but, due to some limitations -26, moreNevertheless, there is not a standard method used to analyse the gene expression by RNA-seq data. In order to assist biologists in the complex task of interpretation of the large amount of data produced by NGS techniques, an investigation technique should be based on computational methods.Clustering methods have been usually used to analyse gene expression data, with the purpose to discover groups of genes (clusters) that are co-regulated with respect to certain experimental conditions. Traditional clustering approaches, like for instance k-means algorithm, however, have proved to perform well only with a small set of expression data. With large datasets, composed of thousands of genes and hundreds of experimental conditions, clustering techniques are not able to correct identify the gene clusters, because they consider the expression profiles under all conditions at the same time. Many regulation mechanisms, indeed, involve only a set of genes and a limited set of experimental conditions ,29. For The use of biclustering algorithm in order to correlate miRNA profiles to specific tumour features was supported by a recent work . It showThe present work, that is an extended version of , will taBiclustering is a computational procedure that separates homogeneous sub-matrices inside a matrix composed of objects vs features. In our study the data matrix is constituted by samples vs miRNAs. The biclustering procedure selects simultaneously a subset of features that characterizes a subset of samples, on the basis of a "local similarity" criterion. On the contrary, traditional clustering groups together objects that are similar with respect to the whole set of features, not to a subset of features . It is aFigure transcription modules by ISA's authors. In our approach, the set of genes are represented by a set of miRNAs and the experimental conditions can be identified with breast cancer samples and healthy tissues.According to recent surveys ,39 that E of size m × n (with m = number of genes and n = number of samples), the ISA algorithm works in the following way. First of all, it is created a random sparse 0/1 vector of size m, called seed vector r0. The elements of r0 are used to select a subset of the genes (rows). E is then transposed (E'), it is multiplied by r0, and the result is thresholded, obtaining as result the vector c0 of size n. This threshold is obtained computing the mean and standard deviations of the vector c0, and considering only the elements that have a standard deviation from the mean higher than the threshold. The vector c0 is then used to select a subset of the input samples, considering only the columns of E corresponding to the non-zero components of c0. After that, E is then multiplied by c0, and the result is again thresholded in order to obtain the vector r1 (of size m). This procedure is iteratively repeated until either ri-1 and ir, and ci-1 and ci are close enough, according to a convergence criteria defined in terms of Pearson correlation. The final vectors final rand final care used to extract a subset of the genes and samples, composing the biclusters, from the expression matrix E. Genes are selected considering the rows of E corresponding to the the non-zero components of finalr; samples are selected considering the columns of E corresponding to the the non-zero components of finalc. Therefore the two most important parameters of ISA algorithm are the two thresholds: r thfor the matrix rows (representing the genes), c thfor the matrix columns (associated to the samples). The threshold values affect the composition of the biclusters: for example, if the row (gene) threshold is high, then the biclusters will have very similar genes. Lower threshold values, in turn, will provide bigger biclusters with less similar genes. The same behaviour applies when considering the column (sample) threshold. Moreover the biclusters are "soft", meaning that they allow overlapping among genes and/or samples, and, according to the threshold values, it is possible that groups of genes and samples do not belong to any bicluster. It is also possible to obtain a bicluster that correlates a single gene with a group of patients and vice versa. As the authors state, ISA algorithm has the best performance processing normalized input data, i.e the input matrix have zero mean and standard deviation equal to one. In addition to the biclusters, ISA algorithm also provides as output two numeric matrices: one for the row (gene) components of the bicluster, and another one for the column (sample) components of the bicluster. Considering for example the resulting matrix for the genes, each column corresponds to a bicluster. The rows with a non-zero value belong to the bicluster related to that column. These matrices have values ranging from -1 and +1 and they express the association degree of a gene, or sample, to that bicluster. If two genes belong to the same bicluster with the same sign, they are correlated, otherwise they are anti-correlated. In the following discussions, for sake of simplicity, we take into account the output matrix having samples as rows, and the miRNA biclusters as column. Since the biclusters are unique, the same conclusions hold when considering miRNA vs sample biclusters.Considering a gene expression matrix isa2 [In our work, we used the ISA implementation provided by the R package isa2 .in vitro experiments, we referred to: 1) an external repositories providing informations about different groups of miRNAs in human species; 2) a different biclustering algorithm to compare our results with.We adopted two different validation criteria: 1) semi-supervised evaluation criterion, based on scientific literature; 2) consensus with another biclustering tool, the SAMBA algorithm , that haIn particular, we investigated over MetaMirClust database  and UCSCIn this section, we introduce the experimental dataset and then we discuss the biclustering results obtained according to the proposed pipeline, introduced in the Section Methods.et al ([http://www.ncbi.nlm.nih.gov/Traces/sra/sra.cgi). The miRNA expression profiles were downloaded from Gene Expression Omnibus (GEO) with accession number GSE28884. The dataset is composed of 169 tumour samples, 10 healthy tissues, 6 cell lines , for a total of 185 samples. We wanted, indeed, to test the efficacy of the ISA biclustering algorithm on a real dataset, not altered by a guided choice of the samples, and on a real life problem as the breast cancer analysis. Another parameter that we took into account was the partition of samples according to the clinical and histo-pathological features, molecular sub-type and the presence of one or more immune-histo-chemical (IHC) markers . Our aim is in fact to discover relationships between specific patterns of patients and specific groups of miRNA.We used a breast cancer dataset composed of a large cohort of patients. RNA-seq data were collected by Farazi et al  and uplOur biclustering approach allowed us to evaluate miRNA deregulation in the context of miRNA abundance, and tissue heterogeneity, important elements in identifying miRNAs that would be useful prognostic and diagnostic markers.et al. [Figure et al. ): a) presf-miR). This approach reduces the complexity of the data. The sequence families are most informative for characterization of seed-sequence-dependent miRNA-target regulation. We considered the miRNA families as identified in the work of Farazi et al. [Regarding the miRNA profiles, we set the threshold for sample comparison to 5,000 total miRNA sequence reads per replicate merged library, as reported in , identifi et al.  and provValidation Technique. Since the analysis of all the biclustering results obtained through SAMBA algorithm is not in the scope of this work, we provide them in the Supplementary Material, Additional file We aimed to evaluate the differential expression of miRNAs in breast cancer samples versus healthy tissues. Figure r thwas set equal to 3.5, c thwas set equal to 1.0. Other threshold values gave few clusters but with many elements, or many clusters with few elements. Considering the random initialization of ISA algorithm we ran it ten times with different initialization values but with the same thresholds, in order to assess the stability of the 12 biclusters. Each time we obtained the same 12 biclusters, although with a different sorting. As it is evident in Figure The two thresholds of the ISA algorithm were regulated in order to have a manageable (not too large) and meaningful number of biclusters. As it will be discussed in the rest of this section, indeed, biclusters composed of one single miRNA family linked to a group of samples, enable us to associate a particular miRNA family to a set of sample tissues with well defined immuno-histochemical and histological features. This characteristic of the algorithm allows in a "one step" process to obtain results that are normally produced with more passages with a classical hierarchical clustering process.et al [Our biclustering procedure highlighted many results from the original work of Farazi et al , it alsoet al. [et al [et al. and Shatseva et al. confirmed our data clustering, showing increased levels of mir-214 in serum of BC patients compared to healthy controls [The first step of our analysis was to check if the healthy tissues were univocally associated to the same sub-class of miRNA. The data showed an exclusive association between these samples and the miRNA cluster sf-hsa-miR-320-RNASEN  are a distinct group of tumour samples characterized from distinct histological and IHC features, as they lack for the presence of estrogen receptor (ER), progesterone receptor (PR), and Her2 protein receptors. They are associated to a specific pattern of miRNA, that is disregulated: sf-hsa-miR-17, sf-hsa-miR-19a, sf-hsa-miR-17*, and sf-hsa-miR-20a* . These results have been confirmed by SAMBA biclustering  -56. MoreAs previously said, sf-hsa-miR-148a* and sf-hsa-miR-148a miRNA clusters were linked by the algorithm analysis to "atypical medullary" and DCIS samples. Also these data differ from literature: scientific works conducted on different cancer types showed a down-regulation in tumour samples compared with healthy controls -62. No aA new association is reported about the miRNA bicluster composed of sf-hsa-miR-451-DICER1, sf-hsa-miR-144, sf-hsa-miR-486: our biclustering showed an over-expression of this bicluster in a sub-class of IDC/her2 positive patients. In literature there are no data regarding an association about this last miRNA bicluster and breast cancer but there are more evidence about an over-expression of miRNA 451 in BC and other tumour samples ,64. InstIn this paper we used a biclustering approach, by means of the ISA algorithm, in order to analyse the differential expression of miRNA in breast cancer samples. Thanks to its features, the biclustering method allows to cluster subset of patients with subset of miRNAs, exploiting this way miRNA biclusters under different samples conditions. Our analysis highlighted 12 miRNA biclusters, each of them involving different types of tumour samples, characterized by immuno-histo-chemical, histopathological and molecular sub-type features. Biclusters were validated in the current scientific literature with the support of the MetaMirClust and UCSC Genome Browser online tools. Further validation was provided by the consensus with another biclustering tool, the SAMBA algorithm. The proposed biclustering methodology has proved a valid instrument for the study of miRNA expression profiles, with the possibility to identify biclusters that can provide novel relationships among groups of miRNAs and patient conditions. Of course, discordant or new results need to be deeply investigated.The authors declare that they have no competing interests.AF: project conception, implementation, discussions, assessment, writing. MLR: project conception, implementation, discussions, assessment, writing. LLP: project conception, experimental tests, discussions, assessment, writing. RR: project conception, discussions, assessment, writing. AU: project conception, discussions, assessment, writing, funding. All authors read and approved the final manuscript.SAMBA biclustering results. Excel file summarizing the whole biclustering results obtained with SAMBA algorithm. Each sheet of the file reports the composition, in terms of miRNA and sample, of each bicluster.Click here for fileBicluster heathmaps. An archive containing the heatmaps of each bicluster obtained with SAMBA algorithm.Click here for file


The four intrinsic subtypes of breast cancer, defined by differential expression of 50 genes (PAM50), have been shown to be predictive of risk of recurrence and benefit of hormonal therapy and chemotherapy. Here we describe the development of Prosigna™, a PAM50-based subtype classifier and risk model on the NanoString nCounter Dx Analysis System intended for decentralized testing in clinical laboratories.514 formalin-fixed, paraffin-embedded (FFPE) breast cancer patient samples were used to train prototypical centroids for each of the intrinsic subtypes of breast cancer on the NanoString platform. Hierarchical cluster analysis of gene expression data was used to identify the prototypical centroids defined in previous PAM50 algorithm training exercises. 304 FFPE patient samples from a well annotated clinical cohort in the absence of adjuvant systemic therapy were then used to train a subtype-based risk model (i.e. Prosigna ROR score). 232 samples from a tamoxifen-treated patient cohort were used to verify the prognostic accuracy of the algorithm prior to initiating clinical validation studies.The gene expression profiles of each of the four Prosigna subtype centroids were consistent with those previously published using the PCR-based PAM50 method. Similar to previously published classifiers, tumor samples classified as Luminal A by Prosigna had the best prognosis compared to samples classified as one of the three higher-risk tumor subtypes. The Prosigna Risk of Recurrence (ROR) score model was verified to be significantly associated with prognosis as a continuous variable and to add significant information over both commonly available IHC markers and Adjuvant! Online.The results from the training and verification data sets show that the FDA-cleared and CE marked Prosigna test provides an accurate estimate of the risk of distant recurrence in hormone receptor positive breast cancer and is also capable of identifying a tumor's intrinsic subtype that is consistent with the previously published PCR-based PAM50 assay. Subsequent analytical and clinical validation studies confirm the clinical accuracy and technical precision of the Prosigna PAM50 assay in a decentralized setting.The online version of this article (doi:10.1186/s12920-015-0129-6) contains supplementary material, which is available to authorized users. These set al. . Polymeret al. . Severalet al. –16.de novo retraining of the PAM50 breast cancer intrinsic subtype classifier should be performed on the nCounter in order to develop the most accurate and robust classifier. The primary aim of this study was to train a PAM50-based subtype classifier and prognostic risk of recurrence (ROR) model on the NanoString nCounter Dx Analysis System that is consistent with the published qRT-PCR-based PAM50 assay using FFPE breast cancer tissue samples obtained specifically for this training. The second aim was to verify that the clinical accuracy of the NanoString Prosigna algorithm is equivalent to the PCR-based classifier and ROR model [Rather than simply implementing the existing PCR-based PAM50 signature on the NanoString platform, scientific best practices dictate that a OR model  using a et al. [Feasibility experiments were conducted to test the concordance between gene expression measured on the NanoString nCounter and qRT-PCR. NanoString probes were designed to match the 50 classifier genes and 5 housekeeper genes defined by Parker et al. . The feaet al. . The PCRet al. .et al. [Reproducibility of the NanoString versus qRT-PCR gene expression measurements was assessed for all 50 target genes and 5 housekeeping genes using 113 samples from archived formalin-fixed tumor blocks that were collected under an Institutional Review Board-approved protocol from Washington University. Cores from each tissue block were obtained using a disposable sampling tool  and total RNA was isolated using the Roche High-Pure RNA Paraffin kit  . RNA inpet al. ) was alset al. . Accuracet al. .The criteria used to select samples for training included tissue amount and quality, along with each FFPE tissue block representing a unique patient. The complete inclusion criteria for algorithm training are described in Additional file The UNC and Wash U samples were included to ensure a broad demographic representation (e.g. approximately 30 % and 40 % African American women respectively) as part of the prototypical subtype centroid training exercise. The 24 FFPE reduction mammoplasty samples were included to ensure the training exercise recapitulated the experimental design of the training for the published PCR-based PAM50 assay , which aet al. [. These samples were included in the algorithm verification to increase the number of samples from the BC TAM study as not all patient blocks contained sufficient tissue for re-isolation.An independent set of FFPE breast cancer patient samples (BC TAM) with clinical outcome data were used for verifying the prognostic value of the trained Prosigna ROR score and subtype centroids. These specimens were also collected from the Genetic Pathology Evaluation Centre from estrogen receptor-positive (ER+), node-negative patients treated with five years of adjuvant tamoxifen . A subseet al. . These sPrior to performing a Prosigna assay, a certified pathologist reviewed all FFPE tissue blocks to identify and circle the area of viable invasive carcinoma. Two 1.0 mm diameter tissue cores were taken from the identified area of the block and shipped from the collection location to NanoString. For a small subset of the training samples, only one core was taken due to limited tumor within the identified area of invasive carcinoma. For the normal tissue samples, the certified pathologist reviewed the FFPE block to confirm the absence of any tumor tissue. Since the normal tissue samples contain a heterogeneous mixture of stromal and epithelial tissue, normal samples were assayed as FFPE tissue scrolls (10 um thickness) rather than core punches.RNA was isolated using a Roche column-based RNA extraction kit manufactured to NanoString’s specifications . Brieflyin vitro transcribed RNA transcripts and corresponding probes, and eight negative quality controls consisting of probes with no sequence homology to human RNA sequences [in vitro transcribed RNAs of the 58 targets that are used for normalization purposes.The RNA was analyzed on the NanoString nCounter Dx Analysis System which delivers direct, multiplexed measurements of gene expression through digital readouts of the abundance of mRNA transcripts , 20. Theequences . Each PrSample processing steps after hybridization are automated on the nCounter Prep Station. The excess probes are removed followed by binding of the probe-target complexes on the surface of the nCounter cartridge via a streptavidin-biotin linkage. Probe-target complexes are aligned and immobilized in the nCounter cartridge. After sample processing has completed, cartridges are placed in the nCounter Digital Analyzer for data collection. Each target molecule of interest is identified by the target specific “color code” generated by six ordered fluorescent spots present on the reporter probe. The Reporter Probes on the surface of the cartridge are then counted and tabulated for each target molecule.a Luminal A centroid that classifies tumors with the best prognosis as Luminal A relative to other putative Luminal A centroidssimilar hazard ratios between Luminal A subtype and other high risk subtypes compared to previously published dataHierarchical clustering was performed using Pearson’s correlation as the distance metric and average linkage clustering. The R package SigClust was used to assess significance of each cluster in order to identify the prototypical samples that were used to derive the centroids. Prototypical subtype centroids were assigned based on gene expression profiles concordant to previously published data , 3, 10 aA multivariable Cox model was fit using ridge regression  in the Ret al. [Using the BC TAM patient samples and following procedures outlined in Parker et al. , the C-iet al. . BrieflyThe median block age of 113 FFPE tumor samples used to compare NanoString and qRT-PCR repeatability was 10 years with a range of 7–13 years old. The gene expression of the 50 classifier genes for each sample was independently normalized to the geometric mean of five housekeeper genes. The median coefficient of variation was used to summarize the reproducibility of the expression measurement of each gene  despite a slight decrease in sensitivity evident by a mean slope between nCounter and qRT-PCR data of 0.88 across the 50 classifier genes  patient FFPE breast tumor samples and 79 previously isolated RNA samples were received at NanoString and met the predefined sample inclusion criteria. After isolation of RNA from the tumor samples and assessment of yield and quality, 854 of these samples were analyzed on the NanoString nCounter Dx Analysis System. Gene expression data from 746 patient samples were determined to be of high quality and were used for algorithm training and verification . A full description of clinical characteristics by cohort is included in Table The four prototypical tumor subtype centroids  were defined by identifying statistically significant (P < 0.001) clusters from hierarchical clustering of the PAM50 genes in 514 samples from the BC no AST, UNC, and WashU cohorts and 24 reduction mammoplasty samples  was applied to the normalized data. The transformed data were then used to set 50 gene centroids for each of four tumor subtypes. Subtypes were subsequently assigned for all 514  tumor samples based on the maximum Pearson’s correlation between the transformed gene expression for each sample and the four tumor centroids  compared to the other three subtypes Fig.  as obserSubtype classifications from BC no AST patient samples were also compared to subtype classifications from an untreated cohort of the Netherlands Cancer Institute (NKI) . Compariet al. [et al. [A 50-gene ROR score model and simplified 46-gene ROR model  were developed using multivariable Cox modeling with ridge regression to learn the ROR score coefficients. These 4 genes were removed and the 46 gene model compared to the 50 gene model as they did not seem to add prognostic accuracy to the predicted risk of recurrence. The ROR score incorporate the biology of the intrinsic subtypes by including the Pearson’s coefficients to the four tumor centroids as factors in the model in addition to a proliferation score and primary tumor size as additive terms to predict risk of recurrence. A number of PAM50 based ROR models have been previously reported  which inThe resulting Prosigna ROR score is calculated using weighted coefficients to the four subtypes, a proliferation score, and tumor size. Tumor size was calculated by assigning a binary classification value to primary tumors measuring 2.0 cm or less in the greatest dimension versus those measuring greater than 2.0 cm. The calculated Prosigna ROR scores were adjusted to a 0–100 scale. Scaling factors were generated using the entire range of Prosigna ROR scores from all tumor samples from algorithm training and verification  where tumor sizes were available.The similarity of the 46 gene ROR and 50 gene ROR scores was evaluated. Data were generated from all the training samples  using both models. When the paired data for each model were plotted against each other the removal of 4 genes had an insignificant impact on the reported ROR score with an R-squared value equal to 0.997 (Fig. The accuracy of the Prosigna classifier was verified in the independent BC TAM cohort by comparing the outcomes of the patients with a Luminal A subtype with the other subtypes. Similar to what was reported for the published PCR-based PAM50 classifier , patientet al. [The distribution of the ROR scores for BC TAM patient tumors from each of these three subtypes show that the model appropriately characterizes the risk associated with PAM50 intrinsic subtype. Only Luminal A tumors were classified as low risk with ROR scores less than 40. Other than a single Luminal A tumor with an ROR score of 63, only Luminal B or HER2-enriched tumors were classified as high risk with ROR scores greater than 60 Fig. . The disThe accuracy of the 46 and 50-gene ROR models were then assessed. Predictions of the C-index for each model were generated from 1,000 bootstrap samples of the risk assignments from the BC TAM verification data using the R package Hmisc. There was no difference in accuracy between the 46 or 50-gene ROR scores Fig. . The facet al. [Consistent with analysis performed by Nielsen et al. , the accet al. .Fig. 10Aet al. [The four breast cancer intrinsic subtypes were first described by Perou et al.  and origet al. , 2. The et al. , 31. Theet al.  and proget al. . PAM50 set al. , 7, 32 aet al. , 10. Addet al. –6.Breast cancer subtypes, as defined using available IHC assessments of ER, PR, and HER2 biomarkers, have been included in treatment guidelines for breast cancer patients as a surrogate for molecular subtyping using PAM50 . These bCurrently, the laboratory-developed BluePrint® test  is the oThere are several other gene expression based tests currently available that output a breast cancer risk classification , 40, 41.in vitro diagnostic version of the PAM50 classifier.The feasibility experiments described herein showed highly concordant results between the NanoString nCounter platform and qRT-PCR. Across 113 FFPE breast cancer specimens, expression of the PAM50 genes, subtypes and ROR scores were highly concordant between qRT-PCR and nCounter using the PCR-based classifier. Additionally, in repeated measures from 71 FFPE tumor specimens, the published classifier on nCounter data gave highly concordant results for ROR and subtype. The similarity of results in the cross platform evaluation and within tissue repeatability was instrumental in the selection of the NanoString nCounter Dx Analysis System as the platform for development and training of the The subtype centroid and ROR model training for the Prosigna algorithm were designed to parallel the training of the published PCR-based PAM50 classifier and ROR model. This retraining was executed using an independent set of FFPE breast tissue samples, with RNA isolated using a GMP-manufactured kit, and using data solely collected on the NanoString nCounter Dx Analysis System. The gene expression profiles of the Prosigna intrinsic breast cancer subtype centroids are similar to the published PAM50 classifier centroids. Additionally, patients assigned a Luminal A subtype were significantly lower risk compared to the Luminal B, HER2-enriched, and Basal-like subtypes in the training population with no adjuvant systemic therapy as previously observed in the published PCR-based PAM50 classifier . The resThe results from both the BC no AST patient population used for Prosigna training and the NKI data used to train the published PCR-based PAM50 classifier  show thaet al. [In contrast to the published PCR-based PAM50 classifier , which wet al.  where thet al. ; howeveret al. [et al. [in vitro diagnostic test, but is suitable for identifying samples with substantial normal RNA contamination in a research setting where pathology review and macrodissection may not always be feasible.Of note, unlike the published classifier of Parker et al. , the Proet al. . An inte [et al.  tested t [et al. . This is [et al. , 53. TheThe analytical  and clinThe Prosigna assay is the only genomic assay that is CE-marked and FDA-cleared that was trained, verified and validated to provide an accurate estimate of the risk of distant recurrence in hormone receptor positive breast cancer using RNA from FFPE breast cancer patient samples. Additionally, Prosigna is the only assay that is capable of classifying patients into one of the four breast cancer intrinsic subtypes using a classifier that was trained and verified to be consistent with the published PAM50 classifier.


Haemophilus influenzae supragenome hybridization (SGH) array that contains probe sequences to 2,890 gene clusters identified by whole genome sequencing of 24 strains of H. influenzae. The array was originally designed as a tool to interrogate the gene content of large numbers of clinical isolates without the need for sequencing, however, the data obtained is quantitative and is thus suitable for transcriptomic analyses. In the current study RNA was extracted from H. influenzae strain CZ4126/02 (which was not included in the design of the array) converted to cDNA, and labelled and hybridized to the SGH arrays to assess the quality and reproducibility of data obtained from these custom-designed chips to serve as a tool for transcriptomics. Three types of experimental replicates were analyzed with all showing very high degrees of correlation, thus validating both the array and the methods used for RNA profiling. A custom filtering pipeline for two-condition unpaired data using five metrics was developed to minimize variability within replicates and to maximize the identification of the most significant true transcriptional differences between two samples. These methods can be extended to transcriptional analysis of other bacterial species utilizing supragenome-based arrays.We previously carried out the design and testing of a custom-built  The tremendous advancement of sequencing technologies combined with rapid reductions in associated costs has allowed researchers not only to sequence more diverse organisms but also to sample individual bacterial species with much greater resolution. Extensive strain sequencing has led to the realization and appreciation that most bacterial species harbour enormous genomic diversity among strains Haemophilus influenzae (Hi) is one such species of bacteria that has been demonstrated to possess enormous genomic variability a priori in order to target genes with sequence-specific probes for measurement. Since two different strains might at a minimum share only the core genes, an array of probes designed for genes found in any single strain does not appropriately represent the species and therefore a significant amount of information will be lost (hundreds of genes) when using an array developed from a single strain. Thus, a more robust strategy for the design of bacterial microarrays is to use probes based on defined supragenomic sequences.H. influenzae supragenome hybridization (SGH) array in order to perform DNA-DNA hybridizations for the purpose of determining gene content in unsequenced strains We previously designed and tested an H. influenzae supragenome hybridization (SGH) array is described by H. influenzae strains were clustered using a custom supragenome pipeline to obtain unique clusters of genes in silico based on uniqueness, distribution and probe manufacturing parameters. In all, 31,307 H. influenzae specific probes were synthesized by Roche/NimbleGen. Each array contained duplicates of each probe and each slide contained 12 arrays. An additional 185 negative control probes based on Streptococcus pneumoniae chromosomal sequences were also attached to the slides in duplicate and a further 9,053 random sequence probes were included. These arrays containing a total of 72,037 probes (72 K) are referred to as the SGH arrays.Design and testing of the H. influenzae SGH arrays as well as for array washing. Images were acquired on an Axon Instruments GenePix 4200AL array scanner.Genomic DNA (gDNA) was isolated from strain CZ4126/02 and Cy3-labeled using a NimbleGen One-Color DNA Labeling Kit. NimbleGen Hybridization Kits and Sample Tracking Control Kits were used to hybridize this labeled DNA to the custom-designed 2 scale) and a measure of probe variance Images were processed and data were normalized within chips using a Robust Multichip Average (RMA) algorithm and quantile normalization via the NimbleScan software v2.5 r [KO]) . These strains were used to test and validate the use of the SGH Array for unpaired two-condition transcriptional microarray analysis. The CZ4126/02 WT and its cognate KO strain were grown in two different media (BHI and CDM) and sampled at multiple time points  for RNA extraction. The RNAs were converted to cDNA, Cy-3 labelled and hybridized to the H. influenzae SGH arrays as described below. For each condition, RNA samples were collected twice which are referred to as replicates A and B. Each condition/replicate was hybridized on two different SGH arrays referred to as chip 1 and chip 2. Finally each array outputs separate information for two duplicate probe-sets referred to as probe-set 1 and probe-set 2. All transcriptional data in this study has been deposited in NCBI's Gene Expression Omnibus (GEO) Parallel work has focused on the role of AI-2 signalling in Hi by comparative studies between CZ4126/02 and an AI-2 sensing mutant (CZ4126/02ΔLsr::CmBrain-Heart Infusion broth (BHI - Oxoid) was made using 37 g of powdered media/L and supplemented with hemin (Sigma-Aldrich) to a final concentration of 10 µg/mL and β-nicotinamide adenine dinucleotide (β-NAD) to a final concentration of 2 µg/mL. Chemically Defined Media (CDM) was made exactly as described 2. Isolated colonies from these plates were used to inoculate 5 mL BHI cultures that were incubated overnight at 37°C with shaking at 200 rpm. The cultures were diluted to an ODA600 of 0.02 in 40 mL BHI or CDM. At selected time-points (Table S3), 1 mL culture samples were collected and transferred immediately into 2 mL RNAProtect (Qiagen). Samples were incubated for 10 minutes at room temperature and then stored overnight at 4°C.For microarray experiments frozen stocks were used to inoculate BHI plates that were incubated overnight at 37°C with 5% COSamples stored in RNAProtect were spun for 10 minutes at 2,500×g , the supernatant removed and the cell pellets resuspended in 100 µL of 1X Tris-EDTA (TE) +1 mg/mL lysozyme  and 1 mg/mL proteinase K (Qiagen). RNA was extracted using a Qiagen RNeasy Mini Plus kit with the standard protocol including genomic DNA (gDNA) eliminator columns. The eluted RNA (∼85 µL) was DNased by adding 10 µL 10X TurboDNase buffer and 5 µL TurboDNase (2 units/µL) (Ambion) and incubating at 37°C for 1.5 hours. 2 µL more TurboDNase was added and incubation continued for an additional 1.5 hours. The DNased RNA samples were cleaned by passing the samples through the RNeasy protocol a second time (including the gDNA eliminator column steps). Samples were eluted in nuclease free (n.f.) water, quantitated on a Nanodrop 1000 spectrophotometer and stored at −80°C. 200 ng of each RNA sample was run on an Agilent 2100 Bioanalyzer using RNA Nano6000 chips to check for RNA degradation. We performed paired reverse transcription reactions on every RNA sample where one reaction received reverse transcriptase  and the other did not (−RT). Both reactions were PCR amplified using primers directed against a housekeeping gene (GAPDH) and observation of amplification in the +RT reaction as well as lack of amplification in the −RT reaction verified removal of gDNA from each RNA sample.T method with the Roche Light Cycler software as well as a linear regression method using the LinRegPCR Single-stranded cDNA was synthesized from the extracted RNA samples using the Roche Transcriptor First Strand Synthesis kit. Specific primers for the housekeeping and experimental genes were designed using Roche Probe Finder online software in order to design ∼75 bp amplicons. qRT-PCR was performed on the Roche Light Cycler 480 using a SYBR green master mix. Reactions were performed in a 20 µl volume containing 2 µl cDNA (1∶5 dilution) and primers at 0.5 µm each. Primer efficiency was determined by testing all primer pairs ahead of time with gDNA template. All reactions were measured in triplicate. The experimental data were normalized using the hpr and ldhA genes as internal standards. Independent data analysis was carried out using both the Pfaffl-ΔΔCH. influenzae SGH arrays as well as for array washing. Images were acquired on an Axon Instruments GenePix 4200AL array scanner.First and second-strand cDNA synthesis was performed using a SuperScript One-Cycle cDNA Kit (Invitrogen) as outlined in the NimbleGen Microarray Experienced User’s Guide including RNaseA and cDNA precipitation steps. 1 µg of cDNA was Cy3-labeled using a NimbleGen One-Color DNA Labeling Kit. NimbleGen Hybridization Kits and Sample Tracking Control Kits were used to hybridize the labelled cDNA to the custom-designed H. influenzae SGH array using the NimbleScan software. These.pair tables were merged with a reference list of subclusters that had been determined to be present in the CZ4126/02 genome (see above) in order to remove non-relevant probe/subcluster data. These parsed.pair files were then normalized within and across chips using a Robust Multichip Average (RMA) algorithm and quantile normalization using the NimbleScan v2.5 software followed by a median polish whereby the 3 to 13 probe values/subcluster were condensed to a single value (in duplicate) Images were processed to.pair files containing expression values for both sets of duplicate probes representing all the subclusters on the H. influenzae SGH array contains 31,307 unique probes that target 2,890 of the 3,100 gene clusters identified in 24 geographically diverse clinical strains. Gene clusters were further subdivided into “subclusters” with more stringent alignment parameters in order to capture allelic differences within more variable genes. The power of this array is its ability to capture information for any strain of H. influenzae since the probes represent the majority of the predicted supragenome  in silico analysis should be performed to ensure that only the relevant subset of probes is included in the final analysis. This is important for the purposes of obtaining a Gaussian distribution of data needed for both normalization and statistical testing. Therefore, it is necessary to remove all data from so-called “non-relevant” gene clusters, defined as those present in one of the 24 strains used to design the array but not present in the strain being interrogated.The custom-designed H. influenzae SGH array. Our first task was to determine the gene content of this strain for the purposes of removing non-relevant data later. This we accomplished by two methods 1) WGS of CZ4126/02 and mapping of identified genes back to the SGH array clusters  and 2) hybridization of genomic DNA (gDNA) to the SGH array and application of a signal threshold to determine whether a gene was present or not For testing purposes we used a strain (CZ4126/02) that was not included in the design of the 2 transformed raw intensity values obtained before and after removal of non-relevant subclusters is shown in From the SGH gene possession experiment, hybridized CZ4126/02 gDNA gave a positive signal for 1702 of the 2890 total gene clusters and 2194 of the 4052 total gene subclusters represented on the array. This list of 2194 gene subclusters was then merged with the raw output from transcriptional experiments to isolate data only associated with those CZ4126/02 strain-specific subclusters. All transcriptional data in this study has been deposited in NCBI's Gene Expression Omnibus (GEO) After removal of non-relevant subcluster data, the remaining data was normalized between and across chips where necessary using the NimbleGen NimbleScan software. In the same representative sample we found only fourteen false positives (14/2194 or 0.64%)  illustrating the consistency across methods of analysis.2 of.9941 (Table S2). In this same dataset applying a fold change cut-off alone is similarly inappropriate. In this comparison of a technical replicate, 30 subclusters are found with a fold change >1.5, all false positives (Table S3). Twenty-three (23)/30 of these subclusters have expression values below 28 (256) raw intensity on both chips. These two parameters applied together (fold changes and t-tests) give some measure of biological and statistical significance that compensate for each others’ weaknesses as far fewer clusters meet both cut-offs of fold >1.5 and p<0.05 than either alone, however, there are still several that slip through as false positives.In analyzing the data here, a Bayesian-corrected variance was applied and t-tests were performed using the web-based microarray analysis tool, CyberT  of.9941 Figure 2Table S1).Therefore, a permutation-based false discovery rate (FDR) was calculated with the Significance Analysis of Microarrays (SAM) excel plug-in 8 (256 = ∼1.5X background in log2) in both conditions were not considered sufficiently above background levels and also removed. This step-wise application of filters complemented each test’s weaknesses and allowed us to observe lists of differentially regulated genes at varying levels of stringency. Fold change was not considered until after the final filter and is presented as is with no cut-off. In most cases this step-wise filtering resulted in final lists of genes up or downregulated by more than 1.5 fold which we consider to be reasonably biologically relevant.Thus, a custom step-wise filtering process was developed and implemented roughly in order of stringency. This involved obtaining the set of genes associated with the SAM permutation-based FDR<10% and then removing genes with a Bayesian-corrected p-value>.05. In some samples the SAM FDR filter was more stringent than the Bayesian-corrected p-value, therefore, the order of filtering was flipped in these cases. After this, genes were removed that had a Benjamini-Hochberg FDR>10%. The remaining genes with Bonferroni corrected p-values<.05 were selected for the next step. Any findings with raw intensity values below 2H. influenzae SGH chips for transcriptomic analyses. Two levels of technical replication were applied. The first level is contained within the array design wherein each probe is represented in duplicate on each chip. Since each subcluster is represented by between 3 and 13 probes on each chip, two sets of 3–13 values are obtained per subcluster. The NimbleScan normalization process includes a median polish whereby the 3–13 probe values are condensed to a single value, which again is calculated in duplicate. We describe this type of technical replication as probe-replication. Therefore the first validation test was to evaluate these duplicate normalized probe values as shown in 2) >0.98. Most variation occurs at expression values at or below background levels . Comparison of probe-sets was performed for an additional 9 conditions  the results of which are displayed in figures S18.An exhaustive analysis of technical replicates was performed to assess the reproducibility of the 2 values ranging from 0.9751 to as high as 0.9951. These extremely high correlation coefficients were found consistently throughout all of the transcriptomic studies performed and are presented in full in figures S26.A second level of technical replication was performed in microarray studies by hybridizing each RNA sample to two separate chips. Therefore this level of replication evaluated the reproducibility of the labeling, hybridization, scanning and normalization processes. Similar expression plots were generated by averaging the two duplicate-probe intensity values for chip 1 and plotting against the average for chip 2 5. We obTable S3). In chemically defined media (CDM), R2-values ranged between 0.9751 and 0.9945 (S28). When grown in the complex media, brain-heart infusion (BHI), the R2-values ranged between.9385 and.9696 . Furthermore, we compared the results of our microarray analysis pipeline with qRT-PCR analysis on pairs of biological replicates  per chip and here we describe a single-color fluorescence analysis pipeline. Strain-specific genome variability in terms of gene possession is captured by the SGH array a priori knowledge of the genome content thereby bypassing any artifacts of gene annotation and potentially identifying novel gene transcripts. In contrast, the described SGH microarray is targeted and those targets are fixed, thus a small amount of data will inevitably be missing for a few genes in any given strain. However, we have demonstrated here that we are able to capture the vast majority of annotated genes in any H. influenzae strain by incorporating the supragenome into the design of the array In this paper we repurpose an established CGH array for microarray-based transcriptional analysis of virtually any , Because of the multiple strain design of the SGH array some extra steps in data analysis are required that are not normally encountered in single organism microarrays. Primarily this involves removing hybridization data that is not relevant to the strain under investigation (ie probes that are designed to genes that are not present in that particular strain). A comparison of data distributions before and after the removal of non-relevant gene clusters confirmed that the transcriptional datasets are converted from non-Gaussian to Gaussian distributions , Table 1Figure S3. To compensate, a statistical comparison of mean values using T-tests is commonly used. However, this test is influenced greatly by the variance of the samples, something that is determined by both the biology of the gene and also by the equipment and methods used to measure the sample. The Bayesian-corrected variance is a method for reducing noise in microarrays which can overwhelm biological differences with limited sampling. This method infers single gene variance based on the variance of other similarly expressed genes. We found in developing our transcriptional comparison pipeline that relying on calculated Bayesian-corrected p-values in conjunction with fold change cut-offs was not stringent enough as false positives were still being detected between replicate samples (S3).For sample comparison we developed an analysis pipeline to be used with the normalized strain-relevant data that incorporates two freely available microarray analysis tools (SAM and CyberT) that together incorporate four diverse methods of statistical analysis. The most logical measure of difference used to analyze transcriptional data is the fold change, obtained simply by dividing the sample means. The weakness in drawing conclusions using fold changes alone lies in the blindness to expression levels illustrated in To further control for the discovery of false positives (FDR) we used both permutation and non-permutation based methods using SAM and CyberT. One weakness of SAM is that it is inappropriate for small numbers of replicates. This is due to the fact that the modified p-values  are calculated based on the number of times the software can randomly rearrange the replicate values (permutations). Thus SAM’s power, accuracy and significance increase with increased replication.H. influenzae microarray studies, however, we found that it was appropriate and useful for removing many genes that had fold changes under 1.5 but were still found significant by the other tests as well as for removing genes for which there was high variance between biological samples. These four filtering steps  used together in conjunction with a raw hybridization value cut-off provide a robust, flexible method for two-condition microarray analyses.As a final filtering step we used Bonferroni-corrected p-values calculated by CyberT. In many microarray analyses applying the Bonferroni correction is not performed because it is too stringent and often removes data that is known to differ between two samples . In these –, S18). We also found that technical replication by hybridization of the same cDNA samples on two different chips was just as robust as probe replication . The correlation between biological replicate experiments is also extremely high, especially when bacteria were grown in the defined media, CDM, where the growth conditions are more tightly controlled . Not surprisingly, the variation observed is somewhat higher when the bacteria were grown in the complex media, BHI, likely due to variation in the growth conditions and not the array (S30).Replicate samples were examined in detail and conclusively demonstrated that duplicate probe sets provide consistent repeatable quantitative data –, S18. WeH. influenzae SGH arrays are consistent and robust within and across chips as well as among biological replicates. Used in conjunction with the filtering pipeline described above we have described the establishment of a unique, robust microarray analysis pipeline for multi-strain comparisons specific to Haemophilus influenzae. We believe this tool could be of great use to others studying this organism and to those studying other organisms that make use of similar supragenome based arrays who may not have the resources to perform RNAseq or the expertise to deal with the associated downstream bioinformatic data analysis required.These results establish that the methods developed to obtain transcriptional data from the Figure S1Comparison of probe replicate values within RNA from condition 1, replicate A. Log2 expression values from the two probe sets in condition 1, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S2Comparison of probe replicate values within RNA from condition 1, replicate B. Log2 expression values from the two probe sets in condition 1, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S3Comparison of probe replicate values within RNA from condition 2, replicate A. Log2 expression values from the two probe sets in condition 2, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S4Comparison of probe replicate values within RNA from condition 2, replicate B. Log2 expression values from the two probe sets in condition 2, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S5Comparison of probe replicate values within RNA from condition 4, replicate A. Log2 expression values from the two probe sets in condition 4, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S6Comparison of probe replicate values within RNA from condition 4, replicate B. Log2 expression values from the two probe sets in condition 4, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S7Comparison of probe replicate values within RNA from condition 5, replicate A. Log2 expression values from the two probe sets in condition 5, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S8Comparison of probe replicate values within RNA from condition 5, replicate B. Log2 expression values from the two probe sets in condition 5, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S9Comparison of probe replicate values within RNA from condition 6, replicate A. Log2 expression values from the two probe sets in condition 6, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S10Comparison of probe replicate values within RNA from condition 6, replicate B. Log2 expression values from the two probe sets in condition 6, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S11Comparison of probe replicate values within RNA from condition 7, replicate A. Log2 expression values from the two probe sets in condition 7, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S12Comparison of probe replicate values within RNA from condition 7, replicate B. Log2 expression values from the two probe sets in condition 7, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S13Comparison of probe replicate values within RNA from condition 8, replicate A. Log2 expression values from the two probe sets in condition 8, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S14Comparison of probe replicate values within RNA from condition 8, replicate B. Log2 expression values from the two probe sets in condition 8, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S15Comparison of probe replicate values within RNA from condition 9, replicate A. Log2 expression values from the two probe sets in condition 9, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S16Comparison of probe replicate values within RNA from condition 9, replicate B. Log2 expression values from the two probe sets in condition 9, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S17Comparison of probe replicate values within RNA from condition 10, replicate A. Log2 expression values from the two probe sets in condition 10, replicate A on two separate chips (left/right).(TIF)Click here for additional data file.Figure S18Comparison of probe replicate values within RNA from condition 10, replicate B. Log2 expression values from the two probe sets in condition 10, replicate B on two separate chips (left/right).(TIF)Click here for additional data file.Figure S19Comparison of chip replicate values for RNA from condition 1. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 1, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S20Comparison of chip replicate values for RNA from condition 2. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 2, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S21Comparison of chip replicate values for RNA from condition 5. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 5, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S22Comparison of chip replicate values for RNA from condition 6. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 6, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S23Comparison of chip replicate values for RNA from condition 7. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 7, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S24Comparison of chip replicate values for RNA from condition 8. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 8, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S25Comparison of chip replicate values for RNA from condition 9. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 9, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S26Comparison of chip replicate values for RNA from condition 10. Log2 expression values (average of two probe-sets) from the same RNA samples run on two different chips: chip1 (x-axes) and chip2 (y-axes). Condition 10, replicate A (left) and replicate B (right).(TIF)Click here for additional data file.Figure S27Biological replicate comparison from Condition 1 and 2. Log2 expression values (averages of both probe-sets from both chips) from the two biological replicate samples for both condition 1 (left) and condition 2 (right).(TIF)Click here for additional data file.Figure S28Biological replicate comparison from Condition 5 and 6. Log2 expression values (averages of both probe-sets from both chips) from the two biological replicate samples for both condition 5 (left) and condition 6 (right).(TIF)Click here for additional data file.Figure S29Biological replicate comparison from Condition 7 and 8. Log2 expression values (averages of both probe-sets from both chips) from the two biological replicate samples for both condition 7 (left) and condition 8 (right).(TIF)Click here for additional data file.Figure S30Biological replicate comparison from Condition 9 and 10. Log2 expression values (averages of both probe-sets from both chips) from the two biological replicate samples for both condition 9 (left) and condition 10 (right).(TIF)Click here for additional data file.Table S1Source of RNA for each “condition”.(DOCX)Click here for additional data file.Table S2False positives found using a p-value<0.05 cutoff. Condition 4, replicate B, chip 1 and chip 2 were compared. Raw expression values are shown. FDR: False discovery rate, BH: Benjamini-Hochberg, Bon. pVal: Bonferroni-corrected p-value.(DOCX)Click here for additional data file.Table S3False positives using a fold change cutoff of >1.5. Condition 4, replicate B, chip 1 and chip 2 were compared. Raw expression values are shown. FDR: False discovery rate, BH: Benjamini-Hochberg, Bon. pVal: Bonferroni-corrected p-value.(DOCX)Click here for additional data file.


The biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes  and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies.Here we quantify the impact of these combined “study-effects” on a disease signature’s predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance.As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification.We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when “sufficient” diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings. There has been substantial effort to develop disease diagnostic strategies based on analyzing large-scale molecular information  from patients. Numerous studies aiming at developing such molecular diagnostics have examined omics data, both directly A major factor hindering the consistency of identified disease classifiers and their performances stems from variability in omics data attributed to technical and biological influences that are unrelated to the specific phenotypic differences under study. Gathering gene expression data from different batches-processed at a specific experimental site and time–introduces technical variability, termed “batch-effects” study-effects herein to describe the joint variability that stems from both technical variation introduced by batch-effects and the biological variation associated with population heterogeneity. Importantly, the presence of these study-effects is not necessarily a reflection of the quality of the laboratories or experimental studies; rather, they emphasize that measured gene expression is sensitive to a broad range of influences. Although numerous excellent studies have examined We use the term inter-study validation (ISV), we identify phenotype-specific classifiers based on data pooled from all studies except for one, and then evaluate the predictive performance on the excluded study. This process is repeated for all studies, leaving each one out for testing and training a predictor on the data combined from all others. This differs from randomized cross-validation (RCV), the standard in machine learning, wherein a random subset of the pooled data, for example ten percent in ten-fold RCV, is set aside for testing, and the predictor is identified from the pooled data excluding this subset. This process is then repeated, for example ten times. The critical difference between these validation strategies is that RCV (and other methods that split data randomly) estimates classification error under a condition (namely random sampling) in which the training and testing data are drawn from the same distribution. In other words, the assumption is made, at least implicitly, that future samples from other studies encountered by the classifier will display the same statistical properties as the training data–a condition that is often violated in real world settings. Hence, randomized sampling obscures systematic differences in expression distributions associated with study-effects. In contrast, ISV is sensitive to study-effects because it preserves the variation among studies . Consequently, estimates of classifier performance derived using ISV are often low when there are expression patterns that vary substantially between studies within phenotypes. Therefore, the magnitude of discrepancies between performance estimates from ISV and RCV reflects the extent of study-effects. We refer to the comparison of ISV and RCV performance estimates as comparative cross-validation analysis (CCVA). CCVA can be applied in subsets of study comparisons in various ways, for example as a measure of how well controlled a multi-site clinical trial is performing in terms of cross-site variation.The motivation of our study is to examine the influence of study-specific variability in gene expression data on disease classification prediction error and suggest how to mitigate this influence to achieve improved classification performance. Our approach to measuring the influence of study-effects on classification involves assessing classification performance with a study-centric validation strategy. In In this study, we use comparative cross-validation analysis to gauge the extent to which study-effects confound gene expression-based disease classification performance. We find that including data from more studies improves representation of biological heterogeneity during the disease signature learning process, which mitigates the influence of study-effects. By tracking the difference between the ISV and RCV performance, we measure the extent to which introducing heterogeneity into the training data alleviates the influence of study-effects on disease classification and can thereby estimate when sufficient data has been incorporated to generate classifiers that are robust to study-specific biases.https://price.systemsbiology.net/measuring-effect-inter-study-variability-estimating-prediction-error), and we have added our uniformly preprocessed dataset to GEO with accession number: GSE60486. It should be noted that this dataset is a compilation and standardization of data generated from other groups.For the microarray data, we download the raw.CEL files of each microarray experiment, either from the Gene Expression Omnibus http://www-huber.embl.de/users/anders/HTSeq/doc/overview.html). We consolidate the data across studies and perform correlation analysis using Python.For the RNA-seq data, we download raw fastq files from the Sequence Read Archive sensitivity, defined as the fraction of samples of that phenotype that are correctly identified by the learned molecular signature.To estimate the significance of study-effects on classification performance, we compare two metrics for evaluating classification sensitivity: inter-study validation (ISV) and the commonly used randomized cross-validation (RCV). For each method, classification performance metric we use is the phenotype-specific In ISV, the expression data from each experimental study is excluded from the training set used to find diagnostic classifiers. Data from the remaining studies is used to train a classifier using an algorithm of choice. The data from the excluded study acts as a test set to evaluate the performance of a classification algorithm. This ISV is repeated for every study included in the analysis, and the sensitivity for each phenotype evaluated on each excluded study is reported. To mitigate sample number bias in training, we implemented a stratified training loop .For each phenotype, we compare the average of the ISV sensitivities across all studies of that phenotype against the phenotype-specific sensitivity obtained by ten-fold RCV. This approach is implemented in MATLAB .We also investigate the effect of including different numbers of studies in the considered dataset on ISV and RCV outcomes. In this cumulative CCVA, we focus on the three phenotypes with greater than five independent studies: ADC, SCC, and NORM. For each of these phenotypes, we combine data from We evaluate predictive performance of support vector machine (SVM) p<0.05 set as the significance threshold.Non-parametric significance tests are used where possible, with We have assembled lung-related expression data from two publicly available online databases: Gene Expression Omnibus We apply CCVA on our lung microarray dataset using two very different multi-class classification schemes–the commonly used linear one-versus-one multiclass support vector machines (SVM) The results in AST = 1.7, CoVarCOPD = 1.1 for SVM; CoVarAST = 0.54, CoVarCOPD = 1.0 for ISSAC), whereas SCC and ADC have higher consistency .Besides yielding improved ISV performance relative to RCV, phenotypes with more studies achieve more consistent ISV results. We quantify ISV consistency by the coefficient of variation (CoVar), which is equal to the standard deviation of ISV sensitivities across studies belonging to a particular phenotype divided by the average. Lower CoVar values translate to higher consistency. Phenotypes with fewer studies, such as AST and COPD, have low consistency , indicating enhanced signature performance and consistency with the addition of data from different studies. Further analysis shows that the trends of ISV performance are reflected in large-scale differences in the expression profiles that can be visualized by principal component analysis, and that these performance trends also have association with consistency of selected gene signatures . Moreover, we observe a converging of ISV sensitivity toward the corresponding RCV sensitivity as additional studies are added, although the rates of convergence differ between phenotypes. As the ISV sensitivity approaches RCV sensitivity, the incremental improvement of average ISV sensitivity drops with the addition of further studies. We also observe a significant negative correlation between the CoVar and the corresponding ISV sensitivity  is significantly lower than corresponding RCV sensitivity (0.98±0.03) , indicating that NORM sensitivity is indeed compromised by study-effects. Notably, we find that NORM ISV sensitivity, which is achieved by learning a molecular signature from two NORM training studies (0.68±0.21), is greater than the sensitivity achieved when only one NORM study was used to learn a molecular signature (0.56±0.26). Increasing the number of RNA-seq studies included in training improves NORM classification performance on untrained studies and does not significantly change ADC performance on untrained studies, suggesting that greater benefit in classification can be gained by the integration of data from additional NORM RNA-seq studies.Generating molecular signatures that yield consistently high predictive capability in diagnostic tests remains a critical challenge in omics-based biomarker discovery. We find that study-effects stemming from both technical and biological variability substantially decrease predictive performance and consistency when data from only a few studies are considered in learning molecular signatures. However, our results show that incorporating data from additional independent studies mitigates the impact of study-effects, thereby reducing the predictive error. The qualitative trends of our study-effects results remained consistent across microarray and RNA-seq datasets. Given that study-effects also account for intrinsic biological heterogeneity, the trends associated with study-effects are relevant to classification even with improved technologies.A significant source of variability arises not from technical batch-effects but rather is an inevitable consequence of the inherent heterogeneity of many disease phenotypes. Given that study-effects associated with disease heterogeneity are biological, they can best be accommodated by collecting data from multiple sources. The key point is that the diversity represented in the training and test set needs to reflect the range of diversity expected in the clinical setting. Subpopulations from different studies have different underlying distributions of expression, so sampling data from multiple independent studies improves the approximation of the global distribution across multiple sites – even when sample sizes are held constant – which aids the identification of consistent classifiers. Therefore, the improvement in classification performance that results from training classifiers on larger numbers of studies highlights the need to incorporate more population heterogeneity in future biomarker discovery studies by integrating data from multiple sites, including additional sites in test validation.Comparative cross-validation analysis provides a quantitative basis for prioritizing strategies for improving classification of different phenotypes. For example, our CCVA results highlight phenotypes for which diagnostic reproducibility was most greatly affected by study-effects. These phenotypes are the most suitable candidates for further data gathering and analysis to immediately yield better classification outcomes. In contrast, in phenotypes with average ISV sensitivities that approach RCV sensitivities, our results suggest that, because of the difficulty of the gene expression-based classification problem, simply gathering more gene expression data and using the same algorithms would not likely substantially improve performance. In these cases, leveraging other strategies for classification, including redefinitions of molecular phenotypes, integration of multi-omic data, and contextualization with biological networks, may be more beneficial to finding classifiers with more consistent performance. By measuring the improvement in classification once study-effects have been mitigated, CCVA can be used to guide future data gathering efforts.In this study, we quantify the degree of impact of technical and biological “study-effects” on disease classification performance. We find that learning diagnostic signatures on larger numbers of studies compensates for study-effects and results in marked improvements in classification performance. Moreover, we can estimate when “sufficient” diversity has been achieved for learning classifiers that are likely to translate effectively to new clinical settings. These results are relevant to phenotype prediction using data across measurement technologies. Our finding that study-effects can be quantifiably mitigated by introducing data collected from additional studies has applicability to disease classification study design strategies. It has clear implications for study design because diversity of samples in the training set (e.g. from multiple sites) shows markedly better consistency in predictive accuracy when taken to new clinical sites, as is a needed step on the path to clinical use. This underscores the need to incorporate more population heterogeneity in future classification studies by integrating data from multiple sources. Additionally, we find that different phenotypes require different degrees of training heterogeneity to mitigate study-effects. Applying comparative cross-validation analysis, we can discriminate between phenotypes that would benefit from for further data gathering to increase training heterogeneity from the phenotypes may require different analysis strategies to reduce predictive error. Therefore, our approach provides a computational tool for prioritizing strategies to improve disease classification.Table S1Microarray Data Details. This contains detailed information of each of the experimental studies that had data included in our analysis.(XLSX)Click here for additional data file.Text S1Supplemental Document. This contains descriptions of supplementary analyses and descriptions of methods.(DOCX)Click here for additional data file.