DGEclust, for differential expression analysis of digital expression data. Our method treats differential expression as a form of clustering, thus unifying these two concepts. Furthermore, it simultaneously addresses the problem of how many clusters are supported by the data and uncertainty in parameter estimation. DGEclust successfully identifies differentially expressed genes under a number of different scenarios, maintaining a low error rate and an excellent control of its false discovery rate with reasonable computational requirements. It is formulated to perform particularly well on low-replicated data and be applicable to multi-group data. DGEclust is available at http://dvav.github.io/dgeclust/.We present a statistical methodology, The online version of this article (doi:10.1186/s13059-015-0604-6) contains supplementary material, which is available to authorized users. Next-generation sequencing (NGS) and high-throughput sequencing are revolutionary tools for the study of the genome, epigenome and transcriptome in a multitude of organisms (including humans) by allowing the relatively rapid production of millions of short sequence tags, which mirror particular aspects of the molecular state of the biological system of interest. A common application of NGS is the study of the transcriptome, which involves a family of methodologies, such as RNA sequencing (RNA-seq) , cap anaClustering is considered an important tool in the study of genomic data and it has been used extensively in the analysis of microarrays -12 (see k-means clustering and self-organising maps [dendrogram. This stands in contrast to partition-based methods (e.g. k-means or self-organising maps), which decompose the data directly into a finite number of non-overlapping clusters [A large class of clustering methods relies on the definition of a distance metric, which quantifies the similarity between any two gene expression data points. Subsequently, clusters are formed, such that the distance between any two data points in the same cluster is minimised. Typical methods in this category are ing maps . Anothering maps . Hierarcclusters .DGEclust) for clustering digital expression data and we show how it can be used in differential expression analysis. From a theoretical standpoint, the significance of the proposed methodology stems from its unification of differential expression and clustering . This makes it possible to bypass the need for a particular statistical test when inferring differential expression or for multiple hypothesis testing correction, such as the Benjamini–Hochberg procedure. From a statistical point of view, the proposed methodology is important because it simultaneously addresses the problem of model selection (i.e. how many clusters are supported by the data) and uncertainty (i.e. the error associated with estimating the number of clusters and the parameters of each cluster). This is made possible by exploiting a hierarchical Dirichlet process mixture model (HDPMM) [In this article, we present a model-based statistical methodology and associated software ( (HDPMM) ,17, a st (HDPMM) ,19, for  (HDPMM) , for int (HDPMM) ,22 and f (HDPMM) . In our  (HDPMM) -27) and  (HDPMM) .Formally, the production of count data using NGS assays can be thought of as random sampling of an underlying population of cDNA fragments. Thus, the counts for each tag describing a class of such fragments can, in principle, be modelled using the Poisson distribution, whose variance is, by definition, equal to its mean. However, it has been shown that, in real count data of gene expression, the variance can be larger than what is predicted by the Poisson distribution -32. An aith feature (e.g. gene) at the jth sample/library with the variable yij. Each library j is assigned to a group of libraries l=λ(j), based on prior information on tissue, experimental condition, disease state, etc. There are a total of N genes, M libraries and L groups of libraries, where 2≤L≤M. We assume that yij is distributed according to a negative binomial distribution, with mean mij and dispersion parameter ϕi:We indicate the number of reads for the mij)= log(cj)+ log(μi)+βiλ(j). In the previous expression, cj is a known a priori normalising factor for library j, μi is the mean expression level for gene i and βiλ(j) is the fold-change of the mean expression level of gene i in group λ(j). A fold-change equal to 0 indicates no change in the mean expression level for feature i in group λ(j), while a value larger  than 0 indicates over(under)-expression.where log for each gene i and group of samples λ(j). We also need to compute the normalising factors cj from the data , followed by giving the priors for the ϕi and μi parameters.A common limitation in experiments using NGS technologies is the low number or even absence of biological replicates, which complicates the statistical analysis of digital expression data. One way to compensate for small sample sizes is to assume that all genes share the same variance . A less l=λ(j), we assume that the gene-specific fold-changes {βil} are random and distributed according to a prior distribution Gl, i.e.Within each sample group Gl is itself randomly sampled from a Dirichlet process with positive concentration parameterγl and base probability distribution G0 [Furthermore, we assume that ution G0 :(3)\docGl from Equation βs. Large values of γl lead to a large number of similarly likely values of β, while small values of this parameter imply a small number of highly probable values of β. This and Equation βil within the lth group of samples are not all distinct. Different genes may share the same value of β or, in other words, genes are grouped in a (not known in advance) number of clusters, based on the value of β they share. Equivalently, the expression profiles of different groups of genes are drawn from different negative binomial distributions, each characterised by its own unique value of β. This clustering effect is illustarted in Figure Dirichlet process priors are distributions over distributions and they have become a popular choice in Bayesian inference studies, since they provide an elegant solution to the problem of determining the correct number of components in mixture models. Standard theoretical results  state thG0 is itself random and sampled from a Dirichlet process with a global scaling parameter δ and a global base distribution H:Up to this point, we have considered clustering of genes within the same group of samples, but not across groups of samples (e.g. tissues or conditions). However, in a given dataset, each cluster might include gene expression profiles from the same, as well as from different groups of samples. In other words, clusters are likely shared between samples that belong to different groups. This sharing of information between sample groups can be expressed naturally in the context of HDPMMs . FollowiG0 is (like each Gl) discrete over a countably infinite set of atoms βk′, which are sampled from H, i.e. βk′∼H. Since G0 is the common base distribution of all Gj, the atoms βk′ are shared among all samples, yielding the desired information sharing across samples . Notice that the base distribution H, which provides the global prior for sampling the atoms μβ and variance ϕi was assumed normal with mean μϕ and variance μi are modelled as ϕi is sampled as above and pi∼Beta. This formulation greatly facilitates the posterior inference of μi given ϕi, since it involves sampling directly from well-known distributions  belongs to. Two different β’s, βil and zil and z|y) of matrix z given the data matrix y.The definition of the HDPMM in Equations DGEclust, which efficiently generates new samples from the posterior p(z|y). The algorithm is an extension of the method presented in [z independently of all others. This not only results in very fast convergence, but it also allows the algorithm to be implemented in vectorised form, which takes advantage of the parallel architecture of modern multicore processors and potentially permits application of the algorithm on very large datasets. Alternative MCMC methods, which are developed on the basis of the popular Chinese restaurant franchise representation of the Hierarchical Dirichlet Process (HDP) [One approach to estimate this distribution is by using Markov chain Monte Carlo (MCMC) methods, which generate a chain of random samples as a numerical approximation to the desired distribution. We have developed a blocked Gibbs sampler in the software package ented in ,40 for iss (HDP) ,41, do ny and a sufficiently large chain of samples z|y) – has been generated, we show how these samples can be used for differential expression analysis. We consider two classes of samples, A and B, which might represent, for example, two different tissues or experimental conditions.Assuming that the above algorithm has been applied on a digital expression dataset not differentially expressed (DE), if its expression measurements in classes A and B belong to the same cluster. In more formal language, we state that the posterior probability πi that gene i is not DE given data y is equal to the conditional probability p(ziA=ziB|y) that the indicator variables of feature i in sample classes A and B have the same value. This probability can be approximated as a simple average over the previously generated MCMC samples A particular gene is said to be  of potentially DE features with probabilities less than this threshold, i.e. πi is calculated as in Equation i.where 1 1(·) is equal to 1 if the expression inside the parentheses is true and 0 otherwise. Given a threshold πi measures the conditional probability that including the ith gene in list  is a Type I error, i.e. a false discovery (FD). This useful property makes possible the calculation of the conditional false discovery rate (FDR) as follows:As observed in , the qua always has an FDR at most equal to From Equation P values, this methodology does not require any correction for multiple hypothesis testing, such as the Benjamini–Hochberg procedure. Although the computation of FDR using Equation πi using Equation Notice that, unlike alternative approaches, which make use of gene-specific DESeq/DESeq2, edgeR and baySeq.To assess the performance of our methodology, we applied it on simulated and actual experimental count data and we compared our results against those obtained from popular software packages, namely generateSyntheticData from the independent Bioconductor package compcodeR [generateSyntheticData were left as their default values, unless stated otherwise.First, we applied our algorithm on simulated data, which provides the advantage that we can control the exact conditions under which the data was generated, including the true differential expression state of each gene. For this purpose, we used the function ompcodeR . Data geompcodeR ,45. EachDGEclust and the other aforementioned methods under the experimental conditions encapsulated by the synthetic data using the following criteria: (a) first, we compared the ability of all methods to identify DE genes using as performance measure the area under the receiver operating characteristic (ROC) curve (AUC) for each method, (b) second, we compared the ability of all methods to keep a low number of Type I errors using FD curves as performance measures and also by measuring the number of Type I errors each method returned given a fixed significance threshold, (c) third, we assessed the ability of all methods to keep a low FDR given a relatively large pre-specified FDR threshold. Furthermore, we applied all methods on actual RNA-seq data from a number of species, as well as on a large CAGE dataset obtained from five different regions of human brains. Details for each category of experiment are given in the following sections.Following the prototypical study in , we compDGEclust to identify truly DE genes. All examined methods rank each gene by providing P values (edgeR and DESeq/DESeq2) or posterior probabilities (DGEclust and baySeq). Given a threshold score, genes on opposite sides of the threshold are tagged as DE or non-DE, accordingly. In an artificial dataset, the genes that were simulated to be DE are considered to be the true positive group, while the remaining genes are considered the true negative group. By computing the false positive rate (FPR) and the true positive rate (TPR) for all possible score thresholds, we can construct ROC and FD curves for each examined method. The AUC is a measure of the overall discriminative ability of a method (i.e. its ability to classify correctly features as DE or non-DE).We first evaluated the comparative ability of DGEclust clearly demonstrates the best performance with an AUC score larger than 80%, followed by DESeq/DESeq2, edgeR and, finally, baySeq. The same trend is observed when we examine datasets with 30% DE genes (second panel), but this time the difference between DGEclust and the other methods is even more prominent. This is because the performance of edgeR, DESeq/DESeq2 and baySeq is negatively affected by the increased proportion of DE genes, while the performance of DGEclust remains essentially the same.Our results are summarised in Figure DESeq, edgeR and baySeq, when datasets with large proportions of DE genes were considered, has already been observed [DGEclust does not seem to be affected by the presence of this asymmetry in the data, although it adopts the same normalisation method as DESeq.The negative effect of DE gene composition on observed  and attrDGEclust ranked first in all cases, but DESeq/DESeq2 and edgeR followed closely, particularly at large sample sizes (n=8). The introduction of outliers with abnormally high or low counts (bottom panel), reduced the AUC for all methods, but again DGEclust ranked first among the alternatives for all sample sizes.We further evaluated the effect of introducing non-over-dispersed genes or outliers, as outlined above. When the fraction of Poisson distributed genes was increased from 0 to 50% (third panel), we observed a significant increase in the AUC for all methods. DGEclust and the other methods to control the rate of Type I errors. First, we assessed this ability through the construction of FD curves, which illustrate the number of false positives as a function of the total number of positives (i.e. as the decision threshold increases). Mean FD curves for each of the cases examined in Figure DGEclust always keeps the number of false positives among the top discoveries below that of the other evaluated methods. As with the AUC scores, this is more prominent when a large number of DE genes (30%) is considered and it holds at all sample sizes. When examining datasets including non-over-dispersed genes (third panel), DGEclust still performs better than alternative methods, at all sample sizes. For datasets with outliers (bottom panel), the difference between DGEclust and the other methods becomes less prominent and effectively disappears at large sample sizes (n=8), at which point DESeq2 also demonstrates good performance.Next, we evaluated the ability of P values (DESeq/DESeq2 and edgeR) and methods that return posterior probabilities (DGEclust and baySeq) and since all methods provide an estimation of the FDR, we compared the number of Type I errors  each method made at a relatively high pre-specified FDR = 0.1. In all datasets not including outliers (top two panels), DGEclust had the lowest proportion of Type I errors at small sample sizes (n=2 and n=4), followed closely by baySeq. At large sample sizes (n=8), DGEclust, DESeq and baySeq performed similarly, followed closely by DESeq2 and edgeR. After the inclusion of outliers (bottom panel), DGEclust again had the lowest Type I error rate along with DEseq at all sample sizes.In a second stage, we examined the ability of all methods to control the number of Type I errors at a pre-specified threshold level in the absence of any truly DE genes  or estimated FDRs  were less than or equal to the aforementioned significance level. If no discoveries were identified, the corresponding true FDR was assumed undefined.Furthermore, we evaluated the ability of DGEclust was the top or among the top-performing methods in terms of controlling the FDR at a pre-specified level. As a general observation, increasing the number of replicates per condition from 2 to 4 to 8, clearly increased the ability of all methods to control the FDR. DGEclust demonstrated excellent control of its FDR, keeping it close to or below the pre-specified significance threshold, in all examined cases. When no outliers and only over-dispersed genes were considered (top two panels), DGEclust was the only method that kept the FDR below the pre-specified threshold at low sample sizes (n=2). At larger sample sizes , DESeq appears to reduce this gap and in one case , it performs as well as DGEclust. In the presence of non-over-dispersed genes (third panel), DGEclust still retains its FDR at  or slightly above (n=2) the pre-specified threshold, although it is now marginally overtaken by baySeq (n=2) or DESeq . When genes with unusually low or high counts were considered (bottom panel), DGEclust managed to retain excellent control of its FDR, even at small sample sizes, and with quite significant differences from all alternative methods. DESeq did not call any DE genes in this case and its FDR was deemed undefined.Our results are summarised in Figure ReCount [In addition to simulated data, we also tested our method on RNA-seq data from mouse, rat, worm and fly and on CAGE data from humans. The RNA-seq data were obtained from the online resource ReCount  and theyn=2) and, after filtering out all rows with zero count sum, a total of 9,502 genes. Similarly, the rat data [n=2) and, after filtering as above, 17,125 genes. The filtered (as above) worm data [n=2) and 19,430 genes. Finally, the fly data [n=3). After removing all rows with zero count sum across all samples, the filtered dataset included 13,188 genes. For more details about how these data were obtained, the reader is referred to [The mouse data  consisterat data  consisteorm data  consisteerred to  and the n=5) and it included 23,448 features, i.e. tag clusters representing promoter regions  and non-DE if this ratio was smaller than 1/2 (corresponding to changes in expression less than approximately 41%). All intermediate cases were considered undefined and they were ignored during computation of ROC curves. For CAGE data, we further validated this approach of approximating the true state of differential expression for each gene by adopting the biological homogeneity index (BHI) [As in , we usedex (BHI)  as an adDGEclust and edgeR  or baySeq (mouse and rat). In several cases , DGEclust achieved almost a perfect classification and in all cases it was the only method that consistently demonstrated top performance. It is interesting to observe that while DGEclust is clearly the best method at small sample sizes (n=2), at larger sample sizes (fly with n=3 and human with n=5) edgeR also performed equally well. Based on this evidence, we conclude that DGEclust has a clear advantage compared to alternative methods at low replication levels.The performance of all tested methods was excellent, reaching a TPR larger than 70% at minimal (≪0.01) FPRs Figure . The twoDGEclust for such datasets by using the CAGE data from human brains as a test case. DGEclust processed the data for 10K iterations, of which the first 5K were rejected as burn-in, while the remaining 5K were used for estimating differential expression, as outlined in a previous section. We also applied edgeR and DESeq2 on the same dataset. baySeq was excluded from this analysis, because this would require examining a large number of possible expression patterns, which we found impractical.Up to this point, we have restricted our analysis to datasets consisting of two groups of samples. While pairwise comparisons have been characterised as the bread and butter of differential expression analyses, multi-group datasets are also quite common. In this section, we demonstrate the applicability of % at FPRs smaller than 1%. In all cases, DGEclust was the top performer as indicated by the AUC, followed by edgeR and then DESeq2 in all cases. Examination of the Venn diagrams constructed from the DE genes, which were identified by each method at an FDR cutoff of 10%, indicates a significant overlap between all three methods, with at least 1K genes commonly identified as DE in most cases. In terms of novel discoveries , DGEclust appears to occupy the middle spot between DESeq2 and edgeR, with DESeq2 calling the largest number of novel discoveries in most cases. Two interesting exceptions were the caudate nucleus/putamen pair and the temporal/frontal lobes, for which DGEclust appears to have called the largest number of novel DE genes.Having established a ground truth as outlined in an earlier section, we constructed ROC curves for all possible pairs of brain regions in the CAGE dataset. It may be observed  at a distance of 0.5, which optimises the expectation of Binder’s loss function [DESeq2’s rlog function.For clustering the DE genes called by each method, we used the function . In bothDGEclust, we additionally performed hierarchical clustering using a similarity matrix computed internally by our software. Specifically, each element i and i′ is defined as follows:For L is the number of brain regions in the dataset.where DGEclust supports BHI scores at least as good as those computed for the other methods  are no worse than those returned from the other methods. This indicates that the clusters supported by the DE genes called by DGEclust are at least as biologically homogeneous as those supported by the other methods, thus increasing our confidence in the ability of DGEclust to identify DE genes correctly as indicated in Figure Our results from this analysis are indicated in Table l and l′ and it is computed as follows:Furthermore, we investigated the relation between different brain regions by constructing a similarity matrix, which we used as input to hierarchical clustering routines for the generation of dendrograms and heat maps. Every element N is the number of genes in the dataset. The similarity matrix calculated as above was used to construct the dendrograms and heat map in Figure where DGEclust for calling DE genes, as presented in Figure Collectively, the evidence presented in Figure IPython’s %memit and %timeit commands to measure peak memory usage and computation time. All simulations were performed on a MacBook Pro with an Intel four-core i7 processor and 8 Gb of memory.Bayesian approaches employing MCMC sampling methodologies for inference are notorious for their increased computational requirements. Being conscious that this might discourage application of this class of methodologies to large genomic datasets, we investigated how our method scales with increasing number of samples and genes. Specifically, we used NumPy’s vector notation and, for this reason, top performance in terms of speed is expected. Groups of samples can also be processed in parallel using multiple cores, if the user wishes to do so. Peak memory usage and computation time also increase linearly with the number of samples (right top and bottom panels), although peak memory usage appears roughly constant over the examined range for the number of samples in the absence of multiprocessing. There is a clear gain in processing speed from using multiple cores to process samples (top right panel). In conclusion, these data suggest that DGEclust is fundamentally applicable even for large genomic datasets, particularly when multiple cores are used.As indicated in Figure digital measure of gene expression. These data are fundamentally different in nature  from the continuous fluorescence intensity measurements obtained from the application of microarray technologies. In comparison, NGS methods offer several advantages, including detection of a wider level of expression levels and independence of prior knowledge of the biological system, which is required by hybridisation-based microarrays [Despite the availability of several protocols (e.g. single vs paired-end) and sequencing equipment , all NGS technologies follow a common set of experimental steps , instead of features (e.g. genes) and they employ distance-based or hierarchical clustering methodologies on appropriately transformed datasets, e.g. ,56,57. Fk-means [The above approaches, although fast and relatively easy to implement, do not always take into account the discrete nature of digital gene expression data. For this reason, various authors have developed distance metrics based on different parameterisations of the log-linear Poisson model for modelling count data, e.g. -60. A mok-means .a priori. For example, both the k-means and the self-organising map algorithms require the number of clusters as input. Similarly, methods that model the data as a finite mixture of Poisson or negative binomial distributions [Although computationally efficient and attractive due to their relative conceptual simplicity, the utility of both distance- and finite model-based clustering methods has been criticised ,41. One ibutions -63 requiDGEclust) for clustering digital gene expression data, which  doesDespite the infinite dimension of the mixture model, only the finite number of clusters supported by the data and the associated parameters are estimated. This is achieved by introducing a blocked Gibbs sampler, which permits efficient processing of large datasets containing more than 20K genes. Unlike MCMC inference methods for HDPMM based on the popular Chinese restaurant franchise metaphor ,41, our The second important contribution of this study is demonstrating how this type of hierarchical clustering can be used for differential expression analysis. We emphasise that differential expression can be thought of as a particular form of clustering. Through comparison with popular alternatives for both simulated and actual experimental data, we demonstrate the applicability of this approach for a wide range of experimental settings and its improved performance, particularly at small sample sizes, which reflects the design of current sequencing experiments.DGEclust, will prove to be a useful addition to the existing tools for the statistical analysis of RNA-seq and similar types of data.In conclusion, we have developed a hierarchical, non-parametric Bayesian method for modelling digital expression data. The novelty of our method is simultaneously addressing the problems of model selection and estimation uncertainty and exposing the intimate relation between clustering and differential expression analysis. We expect our work to inspire and support further theoretical research on modelling digital expression data and we believe that our software, DGEclust, which is written in Python and uses the SciPy stack. DGEclust expects as input and clusters a matrix of unnormalised count data along with replication information, if this is available. The output of the clustering process is required as input to post-processing routines, which compute the posterior probabilities that each particular gene between any two groups of samples in the data are DE. Further post-processing routines can generate gene- or library-wise similarity matrices, which can be used as input to hierarchical clustering routines for the generation of heat maps and dendrograms. DGEclust takes advantage of multicore processors to accelerate computations. All analyses in this article were performed using DGEclust and standard Python/SciPy tools, as well as DESeq/DESeq2, edgeR and baySeq for comparison purposes. When using these packages, all parameters were left at their default values. For baySeq, we used 5K samples when estimating the priors using the quasi-likelihood approach. To make a comparison possible, we processed all datasets with DGEclust for 10K iterations, rejecting the first 5K of them as burn-in.We implemented the methodology presented in this article in the software package DGEclust uses the same normalisation method as DESeq. This behaviour can be overridden by providing a set of library sizes as input. When comparing different software packages, we used the default normalisation method of each package.Internally, Post-mortem human brain tissue from frontal, temporal, hippocampus, caudate and putamen regions from five donors was obtained from the Netherlands Brain Bank . All tissue requests received at the NBB are reviewed by the NBB’s scientific committee and all materials and data collected are obtained with written informed consent. The procedures, information and consent forms of the NBB have been approved by the Medical Ethics Committee of the VU Medical Centre .Total RNA was extracted and purified using the Trizol tissue kit according to the manufacturer’s instructions . CAGE libraries were prepared according to the standard Illumina CAGE protocol . BrieflySequenced reads (tags) were filtered for known CAGE artefacts using TagDust . Low quaDGEclust is available online under the MIT licence [The most recent version of  licence .Additional file 1Supplementary Material for DGEclust: differential expression analysis of clustered count data. In this supplementary material we provide a detailed account of posterior inference in the model summarised by Equations


Currently, in the era of post-genomics, immunology is facing a challenging problem to translate mutant phenotypes into gene functions based on high-throughput data, while taking into account the classifications and functions of immune cells, which requires new methods.Gata3-KO T cells and Stat3-KO T cells) in a data-oriented manner. With a new concept, differentiation variable, CCA provides an automatic classification of cell samples, which had a high sensitivity and a comparable performance to other classification methods. In addition, we elaborate how CCA results can be interpreted, and reveal the features of CCA in comparison with other visualisation techniques.Here we propose a novel application of a multidimensional analysis, Canonical Correspondence Analysis (CCA), to reveal the molecular characteristics of undefined cells in terms of cellular differentiation programmes by analysing two transcriptomic datasets. Using two independent datasets, whether RNA-seq or microarray data, CCA successfully visualised the cross-level relationships between genes, cells, and differentiation programmes, and thereby identified the immunological features of mutant cells  in the context of cellular differentiation. The proposed approach fits with common hypothesis-oriented studies in immunology, and can be used for a wide range of molecular and genomic studies on cellular differentiation mechanisms.The online version of this article (doi:10.1186/1471-2164-15-1028) contains supplementary material, which is available to authorized users. Analysis of mutant phenotypes has been the major means to reveal gene functions in molecular biology . Currentmultidimensional , and each measurement can be high dimensional (e.g. many genes). In addition, it is common in immunological genomic data that the number of experimental groups is larger than that of replicates , bec Project .Signature is defined by the characteristic expression of a set of genes in a particular cell subtype [In order to analyse such multidimensional data across different experiments, currently the gene signature approach is commonly used in immunology.  subtype , 7–10. H subtype , 12. Gen subtype , 14. Thi subtype , 14. Themultidimensional in non-biomedical disciplines such as ecology and sociology, and accordingly, multidimensional analyses including CCA have developed and widely used in these areas [CCA on microarray data, CCAM) to visualise the cross-level relationships between pathological and physiological processes for addressing haematological problems [In this study, we have adapted Canonical Correspondence Analysis (CCA) to cross-analyse a transcriptomic dataset of interest (response data) and another transcriptomic dataset (explanatory data) that defines cellular differentiation programmes. CCA measures and visualises similarities (i.e. correlations) between elements across three different levels: genes, cells, and differentiation programmes. Mathematically, CCA uses linear regression and singular value decomposition (SVD), and thereby identifies the linear combinations of explanatory variables that maximise the dispersions of samples in response variables . Thus, Cse areas , 17. We problems . In the + T cells for T cell differentiation in this study. It is known that CD4+ T cells, upon antigenic stimulation, differentiate into functionally distinct T cell subsets including interferon-gamma (IFN-γ)-secreting helper T cell-1 cells (Th1), interleukin (IL)-4-secreting Th2, IL-17-secreting Th17 cells, and Foxp3-expressing regulatory T cells (Treg), depending on the cytokine and morphogen milieu [We have analysed transcriptomes of CD4n milieu . The linn milieu –20. Accon milieu . Thus, tX), and interpret the results of the analysis by current knowledge (e.g. the literature and annotation databases) on the genes that are related to D  as explanatory variables. The standardised matrix of X, S, is projected onto Z, and thus, the projected space QS is the interpretable part of the main data X by the explanatory variables. SVD is applied to QS, producing sample and gene scores (X and Gene in the new space). Differentiation programmes are visualised as regression coefficients between Z and the new axes. These results are visualised as a triplot that show relationships between cell subsets, genes, and differentiation programmes, facilitating hypothesis-generation based on the interpretation of data in a data-oriented manner  is disturbed in undefined cell subset X , in order to visualise the relationships between the geographical location (site), fish species, and environmental gradients in the ocean , 22. In differentiation variable as explanatory variable, which is equivalent to environmental gradient by ter Braak [d is defined as the responses of a set of transcripts when a cell changes its phenotype from X to Y, d = μy – μx , using mean gene expression profiles of X and Y, μx and μy, which is equivalent to environmental gradient by ter Braak [While visualisation is the primary strength of CCA, we have developed a new approach for characterising and classifying samples using CCA by introducing er Braak , 22. Herer Braak , 22. ThuWe have analysed two immunological problems using two sets of transcriptomic data in this study. In each analysis, we first examine the visualisation ability of CCA and elaborate how CCA results can be interpreted. Next, we compared the classification ability of CCA with other classification methods. Table Gata3-KO and WT T cells including Th1, Th2, Th17, and iTreg (GSE20898 [Gata3 dataset) and a microarray dataset that analysed the same Th subsets from WT mice (GSE14308 [Th dataset). The purpose of this analysis is to identify which Th differentiation programme is most disturbed by the deletion of the Gata3 gene. The results of PCA using these datasets are shown in Figure In this analysis, we analysed an RNA-seq dataset of GSE20898 , designaGSE14308 , designaCCA was applied to the Gata3 dataset, using the Th dataset as explanatory variables. CCA clustered Th1, Th2, Th17, and iTreg RNA-seq samples in the first 3 axes  was similar between two analyses . Thus, even considering that the overall dispersion of the Th1 score was approximately 30% larger, the difference between WT and KO in the Th2 score was most remarkable in these two analyses.In order to further examine the correlations between samples and the Th1 and Th2 differentiation programmes, we analysed the Gata3 dataset using corresponding differentiation variables. Using a Th2 differentiation variable, the CCA solution showed that WT Th2 cells had the highest scores, while KO Th2 and other Th cell populations had low scores , linear discriminant analysis (LDA), K-nearest neighbor (KNN), Naïve Bayes (NB), and Random Forest (RF). The Th dataset was used as a training data (a resource dataset for CCA), and WT data from the Gata3 dataset was used as a test data. Thus, we addressed how efficiently those classification methods identify Th transcriptomes from RNA-seq data, based on those from microarray data. Table Stat3-KO and WT T cells in various culture conditions (GSE21670 [Stat3 dataset) and the Th dataset  were presumably related to either or both of Th17 differentiation and Stat3 signalling. In fact, they were enriched with Th17-related genes: (1) well-known Th17 genes such as ositive , and the Stat3 dataset was used as a test data. We addressed how efficiently those classification methods identify Th17 differentiated T cells in the Stat3 dataset, based on the resource dataset. Using various numbers of genes, CCA had high sensitivities and accuracies and outperformed, or at least was equivalent to, other classification methods Figure a. This rconstrained space) and finds a solution within the constrained space, while ignoring the non-interpretable part of the data (unconstrained space) [In typical immunological hypotheses, genes and cell phenotypes are correlated with immunological processes, and accordingly, genomic data are often ‘filtered’ for the immunological processes of interest  , 22. Thud space) , 5). UsiCCA had high sensitivities for identifying correlated samples in the cross-analysis of two datasets Figure  and 6. Tthe theory of the duality diagram, which is implemented in the CRAN package ade4[The proposed method assumes that gene expression is the measurement of transcripts at each gene (site), which represents a local ‘activity’ for a cell phenotype, and the total set of those transcripts (i.e. transcriptome) collectively shapes the cellular phenotype. Thus, CCA uses two matrices with genes in rows and samples in columns, and the genes must be the same between the two matrices Figure b. Althoukage ade4.Gata3 had the largest impact on Th2 differentiation, and also suggested that it had some effects on Th1 differentiation. In fact, Gata3 has been closely linked to Th2: Th2 differentiation is totally abolished in vitro and in vivo by the conditional deletion of Gata3 [Gata3-KO Th2 cells were closer to Th1 cells than WT Th2 cells in our analysis  . On the of Gata3 . Interesh2 cells . The dysnd Gata3 : Gata3-KStat3-KO T cells. This result is compatible with the findings by Durant et al., which showed that Stat3 was required for Th17 differentiation by in vivo and in vitro experiments using Stat3-KO T cells [Stat3-KO T cells was the iTreg differentiation programme  that facilitates the generation and refinement of hypothesis. In addition, CCA can have a high sensitivity for identifying the differentiated cells in a dataset that are similar to the ones in another dataset. These unique features make CCA competitive with other existing methods. The proposed method can be applied to a wide range of biological problems, providing effective solutions for multidimensional problems with multiple phenotypes and functions.rma of the Bioconductor package, affy. RNA-seq data were normalised for RPKM and log2 transformed. The function dudi.pca of the CRAN package ade4 was used for PCA. The function cca of the CRAN package, vegan, was used for the calculation of CCA.The datasets used in this study are summarised in Table X ∈ ℜk × p and is composed of the measurements of p cellular phenotypes at k genes . The j-th phenotype of X, xj = (x1jx2j … xkj)T is the experiment vector of k genes (i.e. transcripts occurred in ‘k gene sites’), where T indicates transposed vector. Similarly, Z ∈ ℜk × q is a matrix for explanatory variables (differentiation variables) and have k gene sites and q differentiation programmes (do not include replicates). Z is standardized to mean 0 and variance 1. First, X is standardised in the chi-square metric by row sums  and column sums . Thus, the standardised matrix is S = Dr-1/2 (1/nX– rcT) Dc-1/2, where n is the grand total of expression data, Dr and Dc are the diagonal matrices of r and c, respectively. CCA linearly regresses S onto differentiation variables Z, by the projection matrix Q = Dr1/2Z (ZTDrZ)-1ZTDr1/2, and the constrained (projected) space S* = Q S[S[S*, S* = U DαVT, where UTU = VTV = I, and Dα is the diagonal matrix of singular values in descending order (α1 ≥ α2 ≥ …). Principal or standard co-ordinates for gene expression scores  and sample scores are Dr-1/2U DαorDr-1/2U, and Dc-1/2 V Dα or Dc-1/2 V, respectively [Dr-1/2U, however, are in fact linear combinations of differentiation variables [S onto the sample scores, namely S V Dα , or S V[Z and the new co-ordinate axes, or more precisely, the LC scores, [The main transcriptomic data  S* = Q S. Thus, t S* = Q S. Mathema* = Q S[S. By calcectively   by Gene Set Enrichment Analysis (GSEA) [limma[heatmap.2 of the CRAN package, gplots, using the complete-linkage clustering using the Euclidean distance.Th-specific genes and Gata3-regulated genes referred to the lists provided by Wei et al.: a linea23 genes . Th17 sis Figure c referrere et al.. Gene sis (GSEA) , 14, 27:A) [limma) with FDvirtualArray[demi[e1071[e1071. In the analysis in Figure class[, and the number of neighbours was set to be 2. Linear and diagonal discriminant analyses were performed using the CRAN package, sda[randomForest[In order to apply machine learning methods, training and test data were cross-normalised using an empirical Bayes approach of the Bioconductor package tualArray or a ranrray[demi. Each Themi[e1071. For claure class, and thekage, sda. Random domForest, with thk = 2, using the CRAN package, cluster[θ[For the automatic classification of CCA sample scores, a differentiation variable was created by the difference of the mean vector of the cell subset of interest and that of all other subsets. CCA was performed using the differentiation variable, and sample scores were clustered by k-means partitioning, with , cluster. A 95% ce mean, θ.Additional file 1: Figure S1: Shows heatmap analysis and hierarchical clustering of the Stat3 dataset using (a) Th1, (b) Th2 (c) Th17, and (d) iTreg signatures. The cell samples that are known to differentiate into Th17 are shown by magenta. (PDF 171 KB)Additional file 2: Figure S2: Shows the result of PCA using the Stat3 dataset. Sample relationships (sample scores) of the first 3 axes are shown. Percentage indicates that of the variance accounted for by the eigenvalue of the axis. See Colour Key for the expression values. (PDF 162 KB)Additional file 3: Table S1: Shows CCA gene scores of the Stat3 dataset using the Th dataset for the Th differentiation programmes. (XLSX 69 KB)


High-throughput sequencing is gradually replacing microarrays as the preferred method for studying mRNA expression levels, providing nucleotide resolution and accurately measuring absolute expression levels of almost any transcript, known or novel. However, existing microarray data from clinical, pharmaceutical, and academic settings represent valuable and often underappreciated resources, and methods for assessing and improving the quality of these data are lacking.To quantitatively assess the quality of microarray probes, we directly compare RNA-Seq to Agilent microarrays by processing 231 unique samples from the Allen Human Brain Atlas using RNA-Seq. Both techniques provide highly consistent, highly reproducible gene expression measurements in adult human brain, with RNA-Seq slightly outperforming microarray results overall. We show that RNA-Seq can be used as ground truth to assess the reliability of most microarray probes, remove probes with off-target effects, and scale probe intensities to match the expression levels identified by RNA-Seq. These sequencing scaled microarray intensities (SSMIs) provide more reliable, quantitative estimates of absolute expression levels for many genes when compared with unscaled intensities. Finally, we validate this result in two human cell lines, showing that linear scaling factors can be applied across experiments using the same microarray platform.Microarrays provide consistent, reproducible gene expression measurements, which are improved using RNA-Seq as ground truth. We expect that our strategy could be used to improve probe quality for many data sets from major existing repositories.The online version of this article (doi:10.1186/1471-2164-15-154) contains supplementary material, which is available to authorized users. Fragmnscripts , and basnscripts , 7. Currnscripts  and Genenscripts . These dnscripts ; identifnscripts , 12; andnscripts –16. Morenscripts –20. WhilHere we present the largest comparison between microarray and RNA-Seq to date, using samples from the Allen Human Brain Atlas , 21, a phttp://human.brain-map.org) includes transcriptional profiling data from more than 3500 samples comprising approximately 200 brain regions in six clinically unremarkable adult human brains using custom Agilent DNA microarrays [The Allen Human Brain Atlas .Several methods for sequence alignment and gene expression quantification of RNA-Seq data have been developed (for review see ) includiProper normalization of microarray data can remove non-biological differences between samples due to batch effects and differences between arrays. These RNA-Seq data showed minimal batch effects ; however, correlations for both methods were quite high (median R = 0.984 vs. R = 0.994), suggesting a high degree of reproducibility when using either method. Next, we directly compared average TPM values for each gene with corresponding average intensities measured by microarray  based on reproducibility, which could, for example, help eliminate genes from the analysis in which all probes show potential off-target effects or non-specific binding. In this case we score each probe based on the correlation, defining all probes with significant positive correlation as passing Figure A. After th and 95th quantile of expression for each microarray probe (using intensity) and for the corresponding RNA-Seq gene (using TPM), and then linearly scaled the microarray intensities so that these values align with TPMs , the relationship between probe intensities and gene expression levels identified through other experimental strategies is nearly linear . We therTo test whether SSMIs provide more biologically reproducible results than corresponding intensity scores, we repeated all of our quantitative assessments  lines  utilize the same Agilent array to facilitate between age comparisons, and have provided insight into the structure and function of the human brain. Furthermore, many research and clinical laboratories already have standard operating procedures in place for analyzing microarray data—including the required machinery, storage space, analysis tools, and expertise—which could be augmented in a relatively straightforward manner using our method, and it will take time to transition to sequencing-based strategies. Finally, until quite recently RNA-Seq techniques required more total RNA (100 ng-1 μg) than microarray [While RNA-Seq will likely replace microarrays in research and clinical settings in the near future due to the improved dynamic range and potential for finding novel transcripts and sequence level variations, microarray data sets are still highly valuable, and application of this method improves their interpretability. Currently, microarray data from thousands of experiments are available in public databases such as GEO  and Arracroarray , 36. In croarray , 38 or ecroarray  hold greOne limitation of this analysis is that, in order to accurately assess probe quality and define scaling parameters, the variability across samples must be accurately measured. For example, we found that probes targeting genes with very low expression in brain were much more likely to be failed, compared with high expressers, and that such genes also showed less consistent expression patterns between brains. It is likely that with a more diverse tissue panel some of these low-expressing probes would be assessed as higher quality. Another possible limitation is that our scaling parameters derived from brain may not be applicable to other tissues for genes showing a high degree of differential isoform expression. Again, by using a more diverse tissue panel to calculate scaling parameters, we would expect that probes for such genes would not show consistent expression between methods, and would therefore be failed at the quality control step.http://commonfund.nih.gov/GTEx/index). Our results suggest that approximately 16 carefully chosen samples would be sufficient, although increasing the number of samples would moderately improve the power to detect passing probes  Project . In short, tissue for the Allen Human Brain Atlas was provided by the NICHD Brain and Tissue Bank  and the University of California Irvine Department of Psychiatry and Human Behavior Brain Donor Program , under approvals by Institutional Review Boards of the Maryland Department of Health and Hygiene and University of Maryland Baltimore, or the University of California Irvine, respectively, and with consent from next-of-kin. Specimens for microarray and RNA-Seq profiling were 24-year-old (Brain 1) and 39-year-old (Brain 2) African American males. Total RNA from 120 macrodissected samples initially processed for microarray from each brain (as described in [http://www.brain-map.org) in the form of gene counts, as well as TPM values.Methods for post-mortem tissue acquisition and sample processing have previously been described ( and httpribed in ) were alhttp://genome.ucsc.edu; hg19, Feb. 2009). Summary expression levels for each gene were calculated in terms of both counts and TPM using this pipeline. Mapped read files were also converted to BAM file format for visualization using GenomeBrowse .Sequences were aligned to the genome using RNA-Seq by Expectation-Maximization (RSEM)  see Add. Transcrhttp://help.brain-map.org/display/humanbrain/Documentation). In short, data is preprocessed for systematic biases, and quantile normalized to the 75th percentile in each batch. Across batches within each brain, data is normalized by aligning two sets of control samples included in all batches and capturing method related bias is adjusted by a modified quantile normalization method. We note that approximately 2,200 probes failed quality assessments during generation of the Allen Human Brain Atlas and were also excluded from this analysis. Across multiple brains non-biological difference is adjusted again by aligning two sets of control samples. RNA-Seq data was TbT normalized [Microarray data normalization was performed as described on the Allen Human Brain Atlas data portal  was extensively used for all analyses and visualizations  were calculated in several contexts as measures of consistency or reproducibility throughout  or as part of Additional file All data presented in this manuscript are available either at the Allen Institute data portal  to reproduce nearly all figures and statistics presented in this manuscript. (ZIP 11 MB)Additional file 1: Clustering of RNA-Seq samples after TbT normalization shows minimal batch effects. A dendrogram that shows samples hierarchically clustered based on the RNA-Seq data. Also shown are bar plots with biological and technical variables. Samples cluster based on brain region and brain of origin, but not batch or other technical variables. (PDF 3 MB)Additional file 2: Supplementary methods and supplementary figure legends. This file contains the supplementary methods for the manuscript followed by supplementary figure legends corresponding to Additional files Additional file 3: Normalization improves the quality of RNA-Seq and microarray data. Several plots showing that RNA-Seq data becomes progressively more consistent and reproducible after TbT normalization, which scales for the total reads, and ComBat normalization, which removes the systematic bias between brains. Microarray data likewise improved. (PDF 18 MB)Additional file 4: Many probes with specifically high microarray intensity do not accurately measure gene expression. Probes with specifically high expression in microarray show poor between-method agreement, suggesting they do not appropriately measure expression of their assigned gene. These probes are also more likely than chance to be part of gene families. (PDF 3 MB)Additional file 5: Gene expression reproducibility is dependent on transcript length. There is a strong linear relationship between gene length—defined based both on the average transcript length for all RefSeq isoforms of the gene as well as for the number of base pairs spanned in the genome—and reproducibility for genes. This is in part because large genes tend to show higher variability in the brain compared with smaller genes. (PDF 6 MB)Additional file 6: Microarray quality control and scaling can be accurately done with 16 samples. The percent of passing probes rapidly improves when using a small number of matched samples, and starts leveling out at around 16 samples. Excellent between-method correlation and improvements in scaling are also seen with 16 samples or fewer. (PDF 3 MB)Additional file 7: Parameters and quality metrics for microarray probe scaling. This table contains the parameters that can be applied to any Agilent Human GE 44K microarray to scale probes so that they more accurately reflect absolute gene expression levels. It also includes quality metrics which can be used as filters, for example, to omit certain probes from consideration. (XLSX 9 MB)Additional file 8:


A major application of RNA-Seq is to perform differential gene expression analysis. Many tools exist to analyze differentially expressed genes in the presence of biological replicates. Frequently, however, RNA-Seq experiments have no or very few biological replicates and development of methods for detecting differentially expressed genes in these scenarios is still an active research area.In this paper we introduce a novel method, called IsoDE, for differential gene expression analysis based on bootstrapping. We compared IsoDE against four existing methods  on RNA-Seq datasets generated using three different sequencing technologies, both with and without replicates. Experiments on MAQC RNA-Seq datasets without replicates show that IsoDE has consistently high accuracy as defined by the qPCR ground truth, frequently higher than that of the compared methods, particularly for low coverage data and at lower fold change thresholds. In experiments on RNA-Seq datasets with up to 7 replicates, IsoDE has also achieved high accuracy. Furthermore, unlike GFOLD and edgeR, IsoDE accuracy varies smoothly with the number of replicates, and is relatively uniform across the entire range of gene expression levels.The proposed non-parametric method based on bootstrapping has practical running time, and achieves robust performance over a broad range of technologies, number of replicates, sequencing depths, and minimum fold change thresholds. RNA-Seq has become the new standard for the analysis of differential gene expression -3 due toUnfortunately, due to the still high cost of sequencing, many RNA-Seq studies have no or very few replicates . MethodsIn this work, we present a novel method for differential gene expression analysis for RNA-Seq data, called IsoDE. Our method uses the traditional bootstrapping approach  to resamRecent studies such as Rapaport et al.  have reiA of RNA-Seq read alignments for each condition. Bootstrapping can be used in conjunction with any method for estimating individual gene expression levels from aligned RNA-Seq reads, estimation typically expressed in fragment per kilobase of gene length per million reads (FPKM). In IsoDE, we use the IsoEM algorithm [As most differential expression analysis packages, IsoDE starts with a set lgorithm , an expelgorithm .M bootstrap samples by randomly resampling with replacement from the reads represented in A. When a read is selected during resampling, all its alignments from A are included in the bootstrap sample. The number of resampled reads in each bootstrap sample equals the total number of reads in the original sample. However, the total number of alignments may differ between bootstrap samples, depending on the number of alignments of selected reads and the number of times each read is selected. The IsoEM algorithm is then run on each bootstrap sample, resulting in M FPKM estimates for each gene. The bootstrap sample generation algorithm is summarized below:The first step of IsoDE is to generate A by read ID1. Sort the alignment file N of reads and generate a list A2. Compute the number i = 1,..., M do:3. For N read IDs from i Aall their alignments with one linear pass over A (a) Randomly select with replacement i Ato get the th iFPKM estimate for each gene(b) Run IsoEM on To test for differential expression, IsoDE takes as input two folders which contain FPKM estimates from bootstrap samples generated for the two conditions to be compared. In case of replicates, a list of bootstrap folders can be provided for each condition  - IsoDE will automatically merge the folders for the replicates to get a combined folder per condition, then perform the analysis as in the case without replicates.M bootstrap samples is generated for each of the compared conditions. We experimented with two approaches for pairing the FPKMs estimated from the two sets of bootstrap samples. In the "matching" approach, a random one-to-one mapping is created between the M estimates of first condition and the M estimates of the second condition. This results in M pairs of FPKM estimates. In the "all" approach, M2 pairs of FPKM estimates are generated by pairing each FPKM estimate for first condition with each FPKM estimate for second condition. When pairing FPKM estimate i afor the first condition with FPKM estimate j bfor the second condition, we use ia/j bas an estimate for the fold change in the gene expression level between the two conditions. The "matching" approach thus results in N = M fold change estimates, while the "all" approach results in N = M2 fold change estimates.In the following we assume that a total of f and the minimum bootstrap support b. For a given threshold f , we calculate the percentage of fold change estimates that are equal to or higher than f when testing for overexpression, respectively equal to or lower than 1/f when testing for underexpression. If this percentage is higher than the minimum bootstrap support b specified by the user then the gene is classified as differentially expressed (DE), otherwise the gene is classified as non-differentially expressed (non-DE). The actual bootstrap support for fold change threshold f , as well as the minimum fold change with bootstrap support of at least b are also included in the IsoDE output to allow the user to easily increase the stringency of the DE test.The IsoDE test for differential expression requires two user specified parameters, namely the minimum fold change b allows users to achieve a smooth tradeoff between sensitivity and specificity for a fixed fold change f  ≤ α} and X is a binomial random variable denoting the number of successes in N independent Bernoulli trials with success probability of 0.5. For convenience, a calculator for computing the bootstrap support needed to achieve a desired significance level given the (possibly different) numbers of bootstrap samples for each condition has been made available online (see Availability).As discussed in the results section, varying the bootstrap support threshold M of bootstrap samples is another parameter that the users of IsoDE must specify. As discussed in the results section, computing the bootstrap support for all genes takes negligible time, and the overall running time of IsoDE is dominated by the time to complete the 2M IsoEM runs on bootstrap samples. Hence, the overall runtimes grows linearly with M . Experimental results suggest that the "all" pairing approach produces highly accurate results with relatively small values of M , and thus results in practical runtimes, independent of the number of replicates. We also note that for studies involving pairwise DE analysis of more than two conditions, IsoDE only requires M independently generated bootstrap samples per condition. Since the time for computing pairwise bootstrap support values is negligible, the overall running time will grow linearly with the number of conditions.The number The four methods that were compared to IsoDE are briefly described below.Fisher's exact test is a statistical significance test for categorical data which measures the association between two variables. The data is organized in a 2x2 contingency table according to the two variables of interest. We use Fisher's exact test to measure the statistical significance of change in gene expressions between two conditions A and B by setting the two values in the first row of the table to the estimated number of reads mapped per kilobase of gene length  in conditions A and B, respectively. The values in the second row of the contingency table depend on the normalization method used. We compared three normalization methods. The first one is total read normalization, where the total number of mapped reads in conditions A and B are used in the second row. The second is normalization by a housekeeping gene. In this case, the estimated number of reads mapped per kilobase of housekeeping gene length in each condition is used. We also test normalization by ERCCs RNA spike-in controls . FPKMs oThe calculated p-value, which measures the significance of deviation from the null hypothesis that the gene is not differentially expressed, is computed exactly by using the hypergeometric probability of observed or more extreme differences while keeping the marginal sums in the contingency table unchanged. We adjust the resulting p-values for the set of genes being tested using the Benjamini and Hochberg method  with 5% GFOLD  is a genCuffdiff  uses a bedgeR  is a staMAQC Illumina reads were mapped onto hg19 Ensembl 63 transcript library; all other datasets were mapped onto hg19 Ensembl 64 transcript library. Illumina datasets (MAQC and MCF-7) were mapped using Bowtie v0.12.8 . ION TorOn MAQC dataset the ground truth was defined based on the available qPCR data from . Each TaFor experiments with replicates we used the RNA-Seq data generated from E2-treated and control MCF-7 cells in . In thisFor each evaluated method, genes were classified according to the differential expression confusion matrix detailed in Table We conducted experiments on publicly available RNA-Seq datasets generated from two commercially available reference RNA samples and a breast cancer cell line.To compare the accuracy of different methods, we used RNA-Seq data RNA samples that were well-characterized by quantitative real time PCR (qRT-PCR) as part of the MicroArray Quality Control Consortium (MAQC) ; namely We used RNA-Seq data generated for HBRR and UHRR using three different technologies: Illumina, ION-Torrent, and 454. Details about the datasets and their SRA accession numbers (or run IDs for ION Torrent datasets) are available in Table S1 in Additional File The MCF-7 RNA-Seq data was generated (from the MCF-7 ATCC human breast cancer cell line) by Liu et al.  using Ilf of 1, 1.5, respectively 2, and bootstrap support threshold b between 40% and 95%. The results of IsoDE-Match with M = 200 bootstrap replicates per condition are shown in Figure f , varying b results in a smooth tradeoff between sensitivity and PPV, while the F-score changes very little. For the remaining experiments we used a bootstrap support level b computed using a significance level of 0.05 under the binomial null model detailed in the Methods section. Note that the value of b selected in this way depends on the number N of fold change estimates, which in turn depends on both M and the pairing strategy .We evaluated both the "matching" and "all" pairing strategies of IsoDE (referred to as IsoDE-Match and IsoDE-All) for fold change threshold M varying between 10 and 200 (results not shown). For the considered measures, IsoDE-All achieved an accuracy very close to that of IsoDE-Match when run with a comparable value of N . For example, as shown in Tables M = 20 bootstrap samples (N = 400) had similar accuracy with the largest number of bootstrap samples we could use with IsoDE-Match (M = N = 200).To determine the best pairing strategy, we ran IsoDE-Match and IsoDE-All with number of bootstrap samples N IsoDE-Match requires 2N bootstrap samples while IsoDE-All requires only M = 20, respectively M = 200, bootstrap samples for both conditions of several MAQC datasets. All timing experiments were conducted on a Dell PowerEdge R815 server with quad 2.5 GHz 16-core AMD Opteron 6380 processors and 256 Gb RAM running under Ubuntu 12.04 LTS. IsoEM is run on bootstrap samples sequentially, but for each run its multi-threaded code takes advantage of all available cores . As expected, the running time scales linearly with the number of bootstrap samples per condition, and thus generating M = 20 bootstrap samples per condition is nearly 10 times faster than generating M = 200 of them. Overall, IsoDE-Match with M = 20 has reasonable running time, varying between 1 hour for the smallest 454 dataset to 3.5 hours for the Illumina dataset.Since for a fixed We compared IsoDE against GFOLD, Cuffdiff, edgeR, and different normalization methods for Fisher's exact test; namely total normalization, housekeeping gene (POLR2A) normalization, and normalization using External RNA Controls Consortium (ERCC) RNA spike-in controls . Cuffdiff of 1, 1.5, and 2, respectively. Table f . Table Table IsoDE has very robust performance, comparable or better than that of the other methods for differential gene expression. Indeed, IsoDE outperforms them in a large number of cases, across datasets and fold change thresholds. Very importantly, unlike GFOLD and Fisher's exact test, IsoDE maintains high accuracy (sensitivity and PPV around 80%) on datasets with small numbers of mapped reads such as the two 454 datasets. This observation is confirmed on results obtained for pairs of individual ION-Torrent runs, presented in Tables S3 and S4 in Additional File M = 20 bootstrap samples from all 7 replicates as its ground truth is generated using a much larger number of bootstrap samples (M = 70). Figure We also studied the effect of the number of biological replicates on prediction accuracy using the MCF-7 dataset. We performed DE predictions using an increasing number of replicates. IsoDE was run with a total of 20 bootstrap samples per condition, distributed equally  among the replicates, as detailed in Table S5. GFOLD and edgeR were evaluated for 1 through 6 replicates using as ground truth the results obtained by running each method on all 7 replicates (see the Methods section). For IsoDE, we also include the results using Since for this experiment the ground truth was defined independently for each method, it is not meaningful to directly compare accuracy metrics of different methods. Instead, we focus on the rate of change in the accuracy of each method as additional replicates are added. Generally, all methods perform better when increasing the number of replicates. However, the accuracy of IsoDE varies smoothly, and is much less sensitive to small changes in the number of replicates. Surprisingly, this is not the case for GFOLD and edgeR sensitivity, which drops considerably when transitioning from 1 to 2 replicates, most likely due to the different statistical models employed with and without replicates. Although we varied the number of replicates without controlling the total number of reads as Liu et al. , our resWe also studied the effect of gene abundance on the IsoDE, GFOLD, and edgeR prediction accuracy. We selected the subset of genes that are expressed in at least one of the two RNA samples. We sorted these genes by the average of the gene's expression. We used the FPKM values predicted by IsoEM, the FPKM values predicted by GFOLD, and the number of uniquely mapped reads, for IsoDE, GFOLD, and edgeR, respectively. The genes were then divided into quintiles, for each method independently, where quintile 1 had the genes with the lowest expression levels, and quintile 5 had the genes with the highest expression levels. Sensitivity, PPV, and F-score where calculated for each quintile separately.Figure A practical bootstrapping based method, IsoDE, was developed for analysis of differentially expressed genes in RNA-Seq datasets. Unlike other existing methods, IsoDE is non-parametric, i.e., does not assume an underlying statistical distribution of the data. Experimental results on publicly available datasets both with and without replicates show that IsoDE has robust performance over a wide range of technologies, sequencing depths, and minimum fold changes. IsoDE performs particularly well on low coverage RNA-Seq datasets, at low fold change thresholds, and when no or very few replicates are available.http://dna.engr.uconn.edu/software/IsoDE/. A web-based calculator for computing the bootstrap support based on the desired number of bootstrap samples and significance level is available at http://dna.engr.uconn.edu/~software/cgi-bin/calc/calc.cgi.IsoDE has been implemented in Java and can be run on any platform with a Java virtual machine. The source code and installation instructions are available at The authors declare that they have no competing interests.SS implemented and tested Fisher's exact test, prepared the testing data used to generate the experimental results, ran Cuffdiff, GFOLD, and edgeR, and performed comparisons between these methods and IsoDE. SS also contributed to designing the algorithms and wrote part of the manuscript. YTT developed, implemented and tested the algorithms for IsoDE and wrote part of the manuscript. AZ contributed to designing the algorithms and the experiments, writing the manuscript and supervised the project. IM contributed to designing the algorithms and the experiments, writing the manuscript and supervised the project. All authors read and approved the final manuscript.Supplementary figures and tables are supplied in PDF format.Click here for file


Background: Telomere length is a putative biomarker of ageing, morbidity and mortality. Its application is hampered by lack of widely applicable reference ranges and uncertainty regarding the present limits of measurement reproducibility within and between laboratories.Methods: We instigated an international collaborative study of telomere length assessment: 10 different laboratories, employing 3 different techniques  performed two rounds of fully blinded measurements on 10 human DNA samples per round to enable unbiased assessment of intra- and inter-batch variation between laboratories and techniques.Results: Absolute results from different laboratories differed widely and could thus not be compared directly, but rankings of relative telomere lengths were highly correlated (correlation coefficients of 0.63–0.99). Intra-technique correlations were similar for Southern blotting and qPCR and were stronger than inter-technique ones. However, inter-laboratory coefficients of variation (CVs) averaged about 10% for Southern blotting and STELA and more than 20% for qPCR. This difference was compensated for by a higher dynamic range for the qPCR method as shown by equal variance after z-scoring. Technical variation per laboratory, measured as median of intra- and inter-batch CVs, ranged from 1.4% to 9.5%, with differences between laboratories only marginally significant (P = 0.06). Gel-based and PCR-based techniques were not different in accuracy.Conclusions: Intra- and inter-laboratory technical variation severely limits the usefulness of data pooling and excludes sharing of reference ranges between laboratories. We propose to establish a common set of physical telomere length standards to improve comparability of telomere length estimates between laboratories. Rankings are very similar if different laboratories measure telomere lengths in the same samples.However, quantitative results from different laboratories are hardly comparable.Southern Blotting and quantitative PCR are similar in their reproducibility.Laboratories measuring telomere length should use a common set of physical standards.,Telomere length (TL) in peripheral blood has been associated in multiple studies with progression of human ageing, mortality and risk of age-related diseases.,,,in situ hybridization with fluorescent telomere PNA probes (qFISH);,,,A wide range of methods have been developed to measure TL such as: (i) Terminal Restriction Fragment (TRF) analysis by hybridization of digested and electrophoresed DNA with telomere sequence probes (Southern blotting);To comprehensively and independently assess the reproducibility of the method and the degree of consistency between different laboratories and techniques, an international collaborative study was conducted in which a number of coded samples of DNA were shipped to 10 expert laboratories around the world, that performed two rounds of fully blinded telomere length assessments according to their established in-house methodology. DNA samples rather than cells or tissues were used in order to minimize preparative variation, so only laboratories performing Southern blot, STELA or qPCR were included. Results of this study indicate important methodological limitations when attempting to compare data between different laboratories, even on a relative scale.Supplementary Table 1, available as Supplementary data at IJE online). Elsewhere in this report, participating laboratories are distinguished by code numbers which are independent of the order in which they are listed in Supplementary Table 1. Four further laboratories were invited to participate. Two of these teams elected instead to conduct their own joint study of telomere length measurement.Laboratories were invited to participate in the study on the basis of an active publication record in the field. The 10 participating laboratories are listed in Supplementary Table S1A (for qPCR methods) and S1B (for gel-based methods) (available as Supplementary data at IJE online). As STELA combines features of both, it is included in both supplementary tables.Two laboratories (labs 1 and 2) applied their established Southern blotting method (South). One laboratory (lab 3) used the STELA technique, and seven laboratories (labs 4–10) used PCR-based methods (qPCR). Methodological details are given in Samples were selected to provide a good coverage of the various kinds of human DNA material that might be encountered in routine work of this nature and thus included tumour and somatic cell DNA as well as DNA isolated from human tissue and human leukocytes .Table 1260/280 values were from 1.88 to 2.05, and OD260/230 ranged from 1.92 to 2.81. Samples were aliquoted  and sent to an independent distributor team  which individually re-coded and shipped to the participating laboratories and kept the code unbroken until all results had been returned. In the first round, 10 samples  were sent. The second round was started only after all data from the first round had been received, to enable the comparison of measurements performed in independent batches. This round included five repeat samples from the first round , of which samples C, G and H were duplicated, and two new samples (K and L) of actual donor DNA to distinguish from cultured cell-lines DNA. Only the Newcastle laboratory was aware of this information, but was blinded as every other participant to the identity of the samples received from the independent distributor. Once all results were returned the codes were broken and statistical analysis was performed.The study was performed in two fully separated rounds to enable assessment of both intra- and inter-batch variation. All DNA samples were generated at the Newcastle, UK, laboratory by QIAamp DNA extraction  and their quality and concentration were assessed by both UV spectroscopy and agarose gel electrophoresis. ODSince variations between laboratories and methods are expected to give rise to systematic differences in raw estimates of telomere length, the primary focus of this study has been to examine the reliability and consistency of assessment of relative telomere lengths, rather than absolute length. For this purpose, telomere length ratios (TLRs) were calculated using a chosen sample as reference. Unless otherwise indicated, TLR values in the remainder of this paper refer to the ratio of the estimated telomere length for a particular sample, divided by the estimated telomere length for sample G. In round 2, where a blind-coded duplicate of sample G was included, the value of just one of the duplicates was used as the reference sample, since this allowed assessment of the precision of performing repeated assessments of samples which, unknown to the laboratories at the time of assessment, were identical. To additionally compensate for differences in the dynamic range of measurements, z-scores were calculated from raw data. In addition to comparing method- and laboratory-specific coefficients of variation (CVs), a General Linear Model (GLM) analysis with normalized telomere length as dependent variable and method and laboratory as factors was performed; we employed this method to determine if a statistically significant difference in telomere length was evident between laboratories (labs), methods and also to test for a lab vs method interaction. All statistical analyses were performed using IBM SPSS Statistics v19 and STATA v13.Supplementary Table S1(available as Supplementary data at IJE online) and no data were returned. Specifically, lab 1 did not measure sample L (second round) because of low quality restriction digest, lab 3 obtained insufficient DNA molecules for amplification from samples E (first round) and C (2 second round) and sample H failed quality control [as defined in Supplementary Table S1 (available as Supplementary data at IJE online)] in lab 4 (first round) and in lab 6 (second round). Lab 10 was only invited to participate after round 1 was already completed, but performed two separate qPCR assays (one-tube and two-tube). Raw data for telomere length (laboratories 1–3) or T/S ratios (laboratories 4–10) are given in Supplementary Table S2 (available as Supplementary data at IJE online). As expected, the values differed widely. To enable comparisons, the returned values were standardized to TLRs. These data are given in Supplementary Figure S1, available as Supplementary data at IJE online) were strong. Corresponding rank correlation coefficients  between TLRs measured in different laboratories ranged between 0.63 and 0.99. Correlations between laboratories within each technique separately were stronger (with no differences between Southern blot and qPCR) than those between Southern blot and qPCR results .From 190 samples sent out for analysis, results were returned for 185. For five samples  results did not meet the internal quality standards of the laboratory as outlined in Supplementary Table S3, available as Supplementary data at IJE online), a comparison of the inter-lab CVs showed that there is significantly  less inter-laboratory variability between the Southern blotting and STELA techniques than within the qPCR laboratory results (P = 0.001) greater than 1. Importantly, Supplementary Table S4, available as Supplementary data at IJE online). However, these variances were still large with medians amounting to between 23% (qPCR) and 30%  of the standard deviation (SD) of the examined population.To measure the variation of the TLR estimates between laboratories, we calculated CVs for every sample as measured by all laboratories and separately as measured by qPCR or Southern/STELA . This va results . This is results . The sam results . This isP = 0.299). A summary of intra-batch CVs per technique is shown in P = 0.161, Kruskal-Wallis ANOVA on ranks). Even if CVs from South and STELA were combined (median CV = 2.40), the difference to the qPCR results remained non-significant .Variation within laboratories was tested separately for both intra- and inter-batch variation. To test intra-batch variation, three samples in round 2 were duplicated. These samples were measured fully blinded on the same gel (Southern and STELA) or the same plate (qPCR). CVs ranging between 0.000 and 31.299 for individual samples and laboratories are given in P = 0.195, Kruskal-Wallis ANOVA on ranks). Median inter-batch CVs . Interestingly, for the qPCR technique, intra- and inter-assay variation were not different, suggesting intra-assay variation as the major contributor to overall variance, whereas plate-to-plate variation seems minor or well corrected for.Any larger study will rely on comparisons of data generated in separate batches. Therefore, inter-batch variation was tested in each laboratory (excluding lab 10) using five fully blinded duplicated samples between rounds 1 and 2. Results are given in atch CVs b were 3.P = 0.060, Kruskal-Wallis ANOVA on ranks). Similarly, when variation was estimated based on z-scored data, there was no significant difference between techniques or individual laboratories (data not shown).To compare accuracy between all participating laboratories, we combined both intra- and inter-batch estimates c. AlthouP = 0.096, confirming borderline significance for standard deviations between labs and techniques. However, partial eta-squared coefficients were low , indicating that neither technique nor laboratory had strong influence on result variation.To further compare the impacts of technique and laboratory on result variance, a generalized linear model was constructed with technique and laboratory as factors. Testing the null hypothesis of equal variance for normalized telomere length in all groups resulted in an F = 1.650, corresponding to This is the first study to undertake a comparison of telomere length measurements across a wide group of laboratories with expertise in three different techniques. For the present blind coded comparison study we used DNA samples that originated from a single laboratory and therefore differences between laboratories or between methods cannot be attributed to pre-analytical conditions such as cell culture, blood sample anticoagulant or collection procedure, alternative DNA isolation or storage methods, etc. Recently it had been shown that DNA extraction methods can have a significant impact on both mean value and dynamic range of telomere length estimates by qPCR,We did not attempt a comparison between absolute data as returned from the participating laboratories because these varied even more than the TLRs, both between and within techniques., ,,,Our main result is that rank correlations between laboratories are high but there is a large variation of TLR estimates between different laboratories. With a median CV of 24% between laboratories, this variation is much larger than differences between control and case groups in typical telomere biomarker studies, which are generally in the order of 3–10%. The large variation between laboratories is partly driven by systematic differences between qPCR- and gel-based techniques, especially in measuring short telomeres. Systematic differences between Southern and qPCR results have been found before.,,The possibility remains that these large variations and systematic differences are at the root of the inconsistencies found in the literature.,It had been suggested that inherent methodological variation might be higher for the qPCR method as compared with Southern blotting.,,,Laboratory-specific intra- and inter-batch CVs have been reported in the literature over a range from 1.25% to 12% for Southern blotting and 2.27% to 28% for qPCR.Supplementary Table S1, available as Supplementary data at IJE online).The amount of methodological differences between laboratories was large. Six different qPCR labs used four different reference genes  and differed in their application of a duplex or monoplex approach, in use of primers, master mix compositions and thermal cycling profiles, in the brands of qPCR systems used  and in the normalization techniques applied to correct for well-to-well and/or plate-to-plate variations. Similarly, Southern protocols differed in multiple parameters between laboratories, including DNA restriction protocols, electrophoresis conditions, the molecular weight marker and the probe labelling as well as the use (or not) of internal batch-to-batch controls . Whereas results using different reference genes in the same lab correlated highly (rank correlation coefficients >0.85), correlations to the blinded results from different labs using the same reference gene were not better than those using different reference genes. In other words, use of different reference genes did not explain the variation between qPCR labs.In essence, every single laboratory had developed its own combination of interdependent methodological details in an approach to optimize outcomes. This means that an ‘observational’ study like ours was not designed to assess the impact of these methodological differences on result variability, even if it would include larger numbers of samples and/or laboratories. However, the results from our study might be used to suggest a follow-up ‘interventional’ study, in which laboratories change certain methodological details to see whether this might improve variability of results (see conclusions below). One obvious Our results demonstrate large inter-laboratory variation even for relative telomere lengths following internal normalization. This means that reference ranges for telomere lengths that may be applied by all laboratories cannot be given in the present state of the art. In other words, ‘the’ telomere length of an individual  does not exist as a measurable quantity, and even a technically perfect telomere length measurement could only be useful as a risk indicator if reference values were measured by the same laboratory using the same protocols. Z-scoring of data appears at present the best possibility for combining results from different laboratories. However, this may result in large errors, which can easily reach median values around 500 bp telomere length in typical human populations.Our data suggest that it would be both possible and useful to develop optimized protocols that will reduce intra- and inter-lab variation. As a first step, we propose that a set of telomere length standards should be generated to share among interested parties . If these were analysed with each major study, it would for the first time enable standardization of results and their comparison between laboratories. However, natural telomeres (i.e. in telomerizzed cells in culture) are not constant in length between sub-clones  or with Regarding further steps towards inter-lab methodological standardization, our results do not immediately suggest measures that would reduce result variation with high probability. For instance, comparing variation between qPCR labs, we found no preference for a single reference gene, neither appeared a multiplex approach to be more reproducible than a monoplex one. Similarly, it was not clear which (combination) of methodological differences between the two Southern labs could be responsible for the tendency towards a lower CV in lab 2. Moreover, we recognize that groups use different pieces of equipment, for which different reagents and protocols are optimal. However, the groups involved in the present study have started discussions about ways to test protocol variations, and we invite all interested laboratories to join and to contribute to further studies.Supplementary data are available at IJE online.UK Medical Research Council [grant numbers G0601333 and G0500997 to T.vZ. and MC_UU_12019/1 to R.Co. and A.W.]; the New Dynamics of Ageing Initiative [grant number RES-353-25-0001 to R.Co.]; the Swedish Cancer Society [grant number 12 06249 to G.R. and U.S.]; the Swedish Research Council [grant number 90341301 to G.R. and U.S.]; the European Community's Seventh Framework ProgramFP7/2007-2011 [grant number 200950 to G.R. and U.S.]; the British Heart foundation [to V.C. and N.J.S.]; the NIHR Newcastle Biomedical Research Centre in Ageing and Chronic Disease [to C.M.M.R.]; the BMBF GerontoSys Stromal Aging [grant number 0315576A to P.B.]; UVA Konsortium [grant number 03NUK003A to P.B.]; the University of Utah Research Account [to R.Ca.]; the Association for International Cancer Research [grant number 10-0021 to D.B.]; and the Cunningham Trust [to P.S.].The work was funded by the


These sources of noise must be modeled and removed to accurately measure biological variability and to obtain correct statistical inference when performing high-throughput genomic analysis. We introduced surrogate variable analysis (sva) for estimating these artifacts by (i) identifying the part of the genomic data only affected by artifacts and (ii) estimating the artifacts with principal components or singular vectors of the subset of the data matrix. The resulting estimates of artifacts can be used in subsequent analyses as adjustment factors to correct analyses. Here I describe a version of the sva approach specifically created for count data or FPKMs from sequencing experiments based on appropriate data transformation. I also describe the addition of supervised sva (ssva) for using control probes to identify the part of the genomic data only affected by artifacts. I present a comparison between these versions of sva and other methods for batch effect estimation on simulated data, real count-based data and FPKM-based data. These updates are available through the sva Bioconductor package and I have made fully reproducible analysis using these methods available from:  Batch effects and other technological artifacts introduce spurious correlation, create bias and add variability to the results of genomic experiments –3. The bi on sample j (gij) to the phenotype for that sample (yj), the known batch variable for that sample (aj) and an unknown artifact on that sample (uj):In 2007 we introduced surrogate variable analysis (sva) as a conceptual approach to statistical modeling of genomic data when artifacts are unknown or unmeasured  Figure  and subsuj) from the data directly, since all the coefficients  are unknown. But we noticed  and subtracted the mean of each gene to remove the baseline effect, the matrix form of the model is:mu is the number of genes where bi1 = 0 and ci = 0 and n is the sample size. We then pointed out that you could apply a matrix decomposition like the singular value decomposition or principal components analysis to this subset of the data to estimate surrogates for the unknown artifacts u. Later we also showed that if the number of genes that are not affected by phenotype or known batch (bi1 = 0 and c = 0) but are affected by unknown artifacts (di ? 0) is large enough, you can obtain consistent estimates of a linear transformation of ui \dIn Step 5, the covariates may be included in a standard linear regression modeling analysis on an appropriately transformed scale or the covariates can be directly used in software that models counts with generalized linear models (GLMs) including edgeR  and DESeThe general form of the sva algorithm relies on the idea that there is a subset of genes, probes or transcripts that are affected by unknown batch effects or other artifacts but are not affected by the biological relationship of interest. This is the main way that a sva approach is distinguished from a standard principal component regression. Standard principal component-based regression methods, such as EIGENSTRAT , are sufλi = 1 for all negative controls and λi = 0 for all other genes in Step 2 of the sva algorithm. The assumption is that control probes will capture all of the variation due to unknown artifacts and none of the variation due to the phenotype. Control probes may miss biological artifacts. For example, we showed that trans-eQTL that are associated with multiple gene expression levels may act like an artifact when measuring the association between gene expression and phenotype (Supervised sva (ssva) sets henotype . These ahenotype , making henotype .f in (log(gij + c) for a small positive constant. In the analyses that follow I will set c = 1. The choice of the moderating constant is an important one and is beyond the scope of this manuscript. Intuitively a choice of c = 0 corresponds to no moderation and as c increases you decrease the variation in the data. After performing Steps 1–5 of the sva estimation algorithm, the estimated covariates are included in downstream models as adjustment variables. For the analyses that follow, I will use the limma package  to the gene expression count data in Step 1 and setting λi = 1 for all negative controls and λi = 0 for all other genes in Step 2 of the sva algorithm.Supervised svaseq proceeds by applying the transformation zebrafishRNASeq Bioconductor package. I loaded the data and filtered as described in the removing unwanted variation in sequencing data (RUVSeq) package. Then I estimated batch effects using supervised and unsupervised sva for sequencing, principal components analysis, RUV with control probes, RUV with empirical controls and residual RUV. I compared the model estimates and I compared differential expression analysis results when each of the different batch effect estimates was included in the model in place of the study variable.I use data from Zebrafish sensory neurons with three control samples and three gallein treated samples as the comparison groups . These dReCount is a database of pre-processed RNA-sequencing data, processed to be comparable across samples . In thishttps://github.com/alyssafrazee/ballgown_codeI then subset the data to only the non-duplicated samples . Fo. Fo33).  package . Then I http://www.bioconductor.org/packages/devel/bioc/html/sva.html). All data and code used to perform this analysis are available as R markdown files .I estimated simulation parameters from the Zebrafish data as described in the methods. I then performed several checks to confirm that (i) the data generated by the simulated model recapitulated the qualitative behavior of the data used to estimate the model parameters Figure , (ii) thP-values  .In the Zebrafish study there was a single simulated batch effect. I simulated two scenarios, in the first scenario the batch effect and the group effect had low correlation. As expected, all methods that aim to estimate batch effects while taking into account multiple sources of signal (svaseq and RUV methods) produce estimates that are highly correlated with the simulated batch effect. The estimate of batch based on principal components is biased, because the principal component is estimating a linear combination of the group and batch variable Figure . In thisp) and batch effect estimates . But the supervised methods correct the statistical significance calculations accurately.We performed an identical analysis where the batch was now correlated with the group variable. Qualitatively similar results hold in this second simulated scenario with one exception. The empirical RUV methods attempt to define control probes by identifying genes that do not show differential expression with respect to batch. But when batch and group are correlated, this may also through away genes that show signal with respect to the group variable. Similar the residual RUV approach estimates the batch variable after taking the residuals from the model fit of the counts on the group variable. However, when batch and group are correlated, this again removes batch signal and leads to slightly lower performance of the RUV approaches . UnsuperR2 = 1) and that they produced identical differential expression results  to sex variable  and batch effect estimates ..g) to sehttp://jtleek.com/svaseq/recount.html) and produce correspondingly more similar differential expression results to using the true study variable as an adjustment in the differential expression analysis . In this scenario the unsupervised sva and principal components analysis approaches outperform the empirical control RUV approach. The residual RUV approach performs worse than no adjustment for study, because signal due to the batch variable was removed when the residuals from the model relating sex to phenotype was calculated  to the population phenotype  and batch effect estimates (The ballgown  can be uufflinks . I loadeufflinks ,31 that Ballgown . I selechttp://bioconductor.org/packages/devel/bioc/html/sva.html and all analyses are fully reproducible and available as R markdown documents from https://github.com/jtleek/svaseq.Here I have described the general sva framework and I have introduced two extensions of the sva approach. The first takes advantage of known control probes to simplify the sva algorithm and the second addresses the distribution of count and FPKM data typically observed in sequencing experiments. The question of whether to use FPKM or count based approaches for the analysis of RNA-sequencing data is beyond the scope of this paper. However, I have demonstrated in this paper that regardless of the choice for measurement summary, svaseq can be applied to remove batch effects.I have shown that sva-based approaches perform comparably to other batch effect estimation procedures for sequencing when the group and unknown batch variables are uncorrelated and outperform other approaches when the batch and group variable are correlated. These extensions are currently available from the devel branch of the sva software