The analysis of differential splicing (DS) is crucial for understanding physiological processes in cells and organs. In particular, aberrant transcripts are known to be involved in various diseases including cancer. A widely used technique for studying DS are exon arrays. Over the last decade a variety of algorithms for the detection of DS events from exon arrays has been developed. However, no comprehensive, comparative evaluation including sensitivity to the most important data features has been conducted so far. To this end, we created multiple data sets based on simulated data to assess strengths and weaknesses of seven published methods as well as a newly developed method, KLAS. Additionally, we evaluated all methods on two cancer data sets that comprised RT-PCR validated results.Our studies indicated ARH as the most robust methods when integrating the results over all scenarios and data sets. Nevertheless, special cases or requirements favor other methods. While FIRMA was highly sensitive according to experimental data, SplicingCompass, MIDAS and ANOSVA showed high specificity throughout the scenarios. On experimental data ARH, FIRMA, MIDAS, and KLAS performed best.Each method shows different characteristics regarding sensitivity, specificity, interference to certain data settings and robustness over multiple data sets. While some methods can be considered as generally good choices over all data sets and scenarios, other methods show heterogeneous prediction quality on the different data sets. The adequate method has to be chosen carefully and with a defined study aim in mind.The online version of this article (doi:10.1186/s12864-015-1322-x) contains supplementary material, which is available to authorized users. Alternative Splicing is an important mechanism for providing the protein diversity essential for eukaryotes . One of The detection of altered expression on the exon level is more challenging than gene based analyses. On one hand, changes in expression levels might be more subtle, which makes it harder to distinguish signal from noise. On the other hand, changes in the expression of the gene has to be taken into account to avoid false positives as well as false negatives. To accomplish this task, exon expression is usually normalized to the corresponding gene expression. Figure Besides using exon arrays, the challenge of DS detection also can be addressed with next generation sequencing (NGS), i.e. RNA-seq . HoweverBesides, the wealth of existing expression data from exon arrays constitutes an important basis for many scientific questions. This led to a variety of algorithms for differential splicing detection developed over time. Different approaches were taken to solve the task. Most of the methods, such as MIDAS , use a sHere, we report on, to our knowledge, the most comprehensive comparative assessment of algorithms for DS detection on exon arrays. We compared and evaluated nine different methods for the detection of differential splicing from exon arrays. We discerned the performance and challenges for each method over a range of different parameters. Using a comprehensive artificial dataset we compared the impact of different expression levels, numbers of exons per gene, different amounts of differentially spliced samples per condition as well as the influence of different group sizes. Additionally, we applied all methods to two well studied and partly RT-PCR validated cancer data sets ,13.We included, to our knowledge, all published methods where an implementation was available: MADS, MIDAS, SI , PAC 7,,7,14, ANIn the following we give a brief description of each method; for details we refer the reader to the original publications.Splicing Index (SI) is similar to the fold change (FC) often used on the gene level. As opposed to the FC, exon expression is first normalized to the corresponding gene expression before calculating the ratio between two conditions. ARH, an information theoretical approach based on Shannon’s entropy, computes the splicing deviation between conditions for every exon and transforms it into a probability for differential splicing. A gene-wise entropy computed from the probabilities is used as final quantification of DS. As with SI, MIDAS uses gene level normalized exon values. Unlike SI, a statistical test determines whether a significant difference, i.e. DS, is observed. MADS takes advantage of an elaborate gene signal estimator for probe-wise SI computation and assesses its significance with a t-test. The final p-value for an exon aggregates the singular probe-level p-values. As the input to all other methods compared is based on exon level, we adopted MADS to work on this level as well for comparability reasons. We will therefore refer to this modified method as MADS’. The underlying assumption in PAC is the proportionality of exon expression to its corresponding gene expression. Deviation from exon to gene expression results in low correlation and therefore indicates DS. ANOSVA detects DS by applying statistical tests to the parameters of a fitted exon expression model. SplicingCompass, originally developed for NGS data, can easily be adapted to exon array data. The idea is to access the significance of difference between angles spanned by exon tuples in one condition compared to the ones in the other condition [FIRMA deduces scores for DS by searching for a high difference between estimated and observed expression.The ondition . FIRMA dKLAS is a novel method and is therefore described in more detail. It uses a similar approach as ARH, but relies on the Kullback-Leibler divergence in the last step. The Kullback-Leibler divergence is an indicator for the variety of two probability distributions. For each condition ci∈{c1…cn} the deviation d of the expression of every exon e from its gene g as in Equations These deviations are turned into a probability distribution per gene and condition, such that the contribution of every exon to the expression of the gene can be denoted by Equation The performance of each method for differential splicing detection is influenced by many factors. A detailed analysis of the properties inherent to the different methods can only be achieved by using specifically designed artificial test data. To this end, we generated a range of synthetic data sets using the model from  applyingenum∈{10,30}), the expression intensity , the number of samples  per group as well as the percentage of differentially spliced samples . The combination of these four parameters with two allocations each led to a total of 16 scenarios yielding a detailed insight that is important when choosing the adequate method for a given dataset or for a certain purpose.Specifically, we studied the influence of the number of exons per gene ) the remaining 100 genes, designed as true negatives (TN), show no altered exon expression. Thus, probably the most challenging of the 16 data sets for a DS detection method  or AUC in the cases where no binary classification was applicable. Furthermore, we quantified sensitivity and specificity for more fine-grained insights. Note, that in the case of binary classification (DS event / no DS event) accuracy corresponds to the area under the curve (AUC).α=0.05. Nevertheless, we quantify performance on the basis of scores as well.Some of the methods produce p-values indicating the certainty of a DS event taking place, while PAC, KLAS, ARH and SI output a heuristic score. To achieve comparability and avoid cutoff problems, we also derived a p-value for all score-based methods using an exact Monte Carlo permutation test . AppliedAs stated, score based methods exhibit the difficulty of choosing a cutoff at which a result is believed to be relevant. There are best practices for some methods (SI is mostly used with a cutoff of 1.5  or 2 2020) or reFirstly, we report on the results for simulated data. The examined parameters section “” were evSubsequently, the results on the colon and lung cancer data sets were reported with a focus on the RT-PCR validated results. As in the case of simulated data, accuracy, sensitivity and specificity was used to evaluate performance.An overview on the accuracy over all scenarios was visualized in Figure While most of the methods achieved good results in the ‘easy’ cases of equal group size and consistent splicing events, accuracy dropped quickly when sample sizes in groups diverged, less samples per condition were spliced, or expression intensity decreased. MADS’ is closely followed by ARH, SI, SplicingCompass and KLAS, which showed similar behaviour Figure .The third-best method cluster consists of ANOSVA and MIDAS. The two performed well in the easy scenarios of sufficient sample numbers and 100% AS events in one group. As circumstances got more challenging, a rapid decay in accuracy could be observed.MADS’. This algorithm showed a unique performance not only concerning efficiency but also in the sensitivity to parameter influences   . Under such circumstances accuracy is not a good measure for performance, and we thus focused on sensitivity and specificity instead  influences of properties such as the number of samples, expression intensity or exon number has been performed yet. In this work we evaluated the impact of an extensive set of parameter combinations on the performance of eight methods. Additionally, we assessed all methods and a ninth one with respect to validated experimental data. In contrast to related work which focused on the comparison based on experimental data  and thusA rank comparison of accuracy-based results is shown in Table Here, we present the outcome of synthetic, colon cancer and lung data. Concerning the accuracy based results, some methods ranked consistently low , others consistently high  while MIDAS included a positive outlier. Nevertheless, results on experimental data should be handled with care due to the unbalanced nature and small size of the evaluation data in these data sets. Recall that accuracy is highly susceptible to a diverging number of positive and negative examples. Especially in the case of MADS’, which predicted a high number of DS genes, combined with an disproportionate high number of positive examples in the lung cancer data set this is an issue.Clearly, the performance of different algorithms was influenced differently by the various parameters and the different data properties, such as effect size and variance. To shed more light into the cause of these differences, we here sought to explain differences in the method’s performance in terms of their underlying mathematical formulation of the problem.Two methods - ANOSVA and SplicingCompass - were significantly affected by the number of exons per gene, i.e. they display a better performance in the low exon number scenario. This is remarkable, as a major concern of most other algorithms is a rising number of FPs with increasing exon number due to parallel tests. In the case of ANOSVA, the reason is that, the higher the number of exons, the more improbable it becomes to obtain significant predictions for TPs as the number of DS exons remains constant. This is underpinned by the observation, that predictions were better in the second half (genes 50 to 100) of TPs, where two instead of one exon is modeled as DS. The same reason applies to SplicingCompass, a statistics-based method, which accesses the difference between exon angles within and between groups. The higher the number of exons - while the number of DS events is constant - the lower is the ratio of angles representing a DS event. This impedes the detection of differences between groups.Interestingly, MIDAS, also a statistical method, was not affected by these parameters. Unlike SplicingCompass and ANOSVA, MIDAS directly takes into account the gene expression normalized exon expression, i.e. effect size, and applies a separate test for every exon. The number of exons per gene is thus not as important. In contrast, SplicingCompass and ANOSVA operate on a gene based level.For any method based on statistical tests, one expects that a higher number of samples improves performance as it increases test power. As expected, this behaviour was observed for ANOSVA and MIDAS, both inherently statistical methods. However, the same (positive) effect also could be observed for SI, ARH and KLAS, which do not perform tests. The explanation is that all these three methods use permutation tests, which become more stable with increasing numbers of samples. The effect was the strongest for SI with those genes which were not differentially spliced  lowers performance as increased variance prevents effects from being significant.As already mentioned in the previous paragraph, statistical methods in general are rather conservative in predicting DS events. One root of this behaviour is their test-basis, but other effects come on top. MIDAS uses gene-normalized expression values instead of exon expression values and thus requires a fairly great effect as the normalization is rather drastic. ANOSVA applies an ANOVA on a so-called interaction term derived from a fitted linear model which further smoothes away differences. Other methods are less strict in these regards. For instance, ARH uses the median exon ratio between groups for correcting for the underlying gene expression. Compared to MIDAS, which directly uses exon to gene ratio, the approach of ARH often results in a less pronounced correction which better preserves effect strength. Splicing Compass accesses the difference between exon angles within and between groups. It does not perform any explicit gene level correction, but implicitly all pairwise angels are considered, resulting in an indirect and rather weak form of normalization. Again, this helps this method to increase its sensitivity.Combining the results of simulated and experimental data completes the picture of MADS’. While leading performance for simulated data, MADS’ seemed to overrate DS events in the experimental settings. The excellent performance in the artificial scenario reflects the strong sensitivity of the method: relatively ’hard’ scenarios are still positively identified, settings in which other methods clearly voted against an DS event. According to our experiments MADS’ can not be recommended for the pure prediction of DS events, but we consider it highly suitable for ranking DS candidates because genes with a very low MADS’ p value very likely show differential splicing.A comparison of MIDAS, FIRMA, SPLICE , ARH, PAFurther comparative work was done by Laajala et al. . Though and specificity as well as in robustness of the method. According to our evaluation, ARH meets these requirements best.Depending on the research question and the experimental data, different methods pose an appropriate choice. As practically all methods showed a significant dependency on the expression level and the amount of DS samples per class the two parameters are of no help for method selection. If sample number is low and / or imbalanced, SplicingCompass is the most reasonable choice according to our evaluation. Independence on the number of exons is best achieved by ARH, while KLAS, SI and MIDAS pose similarly good choices. High specificity throughout the data sets was provided by ARH, SplicingCompass and MIDAS. When it comes to the most sensitive methods FIRMA, ARH and KLAS fulfill the task best. As validation of results is expensive and time-intensive most studies are interested in high sensitivity Over time a variety of methods for the detection of DS has been published, each of them with different characteristics regarding sensitivity, specificity, interference to certain data settings and robustness over multiple data sets. While some methods, such as ARH, can be considered as generally good choices over all data sets and scenarios, other methods show heterogeneous prediction quality on the different data sets. The adequate method has to be chosen carefully and with a defined study aim in mind.To avoid an unmanageable flood of data scenarios we restricted our simulations to cases, where one and two exons are differentially spliced per gene. Naturally, this does not represent the spectrum of actually occurring DS events. Thus, based on our study, an important question to address in future work would be the susceptibility of the methods to the number of DS exons per gene. Further improvement could be provided by varying the noise level in data generation to assess method robustness.An important topic when discussing exon arrays is its replaceability with RNA-seq. Next generation sequencing is the younger technology and therefore under constant development. While claimed to be the more accurate technology, it still displays difficulties in certain areas such as high FP and FN values in low expression ranges . TherefoAdditional file 1Supplementary materials.Additional file 2Supplementary figures.


Normalization procedures are widely used in high-throughput genomic data analyses to remove various technological noise and variations. They are known to have profound impact to the subsequent gene differential expression analysis. Although there has been some research in evaluating different normalization procedures, few attempts have been made to systematically evaluate the gene detection performances of normalization procedures from the bias-variance trade-off point of view, especially with strong gene differentiation effects and large sample size. In this paper, we conduct a thorough study to evaluate the effects of normalization procedures combined with several commonly used statistical tests and MTPs under different configurations of effect size and sample size. We conduct theoretical evaluation based on a random effect model, as well as simulation and biological data analyses to verify the results. Based on our findings, we provide some practical guidance for selecting a suitable normalization procedure under different scenarios. Pertinent statistical significance tests are applied to these normalized gene expression levels. Parametric tests such as Student’s p-values are adjusted by a multiple testing procedure (MTP) in order to control certain quantity of per-family Type I error, such as familywise error rate (FWER) High-throughput data such as microarray expression data and RNA-seq data have become an indispensable tool for medical research nowadays. Typically, a data pre-processing step called There are different sources of technological noise in high-throughput genomic data stronger than global normalization. On the other hand, it does not change the rankings of genes and retains more parametric information than the rank normalization. So it can be viewed as a compromise between the global and rank normalization procedures.Multiple-array normalizations adjust the scale across arrays to avoid different arrays having undue weight. A typical example is the quantile normalization. Motivated by quantile-quantile plot, it makes the empirical distribution of gene expressions pooled from each array to be the same local because only one gene is needed to normalize a given gene. Interested readers are referred to A data-driven variable transformation called the Surrogate variable analysis (SVA) Although SVA was not originally designed as a normalization procedure, it can serve this role as long as the array-specific technical noise is considered as one of the unmodeled factors. To some extend, SVA can be viewed as an extension of the global normalization, which normalizes the the observed expression levels by removing just one factor, per-array mean expressions.Normalization procedures are designed to remove technological noise and improve the detection of differentially expression genes. However, the testing power improvement from normalization procedures may also come with a price. The performance of a particular normalization procedure in terms of gene selection power and type I error control is heavily influenced by the sample size and true effect size of gene differentiation. Some efforts have been made to evaluate different normalization procedures In this paper, we conduct a thorough study to evaluate the performance of gene selection strategies which consist of four normalization procedures  combined with three statistical tests and two MTPs, with various combinations of sample sizes and effect sizes. It is well-known that variance reduction always comes with the price of bias. We use this bias-variance trade-off principle to study the theoretical statistical properties of normalization procedures. Simulation and biological data analyses are conducted to support the theoretical results in the biological application. Based on our study, we make several concluding points in the Section Discussion. Whenever possible, we provide theoretical explanations to support these conclusions based on a random effect model The following supporting materials are presented in In this paper, we assume all expression levels are log-tranformed. Additive (non-additive) noise are thus multiplicative (non-multiplicative) noise in terms of the original expression levels. For convenience, the words “gene” and “gene expression” are used interchangeably to refer to these log-transformed random variables.Let We denote This high correlation may come from two sources We divide genes into three sets:The set of differentially expressed genes (abbreviated as DEGs) is the union of both up-regulated and down-regulated genes, which is denoted by The following normalization procedures are studied in this paper:GLOBAL): Subtract each element of the data matrix by the mean over all gene expression signals on the array to which this element belongs 1. Global normalization (RANK): Replace every gene expression in one array by its rank in the (ascendingly) ordered array divided by the total number of genes. Denote 2. Rank normalization (QUANT): First, a reference array of empirical quantiles, denoted as 3. Quantile normalization (DELTA): Sort all genes by their sample variances and denote the sorted array by local because it only involves one gene to normalize another. Next, select candidate gene pairs by applying an appropriate hypothesis test and an MTP to ad hoc method was proposed 4. SVA): in this approach, the observed gene expression is modeled as5. Surrogate variable analyses  ordinary linear regression based on least squares; b) a modified linear regression method implemented in LIMMA. An Due to the nature of the SVA method, a combination of SVA with linear regression is comparable to the two-sample NONE).6. As a comparison, we also study the properties of gene selection procedures without normalization  with sample size 10 and effect size 1.8, the average sample variance of expression levels is 0.1286 before global normalization and 0.01367 after. For the biological data HYPERDIP with sample size 10, the average sample variances before and after global normalization are 0.1514 and 0.01395, respectively.This variance reduction is confirmed in both simulations and biological data analyses. In balanced differential expression structure henceforth. In either case, we predict that the testing power will be improved because the induced bias is small and the variances of global normalized genes are reduced.When The bias effect can be substantial if the differential expression structure is highly unbalanced. For example if most DEGs are up-regulated  with sample size 10 and true effect size 1.8 as an example, the average sample mean difference The effect size of DEGs decreases if SIMU3 with sample size 10 and effect size 1.8 before and after quantile normalization are 0.1286 and 0.06378, respectively. Similarly, the average sample variances of biological data  before and after quantile normalization are 0.1514 and 0.0086, respectively. The variance reduction of GLOBAL is better than that of QUANT in SIMU3 since GLOBAL is designed for additive noise structure. On the other hand, the variance reduction of QUANT is better than that of GLOBAL in real biological data since QUANT is robust to non-additive noise structure.The variance of Like the global normalization, the benefit of variance reduction induced by the quantile normalization outweighs the adverse effect of bias introduced by this procedure when According to For NDEGs, the quantile normalization introduces a bias RANK introduces a bias Compared with the quantile normalization, the rank normalization goes even further in the nonparametric direction. Based on derivations in Section 2 of In a highly unbalanced gene differential expression structure, for example RANK comes from a very different mechanism as compared to the other normalization procedures. It removes the variance of noise by only preserving the ordering of observations.The variance reduction effect of DELTA comes from the imperfect “pair-breaking” method. Due to the nature of DELTA, a pairing breaking method must be used to produce list of individual DEGs. Specifically, a gene is labeled as DEG only if it belongs to two significant gene pairs. On average, the probability of a given NDEG to be paired with a DEG twice is approximately The bias increasing effects of DELTA come from the gene pairing and subtraction. This variance reduction is confirmed in both simulation studies and real data analyses. As an example, the average sample variances of data SIMU3 with sample size 10 and true effect size 1.8 before and after HYPERDIP, sample size 10) before and after ceteris paribus. When the sample size or effect size becomes large, the testing power of The variance reduction effects of GLOBAL, QUANT, and RANK), we predict that strategies based on DELTA produce a stable number of false positives approximately equal to independent of either the sample size or the effect size. This property of DELTA can be attractive when the effect size or the sample size is large.Unlike other normalization procedures (Since the SVA method is based on a complex linear model ), directWe apply the following tests to the normalized data to compute unadjusted t).LIMMA).A moderated Wilcox).Wilcoxon rank-sum test (Nstat).The third test is a multivariate nonparametric test which has been successfully used to select differentially expressed genes and gene combinations In order to control Type I errors, a suitable multiple testing procedure (MTP) must be applied to BONF): The adjusted Bonferroni procedure (BH): Let Benjamini-Hochberg procedure (We present our results based on HYPERDIP), 79 patients (arrays) with a special translocation type of acute lymphoblastic leukemia(TEL) and 45 patients (arrays) with a T lineage leukemia (TALL). Since the original probe set definitions in Affymetrix GeneChip data are known to be inaccurate http://brainarray.mbni.med.umich.edu. Each array is represented by an array reporting the logarithm (base 2) of expression level on the set of 9005 genes.The biological dataset used in this study is the childhood leukemia dataset from the St. Jude Children’s Research Hospital database SIMU-BIO, as follows. We apply HYPERDIP and TEL (79 slides chosen from each set) without any normalization procedure or multiple testing adjustment. Under the significance level 0.05, 734 genes are declared to be DEGs with an unbalanced differential expression structure (677 up-regulated and 57 down-regulated). We record the mean difference across HYPERDIP and TEL for each DEG as its effectwww.ixinyiwu.com size  and discover 546 DEGs with a balanced differential expression structure (259 up-regulated and 287 down-regulated). We then apply the above resampling procedure to create simulated data with sample size To match the statistical properties of real gene expression more closely and mimic other noise sources such as non-additive noise, we apply resampling method to the biological data to construct the main simulated data, denoted by In addition, we conduct several simulation studies based on a widely adopted random effect model used in These simulated data can help us to gain better insight into the performance of different normalization procedures. First, we simulate several sets of data with additive noise based on a Each set of data has two groups of We generate three sets of simulated data with the following configurations.SIMU1: The expectations of DEGs in group SIMU1 to study the impact of different effect sizes on gene normalization procedures when the sample size is fixed and relatively small. SIMU2: The expectations of DEGs in group SIMU2 to study the impact of different sample sizes on gene normalization procedures when the effect size is fixed and relatively small. SIMU3: These datasets have the same configuration as SIMU2 except that the expectations of DEGs in group SIMU3 to study the impact of different sample sizes on gene normalization procedures when the effect size is fixed and relatively large. SIMU-BIO, SIMU1, SIMU2, and SIMU3. We apply normalization procedures first and then conduct hypothesis tests to obtain raw p-values. After that, we apply multiple testing procedures to get adjusted p-values. We declare a gene to be differentially expressed if its adjusted p-value is less than a prespecified significance level BONF MTP are presented in We randomly generate HYPERDIP, TEL and TALL). The numbers of positives with BONF are presented in We apply the aforementioned gene selection strategies to detect differentially expressed genes across three different microarray datasets . Overall, gene selection strategies based on NONE  have very good control of type I errors, but their statistical power are poor and the variability of results  are larger. One explanation of this phenomenon is that all normalization procedures reduce the variability caused by the NONE has better statistical power. As a comparison, we reduced SIMU1 and applied the same gene selection strategies as before. The true/false positives produced by gene selection strategies based on normalized data are almost identical to those produced from the original SIMU1 data; but NONE has slightly better power. For example, with true effect size 1.0 and 60 up regulated genes, the mean number of true positives detected by In this study, we find that all four normalization procedures can reduce the variances and covariances of gene expressions so that the statistical power of the subsequent gene selection procedures may be improved. However, they also introduce certain biases which may cause more Type I errors and/or reduce testing power in certain situations. This bias-variance trade-off is common in many different branches of statistics. We found that gene selection strategies based on DELTA is the most robust one. The numbers of false positives produced by the Among all normalization procedures, we found that SVA produces very unstable results. Take SIMU-BIO) as an example, the testing power related with SVA is very good , but it produces far too many false positives in several occasions. Similar unstable behavior of SVA can be observed in Compared with other normalization procedures, Another notable result is that gene selection procedures based on normalizations have better power while retaining less or comparable false positives when the gene differential expression structure is balanced.QUANT and RANK have the best performance when the differential expression structure is balanced; GLOBAL is the clear winner otherwise. NONE is more attractive when the differential expression structure is unbalanced. DELTA has poor performance compared with other options. GLOBAL seems to have the best overall performance for both cases.We also compare the performance of different normalization procedures by ROC curves. ROC curves represent the balance of true and false positives detected at different significance level and is most relevant if the primary goal of expression analysis is to select a fixed number of “top genes” instead of using a formal MTP to control type I error. In Based on these results, our main conclusions can be summarized as follows.We recommend applying normalization when the sample size is relatively small  the statistical power is too low (too few DEGs are identified) and the differential expression structure (in terms of up/down regulated genes) is balanced; b) the goal is to select a fixed number of “top genes”.For large sample data, we recommend conducting differential expression analysis without normalization first. If the statistical power is adequate and a reasonably high percentage Click here for additional data file.


Juvenile Idiopathic Arthritis (JIA) is one of the most common chronic disease conditions affecting children in the USA. As with many rheumatic diseases, there is growing interest in using genomic technologies to develop biomarkers for either diagnosis or to guide treatment . Here, we explore the use of gene expression patterns in peripheral blood mononuclear cells (PBMC) as a first step approach to developing such biomarkers. Although PBMC carry many theoretical advantages for translational research, we have found that sample heterogeneity makes RNASeq on PBMC unsuitable as a first-step method for screening biomarker candidates in JIA. RNASeq studies of homogeneous cell populations are more likely to be useful and informative. The importance of developing biomarkers to assist in the diagnosis or to monitor the efficacy of therapy of adults and children with rheumatic diseases has been long recognized. Thus, the emergence of technologies to study gene expression on a genome-wide scale held considerable promise that, by surveying patterns of gene expression in an unbiased way, novel biomarkers might be developed and/or therapies individualized to maximize efficacy124x is shows higher expression in T cells, but much lower expression B cells in one of the phenotypes compared to the other, then significant differences in expression may not be detected by either count based or fragment length based methods for analyzing RNASeq data1617In the past 5 years, there has been a growing trend toward using RNASeq as the preferred method for transcriptional analysis, even for biomarker development91214We used samples from three independent cohorts  for this study. In cohort A, we studied 8 samples each for children: i) with newly diagnosed active untreated disease (ADU); ii) with active disease who had been on treatment (ADT); iii) who fit criteria for clinical remission on medication (CRM); and 8 healthy controls (HC). For the cohort B, we studied 9 patients with active disease on treatment and 10 patients in clinical remission on medication, each child of cohort B was studied on 2 occasions . In the cohort C, there were 8 samples in each of the three different JIA states and 8 HC subjects.Children designated as having active disease all had synovitis, as indicated by the presence of warmth and synovial thickening, in at least one joint. We used the Wallace criteriaWe first sought to determine whether we could identify biomarker candidates that might unambiguously identify patients whose disease remained active while they were on treatment (ADT) from those who had achieved remission. Having such biomarkers would be useful in guiding decisions about how long to continue therapy and when it might safely be discontinued. In order to identify transcriptome changes that occur after treated patients achieve the remission (CRM) stage, we attempted to identify differently expressed genes (DEG) by comparing ADT and CRM. Criteria for remission are described in the Methods section.Prior to DEG analysis, we previewed the distributions of the ADT and CRM samples by constructing multi-dimensional scaling (MDS) plots over gene expressions. The two groups of samples scattered randomly without forming separable clusters  when we et al.We next adopted the sample weighting method recently reported by Liu There are multiple computational approaches that can be used to identify DEG from RNASeq data2223We next attempted to combine samples across the three cohorts, on the assumption that larger sample sizes might better enable DEG identification. In this analysis, we used voom with sample quality weighting and applied batch effect removal before calling DEG. Once again, we were unable to identify DEG that might serve as useful biomarkers of remission (CRM), given the inter sample variation. Four DEGs were identified with adjusted p-values of ≤0.3 and fold change (FC) ranging between 1.4 and 1.6. These findings and low statistical significance (~25%) demonstrate that merging of the three cohorts’ samples did not improve DEG discovery.The inability to crisply identify DEG for the comparison of treated and remission JIA subjects using RNASeq data from PBMC is likely due to due to mixed transcriptomic signals that emerge from the heterogeneous cell types of PBMC. To test this possibility, we performed MDS analysis using RNASeq data from CD4+ T cells from an independent cohort of JIA samples (11 TCell_ADT and 10 TCell_CRM). We observed two well separated clusters representing children with active, treated disease and treated and children in remission, as shown in Treatment response in JIA is likely to be a continuous process; patients do not suddenly move from active disease to remission in a single step. We thus hypothesized that our inability to distinguish ADT and CRM might be due to the fluid nature of “active disease” and “remission” as treatment response evolves (over the course of active disease) or becomes more established (over the course of remission). We therefore analysed the transcription profiles of PBMC from the patients in cohort B. We first examined the ADT samples over two time points  samples might be distinguishable based on their timing of collection. We examined transcriptomic dynamics of cohort B JIA patients who have achieved remission on medication over two time points . We plotted MDS over gene expressions to preview the cohort B CRM sample distribution. No distinct clusters were observed among the CRM_B samples from the two time points . Based oWe next reduced sample variability using the voomWithQualityWeights function of the limma R package prior to DEG discovery. This analysis resulted two and one DEGs identified before and after the sample quality weighting procedure, with no better performance obtained by the additional four methods for DEG discovery . Taken tAlthough we were unable to identify biomarkers from PBMC expression data that might be useful in guiding therapy , there are other clinical arenas where biomarkers might be useful. For example, a simple diagnostic biomarker would be a significant advance for the field, as broadly used screening tests such as antinuclear antibody and IgM rheumatoid factor assays have little clinical or diagnostic utility in childrenWe next examined whether there might be other uses for PBMC transcriptional analysis in JIA, for example, understanding disease mechanisms or mechanisms of treatment response/non-response. For example, very little is known about how transcriptomes are re-organized in early therapy of JIA patients, although our workFor this analysis, first we previewed sample distributions of ADU and ADT samples from cohort A and cohort C, no time course information for those ADT subjects. MDS analysis showed that most of the ADU samples from cohort A formed sub-cluster within ADT samples , suggestWe next ran DEG analysis on two cohorts (A and C) independently using voom with sample quality weighting. This procedure resulted in the identification of 12 and 131 DEG  for cohort A and C respectively, with no common DEG between the two cohorts. As with the previous analyses, within group sample heterogeneity is one of the strong challenges in discovering candidate biomarkers , as demoet al. demonstrated that ADT subjects could be grouped by treatment response such as partial, full, or non-responsive based on microarray analyses of peripheral blood buffy coat samplesFollow this, we investigated the differences in transcription profiles between treated JIA patientsFinally, we sought to determine whether PBMC transcriptomes generated by RNASeq might give us insights into the biology of remission by comparing transcriptional profiles of JIA individuals who achieved remission on medication (CRM) with healthy controls. MDS and hierarchical clustering analysis resulted in two clusters separating samples from CRM_C and HC_C  but no cet al. adopted a different method for reads mapping When genome-wide expression technologies were introduced, they were heralded as a promising means through which biomarkers might be developed in a broad variety of diseases, including rheumatic disease such as rheumatoid arthritis and JIA4One of the important issues in developing biomarkers for clinical adaptation is the selection of appropriate cells or tissues from which to identify candidates and then develop biomarkers. PBMC remain attractive candidates because they are easy to obtain and simple to prepare. However, PBMC carry significant potential disadvantages from the standpoint of developing expression-based biomarkers. The most significant of these disadvantages is the fact that PBMC represent a heterogeneous cell population, including cells of both myeloid and lymphoid linages, with each of these linages also containing distinct sub-populations. At the same time, transcriptomes show considerable cell-type specificity, reflecting the specific functions of each cell type and subtype. Thus, while there are some broad commonalities that both ENCODE and the Roadmap Epigenomics projects have identified in the transcriptomes of peripheral blood cells14We generated RNASeq data from PBMC using 3 independent patient cohorts in an attempt to determine whether we could identify biomarker candidates that would assist in either disease staging  or in diagnosis . Our data demonstrate that adding an additional layer of variability in the use of a mixed cell population compounds the already-existing challenges in the use of RNASeq data as a first step in identifying clinically useful biomarkers. These challenges include both technical and biological issues.The technical issues that may introduce variability in RNASeq data have been discussed in detail recently39Another issue that the field faces is the clinical and biological heterogeneity of the disease entity we call “polyarticular JIA”. Patients with this disease display a broad range of ages at presentation , differences in disease severity  considerable differences in their responses to specific therapies, differences in autoantibody expression, and differences in the rates of specific complications such as uveitis. These phenotypic differences are accompanied by well documented biological differencesThe heterogeneity of the patient population is not the only issue. Recent studies have found considerable differences in gene expression in healthy individuals that are linked to genetic variance32Taken together, multiple analysis approaches failed to crisply separate ADT and CRM samples from RNASeq of PBMC. This could be a real biological phenomenon, reflecting the fact that the ADT and CRM states are too similar in their transcriptome properties to be distinguished. However, it is equally plausible that PBMC, being a heterogeneous mixture of cell types, produce a wide range of signals that contribute to a noisy transcriptional landscape. The heterogeneity of the sample is further complicated by the known heterogeneity of the patient population designated, “polyarticular JIA”We should point out that neither subject heterogeneity (patients or controls) nor sample sizes interdict proper JIA classification  when relatively homogeneous cell populations are used in RNASeq studies. In the current study, we demonstrate that we can easily classify ADT and CRM samples based on MDS analysis of RNASeq data from CD4+ T cells. Our group has also recently shown that RNASeq expression patterns in neutrophils crisply identify children with untreated JIA and those in remissionet al. have shown that serum level of S100 proteins and high-sensitivity C-reactive protein assays can distinguish JIA patients who will flare within 6 months after coming off therapy from those who will go on to more stable remissionAt the present time, conventional, serum based biomarkers will probably continue to be the most reliable approach for monitoring therapy and providing a basis for clinical decision-making. For example, Gerss While ours was essentially a negative study, it carries important findings for the field of pediatric rheumatology and translational research in rheumatology in general. Several large clinical trials in juvenile arthritis are about to be launched, and there is considerable interest in the field in obtaining appropriate clinical samples to elucidate the basic biology of treatment response or non-response.While PBMC carry many advantages in terms of the ease with which they can be obtained and stored, they are very likely to be limited in what they can tell us in gene expression studies. Our data demonstrate that the heterogeneity of this cell population adds an additional level of variance that already challenges both biomarker development and mechanistic studies aimed at understanding the underlying biology of therapeutic response. These findings should be considered in planning the collection of biological research specimens in the context of pediatric rheumatology clinical trials.We studied children with the polyarticular, rheumatoid factor-negative form of JIA . This set of JIA patients was recruited from Women and Children’s Hospital of Buffalo General Pediatrics Clinics. ADT and CRM patients were all on treatment with both etanercept and methotrexate. CD4+ T cells were prepared from PBMC using negative selection procedures.The research protocols for this study were approved by the institutional review boards (IRBs) of the University at Buffalo, University of Oklahoma Health Sciences Center and Seattle Children’s Medical Center. The research methods were carried out adhering to the approved IRBs protocols. Informed consent was obtained from the parents of all patients and healthy controls prior to obtaining specimens. In order to accomplish the goals of this study, it was necessary to study the same patient on two occasions at the same disease stage  as well as to access technical reproducibility by examining samples from a completely independent patient cohort. To increase the rigor and general applicability of these studies, cohorts A, B and C were sequenced in two different sequencing facilities. Developing biological replicates was obviously not possible for children with active, untreated JIA, as therapy starts as soon as possible after a diagnosis. Similarly, we have found that it is difficult to get more than one sample from a given child who has achieved the inactive disease as defined by the Wallace criteriaWe used the Wallace criteriaFor both PBMC and CD4+ T cell RNASeq sample preparation, whole blood was drawn into 10 mL citrated Cell Preparation Tubes . For PBMC, cell separation procedures were started within one hour from the time the specimens were drawn. PBMC and CD4+ T cells were separated from granulocytes and red blood cells by density-gradient centrifugation. PBMC were then immediately placed in TRIzol™ reagent  and stored at −80 °C. For CD4+ T cell, cells were isolated using negative selection from each sample using StemSep™ Human CD4+ T Cell Enrichment Kit .Total RNA was extracted using TRIzol™ reagent according to manufacturer’s directions. RNA was further purified using RNeasy MiniElute Cleanup kit including a DNase digest according to the manufacturer’s instructions . RNA was quantified spectrophotometrically  and assessed for quality by capillary gel electrophoresis . cDNA libraries were prepared for each sample using the Illumina TruSeq RNA Sample Preparation Kit by following the manufacture’s recommended procedures. Libraries were sequenced using 100 basepair (bp) paired-end reads on Illumina HiSeq 2500 instruments. Library construction and RNASeq were performed at the Columbia University Genome Center (cohort A and C) or at the University at Buffalo Genomics and Bioinformatics Core (both replicates of cohort B and CD4+ T cells samples).http://hgdownload.soe.ucsc.edu/goldenPath/hg19/chromosomes/) using TopHat version 2.0.13Raw paired-end reads were aligned to the human reference genome hg19 downloaded from the University of California Santa Cruz Genome Bioinformatics Site, UCSC (http://www.bioinformatics.babraham.ac.uk/projects/fastqc/), we conducted QC on raw sequenced reads. Prior to downstream analysis, we carried out a series of quality control (QC) checks on the RNASeq reads. Using FASTQC tool  were then used for MDS analysis to determine whether group separations  are possible based on RNASeq expression signatures.http://hgdownload.cse.ucsc.edu/goldenPath/mm10/bigZips/chromFa.tar.gz). Gene expression abundance was calculated using htseq-counts on mapped reads using UCSC Mus Musculus mm10 gene annotation, followed by voom, with sample quality weighting for DEG analysis.We tested the accuracy of the DEG analysis workflow by applying an identical analysis workflow that we used for PBMC RNASeq in SMchD1 experiment dataHow to cite this article: Wong, L. et al. Limits of Peripheral Blood Mononuclear Cells for Gene Expression-Based Biomarkers in Juvenile Idiopathic Arthritis. Sci. Rep.6, 29477; doi: 10.1038/srep29477 (2016).


RNA-seq is a powerful tool for measuring transcriptomes, especially for identifying differentially expressed genes or transcripts (DEGs) between sample groups. A number of methods have been developed for this task, and several evaluation studies have also been reported. However, those evaluations so far have been restricted to two-group comparisons. Accumulations of comparative studies for multi-group data are also desired.We compare 12 pipelines available in nine R packages for detecting differential expressions (DE) from multi-group RNA-seq count data, focusing on three-group data with or without replicates. We evaluate those pipelines on the basis of both simulation data and real count data.As a result, the pipelines in the TCC package performed comparably to or better than other pipelines under various simulation scenarios. TCC implements a multi-step normalization strategy  that internally uses functions provided by other representative packages . We found considerably different numbers of identified DEGs  among the pipelines for the same real dataset but similar distributions of the classified expression patterns. We also found that DE results can roughly be estimated by the hierarchical dendrogram of sample clustering for the raw count data.EEE-E pipeline) for count data with replicates . For data without replicates, the DEGES-based pipeline with DESeq2  can be recommended.We confirmed the DEGES-based pipelines implemented in TCC performed well in a three-group comparison as well as a two-group comparison. We recommend using the DEGES-based pipeline that internally uses edgeR  contains supplementary material, which is available to authorized users. RNA sequencing (RNA-seq) is a basic tool for measuring expressions of multiple genomic loci –5. One iRead counts across technical replicates derived from a single source fit to a Poisson distribution , 21. ForX and DEG identification Y), and each R package has its own methods for the X-Y pipeline [X, then construct the model , and calculate p-values  as Y. Previous studies have demonstrated that X has more impact than Y on the ranked gene list [X-Y for edgeR and DESeq “edgeR-edgeR (or E-E)” and “DESeq-DESeq (or D-D)”, respectively.In general, the DE analysis consists of two steps , Y using an empirical Bayesian method implemented in the baySeq package [X using TMM after elimination of the estimated DEGs (step 3) comprising the TMM-baySeq-TMM normalization pipeline. The key concept is to alleviate the negative effect of potential DEGs before calculating the normalization factors in step 3. As mentioned previously [n procedure. Accordingly, a generalized pipeline with the multi-step normalization can be described as X-(Y-X)n-Y in which the X-(Y-X)n with n > = 2 corresponds to the iterative DEGES-based normalization.We previously proposed a multi-step normalization procedure called TbT . TbT con package  (step 2)eviously , the DEGX-(Y-X)n-Y. Recommendations are made from an extensive simulation analysis: (i) edgeR-(edgeR-edgeR)3-edgeR on two-group RNA-seq data with few replicates and (ii) DESeq-(DESeq-DESeq)3-DESeq on two-group data without replicates [Our TCC package  implemenplicates . Howeverplicates –28, 33, plicates , 34–37, X-(Y-X)n-Y with n = 3 as recommended [XYX-Y with the fixed number of n for short. We basically confine individual methods (X and Y) in each pipeline to functions provided by the same packages  for simplicity. For example, the edgeR-related pipeline is “edgeR-(edgeR-edgeR)3-edgeR”, where X = TMM and Y = the DEG identification method, implemented in edgeR. Although we previously termed this pipeline “iDEGES/edgeR-edgeR” [EEE-E for convenience. Similarly, the “DESeq-(DESeq-DESeq)3-DESeq” pipeline can be shortened to DDD-D. This is because (1) users can select, for example, different DEG identification methods Y for steps 2 and 4 and (2) we will discuss some possible combinations such as DED-S for the “DESeq-(edgeR-DESeq)3-DESeq2” pipeline. In this sense, the DEGES-based pipeline can also be denoted as X-(Y-X)n-Z or XYX-Z.To investigate the performance of DE pipelines for a multi-group comparison, a total of 12 pipelines available in the nine packages were mainly evaluated in this study: TCC (ver. 1.7.15) , DESeq , 5 or 25 % of the genes are DEGs (PDEG = 5 or 25 %), the levels of DE are four-fold in individual groups, and the proportions of DEGs up-regulated in individual groups  are , , , , , , and .Following our previous studies , 12, we EEE-E, DDD-D, and SSS-S), edgeR, edgeR_robust, DESeq, DESeq2, voom, SAMseq, PoissonSeq, baySeq, and EBSeq. Table EEE-E pipeline were the highest and similar across the seven different proportions of DEGs up-regulated in individual groups . The edgeR  performed the second best overall. EEE-E and edgeR performed comparably under the unbiased proportion of DEGs in individual groups . This is quite reasonable because the EEE-E can be viewed as an iterative edgeR pipeline and their theoretical performances are the same under the unbiased condition [EEE-E and edgeR, the DDD-D (or SSS-S) can be viewed as an iterative DESeq (or DESeq2) pipeline. As expected, DDD-D (or SSS-S) consistently outperformed DESeq (or DESeq2) in all simulation conditions except for the unbiased situations.We first assessed the performances of a total of 12 pipelines: three pipelines in TCC n-Z or XYX-Z, where Y and Z are the DEG identification methods and X is the normalization method. We investigated the effect of the individual methods  by analyzing a total of 12 pipelines (eight DEGES-based pipelines and four non-DEGES-based pipelines). Table EEE-E, DDD-D, E-E, and D-D; colored gray in Table Z has more impact on the gene ranking accuracy than choosing Y and that using the DEG identification method provided in edgeR in both Y and Z can be recommended. In comparison with the two normalization methods in X in the eight DEGES-based pipelines, the method in DESeq (denoted as “D”) gave slightly higher AUC values than the TMM normalization method in edgeR (denoted as “E”). However, the superiority of DESeq in X was not observed when the four non-DEGES-based pipelines X-Z were compared, where edgeR  outperformed DESeq. In any case, the different choices in X have less impact than those in Y and Z. Additional file As mentioned above, TCC can perform various combinations for the DEGES-based DE pipeline DED-E, followed by EEE-E and DDD-E 3-edgeR” pipeline, consists of the normalization method in DESeq as X and the DEG identification method in edgeR as Y and Z. These results suggest that in some cases, the suitable choices of the best pipeline may slightly improve DE results. We should note that the current simulation data are generated by the “simulateReadCounts” function in TCC. This is simply because, to the best of our knowledge, TCC only provides the R function that can generate multi-group simulation count data. TCC simulates all counts using NB distributions, implying that this simulation framework advantageously acts on the classical R packages such as edgeR and DESeq. This is probably the main reason for poor performances of two recently published packages  performed the best, followed by DESeq (D-D) and edgeR (E-E). This is completely different from the results in Table XYX-Z are compared, choosing Z for the DEGES-based pipeline has more impact on the gene ranking accuracy than choosing Y (similar to Table S) can be recommended as Z. This result may possibly be explained by the removal of outliers that do not fit the distributional assumptions of the model [Unlike (multi-group) count data with replicates, there are few packages that can manipulate count data without replicates. These include TCC, edgeR, DESeq, DESeq2, and so on. We here evaluated a total of 20 pipelines (13 DEGES-based pipelines and seven non-DEGES-based pipelines). Table he model : DESeq2 he model  implemenhe model . In the XYX step also slightly but reliably improves ranked gene lists. For example, the AUC values higher for SSS-S than S-S  are by virtue of the multi-step normalization strategy originally proposed by Kadota et al. [Y step) in the SSS-S pipeline were nearly zero  in all simulations, although the actual PDEG values were 25 %. This is reasonable because any attempt to work without replicates will lead to conclusions of very limited reliability [DEG value (=5 %) to obtain certain differences between the DEGES-based approach SSS-S and non-DEGES-based approach S-S: at least 5 % of the top-ranked genes are not used when the normalization factors are calculated at step 3 in the XYX pipeline. As an estimated PDEG value of x% tends to work better when simulation data with the same PDEG value are analyzed, accurate estimation is the next important task. Additional file In addition to the model construction only with non-outliers in the Z step, the DEGES-based normalization in the a et al. . Howevera et al. , DESeq aa et al. . Indeed,iability . TCC empIn addition to the simulation study, we also analyzed a real RNA-seq count dataset sequenced from the three species : humans (HS), chimpanzees (PT), and rhesus macaques (RM) . BrieflySSS-S to D-D) show similar ranked gene lists. This is mainly because the seven pipelines from the four packages  commonly employ a generalized linear model (GLM) framework. Indeed, the minimum value of Spearman’s correlation coefficients (r) among the seven pipelines was 0.9240. It is also noteworthy that ranked gene lists produced from TCC’s iterative strategies and the corresponding original non-iterative strategies are particularly similar. For example, the r between EEE-E from TCC and E-E from edgeR was 0.9999, implying that these data are not extremely biased in light of the proportions of DEGs up- and/or down-regulated in individual groups . That is, the proportions of DEGs in these data  are rather closer to  than, for example,  or .Figure p-values and ranked genes in ascending order of the values. This is probably the main reason for EBSeq having lower similarity than the others. We also confirmed this trend with some simulation data. As shown in Sheet 2 in Additional file DEG = 5 %,  for , and Nrep = 9. A typical dendrogram of 12 ranked gene lists obtained from this simulation condition is given in Additional file Note that the dendrogram shown in Fig. EEE-E, DDD-D, and SSS-S) and the corresponding non-DEGES-based ones . The Jaccard coefficients, defined as “intersection/union” for two sets of DEGs, for the three pairs  were top-ranked among a total of 66 possible pairs. For example, both EEE-E in TCC and E-E in edgeR reported the same numbers of DEGs (=7247). Of these, 7208 DEGs (99.46 %) were common, and the Jaccard coefficient was 7208 / 7286 = 0.9893  show the distributions for each of the pipelines. It is reasonable that no DEGs identified by individual pipelines are assigned as a flat expression pattern  for the HS (G1) vs. PT (G2) vs. RM (G3) comparison. We found that most DEGs were assigned preferably to one of four patterns  and unpreferably to one of two patterns (G1 > G3 > G2 and G2 > G3 > G1). That is, up- (or down-) regulation in G1 for DEGs tends to coincide with G2 more than G3. This can also be seen in the results from sample clustering of the raw count data . This kind of classification can also be performed using EBSeq . EBSeq dWe finally assessed the reproducibility of ranked gene lists. Remember that the real dataset we analyzed here consists of three groups, each of which has six BRs (we denote this dataset as “rep1-6”). In addition to the original three-group comparison with six replicates , we also performed three three-group comparisons by dividing the original dataset into three; individual subsets consist of two BRs for each group. For example, the first subset (say “rep1-2”) consists of a total of six samples for comparing HS_rep1-2, PT_rep1-2, and RM_rep1-2. Likewise, the third subset (“rep5-6”) is for comparing “HS_rep5-6 vs. PT_rep5-6 vs. RM_rep5-6”. After performing the DE analysis for the three subsets , we obtained three ranked gene lists for these subsets. Accordingly, there are a total of four ranked gene lists  for each pipeline. We evaluated the reproducibility of ranked gene lists (i) for each subset to the original dataset  and (ii) among the three subsets .EEE-E pipeline  and low reproducibility for two pipelines (SAMseq and EBSeq). This trend is consistent with the simulation results shown in Table Figure E-E and D-D) under one simulation condition  for  shown in Table E-E, we evaluated a total of eight combinations, 4 method options  in “calcNormFactors” function × 2 test options (“chisq” and “F”) in the “glmLRT” function, provided in edgeR. We observed quite similar performances between the two test options (“chisq” and “F”). The average AUC value when using the method = ”TMM” option was the highest (91.47 %), followed by “RLE” (91.46 %), “upperquartile” (91.40 %), and “none” (91.19 %). Since the best practice  is the default in E-E , the choices should be left unchanged. Currently, TCC does not allow these options to be changed when performing EEE-E that can be recommended for multi-group data with replicates.In general, there are multiple options for some functions, and different choices may result in different ranked gene lists. We investigated the effect of different choices for two representative pipelines . We evaluated a total of 18 combinations, 3 method options  × 3 sharingMode options  × 2 fitType options , in “estimateDispersions” function provided in DESeq. While the average AUC value for the suggested combination in DESeq  was 90.60 %, the highest value in the 18 interrogated combinations was 91.69 %. Surprisingly, the best performing combination  did not include any suggested choice.We should note that method = “blind” and sharingMode = “fit-only” was especially important to obtain high AUC value when analyzing count data with replicates. Recall that the combination was the default (or suggested) choices for DESeq when analyzing count data without replicates. The high AUC values with the sharingMode = “fit-only” option can be explained by the nature of simulation data (see the ‘Simulation data with replicates’ subsection). In other words, (i) the sharingMode = “fit-only” option is advantageous when existence of count outliers is not assumed  and (ii) the simulation data generated by the “simulateReadCounts” function in TCC do not have count outliers. The method = “blind” option ignores the group labels (G1 or G2 or G3) and can compute a gene’s empirical dispersion value even if there is no BRs. As described in the manual, this method can lead to loss of power . The use of method = “blind” for count data with replicates cannot practically be recommended in light of potential low sensitivity. Most importantly, DESeq  is no longer recommended by the authors and DESeq2  is recommended instead regardless of the number of BRs [We found that using both r of BRs .S-S under one simulation condition  for  shown in Table type options (“ratio” and “iterate”) in the “estimateSizeFactors” function × 3 fitType options  in “estimateDispersions” function, provided in DESeq2. Overall, we found that different choices for fitType options had more impact than those of type options. Moreover, the use of fitType = “parametric” had the highest AUC values , followed by the uses of “local” (81.53 and 81.31 %), and “mean” (76.02 and 75.84 %). Since the best practice  is the default in S-S , the choices should be left unchanged. Similar to the above described for EEE-E, TCC does not allow these options to be changed when performing SSS-S, which can be recommended for multi-group data without replicates. These results indicate that, as expected, suggested options should basically be used. The AUC values for these combinations are given in Additional file We also investigated the effect of different choices for one pipeline We evaluated 12 pipelines for DE analysis of multi-group RNA-seq count data. Second to two-group comparison, this experimental design has arguably been performed well in practice. To our knowledge, the current evaluation is the first comprehensive study on multi-group count data. Our main findings can be summarized as follows:EEE-E, DDD-D, and SSS-S) were higher overall than the corresponding non-DEGES-based pipelines: E-E (edgeR), D-D (DESeq), and S-S (DESeq2), respectively  perform comparably to or better than the GLM-based pipelines  when a number of replicates are available . We expect the DESeq2-related pipelines  would be recommended for analyzing two-group data without replicates as an updated guideline. The comprehensive evaluation should, of course, be performed as one of the next tasks.Fourth, to analyze three-group data without replicates, we recommend using either -S Table . Both piunique expression patterns as an objective filtering) in the clustering function (“clusterSample”) provided in TCC, further evaluation should also be performed.Fifth, the results of DE analysis (including existence or non-existence of DEGs) can roughly be estimated by the hierarchical dendrogram of sample clustering for the raw count data Table , and 9. We speculate that the current recommendations made from the three-group comparative study can be applied to data consisting of three or more groups. While our preliminary analysis for four- and five-group simulation data has produced similar results to the current study, comprehensive evaluations are the next tasks.All analyses were performed using R (ver. 3.2.0 pre-release)  and BiocV) of the NB distribution can generally be modeled as V = μ + Φμ2. The empirical distribution of read counts for producing the mean (μ) and dispersion (Φ) parameters of the NB model was obtained from Arabidopsis data (three BRs for both the treated and non-treated samples) in [The three-group simulation data analyzed here were produced using the “simulateReadCounts” function in TCC. The variance (ples) in . The outThe real count dataset (“suppTable1.xls”) was obtained from the supplementary website of . The rawGene lists ranked in accordance with the level of DE are pre-required for calculating AUC values. The input data for DE analysis using all R packages are the raw count data where each row indicates the gene (or transcript), each column indicates the sample (or library), and each cell indicates the number of reads mapped to the gene in the sample. The versions of major R packages were TCC ver. 1.7.15, edgeR ver. 3.8.5, DESeq ver. 1.18.0, DESeq2 ver. 1.6.3, limma ver. 3.22.1, samr ver. 2.0, PoissonSeq ver. 1.1.2, baySeq ver. 2.0.50, and EBSeq ver. 1.6.0.X-(Y-X)n-Z or XYX-Z were performed using the TCC package. This pipeline includes EEE-E, DED-E, EDE-E, DDD-E, EEE-D, DED-D, EDE-D, DDD-D, SSS-S, EEE-S, DED-S, EDE-S, and DDD-S. Four other non-DEGES-based pipelines X-Z  were also performed using this package, since they were the hybrid ones originally implemented in different packages. These DEGES-based and non-DEGES-based pipelines were performed using two functions: “calcNormFactors” for X (and Y) and “estimateDE” for Z, in the TCC package. For the DEGES-based pipelines X-(Y-X)n-Z, the options for X, Y, and n in the “calcNormFactors’ function correspond to norm.method, test.method, and iteration, respectively. The E, D, and S for X correspond to norm.method = “tmm”, “deseq”, and “deseq2”, respectively. The E, D, and S for both Y and Z correspond to test.method = “edger”, “deseq”, and “deseq2”, respectively. For n in the DEGES-based pipelines X-(Y-X)n-Z, the iteration = 3 was used as recommended in [DED-S pipeline was performed using the “calcNormFactors” function with norm.method = “deseq”, test.method = “edger”, and iteration = 3 options, followed by the “estimateDE” function with test.method = “deseq2” option 0-Z were accomplished by applying iteration = FALSE. The genes were ranked in ascending order of the p-values. The p-value adjustment for the multiple-testing problem was performed using the “p.adjust” function with method = “BH” option .All the DEGES-based pipelines ended in . For exa“tmm”, “edger”, “deseq”, and “deseq2”) of options in TCC. The options used for individual functions in those three packages, internally used in TCC, are the same as those suggested in the original packages. Accordingly, three pipelines  as the default procedures in edgeR, DESeq, and DESeq2 can also be performed using TCC. For example, the S-S pipeline in TCC can be performed using the “calcNormFactors” function with norm.method = “deseq2”, test.method = NULL, and iteration = FALSE options, followed by the “estimateDE” function with test.method = “deseq2” option  of the three other packages  in accordance with the specific choices  and edgeR_robust, were performed using the edgeR package. The E-E pipeline for analyzing count data with replicates was performed using the following functions: “DGEList”, “calcNormFactors” with the method = “TMM” option, “estimateGLMCommonDisp” with method = “CoxReid” option, “estimateGLMTrendedDisp” with method = “auto” option, “estimateGLMTagwiseDisp”, “glmFit”, and “glmLRT” with test = “chisq” option. When analyzing count data without replicates, the “estimateGLMCommonDisp” function with three options  was used and two functions (“estimateGLMTrendedDisp” and “estimateGLMTagwiseDisp”) were not used, as suggested. The edgeR_robust method was performed using the following functions: “DGEList”, “calcNormFactors” with method = “TMM” option, “estimateGLMRobustDisp” with prior.df = 10, maxit = 6, and record = FALSE options, “glmFit”, and “glmLRT” with test = “chisq” option. The gene ranking and p-value adjustment procedure were performed the same way as described above.Two pipelines, D-D was performed using the DESeq package. The D-D for analyzing data with replicates was performed using the following functions: “newCountDataSet”, “estimateSizeFactors” with locfunc = median option, “estimateDispersions” with method = “pooled”, sharingMode = “maximum”, and fitType = “parametric” options, and “fitNbinomGLMs”. When analyzing data without replicates, the “estimateDispersions” function with the following options was used as suggested: method = “blind” and sharingMode = “fit-only”. The genes were ranked in ascending order of the p-values. The p-value adjustment for the multiple-testing problem was performed using the “p.adjust” function with method = “BH” option .The pipeline S-S in the DESeq2 package was performed using the following functions: “DESeqDataSetFromMatrix”, “estimateSizeFactors” with type = “ratio” option, “estimateDispersions” with fitType = “parametric” option, and “nbinomLRT” with modelMatrixType = “standard’ option. The genes were ranked in ascending order of the p-values. Since this package provides adjusted p-values, the number of DEGs satisfying the 5 % FDR threshold was obtained using the values.The pipeline method = “TMM” option in edgeR, “voom”, “lmFit”, “eBayes”, and “topTable”. The gene ranking was performed using the resultant p-values. Since this package provides adjusted p-values, the number of DEGs satisfying the 5 % FDR threshold was obtained using the values.The pipeline voom in the limma package was performed using the following functions: “DGEList”, “calcNormFactors” with nperms = 100, nresamp = 20, resp.type = “Multiclass”, and fdr.output = 1.0. Since this package only provides adjusted p-values, the gene ranking was performed using the adjusted p-values.The pipeline SAMseq in the samr package was performed using the “SAMseq” function with the following options: npermu = 500 option. The gene ranking was performed using the resultant p-values. Since this package provides adjusted p-values, the number of DEGs satisfying the 5 % FDR threshold was obtained using the values.The pipeline PoissonSeq was performed by using the “PS.Main” function with estimationType = “edgeR” option, “getPriors.NB” with samplesize = 5000 and estimation = “QL” options, “getLikelihoods” with pET = “BIC” option, and “topCounts”. Since this package only provides adjusted p-values, the gene ranking was performed using the values. The ordering information in the output of the “topCounts” function was used for classifying the expression patterns of genes.The pipeline baySeq was performed using the following functions: “new”, “getLibsizes” with maxround = 5, Qtrm = 1.0, and QtrmCut = −1), and “GetMultiPP”. There are five expression patterns to consider when comparing three-group data. The “EBMultiTest” function was performed with the consideration of all the five possible patterns. The posterior probability obtained from the “non-DEG” pattern was used as a surrogate estimate for the adjusted p-values. The gene ranking was performed using the values. The MAP information in the output of the “GetMultiPP” function was used for classifying the expression patterns of genes.The pipeline EBSeq was performed using the following functions: “GetPatterns”, “MedianNorm”, “EBMultiTest” with three options (