N = 3, which often results in statistically underpowered analysis and poor reproducibility. All these issues are compounded by the presence of experimental noise, which is harder to distinguish from instrumental error when sample sizes are limiting , experimental populations exhibit biologically heterogeneous or diffuse expression phenotypes , or when discriminating among transcriptional signatures of closely related experimental conditions . Here, we present a leveraged signal-to-noise ratio (LSTNR) thresholding method, founded on generalized linear modeling (GLM) of aligned read detection limits to extract differentially expressed genes (DEGs) from noisy low-replication RNAseq data. The LSTNR method uses an agnostic independent filtering strategy to define the dynamic range of detected aggregate read counts per gene, and assigns statistical weights that prioritize genes with better sequencing resolution in differential expression analyses. To assess its performance, we implemented the LSTNR method to analyze three separate datasets: first, using a systematically noisy in silico dataset, we demonstrated that LSTNR can extract pre-designed patterns of expression and discriminate between “noise” and “true” differentially expressed pseudogenes at a 100% success rate; then, we illustrated how the LSTNR method can assign patient-derived breast cancer specimens correctly to one out of their four reported molecular subtypes ; and last, we showed the ability to retrieve five different modes of action (MOA) elicited in livers of rats exposed to three toxicants under three nutritional routes by using the LSTNR method. By combining differential measurements with resolving power to detect DEGs, the LSTNR method offers an alternative approach to interrogate noisy and low-replication RNAseq datasets, which handles multiple biological conditions at once, and defines benchmarks to validate RNAseq experiments with standard benchtop assays.To life scientists, one important feature offered by RNAseq, a next-generation sequencing tool used to estimate changes in gene expression levels, lies in its unprecedented resolution. It can score countable differences in transcript numbers among thousands of genes and between experimental groups, all at once. However, its high cost limits experimental designs to very small sample sizes, usually  At their core, RNAseq and qPCR are two flavors of the same principle: by quantifying how many copies of different molecular templates are accumulated after a discrete number of duplication rounds, it is possible to estimate their abundance in the original sample as long as the amplification has occurred in exponential fashion  is poor. In those scenarios, the traditional route is to increase the sequencing depth for each sample with additional rounds of sequencing. Adding sequencing rounds surely increases the number of aligned reads per gene (or coverage); however, no matter how low the rate of misconstrued and contaminant reads that contribute to the total read output may be, they will amount to higher numbers of “phantom reads” undistinguishable from sequencing noise as more sequencing rounds pile up : that RNAseq-based estimates of gene expression differences between groups are as robust as the sequencing depth that underlie them.Here, we present a leveraged signal-to-noise ratio (LSTNR) thresholding method, founded on generalized linear modeling (GLM) of aligned read detection limits to extract DEGs from count-based and noisy low-replication RNAseq experiments. The LSTNR method uses an agnostic independent filtering strategy to define the dynamic range of detected aggregate read counts per gene that can be explained by the variation in sequencing coverage across genes and experimental replicates. This approach not only determines a minimum read count density that true expressed genes must accrue for reliable detection, but also qualifies genes as more or less reliable for differential expression measurements based on how distant they are from the reliable detection minimum. By taking into account that expression measurements based on read counts have different levels of resolution for different genes, the LSTNR method relies on one fundamental difference between the nature of scale-dependent dispersion in sequencing output  and the behavior of differential expression estimates 2-transformed fold changes (Log2FC) in RPM relative to gene-wise mean RPM either from all samples (for phenotype profiling experiments) or from a reference group (for treatment vs. control experiments) with N ≥ 3 replicates per group. Resolution weights of genes corresponded to the cumulative hazard of gene-wise significance scores from two-way ANOVA testing of the linear predictor of RPM from GLM of the natural parameter B(ϑ) to an exponential continuous-valued distribution with a canonical inverse link function ; DEGs equaled the subset of SGs with a minimum practical effect size δLog2FC > δEffect and post-hoc pairwise-significant Log2FC differences between at least two groups (Student's t-test p < 0.05). As a reproducibility benchmark, we refer to LSTNRs as the subset of SGs in which a minimum practical effect size δLog2FC > δEffect was detected and at least one group exhibits average Log2FC signal vs. baseline greater than transcriptome-wide measurement noise (or SNR > 1) where noise is defined as the 95% Tolerance Interval  were identified as those with significantly different weighed ANOVA scores (FDR adj. Log2FC), as well as estimated retrospective statistical power means (≪π≫) and 95% lower confidence limits (πlow). DEGREEs with π > 90% were listed in descending order by ΔLog2FC first, and then by πlow. The difference between successive values of ΔLog2FC going down the list of ranked DEGREEs  were calculated, with the expectation that genes with higher retrospective statistical power also show larger ΔLog2FC values; when true, this expectation results in a list of negative-valued lagging differences. The minimal subset of characteristic genes, or Profiler DEGREEs, corresponds to the subset of DEGREEs ranked by πlow that all show negative ΔLog2FC lag differences before the first positive instance is found.To extract a list of DEGREEs with the highest transcriptional profiling potential and minimal Type II error rates, we calculated within-gene observed effect sizes  based on the list of Profiler DEGREEs. The predictive machine-learning power of partitioned data is reported graphically via receiver operating characteristic (ROC) curves specific to each known phenotype. The minimum number of biomarkers needed for partitioning equals the number of phenotypes being partitioned minus one. Partitioning performance was then evaluated by all-at-once discriminant analysis based exclusively on the data from the biomarkers selected by sequential partitioning. Predictive machine-learning power of the canonical multivariate factors of discrimination, estimated using only biomarker data, is reported graphically via ROC curves for each expected phenotype. Phenotype segregation is depicted in multivariate space using 2-component canonical factor plots.in silico dataset of simulated RNAseq counts—originally assembled in the development of EPIG-seq  was the best candidate value to use for independent filtering across pseudogenes with simulated reads. In the case of this simulated data set, independent filtering against the threshold parameter γ did not exclude any pseudogenes from subsequent analysis—meaning all listed pseudogenes were “detectable” assuming the estimated error model of simulated read counts was properly fit by a lognormal distribution   going further down the LSTNR pipeline.The next step to implement the LSTNR method was to find a metric that places RPM-values from each of the detected pseudogenes  in the context of the entire dataset to describe the relative ) Figure . FurtherB-values were tested by two-way ANOVA to estimate transformant significance scores for each pseudogene; after multiple testing adjustment by the false-discovery rate approach  from that of all pseudoreplicates combined (Table Transformant log(RPM) ed Table . We thened Table . We esti2-fold changes (Log2FC) relative to the average RPM in the baseline group for each gene; then, we combined Log2FC of genes with their resolution weights in a two-way multivariate ANOVA model. We reasoned that scaling Log2FC relative expression measurements by their resolution metric would also homogenize the scale of variation around the means of individual genes in each experimental statistical group. Indeed, we found that the relative dispersion and homogeneity of Log2FC residuals improved after scaling by gene resolution weights . This pool of statistically significant pseudogenes contained all 1,000 differentially expressed pseudogenes with simulated co-expression patterns, as well as an extra pool of 3,541 pseudogenes from the unpatterned group exhibiting only random noise  measurements of all 20,000 pseudogenes in the EPIG-seq simulated data set that passed independent filtering, as well as among the 4,541 statistically significant pseudogenes identified after resolution-weighed ANOVA. We found log2(RPM)95%TI = ±7.7 among all 20,000 pseudogenes, and log2(RPM)95%TI = ±7.0 among 4,541 statistically significant ones. In other words, assuming these data were derived from a “true” sample of biological specimens, one could project with 95% confidence that 95% of differences in expression between groups, and for any particular pseudogene, may be ~200-fold off from their “true” expected value, or ~130-fold when adjusted for noise among significant pseudogenes, due only to experimental variability between replicate experiments  how large should RNAseq expression differences be in validation assays to be reliable; and (b) which genes are the most reliable validation candidates based on their RNAseq expression differences. With that principle in mind, we calculated 95% tolerance intervals around the mean logs Figure .in silico . Altogether, the LSTNR method detected statistically significant pseudogenes belonging to both patterned and unpatterned expression trends, but did not compromise the ability to discriminate between both kinds of pseudogenes, nor their correct expression patterns, by standard clustering analyses.The unpatterned pseudogenes with statistically significant expression levels detected through LSTNR could aggregate to excessive levels of “background noise.” This “noise” could be detrimental to statistical clustering or discriminant analyses, and may undermine the capacity to extract “true” expression patterns. To address this point, we performed naïve hierarchical clustering (Ward's method) and found that, even though unpatterned pseudogenes accounted for most of the detected differential pseudogenes overall, all differentially expressed pseudogenes were agglomerated in the hierarchical tree under simulated patterns. We also found that the 3,541 unpatterned but statistically significant pseudogenes were segregated apart from the 1,000 patterned ones Figure . Furtherin silico dataset, and compared to those obtained by EPIG and EPIG-seq analyses  and EPIG-seq (a modified version of EPIG to handle count-based data) ; in contrast, the EPIG-seq method matched pseudogenes completely to their simulated pattern only for B. The five hierarchical clades identified by LSTNR also matched their respective in silico patterns in terms of average expression levels within statistical groups  to test the LSTNR method. The data set collected for this assessment consisted of 160 primary tumors classified under 4 breast cancer molecular subtypes groups in equal sample sizes  and 40 matching normal breast tissue biopsies as controls  of equal size, performed parallel statistical analyses for each, and identified the subset of DEGs detected in common by all four separate per-realization analyses; and (b) we analyzed the entire dataset all at once in a single run. Altogether, this strategy comprised a total of 5 separate implementations of the LSTNR method on patient-derived breast cancer data: four per-realization analyses and one single-shot analysis. By following this analytical design, we sought to establish the degree of observed concordance between DEGs detected by a single-shot analysis vs. the consensus from multiple tests on subsampled statistical groups. We also aimed at determining whether capturing more DEGs with an all-encompassing single-shot analysis is superior in quality and performance to performing split tests in tandem and extracting a consensus DEG list.We analyzed TCGA dataset following two approaches: (a) we split the specimens from each of the four molecular subtypes into four independent and mutually exclusive sample subsets  where y = RPM  with uniquely aligned reads were represented in any one of the specimens used to test the LSTNR method. Empirically observed RPM averages within realizations were fit to parametric distribution functions independently. In all cases, the best-fit model for within-gene RPM averages corresponded to a 3-parameter Weibull distribution function P(y) ~ WeibullM Figure ; the estB ~ (y–γ)−1 and the normalizing link function η ~ 1/B . Even though the number of resolved genes estimated by single-shot analyses  fell within the order of magnitude resolved genes by the other four per-realization analyses, it was still over 50% larger than the average of 1,899 resolved genes per realization  were quite different in each of these separate analyses (as mentioned earlier).We then performed resolution-weighed multivariate ANOVA tests of Log2FC differences relative to the average of normal tissues, either per realization or single-shot. We found that, by including resolution scaling, the number of detected significant genes (SGs) became more consistent: between 4,465 and 5,086 SGs from the four independently analyzed realizations, and 6,193 SGs by the single-shot analysis  exhibited Log2FC differences with SNR > 1 relative to transcriptome-wide Log2FC measurement error  were also among the 6,193 SGs detected in the single-shot analysis using the entire data set. This means that about three out of four (75.6%) of all SGs identified by single-shot testing of the entire dataset are not replicated in parallel tests of mutually exclusive subsets of samples from the same experiment. We then evaluated the concordance rates between intersecting genes across per-realization vs. single-shot analyses, and detected final overlaps of 908 DEGs , 337 LSTNRs , and 336 DEGREEs overall   of the entire set of 200 specimens based on Log2FC expression differences vs. normal breast biopsies with: (a) 366 consensus DEGREEs across all per-realization analyses; and (b) the 1,511 DEGREEs identified from the single-shot analyses. We found that unguided sorting of specimens in hierarchical clusters was in agreement with the known molecular classification of the specimens, irrespective of the set of DEGREEs used Figure . An alteLog2FC; we termed these Profiler DEGREEs (Table post-hoc pairwise absolute differences (p < 0.05) in at least one of the four breast cancer subtypes vs. normal tissue biopsies in each of the 200 Profiler DEGREEs  and within-gene effect size Δs Figure , which as Figure . The incs Figure .N = 200) than the one reported previously by the TCGA consortium (N = 825)  necessary to discriminate among 4 predetermined subtypes. In practice, diagnostic testing can be carried out to validate findings, to confirm observations by other researchers, or for clinical purposes where sample numbers are limiting. Our data indicate using the 4 biomarkers detected by LSTNR to diagnose breast cancer subtypes would be at least as reproducible as using 50-gene subtype predictor microarrays reported previously  in spite of the heterogeneity in the gene expression profiles among patients because the transcriptional signatures are often overt. In the case of specimens deposited in TCGA, the evidence supporting the classification of each specimens is a combination of both histopathological criteria and transcriptional profiling, an approach pioneered almost two decades ago and refined since by Perou and others based on microarray data . Usually, these studies use large sample sizes, but that is often the case because specimens tested for the same MOA are often exposed to more than one eliciting chemical agents, each with minimal sample sizes. In the end, this means most toxicogenomics datasets cannot be split into realization subsets, and so they must be inspected with single-shot analyses. For all these reasons, identifying MOA from toxicogenomics studies can be more challenging than transcriptional profiling of disease phenotypes.N = 3 per combined MOA × Agent group. In all, the experimental design comprised 45 individual specimens equally split among 15 different agents and stratified under 5 MOAs with N = 9. Because the purpose of this particular TGxSEQC crowdsourced experiment was to create a benchmark training set to attest performance of newly developed statistical pipelines, additional levels of complexity were introduced in the experimental design by controlling their exposure scheme: some of the chemical agents were supplied by intraperitoneal injections, and others by oral gavage using either nutritive (corn oil) or non-nutritive (water) delivery vehicles. In addition to the 45 agent-exposed specimens, an additional set of 9 livers was also collected as a control group from male rats that underwent mock treatments by different combinations of exposure schemes .To assess the performance of LSTNR for toxicogenomics analyses, we used a MOA training RNAseq dataset generated through the MAQC phase III SEQC crowdsource toxicogenomics effort (TGxSEQC). The TGxSEQC training dataset consists of liver transcriptomes from male Sprague-Dawley rats following exposure to hepatotoxicants  where y = RPM. We then performed independent filtering with respect to the Weibull scale parameter α = 6.7 RPM and retained 9,593 transcripts for differential expression analysis −1 and normalizing link function η ~ 1/B as appropriate for fixed-ß Weibull distributions , which we discarded from subsequent analysis. The remaining 1,567 LSTNRs corresponding to mRNA transcripts (NM_ accession prefix) were consolidated into a final list 1,510 DEGREEs with official non-duplicate Entrez gene symbols. Then, after comparing the ranks of retrospective statistical power and within-DEGREE effect size for Log2FC measurements across MOA groups, we identified a minimal set of 65 Profiler DEGREEs for transcriptional profiling  uniquely aligned reads, which we refer to as “genes” hereafter for simplicity. The best-fit model to the average RPM-values within annotated transcripts across all 54 individual liver specimens was a 3-parameter Weibull distribution function P(y) ~ WeibullWe asked to what extent the DEGREEs detected by our method captured the underlying organization of specimens into MOA groups. Using two-way unsupervised hierarchical clustering (Ward method), we found the set of 1,510 DEGREEs sorted not only 43 of the 45 treated specimens into their original MOA and chemical agent groupings, but also recovered their stratification by route of exposure, nutritional status of vehicle, and one of two means of pathway activation: receptor mediated (RM) and non-receptor mediated (NRM). We also detected five co-expression patterns among DEGREEs, which were supported by separate clustering of correlation scores for the differential expression of DEGREEs across MOA groupings Figure . Still, Next, we investigated the capacity of Profiler DEGREEs to segregate MOA groups, and their underlying strata, compared to the pool of DEGREEs. Unsupervised clustering based on the 65 Profiler DEGREEs revealed seven MOA clades among treated specimens (I-VII) and, like its larger counterpart with 1,510 DEGREEs, five minimal gene co-expression patterns (a–e) Figure . Using oThe ability to segregate entire MOA groups from each other based on Profiler DEGREEs allowed comparing transcriptional profiles of each MOA across the five expression patterns. Among RM MOA groups, PPARA showed the strongest and most dissimilar transcriptional response; it was also the MOA with the most pronounced effects across the entire dataset and across all co-expression patterns Figure . Both Ah2+ signaling)  Figure .The results for signaling pathway enrichment analysis were further supported by sets of predicted upstream regulators via activation z-scores Figure . In partInferred regulation scores for upstream regulators were opposite to those in PPARA specimens for all other clades, except for clade IV. The distinct behavior of clade IV, which included two of the three liver specimens treated with carbon tetrachloride, did present distinctive enrichment of LXR/RXR activation and atherosclerosis signaling pathways, along with inferred activation of PPARA and KLF15. Those results for clade IV were consistent with known effects of carbon tetrachloride, a traditional model of chronic liver injury that elicits fibrogenic activity in hepatic stellate cells and loss of fenestration along liver sinusoids due to thickening of basal membranes with fibrillar collagens; both these responses are relayed via LXR signaling  from the DNA damage group  from the cytotoxic MOA group, and p Figure . Still, p Figure . Enrichmp Figure .Last, we performed sequential tree partitioning analysis of Profiler DEGREEs to extract candidate biomarkers and test their ability to discriminate MOA groups compared to using all 65 Profiler DEGREEs. The process of biomarker selection was a two-tier process: first, tree partitioning analysis was performed with all MOA groups; then, to preempt disparately larger transcriptional responses in PPARA MOA groups from undercutting our analysis, we repeated the partitioning analysis without including PPARA specimens. We detected eight biomarkers from both analyses combined: Ucp3, Acaa1b, Acaa1a, Sugct, Hadhb, Tfam, Gsdmd, and Tmem86b Figure . AltogetG2—the likelihood-ratio test—which: (a) represents the relative contribution of a gene among all others to assigning samples into expected clusters across bootstrapped partition trees; and (b) approximates a χ2 distribution to estimate the statistical significance of each gene's predictive capacity. As we surmised, the contributions of biomarker genes to MOA cluster assignment showed increasing significance as the tested candidate gene lists were curated in favor of genes with more robust expression differences  triggers and their secondary  effects in transcriptional responses to toxicants. We believe these types of results from LSTNR implementation, as shown by the analysis of the TGxSEQC dataset, reflect the underlying basis of bottom-up transcriptional networks that become more intertwined as more gene nodes are added—and yet, the LSTNR method can unravel them systematically from the top down.in silico dataset. This outcome, which rarely occurs in the analysis of bona fide experimental data sets, is consistent with using simulated data since: (a) pseudogene averages in statistical groups are prescribed under known patterns for all pseudoreplicates; and (b) read count variation is modeled around those fixed averages across the entire dataset. Given these properties of simulated data, it follows that an explanatory parametric fit of pseudogene averages should include every pseudogene if those gene averages are also parametrically tailored; if true, the estimated dynamic range of average RPM-values should be valid for all pseudogenes and, consequently, all pseudogenes should pass independent filtering—just as we saw in the EPIG-seq in silico dataset.A key element of RNAseq analysis is choosing a sensible threshold that reflects the dynamic range of gene detection based on total aligned reads from a sequencing run. In our analyses of the TCGA and TGxSEQC datasets, we used the scale parameter α of a 3-parameter Weibull fit to gene RPM means as the independent filtering threshold. Mathematically, this ensured no more than 63% of genes with aligned reads will be retained for analysis, since by definition the α parameter is the 37th percentile of a Weibull-distributed variable  in RNAseq tests with small sample sizes, it is difficult to separate between genes whose significant expression differences are more likely to be real, rather than anecdotal, in an underpowered experimental design , which are statistical instruments to account for magnitude and resolution differences all at once in non-linear and non-Gaussian systems  [or B in short]; furthermore, if the chosen linear predictor B yields transformant values whose variation around within-gene averages behave like a classical distribution from the exponential family , then the transformant values of B can be manipulated algebraically into normally distributed scores using what is known as a link functionη(B) [or η in short]. As a result, η is a representation of RPM that, unlike RPM-values themselves, are linear and normally distributed—which meets requirements for ordinary multivariate ANOVA tests of significance; this is the basis of GLM , it is continuous-valued , and monotonically increasing (larger weights for better resolved genes). Also, it is derived from read count data without being a direct transformation of read count values; the benefit is that the same weight can be applied to each gene across samples or experimental realizations whether net read counts are the same or not between individual replicates. In the context of RNAseq output, the behavior of cumulative hazard functions resembles that of the “tightness” of relative expression measurements around the mean from sequencing data: genes with low read counts are poorly detected and yield noisy estimates of log-fold differences; genes with more aligned reads produce more robust estimates with increasing coverage; and genes near saturated sequencing levels plateau to the absolute resolution limit of the sequencing instrument.Based on the TCGA dataset, we found the behavior of resolution weights to be consistent with how read count differences in RNAseq data behave: genes with better resolution will often present either low read counts and large differences between groups, or high read counts with modest differences and tight within-group variability. Since RNAseq is based on PCR amplification, observing tight distributions for abundant transcripts may occur rarely, since amplification errors are propagated exponentially. Using separate implementations of the LSTNR method in each individual realization ameliorated the impact of such features, all too common in noisy RNAseq data. In effect, resolution scaling in the LSTNR method adjusted for differential expression tests for exponential propagation of PCR errors.regardless of sample size or sequencing depth. If true, the GLM basis of the LSTNR method makes it a particularly attractive pipeline to determine consensus resolution weight matrices from small training sets heading into consolidated meta-analyses of much larger cohorts. As our findings suggests, the use of LSTNR-derived gene weight matrices would protect DEG analysis against detrimental batch effects and subsampling errors, both of which are characteristic of epidemiological and clinical studies.In theory, there is an additional benefit worth noting that LSTNR offers. GLM with canonical link functions relies on the statistical sufficiency of exponential family distributions, and the resolution weights of genes are derived from linear predictor ANOVA based on ranks, not scores, of statistical significance. If individual experimental replicates are a true representative sample of a population, and their distribution belongs to the exponential family (as implemented in the LSTNR method), then a notable statistical corollary follows: the resolution power of gene detection from one experiment is valid for separate realizations from the same population. In practical terms, this means the resolution weights for genes detected in one study are valid for any additional sequencing of its replicates, entire repeated experiments, and replicate studies—The customary methods of inferential testing among multiple groups are ANOVA models because of their computational efficiency and ease of implementation. The main caveat of ANOVA models pertains statistical power, which strongly depends on sample sizes, normally distributed responses, and homoscedasticity across statistical blocks. These requirements are a major impediment to the management of RNAseq data sets; for one, sample sizes are limiting in RNAseq applications due to cost constraints, since sequencing reagents, equipment and necessary computational support to convert raw sequencing output into genome-aligned reads are all expensive. In regard to normality of measurements, it is common practice to represent expression levels as log-transformed read counts, such that expression fold-changes between groups become arithmetic differences between means with distributions that approximate normality; this is not only questionable for most RNAseq experiments, mainly due to the impedingly small sample sizes mentioned earlier, but also insufficient as it does not address lack of homoscedasticity.Log2FC values, which are read count ratios, are a measurement of relative expression levels for a gene between conditions; however, Log2FC values by themselves do not offer any information on their own measurement resolution because the read counts used to estimate them are “divided out.” Calculating significance scores based on Log2FC values is only statistically fair if all genes show equal levels of read coverage or, as in the LSTNR method, if relative expression measurements are adjusted to reflect different resolution levels among detected genes. Introducing resolution weights to multivariate ANOVA testing of relative expression differences posited attractive statistical advantages. In principle, resolution scaling of log-fold differences should: (a) improve homoscedasticity across genes, as shown in Figure by instrumentation with respect to each other, not whether accuracy in sequencing was exactly the same for all detected genes between two different replicate experiments. In that sense, our findings showed how coupling of individual sequencing resolution of genes with relative expression differences in the LSTNR method alleviates discrepancies in the number of detectable SGs across replicate experiments, and performs well when the statistical groups are well-defined in advance—e.g., treatments, phenotypes, controls, etc.When we analyzed the breast cancer dataset from TCGA, we found large discrepancies in the number of significantly resolved genes among separate analyses, yet the number of SGs was very consistent across the board. In our view, this illustrates the main strength behind the LSTNR method: whether particular sample groupings lead to statistically significant dispersion in RPM-values in one subset vs. another is less relevant than the actual ranking of their relative resolution within replicate experiments. Our interpretation is based on the premise that estimates of variability in observed RPM-values within genes are representative of how accurately they were detected Benchmarking the expected dispersion range in RNAseq output is the foundation to a power analysis that outlines what are the smallest differences to expect ahead of deeper sequencing; often, though, access to RNAseq and costs are too prohibitive to pursue any benchmarking efforts. In such cases, the same projection for expected dispersion ranges can be used as a practical signal-to-noise (SNR) threshold of reproducibility that a select subset of genes can be validated against, even if the expression differences measured with RNAseq data for those genes were statistically significant or not. A benchmark SNR threshold of within-gene dispersion in sequenced RPM-values is a key metric for experimentalists, because it helps justify or rule out additional rounds of sequencing  when the statistical significance of any transcriptional differences is marginal—or, if all else fails, to shift focus to better experimental alternatives or re-design a project altogether.2(RPM) in a reference control group. The advantage of establishing a fixed “null hypothesis” reference, instead of using the distribution of samples in the control group, is that sample means and variation from individual replicates can be estimated for all groups, including the baseline condition, and before inferential testing of significance. The benefits are many: in practice, RNAseq experiments often produce expression measurements that, in retrospective, are underpowered, biased or unable to detect relevant biological effects because intrinsic biological variability among specimens is simply too large; in other cases, cost constraints limit studies using RNAseq technology to low replication models  while minimizing the commensurate false-positive rate projected from the available experimental data .To define candidate biomarkers for different transcriptional signatures, the LSTNR method defines a subset of Profiler DEGREEs, equal to the largest subset of ranked DEGREEs with monotonically decreasing retrospective statistical power (≪π≫ >90%) and effect size ΔN = 3). Hence, it cannot be split into realizations for parallel testing, and has to be tested through a single-shot analysis. Another challenge to testing the TGxSEQC training set deals with prospective statistical power: given 15 groups, the number of possible comparisons between them is very large. Therefore, if the analysis is performed based on pairwise gene expression differences, there is a large risk of over-correcting for multiple testing. As a result, this would underestimate the number of DEGs, and limit the ability to discriminate specimens from different MOA groups. Consequently, grouping the pool of 45 specimens under too many categories is detrimental to the statistical power of the experimental design as a whole. To circumvent this challenge, and based on the fact that each of the individual chemical agents belongs to one out of five overarching MOAs, we opted to group specimens under one out of five MOA groups with N = 9. Even then, analysis of the TGxSEQC training set using the LSTNR method captured additional sample classifications at different tiers of statistical stringency, based on MOA information alone, and simply by unsupervised hierarchical clustering. In addition, each of the three tiers of discriminant power  were consistent with each other in regards to clustering and detection of co-expression patterns. Most importantly, each increasingly stringent filter improved on the ability of its predecessor to segregate individual MOA groups: clustering by 1,510 separated MOA × Agent groups, but displayed interlaced strata from RM and NRM activation modes; based on 65 Profiler DEGREEs, clustering effectively separated each MOA from all others; and the 8 biomarkers effectively refined the discriminative power between NRM cytotoxic and DNA damage MOA strata.In both experimental datasets, Profiler DEGREEs matched the discriminant capacity of other pipelines, but with considerably less genes overall. The strongest evidence for this came from our analysis of the TGxSEQC training dataset. If stratified by agent, the TGxSEQC dataset is a 15-group experimental design with minimal replication level (We put forth the LSTNR method as an alternative pipeline to tackle current obstacles in RNAseq analysis: first, it defines a detection limit for genes with respect to random errors in instrumental sequencing and read alignment by fitting the observed distribution of aligned read counts; in return, this empirical fit offers a data-driven threshold for independent filtering. Then, the pipeline accounts for non-linear and non-homogeneously distributed variation in read counts per gene; this is used to generate a gene-wise resolution score based on read counts that, when implemented as a weight function, improves the normality and homogeneity of relative expression measurements. With the LSTNR method, the improvements in homoscedasticity by resolution scaling of Log2FC differences allow experimental designs with multiple groups to be tested by standard ordinary ANOVA techniques. Furthermore, DEGs detected by the LSTNR method capture the same transcriptional signatures at different tiers of statistical stringency with high accuracy; we found the performance of the LSTNR method was so robust that it required less genes than previously reported to discriminate breast cancer phenotypes and hepatotoxic MOAs using the same experimental datasets (Perou et al., OL conceptualized and implemented the analytical methodology described in the study, performed data analysis, and wrote the manuscript. OL and JS wrote the initial manuscript drafts leading to submission. JS and RW supervised the selection of simulated and experimental RNAseq data sets to validate and test the statistical approach described herein. All authors contributed in the design of the study and interpretation of the results, as well as revised, read, and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The iPANDA method combines precalculated gene coexpression data with gene importance factors based on the degree of differential gene expression and pathway topology decomposition for obtaining pathway activation scores. Using Microarray Analysis Quality Control (MAQC) data sets and pretreatment data on Taxol-based neoadjuvant breast cancer therapy from multiple sources, we demonstrate that iPANDA provides significant noise reduction in transcriptomic data and identifies highly robust sets of biologically relevant pathway signatures. We successfully apply iPANDA for stratifying breast cancer patients according to their sensitivity to neoadjuvant therapy.Signalling pathway activation analysis is a powerful approach for extracting biologically relevant features from large-scale transcriptomic and proteomic data. However, modern pathway-based methods often fail to provide stable pathway signatures of a specific phenotype or reliable disease biomarkers. In the present study, we introduce the  Pathway analysis aids interpretation of large-scale gene expression data, but existing algorithms fall short of providing robust pathway identification. The method introduced here includes coexpression analysis and gene importance estimation to robustly identify relevant pathways and biomarkers for patient stratification. The application of novel supervised learning algorithms to large-scale transcriptomic data has the potential to transform conventional approaches for disease classification, personalized medicine and development of prognostic models. However, their use as a modality for clinical applications is hindered by several recognized challenges and limitations. One of the most relevant challenges in transcriptomic data analysis is the inherent complexity of gene network interactions, which remains a significant obstacle in building comprehensive predictive models. Moreover, high diversity of experimental platforms and inconsistency of the data coming from the various types of equipment—may also lead to the incorrect interpretation of the underlying biological processes. Although a number of data normalization approaches have been proposed over the recent years1Despite these challenges, various transcriptomic data analysis algorithms have been developed in both academic and commercial settings, and numerous attempts have been made to apply these algorithms clinically, particularly, for predicting patient response to various anti-cancer therapies45910in silico Pathway Activation Network Decomposition Analysis (iPANDA). We demonstrate the performance of this method by using multiple paclitaxel breast cancer treatment data sets obtained from Gene Expression Omnibus (GEO)In this study, we suggest a novel method for large-scale transcriptomic data analysis called Fold changes between the gene expression levels in the samples under investigation (tumour samples) and an average expression level of samples within the normal set is used as input data for the iPANDA algorithm. Since some genes may have a stronger effect on the pathway activation than others, the gene importance factor has been introduced. Several approaches of gene importance hierarchy calculation have been proposed during the last few decades8P values from a t-test performed on groups of normal and tumour samples is applied to the gene expression values. The smooth threshold is defined as a continuous function of P value ranging from 0 to 1. The statistical weights for genes are also derived during this procedure. The topological weights for genes are obtained during the pathway map decomposition. The topological weight of each gene is proportional to the number of independent paths through the pathway gene network represented as a directed graph.Here we propose a novel approach that integrates different analytical concepts described above into a single network model as it simultaneously exploits statistical and topological weights for gene importance estimation . The smoIt is well known that multiple genes exhibit considerable correlations in their expression levelsThe contribution of gene units  to pathway activation is computed as a product of their fold changes in logarithmic scale, topological and statistical weights. Then the contributions are multiplied by a discrete coefficient which equals to +1 or −1 in the case of pathway activation or suppression by the particular unit, respectively. Finally, the activation scores, which we refer to as iPANDA values, are obtained as a linear combination of the scores calculated for gene units that contribute to the pathway activation/suppression. Therefore, the iPANDA values represent the signed scores showing the intensity and direction of pathway activation (see Methods section for details).Although currently there are several publicly available pipelines for benchmarking the transcriptomic data analysis algorithms72324One of the major issues that should be addressed when developing a novel transcriptomic data analysis algorithm is the ability of the proposed method to reduce noise while retaining the biologically relevant information of the results. Since pathway-based analysis algorithms are considered dimension reduction techniques, the pathway activation scores should represent collective variables describing only biologically significant changes in the gene expression profile.t-test P value <0.05) were utilized. Pearson correlation was chosen as a metric to measure the similarity between samples. Sample-wise correlation coefficients were obtained for the same samples profiled on Affymetrix and Agilent platforms. Similar procedure was performed using pathway activation values . The results acquired for the set of samples from MAQC data are shown in In order to estimate the ability of the iPANDA algorithm to perform noise reduction while preserving biologically relevant features, we performed an analysis of the well-known MAQC data set (GEO identifier GSE5350) ref. . It contTaken together, iPANDA demonstrates better performance for the noise reduction test in comparison to other pathway analysis approaches, suggesting its credibility as a powerful tool for noise reduction in transcriptomic data analysis.As a next step we have addressed the iPANDA ability to identify potential biomarkers (or pathway markers) of the phenotype under investigation. One of the commonly used methods to assess the capability of transcriptomic pathway markers to distinguish between two groups of samples  is to measure their receiver operating characteristics area under curve (AUC) values. The capacity to generate a high number of biomarkers with high AUC values is a major requirement for any prospective transcriptomic data analysis algorithm to be used in prediction models.t-test P values for genes were calculated over the whole group of breast cancer samples against healthy tissue samples in order to estimate the statistical weights, which were further used to obtain sample-wise pathway activation iPANDA scores. Cross-validation approach using samples from GSE20194 data set was utilized to obtain the threshold values for calculation of statistical weights and merging the gene modules. To avoid introduction of the artificial inter-data sets bias for the gene weights as a result of this approach, the gene weights in iPANDA were calculated for each of the data sets used for this analysis separately (independently).To estimate the capability of our method to produce potential biomarkers, we have selected several gene expression data sets from breast cancer patients with measured response to paclitaxel treatment. iPANDA algorithm was applied to obtain pathway activation scores for each sample. For each breast cancer data set used in this study, we have carefully selected a tissue specific normal control  HER2-positive (HER2P) and ERN HER2-negative (HER2N) breast cancer types are given in 3031323335Pathways with various numbers of member genes ranging from <10 members  to over 400  can be found in the lists. This illustrates that iPANDA algorithm treats small and large pathways in the same way, indicating the scalability hallmark of valid pathway analysis methods.Similar calculations were performed by using five different third-party pathway analysis algorithms such as GSEA, SPIA, PLAGE, DART and ssGSEA. As demonstrated in To further estimate the ability of iPANDA to detect relevant pathways, we have assessed its performance by using the prioritization criteria according to recently proposed pathway methods benchmarking pipelineOne of the most important shortcomings of modern pathway analysis approaches is their inability to produce consistent results for different data sets obtained independently for the same biological case. Here we show that iPANDA algorithm applied to the breast cancer data overcomes this flaw and produces highly consistent set of pathway markers across the data sets used in the study. In particular, the iPANDA values for 19 and 8 pathways for ERN HER2P and ERN HER2N breast cancer types, respectively, can be utilized as paclitaxel response classifiers with AUC values higher than 0.7 for all data sets examined. Whereas, all third-party algorithms tested  failed to obtain even a single pathway marker  common for all data sets examined (for both cancer types) , suggestThe common marker pathway (CMP) index (see Methods section for details) was applied to paclitaxel treatment response data for both ERN HER2P and ERN HER2N breast cancer types in order to estimate the robustness of the biomarker lists. Pathway marker lists obtained for four independent data sets were analysed. The calculation of pathway activation scores was performed using the iPANDA algorithm and its versions with disabled gene grouping and/or topological weights . The ‘ofHigh AUC values for the pathway markers shown in t-test P<0.05) (logDGE), and fold change in expression levels of most differentially expressed genes (t-test P<0.05) between the training and corresponding normal breast tissue data sets (logDFC). Logarithmic scale was used for training the gene level models. All pathway-level and gene-level data was Z-score normalized separately for each GEO data set used (see online Methods for details).In order to classify the samples as responders or non-responders, the random forest models were developed using iPANDA scores obtained for training sets of samples for each end point (see Methods section). Subsequently, performance of these models was measured using validation sets. Matthew's Correlation Coefficients (MCC), specificity and sensitivity metrics were applied to evaluate performance of the models . MCC metAs demonstrated in Moreover, taxol-based neoadjuvant therapy response prediction with iPANDA-based random forest model was performed using mixed data regardless of the cancer type of the samples. The performance metrics of the model were 0.422, 0.854 and 0.563 for MCC, specificity and sensitivity, respectively. Notably, the performance of the model on samples from various data sets used for validation was not homogeneous. Interestingly, all 28 samples (8 responders and 20 non-responders) from the GSE22513 data set were correctly separated into two groups , whereas for the GSE41998 data set the MCC value generated by the model was only 0.254. This observation demonstrates that the distinctions between data sets can be blurred, but not completely eliminated when developing prediction models. Nevertheless, the results show that patients' classification according to their potential treatment response can be successful even in case when receptor status is unknown. These finding suggest that despite the differences between breast cancer types, both ERN HER2P and ERN HER2N cancer share common features which can define treatment response sensitivity.Although we acknowledge that the methods used for comparison in this study were initially designed for the relevant pathway assessment for a given condition rather than for phenotype prediction, we have selected these methods for comparison as they are among the most cited and highly acclaimed algorithms in the community. Nevertheless, we expect that a good pathway analysis approach would produce robust scores, which should demonstrate a certain degree of discriminative power if the pathway database is chosen accordingly to the biological condition under study. Comparison to other models based on pathway activation (enrichment) scores obtained with other pathway activation algorithms shows that iPANDA-based classifiers outperform these methods for all three end points examined. Moreover, iPANDA surpasses the similar random forest models trained on gene-level data, implying that iPANDA is an effective tool for noise reduction in gene expression data, while preserving the biologically relevant features. Interestingly, using fold changes between tumour samples and normal samples instead of solely gene expression increases the predictive power of the model, while the use of differentially expressed genes only does not lead to significant improvement. Moreover, pathway analysis algorithms that explicitly account for fold changes (iPANDA and SPIA) generate better results when used as inputs for prediction models in comparison to other methods. This observation demonstrates that direct incorporation of the expression data from corresponding normal tissue into the fold change calculation, can be a valuable addition when developing prediction models.To further evaluate the proficiency of iPANDA for reliably classifying a sample with respect to the various clinical parameters, we have used the training and validation data sets refer to  along wiApplication of the pathway activation measurement implemented in iPANDA leads to significant noise reduction in the input data and hence enhances the ability to produce highly consistent sets of biologically relevant biomarkers acquired on multiple transcriptomic data sets. Another advantage of the approach presented is the high speed of the computation. The gene grouping and topological weights are the most demanding parts of the algorithm from the perspective of computational resources. Luckily, these steps can be precalculated only once before the actual calculations using transcriptomic data. The calculation time for a single sample processing equals ∼1.4 s on the Intel (R) Core i3-3217U 1.8 GHz CPU . Thus, iPANDA can be an efficient tool for high-throughput biomarker screening of large transcriptomic data sets.The use of merely microarray data for pathway activation analysis has well-known limitations, as it cannot address individual variations in the gene sequence and consequently in the activity of its product. For example, a gene can have a mutation that reduces activity of its product but elevates its expression level through a negative feedback loop. Thus, the elevated expression of the gene does not necessarily corresponds with the increase in the activity of its product. Nevertheless, comprehensive analysis of the tumour pathway activation profile may be a more clinically relevant strategy to stratify the subset of patients whose tumours could probably respond and who would clinically benefit from anti-cancer therapeutic regimens than other outcome prediction methods based on the gene expression profile. While gene expression levels can be effectively used for phenotype prediction, it is quite possible that the most differentially expressed genes in a given signature will not be part of the pathways that actually drive tumour behaviour. Alternatively, expression of some genes within the cancer driving pathways is not always predictive of the overall pathway activation. Therefore, while there is no single preferential approach for interpreting gene expression results, the proposed method of transcriptomic data analysis on the signalling pathway level may not only be useful for discrimination between various biological or clinical conditions, but may aid in identifying functional categories or pathways that may be relevant as possible therapeutic targets.Although the iPANDA algorithm was initially designed for microarray data analysis, it can also be easily applied to the data derived from genome-wide association studies (GWAS). In order to do so, GWAS data can be converted to a form amenable for the iPANDA algorithm. Single-point mutations are assigned to the genes based on their proximity to the reading frames. Then each single-point mutation is given a weight derived from a GWAS data statistical analysisOne of the rapidly emerging areas in biomedical data analysis is deep learningFrom the GEO database we have downloaded six data sets containing gene expression data related to breast cancer patients treated with paclitaxel and three data sets containing transcriptomic data from normal cancer-free breast tissue which were used as a reference .The breast cancer and normal data from different data sets was preprocessed using GCRMA algorithmERN breast cancer samples were stratified by the HER2 status: HER2-positive (HER2P) and HER2-negative (HER2N). Only samples profiled before any treatment were analysed. In this analysis, we included the samples from patients who subsequently underwent paclitaxel (Taxol) treatment and showed any definite therapy outcome (response or non-response) . Also, GThe MicroArray Quality Control (MAQC) data set (GEO identifier 5350) was obtained from the GEO database. The raw data for 60 samples from Affymetrix platform was preprocessed using GCRMA algorithmhttp://www.sabiosciences.com/pathwaycentral.php). Using a cancer-specific pathway database ensures the presence of multiple pathway markers for the particular phenotype of the breast cancer under investigation. The database contains a set of 374 signalling pathways which cover a total of 2,294 unique genes. Each pathway contains an explicitly defined topology represented as a directed graph. Each node corresponds to a gene or a set of genes while edges describe biochemical interactions between genes in nodes and/or their products. All interactions are classified as activation or inhibition of downstream nodes. The pathway size ranges from about twenty to over six hundred genes in a single pathway.There are several widely used collections of signalling pathways including Kyoto Encyclopedia of Genes and Genomes, QIAGEN and NCI Pathway Interaction Database. In this study, we use the collection of signalling pathways most strongly associated with various types of malignant transformation in human cells obtained from the SABiosciences collection . Therefore, the actual function for the calculation of the pathway Gip) and the gene modules (Mjp) is computed as follows:The contribution of the individual genes . As the expression levels are assumed to be logarithmically normally distributed and in order to convert the product over fold change values to sum, logarithmic fold changes are utilized in the final equation. Activation sign Aip is a discrete coefficient showing the direction in which the particular gene affects the pathway given. It equals +1 if the product of the gene i has a positive contribution to the pathway activation and −1 if it has a negative contribution. The factors wiS and wipT are the statistical and topological weights of the gene i ranging from 0 to 1. The derivation procedure for these factors is described in detail in the subsequent sections. Since log (fci) and Aip values can be positive or negative, the iPANDA values for the pathways can also have different signs. Thus, positive or negative iPANDA values correspond to pathway activation or inhibition respectively. The principal scheme of the iPANDA algorithm is shown in Here wipT), all possible walks through the gene network are calculated on the directed graph associated with the pathway map. The nodes of the graph represent genes or gene modules, while the edges correspond to biochemical interactions. The nodes which have zero incoming edges are chosen as the starting points of the walks and those which have zero outgoing edges are chosen as the final points. Loops are forbidden during walks computation. The number of walks Nip through the pathway p, which include gene i is calculated for each gene. Then wipT is obtained as the ratio of Nip to the maximum value of Njp over all gene unit in the pathway:In order to estimate the topological weight  are estimated according to the following equation:The i refers to the particular member  of the pathway p, N is the number of pathway members, wiS and wipT are the topological and statistical weights of the member i, pi is the group t-test P value for the member i. Since the P values obtained are not used for further pathway marker scoring they are not normalized in any way and rely solely on the statistics for the genes which have non-zero statistical weights.where In order to quantitatively estimate the robustness of the algorithm between data sets, we introduce the CMP index. The CMP index is a function of the number of pathways considered as markers that are common between data sets. It also depends on the quality of the treatment response prediction when these pathways are used as classifiers. The CMP index is defined as follows:n is the number of data sets under study, Ni is the number of genes in the pathway i and AUCij is the value of the receiver operating characteristic area under curve, which shows the quality of the separation between responders and non-responders to treatment when pathway i is used as classifier for the j-th data set. AUCR is the AUC value for a random classifier and equals to 0.5. A pathway is considered as a marker if its AUC value is higher than 0.75. The ln(Ni) term is included to increase the contribution of the larger pathways because they have a smaller probability to randomly get a high AUC value. The larger values of the CMP index correspond to the most robust prediction of pathway markers across the data sets under investigation, while zero value of CMP index corresponds to the empty intersection of the pathway marker lists obtained for the different data sets.where https://cran.r-project.org/web/packages/randomForest/index.html).In order to apply iPANDA to the paclitaxel treatment response prediction over a several independent data sets, the pathway activation values were normalized to the Z-scores independently for each data set. The expected values used for the Z-scoring procedure were adjusted to the number of responders and non-responders in the data set under study. For each end point described in the result section, random forest classifiers were trained using pathway-activation scores or gene-level data for training set. Then the predictive power of the models was evaluated using validation set. RandomForest function from bioconductoR repository (version 4.6–12) with default settings was utilized to train and run the models  and SPIA packages were chosen as the most commonly used. PLAGE was selected as the best performing method in the recently proposed pathway analysis method benchmarking approachwww.broadinstitute.org/gsea/index.jsp). All the input data files were prepared according to the GSEA User guide available at http://software.broadinstitute.org/gsea/doc/GSEAUserGuideFrame.html. Only expression levels for differentially expressed genes  were used as the input. The pathway database was converted to the GSEA file format using the same package. All the calculations were run from the command line for each tumour sample versus all available normal samples for the particular data set. The parameter ‘Number of permutations' was set to 1,000, ‘Collapse data set to gene symbols' was set to ‘false', ‘Permutation type' was set to ‘gene_set', ‘Enrichment statistics' was set to ‘weighted', ‘Scoring scheme' was set to ‘weighted', ‘Metric for ranking genes' was set to ‘Signal2Noise', ‘Gene list sorting mode' was set to ‘real', ‘Gene list ordering mode' was set to ‘descending', ‘Collapsing mode for probe sets' was set to ‘max_probe', ‘Normalization mode' was set to ‘meandiv', ‘Randomization mode' was set to ‘no balance'. Normalized enrichment score values were extracted from GSEA report for further analysis.We used java GSEA package downloaded from the GSEA official web site (http://bioconductor.org/packages/release/bioc/html/SPIA.html). The pathway database was converted to the SPIA file format using the same package. Fold changes between each tumour sample and the mean over the whole set of normal samples for the differentially expressed genes  were used as the input for the calculations. The total net accumulation perturbation (tA) values for each pathway were extracted from SPIA output for further analysis. All the steps of further pathway analysis using GSEA and SPIA algorithms were similar to the ones used for the analysis performed using iPANDA algorithm. The use of tA values for comparison with other methods has certain limitations. tA is only half of the evidence that SPIA considers and is independent of the information about the overall amount of differential expression in the pathway that the other methods use. Besides, tA is based on information derived only from the genes in the pathway that are connected with certain type of relations documented in Kyoto Encyclopedia of Genes and Genomes, while all other methods use information from all genes in the pathway. Nevertheless, SPIA tA values are the only metric besides the iPANDA values, which surpasses gene-level cross-platform correlations between MAQC-I samples and demonstrates better performance when used in phenotype prediction models comparing with gene-level data.SPIA R package was downloaded from Bioconductor Web site according to the instructions on the SPIA Bioconductor page (https://www.bioconductor.org/packages/release/bioc/html/DART.html). From all studied breast cancer samples, we constructed a normalized gene expression data matrix with logarithmic gene expression values. DART was run independently for each signalling pathway. Each pathway was given to DART as a Model Signature—a numeric vector  reflecting if a gene contributes positively or negatively to the pathway. Relevance network was calculated with the default fdr=0.000001 threshold for correlations between gene expressions. According to DART usage guidelines, we evaluated gene network consistency and pruned each network. We next predicted pathway activation scores in every sample for a given pruned gene network  and gene expression matrix. Remarkably, DART doesnot construct gene networks for the pathways in which no gene pair has significant correlation—that is why for these pathways activation scores were set to 0 in each sample. Calculated activation scores for each pathway and each sample were constructed to a numeric matrix.DART R package (version 1.20.0) was downloaded from bioconductoR repository  and normal samples from GSE9574, GSE13591, GSE19422 as corresponding controls obtained using the same profiling platform were downloaded from GEOBreast cancer and multiple myeloma data with corresponding normal samples were processed using AFFY (version 1.46.1) and FRMA (version 1.20.0) algorithms from bioconductoR repository. Neuroblastoma data and corresponding normal samples were processed using SCAN.UPS (version 2.10.9) from bioconductoR repository. We divided data into training and validation groups according to the original MAQC-II experiment designiPANDA scores were calculated for all samples from three cancer-related data sets used in MAQC-II study separately for training and validation sets. Statistical weights were calculated for the whole group of tumour samples compared with corresponding normal samples for training and validation groups independently.In order to build a prediction model for iPANDA scores we used random forest algorithm implemented in randomForest package from bioconductoR repository (version 4.6–12). All the parameters were set as recommended by default. For each phenotype iPANDA scores for MAQC-II training data sets were used as an input to random forest to make a prediction on 10 MAQC-II end points.t-test and moderate t-test were applied during maPredictDSC run. CVP=2,NF=4, NR=1,FCT=1.0 were applied during predictDSC run. The best performing model was selected using validation data sets according to AUC performance metrics. AggregateDSC mode was used to combine the prediction of several models (crowd).MaPredictDSC (version 1.6.0), the best method from IMPROVER DSC challengehttps://github.com/varnivey/ipanda) and is freely available for academic use. Transcriptomic data sets that support the findings of this study are publicly available from the GEO data sets website (http://www.ncbi.nlm.nih.gov/geo/) using corresponding accession numbers.The code of iPANDA package is deposited to github repository  as a method for biomarker development. Nat. Commun.7, 13427 doi: 10.1038/ncomms13427 (2016).Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Supplementary Figures 1-9, Supplementary Tables 1-3, Supplementary Notes 1-2 and Supplementary References.Random forest performance by series. Prediction models performance metrics are calculated separately for each GEO series in validation set including TP - the number of responders samples identified as responders, FP - the number of responders identified as nonresponders, FN - the number of non-responders identified as responders, TN - the number of nonresponders identified as non-responders, MCC -Matthews correlation coefficient, ACC - accuracy, TRP - sensitivity, SPC - specificity.


The Illumina Infinium HumanMethylation450 BeadChip and its successor, Infinium MethylationEPIC BeadChip, have been extensively utilized in epigenome-wide association studies. Both arrays use two fluorescent dyes (Cy3-green/Cy5-red) to measure methylation level at CpG sites. However, performance difference between dyes can result in biased estimates of methylation levels.https://www.bioconductor.org/packages/release/bioc/html/ENmix.html).Here we describe a novel method, called REgression on Logarithm of Internal Control probes (RELIC) to correct for dye bias on whole array by utilizing the intensity values of paired internal control probes that monitor the two color channels. We evaluate the method in several datasets against other widely used dye-bias correction methods. Results on data quality improvement showed that RELIC correction statistically significantly outperforms alternative dye-bias correction methods. We incorporated the method into the R package ENmix, which is freely available from the Bioconductor website  contains supplementary material, which is available to authorized users. M and U) and the measurement of the methylation level, called beta value, is the ratio between methylated and unmethylated intensity and is calculated as M/(M + U + c), where c is an offset usually assigned a value of 100 to regularize the beta value when both M and U are small.DNA methylation arrays are commonly employed in large-scale epigenome-wide studies. The Illumina Infinium HumanMethylation450 BeadChip provides methylation measurements at more than 485,000 individual CpG sites , and itsIt was known that the two channels usually perform differently  with oftHere we propose REgression on Logarithm of Internal Control probes (RELIC), a novel dye-bias correction method based on data properties observed in paired internal normalization control probes. It is simple, accurate and efficient. We comprehensively evaluated the method in multiple datasets for robustness and accuracy and found that RELIC significantly outperformed the alternative methods. RELIC has been implemented in Bioconductor package ENmix , is freeThe Illumina HumanMethylation450 BeadChip contains 93 pairs of internal normalization control probes , while its successor MethylationEPIC BeadChip contains 85 pairs. The two probes in each pair are designed to target the same DNA region within housekeeping genes and contain no underlying CpG sites: one probe will extend to incorporate a base A or T , and the other probe will incorporate a base G or C . These probe pairs were designed to monitor array performance in different color channels and thus can be used for dye-bias correction. If there were no dye-bias, the intensity values from the two probes of each pair would be expected to be the same with a ratio close to 1.Scatterplots of the log transformed intensity values between red and green channels for these normalization control probes demonstrates a clear linear relationship in every sample. A typical plot is shown in Additional file i is the index of probe and Ii is the intensity value of the probe in the green channel and Ii,adj is the adjusted intensity value; where n = 10, 3, 2, 3, 3, 2, 3, 3 and 10, respectively) were independently assayed on different arrays.Dataset 1: 450K dataset of a total of 39 methylation laboratory standard control samples reported by . Human uDataset 2: 450K dataset of 22 samples reported by [Dataset 3: 450K dataset of 24 samples reported by . These sTo demonstrate RELIC dye-bias correction effect, we applied the method to a large 450K dataset with 889 blood samples [The different performance between Infinium I and II probes is partially due to dye-bias . A correTo evaluate RELIC and compare it with alternative methods in terms of data quality improvement, we applied RELIC and six other dye-bias correction methods to the three datasets. The six alternative methods are: the Illumina method (Illumina), a modification of the Illumina method implemented in methylumi (methylumi), ASMN, the nanet method implemented in wateRmelon (nanet), the smooth quantile normalization implemented in lumi (lumi.quantile), the simple scaling normalization implemented in lumi (lumi.ssn). We also included results without adjustment (raw) for comparison. We did not include the nanes method (implemented in wateRmelon) in the comparisons as the nanet method is better than the nanes method, according to the Table two in .p values for the 39 samples range from 4.0 × 10−3 to 1.2 × 10−11. Separate evaluations demonstrated the same pattern  and Infinium II probes .Although all CpG sites in the experimental control samples are expected to have the same methylation level, the actual beta measurements often have skewed  distribution (data not shown), and thus for each sample we used the beta value distribution mode as the methylation level estimate. For each of the seven dye-bias correction methods, we calculated the absolute difference between the expected methylation and the averaged methylation mode across replicates for each methylation level group Fig. . The perp values range from 7.7 × 10−3 to 3.2 × 10−15, as shown in Additional file Bisulfite pyrosequencing (BPS) measurements were available to validate 450K array measures for 15 probes (six Infinium I probes and nine Infinium II probes) in Dataset 2. Similar to [p values ranging from 3.3 × 10−4 to 1.3 × 10−16  in 24 samples and calculated the absolute difference of beta values between the gold standard BPS measures and the 450K estimates. For the total of 72 measurements, RELIC again outperformed other methods  employed in Illumina’s methylation arrays have different performance characteristics, which often result in relatively compressed data distribution in the green channel. The discrepancy between color channels can directly affect methylation beta value calculation, especially for Infinium II probes (accounting for 72% of probes in HumanMethylation450 array and 84% of probes in MethylationEPIC array) where methylated states are evaluated in green channel and unmethylated states are evaluated in red channel. Based on the observed data property that intensity values are linearly correlated at log scale between red and green channels, we here introduced a novel dye-bias correction method, RELIC, to reduce the discrepancy and improve data quality. We demonstrate that RELIC can efficiently extend intensity value distribution in the green channel so the adjusted values are comparable to red channel data.We have evaluated all currently available dye-bias correction methods in several datasets , the results can be used to guide whole genome DNA methylation data analysis. To best demonstrate the effect of dye-bias correction methods, we intentionally avoided using additional preprocessing steps  that may normally be employed in Illumina DNA methylation array analysis. Our evaluations reveal that dye-bias correction methods based on internal normalization control probes perform better and more robust than quantile normalization methods that use all probe-sets on an array. This is because by design, in the absence of dye-bias, the two probes in each normalization control probe pair are expected to have identical measures. In contrast, it may not be valid to assume that intensity value distributions for red and green channels are identical in every biological sample. This is particularly true for Infinium II probes, where all methylated intensities are measured in the green channel and all unmethylated intensities are measured in the red channel. This can be demonstrated in Fig. We are unable to find EPIC data with bisulfite pyrosequencing validation, and thus we only evaluated these methods in 450K datasets. Although EPIC arrays have slightly fewer normalization control probe pairs, the application of the RELIC method is robust: using only a subset of normalization control probe pairs (85 random pairs from 93 pairs) we have done similar evaluations in Dataset 2 with results similar to that based on full normalization control probe pair set (data not shown). Thus we believe RELIC can reliably be applied in the analysis of EPIC data.We also performed a comparison of different dye-bias correction methods in combination with three probe type bias correction methods: SWAN , BMIQ 11 and RCPWe demonstrated that RELIC is a robust and efficient method to correct for dye-bias in Illumina Methylation BeadChip data, and that it statistically significantly outperforms all alternative methods, resulting in improved accuracy of methylation beta value estimates. We incorporated RELIC method into a user friendly R Bioconductor software package ENmix  to facil