Microarray is a powerful technique used extensively for gene expression analysis. Different technologies are available, but lack of standardization makes it challenging to compare and integrate data. Furthermore, batch-related biases within datasets are common but often not tackled. We have analyzed the same 234 breast cancers on two different microarray platforms. One dataset contained known batch-effects associated with the fabrication procedure used. The aim was to assess the significance of correcting for systematic batch-effects when integrating data from different platforms. We here demonstrate the importance of detecting batch-effects and how tools, such as ComBat, can be used to successfully overcome such systematic variations in order to unmask essential biological signals. Batch adjustment was found to be particularly valuable in the detection of more delicate differences in gene expression. Furthermore, our results show that prober adjustment is essential for integration of gene expression data obtained from multiple sources. We show that high-variance genes are highly reproducibly expressed across platforms making them particularly well suited as biomarkers and for building gene signatures, exemplified by prediction of estrogen-receptor status and molecular subtypes. In conclusion, the study emphasizes the importance of utilizing proper batch adjustment methods when integrating data across different batches and platforms. The probes lengths typically vary from 25-mer to 60-mer. In addition to these commercially available platforms, spotted arrays with either cDNA or synthesized oligonucleotide probes deposited onto the array surface comprise a cost effective alternative. Spotted arrays can be either commercially manufactured or in-house manufactured. Because of their relatively low cost and flexibility, the spotted microarray technology has been widely used in many academic laboratories.The microarray technology has been extensively used for genome-wide gene expression analysis. Microarray is a well-established, cost-effective, high-throughput technology able to simultaneously measure the expression levels of thousands of genes and hereby offers an efficient way to generate a snapshot of the entire transcriptome. Several different microarray platforms are commercially available, differing with respect to the fabrication methodologies and length of the oligonucleotide probes. Some manufactures use photolithography (light directed) methods while others rely on ink-jet technology forThe workflow of a microarray gene expression analyses consists of multiple steps. First RNA is extracted, amplified, and either biotin-labeled or labeled with fluorescent dyes Cy3/Cy5) depending on the platform. Subsequently, the labeled RNA is hybridized to the array, typically overnight. After hybridization, the arrays are washed and scanned. Finally, the scanned images are quantified by specialized software. All abovementioned steps comprise a potential source of systematic variation. In addition, there are multiple technical issues that must be controlled in the fabrication and use of spotted arrays. Spotted arrays are often manufactured in relatively small batches under semistandardized conditions. This often introduces systematic differences between the measurements of different batches, often termed “batch effects” . Further/Cy5 depeCombining data from different experiments/platform/batches without reducing possible systematic variant can give rise to misleading results that can be strong enough to mask or even confound true biological signals and lead to misinterpretation of the data. For unmasking, it is necessary to identify and remove the batch effects before proceeding. Several approaches have been developed for removal of systematic batch effects. Single value decomposition and principal component analysis (PCA) have been used by subtracting the component representing the batch effect from the data . DistancDue to its extensive use, thousands of gene expression microarray datasets have been deposited to public databases making these repositories valuable data sources. Microarray gene expression data has been widely used in medical decision-making research. For clinical use of array-based diagnostics, significant numbers of patient samples are needed for training and evaluation. Therefore, reuse, pooling, and integration of multiple microarray dataset are attractive approaches. Because of the heterogeneous nature of microarray datasets, it is important to address the issue of cross-platform variation as demonstrated in several studies when combining different dataset from different platforms analyzed in different laboratories, for example, medical decision-making purposes –9. Furth in-house 29K array platform and Agilent SurePrint G3 microarray platform. The 29K dataset contained known batch effects associated with the fabrication procedure. Using these unique datasets, the aim was to assess the significance of removing systematic batch effects existing within a dataset and when integrating datasets of different platforms.In the current study, we analyzed the same 234 breast cancers on two different microarray platforms, ourThe study was carried out as a retrospective register study and in accordance with the Helsinki Declaration. The study has been approved by The National Committee on Health Research Ethics of Denmark (S-VF-20020142), waiving the requirement for informed consent for the study. BRCA1 (n = 33) or BRCA2 (n = 22) or familial non-BRCA1/2 cases with no detectable germline mutation in BRCA1 or BRCA2 (n = 65) were included in the study. In addition, 123 primary tumor samples from sporadic breast cancers were included in the study. In order to determine the tumor cell content, slides of the frozen tumor biopsies were haematoxylin-eosin-stained and examined by a pathologist at the Department of Pathology, Odense University Hospital. Samples analyzed in the present study contained at least 50% tumor cells. Immunohistochemistry determined estrogen receptor (ER) expression data was obtained from the Danish Breast Cancer Cooperative Group (DBCG). Gene expression analyses were performed using two different microarray platforms: the in-house manufactured 29K oligonucleotide microarray platform and Agilent SurePrint G3 platform.This study was performed on a series of 243 frozen primary breast tumors obtained from the biobanks of the Department of Pathology, Odense University Hospital and the Danish Breast Cancer Cooperative Group (DBCG). The tumor samples comprise a subset of a larger series of 253 tumor samples previously reported , 11. BreμL and spotted onto CodeLink HD (SurModics) activated glass slides by a high-precision spotting robot . The slides have an active polymer coating of amine-reactive groups permitting the 5′-amino modified DNA oligos to covalently attach under high relative humidity. The microarray fabrication was carried out in a controlled environment at 38% humidity and the temperature was held constant at 23°C. SMP 2.5 stealth pins  was used to deposit the oligos onto surface of the slides. Up to 85 arrays were spotted per batch. After printing, slides were incubated overnight at 70% humidity and blocked as recommended by the manufacturer.For manufacturing of the in-house spotted arrays, a human oligonucleotide library consisting of 28,919 DNA oligonucleotide probes (60-mer) was purchased from Compugen-Sigma-Genosys (The Woodlands). The oligonucleotide probes were solubilized in 150 mM sodium phosphate buffer (pH 8.5) to a final concentration of 20 pmol/Total RNA was extracted from freshly frozen tumor tissue. RNA extraction was carried out using Trizol Reagent (Invitrogen) followed by RNeasy Micro Kit (Qiagen) including DNase treatment. RNA concentration was determined using a NanoDrop Spectrophotometer (NanoDrop Technologies) and the quality assessed by the Agilent 2100 Bioanalyzer using Agilent RNA 6000 Nano Kit (Agilent Technologies). RNA integrity numbers (RIN) were calculated, and RNA was amplified and labeled using the Amino Allyl MessageAmp II aRNA Amplification Kit (Ambion) according to the manufacturer's protocol, starting with 750 ng of total RNA. Amplified aRNA from the tumor samples were labeled with Cy5. Universal Human Reference RNA (Stratagene) was labeled with Cy3 and used as reference RNA. Dye incorporation efficiency was measured by NanoDrop Spectrophotometer. Labeled sample aRNA corresponding to 200 pmol Cy5 and reference aRNA 130 pmol Cy3 were hybridized to the in-house spotted arrays, whereas 20 pmol Cy5 sample aRNA and 20 pmol Cy3 reference aRNA were used for hybridization to Agilent SurePrint G3 Human GE 8 × 60K Microarrays (Agilent Technologies). Fragmentation, hybridization, and washing for both spotted arrays and Agilent arrays were carried out using the Agilent Gene Expression Hybridization Kit and Gene Expression Wash Buffer Kit (Agilent Technologies) according to the manufacturer's protocol in a low ozone environment. Subsequently, the arrays were scanned using an Agilent G2565CA Microarray scanner (Agilent Technologies). Microarray data have been deposited into the Gene Expression Omnibus (GSE54275).2 transformed Cy5/Cy3 ratios. Data preprocessing was performed using the R package limma [ k-nearest neighbors averaging (k = 10). The Agilent dataset contained 0.012% missing data points, and the 29K dataset contained 1.144% missing data points.The scanned images of the spotted arrays were quantified by Gene Pix Pro 6.0 (Molecular Devices), whereas images from Agilent arrays were processed by Agilent Feature Extraction Software 10.7.3.1 (Agilent Technologies). In the subsequent data preprocessing, raw intensity data from the spotted arrays and Agilent arrays were treated equally. Raw intensity data were background corrected using normexp method and bad quality features flagged during feature extraction were removed from further analysis . Data wege limma , 16. Rephttp://earray.chem.agilent.com/earray) using the original probe sequences provided by the manufacturer. Based on the overlapping gene set, a data set was created for each platform. In cases of multiple probes per gene symbol, only the probe with the maximum mean intensity  was kept.To obtain the overlapping gene set measured on both platforms and to perform an unbiased cross-platform comparison, reannotation of the probes was performed. For both the platforms, the probes were reannotated to gene symbols by Agilent eArray tool  plots were generated using data where the expression level of each gene had been standardized to zero mean and unit variance by Qlucore Omics Explorer 2.3 (Qlucore).To assess the correlation between sample pairs and between gene pairs from the two platforms, we calculated Pearson's correlation coefficients. To evaluate cross-platform reproducibility after pooling the two datasets, the intraclass correlation coefficients, ICC, were calculated. Wilcoxon rank sum test was used when comparing the correlation coefficients. samr.Differential expressed gene analysis was carried out using significance analysis of microarrays (SAM) method using the implementation found in the R package ESR1 transcript. For each platform, the optimal sensitivity-specificity cutoff was determined based on receiver operating characteristic (ROC) curves, performed by the R package pROC [ER status was predicted by the gene expression data using the probes targeting theage pROC . The sta genefu [The tumors were classified into five intrinsic molecular subtypes using the 50-gene subtype classifier (PAM50) described by Parker et al. . Distanc genefu .V coefficient was used as a measure of the relative strength of the associations, ranging from 0 to 1 (perfect association), using the implementation found in the R package vcd [Cramer's kage vcd . BRCA1, 22 BRCA2, 65 non-BRCA1/2, and 123 sporadic) were analyzed on both the 29K microarrays and Agilent microarrays. The gene expression analyses were conducted as two color analyses with a common reference sample cohybridized to all arrays. We sought to control other possible sources of variations that could influence the comparison and introduce additional nonplatform dependent differences. Thus, identical RNA amplification/labeling method and hybridization/washing conditions were used for both analyses. To avoid that day-to-day or batch-to-batch variation influenced the biological interpretation of the results of the RNA extraction, the RNA amplification steps and array hybridizations steps were performed in randomized sample orders. To provide the most up-to-date annotations, we reannotated all probe sequences and mapped them to gene symbols. To enable comparison of the two platforms, we created data subsets with only the shared gene set. In total, 22171 and 19191 unique gene symbols were represented on the Agilent arrays and spotted arrays, respectively, of which 18424 gene symbols are available on both arrays , indicating that the overall gene expression patterns had become more alike. Similarly, correlations between gene pairs across all samples were calculated (P < 2.2e − 16).For a more quantitative evaluation on the effect of the adjustment of the 29K dataset, correlations between sample pairs and gene pairs from the two platforms were assessed by Person's correlation coefficient prior to and after the batch adjustment Figures . Person'lculated . CorrelaPooling and integration of multiple microarray dataset are attractive to increase the number of sample in order to enhance statistical power. Because of the heterogeneous nature of microarray datasets, it is important to address the issue of cross-platform variation when combining dataset from multiple sources. In the current study, we were in the exceptional situation that a large number of samples had been analyzed on two different microarray platforms. This gave us a unique possibility to evaluate the significance of adjusting for interplatform differences by assessing the cross-platform agreement of the gene expression measurements. We chose to use the ComBat method for standardization of the two datasets. Intraclass correlation coefficients, ICC, were used as a measure of cross-platform agreement of the gene expression levels measured on the two platforms prior to and after interplatform adjustments. ICC is scale sensitive and provides an assessment of the consistency and reproducibility of quantitative measurements.P < 2.2e − 16). To test if filtering of genes with low variance would improve correlation even further, we created a data subset retaining only genes with a standard deviation >0.3 across samples (2125 genes). When using only the high-variance genes, we gained additional improvement ICC coefficient , demonstrating that high-variance genes are more reproducibly measured across different microarray platforms. An ICC coefficient matrix between all possible sample pairs was generated. True for all sample pairs, the highest ICC coefficients were always observed when comparing the measurements of the same sample measured on the two platforms (data not shown).Interplatform adjustment led to a highly significant increase in ICC coefficients  = 0.1; median (after interplatform adj.) = 0.46; A number of studies have been carried out to evaluate the correlation between data produced by different microarray platforms. Reports of both low and high reproducibility between different platforms exist , 27–29. Our results demonstrated that both intraplatform adjustment for batch effects and between-platform adjustments improve the reproducibility of the gene expression measurements. In addition, the analysis showed that by filtering out low-variance genes, we were able to enrich for genes that were highly reproducibly expressed across platforms, making high-variance genes particularly well suited as biomarkers or for building gene signatures.A common application of gene expression profiling studies is to detect candidate genes that are differentially expressed between two or more groups. With the two datasets available derived from the same set of cancer samples, we were in a position to assess the agreement of differentially expressed genes across different platforms. For the purpose we applied the significance analysis of microarrays (SAM) algorithm, frequently used to identify differentially expressed genes between two groups. The samples were divided into estrogen receptor (ER) positive and negative samples. ER status was selected as it is a clinical used biomarker that is able to divide breast tumors into two clinically important and clearly distinguishable groups of cancers. The biological differences between ER+ and ER− samples are very strong and known to have a profound influence on the gene expression pattern. The SAM test was applied to the Agilent dataset and to the 29K dataset both with and without ComBat intraplatform batch-effect adjustment, respectively. The gene lists were then ranked according to their statistics. The proportion of shared top genes comparing Agilent versus 29K unadjusted and Agilent versus 29K intraplatform adjusted, respectively, is visualized in Furthermore, we sought to evaluate whether the high-variance genes also were among the most consistently expressed genes across platforms. By only including high-variance genes, the proportion of shared genes were clearly higher both in the ER+/ER− comparison and in the luminal A/B comparison, respectively Figures . Our resESR1) due to its known clinical importance and the availability of immunohistochemical (IHC) ER expression data for the vast majority of the samples (231 out of 243). Density plots of the gene expression levels of the ESR1 probe showed very clear bimodal distributions when measured on the Agilent platform, representing tumors with low and high expression of ESR1, respectively. The ESR1-low peaks show large overlap with the IHC ER negative tumors, whereas the ESR1-high peaks cover the majority of the IHC ER positive tumors. In the unadjusted 29K dataset, the ESR1 measurement also displayed a bimodal distribution associated with the IHC ER status, although the ESR1-high peak was less distinct. ComBat intraplatform batch adjustment of the 29K corrected this, resulting in a bimodal distribution curve very similar to the Agilent measurements. The ESR1 expression measurements on the Agilent and the 29K platform were highly correlated ( ESR1 was r = 0.93. ComBat batch adjustment further improved the correlation coefficient (r = 0.96). To evaluate the predictive power of ESR1 expression in relation to IHC ER status, we calculated the area under curve (AUC) of ROC curve analyses leading to very comparable results in the two datasets . Using ESR1 cutoff values obtained from the bimodal distribution , we obtained mean balanced accuracies of 89% and 88% for the Agilent dataset and the 29K datasets, respectively   .The high levels of agreement between IHC-based ER status and ER status prediction and the high cross-platform concordance rate indicate that predictions of estrogen receptor status by microarray gene expression analysis are very robust. Because of its high discriminatory power, in a study by Li et al., the authors suggested that gene expression analysis might even be a better choice for assessing ER status compared to IHC-based analysis as it might be more sensitive compared to traditional IHC analysis . They shhttp://dx.doi.org/10.1155/2014/651751). A strong association between subtype classifications was achieved when sample pairs across platforms were compared. Agilent versus 29K (unadjusted) classifications resulted in 203 concordant and 40 discordant subtype predictions (Cramer's V association coefficient = 0.768), and Agilent versus 29K (intraplatform adjusted) classifications resulted in 208 concordant and 35 discordant subtype predictions (Cramer's V association coefficient = 0.782). A high concordance rate was observed among tumors predicted as basal-like, whereas distinguishing luminal B from HER2-enriched and luminal A involves more disagreements. In a study by Waddell et al., they compared gene expression data from frozen breast tumor biopsies and FFPE tumor blocks. In 12 out of 15 cases, the FFPE/frozen sample pairs were concordant in relation to the molecular subtypes classified using the PAM50 classifier [For comparisons of molecular subtype classifications, we applied a commonly used classifier designed for single sample predictions, the 50-gene subtype classifier (PAM50) developed by Parker et al. . PAM50 cassifier . Our res in-house 29K array platform and Agilent SurePrint G3 microarray platform, we demonstrated the importance of detecting and tackling batch-related biases within datasets prior to data analysis. We have demonstrated how tools such as ComBat are suitable to successfully overcome systematic technical variations in order to unmask essential biological signals. Batch adjustment was found to be particularly valuable in the detection of more delicate differences in gene expression. Furthermore, the study shows that prober adjustment is essential for optimal integration of gene expression data obtained from multiple sources. The data also demonstrate that high-variance genes are highly reproducibly expressed across platforms, making them particularly well suited as biomarkers or for building gene signatures. Predictions of estrogen-receptor status and molecular breast cancer subtypes were found to be highly concordant. In conclusion, the study emphasizes the importance of utilizing proper batch adjustment methods to reduce systematically technical bias when analyzing and integrating data from different batches and microarray platforms and by selecting high-variance genes it is possible to enrich for highly reproducible genes.Using data from 243 breast cancer samples analyzed on two different microarray platforms, ourSupplementary Figure S1: shows the PAM50 classification and associated heatmap visualizing the PAM50 genes using the Agilent dataset.Supplementary Figure S2: shows the PAM50 classification and associated heatmap visualizing the PAM50 genes using the 29K (unadjusted) dataset.


RNA-Seq is gradually becoming the standard tool for transcriptomic expression studies in biological research. Although considerable progress has been recorded in the development of statistical algorithms for the detection of differentially expressed genes using RNA-Seq data, the list of detected genes can differ significantly between algorithms. We present a new method (PANDORA) that combines multiple algorithms toward a summarized result, more efficiently reflecting true experimental outcomes. This is achieved through the systematic combination of several analysis algorithms, by weighting their outcomes according to their performance with realistically simulated data sets generated from real data. Results supported by the analysis of both simulated and real data from different organisms as well as correlation with PolII occupancy demonstrate that PANDORA improves the detection of differential expression. It accomplishes this by optimizing the tradeoff between standard performance measurements, such as precision and sensitivity. One of the common applications of RNA-Seq  is genomP-value weighting scheme , a new method for the analysis of RNA-Seq gene expression data which deploys a combination of existing algorithms. We show that PANDORA, as evaluated on simulated and real data sets, achieves significant improvements in both precision and sensitivity, and, in the worst case, similar performance to existing algorithms that have been previously evaluated as most robust  returned by baySeq and the score (1 – q statistic) returned by NOISeq. Specifically, let ijp be the resulting P-value for gene i after the application of the statistical test j. Then, the combined P-value for gene i, is derived using one of the following approaches:metaseqR can combine the ip1, ip2, …, imp be the P-value scores returned for gene i after the application of m statistical tests. Let also ip(1), ip(2), …, ip(m) be the aforementioned P-values sorted in increasing order. Then, according to Simes’ method, the probabilityP-value for a set of m statistical tests  multiplied by –2:f follows an X2 distribution with 2m degrees of freedom, which can be used to derive the combined P-value from m statistical tests. However, Fisher's method was developed with the intention of combining P-values produced by the same statistical test applied in different data sets, where the underlying null hypothesis is the same , rather than combining P-values produced by different statistical tests applied to the same data set. In the context of different statistical tests, the null hypothesis remains essentially the same , although there might be differences in its formulation, according to the context of each statistical test.According to the Fisher's method, let e proved  that f fZ-method , which can be used to derive the P-value of the combined tests. Regarding the usage of the Whitlock method for combining P-values from different statistical tests, the assumptions described above (Fisher's method) also apply.According to Whitlock's weighted Z-method , the weiP-value isα, this approach is equivalent to the ‘intersection’ of the genes for which the null hypothesis of no differential expression is rejected (ijp < α), as these are derived from each statistical test. Thus, the final list will contain genes which have been characterized as differentially expressed by all the statistical tests applied. The maximum P-value ensures that the false positives are minimized at a  cost on the true positives .In this case, the combined P-value isα, this approach is equivalent to the ‘union’ of the genes for which the null hypothesis of no differential expression is rejected (ijp < α), as these are derived from each statistical test. Thus, the final list will contain genes which have been characterized as differentially expressed by at least one of the statistical tests applied. The minimum P-value ensures that the true positives are maximized at a  cost on the false positives (type I error).In this case, the combined P-value isjw represent automatically assigned or user-specific weights for the j statistical tests performed. The weights can be automatically assigned according to the area under the false discovery curve (AUFC) estimated from simulated data based (possibly) on the data set under investigation, or can be user-defined according to previous performance experience . In the case of automatic estimation of weights using the AUFCs, the weights are estimated using the following formula:jAUFC is the area under the false discoveries curve for the results of statistical test j. Apart from the automatic weights assignment, metaseqR allows the user to specify weights, based. for example. on previous own experience, on previous studies or manual inspection of the performance measurements offered by metaseqR. In all cases, the supplied weights must have a unit sum.In this case, the combined F1-score (or F-measure). It comprises a statistical measurement used to assess the performance of statistical tests or binary classifiers . The FDT is defined as:In order to quantify the tradeoff between precision and sensitivity when assessing methods, we use the ssifiers  by combiWe used false discovery curves (FDCs) and false negative curves (FNCs) to measure the progression of type I and type II errors, respectively, while traversing lists ranked according to statistical significance from top-to-bottom for FDCs and from bottom-to-top for FNCs.We used receiver operating characteristic (ROC) analysis (the area under the curve) and assessment of the true false discoveries to demonstrate that weighting and combining statistical tests according to their performance yields results as good as or better than the best performing algorithms in the majority of test cases.F1-score and the area under F1-score curves to demonstrate its maximization in final gene lists, produced by weighted P-value combination, in most simulated cases. The area under the F1-score curve is constructed by ranking the genes according to their combined P-value in increasing order and calculating the respective F1-score while traversing the ordered list.We used the F1-score analysis in real data from the SEQC project and true false discovery rate (FDR) analysis in a data subset from Brawand et al.We applied ROC and F1-score analysis in two RNA-Seq data sets , of which 50% are up-regulated in the first condition and 50% are up-regulated in the second.A simulated RNA-Seq data set with 10 000 genes, consisting of two experimental conditions, each with seven replicates. The percentage of DEGs was set to 12% , of which ∼42% are up-regulated in the first condition and ∼58% are up-regulated in the second, comprising a slightly unbalanced set of deregulated genes among the two conditions.Based on the above data sets and the implemented simulator, we used two main synthetic data configurations from each organism:a priori ground truth regarding the specific transcripts. Furthermore, the SEQC project includes a set of ∼1000 genes validated by TaqMan quantitative polymerase chain reaction (qPCR); thus the fold change of these genes as calculated by qPCR can be used to estimate true levels of differential expression and provide measurements regarding the performance of statistical algorithms and the combination methods of metaseqR. Details regarding the SEQC data set are discussed in detail elsewhere  by separating each group to two subgroups  with two replicates each. As both the groups contained five technical replicates, we excluded from each group the replicate which was clustered further from the other two in a sample-wise hierarchical clustering.Moreover, to assess the performance of metaseqR-supported algorithms and et al.  data sets: (i) RNA-Seq and PolII data from Mokry et al. , where a , Pan troglodytes (Chimpanzee), Mus musculus (Mouse), Drosophila melanogaster (Fruitfly) and Arabidopsis thaliana (Arabidopsis). We used a limited set of simulations and real data, which are sufficient to prove the added value of PANDORA. Specifically, we restricted the properties of synthetic data sets to two main configurations: a data set with two conditions of three biological replicates each and balanced differential expression between conditions  and a data set with two conditions of seven biological replicates each and slightly unbalanced differential expression between conditions .We evaluated the performance of PANDORA using experimental settings based on simulated data or data acquired from the SEQC study  and Brawet al. . The synet al. ,23 basedP-value weighting scheme . However, as performance is not only measured by power, but sensitivity as well, PANDORA offers an optimal tradeoff between precision and sensitivity, when multiple statistical tests are applied on the same data set and their resulting statistical scores are properly weighted. To demonstrate this, we used established measurements, such as ROC analysis and FDCs, as well as the harmonic mean of precision and sensitivity . To avoid confusion, we will use the term ‘P-value’ to refer to all the statistical scores , even if not all statistical scores comprise nominal P-values.As shown below, the combined usage of statistical tests for RNA-Seq data does not necessarily perform better in terms of statistical power than certain algorithms reported to stand out in related evaluations. With certain evaluation metrics, this is something to be expected because of the convex P-value combination method). Specifically, we applied the following methods:P-value combination method proposed by Simes  or mined from the related statistical literature and used in the appropriate context  wP-value combinations, applied on the synthetic data sets generated based on real data from five organisms. To avoid possible biases introduced by major (DESeq) or minor  differences in the normalization approaches followed by each algorithm, we used EDASeq  and the Simes method, all perform quite poorly, presenting a large number of false positives among the top genes, very similar to the Union method which is the most liberal, and thus expected to return a large number of false discoveries. PANDORA performs quite well but not better than limma voom because of the convex nature of the weights. Notably, the Intersection, which is supposed to be the strictest selection strategy, appears inferior to limma voom and sometimes baySeq, apart from the Fruitfly simulations. This observation suggests that the often intuitive use of the common DEGs, returned by a variety of tests, as a robust signature to distinguish among biological conditions and sometimes as a ‘ground truth’ for evaluation  returning nominal P-values . Finally, we examined the progression of Type II errors using FNCs for each algorithm and simulation configuration. In this case, the above performance evaluation is almost reversed and many of the best performing algorithms exhibit a large number of false negatives, with certain combination methods (PANDORA among them) showing quite stable results (Supplementary Results).Figure d EDASeq  as a comaluation , is not Next, we evaluated the performance of PANDORA as compared to the other methods using ROC analysis and assessment of the true FDR, inspired by previous comparison work . ROC anaP-value combination methods, Intersection performs best, as expected due to stringency of selection, while the Simes method performance is between the performances of individual tests. Remarkably, the Union method, which is the most liberal and expected to exhibit the highest number of true false discoveries, performs better than the Fisher and Whitlock methods. Finally, and most importantly, PANDORA exhibits true FDR values below 5%, which is better than limma voom, the best individual algorithm so far according to our study. All the aforementioned observations do not change when using each package's specific normalization algorithm instead of EDASeq .We also assessed the accuracy of individual as well as combined tests by measuring their true FDR using simulated data Figure . From thF1-score. Figure F1-score levels slightly higher than the best performing individual tests for Human, Chimpanzee and Arabidopsis. The F1-score levels are much higher in Mouse and Fruitfly simulations. In the ‘7 replicates—unbalanced DEG’ study, the PANDORA method markedly stands out for all five organisms. Moreover, the good performance of PANDORA is supported by the area under the F1-score curve , which depicts the evolution of the F1-score while traversing ranked gene lists from higher to lower statistical significance. Regarding the other combination methods, in the majority of cases they demonstrate lower F1-scores than PANDORA, but higher F1-scores than certain rather conservative individual tests . Specifically, among the other methods, Simes presents the highest F1-score, followed by Whitlock, Fisher, Union and Intersection. Although Whitlock performs quite well, if we break down the F1-score value to its components , we see that, although it takes into account the performance of each algorithm, the Whitlock method is poor because of the high number of false positives as compared to the gain in true positives. This suggest that the use of weighted Z-scores is not appropriate for combining statistical tests for RNA-Seq data and that PANDORA manages to accomplish this task in simulated data. Overall, our observations strengthen the main rationale behind PANDORA: as each statistical test presents certain advantages , a more robust gene list can be derived by applying an objective methodology to combine them. PANDORA achieves that by AUFC weighting, which ranks the statistical tests according to their ability to provide a reasonably low number of false positives in the top genes (500 in simulation studies). Finally, as expected, the Intersection method presents the lowest F1-score in all simulations across all organisms, being overly conservative and ignoring a lot of true hits returned by each statistical test .We evaluated the ability of individual as well as proposed combined tests to maximize the number of true positives, while at the same time keeping the number of false hits at a low level thus to maximize the F1-score values, while baySeq and NOISeq show consistently lower values in most cases. This suggests that although the latter two are both successful in a satisfactory top-to-bottom statistical ranking of genes, their final outcome is excessively strict, resulting in significant loss of true positives . Notably, baySeq presents an F1-score as low as the conservative strategy of constructing final gene lists by intersecting lists derived by individual tests. This is also supported by FNCs , where in the majority of the cases baySeq presents a higher area under the FNC as compared to the other algorithms, either combined or individual. The tradeoff between precision and sensitivity is also assessed using corrected P-values (Supplementary Results) and by measuring the ratio of true positives to false hits . Overall, the F1-score analysis reveals what more classical statistical algorithm performance evaluation measures, such as FDCs and ROC, do not: PANDORA successfully provides optimized gene lists, with respect to the tradeoff between true positives and false hits.From the individual tests, limma voom and edgeR demonstrate the highest et al.  its ability to identify true or false differential expression with RNA-Seq data generated in the latest versions of the MAQC study, namely, the SEQC project for the evaluation of sequencing platforms regarding gene expression and (ii) its ability to control false discoveries with mock comparisons using data from SEQC and four male chimpanzee prefrontal cortex samples from Brawand et al.  which haet al. . The SEQet al. , apart from the performance of DESeq regarding the F1-score, which drops even more for ERCC spike-in data.From the individual algorithms, DESeq and NBPSeq show the highest area under the curve, followed by edgeR, limma voom, baySeq and NOISeq when using the TaqMan data. When using ERCC spike-ins, DESeq is again top, followed by NOISeq, edgeR, limma voom and NBPSeq. Regarding the P-value combination methods using three ‘same versus same’ comparisons, two by separating the SEQC data groups into two subgroups each and one more using data from Brawand et al. , followed by PANDORA. The worst combination method is Fisher, followed by Whitlock, Union and Simes. Notably, PANDORA achieves one of the best rankings, presenting also several advantages, as described in previous sections. The aforementioned performance trend does not change when using each package's normalization algorithm instead of EDASeq (Supplementary Table S2).Finally, we evaluated the approximation of the true FDR of individual algorithms and l. Table , excludil. Table . All theF1-scores for each statistical algorithm and P-value combination method were calculated by using PolII occupancy as a ‘ground truth’ representation. To evaluate the tradeoff between the correlations of transcription changes, the above were deployed as a graphical evaluation method of each algorithm  show a larger F1-score but a much lower correlation with PolII occupancy. It is evident that PANDORA achieves an optimal tradeoff between precision-sensitivity (F1-score) as defined by PolII occupancy and correlation of gene expression as measured by two independent techniques. The optimization is clearer for the Lin et al. data set , it is most frequently applied to investigate changes in a ‘summarized gene’ expression across several experimental conditions. In addition, looking into more complex events, such as alternative isoform representation, often benefits from having established robust calls for deregulated genes. This is partly justified by a recent publication  where thP-values where appropriate. The issue of multiple testing correction and its application has been exhaustively treated in the literature. Its effect can be quite misleading with certain algorithms . Specifically, using simulated data sets for five organisms and two experimental configurations, the performance of DESeq, edgeR, NBPSeq and limma voom is not improved when using adjusted P-values, as measured by FDCs . On the other hand, when using adjusted P-values for these four statistical tests, the performance as measured by FNCs dramatically drops, with limma voom exhibiting the most moderate accuracy loss. We thus believe that such results/phenomena are usually not noted, as most efforts in statistical algorithms are directed toward the minimization of false positives—without regard to the false negatives. This leads to overly strict outcomes and restricted downstream analyses .Our results were described based on raw, rather than corrected for multiple testing, P-values which can be subjected to multiple testing correction  and, for these reasons, raw P-values have been used before for evaluation purposes  or more empirical ones (e.g. NOISeq), where ‘traditional’ multiple testing correction is not applied. We demonstrate that although these methods perform quite well as assessed by FDCs, FNCs and ROC analysis, when looking at actual numbers of true and false hits at specific statistical thresholds, NOISeq and baySeq perform badly: they generate an excessively high number of false hits, particularly false negatives. For the researcher who wants to look at a strict list of top DEGs, this is welcome. It, however, can also be achieved by imposing stricter statistical thresholds with other algorithms. In addition, not all methods return nominal purposes . Using PF1-score analysis, PANDORA is not the best performer when combining P-values adjusted for multiple testing correction. This, in our opinion is not a bottleneck, because, first, in a realistic experimental setting, the inclusion of more than three biological replicates is rare. Our analysis indicates that in the presence of a low number of replicates per experimental condition, and when using adjusted P-values for DESeq, edgeR, limma voom and NBPSeq: (i) the restriction of false positives is rather low as demonstrated by FDCs, (ii) the loss in numbers of false negatives is high as demonstrated by FNCs, (iii) ROC analysis does not show notable differences in performance and (iv) F1-scores slightly drop in most cases for the four tests, especially in the human and mouse cases, two organisms that dominate current biological studies. Second, in some cases a PANDORA type of combining tests may not be desirable. This would occur either because it does not perform well for a specific data set  or a specific statistical test known to work well for a biological system under investigation. In such cases, metaseqR still provides an easy interface to several normalization algorithms, statistical tests and quality control measures, combined with very rich and detailed reporting capabilities.As shown by In simulation studies, where the synthetic data sets are produced by strictly controlled computational means, PANDORA works considerably well. This may not always be the case with real data. The analysis of real RNA-Seq data from the SEQC project demonstrated a more moderate performance of the PANDORA method. At this point, it should be stressed that (i) we used Human weights estimated from the Montgomery–Pickrell combined data sets, which give greater weights to limma voom, edgeR and baySeq algorithms and (ii) the assessment is based on the TaqMan data and the ERCC spike-in controls, which both constitute a fixed set of ‘ground truth’ transcripts and the performance is based on these fixed numbers. In a real-life situation, there is no fixed number of DEGs that are expected from an experiment. This number is rather defined by statistical and/or empirical thresholds chosen by the analyst, without any guarantee that the false positives and false negatives are minimized. In this sense, methodology assessment with simulated data sets, where the number of differential synthetic genes is known but each test is free to return any number of genes, may well prove more crucial in the determination of which methods perform best.Furthermore, technical replication in the SEQC data set is suitable for assessing technology reproducibility. However, it fails to capture biological noise present in more realistic experimental settings, which can be partially caught by simulated data created with parameters estimated from real data. We thus advocate the use of PANDORA, which excels in the majority of test cases, as its usage can offer a very good starting point for downstream analyses. The gene lists with minimized false hits it generates should be appreciated especially in low replication data sets, such as clinical applications.F1-score). At the same time it achieves a good correlation with PolII occupancy. On the other hand, very liberal methods  fail to correlate well with PolII, even if they generate higher F1-scores. This is the expected outcome of the inclusion of more false positives. Stricter methods correlate better with PolII (again to be expected because of stringency), but fail to offer a good tradeoff between precision and sensitivity.The robustness of PANDORA is further supported when using PolII occupancy as a measurement of transcription. When the correlation of differential gene expression, as detected by RNA-seq, with differential average PolII occupancy across gene bodies was used as an independent accuracy measurement, we observed that PANDORA offers a good tradeoff between precision and sensitivity  package for the analysis of RNA-Seq gene expression data providing an interface for several normalization methods and statistical tests, methods for combining statistical tests as well as detailed and comprehensive reporting facilities.PANDORA is implemented in metaseqR, a Bioconductor (Supplementary Data are available at NAR Online.


Analysis of Hi-C data has shown that the genome can be divided into two compartments called A/B compartments. These compartments are cell-type specific and are associated with open and closed chromatin. We show that A/B compartments can reliably be estimated using epigenetic data from several different platforms: the Illumina 450 k DNA methylation microarray, DNase hypersensitivity sequencing, single-cell ATAC sequencing and single-cell whole-genome bisulfite sequencing. We do this by exploiting that the structure of long-range correlations differs between open and closed compartments. This work makes A/B compartment assignment readily available in a wide variety of cell types, including many human cancers.The online version of this article (doi:10.1186/s13059-015-0741-y) contains supplementary material, which is available to authorized users. Hi-C, a method for quantifying long-range physical interactions in the genome, was introduced by Lieberman-Aiden et al. , and it In Lieberman-Aiden et al. , it was The A/B compartments are estimated by an eigenvector analysis of the genome contact matrix after normalization by the observed–expected method . SpecifiOpen and closed chromatin can be defined in different ways using different assays such as DNase hypersensitivity or chromatin immunoprecipitation (ChIP) sequencing for various histone modifications. While Lieberman-Aiden et al.  establisIn this manuscript, we show that we can reliably estimate A/B compartments as defined using Hi-C data by using Illumina 450 k DNA methylation microarray data  as well As an application, we show how the somatic mutation rate in prostate adenocarcinoma (PRAD) is different between compartments and we show how the A/B compartments change between several human cancers; currently TCGA does not include assays measuring chromatin accessibility. Furthermore, our work reveals unappreciated aspects of the structure of long-range correlations in DNA methylation and DNase hypersensitivity data. Specifically, we observe that both DNA methylation and the DNase signal are highly correlated between distant loci, provided that the two loci are both in the closed compartment.% (ranges from 92.6 % to 96.0 %) on chromosome 14. These measures vary little between chromosomes; a full depiction is available in Additional file We obtained publicly available Hi-C data on Epstein–Barr virus (EBV)-transformed lymphoblastoid cell lines (LCLs) and fibroblast cell lines and estimated A/B compartments by an eigenvector analysis of the normalized Hi-C contact matrix “”. The coUsing high-resolution data does not change the estimated A/B compartments as seen in Additional file Figure ICE normalization removes any marginal dependence of the contact matrix on GC content by forcing the marginal sums of the contact matrix to be constant . DespiteIn the remainder of the manuscript, we use the most recent data, i.e. HiC-EBV-2014 and HiC-IMR90-2014, to represent eigenvectors and A/B compartments derived from Hi-C data in these cell types.To estimate A/B compartments using epigenetic data other than Hi-C, we first concentrate on DNA methylation data assayed using the Illumina 450 k microarray platform. Data from this platform are widely available across many different primary cell types. To compare with existing Hi-C maps, we obtained data from 288 EBV-transformed LCLs from the HapMap project .% on this chromosome. In this analysis, we implicitly assume that there is no variation in compartments between different individuals for the same cell type.DNA methylation is often described as related to active and inactive parts of the genome. Most established is high methylation in a genic promoter leading to silencing of the gene . As a fiSurprisingly, we found that we could improve considerably on this analysis by doing an eigenvector analysis of a suitably processed between-CpG correlation matrix Fig. . This ma% (14.2 % of the bins excluded). Excluding bins where either the 450 k-based prediction is close to zero or the Hi-C eigenvector is close to zero, we got an agreement of 93 % (24.8 % of the bins excluded).Closely examining differences between the 450 k-based predictions and the Hi-C-based estimates, we found that almost all disagreements between the two methods occur when an entry in one of the two eigenvectors is close to zero; in other words, where there is uncertainty about the compartment in either one of the two analyses. Excluding bins where the 450 k-based prediction is close to zero, that is bins that have an absolute eigenvector value less than 0.01, we got an agreement of 88.8 Our processing of the correlation matrix is as follows  do not decay with distance between bins ; see Fig. For the 450 k-fibroblast experiment, we had access to the raw IDAT files and therefore to the control probes located on the array. For this dataset, we examined if the striking global ranking between different samples using the open sea probes in the closed compartment could be explained by technical factors such as bisulfite conversion. To test this, we regressed the mean (and median) methylation levels against each of the following five variables: chip and well variables (surrogates for batch), Bisulfite I and Bisulfite II control probes and negative control probes (background noise). None of these variables was significantly associated with the mean of the median methylation levels  but different samples . If possible, we have preferred to process the data ourselves starting from the Illumina IDAT files. However, for several datasets, we had to use the original authors’ preprocessing pipeline; see “We examined the impact of preprocessing methods on the estimated eigenvectors by using functional normalization , quantilWe examined the binning resolution of our approach using data from the 450 k methylation array. As resolution increases, the number of bins with zero or few probes per bin increases. In Additional file We applied these methods to Illumina 450 k data on PRAD from TCGA. Quality control shows both normal and cancer samples to be of good quality. Since the normal prostate samples represent uncultured primary samples, we confirmed that this dataset has the same information in its long-range correlation structure as established above Fig. .Fig. 11We obtained a list of curated somatic mutations from TCGA and used them to compute simple estimates of the somatic mutation rate in each 100-kb bin of the genome . Since the list of somatic mutations was obtained using whole-exome sequencing, we identified the capture assay used in these experiments and used the capture regions from this specific assay to compute somatic mutation rates for each 100-kb genomic bin by computing the number of somatic mutations per base captured in that bin. Because the capture assay is biased towards coding regions, the somatic mutation rates we computed can roughly be interpreted as the somatic mutation rate in coding regions per genomic bin. Many genomic bins have a somatic mutation rate of zero, and the number of bases captured varies between bins. In Fig. While open and closed chromatin are cell-type specific, it is not surprising that a large percentage of the genome 74 %) is in the same compartment in both normal and cancer samples. To illustrate the added value of a cancer-specific map of open and closed chromatin, we focused on the somatic mutation rate of bins that change compartment between normal and cancer. These bins are displayed in color in Fig.  % is in Using the method we have developed in this manuscript, it is straightforward to estimate A/B compartments across a wide variety of human cancers using data from TCGA. Figure Lieberman-Aiden et al.  establisWe obtained DNase sequencing (seq) data on 70 samples  from EBVFor each sample, we computed the signal in each 100-kb genomic bin. To obtain the average DNase signal, we averaged the signal across samples. The resulting mean signal is skewed towards positive values in the open compartment, and we therefore centered the signal by the median. The median was chosen as this has the best compartment agreement with Hi-C data. Figure Inspired by the success of considering long-range correlations for the 450 k data, we examined whether this approach is useful for DNase data. We therefore computed the Pearson correlation matrix of the binned DNase signal; in contrast to the 450 k data, we did not bin the correlation matrix as the signal matrix was already binned. The first eigenvector of this correlation matrix is highly skewed; we centered it by its median. Figure We observed an association between GC content and average DNase signal  sequencing , which g161616Single-cell DNA methylation can be measured using a form of whole-genome bisulfite sequencing (WGBS) as described in Smallwood et al. . Due to 171717171717In this work, we show how to estimate A/B compartments using long-range correlations of epigenetic data. We have comprehensively evaluated the use of data from the Illumina 450 k DNA methylation microarray for this purpose; such data are widely available on many primary cell types. Using data from this platform, we can reliably estimate A/B compartments in different cell types, as well as changes between cell types.This result is possible because of the structure of long-range correlations in this type of data. Specifically, we found that correlations are high between two loci both in the closed compartment and low otherwise, and do not decay with distance between loci. This result only holds true for array probes measuring CpGs located more than 4 kb from CpG islands, so-called open sea probes. This high correlation is the consequence of a surprising ranking of DNA methylation in different samples across all regions belonging to the closed compartment. We have replicated this result in an independent experiment using the Illumina 27 k DNA methylation microarray.We have furthermore established that A/B compartments can be estimated using data from DNase hypersensitivity sequencing. This can be done in two ways: first by simply computing the average DNase signal in a genomic region, and second by considering long-range correlations in the data, like for 450 k array data. Again, we exploited the structure of long-range correlations in this type of epigenetic data and, as for DNA methylation data, we found that correlations between loci both in the closed compartment are high, whereas correlations between other loci are approximately uniformly distributed. Again, this correlation is caused by a ranking of the DNase signal in different samples across all regions belonging to the closed compartment. Surprisingly, our method works both for biological replicates (EBV-transformed lymphocytes) but also on technical between-lab replicates of the same cell line (IMR90).Finally, we have established that our method works on single-cell epigenetic data, including scATAC-seq and scWGBS. These experimental techniques are in their infancy; it is likely that additional data will allow us to tune aspects of our method to this type of data. Now, the correlation is between single cells as opposed to biological replicates of bulk cells. This potentially allows our method to be used on rare types of cells. During the review of this paper, Buenrostro et al.  appearedRecently, clusters of DNA methylation under genetic control (GeMes) have been described . These cOur approach is based on computing the first eigenvector of a (possibly binned) correlation matrix. It is well known that this eigenvector is equal to the first left-singular vector from the singular value decomposition of the data matrix. The right-singular vector of this matrix is in turn equal to the first eigenvector of the sample correlation matrix, also called the first principal component. This vector has been shown to carry fundamental information about batch effects . BecauseWe have examined the impact of GC content on our method. It has previously been established that GC content is associated with A/B compartments . This asThe reason our method works is a surprising, consistent ranking of different samples across all regions belonging to the closed compartment (and only the closed compartment). By comparison with additional 27 k methylation array experiments, we have shown that this ranking is not a technical artifact caused by (for example) hybridization conditions.We caution that while we have had success with our method on many datasets, we have seen failures as we described in our analysis of the dataset on whole blood measured on 450 k. This raises the issue of when and why the method fails. In recent work, we studied colon cancer and EBV transformation of lymphocytes using WGBS , 23. In 17The functional implications of A/B compartments have not been comprehensively described; we know they are associated with open and closed chromatin , replicaβ=M/(M+U+100) for estimating percentage methylation given (un)methylation intensities U and M. Traditionally, the term M value is used for the logit transform of the beta value, and we do the same.We use the standard formula % of the array) are probes located in CpG islands, shore probes (23.1 %) are probes within 2 kb of CpG islands, and shelf probes (9.7 %) are probes between 2 kb and 4 kb from CpG islands. Open sea probes (36.3 %) are the rest of the probes. We use the term CpG resort probes to refer to the union of island, shore and shelf probes; in other words non-open sea probes.With respect to CpG density, the 450 k array probes fall into four categories that are related to CpG islands. CpG island probes (30.9 Methylation data are given in Table The 450 k-fibroblast dataset The study contains 62 samples from primary skin fibroblasts from .p×n signal data matrix where p is the number of bins for the chromosome and n the number of samples. We defined the average DNase signal as the across-sample mean of the signal matrix. The DNase correlation matrix is the p×p Pearson correlation matrix of the signal matrix.For the DNase-EBV dataset from , we downFor GC content correction of the DNase data, we fitted a LOESS curve of the DNase signal against the bin GC content for each sample differently and regressed out the fitted relationship.y=x/3 line in the figure. Subsequently we discarded hypersensitive sites that had no reads in any of the cells and obtained 631 bins at a resolution of 100 kb on chromosome 14. Eigenvectors were computed and smoothed as described below.scATAC-seq data were obtained from GEO under the accession number GEO:GSE68103] described in ; see Tab103 descr16scWGBS data were obtained from GEO under the accession number GEO:GSE56879] described in ; see Tab79 descriTo obtain eigenvectors of the different matrices from Hi-C, DNA methylation and DNase data, we use the non-linear iterative partial least squares (NIPALS) algorithm implemented in the mixOmics package in R . Each eiWhen we compare eigenvectors from two different types of data, we only consider bins that exist in both data types; some bins are filtered out in a data-type-dependent manner, for example, because of the absence of probes or low coverage. This operation slightly reduces the number of bins we consider in each comparison.Because the sign of the eigenvector is arbitrarily defined, we use the following procedure to define a consistent sign across different chromosomes, datasets and data types. For Hi-C data and DNase data, we correlate the resulting eigenvector with the eigenvector from Lieberman-Aiden et al. , changinTo measure the similarity between two eigenvectors, we use two measures: correlation and compartment agreement. The correlation measure is the Pearson correlation between the smoothed eigenvectors. The compartment agreement is defined as the percentage of bins that have the same eigenvector sign, interpreted as the percentage of bins that belong to the same genome compartment (A or B) as predicted by the two eigenvectors. Occasionally, this agreement is restricted to bins with an absolute eigenvector value greater than 0.01 to discard uncertain bins.Because open chromatin regions have a very high DNase signal in comparison to closed chromatin regions, the DNase signal distribution is highly skewed to the right; therefore, we center both the average signal and the first eigenvector by subtracting their respective medians, before computing the correlation and agreement.We obtained a list of somatic mutations in PRAD from the TCGA data portal . SeveralSomatic mutation rates in each 100-kb genomic bin were computed as the number of mutations inside each bin, divided by the length of the capture regions inside the bin.Estimated compartments for TCGA cancer data are available in Additional file Software for performing the analysis of 450 k methylation arrays described in this manuscript have been added to the minfi package  version


The advent of next-generation RNA sequencing (RNA-seq) has greatly advanced transcriptomic studies, including system-wide identification and quantification of mRNA isoforms under various biological conditions. A number of computational methods have been developed to systematically identify mRNA isoforms in a high-throughput manner from RNA-seq data. However, a common drawback of these methods is that their identified mRNA isoforms contain a high percentage of false positives, especially for genes with complex splicing structures, e.g., many exons and exon junctions.We have developed a preselection method called “Non-negative Matrix Factorization Preselection” (NMFP) which is designed to improve the accuracy of computational methods in identifying mRNA isoforms from RNA-seq data. We demonstrated through simulation and real data studies that NMFP can effectively shrink the search space of isoform candidates and increase the accuracy of two mainstream computational methods, Cufflinks and SLIDE, in their identification of mRNA isoforms.NMFP is a useful tool to preselect mRNA isoform candidates for downstream isoform discovery methods. It can greatly reduce the number of isoform candidates while maintaining a good coverage of unknown true isoforms. Adding NMFP as an upstream step, computational methods are expected to achieve better accuracy in identifying mRNA isoforms from RNA-seq data.The online version of this article (doi:10.1186/s12864-015-2304-8) contains supplementary material, which is available to authorized users. In molecular biology and transcriptomics, an important question is to understand the alternative splicing process, a key step of gene transcription in diverse species, from invertebrates to mammals. Alternative splicing generates at least two mRNA isoforms  for one gene, and these mRNA isoforms as well as their protein products can play different roles in various biological phenomena. For example, studies have found that aberrant structures or abundance of mRNA isoforms can cause various human diseases , 2. HencMore than a decade ago, microarray technologies established a high-throughput platform for identifying and quantifying mRNA isoforms of genes with known sequences. SPACE , 4 is a de novo methods,” because their isoform identification approach can completely rely on RNA-seq reads without using existing annotations. Another reference-based method SLIDE uses a different approach. It constructs a search space of possible isoforms by using known boundaries of genes and exons from existing annotations, with the possible addition of novel genes and exons inferred from RNA-seq data. SLIDE then identifies isoforms from the search space based on a regularized linear model, which accounts for the RNA-seq read generating mechanism and read distributions. Because of its use of annotations, SLIDE is also called an “annotation aided method”, in contrast to the previous “de novo methods.” Note that the recent versions of Cufflinks also added an annotation aided option, which can simulate RNA-seq reads from annotations and use the simulated reads together with RNA-seq data for isoform discovery , representing the biases of read counts in different bins. Such biases are attributable to the features of the bins, such as GC contents and the bins’ positions in transcripts. G is a p×s indicator matrix with binary values to describe the existence of bins in isoforms. Each column of G represents an isoform.where V, our goal is to estimate G, whose columns will be regarded as isoform candidates. To achieve this goal, we have to first estimate W and A. We adopt the SPACE approach to estimate G. In the SPACE method [A was estimated as With the normalized read count matrix E method , given ai-th bin, is considered as i-th bin in all isoforms.In other words, G is estimated as With I(·) is an indicator function, and c∈IR is a threshold to make c=0.4 in this paper.where W by decomposing V using NMF. There exist many available NMF algorithms [V, one constraint we need is the compatibility of bins in every column of Prior to that, we estimate gorithms , 20–22, To realize this constraint in NMF, we add a penalty term to the standard NMF objective function as follows p is the number of bins, m is the number of samples, and α is a regularization parameter controlling the weight of the penalty term . i⊥j denotes that Bin i and Bin j are conflicting  subject to a pre-specified s=rank(W), which is equivalent to the number of columns of W. How to determine the rank remains an open question. In NMFP, we choose the rank by gap statistic, which was originally designed to select the number of clusters in k-means clustering [where ustering   and aggregated the identified isoform candidates into a candidate pool. In the filtering step, we first remove the isoform candidates, which contain subexon junctions not supported by reads, from the pool. Then among the rest of candidates in the pool, we only retain the high-frequency ones, which are selected in at least r NMF runs , as final isoform candidates.To address the issue of non-unique NMF results, we use a high-frequency filtering approach to find isoform candidates identified in a number of NMF runs. Specifically, we implement the NMF estimation procedure described in the previous two subsections for The NMFP method is illustrated in Fig. We conduct simulations and real-data analyses in many scenarios to demonstrate the effectiveness of NMFP in improving the performance of downstream isoform discovery methods. The following results show that NMFP can help Cufflinks and SLIDE achieve better isoform discovery accuracy. We also study the robustness of NMFP to the choice of NMF rank, the number of input samples in the read count matrix, and the gene expression levels.D. melanogaster (with reference genome dm6 and Ensembl annotation BDGP6 of release 80). We focus on 4125 genes on chr3R, in accordance with the SLIDE paper [% (2132) genes contain 3 to 10 subexons (Table % (951) have more than one isoforms in the annotation. We apply NMFP to these 951 genes.We use Flux Simulator  to simulDE paper , which sn−1 possible isoforms are enumerated for an n-subexon gene, and SLIDE uses a simple preselection to filter out the isoforms with subexons or subexon junctions not supported by reads. NMFP + SLIDE replaces this preselection step by NMFP.To check the effects of NMFP on improving the isoform discovery performance of SLIDE, we define a new procedure called “NMFP + SLIDE”, which combines NMFP and SLIDE by inputting the isoform candidates found by NMFP into SLIDE. In the original SLIDE method, 2Similarly, we define “NMFP + Cufflinks”, which inputs the isoform candidates found by NMFP as a GTF file into Cufflinks and uses Cufflinks with the “–GTF” option to estimate the abundance of these candidates. The candidates with non-zero estimated abundance are kept as the isoforms identified by NMFP + Cufflinks. This is different from the original Cufflinks, because it does not rely on a connectivity graph or the parsimony assumption as Cufflinks does.λ in the Lasso regression (λ=0.2 for fewer isoforms and λ=0.01 for more isoforms), we include two versions of SLIDE in this comparison: “SLIDE(fewer)” for λ=0.2 and “SLIDE(more)” for λ=0.01. This results in a comparison of seven methods in total: CufflinksNMFP + CufflinksSLIDE(fewer)NMFP + SLIDE(fewer)SLIDE(more)NMFP + SLIDE(more)NMFPTo compare SLIDE with NMFP + SLIDE, and Cufflinks with NMFP + Cufflinks, we evaluate their isoform discovery performance at three different levels: the nucleotide, exon and transcript levels, similar to the RGASP evaluation . In thisk identified isoforms and r annotated isoforms and k>r, only the top r identified isoforms would be paired with an annotated isoform, and the rest k−r identified isoforms would be counted as false positives with zero precision rates. For each identified isoform paired with an annotated isoform, we calculate its precision rate as We first compare the seven methods at the nucleotide level, that is, how many nucleotides are shared by the identified isoforms of each method and the true isoforms in the annotation. For each identified isoform, we match it to the annotated isoform that shares the most nucleotides with it. We rank the identified isoforms by their percentages of overlapping nucleotides with their matched annotated isoforms, from the highest percentage to the lowest. If there are Similarly, we match each annotated isoform with an identified isoform. We calculate the recall rate of each annotated isoform with a paired identified isoform as We calculate the precision and recall rates of every method in each of the 50 samples by taking the average of 951 genes. In the results, all methods except NMFP have precision rates above 0.76 in all samples. Although NMFP has low precision rates, its recall rates are as high as 0.96, implying that its identified isoform candidates has a good coverage of annotated isoforms Fig. . NMFP + In terms of F scores, which combine precision and recall rates, NMFP is shown to have greatly improved the performance of Cufflinks, by increasing the F scores of approximately 0.6 for Cufflinks to more than 0.8 for NMFP + Cufflinks. Strikingly, with the addition of NMFP, Cufflinks outperforms SLIDE, which originally had better F scores than Cufflinks without NMFP . NMFP + % of an annotated exon, we call the two exons “overlapping”. Similar to our matching scheme at the nucleotide level, we match identified isoforms with annotated isoforms at the exon level. Then for each identified isoform, its precision rate at the exon level is defined as We next evaluate the performance of these seven methods at the exon level. If an exon in an identified isoform overlaps at least 50 Similar to the performance at nucleotide level, NMFP significantly improves the F scores of Cufflinks from approximately 0.6 to over 0.8, yet it does not improve the performance SLIDE Fig. . The reaCompared to the previous evaluation at nucleotide and exon levels, the evaluation at the transcript level is more important, as the ultimate goal of isoform discovery methods is to correctly identify full-length mRNA isoforms. At the transcript level, if an identified isoform and an annotated isoform have every exon matched by the criterion we defined for matching at the exon level, we called these two isoforms “matched”. Then for every gene, its precision and recall rates at the transcript level are defined as In the results, NMFP has high recall rates greater than 0.85, indicating that the isoform candidates found by NMFP have a good coverage of annotated transcripts Fig. . MeanwhiD. melanogaster gene FBgn0039955 as an example and apply NMFP with different NMF ranks to the simulated RNA-seq data. This gene has 4 annotated isoforms (in Ensembl BDGP6 of release 80) and 10 subexons. In the result  (reference genome mm10 and annotation GRCm38 of release 81), a mammalian model organism with more complex gene structures than D. melanogaster. We generate 100 RNA-seq samples from mouse chr1 using Flux Simulator, with paired-end reads of length 2×76 bp. The 100 samples have 10 different numbers of RNA molecules and 10 different numbers of reads. The numbers of RNA molecules range from 4,200,000 to 6,000,000, increasing in steps of 200,000. The number of reads range from 11,000,000 to 20,000,000, increasing in steps of 1,000,000. More detail on the 100 samples is summarized in Section 3.2 and Table S1 of the Additional file D. melanogaster.We also evaluate the performance of NMFP in λ in the Lasso estimation. We conduct case studies on three mouse genes, for which we apply NMFP + SLIDE and SLIDE with 150 λ values, ranging from 0.001 to 0.15 by 0.001. The results show that NMFP + SLIDE has better isoform discovery accuracy than SLIDE for smaller λ values  and vary the gene expression levels in the other nine samples from 0.5, 1, 2 to 10 RPKM in 11 different settings. In each setting, we calculate a read count matrix U from the gene expression levels, the isoform proportions, and the uniformity assumption. Then we add white Gaussian noise N to U, and normalize the resulting matrix as V, for which we apply NMFP. Figure We use another simulation study to demonstrate that NMFP is capable of increasing the isoform discovery accuracy for a lowly expressed gene in a sample by leveraging other samples Fig. . In thisOnce the gene expression level goes above 3 RPKM, the number of needed candidates decreases to approximately 30 and has a reduced variance. This result demonstrates that for a gene with low expression in a sample, NMFP can take advantage of other samples to help reduce the number of isoform candidates of this gene. This property makes it possible for downstream isoform discovery methods to explore isoform structures of lowly expressed genes in a sample given the availability of other samples where the genes are more highly expressed.D. melanogaster (reference genome dm5 and annotation BDGP5 of release 66). Description of the 74 data sets is in Section 5 of the Additional file D. melanogaster pupae 3 days) and the isoforms identified by the seven methods.We further demonstrate the performance of NMFP and compare the seven methods on 74 real RNA-seq data sets of D. melanogaster gene FBgn0037643, which has 11 subexons and 6 annotated isoforms, NMFP finds 20 isoform candidates, which contain 4 annotated isoforms and miss the other two by a very short subexon (of only 3 bp). Cufflinks only identifies one annotated isoform, which cannot explain the observed read distribution in the last subexon. In contrast, NMFP + Cufflinks successfully identifies four annotated isoforms, which successfully capture the last subexon. NMFP also helps SLIDE discover full-length annotated isoforms instead of isofrom fragments  by adding a penalty term to NMF and decomposing the invalid isoform candidates with conflicting bins into biologically valid isoforms.NMF is inherently capable of providing interpretable results for the problem of mRNA isoform discovery. From an input matrix of RNA-seq read counts, NMF can find a sparse structure to indicate which bins are likely to form one isoform. However, NMF alone is not competent for recovering isoforms from RNA-seq data because of several reasons: (i) inaccurate read coverages and missing junction reads due to RNA-seq biases and mapping difficulties, (ii) non-uniqueness of NMF solutions leading to ambiguous isoform discovered, (iii) conflicting bins co-existing in discovered isoforms making the results biologically invalid, and (iv) difficulty in determining the NMF rank. Those issues can be largely alleviated or resolved when NMF is used as a method to pre-select isoform candidates rather than determining final isoforms. For (i), (ii) and (iv), since each non-unique NMF solution provides isoform candidates that may include some true isoforms among many false positives, aggregating multiple NMF solutions to find the high-frequency isoforms will increase the chance of discovering the true isoforms. We demonstrate in the simulation study of We propose NMFP as a preselection method for improving the accuracy of downstream isoform discovery methods. NMFP takes input from annotated genes and exon boundaries (e.g. UCSC and Ensemble annotations) and/or de novo assemblies (e.g. Cufflinks output). It outputs a small pool of isoform candidates, which are expected to well cover true isoforms and can form a reduced search space for downstream methods. From a general perspective, combination of methods with opposing advantages and drawbacks may improve over individual methods. NMFP and SLIDE are both built on the idea of selecting isoforms from all possibilities, while Cufflinks assembles transcripts directly via an overlap graph. A possible reason for NMFP + Cufflinks to become the top performer is that it combines the strengths of two different approaches: selection and assembly. This may also explain why NMFP + SLIDE, which combines two selection based methods, is outperformed by NMFP + Cufflinks, though SLIDE alone outperforms Cufflinks without NMFP. However compared to SLIDE itself, NMFP + SLIDE still has improved performance at the transcript level, and even becomes more robust to the regularization parameter choices. Overall, we demonstrate by simulation and real data studies that NMFP is a useful tool for improving isoform discovery accuracy and robustness in many aspects.Nevertheless, NMFP still encounters several problems and there remains substantial room for improvement. The noise and bias generated during RNA-seq still has large influence on NMFP results. Some methods for denosing and debiasing may help the performance of NMFP , 27. In