Transcriptome sequencing (RNA-Seq) has become the assay of choice for high-throughput studies of gene expression. However, as is the case with microarrays, major technology-related artifacts and biases affect the resulting expression measures. Normalization is therefore essential to ensure accurate inference of expression levels and subsequent analyses thereof.p-value distributions for tests of differential expression. The exploratory data analysis and normalization methods proposed in this article are implemented in the open-source Bioconductor R package EDASeq.We focus on biases related to GC-content and demonstrate the existence of strong sample-specific GC-content effects on RNA-Seq read counts, which can substantially bias differential expression analysis. We propose three simple within-lane gene-level GC-content normalization approaches and assess their performance on two different RNA-Seq datasets, involving different species and experimental designs. Our methods are compared to state-of-the-art normalization procedures in terms of bias and mean squared error for expression fold-change estimation and in terms of Type I error and Our within-lane normalization procedures, followed by between-lane normalization, reduce GC-content bias and lead to more accurate estimates of expression fold-changes and tests of differential expression. Such results are crucial for the biological interpretation of RNA-Seq experiments, where downstream analyses can be sensitive to the supplied lists of genes. RNA-Seq , under the accession number SRA048710.The Yeast data are available in the NCBI’s Sequence Read Archive (SRA) [et al. [Illumina's Genome Analyzer II high-throughput sequencing system was used to sequence RNA for two types of biological samples from the MicroArray Quality Control (MAQC) Project : Ambion'et al. .In dataset MAQC-2, Brain and UHR RNA were sequenced each using a single library preparation and seven lanes distributed across two flow-cells . There is no biological replication, but various types of technical replication . Library preparation effects are confounded with the extreme differential expression one expects when comparing such different samples as Brain and UHR. Nonetheless, the availability of qRT-PCR measures for a subset of circa 1,000 genes makes this a valuable benchmarking dataset.In dataset MAQC-3, four different library preparations of UHR RNA were each sequenced using three or four lanes from only one of two flow-cells. There is again no biological replication, but one can use this dataset for examining technical effects such as library preparation and lane effects. However, library preparations are nested within flow-cells, so that differences between flow-cells are confounded with library preparation effects.union-intersection (UI) gene model of [For both the MAQC-2 and MAQC-3 datasets, reads were mapped to the genome (GRCh37 assembly) using Bowtie , with unIn the original MAQC paper , 997 genp-values for UHR vs. Brain differential expression using the limma package [lmFit and eBayes pipeline. There are 11,081 genes detected by RNA-Seq and present on the Affymetrix chip. The MAQC data are available in the Sequence Read Archive, under the accession number SRA010153.Moreover, as reported in , a numbe package , with thregression procedure; the global-scaling and full-quantile normalization methods involve stratifying genes in equally-sized bins  based on GC-content and then "matching" parameters of the count distributions across bins.We propose three novel within-lane normalization approaches to account for the dependence of read counts on GC-content. The first method is based on the simple idea of regressing gene-level counts on GC-content and is implemented using the loess robust local We choose to normalize the logarithms of the gene-level counts for at least two reasons. Firstly, the logarithm is the canonical link for the Poisson  distribution, hence it seems natural to work on the log-scale when considering regression for count data. Moreover, regression on the log-scale is more robust to the presence of outliers  that can bias the fit.j yand j xdenote, respectively, the logarithm of the read count and the GC-content  for gene j = 1, ..., J.In what follows, let j yare regressed on GC-content j xusing the loess robust local regression method [Gene-level read counts  n method  and normT a summary statistic such as the median.where K equally-sized bins based on GC-content. The normalized expression measures are defined asGenes are stratified into k(j) denotes the GC-content stratum to which gene j belongs and T denotes a summary statistic, e.g., median, upper-quartile, or count for control genes. For instance, on the original (unlogged) scale, the normalized count for a particular gene could be its raw count divided by the ratio of the median count in its GC-bin to the overall median count of all genes.where et al. [et al. [In full-quantile (FQ) normalization, genes are stratified according to GC-content as for global-scaling normalization. The quantiles of the read count distributions are then matched between GC-bins, by sorting counts within bins and then taking the median of quantiles across bins. This approach is analogous to the microarray between-chip normalization of Irizarry et al.  and the  [et al. .within a lane. However, other technical effects, such as between-lane differences in sequencing depth, can strongly bias differential expression results. We therefore apply a between-lane normalization procedure, as in Bullard et al. [within-lane normalization and before differential expression analysis.GC-content normalization is designed to reduce the dependence of gene-level read counts on sequence composition d et al. , after wet al. [et al. [et al. [Between-lane normalization methods inherently make count distributions more similar between lanes, at the risk of dampening down true differential expression. Full-quantile normalization is the most aggressive of the methods we have proposed and both FQ and total-count (cf. RPKM) normalization force equal library sizes  across lanes. For the Yeast dataset, all four between-lane normalization methods  appear to yield similar results (data not shown). Since the CQN approach of Hansen et al.  involves [et al. . A thoro [et al. .withinLaneNormalization and betweenLaneNormalization, respectively, of the EDASeq package. For GC-content normalization, the withinLaneNormalization function takes as input a genes-by-lanes table of counts and a vector of gene GC-content values and returns a genes-by-lanes table of normalized counts, on the original unlogged scale and rounded to the nearest integer. There is also the option to output a table of normalization offsets, equal to the difference between the normalized and unnormalized counts. The normalized counts (with offset set to zero) or the unnormalized counts and corresponding offsets can then be supplied to standard R packages for differential expression analysis, such as DESeq [The above within-lane and between-lane normalization procedures are implemented in the functions as DESeq  or edgeRas DESeq . DetailsDifferential expression (DE) analysis is performed using likelihood ratio tests (LRT) based on a negative binomial model for gene-level read counts [over-dispersion by modeling the variance as a quadratic function of the mean μ, V(μ) = μ + ϕμ2, with dispersion parameter ϕ. For ϕ = 0, one recovers the Poisson distribution.d counts ,33. The We use the Bioconductor R package edgeR  to fit aet al. [While Bullard et al.  found thp-value distributions for likelihood ratio tests of DE based on a negative binomial model for gene-level read counts [Our aim is to evaluate GC-content normalization approaches in terms of their impact on differential expression results. To achieve this, we consider bias and mean squared error in expression fold-change estimation. We also compare normalization methods in terms of their Type I error rates and d counts .K = 10 GC-content bins for the MAQC datasets and K = 50 bins for the Yeast dataset, to reflect the strength of the GC-content effect for each dataset.The global-scaling and full-quantile within-lane normalization approaches were implemented using Expression log-fold-changes are estimated by log-ratios of average normalized read counts between two sets of lanes corresponding to the two conditions of interest.bias and mean squared error (MSE), one needs to know the true value of the expression fold-change. For the MAQC-2 dataset, one can use the estimate of the UHR/Brain fold-change from qRT-PCR as the true value, since qRT-PCR is often considered as a gold standard for producing accurate estimates of expression levels. The RNA-Seq estimated fold-change is the ratio of the average of the normalized counts for the seven UHR lanes to the average of the normalized counts for the seven Brain lanes. For a given gene, bias is then estimated as the difference between the estimated log-fold-changes from the two technologies.In order to compute For the Yeast dataset, we consider only the eight YPD lanes Table , for whiTo evaluate the impact of normalization on differential expression results, we use the edgeR package  to perfoType I error by considering again all 35 YPD null pseudo-datasets and by testing for DE between each of the corresponding two groups. Such a setting is intended to mimic the null hypothesis of no DE and any gene called DE yields a false positive. For a given pseudo-dataset and nominal Type I error rate α, the actual Type I error rate is defined as the proportion of genes with unadjusted p-values not exceeding α.For the Yeast dataset, we assess p-value distributions and numbers of genes declared DE between different normalization procedures.It is not possible to assess power with the Yeast dataset, as one cannot identify with certainty the set of all genes expected to be DE between growth conditions. Nonetheless, we perform gene-level tests of DE between the three growth conditions using edgeR and compare p-value distributions and numbers of genes declared DE for the two technologies.For the MAQC samples, we compare UHR vs. Brain differential expression results based on Illumina RNA-Seq and Affymetrix chip data. For RNA-Seq, we perform tests of DE between the seven Brain and seven UHR lanes using edgeR. Tests of DE between the five Brain and five UHR chips are performed using limma. We then examine et al. [et al. [sample-specific, meaning that the dependence of gene counts on GC-content may vary between lanes. For the Yeast dataset, Figures unimodal, in the sense that read counts first increase, then decrease with GC-content. Likewise, for the MAQC datasets, Figure S5 illustrates that the GC-content effect varies between different library preparations, but not within library preparations, although the effect is weaker than for the Yeast dataset. These observations suggest that the GC-content bias is likely to be introduced at the library preparation step , likely due to culture/library preparation effects. There is a clear bias for unnormalized counts, with log-fold-change estimates as high as 2. The full-quantile GC-content normalization method seems to be the most coherent in estimating the log-fold-change around zero.To assess the impact of normalization on DE tests, we first consider Type I error rates. For the Yeast data, the 35 YPD pseudo-datasets simulate an experiment in which all genes satisfy the null hypothesis of constant expression and hence any gene called DE is considered a false positive.Figure S12 displays, for each of the 35 pseudo-datasets, the difference between the actual Type I error rate  and the nominal Type I error rate for the negative binomial LRT implemented in the edgeR package . The figFigure The 35 pseudo-datasets do not actually fully mimic the null hypothesis of no differential expression, due to culture/library preparation effects. Indeed, regardless of the normalization approach, and as expected from biology, the eight YPD lanes cluster according to culture. Only with median normalization does the clustering first reflect library preparation protocol . After verification, it turns out that the top curves in Figure p-values on GC-content for all but the full-quantile GC-content normalization method. Figure p-value cut-off of 10-5). Similar results are observed with unadjusted p-value cut-offs of 0.01 and 0.001 (data not shown).To evaluate normalization methods in a biologically meaningful context, we consider the full Yeast dataset  and perform gene-level LRT of growth condition effects on gene expression using edgeR. The stratified boxplots in Figure S15 reveal a clear dependence of p-value distributions when testing for DE between Brain and UHR using both RNA-Seq and Affymetrix chip data. Figure technology-specific. Indeed, for microarrays, p-values do not depend on GC-content. By contrast, with only between-lane normalization, RNA-Seq p-values tend to decrease with GC-content. Full-quantile within-lane GC-content normalization removes this dependence.For the MAQC-2 dataset, we examine number of GC-content bins. This parameter is analogous to the span in loess robust local regression, thus the same considerations of bias/variance trade-off should guide its selection. Intuitively, the larger the number of bins, the more adaptive and possibly noisy the normalization. The boxplots of bias in Figure S16 indicate that DE results are robust to the number of GC-content bins in FQ normalization. We used K = 10 bins for the MAQC datasets and K = 50 for the Yeast dataset; a selection which reflects the stronger GC-content effect observed for the latter dataset.The main tuning parameter in our proposed global-scaling and full-quantile GC-content normalization procedures is the et al. [We have compared differential expression results based on our three proposed within-lane GC-content normalization methods and the CQN method of Hansen et al. , on the p-values on GC-content is technology-specific, i.e., unlike RNA-Seq p-values, microarray p-values do not depend on GC-content. Full-quantile within-lane normalization reduces the dependence of p-values on GC-content. Interestingly, and encouragingly, the much smaller p-values for the RNA-Seq data suggest that this newer assay is more powerful than microarrays for DE analysis .To rule out a biological reason for the dependence of DE on GC-content, we compared UHR vs. Brain DE results based on the MAQC-2 RNA-Seq data to those based on Affymetrix chip data . Figure Another well-known selection bias in RNA-Seq is due to gene length ,14. For In addition to their good performance noted above, our proposed normalization methods offer a number of advantages. They are very simple to implement and extend and lead to DE results that are robust to tuning parameters such as the number of GC-content bins . They could be applied to other genomic regions , either "from scratch" or by retaining the scaling from a previous gene-level normalization. They can easily be adapted to incorporate other sequence features such as gene length and mappability. Note, however, that in the process of adjusting for GC-content one may already be adjusting indirectly for other covariates such as length. Controls  could also be included.offset for normalization purposes in the statistical model for read counts [et al. [Our normalization procedures return genes-by-lanes tables of normalized counts, on the original unlogged scale and rounded to the nearest integer. Some authors have argued that it is better to leave the count data unchanged to preserve their sampling properties and instead use an d counts ,22. It i [et al. . The EDAet al. [We identified differentially expressed genes using a likelihood ratio test based on a negative binomial model for read counts. For the MAQC datasets, Bullard et al.  found thproxy for sample size, in a similar manner as length, and relates to power: as GC-content increases, read counts first increase then decrease, and evidence in favor of DE increases. If the effect was not sample-specific and simply a proxy for sample size, one would expect no dependence of expression fold-changes on GC-content and the effect on p-values to be due to the dependence of the variance on GC-content . One could therefore argue that it is not justified after all to normalize for GC-content and, in particular, that FQ normalization is too aggressive. Indeed, as seen in Figure p-values, i.e., p-values that explicitly account for sample size [p-value n pbased on a sample size of n to sample size 100 is n, one could use gene length or GC-content. The second and more insidious effect, however, is sample-specific and hence biases fold-changes and the resulting DE statistics . In particular, the standardized p-value approach does not address the sample-specificity (and complexity) of the GC-content effect and would still lead to biased DE results. Likewise for methods that correct for the GC-content bias after performing DE tests, e.g., in a fashion similar to that proposed in Young et al. [p-values based on microarray data do not vary with GC-content and hence suggests that the GC-content effect is a technology-related artifact. Of the normalization procedures we considered, full-quantile normalization seems most effective at removing the dependence of DE results on GC-content. However, results may vary in a dataset-specific manner and less aggressive approaches, such as loess or median normalization, may be robust alternatives. In the absence of controls, we recommend a thorough exploration of the data before choosing an appropriate normalization. In summary, there is a trade-off between bias removal and power: without within-lane GC-content normalization, fold-changes are biased, however normalization may mask true DE.There are two different types of GC-content effects. The first effect is to act as a ple size . A rule-g et al.  for geneS. bayanus and S. cerevisiae [GC-content bias is even more of an issue when comparing read counts between species, e.g., allele-specific expression in diploid hybrid of revisiae . We are It would also be interesting to consider adaptations of our methods to other sequencing assays, such as ChIP-Seq and DNA-Seq.controls  are essential for conclusive validation and comparison of any inference method, e.g., in terms of bias, variance, Type I error, and power. Controls could also be incorporated as "anchors" within the normalization procedure itself [et al. [Finally, as with microarrays, positive and negative e itself . The use [et al.  is an enet al. [et al. [p-value distributions for tests of differential expression. Our proposed within-lane procedures, followed by between-lane normalization as in Bullard et al. [We have reported the existence of strong sample-specific GC-content effects on RNA-Seq read counts, which can substantially bias differential expression analysis, and have proposed three simple within-lane gene-level GC-content normalization approaches. The GC-content effect seems to be the same for lanes assaying the same library preparation, but tends to vary between library preparations for the same type of biological sample. Hence, the bias is likely to be introduced at the library preparation step , on botd et al. , reduce The normalization methods proposed in this article are implemented in the open-source Bioconductor R package EDASeq. The resulting normalized counts  can then be supplied seamlessly to other R packages for differential expression analysis, such as DESeq  or edgeRhttp://www.bioconductor.org. This package, for exploratory data analysis and normalization for RNA-Seq, implements a variety of numerical and graphical summaries of read data, within-lane normalization procedures to adjust for GC-content or other gene-level effects, and between-lane normalization procedures to adjust for distributional differences between lanes.The methods proposed in this article are implemented in the R package EDASeq, released as part of the Bioconductor Project DR co-developed the statistical methods, wrote the EDASeq R package, performed data analysis, and drafted the manuscript. KS designed and performed the yeast experiments and prepared the DNA sequencing libraries. GS designed the yeast experiments and edited the manuscript. SD co-developed the statistical methods, performed data analysis, drafted the manuscript, and designed and coordinated the research. All authors read and approved the final manuscript.Supplementary Figures. Additional figures referred to in the main article as Figures S1-S18.Click here for file


Many missing-value (MV) imputation methods have been developed for microarray data, but only a few studies have investigated the relationship between MV imputation and classification accuracy. Furthermore, these studies are problematic in fundamental steps such as MV generation and classifier error estimation. In this work, we carry out a model-based study that addresses some of the issues in previous studies. Six popular imputation algorithms, two feature selection methods, and three classification rules are considered. The results suggest that it is beneficial to apply MV imputation when the noise level is high, variance is small, or gene-cluster correlation is strong, under small to moderate MV rates. In these cases, if data quality metrics are available, then it may be helpful to consider the data point with poor quality as missing and apply one of the most robust imputation algorithms to estimate the true signal based on the available high-quality data points. However, at large MV rates, we conclude that imputation methods are not recommended. Regarding the MV rate, our results indicate the presence of a peaking phenomenon: performance of imputation methods actually improves initially as the MV rate increases, but after an optimum point, performance quickly deteriorates with increasing MV rates. Microarray data frequently contain missing values (MVs) because imperfections in data preparation steps  create erroneous and low-quality values, which are usually discarded and referred to as missing. It is common for gene expression data to contain at least 5% MVs and, in many public accessible datasets, more than 60% of the genes have MVs . MicroarIn most of the studies about MV imputation, the performance of various imputation algorithms is compared in terms of the normalized root mean squared error (NRMSE) , which mWhile much attention has been paid to the imputation accuracy measured by the NRMSE, a few studies have examined the effect of imputation on high-level analyses , which demand that the dataset be complete. For example, the effect of imputation on the selection of differentially expressed genes is examined in , 11, 12 Wang et al. study the effects of MVs and their imputation on classification performance and report no significant difference in the classification accuracy results when KNNimpute, BPCA, or LLS are applied . Five daBy studying two real cancer datasets , Shi et al. report that the gaps between different imputation methods in terms of classification accuracy increase as the MV rate increases . They teA critical problem within both aforementioned studies is that all training data and test data are imputed together before classifier design and cross-validation is adopted for the classification process. The test data influences the training data in the imputation stage and the influence is passed to the classifier design stage. Therefore, the test data are involved in the classification design process, which violates the principle of cross-validation.In this paper, we carry out a model-based analysis to investigate how different properties of a dataset influence imputation and classification, and how imputation affects classification performance. We compare six popular imputation algorithms, namely, RAVG, KNNimpute, LLS.L2, LLS.PC, LS, and BPCA, by measuring how well the imputed dataset can preserve the discriminant power residing in the original dataset. An empirical analysis using real data from cancer microarray studies is also carried out. In addition, the NRMSE-based comparison is included in the study, with a modification in the case of the synthetic data to give an accurate measure. Recommendations for the application of various imputations under different situations are given in Section 3.Many studies have shown the log-normal property of microarray data, that is, the distribution of log-transformed gene expression data approximates a normal distribution , 17. In There are other models proposed for gene expression data, including a multiplicative model for gene intensities , a hieraBased on the log-normal property and inspired by ANOVA, we propose a model for the normalized log-ratio gene expression data which is centered at zero, assuming that any systematic dependencies of the log-ratio values on intensities have been removed by methods such as Lowess , 24. HerLet The true signal is given bywhere The log-transformed fold-change under the constraint that detected . The staThe distribution of tructure  is used whereIn the above equations, the gene block standard deviation The additive noise eal data . The noiFollowing the model above, we generate synthetic gene expression datasets for the true signal, The threshold Following the notation of , a gene We will discuss three more complex methods, namely, KNNimpute, LLS, and LS imputation, which follow the same two basic steps.(1) For each target gene (2) The missing part of the target gene We will additionally describe briefly the BPCA imputation method.In the first step, the The weight for each candidate gene is proportional to the reciprocal of the whereThe performance of KNNimpute is closely associated with the number of neighbors http://gsp.tamu.edu/Publications/supplementary/sun09a).In the first step, either the te genes , resultiIn the second step, the missing part of the target gene is estimated as a linear combination (which need not be a convex combination) of the corresponding parts of its candidate genes :where the vector of weights As is well known, the solution is given bywhere In the first step, similar to LLS.PC, the get gene .In the second step, the least squares estimate of the target given each of the where the regression coefficient where The missing part of the target gene is then approximated by a convex combination of the The weight of each estimate is a function of the correlation between the target and the candidate gene:The normalized weights are then given by BPCA is built upon a probabilistic PCA model and employs a variational Bayes algorithm to iteratively estimate the posterior distribution for both the model parameters and the MVs until convergence. The algorithm consists of three primary processes, which are (1) principle component regression, (2) Bayesian estimation, and 3) an expectation-maximization-like repetitive algorithm [ an expecBased on the previously described data model, we generate various synthetic microarray datasets by changing the values of the model parameters, corresponding to various noise levels, gene correlations, MV rates, and so on (more details are given in Section 3). The MVs are determined by (6), with the threshold A critical issue concerns the manner in which the test data are employed. As noted in the introduction, imputation cannot be applied to the training and test data as a whole. Not only does this make the designed classifier dependent on the test data, it also does not reflect the manner in which the classifier will be employed. Testing involves a single new example, independent of the training data, being labeled by the designed classifier. Thus, error estimation proceeds in the following manner after imputation has been applied to the training data and a classifier designed from the original and imputed values:  while the latter is a standard procedure used in the wrapper method . SFFS is a development of the sequential forward selection(SFS) method. Starting with an empty set "looking-back" for the features already in set The classifier training process includes feature selection, and classifier design based on a given classification rule. Three popular classification rules are used in this paper: Linear Discriminant Analysis (LDA), 3-Nearest Neighbor (3NN) and Linear Support Vector Machine (LSVM). Two feach (SFFS), are conrocedure . For therocedure . In addind stage . This twnd stage . In summEach experiment is repeated 150 times, and the average classification error is recorded. The averaged classification error plots for different datasets, feature selection methods and classification rules are shown in Section 3. Besides the classification errors, the NRMSE between the signal dataset and each of the 6 imputed datasets is also recorded. The simulation flow chart is shown in Figure As previously mentioned, there can be drawbacks associated with the NRMSE calculation; however, in our simulation study, the MVs are marked according to the data quality and the NRMSE is calculated based on the true signal dataset which can serve as the ground truth: In this way, the aforementioned drawbacks about using NRMSE are addressed.In addition to the synthetic data described in the previous section, we used the two following publicly available datasets from published studies.(i) Breast Cancer Dataset (BREAST)Tumor samples from 295 patients with primary breast carcinomas were studied by using inkjet-synthesized oligonucleotide microarrays which contained 24,479 oligonucleotides probes along with 1281 control probes . The sam(ii) Prostate Cancer Dataset (PROST)Samples of 71 prostate tumors and 41 normal prostate tissues were studied, using cDNA microarray containing 26,260 different genes . In addiwhere In the above equations, For the patient data study, the schemes used for imputation, feature selection and classification are similar to those applied in the synthetic data simulation, except that we use hold-out-based error estimation, that is, in each repetition,  samples , 35; ind samples . The resUnlike the synthetic study, the true signal dataset is unknown in the patient data study since the data values are always contaminated by measurement errors. Therefore, in the absence of the true signal dataset, the NRMSE is calculated between the measured dataset and each of the imputed datasets . Thus the NRMSE result is less reliable in the patient data study, which highlights further the need for evaluating imputation on the basis of other factors, such as classification performance.We have considered the model described in the previous section, for different combinations of parameters, which are displayed in Table Figure The effect of variance  but perform relatively poorly for the other combinations.The NRMSE values of different imputation methods generally decrease first and then increase as the MV rate increases (see Figure It is also found that there is no strong correlation between the low-level performance measure NRMSE and the high-level measure classification error. A small NRMSE may not necessarily suggest a small classification error, that is, an imputation method may perform better than another imputation method in terms of estimation accuracy, but the former may not be as good as the latter in terms of classification performance. In other words, although a given imputation method may be more accurate than another when measured by NRMSE, it might decrease more the discrimination power presents in the original data.We study the effects of MVs and their imputation on classification by using a model-based approach. The model-based approach is employed because it enables systematic study of the complicated microarray data analysis pipeline, including imputation, feature selection and classification. Moreover, it gives us ground truth for the differentially expressed genes, allowing the computation of imputation accuracy and classification error. We also carry out a simulation using real patient data from two cancer studies to complement the findings of the synthetic data study.Our results suggest that it is beneficial to apply MV imputation on the microarray data when the noise level is high, variance is small, or gene-cluster correlation is strong, under small to moderate MV rates. In these cases, if data quality metrics are available, then it may be helpful to consider the data point with poor quality as missing and apply one of the most robust imputation algorithms, such as LLS, and BPCA, to estimate the true signal based on the available high-quality data points, in which case the classifier designed on the imputed dataset with reduced noise may yield better error rates than the one designed on the original dataset.However, at large MV rates, we observed that imputation methods are NOT recommended, and the original measured data yields better classification performance. Regarding MV rate, our results indicate the presence of a peaking phenomenon: performance of imputation methods actually improves initially as the MV rate increases, but after an optimum point is reached, performance quickly deteriorates with increasing MV rates. This was observed very clearly in the synthetic data simulation, and less so with the patient data, even though the phenomenon is still noticeable.As for the NRMSE criterion, which is the figure of merit employed by most studies, we also observe a peaking phenomenon with increasing MV rate, in contrast to previous studies that report the NRMSE to increase monotonically with increasing MV rate; this may be a consequence of the different ways in which the MVs are selected in those studies as compared with the present one; in the former, MVs are picked randomly, whereas we pick MVs based on quality considerations.


Well-established statistical theory is available that simultaneously accounts for library size differences and biological variability using an appropriate mixture model. Moreover, specific implementations for DNA sequencing read count data  are already available in RNA-Seq focused R packages such as edgeR and DESeq. Here we summarize the supporting statistical theory and use simulations and empirical data to demonstrate substantial improvements provided by a relevant mixture model framework over simple proportions or rarefying. We show how both proportions and rarefied counts result in a high rate of false positives in tests for species that are differentially abundant across sample classes. Regarding microbiome sample-wise clustering, we also show that the rarefying procedure often discards samples that can be accurately clustered by alternative methods. We further compare different Negative Binomial methods with a recently-described zero-inflated Gaussian mixture, implemented in a package called metagenomeSeq. We find that metagenomeSeq performs well when there is an adequate number of biological replicates, but it nevertheless tends toward a higher false positive rate. Based on these results and well-established statistical theory, we advocate that investigators avoid rarefying altogether. We have provided microbiome-specific extensions to these tools in the R package, phyloseq.Current practice in the normalization of microbiome count data is inefficient in the statistical sense. For apparently historical reasons, the common approach is either to use simple proportions (which does not address heteroscedasticity) or to use  microbiome refers to the ecosystem of microbes that live in a defined environment. The decreasing cost and increasing speed of DNA sequencing technology has recently provided scientists with affordable and timely access to the genes and genomes of microbiomes that inhabit our planet and even our own bodies. In these investigations many microbiome samples are sequenced at the same time on the same DNA sequencing machine, but often result in total numbers of sequences per sample that are vastly different. The common procedure for addressing this difference in sequencing effort across samples – different library sizes – is to either (1) base analyses on the proportional abundance of each species in a library, or (2) rarefy, throw away sequences from the larger libraries so that all have the same, smallest size. We show that both of these normalization methods can work when comparing obviously-different whole microbiomes, but that neither method works well when comparing the relative proportions of each bacterial species across microbiome samples. We show that alternative methods based on a statistical mixture model perform much better and can be easily adapted from a separate biological sub-discipline, called RNA-Seq analysis.The term  PLOS Computational Biology Methods article.This is a RNA-Seqshotgun sequencing of microbiome metagenomic DNA reads rather than continuous values derived from the fluorescence intensity of hybridized probes. In recent generation DNA sequencing the total reads per sample  can vary by orders of magnitude within a single sequencing run. Comparison across samples with different library sizes requires more than a simple linear or logarithmic scaling adjustment because it also implies different levels of uncertainty, as measured by the sampling variance of the proportion estimate for each feature . In this article we are primarily concerned with optimal methods for addressing differences in library sizes from microbiome sequencing data.Modern, massively parallel DNA sequencing technologies have changed the scope and technique of investigations across many fields of biology K, for gene i, in sample j according to:js is a linear scaling factor for sample j that accounts for its library size, iμ is the mean proportion for gene i, and iφ is the dispersion parameter for gene i. The variance is φ = 0. Recognizing that φ>0 and estimating its value is necessary in gene-level tests in order to control the rate of false positive genes. Many false positive genes appear significantly differentially expressed between experimental conditions under the assumption of a Poisson distribution, but are nevertheless not-significant in tests that account for the larger variance that results from non-zero dispersion.Variation in the read counts of features between technical replicates have been adequately modeled by Poisson random variables iφ for every gene when there is a small number of samples — or a small number of biological replicates — can be mitigated by sharing information across the thousands of genes in an experiment, leveraging a systematic trend in the mean-dispersion relationship The uncertainty in estimating ad hoc library size normalization by random subsampling without replacement, or so-called rarefyingrarefactionindividual-based taxon re-sampling curvesrarefying or rarefied counts when referring to the normalization procedure, respecting the original definition for rarefaction. Rarefying is most often defined by the following steps Instead, microbiome analysis workflows often begin with an rarefaction levelSelect a minimum library size, Discard libraries (microbiome samples) that have fewer reads than Subsample the remaining libraries without replacement such that they all have size defective, and the process of identifying defective samples comes with a risk of subjectivity and bias. In many cases researchers have also failed to repeat the random subsampling step (3) or record the pseudorandom number generation seed/process — both of which are essential for reproducibility. To our knowledge, rarefying was first recommended for microbiome counts in order to moderate the sensitivity of the UniFrac distance differential abundance of OTUs between predefined classes of samples differential abundancedifferential expression from RNA-Seq. Like differentially expressed genes, a species/OTU is considered differentially abundant if its mean proportion is significantly different between two or more sample classes in the experimental design.Often rarefying biological count data is statistically inadmissible because it requires the omission of available valid data. This holds even if repeated rarefying trials are compared for stability as previously suggested Despite its current popularity in microbiome analyses A and B, comprised of 100 and 1000 DNA sequencing reads, respectively. In statistical terms, these library sizes are also equivalent to the number of trials in a sampling experiment. In practice, the library size associated with each biological sample is a random number generated by the technology, often varying from hundreds to millions. For our example, we imagine the simplest possible case where the samples can only contain two types of microbes, called OTU1 and OTU2. The results of this hypothetical experiment are represented in the Original Abundance section of χ2 test  or a Fisher exact test. By first rarefying (Rarefied Abundance section) so that both samples have the same library size before doing the tests, we are no longer able to differentiate the samples  because they have unequal variances. This problem of unequal variances is called heteroscedasticity. In other words, the uncertainty associated with each value in the table is fundamentally linked to the total number of observations (or reads), which can vary even more widely than a 10-fold difference. In practice we will be observing hundreds of different OTUs instead of two, often with dependendency between the counts. Nevertheless, the difficulty caused by unequal library sizes still pertains.The variance of the proportion's estimate repeated rarefying trials , or sets of proportions . Although rarefying does equalize variances, it does so only by inflating the variances in all samples to the largest (worst) value among them at the cost of discriminating power (increased uncertainty). Rarefying also adds artificial uncertainty through the random subsampling step, such that In order to quantify the relative statistical costs of rarefying, and to illustrate the relative benefits of an appropriate mixture model, we created two microbiome simulation workflows based on repeated subsampling from empirical data. These workflows were organized according to Principal Coordinate Analysis, PCoA), which is useful if the desired effects are clearly evident among the first two or three ordination axes. In some cases, formal testing of sample covariates is also done using a permutation MANOVA .For each of the previous normalizations we calculated sample-wise distance/dissimilarity matrices using the following methods, if applicable.Bray-Curtis. The Bray-Curtis dissimilarity first defined in 1957 for forest ecology Euclidean. The euclidean distance treating each OTU as a dimension. This has the form K and i as defined in the n the number of distinct OTUs.PoissonDist. Our abbreviation of PoissonDistance, a sample-wise distance implemented in the PoiClaClu package top-MSD. The mean squared difference of top OTUs, as implemented in edgeR UniFrac-u. The Unweighted UniFrac distance UniFrac-w. The Weighted UniFrac distance Ocean or Feces microbiomes of the Global Patterns empirical dataset Ocean and Feces sample classes were chosen because they have negligible overlapping OTUs, and mixing them by a defined proportion allows arbitrary control over the difficulty of the subsequent sample clustering task from trivial (no mixing) to impossible . See simulation-design section of In order to consistently evaluate performance in this regard, we generated microbiome counts by sampling from two different multinomials that were based on either the Clustering was performed independently for each combination of simulated experiment, normalization method, and distance measure using partitioning around medoids (PAM The number of samples (40) to include for each template in Simulation A was chosen arbitrarily after some exploration of preliminary simulations. It was apparent that the classification results from Simulation A were most informative when we included enough samples per simulated experiment to achieve stable results, but not so many that it was experimentally unrealistic and prohibitively slow to compute. Conversely, preliminary trials of Simulation A that included only a few samples per experiment resulted in a large variance on each performance measure that was difficult to interpret.a priori knowledge about the microbiological conditions, and we want to enumerate the OTUs that are different between these microbiomes, along with a measure of confidence that the proportions differ. For this form of analysis, the microbiome counts for each simulated experiment are generated by sampling from a single multinomial derived from the OTU proportions observed in one environment of the Global Patterns dataset. To create an effect, the simulated samples of an experiment were divided into two equally-sized classes, test and null, and a perturbation was applied  to the count values of a random subset of OTUs in the test class only. See part B of Simulation B is a simple example of microbiome experiments in which the goal is to detect microbes that are differentially abundant between two pre-determined classes of samples. This experimental design appears in many clinical settings , and other settings for which there is sufficient For each simulated experiment, we used the following normalization/modeling methods prior to testing for differential abundance.Model/None. A parametric model was applied to the data, or, in the case of the t-test, no normalization was applied .Rarefied. Rarefying is performed as defined in the introduction, using rarefy_even_depth implemented in the phyloseq package th-percentile of library sizes within each simulated experiment.Proportion. Counts are divided by total library size.For each OTU of each simulated experiment we used the following to test for differential abundance.two sided Welch t-test. A two-sided t-test with unequal variances, using the mt wrapper in phyloseq edgeR - exactTest. An exact binomial test (see base R's stats::binom.test) generalized for overdispersed counts DESeq - nbinomTest. A Negative Binomial conditioned test similar to the edgeR test above, implemented in the nbinomTest method of the DESeq package Testing for differential expression in Anders and Huber, 2010 DESeq2 - nbinomWaldTest. A Negative Binomial Wald Test using standard maximum likelihood estimates for GLM coefficients assuming a zero-mean normal prior distribution, implemented in the nbinomWaldTest method of the DESeq2 package.metagenomeSeq - fitZig. An Expectation-Maximization estimate of the posterior probabilities of differential abundance based on a Zero Inflated Gaussian model, implemented in the fitZig method of the metagenomeSeq package library size is altogether different from effect size; the former being equivalent to both the column sums and the number of reads per sample. The library sizes for both categories of simulation were randomly sampled from the original distribution of library sizes in the Global Patterns dataset, and then scaled according to the prescribed median library size of each simulated experiment.All tests were corrected for multiple inferences using the Benjamini-Hochberg method to control the False Discovery Rate exactly. Interested investigators can inspect and modify this code, change the random seed and other parameters, and observe the results (including figures). For ease of inspection, we have authored the source code in R flavored markdownknit2html function in the knitr package. This function will take the location of the simulation source files as input, evaluate its R code in sequence, generate graphics and markdown, and produce the complete HTML5 output file that can be viewed in any modern web browser. These simulations, analyses, and graphics rely upon the cluster We have included in φ>0), especially at larger values of the common-scale mean. By definition, these OTUs are the most abundant, and receive the greatest interest in many studies. For rarefied counts the absolute scales are decreased and there are many fewer OTUs that pass filtering, but overdispersion is present in both cases and follows a clear sample-wide trend. See the dispersion-survey section of iφ, can be adequately estimated among biological replicates of microbiome data, despite a previous weak assertion to the contrary We surveyed various publicly available microbiome count data to evaluate the variance-mean relationship for OTUs among sets of biological replicates, a few examples of which are shown here . In everIn simulations evaluating clustering accuracy, we found that rarefying undermined the performance of downstream clustering methods. This was the result of omitted read counts, added noise from the random sampling step in rarefying, as well as omitted microbiome samples with small library sizes that were accurately clustered by alternative procedures on the same simulated data . The extth-percentile of library sizes for most simulations and normalization/distance procedures  but not with RLE normalization (shown), and was similarly absent in DESeq(2) results. The false positive rate for edgeR and DESeq(2) was near zero under most conditions, with no obvious correlation between false positive rate and effect size. Although rarefied counts and proportions both performed relatively poorly, count proportions outperformed rarefied counts in most simulations due to better sensitivity, but also suffered from a higher rate of false positives at larger values of effect size  is large enough that the null (unmodified) OTUs appear significantly more abundant in the null samples than in the test samples. This explanation is easily verified by the sign of the test statistics of the false positive OTU abundances, which was uniformly positive . Importaect size .The rarefying normalization procedure was associated with performance costs in both sample-clustering and differential abundance statistical evaluations, enumerated in the following.a loss of power or decreased sensitivity . Overdispersion is theoretically expected for counts of this nature, and we unambiguously detected overdispersion in our survey of publicly available microbiome counts . EstimatRarefying counts requires an arbitrary selection of a library size minimum threshold that affects downstream inference , but forThe random step in rarefying is unnecessary and adds artificial uncertainty  and the original library sizes are large enough to withstand the loss of data. Many early descriptive studies fall into this category – for example comparing functionally distinct human body sites or environments Due to these demonstrated limitations and proven sub-optimality, we advocate that e.g. Benjamini-Hochberg In the case of differential abundance detection, it seems unlikely that the cost of rarefying is ever acceptable. In our simulations, both rarefied counts and sample proportions resulted in an unacceptably high rate of false positive OTUs. As we described theoretically in the introduction, this is explained by differences among biological replicates that manifest as overdispersion, leading to a subsequent underestimate of the true variance if a relevant mixture model is not used. We detected overdispersion among biological replicates in all publicly available microbiome count datasets that we surveyed . Failureincreased with the effect size, e.g. the fold-change in OTU abundance among the true-positive OTUs. For rarefied counts, we also detected a simultaneous increase in Type-II error attributable to the forfeited data. It may be tempting to imagine that the increased variance estimate due to rarefying could be counterbalanced by the variance underestimate that results from omitting a relevant mixture model. However, such a scenario constitutes an unlikely special case, and false positives will not compensate for the false negatives in general. In our simulations both Type-I and Type-II error increased for rarefied counts Click here for additional data file.Text S1A supplemental appendix of the statistical mathematics described in the article.(PDF)Click here for additional data file.


MicroRNAs (miRNAs) are small non-coding RNAs that regulate gene expression. It has been proposed that miRNAs play an important role in cancer development and progression. Their ability to affect multiple gene pathways by targeting various mRNAs makes them an interesting class of regulators.We have developed an algorithm, Classification based Analysis of Paired Expression data of RNA (CAPE RNA), which is capable of identifying altered miRNA-mRNA regulation between tissues samples that assigns interaction states to each sample without preexisting stratification of groups. The distribution of the assigned interaction states compared to given experimental groups is used to assess the quality of a predicted interaction. We demonstrate the applicability of our approach by analyzing urothelial carcinoma and normal bladder tissue samples derived from 24 patients. Using our approach, normal and tumor tissue samples as well as different stages of tumor progression were successfully stratified. Also, our results suggest interesting differentially regulated miRNA-mRNA interactions associated with bladder tumor progression.The need for tools that allow an integrative analysis of microRNA and mRNA expression data has been addressed. With this study, we provide an algorithm that emphasizes on the distribution of samples to rank differentially regulated miRNA-mRNA interactions. This is a new point of view compared to current approaches. From bootstrapping analysis, our ranking yields features that build strong classifiers. Further analysis reveals genes identified as differentially regulated by miRNAs to be enriched in cancer pathways, thus suggesting biologically interesting interactions. Bladder cancer is the fourth most common cancer in industrialized countries Apart from co-expressed genes, differentially regulated pairs of miRNAs and mRNAs play an important role in several cellular processes and diseases. To assess this issue, several methods have been developed to predict interactions between miRNAs and mRNAs based on their sequences. Most of the tools exploit the seed complementary between miRNAs and the 3′UTR of specific mRNA, information about the sequence conservation of adjacent bases and thermodynamic properties of miRNA-target mRNA interactions. The different methods have been recently reviewed There is a need for methods which consider the specific nature of miRNA induced regulation. miReduce In this study, we present a novel approach that evaluates differential miRNA-mRNA regulation combined with the distribution of samples for a single interaction. We hypothesize that single miRNA-mRNA interactions are characteristic for a particular state of tumorigenesis. We consider differential miRNA induced gene regulation as a two class problem and use the following assumption. Given an interaction between a miRNA and mRNA which is characteristic for a difference between two groups of samples, the miRNA is up-regulated and the mRNA down-regulated in the first group with respect to the second group, or reciprocal. Our approach classifies each predicted interaction for each sample independently of group knowledge. By this way, one can analyze individual differences inside a collective of samples for a specific set of interactions. Furthermore given an interaction, we can partition samples into expected groups which reflect the miRNA induced gene regulation. The agreement between the expected groups and the experimental ones yields a meaningful ranking to distinguish potential interactions from those which are unlikely to occur. In a final step, we incorporate information about negative correlation between miRNA and mRNA expression to eliminate false positives.Identifying differentially regulated miRNA-mRNA interactions is a basically a form of feature selection. To validate the different steps of our approach, we have performed a principal component analysis to analyze the separation of samples after assignment of interaction states and evaluated the performance of our ranking to build classifiers.In particular, we have applied our approach to a collective of healthy bladder tissue samples and bladder tumor samples at different stages. In addition, we have examined the ability of our approach to classify prostate cancer tumors and healthy tissue, as well as colon cancer samples and healthy tissue using small sample sizes A selection of 24 urothelial samples from a collective of bladder cancer patients described previously was used in this study The analyzed tumor tissues samples contained more than 80% tumor cells as previously described miRNA expression analysis was performed by one-color hybridizations on human catalog 8-plex 15 K microRNA microarrays (AMADID 019118) from Agilent  which enclosed 723 human and 76 viral microRNAs from the Sanger miRBase (release 10.1). All reaction steps were carried out as previously described in detail mRNA expression analysis was performed by one-color hybridizations on whole human genome microarray 4×44 K v2 (026652) from Agilent comprising probes for human 34184 mRNA transcripts. After hybridization, microarrays were washed, scanned, and processed according to the supplier’s protocol. The raw data were normalized using Genespring GX11 Software (Agilent) with default parameters . All microarray data has been deposited in the NCBI GEO database with accession number GSE40355.Validated human miRNA-mRNA interactions were obtained from Tarbase 5.0 and miRecords (version 11-2010) The goal of the algorithm is to partition the expression values corresponding to each probe into three sets: “high”, “medium” and “low”.Let We define the absolute fold change as There are two preliminary considerations. The first assumption is that two expression values are differentially expressed if their absolute fold change B where B, we define the absolute fold change between a and the mean of set B as B is nonempty, and Given A is the neighborhood of a if and only if We define that set a as representative of a set A if and only if A is the neighborhood of a. Please notice, that there can be more than one representative for a set A, i.e. for two values A is neighborhood of a and B is neighborhood of b, if We define a and b and their neighborhoods A and B as follows:We define a scoring function on two elements, We add following constraint to determine the final score, where The rationale behind this scoring function is to find two sets of similarly expressed values which cover most of the data, thus also which overlap little as possible, i.e. the data coverage term To sum up the essential meaning of the scoring function, we identify two different neighborhoods, i.e. values of similar expression. These neighborhoods differ by at least a defined absolute fold change, but then the absolute fold change can be arbitrary large. The scoring function evaluates to what extend these neighborhoods are useful to represent the data, based on data coverage not absolute values.Given the two resulting sets and their corresponding representatives which produce the highest final score, we denote the representative with the lower value as The rationale for this is as follows. The boundaries are defined by the upper limit of the lower set, and lower limit of the upper set; if the sets overlap, the boundaries are switched.v is defined by:Finally, for each This classification will be referred to as state in the following.For the actual classification of expression values, the fold threshold and neighborhood threshold is determined dynamically from a list of predefined paired values, i.e. a pair t is an arbitrary real value and the cardinality S  =  {LH, HL, LM, HM, MH, ML, HH, LL, MM}.Only those miRNA or mRNA probes are considered that exceed a certain score higher than a threshold We group these combinations by their biological meaning:compHLS  =  {HL, ML, HM}; up-regulated miRNA cause hypothetic down-regulation of mRNA.Down-regulated states compLHS  =  {LH, MH, LM};. Down-regulated miRNA cause hypothetic up-regulation of mRNA.Up-regulated states undefS  =  {HH, LL, MM} which do not follow the biological interpretation mentioned above.Undefined states Interactions with a frequency of undefined states A and B, it was defined that an interaction is differentially regulated for A and B, if the state with the maximum frequency of group A is an element of B is an element of Given two pre-defined groups For each interaction, a Jaccard-index is calculated to evaluate the agreement between the predefined experimental groups Therefore, a partition The Jaccard-Index is then the similarity between the two partitions To evaluate the applicability of our ranking by Jaccard-indexes, based on the set of selected differentially regulated interactions a simple classification model is constructed which predicts the first group of an ensemble of samples, e.g. cancer samples from a collective of cancer and non-cancer samples.i of the set of selected interactions I, where i where the state of the sample i where the state of the sample Such a model contains a set of states A model is generated from the highest ranked interactions within a threshold value for the Jaccard-index of an interaction or by a defined number of randomized interactions within a range of Jaccard-indexes. The states The normalized expression values were randomly divided into training and test sets where each training set contains half of the samples of each group without replacement. If the number of samples was odd for a group, the trainings sets were assigned one sample more than the test sets for that group. Concerning the bladder cancer data set, for the collective of all samples, each training and each test set contains eight samples from either the group of invasive or non-invasive bladder cancer samples and four normal tissue samples. For the collective of bladder cancer samples, each training and each test set contains four invasive bladder cancer samples and four non-invasive bladder cancer samples. For the two collectives, 100 different data sets of training and test sets were generated by randomly dividing the samples under the constraints mentioned above. miRNA-mRNA interactions were computed and classified separately for each training and each test set. For each of the 100 data sets a model was computed based on the training set and applied to the corresponding test set. Mean specificities, sensitivities and false positives rates were computed over all 100 data sets.Similarly to the bladder cancer data set, a colon cancer and prostate tumor data set that contain paired miRNA/mRNA micro array expression data was used to estimate specificities and sensitivities. Colon tissue samples and prostate tissue samples were extracted from the data set provided by Lu et al. In addition for the cancer tissue sample collective of the bladder cancer data set, the whole procedure was performed with an outlier removed and the same outlier re-assigned to the expected group according to the results of our examination.To compare the results of our classifiers to another method, Prediction Analysis of Microarrays for R (pamr) Mean specificities and sensitivities were calculated in the same way as mentioned above.For each of three experimental groups, i.e. invasive bladder cancer samples, non-invasive bladder cancer samples and normal tissue samples, Spearman correlation coefficients, ρ, were calculated between the miRNA and mRNA expression. The log-normalized expression values were used as input data. Pairs of miRNA-mRNAs were defined by the same set of interactions, as mentioned above. The expression values were treated separately for each of the three experimental groups. Spearman correlation coefficients were calculated for each pair of miRNA-mRNA interactions for each group.We applied our approach to two different collectives, a collective of all samples (8 non-invasive- and 8 invasive tumor samples as well as 8 control persons) and a collective of tumor samples with different tumor stages (8 non-invasive and 8 invasive samples) without healthy persons. For both collectives, only miRNAs and mRNAs expression values were processed showing in at least 20% of the used samples a “present call”, indicated by the microarray normalization software Genespring GX. Next, we applied our approach to identify differentially regulated interactions. In a further step, we selected only interactions that show a negative correlation, i.e. ρ≤−0.4, between normalized miRNA and mRNA expression values for at least one experimental group. For the collective of cancer tissue samples these groups are the invasive bladder cancer samples and non-invasive bladder cancer samples. For the collective of all samples the groups comprise both bladder cancer sample groups and the group of normal tissue samples, i.e. three different groups.N N-dimensional vectors Based on the interaction states a principal component and clustering analysis was performed. For this purpose, the interaction states were substituted into real values as mentioned in Genes which are involved in the differentially regulated interactions between miRNA and mRNA were analysed using the database for annotation, visualization and integrated discovery (DAVID) For comparative analysis, we applied four additional approaches to analyze the two collectives of bladder cancer samples, the collective of healthy and tumor samples and the collective of invasive and non-invasive tumor tissue samples. The TaLasso web server was used to identify miRNA-mRNA interactions with the TaLasso method and GenMiR++ algorithm In addition, Spearman correlations and a Meta analysis approach using the Magia2 web server were used to analyze the data sets For all approaches and both collectives, only miRNAs and mRNAs expression values were processed showing in at least 20% of the used samples a “present call”, indicated by the microarray normalization software Genespring GX. Log-normalized expression values were used for analysis, as mentioned above. When more than one probe was mapped to the same EntrezGeneID, for each sample, the median of the log-normalized expression values over the probes was calculated.To determine the set of “best” interactions, interactions were ranked by highest score when TaLasso or GenMiR++ was used, by most negative correlation when Spearman correlations were calculated and by lowest q-value when the Meta analysis approach was used.http://sourceforge.net/projects/caperna. It can be used under the terms of the GNU General Public License v3.0.An implementation of our algorithm as package of command-line tools is available at After classification and filtering, our algorithm identified 11562 interactions for the whole collective of bladder tissue samples and 9075 interactions for the collective of bladder tumor samples that exhibit different states of regulation over the samples and assigned interaction states. At this stage, no a priori group knowledge is inferred. Distances between the samples were calculated using interaction states. Comparing the spread of samples in the first two principal components, one can clearly distinguish samples of the control group from tumor samples . HoweverA cluster analysis reveals similar results. Samples of the control group form an isolated cluster that is far from the groups of tumor samples . ConcernAs identifying altered miRNA-mRNA interactions between two groups of tissues is a form of feature selection we argue that a valid ranking of putative miRNA-mRNA interactions should be reflected by the selected features. The quality of selected features can be measured by the performance of classifiers constructed from these features. We have generated randomized non-overlapping trainings and test sets for four different collectives, I. healthy bladder cancer tissue and bladder tumor tissue, II. non-invasive bladder tumor tissue and invasive bladder cancer tissue, III. healthy colon tissue and colon tumor samples and IV. healthy prostate tissue samples and prostate cancer samples. Next, we created simple models based on the interactions with the highest Jaccard-indexes using our method, CAPE RNA, and trained models on the same sets, but separately for miRNAs and mRNAs, using Prediction Analysis of Microarrays for R (pamr) For collective I, our method exhibits mean specificities between 0.99–1 and mean sensitivities between 0.93–0.99 depending on the lowest allowed Jaccard-index, thus the size of the model, For the colon and prostate collectives , our method shows mean specificities between 0.89–1 and mean sensitivities between 0.87–0.97, We have examined the miRNA-mRNA interactions inside the bladder cancer data set more elaborately. The outlier which has been identified by PCA was removed for analysis of the non-invasive and invasive bladder tumor collective. After inferring a priori knowledge of the groups and selecting interactions by interaction states that indicate a differential regulation between the two groups, the set of selected differentially regulated interactions contains 9180 interactions for the collective of all samples and 5963 differentially regulated miRNA-mRNA interactions for the collective of tumor samples. Next, the interactions were selected that exhibit a negative correlation between miRNA and mRNA expression. There were 5583 interactions for the collective of all samples and 2938 interactions for the collective of cancer samples that satisfy these criteria. Compared to these numbers, there are relatively few interactions which are suitable to partition the samples into their pre-defined groups when used as single interaction, i.e. interactions that exhibit a high Jaccard-index .For the collective of all samples, there are 29 interactions which yield a perfect classification of the collective as single interaction, For the differences in regulation between the non-invasive and invasive tumors, there are no interactions with a perfect Jaccard-index of 1. However, there are twelve interactions with a Jaccard-index higher than 0.70, see By pathway enrichment analyses we could show that corresponding genes of the detected miRNA-mRNA interactions were strongly associated with different KEGG cancer pathways including bladder cancer. The detailed list can be found in In this section, we would like to analyze two selected pathways in more detail. First, the growth factor receptor signaling pathway in bladder cancer (FGFR3), as defined in the review by Fendler et. al, is examined For both pathways, we selected only interactions with a Jaccard-index equal to or higher than 0.4 that were selected by negative correlation between miRNA and mRNA expression. Based on these criteria, we have identified 13 differentially regulated interactions in the FGFR3 pathway . In bladFor the KEGG bladder cancer pathway there are sixteen interactions that satisfy the conditions . In addiFurthermore, the oncogenes NRAS and ERBB2 are up-regulated in bladder cancer, hypothetically, due to lower of expression of miR-28-5p and miR-125b, respectively.In general to assess the validity of a prediction method, one could evaluate its performance on a data set where the correct result is known for every instance, then, for example, specificities and sensitivities can be calculated. For miRNA-mRNA prediction methods this proves very problematic because there is not such a data set. In addition, even experimental results may have a bias due to the researchers’ interpretation and may have a limited reproducibility. Furthermore, it is almost impossible to create such a data set in the first place, because it would, for instance, involve knockout experiments for every predicted interaction and every non-predicted interaction. Nonetheless, one can look at the problem from a different point of view. We stated earlier in this manuscript that predicting miRNA-mRNA interactions is a form of feature selection and one can at least evaluate the validity of selected features in the sense that they can generate strong classifiers. This does not imply that these interactions do actually occur in biological systems, but one may at least suggest that they are more likely to have a biological meaning. An important aspect is that one evaluates the performance to classify unknown instances, i.e. training sets and test sets must be separated, otherwise a classifier could be over-fitted and learn the data by heart. We argue that the method for the evaluation of the performance of a predictor for miRNA-mRNA interactions as presented by this study is more general and may be more reasonable than calculating statistics over a data set itself or comparing a limited number of literature results.For our approach, CAPE RNA, following observations may indicate its validity. The validity of interaction states is reflected in the results of the bootstrapping analysis since interaction states are the foundation of further steps in our approach. In addition, the results of the PCA and cluster analysis suggest they perform more reasonable to discriminate between the different groups of samples than the log-normalized expression data, The results of the bootstrapping analysis show that a simple model based on our approach and assumptions outperforms a standard method for gene expression as a classifier. Furthermore, our method is applicable to very small sample sizes, i.e. for the prostate cancer collective a training set contains three samples for each group and for the colon cancer collective four tumor samples and two healthy samples.It was our assumption that the Jaccard-index can be used as a ranking for interactions. To examine whether there is a relationship between the performance of classifiers and Jaccard-indexes independently of the size of the model, we generated models by randomly selecting a defined number of interactions within a range of Jaccard-indexes using the bladder cancer training and test sets (collective I and collective II) and evaluated their performance. Also, instead of randomly determining relationships between miRNAs and mRNA we used a set of putative miRNA-mRNA interactions which has a scientific basis. Furthermore, the genes involved in predicted interactions are enriched in cancer pathways.In this study, we present an approach that introduces a high level of abstraction by classifying expression data in three different states. Consequently, there is a loss of information. However, our method has several advantages. As gene expression data is very noisy data in general, the classification of data into few discrete states makes it much more feasible to analyze. The classification closely follows a biological assumption that there exist two groups, one group which exhibits a low expression level and another one which exhibits a high expression level, thus reducing the information content to a reasonable level of abstraction. It provides a simple way to classify miRNA-mRNA interactions into a small number of states that can be grouped by their biological meaning in down-regulated, up-regulated and undefined interactions as direct interpretation of siRNA mechanism. Using interaction states, individual differences between samples can be compared for a selected set of genes. There is no other approach known to us which provides such intuitive classifications for individual samples.The scoring function for the classification depends on data coverage. Hence, outliers have almost no influence on the classification of the expression data of the other samples. Instead of using predefined groups, our algorithm identifies two groups, i.e. neighborhoods of similarly expressed values, by the data itself and gives a score which describes how much the two groups represent the expression values for a given probe. For other purposes, this scoring function could be generalized to multi-class problems, e.g.However, we are uncertain whether it would perform as well as for a two class problem. In addition, the combinatorial complexity increases with the number of classes. Especially, for paired data the number of interaction states increases quadratically and one would compromise the simplistic interpretation of those states. For differential regulation between two groups of samples, we are only interested in a two class problem.Compared to most approaches based on statistical tests, e.g. a t-test, our method does not compare mean or median values of groups which may underestimate the regulation of single samples and also is error-prone to outliers. Our approach evaluates miRNA-mRNA expression data on the level of interactions. While the evaluation of miRNA and mRNA expression data as separate sets needs not be meaningful, one can identify reasonable differences in regulation by examining the combined set of miRNA-mRNA interaction states.The examination of differential expression levels alone may be insufficient to identify direct interactions between miRNAs and mRNAs. On the other hand, biological systems are very complex systems. Several miRNAs may target the same mRNA and other factors such as transcription factors may influence the expression of a specific mRNA. Therefore, one needs not necessarily find a strong negative correlation between the expression levels of a miRNA and the target mRNA. However, the identification of even low or moderate negative correlation between miRNA and mRNA expression increases the certainty of a predicted interaction.The augmentation of our approach by testing for negative correlation, is a reasonable choice not to over- or underestimate differential expression levels and direct correlation between miRNA and mRNA expression.We have compared our approach, CAPE RNA, to four related methods, TaLasso, GenMiR++, Spearman correlation and a Meta analysis approach. The TaLasso web server offers an implementation of both TaLasso and GenMiR++ To compare the four different methods mentioned above and our approach, we formulated a simple problem: given a data set of paired miRNA/mRNA expression values and two known groups of samples give us the 500 most promising candidates for interesting miRNA-mRNA interactions. For this purpose, we analyzed the collective of healthy tissue samples and bladder tumor samples (collective I) and the collective of invasive and non-invasive bladder cancer samples (collective II).Although similar sets of potential interactions were used, the results of this comparison are clearly biased since the set of putative interactions differ for different approaches. The Meta analysis and correlation based analysis were performed with the same set of interactions. For TaLasso and GenMiR++ the same set of interactions was used, but for our approach another set of interactions was used. This is a general problem when different tools for the identification of miRNA-mRNA interactions are compared.Our results suggest that there is little overlap between predictions for the each best 500 interactions when different approaches are applied .As for GenMiR++, TaLasso and Spearman correlation the data set is analyzed as a whole, they do not take differential expression between two groups into account. Therefore, we do not expect them to exhibit the same results as a method that considers differential expression does. Compared to our method, the Meta approach considers differential expression between groups, but does not analyze paired miRNA and mRNA expression per sample. Since we apply correlation as a filtering criterion and our method may show some similarities to correlation in general, we expected an overlap between analysis based on correlation and our approach. However, there are several differences between both methods. We calculate correlation for each experimental group separately and not over the data set as a whole which may be more reasonable when one considers a complex pattern of de-regulation and perhaps also loss or gain of function. Obviously, the main difference is our procedure in general. We consider differential expression and evaluate to what extent the partition of samples into expected classes resembles the defined groups. Concerning the relationship between all samples, our approach is less strict than correlation over a data set as a whole, but it emphasizes on the distribution of each sample based on paired miRNA-mRNA expression.In conclusion, most approaches address similar problems, but not a two class problem for differential miRNA induced regulation, e.g. between healthy and disease tissue samples. Even when methods are applied that address the same problem, there is much less overlap between the top ranked interactions than one might have expected. Although there is a bias introduced by the set of potential interactions, this observation remains valid, e.g. when TaLasso and GenMiR++ are compared which share the same set of interactions. Correlation analysis, TaLasso and GenMiR are interesting methods to identify interactions in general or to analyze time series data while methods like the Meta approach most likely do not put enough emphasis on the regulation of individual samples. With CAPE RNA, we present a method that combines both differential expression and paired miRNA-mRNA regulation per sample. None of the methods or tools yields an individual classification of each sample similar to the interaction states based on our approach. Our method to evaluate the distribution of samples based on the agreement between expected and predefined  groups is unique among those approaches, as far as we know.Among the highest ranked differentially regulated interactions we found several genes which are known to be involved in bladder tumor progression. For instance, the MMP repressor RECK , which decrease was found associated with more invasive forms of bladder cancer One family of miRNAs appears to be most important for the differential regulation in bladder cancer, the family of miR-200b/c and miR-429. Other highly ranked interactions , include miR-1,miR-143, miR-145, miR-183, miR-19a, miR-19b, miR-200a, miR-200b, miR-200c, miR-204,, miR-222, miR-23b, miR-33a, miR-425, miR-429. There are 88 miRNAs involved in 2480 interactions with a Jaccard-index equal to or higher than 0.4, and 25 miRNAs involved in 252 interactions with a Jaccard-index of at least 0.7. These numbers show us that there is a large network of differentially regulated potential interactions of which several different interactions may play in important role for tumorgenesis.To distinguish invasive from non-invasive bladder cancer, 319 interactions with a Jaccard-index equal to or higher than 0.4 remain involving 34 miRNAs, but there are just twelve interactions with a Jaccard-index equal to or higher than 0.7 that comprise six miRNAs, namely miR-24, miR-26b, miR-29b, miR-29c, miR-30b and miR-7. These six miRNA are the most promising candidates to investigate tumor progression to invasive bladder cancer based on our examination.miRNA-mRNA interactions in two selected pathways, were analyzed in more detail. First the FGF3R pathway as defined in the review by Fendler et. In KEGG bladder cancer pathway, the oncogenes NRAS and ERBB2 were found up-regulated, hypothetically, due to lower of expression of miR-28-5p and miR-125b, respectively. The up-regulation of cyclin D1, which is involved in cell cylcle progression, is probably promoted by down regulation of miR-497. The Cyclin D/CDK4 complex phosphorylates pRB allowing it to disassociate from the transcription factor E2F. As previously reported In this study, the identification of interactions is limited by the used set of validated and predicted miRNA-mRNA interactions and by our approach in the sense that we only consider probes which exhibit two distinct expression levels based on our definition. Also, the certainty of identified differentially regulated interactions depends on the number of samples and the distribution of samples over the two groups. Based on the definition of our scoring function, this approach is unlikely to perform well if the number of samples differs too much between the two groups. Although we used negative correlation between miRNA and mRNA expression as filtering criterion, one cannot rule out the influence of other factors on gene expression such as transcription factors.While the identified interactions present many interesting candidates for biomarkers, the underlying mechanisms can only be understood partially by the examination of a limited of subset of genes. The analysis of the FGF3R and bladder cancer pathway indicates a few differentially regulated switches. A more elaborate understanding is adjourned to future examinations.Figure S1Spread of samples for the bladder cancer data set using normalized expression values. The first two principal components (PC) of the normalized data are shown for the collective of all samples (A and C) and the collective of tumor samples (B and D) using normalized miRNA (A and B) or mRNA (C and D) expression data, respectively. Circles refer to samples of the control group, triangles are samples revealed invasive tumors and crosses refer to samples with non-invasive tumors.(TIF)Click here for additional data file.Figure S2Hierarchical clustering of the bladder cancer data set using normalized expression values. Hierarchical clustering of the normalized expression data was performed using Ward’s method for the collective of all samples (A and C) and the collective of tumor samples (B and D) using normalized miRNA (A and B) or mRNA (C and D) expression data, respectively.(TIF)Click here for additional data file.Figure S3Mean specificities and sensitivities for the colon and prostate cancer data. Models for the colon collective (A) and prostate collective (B) based on our approach, CAPE RNA, were generated from training sets by selecting all interactions with a Jaccard-index equal to or higher than a threshold. Prediction Analysis of Microarrays for R was used to train models for the colon tissue samples based on miRNA (C) and mRNA expression (D), as well as for the prostate collective based on miRNA (E) and mRNA (F) expression. Models were generated with different thresholds. A-F) illustrate the mean specificities and mean sensitivities to classify unknown test sets.(TIF)Click here for additional data file.Figure S4Mean specificities and sensitivities for the bladder cancer data set by randomized models. Models were generated by randomly picking a defined of interactions within a specific range of Jaccard-indexes using our approach, CAPE RNA. The performance of different models was compared to classify unknown test sets: a) specificities and b) sensitivities to discriminate tumor samples from healthy tissue samples, c) specificities and d) sensitivities to discriminate invasive tumor samples from non-invasive tumor samples.(TIF)Click here for additional data file.Figure S5Venn diagram of predicted miRNA-mRNA interactions in bladder cancer derived from five different methods for integrative analysis . Two different collectives were analysed: (A) the entire bladder cancer dataset of normal (n = 8) and tumor samples (n = 16) and (B) only the collective of invasive (n = 8) and non-invasive (n = 8) bladder cancer tumor samples. The number of all predicted miRNA-mRNA interactions by each method were visualized in the Venn diagram.(TIF)Click here for additional data file.Table S1List of all selected differentially regulated interactions and their Jaccard-indexes for all samples for the bladder cancer data set. Only, interactions were selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group.(CSV)Click here for additional data file.Table S2List of all selected differentially regulated interactions and their Jaccard-indexes for non-invasive and invasive bladder tumor samples. Only, interactions were selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group.(CSV)Click here for additional data file.Table S3Spearman correlation coefficients of all interactions for all samples of the bladder cancer data set. Only, interactions were selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group.(CSV)Click here for additional data file.Table S4Spearman correlation coefficients of allinteractions for non-invasive and invasive bladder cancer tissue samples. Only, interactions were selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group.(CSV)Click here for additional data file.Table S5Functional annotation clustering of selected mRNAs using DAVID v6.7. Starting with the whole tissue collective  we obtained 2480 miRNA-mRNA interactions with a JI > = 0.4. The corresponding 1459 mRNAs of these interaction pairs were analyzed by DAVID software with standard classification stringency. 28.4% (414 mRNAs) of the analyzed mRNAs could be matched into the Kegg pathways.(PDF)Click here for additional data file.Table S6Differentially regulated interactions for all samples inside the FGF3R pathway. Only, interactions were selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group. Interactions with a Jaccard-index ≥0.40 are shown. The regulation in bladder cancer tissue samples compared to normal tissue samples is indicated.(PDF)Click here for additional data file.Table S7Differentially regulated interactions for all samples inside the bladder cancer pathway. Only, interactions are selected that exhibit a negative correlation, i.e. ρ≤−0.4, between the normalized miRNA and mRNA expression values for at least one experimental group. Interactions with a Jaccard-index ≥0.40 are shown. The regulation in bladder cancer tissue samples compared to normal tissue samples is indicated.(PDF)Click here for additional data file.TableS8List of the shared interactions between our approach CAPE_RNA and each of the other used prediction methods. Two different collectives were analysed: the entire bladder cancer set of normal (n = 8) and tumor samples (n = 16) and the collective of invasive (n = 8) and non-invasive (n = 8) bladder cancer tumor samples without normal samples. Each of the 500 best interactions were compared. A) Intersection TaLasso - CAPE RNA; B) Intersection GenMiR++ - CAPE RNA; C) Intersection Spearman correlation - CAPE RNA: D) a Meta analysis approach – CAPE RNA.(CSV)Click here for additional data file.