ROTS has already been successfully applied in a range of different studies from transcriptomics to proteomics, showing competitive performance against other state-of-the-art methods. To promote its widespread use, we introduce here a Bioconductor R package for performing ROTS analysis conveniently on different types of omics data. To illustrate the benefits of ROTS in various applications, we present three case studies, involving proteomics and RNA-seq data from public repositories, including both bulk and single cell data. The package is freely available from Bioconductor (https://www.bioconductor.org/packages/ROTS).Differential expression analysis is one of the most common types of analyses performed on various biological data (e.g. RNA-seq or mass spectrometry proteomics). It is the process that detects features, such as genes or proteins, showing statistically significant differences between the sample groups under comparison. A major challenge in the analysis is the choice of an appropriate test statistic, as different statistics have been shown to perform well in different datasets. To this end, the reproducibility-optimized test statistic (ROTS) adjusts a modified  PLOS Computational Biology Software paper.”“This is a t-test, which has been later shown not to be the most optimal solution in many cases [a priori.Differential expression analysis between two groups of samples is perhaps the most common type of analysis that is performed on various types of omics data. The aim of differential expression analysis is to detect features (e.g. genes or proteins) showing statistically significant changes between the groups. A commonly used approach has been the Student’s ny cases , 2. Althny cases –5, a majny cases –9 and tht-type statistics. In particular, ROTS optimizes the ranks of the features because usually the final ranking determines if the differentially expressed features are selected for further validation studies.To address this need, we have introduced a reproducibility-optimized test statistic (ROTS) that optimizes the choice of the statistic directly from the data instead of using a fixed predefined statistic . This isThe ROTS method has already been used in various applications, such as microarrays , mass spt-statistics:α1 and α2 are non-negative parameters to be optimized, and s is the pooled standard error [t-statistic  and the signal log-ratio . The optimal statistic is determined by maximizing the reproducibility Z-score:α1 ∈ {0, 0.01, …, 5}, α2 ∈ {0, 1}, k ∈ {1, 2, …, G}. Here, Rk (dα) is the observed reproducibility of statistic dα at top list size k in bootstrap datasets, sk (dα) is the standard deviation of the bootstrap distribution, and G is the total number of features in the data. Reproducibility is defined as the average overlap of k top-ranked features over pairs of bootstrapped datasets.ROTS optimizes the reproducibility of top-ranked features in group-preserving bootstrap datasets among a family of modified rd error . Specialα1 and α2 giving the highest reproducibility Z-score. The false discovery rate (FDR) is estimated by randomly permuting the sample labels.The final ROTS output is calculated from the original data using the optimized parameters https://www.bioconductor.org/packages/ROTS) and runs in R environment. Both the package and the R environment can be used on Windows, MacOS or UNIX platforms. After installing the package, the differential expression analysis can be performed within R. A preprocessed and appropriately normalized data matrix is required for input with columns representing different samples and rows representing the features. It can be various types of omics data, such as gene expression microarray data, RNA-seq data or mass spectrometry based proteomics data. As an example, an excerpt from a typical expression matrix of a label-free proteomics study is shown in The ROTS package is freely available from Bioconductor (data), only one line of code is required after loading the package:To perform differential expression analysis on an expression matrix (here library(ROTS)rots.out <- ROTS, B = 1000, K = 500)groups defines the columns of the data matrix belonging to the two different sample groups under comparison, B denotes the number of bootstraps to perform, and K is the maximum top list size to consider in reproducibility optimization. Setting this number to a smaller value may improve the running time drastically. However, we recommend that the value should always be considerably higher than the number of features expected to be differentially expressed.Here the vector rots.out) contains the test statistics and additional details for all the features in the input data, including the optimized parameters a1 and a2. If the reported top list size k (rots.out$k) is close to the given parameter K, it suggests that the maximum top list size to be tested might have been too small, and increasing it should be considered. The reproducibility value (rots.out$R) and the reproducibility Z-score (rots.out$Z) are also included. All the results including p-values , false discovery rates (rots.out$FDR) or fold changes (rots.out$logfc) can be exported by the user and used for further external analysis, such as gene-set or pathway enrichment analysis.The generated ROTS object (here plot function. The type of plot can be selected using the type parameter of the function. type=‘volcano’), which visualizes the relationship between fold changes and p-values . It can be used to select the most promising candidate features for further validation studies. type=‘ma’), which shows the relationship between the average intensities (A) and intensity ratios (M) calculated across and between the sample groups for each feature, respectively. It can be used, for instance, to assess the quality of normalization used in preprocessing the data. k (type=‘reproducibility’). It can be used to look for possible alternative peaks of Z-score, which could suggest, for example, subgroups of differentially expressed features or artifacts from data normalization. p-values , which enables assessing the overall performance of the hypothesis testing. Under the null hypothesis, p-values are uniformly distributed, but if there is a large number of differentially expressed features present, the distribution of p-values is likely skewed towards smaller values. type=‘pca’). It is a transformation, where the data is projected into a new coordinate system of principal components retaining the highest variance. It can be used as a tool to evaluate similarities between samples or groups. By setting the FDR parameter to 1, the principal components are calculated using all the features. type=‘heatmap’).The ROTS package includes also versatile built-in options for visualization that can be accessed using the standard R For additional details and examples of using the ROTS package, the reader is referred to the package manual and the three case studies discussed below.The benefits of ROTS over other state-of-the-art tools have already been shown in various applications –12. HereThe ROTS method has previously been benchmarked in label-free shotgun proteomics using spike-in mixtures and complex mouse liver samples, where it has shown competitive performance against other state-of-the-art methods . Here, tS. cerevisiae) background at different concentration levels ranging from 0.25 to 20 fmol/μL to create five distinct sample groups each with three technical replicates. From the different datasets available, we processed Orbitrap raw files produced at site 86, from which a total of 736 proteins were quantified using the Progenesis software with peptide identifications from the Mascot search algorithm in Proteome Discoverer software. Threshold for peptide identifications was set to FDR < 0.01 and relative protein quantitation was done using non-conflicting peptides, followed by median normalization. Progenesis was unable to align one of the three replicates in one of the sample groups (sample group E). Only the sample groups with all three replicates (sample groups from A to D) were used here for performance benchmarking.The data are from an inter-laboratory spike-in study of the Clinical Proteomic Tumor Analysis Consortium (CPTAC technology assessment study 6) –16. It ct-test. Performance was measured using receiver operating characteristic (ROC) curves, which were created by merging the results from the six possible individual pairwise comparisons involving sample groups from A to D. Overall, ROTS produced a significantly better ROC-curve compared to all other tested methods (DeLong’s test p < 0.001 for each method), which supports the applicability of ROTS in proteomics studies to distinguish differentially expressed proteins.Similarly as with proteomics data, the ROTS method has been extensively benchmarked against other software packages in bulk RNA-seq data . BesidesThe data are from the sequencing quality control (SEQC) project , which ip < 0.001 for each method), confirming the applicability of ROTS in bulk RNA-seq studies to detect differentially expressed genes.Recently, performance of the ROTS method in comparison to other state-of-the-art methods has also been tested in single-cell RNA-seq data. ROTS showed good performance without requiring any single-cell-specific modifications, whereas no systematic benefits of the recent single-cell-specific methods were found . Here, wThe data are from a previously published single-cell study on innate lymphoid cells (ILC), containing single cell samples sequenced using Illumina HiSeq 2000 platform . Similart-statistic, it is possible to further extend the method by allowing multiple sample groups by using, for example, a modified F-statistic. Finally, to enhance the running time of the algorithm, parallelization within the package or improved heuristics could be implemented for optimizing the parameters.ROTS has been successfully applied in multiple studies in a diversity of applications and the results on different types of omics data have shown its overall robustness. A major benefit of ROTS is its ability to automatically select an appropriate test statistic for a specific data under study by maximizing the reproducibility of the differentially expressed features. Therefore, it would be beneficial to integrate ROTS into various existing workflows to perform the differential expression analysis. Besides being able to select a test statistic, ROTS could possibly aid also in selecting, for instance, an appropriate normalization method based on the data. While ROTS is based on a modified https://www.bioconductor.org/packages/ROTS) and it conveniently allows to perform statistical testing and result visualization using simple commands. A complete reference manual for the package and a vignette with examples are also available from Bioconductor.The R package ROTS is freely available from Bioconductor (


We applied two state-of-the-art, knowledge independent data-mining methods – Dynamic Quantum Clustering (DQC) and t-Distributed Stochastic Neighbor Embedding (t-SNE) – to data from The Cancer Genome Atlas (TCGA). We showed that the RNA expression patterns for a mixture of 2,016 samples from five tumor types can sort the tumors into groups enriched for relevant annotations including tumor type, gender, tumor stage, and ethnicity. DQC feature selection analysis discovered 48 core biomarker transcripts that clustered tumors by tumor type. When these transcripts were removed, the geometry of tumor relationships changed, but it was still possible to classify the tumors using the RNA expression profiles of the remaining transcripts. We continued to remove the top biomarkers for several iterations and performed cluster analysis. Even though the most informative transcripts were removed from the cluster analysis, the sorting ability of remaining transcripts remained strong after each iteration. Further, in some iterations we detected a repeating pattern of biological function that wasn’t detectable with the core biomarker transcripts present. This suggests the existence of a “background classification” potential in which the pattern of gene expression after continued removal of “biomarker” transcripts could still classify tumors in agreement with the tumor type. These databases are deep and diverse, containing data ranging from DNA sequence to epigenetic state to dynamic gene expression output. TCGA, for example, contains multiple measurements for over 14,000 tumors of multiple types. GTEx contains the gene expression patterns and matching DNA sequences for over 11,000 human tissue samples. dbGaP contains over 2.4 million molecular assays relevant to human disease and phenotypes. Clearly, there is massive opportunity to mine these and future databases for biological insight.Dozens of public genomic data repositories relevant to human biology have emerged to support biomedical science. These repositories include The Cancer Genome Atlas (TCGAm × n gene expression matrices (GEMs) where rows (m) are gene or RNA transcript identifiers and columns (n) are samples. One can think of a GEM as a compendium of molecular snapshots from different points of anatomy, developmental stage, and environment. While a liver gene vector, for example, has the same genes as a neighboring pancreas gene vector, their gene expression intensities vary in a manner reflective of their underlying biology.A powerful quantitative measurement of genome information flow within a biological sample is the steady state RNA-based gene expression profile; this profile is captured in a gene expression vector representing the number of RNA molecules produced by tens of thousands of genes in the specimen. A researcher can aggregate profiles from multiple samples retrieved under varied biological contexts into 6. Correlation analysis can be performed to identify genes whose expression patterns are synchronized across samples using software such as WGCNA7 and KINC8. Similar gene expression patterns are predicted to have a common biological purpose. However, the brute force interrogation of all samples in a GEM may not be the best approach to detect context-specific gene interactions; overrepresentation of one biological condition might drown out rare gene interactions. One approach to addressing variation between samples involves sorting the gene expression vectors into sample clusters of similar contexts. Thus, liver expression profiles might produce one sample cluster, as will pancreas profiles, etc. Correlation analysis can be performed in sorted sample clusters. Sample sorting is becoming increasingly relevant given the rapid growth of samples in databases that are being interrogated for biological function. If a sample sorting technique can robustly sort mislabeled or outlier samples in to alternate groups, then noise should be reduced for each group possibly making it easier to identify meaningful biomarkers and pathways.GEMs can be normalized to reduce technical variation between biological samples9.Two approaches exist for clustering GEMs into sample groups of similar global gene expression patterns: knowledge-dependent and knowledge-independent methods. Both can increase the probability of detecting gene expression patterns relevant to the sorted biological contexts. Knowledge-dependent methods sort the GEM into sub-GEMs based upon annotations associated with the samples. Thus, one could prepare a mixed knowledge-independent condition GEM from a data repository like TCGA and then sort the GEM into tumor types prior to analysis. This approach, while logical – the metadata associated with TCGA is well curated – weakens if the sample label is assigned incorrectly or is representative of multiple subgroups. This can happen, for example, when a tumor type is incorrectly annotated or when tumors are related by molecular architecture as opposed to tissue of origin12. Sample metadata is assigned after sorting to provide conditional context to the clusters. One sample cluster might be enriched for a specific biological context label, such as a tumor type, and any genetic relationships from that cluster can be associated with that label. A drawback to k-means clustering is that the number of clusters should be known beforehand, resulting in bias in the sample grouping if an inappropriate number of distinct clusters is chosen.A less-biased, knowledge-independent approach uses clustering methods to sort datasets into groups based only upon their global expression pattern. For example, k-means clustering of global gene expression profiles sorts samples into a pre-defined number of groups and has been shown to improve gene-gene interaction detection14) – like most knowledge independent sample clustering approaches – relies upon strongly reducing the dimensionality of the gene expression space prior to sample comparison. Two common algorithms used to perform this task are principal component analysis (PCA15) and singular value decomposition (SVD15). Typically, this machine learning technique projects high dimensional data into two or three dimensions. It should be noted that t-SNE has been applied to GTEx16 and TCGA17 datasets, where the TCGA study used t-SNE as part of an integrated omics sorting workflow called MEREDITH17.There are emerging knowledge-independent clustering methods that can be applied to GEMs and that do not introduce as much bias. One method, t-distributed stochastic neighbor embedding (t-SNE18), unlike other clustering approaches, does not need to use strong dimensionality reduction to analyze high-dimensional data; however, it is common in a DQC analysis to use modest SVD-based dimensionality reduction to speed up the analysis. DQC begins by replacing each column of a GEM  by an analytic function in m-variables; specifically, each column of the GEM is replaced by a Gaussian function centered on the m-dimensional location specified by the corresponding gene expression vector. The sum of these functions is then used to create a potential function, V, that is a proxy for the density of the data in feature space19. By construction, the local minima and saddle points of this potential function represent regions of higher local density of the data. This potential function is then used to create a Hamiltonian operator as Equation , is then evolved according to the corresponding Heisenberg equation of motion20 in Equation , glioblastoma multiforme (GBM), ovarian serous cystadenocarcinoma (OV), urothelial bladder cancer (BLCA), and papillary thyroid carcinoma (THCA). We discuss the clustering and biomarker discovery potential using t-SNE and DQC approaches. In addition, we examine the effect of salient biomarker removal and the ability of both techniques to continue to classify the tumors into meaningful groups in the absence transcripts with high classification potential. When applied to deeply sequenced tumors, our approach can be used to detect biomarker combinations that sort tumor types without prior knowledge of where the tumor was initiated.18 and compared it to an approach that performs strong dimensional reduction, t-SNE13. To do this, we constructed a mixed tumor GEM consisting of 2,016 samples from TCGA1. Specifically, we mixed tumor expression profiles from five TCGA labeled sample groups: bladder cancer , glioblastoma , lower grade glioma , ovarian carcinoma , and thyroid cancer . It should be noted that some of the groups contained low numbers of non-tumor samples with the same label as tumor type . This GEM was used as the input for both the DQC and t-SNE analyses  to discover 48 “core transcripts”  for at least one tumor type label. Clusters 6, 7, and 8 were enriched for both GBM and LGG tumor type labels. Thyroid tumor clusters were enriched for the female gender. One of the bladder tumor clusters and two of the mixed brain tumor clusters were enriched for the male gender. The bladder and ovarian tumor clusters were uniformly enriched for age above 40 years, while several of the thyroid and brain tumor clusters were enriched for age below 40. The age enrichment corresponded roughly to stage enrichment: both thyroid clusters were enriched for tumors in their earliest stage while the mixed BLCA-OV clusters 2, 3, and 4 (where data was available) were enriched for stage IV tumors. Little enrichment was observed based on race or ethnicity although thyroid-enriched cluster 0 was also enriched for “Asian” as a race. The mixed GBM-LGG tumor cluster 6 was enriched for the annotation term “white”. It is worth noting that we did not attempt to correct for bias that may exist within the TCGA database itself with regard to sample collection.i.e. the one obtained from the original GEM by removing the 48 core transcripts) can still be used to successfully segregate the tumors into five groups via either t-SNE or DQC. Moreover, the DQC analysis once again successfully separated the GBM tumors from the LGG tumors. So, we see that – at least for the case of glial cell tumors – information about tumor type and not just tissue of origin is being encoded in the reduced GEM. Still, brain tumors aside, the observed DQC and t-SNE substructure both distinguished the other tumors. Why would classification occur in the absence of the core biomarkers? It is well known that hundreds to thousands of genes might be differentially expressed between tumor types. In the case of GBM and LGG, it has previously been reported that 2275 genes are differentially expressed between these tumor types21. Therefore, one would expect several thousand transcripts to have a combined sorting ability as the gene set without core biomarkers is still likely embedded with differentially expressed genes. However, we were curious about the effect of deeper biomarker removal and the effect on sample sorting. Thus, we performed successive rounds of DQC-based feature (i.e. important transcripts) selection and removal.As can be seen in Fig. Since this part of the analysis was only focused on how systematic removal of features would affect the ability to separate tumors, we decided to expedite the analysis by limiting DQC-based feature selection to the first 12 SVD dimensions to identify the next layer of important transcripts. For these 12 eigenvectors, we plotted the sorted absolute values of their components and set a threshold based upon breaks in the plot. These thresholds steadily decreased as we repeated the process of important transcript selection and removal. At each iteration, we checked that the new set of important transcripts alone could still separate the glial cell tumors by diagnosis Fig.  and not Remaining transcripts after iterative removal of important transcripts were studied by both DQC and t-SNE . We expected that these dimensions would confer biological classification potential which in this study was the separation of tumor types without prior knowledge of the sample origin. To achieve our goal, we tested two approaches, DQC and t-SNE, both of which grouped samples into clusters that made biological sense based on sample annotation enrichment.22. Clusters 1, 6, 8 showed a male bias and clusters 6 and 8 are associated with brain cancer; this finding corroborates the male bias seen in brain tumor incidence23. None of the brain tumor samples were annotated as normal. With a deeper attribute dataset, we believe this approach would reveal higher quality sample groups and even more insight for the exploration of tumor biology.The biological relevance of the tumor clusters was evidenced by enrichment of annotations relative to all tumors. The consensus clustering technique applied to one thousand t-SNE embeddings revealed nine tumor subpopulations Figs  and 4. F24) versus 7 years (LGG25).While both t-SNE and DQC performed well in clustering the tumors, there were some interesting differences. DQC revealed interesting patterns that were not observed with t-SNE. For example, the DQC “brain arch” driven by the 48 core transcripts revealed an interesting substructure connecting LGG and GBM brain tumors Fig. . When weet al. identified eight transcriptional superclusters using unsupervised hierarchical clustering of expression profiles between tumor sub-types across twelve TCGA tumor types26. In contrast to our approach, their study analyzed the top 1500 genes and used prior knowledge of tumor type in their analysis whereas our study input the full GEM and sorted tumors without using tumor type knowledge in the sorting process. Li et al. examined TCGA RNAseq data using a classification strategy where they classified 9096 tumor samples from 31 tumor types using GA/KNN as classification engine27. That classification study used prior knowledge of tumor type in the classification process whereas our study was blind to the sample labels. In Hoadley et al., a cluster-of-cluster (COCA) technique was used on RNAseq profiles (and other tumor molecular measurements) to determine a molecular taxonomy of 12 cancer types, discovering 11 molecular signature-based subtypes9. Their study showed that tissue of origin labels was not always indicative of the tumor’s molecular basis. In contrast to our study, they only used the 6000 most variable genes in their analysis. It is also typical to use many differentially expressed genes to characterize tumor type or subtype. Ceccarelli et al., for example, used 2275 differentially expressed genes to build “molecular profiles” of LGG and GBM tumors21 and Verhaak et al. identified 840 genes predictive of subtype in the case of GBM tumors alone28. We are aware that the 48 core transcripts used in this analysis are probably a subset of such a larger population of classifier genes.This is not the first study to identify biomarkers and classify TCGA tumor types. Martinez Our results confirm there are two means of classifying samples: (1) using a small number of transcripts of high significance or 2) many transcripts of low relative significance. By examining the functions of genes in the first iteration of DQC-based transcript selection  which would imply tissue specificity as opposed to a tumor type property.While the “background classification” effect suggests that a sufficiently large set of random gene expression vectors has classification potential, it seems unlikely that all these genes would be involved in tumor specific biology. We assumed that through successive levels of DQC-based transcript selection we would remove differentially expressed genes and detect essentially random genes without collective function that merely contain the background classification potential. In fact, we did find that as we continued this process, the number of pathways we detected decreased after correcting for the number of transcripts found in the pathway Fig. . However19.In conclusion, we describe and contrast two very different sample clustering algorithms: DQC and t-SNE. We implemented dimensionality reduction to identify the important transcripts and all subsequent DQC analysis was done without further dimensionality reduction. t-SNE analysis involved further embedding into two spatial dimensions. Both techniques are effective at clustering samples to detect substructures in a GEM. The fact that DQC worked in the full 48-dimension space of core transcripts is likely the reason it reveals more subtle aspects of the data. Unexpectedly, we discovered the confounding effect many random transcripts can classify and sort samples. We also repeatedly showed that this ability is not merely identifying tissue of origin as opposed to tumor type. This is an early but intriguing concept that should be addressed if a researcher seeks cause-and-effect as opposed to tumor type-associated biomarkers. Finally, while we applied these techniques to tumor data, the same approach can be applied beyond genomics contexts, a fact that has been previously shown for DQChttps://gdc-portal.nci.nih.gov. A total of 2,016 datasets were obtained comprised of the types as labeled by TCGA: BLCA (n = 427), GBM (n = 174), LGG (n = 534), OV (n = 309), and THCA (n = 572). Each expression profile was merged into a single gene expression matrix (GEM) with 73,599 transcripts labeled with knowngene5 UC-Santa Cruz genome database gene model identifiers. It is important to address potential batch effects – technical and biological variation between samples of the same group – in a high throughput genomics study6. The TCGA Batch Effects webserver (http://bioinformatics.mdanderson.org/tcgambatch/) was queried to gain insight on batch effects present in each cancer subtype. It was found that the Dispersion Separability Criterion (DSC) scores for the RNAseqv2 isoform data for each cancer subtype indicate a mild presence of batch effects in the data used in our study (p < 0.0005). The DSC scores for BLCA, GBM, LGG, OV, and THCA are 0.310, 0.000, 0.298, 0.089, and 0.252, indicating a low ratio of dispersion between vs. within batches for these cancer types. Lauss et al. found that performing a quantile normalization on colon cancer RNAseqv2 data from TCGA helped to reduce batch effects present in this data6. We performed a similar quantile normalization on the GEM used in our study. Furthermore, we did not detect any outliers using a Kolmogorov–Smirnov test as performed in8. The TCGA GEM was quantile normalized and randomly sorted to create a single tumor GEM for input into the DQC and t-SNE pipelines. A small fraction of the subtype labeled samples  included “Solid Tissue Normal” samples. These and other clinical annotations associated with each TCGA sample are included in Supplemental Table RNAseq profiles, rsem processed at the transcript level, for five public tumor types were downloaded on April 1, 2016 from the TCGA Data portal at LGG n = 54, OV . This separation allows for a novel form of feature selection where we select a subset of the RNA transcripts that play the most important role in the evolution of the data.m × n GEM  as Equation  using the Python implementation from https://github.com/DmitryUlyanov/Multicore-TSNE. For each t-SNE run, one thousand two-dimensional randomly initialized embeddings were created. Each embedding was clustered individually using Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN30). A consensus of nine clusters was determined over the whole set of clustered embeddings using the Cluster Ensembles method31. Label enrichment of these consensus clusters for patient attributes associated with the tumor samples was evaluated using a Chi-squared test (p < 0.001).The full or partial GEM was evaluated separately by a dimensionality reduction (embedding), clustering, consensus, and enrichment pipeline. Embedding was performed using t-Distributed Stochastic Neighbor Embedding 34, the Kyoto Encyclopedia of Genes and Genomes (KEGG)35, Reactome36 and MIM37. Terms that were present in a gene list more often than in the genomic background were considered enriched (FDR <0.01). The full enrichment list is shown in Supplemental Table Functional enrichment of the core and subsequent iterations of transcripts was performed using an in-house Perl script modeled after the online DAVID tool at Supplemental Video 1Supplemental Video 2Supplemental Table 1Supplemental Table 2Supplemental Table 3


RNAs (miRNAs) belong to a family of small non‐coding RNAs (sncRNAs) playing important roles in human carcinogenesis. Multiple investigations reported miRNAs aberrantly expressed in several cancers, including high‐grade serous ovarian carcinoma (HGS‐OvCa). Quantitative PCR is widely used in studies investigating miRNA expression and the identification of reliable endogenous controls is crucial for proper data normalization. In this study, we aimed to experimentally identify the most stable reference sncRNAs for normalization of miRNA qPCR expression data in HGS‐OvCa. Eleven putative reference sncRNAs for normalization  were analysed on a total of 75 HGS‐OvCa and 30 normal tissues, using a highly specific qPCR. Both the normal tissues considered to initiate HGS‐OvCa malignant transformation, namely ovary and fallopian tube epithelia, were included in our study. Stability of candidate endogenous controls was evaluated using an equivalence test and validated by geNorm and NormFinder algorithms. Combining results from the three different statistical approaches, SNORD48 emerged as stably and equivalently expressed between malignant and normal tissues. Among malignant samples, considering groups based on residual tumour, miR‐191‐5p was identified as the most equivalent sncRNA. On the basis of our results, we support the use of SNORD48 as best reference sncRNA for relative quantification in miRNA expression studies between HGS‐OvCa and normal controls, including the first time both the normal tissues supposed to be HGS‐OvCa progenitors. In addition, we recommend miR‐191‐5p as best reference sncRNA in miRNA expression studies with prognostic intent on HGS‐OvCa tissues.Micro Epithelial ovarian cancer is the fifth most common type of cancer in women and the leading cause of mortality for gynaecological neoplasms, with high‐grade serous carcinoma (HGS‐OvCa) being the most frequent and aggressive histological type Currently, a proper endogenous control‐based normalization strategy to be used in miRNA qPCR studies on cancer tissues has not been identified To fill this gap of knowledge, the present investigation aims at identifying the most suitable sncRNAs as the endogenous controls for miRNA expression normalization in a wide and well‐characterized cohort of HGS‐OvCa tissues. Furthermore, as HGS‐OvCa histogenesis is still a matter of debate The candidate endogenous sncRNAs were chosen based on the literature suggesting their use for normalization of qPCR studies in HGS‐OvCa tissue samples, carrying out a Medline search using the MeSH terms ‘ovarian cancer’/‘ovarian carcinoma’ and ‘microRNA’/‘miRNA’ and ‘real‐time PCR’. In addition, endogenous miRNA recommended from Exiqon , based on their stable and constitutive expression across different cells and tissues, were included in our study This study was performed following the Declaration of Helsinki set of principles and approved by the Research Review Board‐ the Ethic Committee‐ of the Spedali Civili, Brescia, Italy (study reference number: NP1676). Written informed consent was obtained from all patients enrolled. HGS‐OvCa tissue samples were obtained from 75 patients diagnosed and treated at the Division of Gynecologic Oncology of the University of Brescia , between 2003 and 2013. Normal control tissue samples were obtained from a total of 30 patients, undergoing surgery for benign pathologies , using TRIZOL reagent  and further purified using RNeasy MiniElute Cleanup kit  with a modified protocol for co‐purification of small RNAs according to the manufacturer's instructions. RNA concentration and integrity was assessed as previously described Stability of candidate endogenous miRNA was evaluated using two different software programs commonly used in the experimental validation of reference genes, NormFinder P values and confidence intervals (CIs) estimation based on ‘White‐Huber’ heteroscedasticity corrected covariance matrices L,εU] is defined. The null hypothesis is set up so that if the 90% CI for the parameter of interest (e.g. the difference among group means) falls completely within the equivalence range, the null hypothesis can be rejected. Two one‐sided tests were conducted for both boundaries of the range. The overall null hypothesis is rejected at level α if the associated P‐value for each of the individual hypotheses is less than α . Prior to conducting an equivalent test, we must define an equivalence range, in which we can consider the parameter of interest in the two groups to be substantively equal. While no fixed objective rules exist to guide the choice of the equivalence range because such choice may depend on substantive considerations, Wellek t‐test of ±0.36 on log scale, and we decided to adopt this latest criteria in our data analysis.Differences in sncRNA expression between groups were tested using linear models on log‐transformed sncRNA expression values, with robustbaseStatistical analysis were performed with R According to our Medline search criteria, we found 114 papers published from November 2007 to August 2015. Within these reports, we removed from analysis papers evaluating gene expression and genotyping studies, as well as miRNA expression studies performed on cell cultures, formalin‐fixed paraffin‐embedded tissues or plasma/serum/urines. Additional articles were excluded because full text was not available in the English language. The remaining 29 papers focused on miRNA expression evaluation in fresh frozen ovarian tumour tissues by real‐time PCR, using the following sncRNAs for data normalization, alone or in combination of two or three: U6 (25 times), SNORD48 (two times), miR‐92a‐3p (one time), let‐7a‐5p (one time), SNORD61 (one time), SNORD72 (one time), SNORD68 (one time). This panel of seven potential endogenous controls derived from the literature was integrated with four additional reference miRNAs, miR‐103a‐3p, miR‐423‐3p, miR‐191‐5p and miR‐16‐5p, selected and validated from Exiqon for normalization of miRNA expression levels in human tissue samples during real‐time PCR analysis (Table S1).Quantitative real‐time PCR was performed on 75 HGS‐OvCa, 15 normal luminal fallopian tube and 10 normal ovarian surface epithelia samples (representative of 15 patients) to assess the expression pattern of the 11 selected reference sncRNAs. All RNA samples isolated from malignant and normal specimens met the criteria of purity and integrity defined by A260/280 ratio  and RIN values (RIN ≥ 8). All RNA samples were verified to be free of any DNA contamination, by analysis of minus‐reverse transcriptase (‘‐RT’) controls in real‐time RT‐PCR experiments (Table S2). For the quantitative comparison of investigated candidate reference sncRNAs, the raw Cq values were converted to CNRQ, a normalization procedure which removes variations related to different cDNA starting quantities and correct for run‐to‐run variation using an internal control sample L,εU] range, if the CI is fully included in the equivalence range the null hypothesis of non‐equivalence can be rejected. Table P values as well as an indicator of null hypothesis rejection based on |ε| = 0.36. Accordingly, considering the difference in sncRNA expression between malignant and non‐malignant samples, all sncRNA were substantially varying but SNORD48, whose ratio of the two groups averages  falls within the fixed equivalence range. Moreover, we performed the same analysis of statistical equivalence considering only the tumour sample cohort, grouped based on residual tumour (RT). Among malignant samples considering groups based on RT (RT = 0 versus RT > 0), miR‐191‐5p emerged as the most equivalent sncRNA .Applying the TOST method equates at computing 1‐2α)% CI (90% when α = 0.05) for the contrast of interest (Δ = difference between group means) and comparing it to the [εα% CI  to the most stable . As an M value of 1.5 was defined as the upper limit for candidate reference, and 0.5 was the M value typically observed for stably expressed reference genes in homogeneous sample groups, values between 0.5 and 1 were considered as a characteristic when evaluating endogenous controls on heterogeneous samples like cancer and normal tissues http://cancergenome.nih.gov/). RNA‐Seq data regarding 292 stage III‐IV HGS‐OvCa snap‐frozen tissues were available for analysis, with similar clinic‐pathologic characteristics compared with our cohort of samples. Among malignant samples, considering groups based on RT (RT = 0 versus RT > 0), miR‐191‐5p was confirmed to be the most equivalent sncRNA , as shown in Table To validate the stability of our candidate invariant sncRNAs, we performed the same analysis of statistical equivalence on RNA‐Seq data, based upon level 3 data generated by the TCGA research network  and tissue composition  was provided. Due to this lack of information, we were prevented to perform the analysis of statistical equivalence on healthy tissue RNA‐Seq data.Conversely, only six normal fallopian tube and four normal ovary RNA‐Seq data were present in the TCGA database and no information about sample collection  wide cohort of snap‐frozen tumour tissue samples of HGS histological type, the most frequent and aggressive ovarian cancer, belonging to a single institution, (ii) inclusion of both ovarian surface and fallopian tube epithelia, as source of normal control tissues, (iii) stringent quality control of isolated total RNA, (iv) careful selection of putative reference sncRNAs, combining a Medline search result and validated endogenous controls from Exiqon, (v) use of optimized Exiqon primer sets with LNA technology, maximizing sensitivity and specificity in detecting sncRNA amplicons, (vi) qPCR performed with SYBR Green technology, that is cost‐effective and easy to apply in every laboratory setting, (vii) use of an inter‐run calibration sample, that showed an optimal correlation in sncRNA expression among all plates and (viii) experiments performed in triplicate for every sncRNA and every sample.To date, no consensus strategy has been reached on the optimal normalization for miRNA expression studies, and irrespective of the guidelines suggesting a validation screening test on a subset of samples under analysis, endogenous references continue to be arbitrarily chosen without any experimental evidence. To our knowledge, no validated endogenous references have been identified for normalization of miRNA expression data in ovarian cancer tissues. Confirming this report, our comprehensive MEDLINE search of miRNA expression studies published between 2007 and the middle of 2015 showed that there was no uniform opinion on which reference should be used for reliable normalization in HGS‐OvCa samples. On the basis of the literature review, a total of seven sncRNAs were identified as putative references for miRNA expression studies in ovarian cancer tissues. Almost all of the published studies (25 out of 29) reported U6 as the reference of choice, while other sncRNAs, such as SNORD48, miR‐92a‐3p, let‐7a‐5p, SNORD61, SNORD72 and SNORD68, have been evaluated alone or in combination with U6 in the remaining papers. To increase the number of potential endogenous sncRNAs to be evaluated in our cohort of 75 HGS‐OvCa and 30 normal tissues, we decided to adopt a combination strategy integrating data from the literature with the following panel of reference sncRNAs: miR‐103a‐3p, miR‐423‐3p, miR‐191‐5p and miR‐16‐5p. They have been reported to be constitutively expressed in a variety of normal and pathological tissues, and validated from Exiqon for normalization of miRNA expression levels during real‐time PCR analysis. None of these potential endogenous controls had been previously evaluated in ovarian tissue samples. The particular design of our study has been characterized by several features, including:  Substantial contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (ii) Drafting or revising the article for intellectual content; and (iii) Final approval of the published article.All authors confirmed that they have contributed to the intellectual content of this paper and have met the following three requirements: (Table S1 Characteristics of candidate sncRNAs selected for evaluation of expression stability.Click here for additional data file.Table S2 Median and range of Cqs with and without reverse transcriptase for each sncRNA analysed.Click here for additional data file.


We compared five statistical methods to detect differentially expressed genes between two distinct single-cell populations. Currently, it remains unclear whether differential expression methods developed originally for conventional bulk RNA-seq data can also be applied to single-cell RNA-seq data analysis. Our results in three diverse comparison settings showed marked differences between the different methods in terms of the number of detections as well as their sensitivity and specificity. They, however, did not reveal systematic benefits of the currently available single-cell-specific methods. Instead, our previously introduced reproducibility-optimization method showed good performance in all comparison settings without any single-cell-specific modifications. Gene expression profiling has traditionally focused on bulk populations of millions of cells, measured either using microarrays or next-generation sequencing technologies. Although an average expression level for each gene in the cell population can be sufficient in many applications, such as determining disease biomarkers, bulk RNA-seq analysis lacks the detail of cell-specific functionality. Consequently, with the rapid technological developments, single-cell RNA-seq (scRNA-seq) has quickly grown into a popular field of RNA-sequencing, enabling novel biological discoveries such as detecting novel cell types with distinct expression signatures, or understanding of the stochasticity of gene expression in a cell population .To effectively use the scRNA-seq data, it is crucial to use appropriate tools for the data analysis. Although several sophisticated methods are already available for analyzing bulk RNA-seq data , their det al. [A common task in many single-cell studies is to detect differentially expressed (DE) genes between cell populations. There are already multiple methods that can be used for that purpose. Some of the methods are new and designed specifically for scRNA-seq data, some have been originally developed for bulk RNA-seq data, and some are more general. Recently, Kharchenko et al.  briefly et al. [et al. [To address this need, we compared five representative statistical methods that detect DE genes between single-cell populations: (1) single-cell differential expression (SCDE) , (2) modet al.  and Björ [et al. .et al. [SCDE (v. 1.99.0) was downloaded fromIn this section, we shortly introduce the five tested methods with both a general summary and by a brief description of the mathematical basis of the methods. Detailed explanations are available in the original publications . The difet al. . MAST iset al. ; CuffDifet al. , 7. Two http://hms-dbmi.github.io/scde/package.htmlMAST (v. 0.931) was downloaded fromhttps://github.com/RGLab/MASTDESeq (v. 1.20.0) is available inhttp://www.bioconductor.org)Limma (v. 3.24.15) is available inBioconductor (http://www.bioconductor.org)ROTS (v. 0.99.9) is available inBioconductor (http://www.bioconductor.org)Bioconductor . In Bayesian approach, the probability E):B is a bootstrap sample of S and g being expressed at level x in cell c conditioned on observed expression level cr and fitted error model Ωc for cell c. The posterior probability can be further written asc when a gene is expressed at an average level of x, and poissonp and NBp are probabilities of observing expression magnitude of x in case of a dropout (Poisson) or successful amplification , with the parameters of the distributions determined by the error model Ωc. The probabilities dp, poissonp and NBp are calculated based on distributionse is the expected expression magnitude estimated as a median observed magnitude in cells where the gene is amplified.When comparing two subgroups of cells g.The MAST method has originally been designed for quantitative polymerase chain reaction-based fluidigm single-cell gene expression assay, but it can also be used for scRNA-seq data . The theg is expressed in cell c (gcZ = 1) or not (gcZ = 0), and let gcY be the gene expression level when gcZ = 1. MAST fits a logistic regression model for Z and a Gaussian linear model for Let cX is the design matrix. The regression coefficients of both components are regularized. In the discrete case the regularization is done using a Bayesian approach in R-package arm. In the continuous case the variance parameter is regularized by cellular detection rate (CDR) defined as:N is number of measured genes. Differential expression is determined using the likelihood ratio test.where The DESeq is a widely applied method originally developed for bulk RNA-seq data. It is based on a NB model, with mean and variance linked by local regression .gjK of gene g in sample j is modeled with NB distribution gjμ and A and B) to each other, termsBgK similarly, and the parameters Agμ and gjK. Let us denote the probability that p. Then the P-value gp of observed AgK and BgK is defined asg. Function fc denotes fold changeIn DESeq, count The LIMMA method was originally designed for gene expression microarray data, but has recently been extended to RNA-seq data. In case of RNA-seq data, Limma uses Voom preprocessing . Limma iLimma considers gene-wise linear modelsgy is a vector of expression values from different samples, X is a design matrix, gα is a coefficient vector and gW is a known weight matrix. The variable that describes possible differences between test groups isC is a contrast matrix. The linear model is fitted to the responses to obtain coefficient estimators gV is unscaled covariance matrix. Some assumptions about the distributions of gd degrees of freedom. The term gjv is the jth diagonal element of d0 are estimated from the data using empirical Bayes approach. If t-test (14) is the ordinary t-test.where The ROTS is the only method among the tested ones that does not have any single-cell or sequencing-specific functions. ROTS optimizes the parameters among a family of modified t-statistics by maximizing the reproducibility of the detections across bootstrap samples. It has previously been shown to perform well in gene expression microarray and bulk RNA-seq data as well as in mass spectrometry-based proteomics data .ROTS maximizes the scaled reproducibilityk; k defines the top list size. The denominator B is the number of bootstrap rounds (two data sets b), gr returns the rank of a gene (or other element) g when test statistic t-test-like statisticg over the samples in group x and y, respectively. The term gs is the standard error.over the parameters The five methods were tested on three different scRNA-seq data sets with the objective of determining DE genes between different types of cells.et al. [et al. [et al. [http://carlosibanezlab.se//Data/Moliner_CELfiles.zip (accessed 30 October 2015). The original .CEL files were preprocessed using the Bioconductor package affy with Robust Multi-array Average (RMA) normalization. Only the genes appearing in both data sets were used in this study. Top 1000 DE genes were detected from the validation data using the Bioconductor package Limma.The first scRNA-seq data set, originated from a study by Islam et al. , is avai [et al.  was used [et al. . The .CEet al. [The second scRNA-seq data set came from the study by Kowalczyk et al. . The couet al. [The third data set by Björklund et al.  containsIn each analyzed scRNA-seq data set, genes that were never expressed were filtered out. With SCDE, MAST, DESeq and Limma, we followed the instructions of the respective authors for data preprocessing and filtering. With ROTS we performed Trimmed Mean of M values normalization  before pet al. [et al. [et al. as a gold standard [P-values) as predictor, and a binary vector, which indicates whether a gene belongs to the top 1000 validation genes, as response. The AUC values were compared between the five methods by P-values computed using bootstrap method.Our first comparison setting used the comparison design and data sets from the study by Kharchenko et al. . More sp [et al.  was usedstandard . The AUCet al. [fullDE denote the set of detected DE genes in the full data set, and subsetDE the set of detected DE genes in a subset of the data. The precision was defined asOur second comparison setting used the relatively large single-cell data set by Kowalczyk et al. , which eTo estimate the rate of false-positive detections, we generated mock comparisons from the 135 LTHSC cells of the old mice by randomly dividing the cells into two non-overlapping groups that represent two cell populations. These two fake populations were compared with each other and the procedure was repeated 10 times. Because all the cells were from the same population, there should not be any real differences. Therefore, any possible detections would be considered false positive.et al. [To consider data from other organisms than mouse, the third comparison was applied to the human data set by Björklund et al. . Similaret al. , we compP-values for comparing the AUC values are shown in The ROC curves of the different methods are shown in et al. [It is worth noting that the results were not identical to those reported by Kharchenko et al. , which iNext, we measured the precision and recall of the detections using the detections from the full data of 135 + 89 LTHSC cells and 308 + 266 ILC cells as the gold standard. The precision and recall values in both data sets for different methods and different numbers of cells per group are illustrated in The recall values were systematically lower than the precision values for all the methods . The medIn et al. data [et al. data [et al. [Finally, we compared the overlaps of the detections between the methods in the three different data sets by extracting ∼10% of the most DE genes from the results of each method . OThe modest overlaps between top detections from different methods indicate that the methods perform differently from one another.Among the tested methods, Limma is the fastest to run; MAST and DESeq are also relatively fast, whereas ROTS and SCDE are more computationally intensive. The time-consuming steps of SCDE and ROTS are fitting an error model and permuting the samples for bootstrapping, respectively. The remaining three methods, MAST, DESeq and Limma, do not include any heavily time-consuming steps. Details about running times with regard to different sizes of input data are available in The times were recorded using system.time function in R. The user times listed in the table correspond to a system with Intel i7-4790 processor (3.60 GHz) and 32 GB of RAM. For the sake of comparability, only one core was used for the runs in et al. data [et al. data, but it performed well in both LTHSC and ILC data sets. Based on our comparisons, DESeq and Limma without any modifications are not suitable for scRNA-seq data analysis, and yet, they have performed well in the context of bulk RNA-seq data [We have performed a systematic comparison of five different statistical methods to detect DE genes between single-cell populations. The single-cell-specific methods SCDE and MAST performed differently from each other. SCDE performed well in the first comparison setting in the Islam al. data , but wasseq data , 6. ROTSMAST detected much more DE genes than the other tested methods , whereas DESeq yielded the smallest number of detections. SCDE and ROTS found similar numbers of DE genes and the numbers were between the two extremes. Overall the relative differences between the methods were similar in the LTHSC and ILC data sets. It is based on the application whether a high or low number of detections is more favorable. The conclusions are summarized in http://hms-dbmi.github.io/scde/tutorials.html). All the methods are available as R packages.In addition to statistical properties, other features are also relevant when deciding which method to use , 19. ForOverall, our comparisons in three diverse data sets showed marked differences between the five tested methods in terms of the number of detections as well as their sensitivity and specificity, but did not reveal systematic benefits of the single-cell-specific methods in general. Instead, our previously introduced reproducibility-optimization method ROTS showed good performance in all comparisons even without any single-cell-specific modifications.Key PointsSelection of the method had a large impact on the results.Single-cell-specific methods did not have systematically better performance than other methods.Methods developed originally for bulk RNA-seq, DESeq and Limma, were not suitable for analyzing scRNA-seq data.ROTS performed well in all the comparisons and MAST had good statistical properties  as well.http://bib.oxfordjournals.org/.Supplementary DataClick here for additional data file.


Differential expression analysis on the basis of RNA-Seq count data has become a standard tool in transcriptomics. Several studies have shown that prior normalization of the data is crucial for a reliable detection of transcriptional differences. Until now it has not been clear whether and how the transcriptomic approach can be used for differential expression analysis in metatranscriptomics.We propose a model for differential expression in metatranscriptomics that explicitly accounts for variations in the taxonomic composition of transcripts across different samples. As a main consequence the correct normalization of metatranscriptomic count data under this model requires the taxonomic separation of the data into organism-specific bins. Then the taxon-specific scaling of organism profiles yields a valid normalization and allows us to recombine the scaled profiles into a metatranscriptomic count matrix. This matrix can then be analyzed with statistical tools for transcriptomic count data. For taxon-specific scaling and recombination of scaled counts we provide a simple R script.When applying transcriptomic tools for differential expression analysis directly to metatranscriptomic data with an organism-independent  scaling of counts the resulting differences may be difficult to interpret. The differences may correspond to changing functional profiles of the contributing organisms but may also result from a variation of taxonomic abundances. Taxon-specific scaling eliminates this variation and therefore the resulting differences actually reflect a different behavior of organisms under changing conditions. In simulation studies we show that the divergence between results from global and taxon-specific scaling can be drastic. In particular, the variation of organism abundances can imply a considerable increase of significant differences with global scaling. Also, on real metatranscriptomic data, the predictions from taxon-specific and global scaling can differ widely. Our studies indicate that in real data applications performed with global scaling it might be impossible to distinguish between differential expression in terms of transcriptomic changes and differential composition in terms of changing taxonomic proportions.As in transcriptomics, a proper normalization of count data is also essential for differential expression analysis in metatranscriptomics. Our model implies a taxon-specific scaling of counts for normalization of the data. The application of taxon-specific scaling consequently removes taxonomic composition variations from functional profiles and therefore provides a clear interpretation of the observed functional differences. Metagenome analysis can provide a comprehensive view on the metabolic potential of a microbial community . In addiIn single organism transcriptome studies, differential expression analysis based on RNA-Seq data has become an established tool . For theIn metatranscriptomics, already the transcript identification step can be challenging. In many cases, RNA-Seq faces a mixture of organisms for which no reference genome sequence is available. Several strategies have been suggested: de novo transcriptome assembly combined with successive homology-based annotation , the dirHere we present an extended statistical model for count data from metatranscriptomic RNA-Seq experiments. The model provides a novel view on metatranscriptome data, which negates the impact of taxonomic abundance differences between samples for differential expression analysis. Theoretical considerations, as well as studies on simulated and real count data, show that our approach can help to identify differentially expressed features which actually reflect a changing behavior of the organisms. For our model to work, accurate normalization of the data is crucial and in general, requires an organism-specific rescaling of expression profiles. The application of differential expression analysis to mixed-species data without prior separation can be found in several metatranscriptomic studies  as well i and sample j arises from a product of the per-gene quantity λicj under condition cj and a size factor sj: λicj is proportional to the mean concentration of feature i under condition cj. The size factor sj represents the sampling depth or library size. Usually, both factors are unknown. If we assume the ith feature to be non-differentially expressed (NDE) we can represent the corresponding row of the count matrix by s. Thus, for NDE features the size factors are proportional to the expected counts. If we knew which features are actually NDE, we would be able to estimate the required size  factors for normalization from the corresponding counts.To clarify our arguments for an alternative normalization of metatranscriptomic data we need to explain the statistical nature of the normalization problem. We first follow the approach of Usually this is not the case and we need to make some assumptions. A common assumption that is used in current tools is that most of the features are NDE. Then it is possible to estimate the scaling factors by some robust statistics. In DESeq for each sample the putative scaling factors from all features are calculated and then the median of all these values is used as an estimator of the sample-specific scaling factor . With a Without any distinction between DE and NDE features the scaling factors have also been estimated from the count sums of all samples. However, the potential shortcomings of this total count normalization have widely been discussed .N-organism mixture that includes a matrix S of organism-specific scaling factors sjk: i, j, k are the feature, sample and organism indices, respectively. We omitted the condition dependency (cj) for convenience.In metatranscriptomics the situation is more complicated because for each organism we can have a different scaling factor. So we have to extend the above sampling model to an i of the count matrix: λi contains all organism-specific rates for feature i and In analogy to α is a column vector of organism-specific abundances and s contains the sample-specific scaling factors, now in a column vector, which is equal for all organisms. Then we can write S has column rank 1, i.e., all column vectors are collinear, would be hard to justify in practice. Implicitly we would assume that the relative contributions of all organisms are constant across all samples. In general, this assumption is not met, because for a real metatranscriptome, the organism composition of transcripts cannot be controlled and will be different for different samples.Application of the above single-organism scheme for estimation of scaling factors is only valid if the matrix of scaling factors has the following form: In the following we show how to normalize metatranscriptomic counts so that the data actually meet the former assumption.We propose a method to prepare metatranscriptomic data for differential expression analysis. The method is referred to as taxon-specific scaling. As an essential prerequisite our approach requires that the data is first partitioned according to the contributing organisms. Then the count data matrix from each partition is normalized separately. Here, established tools from transcriptomics can be used to estimate the corresponding scaling factors.Finally, the normalized count data matrices are summed up to provide normalized metatranscriptomic count data which can be analyzed in terms of differential expression . Here alk as Yk and the associated vector of estimated scaling factors as −1 operator transforms the scaling vector to a diagonal matrix with inverse scaling factors on the diagonal and zeros everywhere else. We provide an R script where we use DESeq2 for scaling factor estimation and identification of significant differences  by scaling the profiles of each organism individually. In contrast, global scaling normalizes the data with respect to the total library size, i.e., the sample-specific count sum, and taxonomic abundance shifts can strongly influence the identification of DEF. With global scaling, the researcher expects the taxonomic composition of the samples to be informative for differential expression analysis. However, it is not always clear how this information can be utilized. When relative taxonomic abundance varies within a condition, this implies a higher dispersion of the corresponding count data, which in turn may prevent the identification of DEF. Furthermore, if DEF are identified, it is unclear whether they result from organisms which show a differing functional profile between conditions or whether they result from organisms with unchanged functional profiles but with varying relative abundances. Finally, if taxonomic composition variations in the data are maintained it may happen that for organisms that are only abundant in one condition all of their associated functions become differentially expressed. If this affects the majority of features, the model assumptions of statistical tools for DEF identification may be violated (see above).Normalization methods have also been used in the field of metagenomics to make the taxonomic compositions between samples comparable. Here, similar techniques can be applied  to identThe aim of our simulations was to demonstrate and explain the differences between taxon-specific and global scaling in practice, using a fully controlled data generation setup. To make the differences visible and understandable it is sufficient to simulate small communities as an increased complexity requires more parameters to be controlled which may even complicate interpretation. In some cases even a minimal community with two organisms may be sufficient to clearly show the different effects of the two scaling methods.In our simulations, a metatranscriptome arises from a mixture of various organisms, each with individual features. We assume that all organisms somehow react to a change of experimental conditions in terms of changing feature profiles. A metatranscriptome can include features covered by all taxa as well as features occurring only in few or a single organism. The sum of contributed counts from a single organism we refer to as the organism-specific library size. Generally, the count contributions from different organisms are not equal and vary across samples. We refer to this as the variation of the library size. To simulate the variation of organism-specific library sizes, we generated multiple transcriptomic count data sets with different total count numbers. Thereby, each generated data set mimics the contribution of a single organism. The data sets were then combined to simulate a metatranscriptomic count matrix.As with all simulations, the generated data can only provide a coarse approximation of real data and the obtained counts depend on particular parameters. Therefore, settings for the number of features and the total count values influence the results. Each organism is simulated with 100 differentially expressed features (DEF), 50 of them upregulated, and with 900 features that were non-differentially expressed (NDE).Li is DE or NDE according to feature i being differentially expressed or non-differentially expressed. The statistical test used to detect DEF, provides a p-value for each feature. The predicted label p-value  per condition and up to five organisms (Org1 to Org5) per sample. In the first three simulations, the different organism profiles are stacked, to exclude any interference between features from different organisms in the combined data. Accordingly, the final count matrix has 12 columns and 5,000 rows that correspond to samples and features, respectively. In our simulated metatranscriptome experiments we show under which circumstances, the differences between global scaling and taxon-specific scaling increase in respect to the number of identified DEF. With the stacked profiles we focus on the ability to recover DEF which the data generation process labeled as DE for the simulated transcriptomes. Within this context, the data generation provides the necessary information to calculate the number of true positives (TP) and false positives (FP). The label  p-value  is belowThe tool compcodeR  was usedIn the first experiment we simulate the case where the library size (LS) for each of the five organisms does not vary across different samples. Although this is an unrealistic case we performed this simulation to verify that both normalization approaches produce the same results under idealized conditions.e7 followed by organism Org2 with 5e6, 1e6 for organism Org3 and organism Org4, Org5 with 5e5, 1e5 respectively. This results in relative abundance fractions of 0.60, 0.30, 0.06, 0.03 and 0.01 for the 5 organisms. Because data is generated without variation for the number of counts per sample, no normalization is required, i.e., the correct scaling factors for all samples and all organisms are the same (=1).In addition, we wanted to investigate how different organism abundances affect the identification of DEF. Each organism was assigned a fixed total count number across all samples, without variation in library size. We simulated organism Org1 with a base count of 1In the second experiment we simulated a more realistic situation, with varying LS for all included organisms. Organism base counts are identical to the first simulation but the LS is randomly increased or reduced according to a random factor between 0.5 and 2. Due to the different library sizes in the samples, a prior normalization for both global scaling and taxon-specific scaling is required.In the third simulation, we investigated to what extent a condition dependent variation of LS can affect the normalization results. Under condition A we increase LS of Org1 by a random factor between 1.5 and 2 while under condition B we decrease the LS by a random factor between 0.5 and 0.667. For Org2 the direction of change is reversed, with a random decrease under condition A and an increase for condition B. For Org1 and Org2 the same base count as for Simulation I and II is used and for Org3–5 all parameters from Simulation II are used.In this simulation, we investigated the different effects that can be observed for taxon-specific scaling and global scaling when using superimposed (mixed) counts. For the superposition, we assume that the corresponding counts of equal functional categories are summed up across different species.a for Org1 and b for Org2) of both organisms i.e., feature a1 + b1, a2 + b2, …, a1,000 + b1,000. Thus, the feature combination results in summing up DEF with the same direction for both organisms.For an easier visualization, we reduced the setup to two organisms with a base count of 1e7 for both. Here, we used the LS variation range for Org1 and Org2 from Simulation III. CompcodeR generated each organism profile with first 50 features as upregulated DEF, next 50 as downregulated DEF, followed by additional 900 NDE features. As a result, we obtain a total of 1,000 features for the superimposed data. For the mixed features, we sum up the corresponding features  and contains gene names and the associated numbers.For a real data study, we chose a metatranscriptome dataset from mice gut . The expBacteroides cellulosilyticus WH2 did not map to gene names, all related counts are excluded from the analysis.Because the gene to Pfam  mapping p-value of 0.05 for the prediction of DEF.A differential expression analysis for all pairwise combinations of distinguished conditions was performed to compare the results of global and taxon-specific scaling. We calculated the number of DEF predicted (a) with both methods with the same fold change direction, (b) with both methods but with an opposite fold change direction and (c) with only one scaling method. In addition, we investigated the overlap between the single organism transcriptome analyses and the differential expression analysis for the mixture. We applied a significance threshold on the adjusted In the first part of our evaluation we examined the performance of taxon-specific and global scaling methods on simulated data. Because simulations I to III had been designed to provide a clear ground truth from a transcriptomics perspective we were able to distinguish true positive predictions of DEF from falsely classified features. In the second part we show results on real metatranscriptomic count data. Here the ground truth is not known and therefore we restrict the analysis on the comparison of the results from the two normalization approaches. Because it is impossible to verify the correctness of predictions we focus on analyzing the agreement or disagreement on DEF detection in this case.In this experiment, we measured the ability to detect DEF in a metatranscriptome without variation of organism-specific library sizes across different samples. This situation, in principle, does not require any normalization and therefore we expected taxon-specific (“tax”) and global (“glo”) scaling to yield similar results. This is confirmed by the resulting true positive (TP) predictions of DEF for the included organisms . For botWe repeated the analysis with edgeR with the included normalization and DESeq2/edgeR with TC normalization to estimate the scaling factors see : Fig. 1.When introducing organism-specific LS variation across samples the picture changes. For the global scaling approach the results show a decrease in the average TP rate for all organisms see . This trThe receiver operating characteristics (ROC) shows a higher area under curve (AUC) value for taxon-specific scaling (0.8776) than for global scaling (0.8282). The curve for global scaling also shows a higher degree of variation across different simulation runs  is high and in a typical application of global scaling on mixed counts from different organisms the two types of transcriptional differences will not be recognized.x-axis) and B (y-axis). While the feature distribution for taxon-specific scaling shows an elliptical distribution with a higher concentration near the diagonal, for global scaling we observe a clustering of features, with a lower density near the diagonal. The two clusters arise from the two organisms, which show a different ratio of condition-specific means, due to different LS under the two conditions. Here we see that it can be problematic to infer the scaling factors which are typically estimated from putative NDE features on and near the diagonal. In this case, for transcriptomics tools like DEseq2 and edgeR the assumption of a majority of NDE features is obviously violated.The last simulation only includes two organisms for a better illustration of the effects that arise from a superposition of counts from different organisms. The normalized superimposed counts show a characteristic picture  and 8B wTo illustrate how the superimposed (mixed) counts are obtained, we used arrows to show the contribution of both organisms to the mixed counts for the first 50 features, i.e., all up-regulated DEF from the generated transcriptome profiles see  and 8D. While the objective of the simulation studies was to evaluate and compare the two normalization approaches in terms of combined transcriptomic DEF, we do not have DE and NDE labels for the analysis of the real data. Therefore we focused on an analysis of the (dis-)agreement in results between both approaches without assessing the detection performance. The analyzed data comprises Pfam counts from 11 organisms, six conditions according to different time points and four replicates per condition. An overview on predicted DEF in all pairwise condition comparisons is shown in For both approaches, the number of predicted DEF peaked at “day 13” vs. “day 27” with 512 and 756 significant features for taxon-specific and global scaling. The number of DEF was low when conditions close together on the time line were compared (“day 15” vs. “day 16” or “day 29” vs. “day 30”).For global scaling, the number of predicted DEF was generally higher than for taxon-specific scaling. For some of the comparisons, the number of extra predictions under global scaling was even higher than the number of shared predictions see . The higWe also compared the results of the mixture analysis for global scaling and taxon-specific scaling to the transcriptome analyses of the individual organisms. For 10 of the 15 comparisons, the majority of predicted DEF with global scaling were not predicted as DEF in any transcriptome see : Fig. 6.As described in the original study, “day 13” and “day 27” each correspond to the final day of a particular diet. Because the number of predicted DEF for both scaling methods  was the highest here, we analyze the results for this comparison in more detail.We mapped the Pfam-annotated features which were predicted as DEF for global scaling and taxon-specitic scaling to Gene Ontology (GO) terms and compared the results. With taxon-specific scaling 250 GO terms with at least one DEF mapping were identified, while global scaling resulted in 311 GO terms. GO terms associated to biological processes with a high agreement between the two methods were for example cellular amino acid metabolic process where both methods identified seven of the nine associated Pfams as DEF and carbohydrate metabolic process with 11 DEF shared between taxon-specific scaling and global scaling. In this category, taxon-specific scaling and global scaling predicted five additional DEF uniquely.GO terms with predicted DEF from taxon-specific scaling alone included magnesium ion binding and fucose metabolic process with three predicted DEF each. For global scaling alone, we found that DNA modification, molybdopterin cofactor biosynthetic process, and RNA modification with 3, 2 and 2 predicted DEF respectively  exceed 80% of the normalized counts for that feature. For extra predictions obtained only with global scaling without evidence from the transcriptome analyses, the counts from a single organism are the main contribution for 82 of 199 features. Additional 43 features are predicted from the summed counts from organisms with the same scaling shift. An example of a feature that shows a dominating single organism is shown in The simulations allowed us to highlight the effects of the two normalization approaches that can lead to results with a varying degree of divergence between taxon-specific and global scaling. Here, we could show that organism-specific transcript abundance and its variation between and within conditions is the key factor for the divergent results.Other important factors are the accuracy of the functional and taxonomic annotation and the number of observed features with sufficient counts to estimate the scaling factors. In this regard, the simulations correspond to idealized conditions which cannot reflect the difficulties of a real data analysis. The number of organisms was chosen with respect to a clear visualization of results. In principle, an increase of the organism number does not add any effects that could not be observed with a lower complexity.All simulations were designed to demonstrate the very different nature of the two normalization approaches. Taxon-specific scaling is closely related to the transcriptomic approach to differential expression analysis and can be viewed as a possible generalization of the single-organism setup to a multi-organism analysis. It equalizes all variations of the taxonomic composition of transcripts across samples to identify functional effects that are consistent across different organisms. If the functional profiles do not show systematic variation between conditions, no meaningful differences will be found with taxon-specific scaling.In contrast, global scaling assumes that variations of the taxonomic composition are meaningful and important for differential analysis. Depending on the data, this assumption may be correct and then many of the false positives for global scaling in Simulation III may actually correspond to true positives. The objective of the simulations was to compare the two normalization approaches under the assumption that the relevant information is in the variation of the organism-specific functional profiles, similar to the transcriptomic approach. In contrast, the taxonomic variations were not assumed to be indicative of the experimental conditions and might for instance result from sample preparation. We chose this setup because it reveals a potential complication of the global scaling approach. As the main problem, the effects of taxonomic and functional variations will always be mixed up in the results when applying global scaling. Without further analysis, it is impossible to determine whether a predicted DEF results from a taxonomic or from a functional variation. Another problem with global scaling has been shown in Simulation IV, where the underlying model assumptions for the normalization tend to be violated. With an increasing number of DEF it becomes increasingly unclear which features should be used as a reference for estimation of the scaling factors. Therefore, if taxonomic variation is informative it might be a better strategy to estimate the species composition for each sample and then compare the taxonomic profiles for the identification of significant differences. In fact a differential composition analysis as used in metagenomics  will welIt is important to realize that while taxon-specific scaling negates library size variation of the organisms between samples, it retains the overall taxonomic composition of the observed transcripts. Therefore, the feature counts of rare organisms will have less impact on the combined metatranscriptome data than feature counts from highly abundant organisms. The minimum required coverage of an organism for reliable estimation of the scaling factors highly depends on the data. For rare species it is possible that only a few highly expressed features are observed. As a special case, organisms only observed in one condition cannot be included in the differential expression analysis of the combined counts for taxon-specific scaling. Therefore, the user has to check what fraction of the features can actually be used in the analysis and if identified DEF could be the result of the exclusion of features from organisms not observed in both conditions. With global scaling such single-condition organisms can be included but without taxonomic separation it can become difficult to explain a possibly large number of differences from a change of experimental conditions.For taxon-specific scaling our approach relies on techniques implemented in tools that have been developed for differential expression analysis in transcriptomics. As a consequence, our method also inherits the assumptions and requirements of these tools for the normalization of transcriptomic count data. In general, both DESeq2 and edgeR require a sufficient number of NDE features to reliably perform the normalization. In particular, both methods expect the number of DEF to be lower than the number of NDE features. Also the proportion of up- and down-regulated features may have an influence on the detection of DEF. When the ratio is shifted towards one regulation direction, the detection performance may decrease. Other normalization methods might provide better results under these conditions . The appThe optimal situation for taxon-specific scaling would enable the separation of all feature profiles from different organisms. This organism-specific separation can be viewed as the most conservative approach, as in principle it would be sufficient to separate all differing expression profiles. Therefore, if organisms show the same transcriptional activities, the normalization of their combined count data is possible without problems. For example, the metatranscriptome might include multiple strains of the same species occurring simultaneously but showing the same activities.The choice of features is also an important factor to consider before the experiments are performed. The transformation of the organism-specific genes to a common feature space, such as the Pfam protein domain categories, allows us to first normalize and then sum up the features of different organisms to obtain a superimposed count matrix. The mapping can also include the assignment of multiple genes from the same organism to a single feature representation. As a consequence, it is no longer possible to identify differential expression with respect to a single gene. This limitation is inherent to many metatranscriptome studies and does not depend on the chosen normalization approach. Again, parallel sequencing of the metagenome might be considered, to obtain a higher resolution. Then, a combined assembly and binning strategy can yield the required draft genomes as reference to realize the highest taxonomic and functional resolution for differential expression analysis.Differential expression analysis in metatranscriptomics is challenging. Metatranscriptomic count data from RNA-Seq experiments show two main modes of biological variation. The functional composition of transcripts reflects the activity of organisms and systematic changes might indicate a metabolic response to experimental conditions. The taxonomic composition of transcripts can change as well and a change may not necessarily be explainable in terms of controlled experimental conditions. In contrast to metagenomics, in metatranscriptomics the questions “who is there?” and “what are they doing” are not necessarily connected and should be answered separately for a clear interpretation of results. If the two questions are not separated, just from a significant difference it is not possible to decide whether this difference goes back to a variation of taxonomic or functional profiles or if it reflects both kinds of variation. Our approach to normalization of metatranscriptomic data eliminates the influence of taxonomic variations from functional analysis. For realization of this approach, the metatranscriptome needs to be decomposed to normalize the organism profiles independently. Then the metatranscriptomic count data can be recombined from the normalized profiles to look for any general tendencies in the superimposed count data. If differential expression tools are directly applied to the metatranscriptomic count matrix, the interpretation of results can be difficult. Our simulations indicate that it is easily possible to obtain a large number of putative functional differences which just arise from taxonomic abundance variations across samples. We would like to point out that our findings do not affect metatranscriptome studies that just aim to analyze the functional repertoire from RNA-Seq data. The question which functions or genes are expressed is much easier to answer than the question what is the functional response to a change of experimental conditions. However, it is important to note that our results do not only apply to the classic two conditions setup that we used throughout our study. Also for multiple conditions and time series our normalization approach will be beneficial to separate functional from taxonomic trends in metatranscriptomic count data.10.7717/peerj.3859/supp-1File S1Click here for additional data file.10.7717/peerj.3859/supp-2File S2Click here for additional data file.10.7717/peerj.3859/supp-3File S3Click here for additional data file.10.7717/peerj.3859/supp-4File S4Click here for additional data file.10.7717/peerj.3859/supp-5File S5Click here for additional data file.10.7717/peerj.3859/supp-6File S6Click here for additional data file.