The mining of weak correlation information between two data matrices with high complexity is a very challenging task. A new method named principal component analysis-based multiconfidence ellipse analysis (PCA/MCEA) was proposed in this study, which first applied a confidence ellipse to describe the difference and correlation of such information among different categories of objects/samples on the basis of PCA operation of a single targeted data. This helps to find the number of objects contained in the overlapping and nonoverlapping areas of ellipses obtained from PCA runs. Then, a quantitative evaluation index of correlation between data matrices was defined by comparing the PCA results of more than one data matrix. The similarity and difference between data matrices was further quantified through comprehensively analyzing the outcomes. Complicated data of tobacco agriculture were used as an example to illustrate the strategy of the proposed method, which includes rich features of climate, altitude, and chemical compositions of tobacco leaves. The number of objects of these data reached 171,516 with 14, 4, and 5 descriptors of climate, altitude, and chemicals, respectively. On the basis of the new method, the complex but weak relationship between these independent and dependent variables were interestingly studied. Three widely used but conventional methods were applied for comparison in this work. The results showed the power of the new method to discover the weak correlation between complicated data. Information extraction and mining of data with high complexity has increasing interests in both academic and industrial sectors. With rapid development and use of smartphones and sensing equipments and advanced scientific instruments such as chromatography, mass spectrometry, spectroscopy analysis and their coupled techniques, theoretical calculations, and simulation, of course, the difficulty of generation of data has been largely overcome to date –5. Thus,Traditionally, the process for quantitative modeling is applied as follows. First, univariate statistical analysis of each feature in a single data is performed, and hypothesis testing between different types of samples is used to obtain statistical results of a single feature. This helps to find and further remove the outliers from both the dimensions of objects and variables . Next, dMany researchers reported the studies to correlate the relationship between multiple datasets. PCA should be one of them with the highest concerns used in different fields, which has also been extensively developed in theory and applications, including sparse PCA generated by an elastic network (LASSO), probabilistic method on the basis of an associated likelihood function, and the robust PCA for processing of data with outliers –28. SincIn this work, a new method called PCA/MCEA  was developed for correlation analysis, which was based on PCA to discover weak information between datasets. It first utilized PCA for dimensionality reduction after data quality improvement. The confidence ellipse analysis then quantitatively analyzed the similarity and difference among different categories of samples. Next, a quantitative index was defined to introduce the correlation of samples in terms of the sample/objects distribution in the individual ellipse of each sample class, as well as their results in the overlapping zone of the ellipses. In the end, the correlation between independent variables and dependent variables was attained after comprehensively analyzing the performance of these findings between different samples. The findings showed that the PCA/MCEA method can be used as an effective tool to mine out the correlation information between multidimensional data with high complexity. Example data of tobacco agriculture were applied to deliver the strategy, which included planting climate and altitude, and the corresponding concentrations of chemical compositions. The results of visualization analysis further explained the findings and show rich characteristics and relationships of planting production of tobacco collected in different locations/regions. Three traditional methods were used for comparison. The strategy and procedure of the PCA/MCEA method can be extensively used for analysis of other types of datasets.PCA has been widely employed for data processing, which has the power to reduce data size by projecting the raw data to low-dimensional space containing most of the original variance and ignoring part of the features with small variance , 27. TheA denotes the raw data with a size of m × n for decomposition and the three matrices U, Σ, and VT denote scores, singular values, and loadings with sizes of m × m, m × n, and n × n, respectively.Singular value decomposition (SVD) is a strategy to realize PCA analysis to obtain the orthogonal principal components (PCs) with the results shown in the following equation :(1)A=UΣVU, VT, and Σ obtained after PCA operation, respectively, represent the left and right singular vector and singular values of original data A. Through analyzing these matrices, the correlation and difference between objects/samples in A can be discovered. The significant difference amongst these objects or features of A can be achieved with the help of confidence interval analysis [U, and furthermore, the confidence ellipse, for example, 95% ellipse on the plot, can be obtained to show the distribution of samples inside or out of the ellipses or included inside of the overlapping zone of two ellipses. Then, multiple confidence ellipses of different types of samples can be applied to deeply mine out the correlation of distribution and further find the characteristics of samples and the correlation and difference among these samples. Typically, the confidence ellipse analysis utilizes the smallest ellipse that covers 95% of data points (objects/samples) in the score plot to a certain class of samples. The ellipse has two important parameters, that is, area of the ellipse and inclination of the principal axis relative to the x-axis or y-axis in the plot representing the direction of change of the ellipse. The ellipse can be calculated by assuming an approximate Gaussian distribution of the objects coordinating around its average value.As described above, the three matrices analysis . Using tIn this paper, a quantitative method to evaluate the correlation of two matrices was proposed on the basis of PCA operation, and thus, it was originally named as multidimensional confidence ellipse analysis (PCA/MCEA). The flowchart of the PCA/MCEA method is shown in First, the original data are divided into multiple matrices according to the actual situation of data information, which is included in the introduction section of matrix. This step is applied to obtain the multiple data for processing and correlation assessment. After this, the PCA method is independently performed to reduce the dimensionality of each data. After this, determination of class of samples can be attained by using the priori knowledge. Next, the multidimensional confidence ellipse analysis can be used to analyze the independent variables between samples of each category and similarity of the dependent variables. The potential correlation between independent variables and dependent variables, namely, integrated analysis of different matrices to be analyzed, can be achieved on the basis of these results.n × m as an example, the original data can be expressed as follows:In terms of the source and type of different indexes of the original data, it is divided into several matrices for correlation analysis. Using data with size of X={x1,…, xm1}, Y={y1,…, ym2}, and D={d1, d2,…, dm3}, respectively, represent data with independent variables, dependent variables, and sample description information. The sizes of the three matrices are m1, m2, and m3. The independent variables and dependent variables of data X and Y can be further subdivided, as shown in equations , Y1 ∪ Y2 ∪ ⋯∪YM2=Y, and Yi∩Yj=ϕ. After these processing steps, two matrices of independent variables and dependent variables can be mathematically obtained for analysis.The three matrices quations .(3)X=X1,As introduced above, the PCA operation can be used for analysis of two datasets with independent variables and dependent variables, and the PCs with the largest variance can be extracted with removal of interference information.Z=X+Y, the submatrix Zi ∈ Z is decomposed and the latent variables with the largest variance are selected for subsequent analysis.Zi‖pca represents results of Zi by using PCA and MinPriColσ(‖Zi‖pca) represents the minimum number of PCs, in which cumulative variance contribution exceeds to a predefined threshold σ. That is to say, the cumulative variance contribution of the first k PCs of ∀‖Zi‖pca is greater than the threshold σ, and ∃‖Zi‖pca, the cumulative variance contribution of the first k − 1 PCs, is less than the threshold σ.Supposing the data to be analyzed as k latent variable after PCA analysis is extracted to construct a new dataset of latent variable T={Ti}, in which the matrix Ti includes the first k columns of ‖Zi‖pca.Finally, the first d ∈ D, d classes of samples can be divided for analysis. Similarly, each data matrix Ti ∈ T can be divided into D={Dj} numbers of classification. Then, a total of (M1+M2) × |d| data matrices were analyzed to a data matrix with M1+M2 independent and dependent variables and recorded as P={Pi}. The set Pi={Pij}  is the |d| classification result.According to the sample description information Pij will generate (M1+M2) × |d| number of multidimensional ellipses. It is denoted as Θi={Θij}, in which Θi={Θij} represents the multidimensional ellipse corresponding to the |d| numbers of sample classification, as shown in Based on the results of PCA operation, the multidimensional confidence ellipse of each data matrix d| ellipses in the multidimensional confidence ellipse {Θij} corresponding to data Pi of classification Zi, it can be divided into two spaces, that is, inside and out of the space. Through statistical analysis of sample classification Dp, Dq, the number of different types of samples existing in the confidence ellipse Θip and Θiq respectively, describe the classification information of Zi, sample classification Dp and Dq, and the degree of sample internal aggregation corresponding to the quantification of similarity and difference evaluation of samples of Dp and Dq.For the |Dp‖ indicates Dp number of samples in sample classification and ‖Θip∩Θiq‖p indicates the number of samples simultaneously existing in sample classification Dp and Dq, which corresponds to the overlapping area in the multidimensional confidence ellipse. Si,p,q denotes the similarity between sample classes of Dp and Dq in the case of classification Zi. In particular, it indicates the degree of aggregation of the sample classification Dp in the case of index classification Zi, if p=q.The quantitative index for evaluation is provided as follows:Zi here.By using the multidimensional confidence ellipse analysis method introduced above, the similarity and difference among multiple classes of samples can be quantitatively evaluated, and this helps to discover the characteristics of sample classification, such as Zi , degree of aggregation of different types of samples, and similar characteristics between classes. This helps to determine the association between multiple data matrices. By the confidence ellipse analysis of different classes of samples and then the correlation coefficient between the number of samples included in the ellipse of each class, the correlation between data can be discovered by analyzing the number of different groups of samples. It is especially helpful to mine out weak correlation, for example, while the classification or regression relationship is not significant.The PCA/MCEA method quantitatively analyzes the results of each classification of samples X) and dependent variables (Y), which seeks to find the simple relationship. This is probably unsuitable for data matrices with a weak relationship. The PCA/MCEA method completely avoids the difficulties and challenges encountered in constructing multivariate classification or regression models, but constructs a confidence ellipse to analyze relationship after PCA analysis. The samples included in the overlapping and nonoverlapping samples of the ellipse effectively denote the correlation of similarity and difference between different classes of samples. After this, the correlation between more than one matrix can be found through individually analyzing the contribution of different independent variables on the influence of dependent variables.The proposed method is unlike the traditional strategies for direct analysis of correlation of independent variables ,…, Fp(1), respectively, and it is assumed that Fi1(1)=max{F1(1),…, Fp(1)}. The independent variable is introduced to the regression model, if Fi1(1) is greater than the critical value F(1), which corresponds to the given significance level α. Otherwise, it will be excluded to the model [In the process of stepwise regression analysis, an independent variables he model .The stepwise regression method realizes the screening of “optimal” independent variables by gradually introducing variables and further calculates the square sum of partial regression analysis. It avoids the multicollinearity problem that occurs while using full independent variable analysis.In PLS regression analysis, the independent and dependent variables are projected into a new space to generate a linear regression relationship in a new space . The PLSThe support vector regression (SVR) analysis uses the optimal model shown in formula , which hε-insensitive loss function. Especially, it has strong generalization ability to reduce the requirement for balanced data sampling of samples.The SVR regression analysis is the first on the basis of structured risk minimization, which introduces an In this article, an example data of tobacco agriculture was employed to deliver the strategy proposed in this work. The purpose is to study the impact of an ecological environment, including climate and altitude, on the quality of tobacco leaves, which was originally collected from Yunnan Province, one of the largest districts for tobacco planting in China. The detailed information of the dataset is given in   Deletion of missing values: the samples with missing values of climate, altitude, and/or chemicals of tobaccos were eliminated before the next step.  Removal of outliers: the strategy of box-plot was applied to remove the samples of outliers.μ) and variance (σ) of the each variable were calculated and standardized according to the following formula: x′=(x − μ)/σ.  Data normalization: the Z-score method was used for data normalization. That is, the mean , altitude (X2), and dependent variables, including 5 indices of tobacco (Y1)The original data was divided and reduced into independent variables including climate , altitude (X2), and chemical compositions of tobacco leaves (Y1) were analyzed by the PCA operation. Based on the cumulative variance threshold, the new variables with the largest variance were selected for analysis. As shown in vice versa. If the samples of the same category of samples were more concentrated, the curve will be sharper with a small value of standard deviation (SD). That is, SD of the curve is smaller, and vice versa. The results of Figures The three datasets of climate , was applied as an example to illustrate the process for quantitative analysis of data with weak correlation by using the PCA/MCEA method. As mentioned above, Figures On the basis of the formula given in –5), the, the5), Furthermore, we conduct an integrated analysis of the correlation between the three data on the basis of the correlation coefficients defined in Figures In the next step, we further adopted radar chart analysis to visually analyze the results of a multidimensional confidence ellipse, which helps to intuitively show the differences of data in climate, altitude, and chemical of tobacco leaves among locations. Similarly, using the sample classification of GCQ as an example, the radar chart is used to compare the differences between the samples of GCQ and all other locations in terms of climate, altitude, and chemicals, as shown in As introduced above, the PCA/MCEA method was used to explain the effect of climate and altitude on the chemical compositions in tobacco leaves. In this section, three traditional regression methods were attempted to be used for building of more accurate and reliable models to the same data. With the help of the leave-one-out (LOO) method for validation, stepwise regression, PLS regression, and SVR analysis were utilized to construct a quantitative model for possible accurate prediction of chemicals, which are obtained by using the model between climate, altitude, or other independent variables and the known chemical indices of tobacco leaves. Here, the influence of climate and altitude on the content of element K (potassium) of tobacco leaves was demonstrated with the results given in In this work, the PCA/MCEA method was constructed based on division and pretreatment of original data, and PCA is first performed for dimension reduction on different data obtained from samples classification. Then, the relationship of samples is constructed by using multidimensional confidence ellipse analysis through finding the samples of different classes existing in the inside and out of these ellipses. The comparative analysis of samples between different groups can be achieved for quantitative analysis of difference and similarity between independent and dependent variables. This largely helps to effectively explore the hidden information of weak correlation between datasets. The dilemma of the traditional methods for quantitative modeling is overcome for processing of data with weak correlation and low capacity for new prediction.The PCA/MCEA method proposed in this work seeks to discover the weak correlation between complex data with the help of a multidimensional confidence ellipse obtained from the results of PCA operation. The common features and difference of samples between different types of samples are characterized by the number of samples existing in the overlapping or nonoverlapping areas of ellipses, which mathematically contains the characteristics of one or multiple classes of samples. The quantitative correlation between independent and dependent variables is comprehensively evaluated through individually analyzing the relationship of data pairs for information discovery. Data containing 171,516 tobacco leaves were handled as an example to deliver the strategy of the proposed method. In contrast to the conventional methods for classification and regression analysis, the results obtained from PCA/MCEA are more helpful to generate rich and informative conclusions. It can be also widely used for more types of complicated datasets with low but potential correlations.


The correct identification of differentially abundant microbial taxa between experimental conditions is a methodological and computational challenge. Recent work has produced methods to deal with the high sparsity and compositionality characteristic of microbiome data, but independent benchmarks comparing these to alternatives developed for RNA-seq data analysis are lacking.We compare methods developed for single-cell and bulk RNA-seq, and specifically for microbiome data, in terms of suitability of distributional assumptions, ability to control false discoveries, concordance, power, and correct identification of differentially abundant genera. We benchmark these methods using 100 manually curated datasets from 16S and whole metagenome shotgun sequencing.The multivariate and compositional methods developed specifically for microbiome analysis did not outperform univariate methods developed for differential expression analysis of RNA-seq data. We recommend a careful exploratory data analysis prior to application of any inferential model and we present a framework to help scientists make an informed choice of analysis methods in a dataset-specific manner. Study of the microbiome, the uncultured collection of microbes present in most environments, is a novel application of high-throughput sequencing that shares certain similarities but important differences from other applications of DNA and RNA sequencing. Common approaches for microbiome studies are based on deep sequencing of amplicons of universal marker-genes, such as the 16S rRNA gene, or on whole metagenome shotgun sequencing (WMS). Community taxonomic composition can be estimated from microbiome data by assigning each read to the most plausible microbial lineage using a reference annotated database, with a higher taxonomic resolution in WMS than in 16S . The finDifferential abundance (DA) analysis is one of the primary approaches to identify differences in the microbial community composition between samples and to understand the structures of microbial communities and the associations between microbial compositions and the environment. DA analysis has commonly been performed using methods adapted from RNA sequencing (RNA-seq) analysis; however, characteristics specific to microbiome data make differential abundance analysis challenging. Compared to other high-throughput sequencing techniques such as RNA-seq, metagenomic data are sparse, i.e., the taxa count matrix contains many zeros. This sparsity can be explained by both biological and technical reasons: some taxa are very rare and present only in a few samples, while others are very lowly represented and cannot be detected because of an insufficient sequencing depth or other technical reasons.In recent years, single-cell RNA-seq (scRNA-seq) has revolutionized the field of transcriptomics, providing new insight on the transcriptional program of individual cells, casting light on complex, heterogeneous tissues, and revealing rare cell populations with distinct gene expression profiles –6. HowevSome benchmarking efforts have compared the performance of methods –12 both We benchmarked a total of 18 approaches . The diversity of the data allowed us to test each method on a variety of circumstances, ranging from very sparse, very diverse datasets, to less sparse, less diverse ones.W = 734, p = 0.377, Additional File We first analyzed 18 16S, 82 WMS, and 28 scRNA-seq public datasets in order to assess whether scRNA-seq and metagenomic data are comparable in terms of sparsity. We observed overlap in the fractions of zero counts between the scRNA-seq, WMS, and 16S, but with scRNA-seq datasets having a lower distribution of sparsities (ranging from 12 to 75%) as compared to 16S (ranging from 55 to 83%) and WMS datasets (ranging from 35 to 89%) whose distributions of zero frequencies were not significantly different from each other  of the statistical models underlying each method on the full set of 16S and WMS data. For each model, we evaluated its ability to correctly estimate the mean counts and the probability of observing a zero Fig. . We evalThe NB distribution showed the lowest root mean square error , we found that Hurdle models provided good estimates of the observed zero proportion for 16S Fig. c and WMS8 (3 × 108 for HMP). To confirm this observation, we carried out a simulation experiment by down-sampling reads from deep-sequenced WMS samples (rarefaction): while the need for zero inflation seemed to diminish as we got closer to the number of reads typical of the corresponding 16S experiments, the profile did not completely match between approaches , which is the observed proportion of significant tests, should match the nominal value .We next sought to evaluate type I error rate control of each method, i.e., the probability of the statistical test to call a feature DA when it is not. To do so, we considered mock comparisons between the same biological Stool HMP samples (using the same Random Sample Identifier in both 16S and WMS), in which no true DA is present. Briefly, we randomly assigned each sample to one of two experimental groups and performed DA analysis between these groups, repeating the process 1000 times (see the “ 1 (ref. ) and theTo evaluate the impact of both the normalization step and the estimation and testing step in bulk RNA-seq inspired methods, we included in the comparison both edgeR with its default normalization (TMM), as well as with DESeq2 recommended normalization  and vice versa (Table S2). Similarly, because the zinbwave observational weights can be used to apply several bulk RNA-seq methods to single-cell data , we havep value distribution reasonably close to the expected uniform. Notable exceptions in the 16S experiment were edgeR with TMM normalization and robust dispersion estimation (edgeR_TMM_robustDisp), metagenomeSeq, and ALDEx2 . In the WMS data, departure from uniformity was observed for metagenomeSeq and edgeR_TMM_robustDisp, and limma_voom_TMM_zinbwave, which employed liberal tests, as well as corncob_LRT, ALDEx2, and scde, which were conservative in the range of interest  statistics in Fig. Ex2 Fig. a, b. Whiest Fig. c, d. We We next recorded the FPR by each method  and compared it to its expected nominal value. This analysis confirmed the tendencies observed in Fig. p value when available or by importance in the case of the songbird and mixMC; see We measured the ability of each method to produce replicable results in independent data in six datasets , 28–30 , i.e., the amount of concordance of the results of each method on the two random subsets.WMC was clearly dataset-dependent, showing high levels of concordance in datasets with a high differential signal  from the HMP , as their number of identified DA taxa was very low Fig. a. This ap value threshold may be applied. However, metagenomeSeq’s performance deteriorates after the 10% threshold, suggesting that this method starts to identify more false positives , enrichment and DA analysis were largely consistent, including several strains of Neisseria and several species of Treponema found to be DA .Our GOF analysis highlighted the advantages of using count models for the analysis of metagenomics data. The goodness of fit of zero-inflated models seemed dependent on whether the data come from 16S or WMS experiments. The difference between these two approaches translates to different count data structures: while for WMS many features are characterized by a clearly visible bimodal distribution , 16S data are as sparse as or even more sparse than WMS data, presenting for many features a less clearly bimodal distribution  and a vector p indicating the probability of the reads to be mapped to each taxon. In metagenomic studies, we have a large n (number of sequenced reads) and small p . In this setting, the Poisson distribution is a good approximation of the multinomial. Similarly, the negative binomial is a good approximation of the Dirichlet-Multinomial [Metagenomic data are inherently compositional, but whether incorporating compositionality into the statistical model provides benefits greater than the tradeoffs they may introduce is a debated topic in the literature , 40–42. tinomial . Secondltinomial . This hatinomial . We did tinomial  because The lack of ground truth makes the assessment of DA correctness very challenging. However, we can rely on mock datasets, within-method concordance, and enrichment analysis to obtain a principled ranking of method performances Fig. . AlthougThe parametric simulation framework is useful to inspect how individual characteristics of the data-generating distribution impact the sensitivity and specificity of the methods. As the entire analysis was supported by real data, we decided to focus only on a very simple but easily reproducible implementation of the NB and ZINB distributions for the simulations. The choice was justified by our GOF analysis on real datasets. Unsurprisingly, the sample size and the effect size were the characteristics that had the most impact on method performances. This translates into an evident suggestion for experimental design: large sample sizes are needed to detect low effect sizes. Our simulation framework can in principle be used for power calculations in the context of DA analysis.In the 16S dataset used for the enrichment analysis, with a total of 76 samples and almost 900 unique taxa, the most time-consuming methods were scde and songbird with more than 5 min needed to identify DA taxa. ALDEx2 and corncob-based methods took about 40 s, zinbwave-weighted methods took approximately 20 s while mixMC, MAST and seurat_wilcoxon around 10 s. DESeq2 and edgeR were under the 10 s with limma-voom which was the fastest method taking less than a second While the latter datasets are used only for a comparison between technologies, the former are widely used for all the analyses. A complete index with dataset usage is reported in Additional file as_phyloseq and setting the bugs.as.phyloseq = TRUE argument, respectively. The otu_table and sample_data slots of the phyloseq objects that contain, respectively, the taxa count table and the metadata associated to each sample were used for all downstream analyses. For the WMS datasets, absolute raw count data were estimated from the metaPhlAn2-produced relative count data by multiplying the columns of the ExpressionSet data by the number of reads for each sample, as found in the pData column “number_reads” (counts = TRUE argument).Phyloseq objects were obtained from the HMP16SData and curatedMetagenomicData packages using the function HMP16SData was split by body subsite in order to obtain 18 separated datasets. Stool and Tongue Dorsum datasets were selected for example purposes thanks to their high sample size. The same was done on curatedMetagenomicData HMP dataset, obtaining 9 datasets. Moreover, for the evaluation of type I error control, 41 stool samples with equal RSID, in both 16S and WMS, were used to compare DA methods. For each research project, curatedMetagenomicData was split by body site and treatment or disease condition, in order to create homogeneous sample datasets. A total of 82 WMS datasets were created.A total of 100 datasets were evaluated; however, for the CAT analysis, datasets not split by condition or body subsite were evaluated .To consider the complexity and the variety of several experimental scenarios, an attempt to select a wide variety of datasets for the analysis was done. The datasets were chosen based on several criteria: sample size, homogeneity of the samples, or availability of the same subjects (identified by RSID) assayed by both technologies.Negative binomial (NB) model, as implemented in the edgeR (v3.24.3) Bioconductor package (on read counts);Zero-inflated negative binomial (ZINB), as implemented in the zinbwave (v1.4.2) Bioconductor package (on read counts);Truncated Gaussian hurdle model, as implemented in the MAST (v1.8.2) Bioconductor package (on log count);Zero-inflated Gaussian (ZIG), as implemented in the metagenomeSeq (v1.24.1) Bioconductor package (on log count).Dirichlet-multinomial (DM), as implemented in the MGLM (v0.2.0) CRAN R package.The following distributions were fitted to each dataset, either by directly modeling the read counts or by first applying a logarithmic transformation:calcNormFactors function; common, trended, and tagwise dispersions were estimated by estimateDisp, and a negative binomial generalized log-linear model was fit to the read counts of each feature, using the glmFit function.The edgeR Bioconductor package was used to implement the NB model. In particular, normalization factors were calculated with the Trimmed Mean of M-values (TMM) normalization  using thzinbFit function. As explained in the original paper, the method can account for various known and unknown technical and biological effects [K = 0). We estimated a common dispersion for all features (common_dispersion = TRUE) and we set the likelihood penalization parameter epsilon to 1e10 .We used the implementation of the MAST Bioconductor package. After a log2 transformation of the reascaled counts with a pseudocount of 1, a zero-truncated Gaussian distribution was modeled through generalized regression on positive counts, while a logistic regression modeled feature expression/abundance rate. As suggested in the MAST paper , cell deCumNormStat and CumNorm functions were used to perform Cumulative Sum Scaling (CSS) normalization, which accounts for specific data characteristics. Normalization factors were included in the regression through the fitZig function.The metagenomeSeq Bioconductor package was used to implement a ZIG model for log2 transformed counts with a pseudocount of 1, rescaled by the median of all normalization factors or by 1e03 which gives the interpretation of “count per thousand” to the offsets. The Note that both MAST and metagenomeSeq were applied to the normalized, log-transformed data. We evaluated both models, using their default scale factor ary size , \documeMGLMreg function with dist = “DM” allowed the implementation of the above model and the estimation of the parameter values.The MGLM package was used to fit a Dirichlet-Multinomial regression model for counts. The To evaluate the goodness of fit of the models, we computed the mean differences between the estimated and observed values for several datasets.For each model, we evaluated two distinct aspects: its ability to correctly estimate the mean counts  and its ability to correctly estimate the probability of observing a zero, computed as the difference between the probability of observing a zero count according to the model and the observed zero frequencies . We summarized the results by computing the root mean squared error (RMSE) of the two estimators. The lower the RMSE, the better the fit of the model.This analysis was repeated for 100 datasets available in HMP16SData and curatedMetagenomicData  as proposed in the phyloseq package [estimateSizeFactors function with option type = “poscounts”. We also tested DESeq2 with TMM normalization (see below). As proposed in [weights slot of the DESeqDataSet class object to account for zero inflation. Observational weights were computed by the computeObservationalWeights function of the zinbwave package. To test for DA, we used a likelihood-ratio test (LRT) to compare the reduced model (intercept only) to the full model with intercept and group variable. The p values were adjusted for multiple testing via the Benjamini-Hochberg (BH) procedure. Some p values were set to NA via the cooksCutoff argument that prevents rare or outlier features from being tested.The DESeq2 (v1.22.2) Bioconductor package fits a negative binomial model for count data. DESeq2 default data normalization is the so-called Relative Log Expression (RLE) based on scaling each sample by the median ratio of the sample counts over the geometric mean counts across samples. As 16S and WMS data sparsity may lead to a geometric mean of zero, it is replaced by  package  and implposed in , observacalcNormFactors and estimateDisp functions, respectively . Analogously to DESeq2, “poscounts” normalization was used in addition to TMM in edgeR_poscounts_standard to investigate the normalization impact. We also evaluated the impact of employing a robust dispersion estimation, accompanied with a quasi-likelihood F test through the estimateGLMRobustDisp and glmQLFit functions respectively (edgeR_TMM_robustDisp). As with DESeq2, zinbwave observational weights were included in the weights slot of the DGEList object in edgeR_TMM_zinbwave to account for zero inflation, through a weighted F test. Benjamini-Hochberg correction was used to adjust p values for multiple testing.The edgeR Bioconductor package fits a negative binomial distribution, similarly to DESeq2. The two approaches differ mainly in the normalization, dispersion parameter estimation, and default statistical test. We examined different procedures by varying the normalization and the dispersion parameter estimation: edgeR_TMM_standard involves TMM normalization and tagwise dispersion estimation through the voom function that (i) transforms previously normalized counts to logCPM, (ii) estimates a mean-variance relationship, and (iii) uses this to compute appropriate observational-level weights [voom weights as done previously [p values.The limma Bioconductor package (v3.38.3) includes a  weights . To adapeviously . The resaldex method infers biological and sampling variation to calculate the expected false discovery rate, given the variation, based on several tests. Technical variation within each sample is estimated using Monte-Carlo draws from the Dirichlet distribution. This distribution maintains the proportional nature of the data while scale-invariance and sub-compositionally coherence of data is ensured by centered log-ratio (CLR). This removes the need for a between-sample normalization step. In order to obtain symmetric CLRs, the iqlr argument is applied, which takes, as the denominator of the log-ratio, the geometric mean of those features with variance calculated from the CLR between the first and the third quantile. Statistical testing is done through Wilcoxon rank sum test, even if Welch’s t, Kruskal-Wallis, generalized linear models, and correlation tests were available. Benjamini-Hochberg correction method was used to correct the p values for multiple testing.ALDEx2 is a Bioconductor package v1.14.1) that uses a Dirichlet-multinomial model to infer abundance from counts [.1 that ulog2(count + 1) is assumed to be a zero-inflated Gaussian mixture model. The mixture parameter is modeled through a logistic regression depending on library sizes, while the Gaussian part of the model is a generalized linear model with a sample-specific intercept which represent the sample baseline, a sample-specific offset computed by Cumulative Sum Scaling (CSS) normalization and another parameter which represents the experimental group of the sample. We opted for the implementation suggested in the original publication [fitZig function to estimate all the parameters. An empirical Bayes approach is used for variance estimation and a moderated t test is performed to identify differentially abundant features between conditions. Benjamini-Hochberg correction method was used to account for multiple testing.metagenomeSeq is a Bioconductor package designed to address the effects of both normalization and under-sampling of microbial communities on disease association detection and testing feature correlations. The underlying statistical distribution for lication , where Ccorncob is an R package (v0.1.0 ) for thesongbird is a python package  that ranmixMC is a multivariate framework implemented in mixOmics, a Bioconductor package (v6.6.1), for omic data analysis . It handi and j represent the ith feature and the jth sample, respectively, is used for the continuous part, modeled as a Gaussian distribution. The kind of data considered, different from scRNA-seq, does not allow the usage of the adaptive thresholding procedure suggested in the original publication [zlm fits the Hurdle model for each feature: the regression coefficients of the discrete component are regularized using a Bayesian approach as implemented in the bayesglm function; regularization of the continuous model variance parameter helps to increase the robustness of feature-level differential expression analysis when a feature is only present in a few samples. Because the discrete and continuous parts are defined conditionally independent for each feature, tests with asymptotic χ2 null distributions, such as the likelihood-ratio or Wald tests, can be summed and remain asymptotically χ2, with the degrees of freedom of the component tests added. Benjamini-Hochberg correction method was used to correct p values.MAST is a Bioconductor package for managing and analyzing qPCR and sequencing-based single-cell gene expression data, as well as data from other types of single-cell assays. The package also provides functionality for significance testing of differential expression using a Hurdle model. Zero rate represents the discrete part, modeled as a binomial distribution while lication . Indeed,FindMarkers function. Rare features, which are present in a fraction lower than 0.1 of all samples, and weak signal features, which have a log fold change between conditions lower than 0.25, are not tested. Benjamini-Hochberg correction method was used to correct p values.Seurat (v2.3.4) R package is a data analysis toolkit for the analysis of single-cell RNA-seq . Brieflyscde.error.models function was used to fit the error models on which all subsequent calculations rely. The fitting process is based on a subset of robust genes detected in multiple cross-cell comparisons. Error models for each group of cells were fitted independently (using two different sets of “robust” genes). Translating in a metagenomic context, cells correspond to samples and genes to taxa or amplicon sequence variants. Some adjustments were needed to calibrate some function default values such as the minimum number of features to use when determining the expected abundance magnitude during model fitting. This option, defined by the min.size.entries argument, set by default at 2000, was too big for many 16S or WMS experiment scenarios: as we usually observe around 1000 total features per dataset (after filtering out rare ones), we decided to replace 2000 with the 20% of the total number of features, obtaining a dataset-specific value. Particularly, poor samples may result in abnormal fits and were removed as suggested in the scde manual. To test for differential expression between the two groups of samples a Bayesian approach was used: incorporating evidence provided by the measurements of individual samples, the posterior probability of a feature being present at any given average level in each subpopulation was estimated. To moderate the impact of high-magnitude outlier events, bootstrap resampling was used and posterior probability of abundance fold-change between groups was computed.The scde Bioconductor package (v1.99.1) with flexmix package (v2.3-13) implements a Bayesian model for scRNA-seq data . Read coFor this analysis, we used the collection of HMP Stool samples in HMP16SData and curatedMetagenomicData. The multidimensional scaling (MDS) plot of the beta diversity did not show patterns associated with known variables , or convergence issues.Starting from the 41 samples, we randomly split the samples into two groups: 21 assigned to group 1 and 20 to group 2. We repeated the procedure 1000 times. We applied the DA methods to each randomly split dataset. Every method returned a p values to the theoretical uniform distribution, as no truly DA features are present. This was summarized in the qq-plot where the bisector represents a perfect correspondence between observed and theoretical quantiles of p values. For each theoretical quantile, the corresponding observed quantile was obtained averaging the observed p values’ quantiles from all 1000 datasets. Departure from uniformity was evaluated with a Kolmogorov-Smirnov statistic. p values were also used to compare the number of false discoveries with 3 common thresholds: 0.01, 0.05, and 0.1.We compared the distribution of the observed p values, fold-changes, or other measures), the CAT statistic was computed in the following way. For a given integer i, concordance is defined as the cardinality of the intersection of the top i elements of each list, divided by i, i.e., L and M represent the two lists. This concordance was computed for values of i from 1 to R.We used the Concordance At the Top (CAT) to evaluate concordance for each differential abundance method. Starting from two lists of ranked features , in which a method was compared to other methods in the same dataset;Within-method concordance (WMC), in which a method is compared to itself in random splits of the datasets.We used CAT for two different analyses:i from 1 to 100.To summarize this information for all pairwise method comparisons, we computed the area under the curve, hence giving a better score to two methods that are consistently concordant for all values of We selected several datasets, with different alpha and beta diversity, for our concordance analysis. Additional file Each dataset was randomly divided in half to obtain two subsets (Subset1 and Subset2) with two balanced groups;DA analysis between the groups was performed with all evaluated methods independently on each subset;p values  obtained from Subset1 was compared to the analogous list obtained from Subset2 and used to evaluate WMC;For each method, the list of features ordered by p values  obtained from Subset1 was compared to the analogous list obtained from Subset1 by all the other methods and used to evaluate BMC for Subset1. The same was done in Subset2;For each method, the list of features ordered by Steps 1–4 were repeated 100 times; andWMC and BMC were averaged across the 100 values (and between Subset1 and Subset2 for BMC) to obtain the final values.The concordance evaluation algorithm can be easily summarized by the following steps:For each dataset, a subset was chosen in order to have a balanced number of samples for each condition. In lower diversity studies  different biological samples from the same subject may be strongly correlated. Hence, we selected only one sample per individual, no matter the condition. To further increase the homogeneity of the datasets, we selected only samples from the same sequencing center.p value less than 0.1 were chosen as DA, for all the methods except songbird and mixMC that return a list of differentials and loadings, respectively. For songbird, a threshold corresponding to the 10% of the total number of taxa was chosen to select the most associated taxa for the considered comparison. mixMC implements a variable selection procedure that automatically selects the most discriminant taxa. We annotated each taxon with the information on genus-level metabolism , classifying each taxon in aerobic, anaerobic, facultative anaerobic, or unassigned.The same low-diversity dataset used in the concordance analysis  was used for the enrichment analysis. The dataset is balanced as it is composed of 38 samples for each body subsite, for a total of 76 samples. DA analysis was performed using Subgingival Plaque as the reference level. Taxa with an adjusted fisher.test  where table is a contingency table. Six contingency tables were built for each method to inspect enrichment of the following:Over-abundant (UP) aerobic taxa in Supragingival Plaque;Under-abundant (DOWN) aerobic taxa in Subgingival Plaque;Over-abundant (UP) anaerobic taxa in Supragingival Plaque;Under-abundant (DOWN) anaerobic taxa in Subgingival Plaque;Over-abundant (UP) facultative anaerobic taxa in Supragingival Plaque; andUnder-abundant (DOWN) facultative anaerobic taxa in Subgingival Plaque.Enrichment analysis was performed via a Fisher exact test, using the function All the information retrieved from the enrichment analysis was summarized in a bar plot, where for each method, the number of differentially abundant taxa together with their direction were represented as a positive (negative) bar for over- (under-) abundant taxa in Supragingival Plaque samples, colored by genus level metabolism.To calculate log odds-ratio for each contingency table, the Haldane-Anscombe correction is applied since it allows the odds-ratio calculation in presence of zero cells. Briefly, it consists in adding a pseudo-count value of 0.5 to each cell of the contingency table to calculate the odds-ratio and a pseudo-count value of 1 to calculate the variance.p values, songbird’s differentials, and mixMC’s loadings were properly ordered;Raw Several thresholds from 1 to 20% of the top ranked taxa in the previously ordered lists were used to select the DA taxa for each method;Putative true positives (TP) were calculated as the sum of aerobic taxa over-abundant in Supragingival Plaque and anaerobic taxa under-abundant in Supragingival Plaque;Putative false positives (FP) were calculated as the sum of aerobic taxa under-abundant in Supragingival Plaque and anaerobic taxa over-abundant in Supragingival Plaque; andThe differences between Putative TP and Putative FP were plotted.To compare all the evaluated methods without considering their power, the followings steps were followed:p values less than 0.1 and the 10% threshold for songbird.To rank all the methods, the same difference was computed, this time using the list of DA taxa based on the adjusted To inspect the concordance of DA taxa between methods, mutual findings were collected and added between the methods. As similar methods tend to identify the same taxa, only one method for each normalization or weighting procedure was considered as representative. This subset contains edgeR with TMM normalization, DESeq2 with poscounts normalization, limma-voom with TMM normalization, MAST, scde, seurat-wilcoxon, corncob , mgsZig, ALDEx2, mixMC, and songbird. The taxa found by most methods in this subset were extracted, but for the graphical representation, all methods were reintroduced.The same analysis was performed in the WMS dataset. However, the sample size was limited to only 5 for the subgingival body subsite, while 88 (with unique RSID) for the supragingival site. For this reason, a 5 vs. 5 sample analysis was performed, randomly selecting five samples from the supragingival dataset. Songbird was not included in the analysis because of an error during the parameter estimation that we were not able to solve. Given the low sample-size, corncob methods with bootstrap were added to the analysis.41 Stool samples available for both 16S and WMS from HMP.208 16S samples and 90 WMS samples of Tongue Dorsum body subsite from HMP.67 Stool and 56 Oral cavity WMS data of Fijian adult women from BritoIl_2016.Several real datasets were used as templates for the simulations:3 and 106, respectively. Moreover, only features present in more than 1 sample with more than 10 reads were kept. After the data filtering step, the simulation framework was established, by specifying the parametric distribution and other data characteristics, described in Additional file Each dataset was filtered to obtain only a sample per individual. 16S and WMS samples were pruned to keep sequencing runs with library sizes of more than 10For each combination of parameters, we simulated 50 datasets, yielding a total of 28,800 simulations. Variables to be included in the simulation framework were chosen based on the role they may play in the analysis of a real experiment.zinbSim function from the zinbwave Bioconductor package easily allows the user to generate both NB and ZINB counts after the zinbFit function estimates model parameters from real data. The user can set several options in zinbFit, we used epsilon = 1e14, common_dispersion = TRUE, and K = 0.NB and ZINB are simple parametric distributions, easy to fit on real data through a reliable Bioconductor package, and above all, seemed to fit 16S and WMS data better than other statistical models see Fig. . The zinGenerating two experimental groups requires the specification of enough samples for each condition and a more or less substantial biological difference between them.Sample size is a crucial parameter: many pilot studies start with 10 or even fewer samples per condition, while clinical trials and case-control studies may need more samples in order to achieve the needed power. We included 10, 20, and 40 samples per condition in our simulation framework.The relative mean for each feature is computed using estimated mean parameter of NB.10 or 50% of features are randomly sampled.foldEffect while the remainings are divided by foldEffect generating up- and downregulated features, respectively. If there is compensation, 1/(1 + foldEffect) of the selected feature relative means are multiplied by foldEffect while the remaining ones are multiplied by (a/b)*(1-foldEffect) + 1, where a is the sum of the relative means of the features that will be upregulated while b is the sum of the features that will be downregulated.If there is no compensation, half of their relative means are multiplied by The resulting relative means are normalized to sum to 1.We considered two different scenarios for the number of features simulated as DA: 10%, representing a case where the majority of the features are not DA, a common assumption made by analysis methods; and 50%, a more extreme comparison. Similarly, we simulated a fold change difference for the DA features of 2 or 5. This is obviously a simplification, since in reality, a continuum gradient of fold effects is present. Nevertheless, it allowed us to characterize the role of the effect size in the performance of the methods. For the DA features, the fold change between conditions was applied to the mean parameter of the ZINB or NB distributions, with or without “compensation” as introduced by [sparsityEffect variable. Acting on the mixture parameter of the ZINB model it is possible to exacerbate downregulation and upregulation of a feature, adding zeros for the former and reducing zeroes for the latter. This scenario provided by 0 , 0.05 and 0.15 of sparsity change should help methods to identify more differentially abundant features. As the mixing parameter can only take values between 0 and 1, when the additive sparsity effect yielded a value outside this range, it was forced to the closer limit.Sparsity is a key characteristic of metagenomic data. The case in which a bacterial species presence rate varies between conditions was emulated in the simulation framework via the so called p values were used to compute the false positive rate (FPR = 1 – Specificity) and the true positive sate (TPR = Sensitivity). Partial areas under the receiver operating characteristic (pAUROC) curve with an FPR from 0 to 0.1 values were computed and then averaged in order to obtain a single value for each set of variables.The previously described DA methods were tested in each of the simulated datasets (50 for each set of simulation framework parameters) and the adjusted To measure the computational times for all the 18 methods, we used the Subgingival vs. Supragingival Plaque HMP 16S dataset where a total of 76 samples and approximately 900 taxa were available. The evaluation was performed on a laptop computer with O.S. Windows 10 64bit, Intel® i7-8th Gen CPU with 16GB of RAM. Moreover, the Stool 16S and WMS parametric simulation datasets  were used in order to measure each method’s computational complexity (except for mixMC and songbird). Time evaluation was performed on a single core for each dataset where all methods are tested sequentially and then properly averaged with the values of all the simulations. The methods’ performance evaluations in power analysis on the 28,800 total parametric simulations were performed in the same way, equally dividing the simulated datasets across 30 cores. The working machine was a Linux x86_64 architecture server with 2 Intel® Xeon® Gold 6140 CPU with 2.30 GHz for a total of 72 CPUs and 128 GB of RAM.Additional file 1: Supplementary Figures and Tables. Contains figures from S1 to S9 and tables from S1 to S3.Additional file 2:Parametric simulations. Contains the description and the results of the parametric simulations.Additional file 3: Review history.


Gliomas are currently classified through integration of histology and mutation information, with new developments in DNA methylation classification. However, discrepancies exist amongst the major classification methods. This study sought to compare transcriptome-based classification to the established methods. RNAseq and microarray data were obtained for 1032 gliomas from the TCGA and 395 gliomas from REMBRANDT. Data were analyzed using unsupervised and supervised learning and other statistical methods. Global transcriptomic profiles defined four transcriptomic glioma subgroups with 91.4% concordance with the WHO-defined mutation subtypes. Using these subgroups, 168 genes were selected for the development of 1000 linear support vector classifiers (LSVC). Based on plurality voting of 1000 LSVC, the final ensemble classifier confidently classified all but 17 TCGA gliomas to one of the four transcriptomic profile (TP) groups. The classifier was validated using a gene expression microarray dataset. TP1 cases include IDHwt, glioblastoma high immune infiltration and cellular proliferation and poor survival prognosis. TP2a is characterized as IDHmut-codel, oligodendrogliomas with high tumor purity. TP2b tissue is mostly composed of neurons and few infiltrating malignant cells. TP3 exhibit increased NOTCH signaling, are astrocytoma and IDHmut-non-codel. TP groups are highly concordant with both WHO integrated histology and mutation classification as well as methylation-based classification of gliomas. Transcriptomic profiling provides a robust and objective method to classify gliomas with high agreement to the current WHO guidelines and may provide additional survival prediction to the current methods. They exhibit highly variable response to chemoradiation therapy and survival prognosis4. It is important to accurately classify these tumors for better treatment and prognostication. Gliomas are classified using a combination of histology and molecular testing5. Histologic diagnoses include glioblastoma, astrocytoma, and oligodendroglioma6. However, inter-observer disagreement with histologic diagnosis is quite high7. Glioma classification became more automated and systematic9 with the discovery of mutations in the isocitrate dehydrogenase (IDH) gene10 5 of non-gliobastomas, as compared to IDH wild type (IDHwt) glioblastomas13 and co-deletion of chromosome arms 1p/19q in oligodendroglioma16. IDHwt glioblastomas are additionally evaluated for O(6)-methylguanine-DNA methyltransferase (MGMT) promoter methylation to determine likelihood of tumor response to the alkylating agent temozolomide18.Gliomas are neoplasms that mostly arise from the cerebral hemisphere of adults19 and a subgroup within IDHwt glioblastomas has been described as pilocytic astrocytoma-like, with good survival prognosis9. WHO has identified the need of additional molecular markers to delineate the poor prognosis IDHwt patients19. Furthermore, secondary glioblastomas occasionally arise from a primary astrocytoma lesion20. These carry the poor prognosis of glioblastomas, but are IDHmut8.Yet, mutational classifications are suboptimal prognostic markers. Glioblastoma patients with IDHwt are among the worst prognosis gliomas; however, many good prognosis gliomas (i.e. grade 1 pilocytic astrocytomas and gangliogliomas) are IDHwt22 and biases24, which are especially apparent under time constraints25. It is therefore necessary to develop objective and automated measures of tumor classification. Much progress has been made to use machine learning to automate and objectify cancer diagnosis and subtyping28. In 2010, Verhaak et al.29 described the transcriptomic classification of glioblastomas into four clusters using consensus clustering30. However, since this method was developed from glioblastomas, it may provide limited use in classifying other gliomas. Additionally, some reports have described limitations and potential inaccuracies in using the consensus clustering algorithm31. Recently, two reports have highlighted the potential use of DNA methylation to classify central nervous system (CNS) tumors with high accuracy and then to identify new subclasses32. Both reports showed overall agreement with IDH-codeletion classes, but the minor disagreements and new subclasses are associated with changes in grade or prognosis. The TCGA reported six glioma DNA methylation classes. Three of the clusters mainly subclassify glioblastomas. Interestingly, even though patients with IDH mutated gliomas usually have better survival prognosis than patients with IDH wt gliomas, TCGA reported the LGm1 DNA methylation cluster identifies IDH mutated cases associated with worse survival prognosis9.Current glioma diagnosis using histology and IDH-codeletion testing is sufficient for most cases, but there are inconsistencies. Histologic diagnoses depend on human pattern recognition and is prone to errors33. Second, accurate tumor classification is necessary to perform many statistical analyses involving these tumor types. Differential frequencies and expressivities innate in tumor subtypes may confound statistical association analyses. For example, if a cancer type has an un-identified poor prognosis molecular subtype, a project to determine genes associated with poor prognosis in a cancer may only identify genes enriched in this molecular subtype. This is well studied in terms of population stratification for genome wide association studies (GWAS)34.While these molecular classifications provide an important step forward for automated and objective glioma classification, a diagnosis from one “-omic” platform is not enough for defining a tumor class. Logically, with each -omic platform which agrees on the classification of sample “x” in class “y”, the confidence in sample “x” truly belonging to class “y” increases. Hence, defining distinct tumor classes requires multiple -omics platforms to corroborate the same classification. Confident tumor classification from multiple -omics platforms provides two potential benefits to cancer research. First, the identification of molecular classes may highlight new targetable pathways for therapeutic intervention35, this has not been performed comprehensively for all gliomas.In order to better define molecular subtypes of gliomas, we sought to develop a transcriptome-based glioma classification method using unsupervised classification techniques and compared the findings to other glioma classification methods. While glioblastomas have been characterized using transcriptome subtypes9. The final dataset contained 1032 samples and 12,717 genes. TCGA Glioma data was centered and scaled. Clinical data were downloaded from the same source and matched to the processed TCGA Glioma data.TCGA Glioma gene expression data, which contains both RNAseq and gene expression microarray data combined through Empirical Bayes, was downloaded from Ceccarelli et al. 201637 (https://gdoc.georgetown.edu/gdoc/), both on September 19, 2018. The expression data were normalized using Expression Atlas, which applied RMA from “oligo”38 (v 1.36.1). Outlier Detection was performed with boxplots (Kolmogorov–Smirnov statistic Ka), distances between arrays (35 detected), and MA plots (Hoeffding’s statistic Da > 0.15)8 on Expression Atlas. In total, 43 outlier samples were removed. Three hundred and ninety five cases overlapped between the outlier removed expression data and clinical data. For genes with multiple probes, the probes with highest mean intensity was used and all others removed. REMBRANDT data was standardized before supervised classification. Only 161 of the 168 genes used for supervised classification were present in the microarray dataset.Normalized gene expression data from fresh frozen tumor were downloaded from E-MTAB-3073 through ArrayExpress and clinical data were downloaded from G-DOC39 package were determined using a grid search. The grid search parameter optimization was performed by visual inspection to maximize cluster density and intercluster distances on UMAP plots. Parameters were number of nearest neighbors (n_neighbors), minimum distance (min_dist), number of genes, and distance metric (metric). The final selected parameters were 1000 random genes, 5 nearest neighbors, 0 minimum distance, and the “manhattan” metric.The optimal parameters for uniform manifold approximation and projection (UMAP) implemented in the “umap”40. The minimum points parameter was empirically determined as 100 based on the approximate number of cases in each of the three groups and the optimal eps parameter was 1.55 based on the elbow of the k nearest neighbors distance plot. The density-based UMAP (DBU) algorithm combines the UMAP and DBSCAN steps repeated 1000 times based on subsampling of 1000 random genes. Hierarchical clustering of the DBU iterations revealed a subset of the iterations required manual switching of group assignment to agree with the majority of iterations. Thirty-four models that only identified two groups were removed from subsequent analyses.The two dimensional data from UMAP was entered into the DBSCAN algorithm from the “fpc” packageThe samples were classified based on plurality voting, where the sample class is assigned as the class with the most votes from the DBU iterations. Samples are designated as “ambiguous” if less than 70% of the models assign the sample to one specific group or if the most commonly assigned group was Group 0 . Most (96.6%) iterations/models classified the vast majority of patients into three main groups with a small number of patients that cannot be unambiguously assigned to one of the three groups. Some UMAP models showed potentially further heterogeneity within Group 2. Further analysis identified relevant molecular differences for two clusters within Group 2. Thus, the final groups were designated TP1, TP2a, TP2b, and TP3 to delineate the two subgroups of group 2.41 and 26 genes were manually selected based on relevance to brain cancer. The remaining 142 genes were selected using recursive feature elimination with a support vector classifier. These genes were divided into six groups based on the expression differences among the subtypes. For each supervised model, half of the genes in each of six gene groups were randomly selected and then recursive feature elimination was applied removing five genes per iteration until optimal accuracy is reached with the minimal number of genes using “sci-kitlearn42”. This was repeated 1000 times, resulting in a dictionary with 1000 entries each with between 29 and 79 genes of the 168 genes. One thousand linear support vector classifiers (LSVC) were developed from the dictionary and the mean accuracy from threefold cross validation was used to remove any models with average accuracy less than 95%. All 1000 models passed this step and average model accuracy was 97.6%.One hundred and 68 genes were identified for use in supervised classification using two complementary methods. First, significantly differentially expressed genes amongst the four DBU groups were identified through LIMMA analysisIn order to decrease the potential of overfitting, data were split into fourfolds, where three folds were trained on the unsupervised model classes and the supervised models predicted on the remaining fold. In this way, no sample was used for both training and making classification calls. This results in calls for each sample from 1000 linear SVC models. Models were combined into one ensemble model using a plurality voting method which reports the most popular class and the proportion of the 1000 LSVC models which agree on this most popular class. A confidence score is calculated by taking the proportion of models classifying samples into the most popular class divided by the proportion of models classifying samples into the second most popular class. If the confidence score is greater than 3, then the ensemble model classifies the sample into the most popular class. If the confidence score is less than or equal to 3, than the ensemble model prediction is “ambiguous”.43 (https://xenabrowser.net/datapages/) GBMLGG somatic non-silent mutation (gene-level) PANCAN AWG TCGA Hub for 461 cases. “0” is “no mutation” and “1” is “mutated.” Genes with no mutation in any samples were removed. Chi-squared analysis for one vs rest was performed and p-values were adjusted with Benjamini-Hochberg44.Mutation data for TCGA Glioma data was downloaded from UCSC Xena41 was used for differential expression analysis comparing each group vs the rest  after removal of ambiguous samples based on the supervised classification.LIMMA45 pathway analysis. Gene Set Enrichment Analysis (GSEA) was implemented with the “fgsea”46 R package. Reactome47 pathways were downloaded from MSigDB (674 pathways). Pathways were filtered for minimum size of 15 genes and maximum size of 500 genes, resulting in 440 pathways. One hundred and ninety-three pathways were enriched in at least one group. There were 123 significantly enriched pathways in TP1, 57 significantly enriched pathways in TP2a, 146 significantly enriched pathways in TP2b, and 26 significantly enriched pathways in TP3. Significant pathways are those with Benjamini-Hochberg-adjusted p-values < 0.05.Results from LIMMA were used for GSEA48. KM plots were made through the “survminer” package49.We modeled survival with Kaplan–Meier (KM) and Cox proportional hazards and tested for significance with the log rank test all using the “survival” R package50, which uses DNA copy number data to calculate tumor purity, and the ESTIMATE algorithm51, which uses DNA methylation data to calculate the proportion of immune and stromal cells in the bulk tumor.Tumor purity measures were obtained from TCGA Ceccarelli et al. data based on the ABSOLUTE algorithmhttps://ww.r-project.org)52.All statistical analyses were performed using the R language and environment for statistical computing 39 (termed “DBU”). Our pipeline randomly samples 1,000 out of 12,717 genes and applies the UMAP dimension reduction algorithm, followed by a DBC algorithm, for 1000 iterations. Most (96.6%) iterations/models classified the vast majority of patients into three main groups with a small number of patients that cannot be unambiguously assigned to one of the three groups The final DBU group assignment is determined by plurality voting, where the “winner” needs support from 70% or more of the 966 models. If no group wins at least 70%, the patient is called “ambiguous”. Plurality voting assigned 603 (58.4%) patients to “Transcriptome Profile 1” (TP1), 174 (17.0%) patients to “TP2”, 208 (20.1%) patients to “TP3” and 47 (4.6%) samples as “ambiguous.” The ambiguous samples may have been more difficult to classify because they had lower tumor purity than other samples that is not related to immune infiltration  status in combination with histology to classify infiltrating gliomas. Our transcriptomic classification is 91.4% concordant with IDH-codel classification  IDHmut-non-codel patients were classified as TP1. As shown in Fig. 0.7% IDHmThe second major discrepancy between the transcriptomic and IDH-codeletion-based classification systems involves 31 patients (23 IDHmut-non-codel and 8 IDHwt) that were classified in the TP2 group, which is predominantly IDHmut-codel (144 of 175 or 82.3%). These 31 patients are within a distinct subgroup on a subset of UMAP models , each 38 genes on average from the 168 gene set  as proneural, which is associated with better prognosis. In this case, Verhaak classification agrees with the mutation classification and disagrees with both the TP subtypes and histology, even though the survival prognosis for these cases favors the latter classification. Our transcriptomic classification is distinct from the method reported by Verhaak et al. Our method also reflects the mutation system and patient survival prognosis more closely than the Verhaak classification.We compared our ETC classifications to the three transcriptomic clusters previously reported by Verhaak et al., “Classical”, “Mesenchymal”, and “Proneural”9, LGm1 through LGm6. We refer to these six groups as the “TCGA methylation” classification. In the TCGA cohort, the concordance between our two methods was 87% while the concordance between the IDH-codel and the TCGA methylation classification was 94%  we analyzed are classified as LGm4 or LGm5, which are both associated with worse survival prognosis in the TCGA methylation classificationTCGA methylation classifies the IDHmut-non-codel glioblastoma TP1 cases (†) as LGm1, which is associated with worse prognosis. In this case, TCGA methylation classification agrees with both our TP subtypes and histology and disagrees with the mutation and Verhaak classifications. The former set of classifications better reflect survival prognosis. Our transcriptomic classification is similar to the TCGA methylation classification.In order to discern the distinct contributions of each molecular classifier to survival prognosis, we developed a multivariable cox regression model which includes TP, mutation, Verhaak, and TCGA methylation subtypes. We found that only the TP1 group from our classification and the LGm4 and LGm5 groups from the TCGA methylation classification remained significant for survival prognosis in the full survival model Fig. . Thus, oGene set enrichment analysis (GSEA) of TCGA transcriptomic data revealed TP1 is enriched for cell cycle, DNA repair, transcription, immune, ECM, and telomere maintenance pathways. TP2b is enriched for neurotransmission, GPCR signaling, and insulin secretion pathways. TP2a is enriched for similar pathways as TP2b albeit less significant. TP3 is enriched for immune, cell cycle, NOTCH signaling, transcription, and translation Fig. A. Full G19. TP2a is enriched for TERT promoter, FUBP1, NOTCH1, CIC, IDH1, and IDH2 mutations. FUBP1 and CIC are both often mutated in TP2 and are known to be associated with oligodendroglioma53. NOTCH1 mutation has been more recently associated with 1p/19q codeletion8 and may play a role in gliogenesis54 but its role in cancer development is unclear. TP3 is enriched for IDH1, ATRX, and TP53 mutations, all of which have reported associations with IDHmut-non-codel gliomas. DAXX, BRAFV600E, and PDGFRA mutations were not enriched in any groups. TP2a is enriched for PDGFRA copy number deletions (26%) and TP1 is enriched for EGFR copy number amplifications (88%) . Thus, the mutational profile of IDHmut TP1 cases is more similar to IDHmut/TP3 cases than to IDHwt/TP1 cases .The REMBRANDT data is a microarray dataset consisting of 395 samples diagnosed as WHO grade 2, 3, or 4 gliomas. UMAP embedding of the combined REMBRANDT gene expression and TCGA data validated four distinct transcriptomic clusters Fig. .Figure 5We noted more astrocytoma and oligodendroglioma cases were classified as TP1 than expected and assessed if this reclassification from ETC reflected survival differences. TP1 cases that are histologically astrocytoma or oligodendroglioma have worse survival prognosis than non-TP1 cases Fig. D,E. In f22, and can be used in establishing gold standard datasets devoid of erroneous and questionable samples for the development of automated tumor classifiers (Fig. We developed and validated an unbiased, automated pipeline for transcriptomic clustering. Without any domain knowledge, our classifier recapitulated known glioma subtypes from histology and mutation status. Our analytical pipeline avoids the potential of overfitting a supervised model to misclassified or mishandled samplesers Fig. .We developed supervised classifiers from TCGA data and validated with the REMBRANDT microarray dataset Fig. . Most prETC clustered all TCGA samples except 17 into four transcriptome profile groups. Characteristics for TP1 cases include IDHwt, glioblastoma Fig. A, high iOur study can explain the transcriptomic nature of the discrepant groups between histology and mutation. Specifically, both IDHmut glioblastomas and IDHwt astrocytomas are both TP1 and therefore transcriptionally more similar to grade IV glioblastomas.55. Our data agree with these findings, since the IDH mutation associated groups TP2a and TP3 have higher tumor purity and lower immune infiltration compared to the other groups in TCGA data. Interestingly, while TP2a was enriched for NOTCH1 mutations, TP3 was enriched for NOTCH signaling gene pathways (Fig. 56.TP2a is characterized as IDHmut-codel, oligodendrogliomas Fig. D. Meanwhays Fig. . The con35 in that the “proneural” group includes tumors from TP1, TP2a, and TP3 while the “mesenchymal”, and “classical” groups mostly include TP1 tumors. However, it is similar to the DNA methylation classification reported by TCGA9. TP1 mostly includes LGm4, LGm5, and LGm6, TP2a mostly includes LGm3, and TP3 mostly includes LGm2 (Fig. Our classification is unique from the transcriptomic classification reported by Verhaak et al.36. The classifier confidently assigns most samples to one of the TP groups, a remarkable observation given the difference in array platforms. More importantly, survival is consistent with patient’s TP status, even within histological subtypes, further validating the utility of ETC (Fig. The validity of ETC was demonstrated using gene expression data from a 395-case microarray datasetIn conclusion, transcriptomic profiling provides an unbiased classification method that can aid in tumor classification. When applied to glioma transcriptome data, the method agreed with the major classification methods, including the WHO integrated histology and mutation classification as well as methylation-based classification of gliomas.Supplementary tables.


The development of single-cell methods for capturing different data modalities including imaging and sequencing has revolutionized our ability to identify heterogeneous cell states. Different data modalities provide different perspectives on a population of cells, and their integration is critical for studying cellular heterogeneity and its function. While various methods have been proposed to integrate different sequencing data modalities, coupling imaging and sequencing has been an open challenge. We here present an approach for integrating vastly different modalities by learning a probabilistic coupling between the different data modalities using autoencoders to map to a shared latent space. We validate this approach by integrating single-cell RNA-seq and chromatin images to identify distinct subpopulations of human naive CD4+ T-cells that are poised for activation. Collectively, our approach provides a framework to integrate and translate between data modalities that cannot yet be measured within the same cell for diverse applications in biomedical discovery. Integration of single cell data modalities increases the richness of information about the heterogeneity of cell states, but integration of imaging and transcriptomics is an open challenge. Here the authors use autoencoders to learn a probabilistic coupling and map these modalities to a shared latent space. High-throughput and high-content single-cell technologies have provided important insights into genome architecture (using imaging and chromosome capture methods)5 as well as detailed genome-wide epigenetic profiles and expression maps (using various sequencing methods)8. However, obtaining high-throughput paired measurements of these different data modalities within single cells is still a major challenge requiring significant breakthroughs in single-cell technologies.Recent evidence has highlighted the importance of the 3D organization of the genome to regulate cell-type-specific gene expression programs11 or of different modalities with the same data structure such as various sequencing measurements15. We here present a computational framework based on autoencoders for integrating and translating between different data modalities with very distinct structures. Several works have proposed using autoencoders for domain adaptation (in particular batch correction) in the context of biological data17. Different from these works, our method uses autoencoders to integrate and translate between different data modalities that may have very different representations. A separate line of work has proposed using neural networks to directly translate between pairwise modalities in an unsupervised manner19 or with side information21. These methods tend to focus on modalities with similar representations  and directly translate between pairs of modalities without learning a common latent representation of the data. In contrast, our work maps each data distribution to a common latent distribution using an autoencoder. This not only enables data integration and translation between arbitrary modalities in a globally consistent manner, but, importantly, it also enables performing downstream analysis such as clustering across multiple modalities at once. Other work has proposed coupled autoencoders to translate between paired biological data22, which differs from our method that does not require paired data. Building on Makhzani et al.23, we align the latent space of an autoencoder using adversarial training and leverage this technique for data integration and/or translation. In particular, our framework can be applied to integrate and translate imaging and sequencing data, which cannot yet be obtained experimentally in the same cell, thereby providing a methodology for hypothesis generation to predict the genome-wide expression profile of a particular cell given its chromatin organization and vice-versa. Such a methodology is valuable to understand how features in one dataset translate to features in the other.Different data modalities provide different perspectives on a population of cells and their integration is critical for studying cellular heterogeneity and its function Fig. a. Curren26, neural networks consisting of an encoder (mapping to the latent space) and a decoder , whose architectures can be customized to the specific data modality  of the data over all modalities:pZ is the probability density of Z, and Xi given Z that reflects the generative process. Multi-modal data integration can then be formalized as the problem of learning conditional distributions PZ based on samples from the marginal distributions Xi is obtained via a deterministic function of Z implies that the latent distribution of each dataset is the same. However, by including the noise variables Ni as in Equation  to minimize the lossλ > 0 is a hyperparameter, L1 is the (Euclidean) distance metric, L2 represents a divergence between probability distributions, and Xi after embedding to the latent space i to j is accomplished by composing the encoder from the source domain with the decoder from the target domain, i.e.,27.When the latent distribution PZ is not usually known in practice, it must also be estimated from the data. This can be done using the following approaches: (i) learn PZ by training a regularized autoencoder on data from a single representative domain; or (ii) alternate between training multiple autoencoders until they agree on an invariant latent distribution. The first approach is typically more stable in practice, while the second captures variability across multiple domains and is therefore more suitable for integrating multiple datasets. Note that PZ is by no means unique; there are multiple solutions that can result in the same observed data distributions in the different domains.Since i, j ∈ {1, …, K} is learned as follows. Let i, we optimize the objectivej, we optimize the objectiveEi, Di) by neural networks and minimize the objective function via stochastic gradient updates. In particular, L2 can be chosen to be the discriminative loss,To be concrete, an invariant latent distribution based on two domains Prior knowledge is sometimes available to guide the integration of different data modalities. For example, there may be knowledge of alignment of specific markers or clusters, or knowledge of certain samples from different datasets corresponding to the same cell, i.e., the same point in the latent space. In this case, training of the autoencoders can be guided by additional loss functions that incorporate the prior knowledge.L2 above with the following discriminative loss that is conditioned on these factors:P and Q are now joint distributions over the data and the markers and/or clusters. This approach is valid for both discrete and continuous values of the cluster/marker y. For example, in ref. 27, this approach was used to align a continuous differentiation marker between RNA-seq and ChIP-seq data. Alternatively, if the markers or clusters can take m discrete values , then we can add a simple classifier model pθ(Y∣Z) with parameters θ and minimize the lossθ and the parameters of the encoders Ei; here Pi is the distribution of the ith data modality. This loss function encourages data points with the same class label irrespective of the data modality to be clustered together in the latent space.Discriminative loss to align shared markers/clusters among datasets: If there are shared markers or clusters that are present in two datasets, they can be aligned by replacing E and Anchor loss to match paired samples: If 28. While our autoencoder framework is designed to integrate vastly different data structures, in the following we show that our framework is competitive with previous methods for the simpler problem of integrating different modalities with similar data structures. For details on the implementation see Supplementary Methods, Supplementary Table k-nearest neighbors accuracy, i.e., the proportion of cells whose true match is within the k closest samples in the latent space (in ℓ1-distance) or in the original space for methods that do not rely on the latent space.Recent technological advances have made it possible to obtain paired single-cell RNA-seq and ATAC-seq data. Such paired data was collected from human lung adenocarcinoma-derived A549 cells treated with dexamethasone (DEX) for 0, 1, or 3 hours in ref. 29, which determines a nonlinear transformation of the two datasets to maximize the correlation of the resulting representations, as well as Seurat, a prominent method for biological data intergration of similar modalities12. In addition, we compare our cross-modal autoencoder model to two additional methods that do not rely on the latent space for alignment of modalities, namely CycleGAN19 and MAGAN21. Similar to CycleGAN, our cross-modal autoencoder does not require paired samples, which is advantageous for many modalities, where the process of data collection results in destruction of the cell  and thus the same cell cannot be used in another assay to measure a different modality . However, if additional information is available such as shared markers measured in all modalities and/or paired data, similar to the MAGAN approach, this prior information can be incorporated through additional terms in the loss function (see section on incorporating prior knowledge). In terms of comparisons with methods that align modalities in the latent space, our autoencoder framework outperforms Seurat and is competitive with DCCA for integrating single-cell RNA-seq and single-cell ATAC-seq data both in terms of fraction of cells assigned to the correct cluster  T-cells  from ref. lls Fig. a and by lls Fig. b. The co32, we hypothesized the presence of two subpopulations of naive T-cells with distinct chromatin packing features. To test this, we carried out DAPI-stained imaging experiments of naive CD4+ human T-cells and analyzed their chromatin organization  and chromatin packing (from single-cell imaging data). To link the identified subpopluations from the unpaired datasets, we used our cross-modal autoencoder framework to integrate the single-cell RNA-seq data with the chromatin images (Methods and Supplementary Table In order to assess whether translated image (or RNA-seq respectively) datasets are still able to separate poised and quiescent subpopulations  and analyze if the clusters obtained separately from gene expression and imaging datasets align with each other, we performed Receiver Operating Characteristic (ROC) analysis on the translated datasets. For RNA-seq, we first trained a random forest classifier (using 100 trees in a forest with 2 as the maximum depth of a tree) on the RNA-seq data with labels based on poised versus quiescent clustering of naive CD4+ T-cell gene expression data. This classifier learned the genes that separate the two clusters. Next, we translated chromatin images into RNA-seq using our autoencoder method and assessed the performance of the pre-trained classifier on its ability to separate central versus peripheral clusters on images translated to RNA-seq . Note the considerable overlap between the true and predicted marker genes . In summary, in the gene expression matrix translated from the imaging dataset, we found that the differential expression of genes was strongly correlated with the true observed differential gene expression and that the predicted and observed marker genes showed considerable overlap.Imaging datasets can provide a rich quantification of cells, such as their chromatin organization. Based on image analysis, subpopulations of cells with different characteristics may be found , and it is often of interest to study which genes might be markers of each subpopulation such that these subpopulations can be separated for example using antibodies against the marker genes. However, generally the full gene expression and imaging features cannot be measured in the same cell. Our computational framework can translate chromatin images into RNA-seq and calculate the predicted mean difference in expression between the subpopulations . As shown in Fig. nes Fig. e. We alsCORO1A and RPL10A, which were predicted to be strongly upregulated in the naive T-cell subpopulations with central and peripheral patterns of chromatin density respectively  as per the manufacturer’s instructions. The cells for the experiments were used within two days upon revival.Cells in media were allowed to adhere to Poly-lysine coated slides for 30 minutes. Cells were then fixed with 4% Paraformaldehyde (Sigma) for 30 minutes and washed with PBS three times, which also removed unattached cells. Permeabilization was done with 0.5% Triton X-100 (Sigma) for 10 minutes followed by PBS washes. Blocking was done with 5% BSA in PBS for 30 minutes and incubated with primary and secondary antibodies as per the dilution and incubation time recommended by the manufacturer. The primary antibodies used in this study are anti-RPL10A antibody  and Anti-Coronin 1a/TACO antibody . Cells were washed with PBS (+0.1% Tween) three times after primary and secondary antibody incubation. During the final step, excess liquid was removed by slanting the slides. ProLong® Gold Antifade Mountant with DAPI (ThermoFischer Scientific) was added to these slides and allowed to cure for 24 hours. Coverslips were then sealed and imaged using a confocal microscope.μm and 0.5 μm depth. Images were processed and further analyzed using custom programs in Fiji and R (see below in code availability).1024  ×  1024 and 12-bit multi-channel images were obtained using a Nikon A1R confocal microscope. Z-stack images were captured using a 100×  objective with a pixel size of 0.1 x, y, and z iteratively until the volume of the eroded nucleus was less than 10 cubic microns. Then the mean intensity of each 3D ring (width 0.5 microns) in the nucleus was computed for all cells. The intensity fraction was calculated by normalizing the mean ring intensity for each nucleus (maximum = 1). Linear interpolation was then used to compute the intensity fraction of rings that occupy 0–10% to 90–100% volume fraction of the nucleus. The heatmaps were visualized using functions from gplots, RColorBrewer and dendextend.The nuclear boundaries were segmented in 3D using the DAPI channels to identify individual nuclei. These nuclei were eroded by 0.5 microns in x, y, and z. This was efficient as the cells were all spherically shaped with high karyoplasmic index. The total intensity in the 3D cellular object was computed for each protein channel and their ratio was obtained for each cell.In order to calculate the cellular levels of proteins, the 3D nuclear object was dilated by 2 microns in p-value of  <0.05. Fig. 38, keeping the top 5 gene ontology biological process terms with lowest p-values  were ≤0.05.We aimed to explore potential heterogeneity in naive CD4+ T-cell gene expression in relation to CD4+ T-cells that already underwent activation. We performed a feature selection step, keeping genes which had average log-fold change of  >0.05 between naive and activated CD4+ T-cells (and vice-versa), resulting in 1187 genes. Similar to the analysis of PBMCs (see Supplementary Methods), we applied PCA for dimensionality reduction on the selected genes, keeping the top 30 components and clustered the naive CD4+ T-cells using the default clustering method in Seurat version 2.3.0 with resolution of 0.8 (Supplementary Fig. k-nearest neighbor graph with k ∈ {10, 20, 50, 100} to cluster the data. We performed 100 different initializations for each method and computed the co-association matrix, which quantifies how often each pair of cells was clustered together; the result is shown in Fig. Since the identification of the two subpopulations of naive T-cells is an important step in our analysis, we thoroughly evaluated its robustness with respect to number of clusters and clustering methods. We re-clustered the data corresponding to naive CD4+ T-cells using Seurat version 2.3.0 with different resolution parameters, i.e., 0.9, 1.1, and 1.15 to obtain 3, 4, and 5 clusters respectively. We computed the silhouette coefficient for each clustering, observing that the number of clusters corresponding to 2 gave the highest score (Supplementary Fig. λ1 = 0.1, λ2 = 1. The architectures of all networks are shown in Supplementary Table Images were normalized to range between  and RNA-seq matrix was Supplementary Methods, Supplementary Figs. Further information on research design is available in the Supplementary InformationPeer Review FileDescription of Additional Supplementary FilesSupplementary Data 1Supplementary Data 2Reporting Summary