High-dimensional multi-omics data are now standard in biology. They can greatly enhance our understanding of biological systems when effectively integrated. To achieve proper integration, joint Dimensionality Reduction (jDR) methods are among the most efficient approaches. However, several jDR methods are available, urging the need for a comprehensive benchmark with practical guidelines. We perform a systematic evaluation of nine representative jDR methods using three complementary benchmarks. First, we evaluate their performances in retrieving ground-truth sample clustering from simulated multi-omics datasets. Second, we use TCGA cancer data to assess their strengths in predicting survival, clinical annotations and known pathways/biological processes. Finally, we assess their classification of multi-omics single-cell data. From these in-depth comparisons, we observe that intNMF performs best in clustering, while MCIA offers an effective behavior across many contexts. The code developed for this benchmark study is implemented in a Jupyter notebook—multi-omics mix (momix)—to foster reproducibility, and support users and future developers. Advances in omics technology have resulted in the generation of multi-view data for cancer samples. Here, the authors compare dimensionality reduction techniques using simulated and TCGA data and identify the features of the methods with superior performance. Moreover, multiomics profiling approaches are currently being transposed at single-cell level, which further stresses the need for methods and tools enabling the joint analysis of such large and diverse datasets2.Due to the advent of high-throughput technologies, high-dimensional “omics” data are produced at an increasing pace. In cancer biology, in particular, national and international consortia, such as The Cancer Genome Atlas (TCGA), have profiled thousands of tumor samples for multiple molecular assays, including mRNA, microRNAs, DNA methylation, and proteomics3. Achieving proper multiomics integration is crucial to bridge the gap between the vast amount of available omics and our current understanding of biology. By integrating multiple sources of omics data, we can reduce the effect of experimental and biological noise. In addition, different omics technologies are expected to capture different aspects of cellular functioning. Indeed, the different omics are complementary, each omics containing information that is not present in others, and multiomics integration is thereby expected to provide a more comprehensive overview of the biological system. In cancer research, omics have been profiled at different molecular layers, such as genome, transcriptome, epigenome, and proteome. Integrating these large-scale and heterogeneous sources of data allows researchers to address crucial objectives, including (i) classifying cancer samples into subtypes, (ii) predicting the survival and therapeutic outcome of these subtypes, and (iii) understanding the underlying molecular mechanisms that span through different molecular layers4.While multiomics data are becoming more accessible, studies combining different omics are more common. This multiomics integration is frequently performed by sequentially combining results obtained on single omics (a.k.a. late or early integration), but the genuine joint analysis of multiomics data (a.k.a. intermediate integration) remains very rare5. Indeed, the different types of omics have a large number of heterogeneous biological variables and a relatively low number of biological samples, thereby inducing statistical and computational challenges, in addition to the typical challenges of “Big Data”. Moreover, each omics has its own technological limits, noise, and range of variability. All these elements can mask the underlying biological signals. Multiomics integrative approaches should be able to capture not only signals shared by all omics data, but also those emerging from the complementarity of the various omics data.Designing theoretical and computational approaches for the joint analysis of multiomics datasets is currently one of the most relevant and challenging questions in computational biology6. Bayesian methods, such as Bayesian Consensus Clustering (BCC)7, build a statistical model by making assumptions on data distribution and dependencies. Network-based methods, such as Similarity Network Fusion (SNF)8, infer relations between samples or features in each omics layer, and further combine the resulting networks. Dimensionality Reduction (DR) approaches decompose the omics into a shared low-dimensional latent space10. Four recent reviews tested and discussed some of these methods from a clustering performance perspective14. Pierre-Jean et al.13, Rappoport et al.11, and Tini et al.14 selected one method from each of the aforementioned three categories, while Chauvel et al.12 focused on Bayesian and DR approaches.The joint analysis of multiple omics can be performed with various integrative approaches, classified in broad categories15.From these initial reviews, DR approaches emerged as particularly well-performing. Of note, DR is employed in computational biology in different contexts, such as data visualization or matrix completion. We focus here on DR for multiomics data integration. DR methods are well-adapted to solve high-dimensional mathematical problems. Furthermore, the richness of the information contained in their output enhances their relevance for multiomics integration. Indeed, DR methods enable the classification of samples (clustering/subtyping), the clinical characterization of the identified clusters/subtypes and a variety of other downstream analyses, including the analysis of cellular processes and/or pathways Fig. . Thus, DHere, we report an in-depth comparison of nine representative state-of-the-art multiomics joint Dimension Reduction (jDR) approaches, in the context of cancer data analysis. We extensively benchmark these approaches, spanning the main mathematical formulations of multiomics jDR, in three different contexts Fig. . First, All these analyses allow formulating recommendations and guidelines for users, as well as indications for methodological improvements for developers. We also provide the Jupyter notebook multiomics mix (momix) and its associated Conda environment containing all the required libraries installed . Overall, momix can be used to reproduce the benchmark, but also to test jDR algorithms on other datasets, and to evaluate novel jDR methods and compare them to reference ones.P omics matrices Xi, ni × m with ni features  and m samples. A jDR jointly decomposes the P omics matrices into the product of ni × k omics-specific weight/projectionmatrices (Ai) and a k × m factor matrix (F) . Factors and metagenes represent the projections on the sample space and gene space, respectively, of the biological signals present in the profiled samples. The factor matrix (F) can be used to cluster samples, while the columns of the weight matrices (Ai) can be used to extract markers by selecting the top-ranked genes, or to identify pathways by applying preranked GSEA (see ref. 10 for further details). A description of the mathematical formulations of the nine jDR approaches is provided in the Methods section.Joint Dimensionality Reduction (jDR) approaches aim to reduce high-dimensional omics data into a lower dimensional space. The rationale behind the use of jDR in biology is that the state of a biological sample is determined by multiple concurrent biological factors, from generic processes  to cell-specific processes. When measuring omics data, we take a snapshot of the state of a biological sample and thus detect a convoluted mixture of various biological signals active in the sample. The goal of jDR is to deconvolute this mixture and expose the different biological signals contributing to the state of the biological sample. We consider (F) Fig. . Here an16, Integrative NMF (intNMF)17, Joint and Individual Variation Explained (JIVE)18, Multiple co-inertia analysis (MCIA)19, Multi-Omics Factor Analysis (MOFA)15, Multi-Study Factor Analysis (MSFA)20, Regularized Generalized Canonical Correlation Analysis (RGCCA)21, matrix-tri-factorization (scikit-fusion)22, and tensorial Independent Component Analysis (tICA)23.Various methods exist to perform jDR , or their co-inertia (MCIA). Finally, JIVE and MSFA consider mixed factors, decomposing the omics data as the sum of two factorizations, one containing a unique factor matrix shared across all omics, and the second having omics-specific factor matrices. Of note, jDR methods considering omics-specific and mixed factors take into account the complementarity of the multiomics data.Seven of the nine jDR approaches are extensions of DR methods previously used for single-omics datasets: intNMF is an extension of non-Negative Matrix Factorization (NMF); tICA is an extension of Independent Component Analysis (ICA); MCIA and JIVE are different extensions of Principal Component Analysis (PCA); and MOFA, MSFA, and iCluster are extensions of Factor Analysis. As a consequence, the different jDR algorithms make different assumptions on the distribution of the factors (Methods). The different jDR approaches also make different assumptions on the across-omics constraints on the factors Fig. . Some alXi matrices, see Fig. Xi have exactly the same samples and features. Nonetheless, the features of multiomics data are frequently different . A possible strategy to have the same features for all omics would be to convert all the features to the same level, e.g., gene symbols. This is sometimes unfeasible: miRNAs cannot be converted to gene symbols, for instance. We applied here another strategy, where we considered for each omics the matrix of correlation-of-correlation between samples (Methods). Both strategies imply a loss of information, which can affect the results of the omics integration. In addition, the number of features ni in the various omics is highly variable, going for instance in TCGA from 800 microRNAs to 5000 CpGs to 20.000 genes. Omics containing more features will have a higher weight in the jDR output. To overcome this issue, in the following, we will first select features based on their variability, and thus make the number of features of the various omics comparable.Most of the jDR approaches can manage different features , but require a match between the samples of the different omics datasets . However, for the sake of comparisons with other algorithms, scikit-fusion is applied here without side information.InterSIM CRAN package24. This package generates three omics datasets with imposed reference clustering. Starting from real omics  extracted from TCGA ovarian cancer datasets, InterSIM generates clusters and associates features to these clusters by shifting their mean values by a fixed amount. InterSIM preserves the covariance matrix between all pairs of omics and thereby maintains realistic inter- and intraomics relationships. Importantly, we selected this approach to avoid making assumptions on the distribution of the data. Indeed, alternative simulation approaches assume specific sample distributions . Assuming a Gaussian distribution, for instance, would favor jDR methods that also make the assumption of Gaussian sample distribution.We first evaluated the jDR approaches on artificial multiomics datasets Fig. . We simuWe simulated multiomics data with five, ten, and fifteen clusters. In addition, each set of clusters is simulated in two versions, either with all clusters of the same size, or with clusters of variable random sizes (Methods).We applied the nine jDR methods, requiring the decomposition of multiomics data into five, ten, and fifteen factors, depending on the simulated datasets. The performances of the nine jDR approaches are then compared based on their clustering of samples. As mentioned before, intNMF and iCluster are intrinsically designed for sample clustering, while the remaining seven algorithms detect factors without providing a direct clustering. Accordingly, we applied directly intNMF and iCluster. For the seven other algorithms, we obtained the clustering of the samples by applying k-means consensus clustering to the factor matrix (Methods).The agreement between the clustering obtained with the various jDR algorithms and the ground-truth clustering is measured with the Jaccard Index (JI) and Adjusted Rand Index (ARI) (Methods). First, we observed that all methods perform reasonably well in the different simulated scenarios . These data are composed of three omics layers: gene expression, DNA methylation, and miRNA expression. The number of samples ranges from 170 for Acute Myeloid Leukemia (AML) to 621 for Breast cancer. Importantly, we do not have ground-truth cancer subtypes to evaluate the performances of the jDR methods. However, in Breast cancer, we compared the jDR clustering results with two subtypings: the ER/PR/HER-2 subtyping based on Estrogen Receptor (ER), Progesterone Receptor (PR) and HER-2 immunohistochemistry markers25, and the Cluster of Cluster Assignment (COCA) integrative classification performed by the TCGA consortium26. Both subtypings cannot be considered as ground-truth for evaluating jDR clustering performances. The ER/PR/HER-2 overlaps with the PAM50 subtyping, which is obtained using only transcriptomics data, and composed of four subtypes: Basal, Her2, Luminal A and Luminal B27. The COCA subtyping is integrative but has been obtained by separately clustering different omics and then performing a consensus of the obtained results. Thereby, it does not take into account the complementarity of the various omics.In the second step, we downloaded TCGA multiomics data for ten different cancer typesWe decomposed the multiomics Breast cancer datasets in four factors, and used the Jaccard Index (JI) and Adjusted Rand Index (ARI) to evaluate the overlap between the clustering obtained from these four factors and the Breast cancer subtypings ER/PR/HER-2 and COCA . We hence compared the performances of the eight jDR algorithms regarding their ability to identify factors predictive of survival, as well as factors associated with clinical annotations. We also evaluated the weight matrices resulting from the jDR methods by assessing their enrichment in known biological pathways and processes.In order to evaluate the methods on the full set of cancer multiomics, we then tested the associations of the factors with survival, clinical annotations and biological annotations Fig. . It is t11. In four other cancer types , all jDR algorithms identified one or two survival-associated factors. Finally, in Melanoma, GBM, and AML, the majority of the jDR methods identified three or four survival-associated factors. In general, MCIA, RGCCA, and JIVE achieved the best performances, finding factors significantly associated with survival in seven out of ten cancer types. These approaches also offered the most significant p-values in the higher number of cancer types. MCIA performed the best for kidney cancer , sarcoma (10−5), and melanoma (10−5); JIVE performed the best in AML (10−3) and liver cancer (10−4); and RGCCA performed the best in Breast cancer (10−3). Furthermore, RGCCA, MCIA, and JIVE showed the most promising results for the cancer types having overall less survival-associated factors  have been designed and applied to bulk multiomics data. It is therefore crucial to evaluate and benchmark the performances of these jDR algorithms for single-cell multiomics integration.Similarly to bulk multiomics analyses, the joint analysis of single-cell multiomics is expected to provide tremendous power to untangle the cellular complexity. jDR approaches are expected to compensate for the strong intrinsic limitations of single-cell multiomics, such as small number of sequencing reads, systematic noise due to the stochasticity of gene expression at single-cell level, or dropouts32. As these cells have been obtained from three different cancer cell lines, we expect that the first two factors of the various jDR approaches would cluster single-cells according to their cancer cell line of origin.To test the jDR approaches on single-cell omics, we fetched scRNA-seq and scATAC-seq, simultaneously measuring gene expression and chromatin accessibility on three cancer cell lines  for a total of 206 cells, and reported in the study of Liu and colleagues33 and LIGER34. Importantly, Seurat does not output factors as the other methods. We thus compared the methods based on their clustering abilities following the same procedure as in the simulation benchmark (Methods). Strikingly, although initially not designed for single-cell data analysis, jDR methods perform equally well or better than Seurat and LIGER . Importantly, this notebook can be easily modified to test the nine jDR algorithms on user-provided datasets. The notebook can also be easily extended to benchmark novel jDR algorithms on our three test cases. Full documentation to achieve these goals is included in the notebook.To foster the reproducibility of all the results and figures presented in this benchmark study, we implemented the corresponding code in a Jupyter notebook available at 14, our benchmark not only focuses on the evaluation of the clustering outputs, but also evaluates the biological, clinical, and survival annotations of the factors and metagenes. Existing comparisons also mainly use simulated data, while we here further consider large datasets of bulk cancer multiomics, as well as single-cell data.We benchmarked in-depth nine jDR algorithms, representative of multiomics integration approaches, in the context of cancer data analysis. In contrast to previous comparisonsWhen performing clustering on simulated multiomics datasets, intNMF, intrinsically designed as a clustering algorithm, displayed the most promising results. MCIA, MOFA, and RGCCA showed the best performance among the set of methods not intrinsically designed for clustering. In the cancer data benchmark, when we evaluated the associations of the factors with survival or clinical annotations, MCIA, JIVE, MOFA, and RGCCA were the most efficient methods. When assessing the associations of the metagenes with biological annotations, MCIA and tICA were the most efficient. Finally, in the last benchmark, when clustering single-cell multiomics data, MSFA and tICA, as well as MCIA and intNMF, outperformed other approaches.As mentioned earlier, intNMF, representative of the Non-negative Matrix Factorization (NMF) approaches, performs well for the clustering tasks, i.e., for detecting substantial patterns of variation across the omics datasets. This is observed for both simulated bulk data clustering and single-cell data clustering. Hence, intNMF should be prioritized by researchers focusing on clustering samples. However, intNMF is not effective when assessing the quality of individual factors and metagenes, as observed in the bulk cancer sub-benchmark. Our results rather suggest that for the study of factor-level information, such as associations with clinical annotation or survival, MCIA, JIVE, MOFA, and RGGCA are the best algorithms to use. When focusing on the underlying biology of the metagenes, tICA and MCIA should be prioritized. Indeed, we showed that these approaches are efficient to detect pathways or processes, but they could also be interesting to identify biomarkers or other molecular mechanisms. Finally, our study highlights the versatility of MCIA. Indeed, MCIA is the method with the most consistent and effective behavior across all the different subbenchmarks. MCIA can thereby be particularly useful for researchers interested in applying jDR with multiple or more open biological questions.14. Finally, multiomics data are frequently profiled from different sets of patients/samples, leading to missing data, and further extensions of the benchmark could take this point into account.In the future, it would be interesting to extend our benchmark to evaluate jDR methods on discrete omics data. Indeed our current benchmark focuses on continuous data , whereas many omics and annotations can be formalized as discrete data . Further extensions of our benchmark could also investigate the impact of different variables on the jDR methods, such as the stability of the methods with respect to variations in the structure of omics data ; or optimal performances according to different combinations of omics data . In addition, to make a fairer comparison, we imposed the same numbers of factors to all of jDR methods, but we could alternatively use the optimal number of factors directly computed by each method, as in the work of Tini and colleagues15 for mouse embryonic stem cells. As a result, future developments should be directed towards methods that can capture the nonlinear signals present in the data. Developers could take advantage of the momix Jupyter notebook using it to compare novel methods with established ones.From a technical perspective, we observed that the methods that seek for omics-specific factors often led to a better performance than the methods designed for finding shared or mixed factors. We hypothesize that jDRs with omics-specific factors could successfully detect not only biological processes shared across multiple omics, but also those processes that are complementary in multiple sources of omics data. In addition, when using algorithms producing omics-specific factors, we have only evaluated the transcriptome-associated omics-specific factors (Methods). However, the outputs of these methods can often contain additional relevant information in other omics-specific factors. As an example, in our benchmark, the jDR methods did not retrieve factors significantly associated with survival in ovarian cancer. However, a significant association is retrieved with one miRNA-specific factor of RGCCA. We thus suggest developers to prioritize omics-specific factors for further methodological developments. Accordingly, the use of co-inertia (as implemented in MCIA) appears more efficient to enforce relationships across omics than the use of correlation (as implemented in RGCCA). In addition, there is room for development of approaches managing missing data, as many of the best-performing approaches can work only on omics profiled from the same samples. This is also true for the consideration of discrete data as, among the methods considered here, only MOFA and scikit-fusion have been previously applied to such data. Finally, most of the considered methods detect only linear signals. MOFA is the only algorithm in our benchmark that can also detect slightly nonlinear signals, as shown in ref. P omics matrices Xi, n × m, with ni features  and m samples. A jDR jointly decomposes the P omics matrices into the product of ni × k omics-specific weight/projection matrices (Ai) and a k × m factor matrix (F) and .Once the matrices 18 is an extension of PCA to multiomics data. JIVE decomposes each omics matrix into three structures: a joint factor matrix (J), a omics-specific factor matrix (A) and a residual noise (E):Ei, Ai and Ui are S is a common score matrix explaining variability across multiple data types.JIVEThe algorithm minimizes r.jive (https://cran.r-project.org/web/packages/r.jive/index.html).JIVE is implemented into the R package 19, is an extension of co-inertia analysis (CIA) to more than two omics datasets. MCIA factorizes each omics into omics-specific factorsXi separately and then maximizing their co-inertia, i.e., the sum of the squared covariance between scores of each factor:omicade4 (https://bioconductor.org/packages/release/bioc/html/omicade4.html).MCIA21 is one of the most widely used generalizations of CCA to multiomics data. Similarly to MCIA, RGCCA factorizes each omics into omics-specific factors:RGCCA (https://cran.r-project.org/web/packages/RGCCA/index.html).RGCCA16 decomposes each omics into the product of a factor matrix that is shared across all omics, and omics-specific weight matrices:Ei and the factor matrix F are normally distributed. Finally, clusters are obtained from the factor matrix by applying K-means. The algorithm is implemented into the CRAN package iCluster (https://rdrr.io/bioc/iClusterPlus/man/iCluster.html).iCluster15 decomposes each omics into the product of a factor matrix and omics-specific weight matrices:F is shared across all omics, the sparsity priors in the in Ai ensure that both omics-specific and shared factors are retrieved. MOFA solves the probabilistic Bayesian model by maximizing the Evidence Lower Bound (ELBO), which is equal to the sum of the model evidence and the negative Kullback–Leibler divergence between the true posterior and the variational distribution. Despite having a factor matrix Fshared among all omics, the sparsity priors in the weights ensure that MOFA will detect both omic-speciifc and shared factors. MOFA is an extension of Factor Analysis to multiomics data, but it is also partially related to iCluster. However, differently from iCluster, MOFA does not assume a normal distribution for the errors but supports combinations of different omics-specific error distributions. The code to run MOFA is available at https://github.com/bioFAM/MOFA. The MOFA package further implements an automatic downstream analysis pipeline for the interpretation of the obtained factor and weight matrices through pathways, top-contributing features or percentage of variance-explained interpretation.MOFAA natural extension of DR methods to multiomic data is based on the use of tensors, i.e., higher-order matrices. Indeed, all the methods designed for single-omics can be naturally extended to multiomics with tensors. However, this requires to work with omics data sharing both the same samples and features. Here, to overcome this limitation we ran the tensorial algorithm on the correlation-of-correlation matrix, i.e., the matrix having on rows and columns the samples that are common to all the omics data and having in position  the correlation of sample i with sample j.23 to represent the tensor-based methods in our benchmark. Considering the multiomics data organized into a tensor X, the equation solved by tICA is:S is a tensor, with the same dimension of X, and composed of We chose tensorial ICA (tICA)23.Thus, tICA searches for independent signals. Since biological processes are generally non-Gaussian and often sparse, the assumption of tICA can improve the deconvolution of complex mixtures and hence better identify biological functions and pathways underlying the multiomics data. Given that multiple tensorial versions of ICA exist, we considered the tensorial fourth-order blind identification (tFOBI), whose implementation in R was reported by Teschendorff et al.20 is a generalization of Factor Analysis (FA), which models the omics matrices Xi as the sum of data-specific and shared factors:Ei has a multivariate normal distribution and the marginal distributions of Fi, Li, and Xi are multivariate normal. MFSA is implemented in R and available at https://github.com/rdevito/MSFA.MSFA22 is based on two steps. First, two groups of matrices are constructed from the multiomics data: relation (R) and constraint (C) matrices. The R matrix encodes relations inferred between features of different omics , while the matrix C describes relations between features of the same omics . The matrix C thus corresponds to the side information considered by scikit-fusion in the factorization. Then, tri-matrix factorization is used to simultaneously factorize the various relation matrices R under constraints C. Given that the R and C matrices are block-matrices, with element Rij containing a relation between the elements of the i-th omics and those of the j-th, the matrix-tri-factorization is applied separately to each block:Gi shared across all the Rip for The data-fusion approach (scikit-fusion)https://github.com/marinkaz/scikit-fusion.Hence, scikit-fusion can naturally combine additional side information in the factorization of the multiomics data, such as protein–protein interactions, Gene Ontology annotations. It is implemented in Python and available at F. As a consequence, all jDR methods with omics-specific and mixed factors contain more information in their factorization than that considered here for sake of comparison.The jDR approaches make different assumptions on the cross-omics constraints of the factors. The various jDR can be thus classified in shared factors, omics-specific factors and mixed factors approach. To use the factor matrices to compare the performances of the various jDR methods, e.g., to cluster the samples based on the factors, we had to select which factor matrix to use for each jDR. Shared factors jDR methods compute a unique factor matrix, which is used in our benchmark. Omics-specific jDR methods compute a factor matrix for each omics dataset. In these cases, we selected the factor matrix associated with transcriptomic data for our benchmark. However, jDRs with omics-specific factors maximize correlation or co-inertia between the various omics-specific factor matrices. The values of the transcriptomic factor matrix are then influenced by the other omics. Finally, for mixed factors jDRs methods, we considered the joint factor matrix InterSIM CRAN package24. InterSIM simulates multiple interrelated data types with realistic intra- and inter-relationships based on the DNA methylation, mRNA gene expression, and protein expression from TCGA ovarian cancer data. We generated 100 simulated datasets, with a number of clusters set by the user. We considered five, ten, and fifteen clusters in this study. The proportion of samples belonging to each subtype is also set by the user, while we considered here two conditions with equally sized clusters and variable random sizes, respectively.The simulated multiomics datasets have been produced by the kmeans function in R). We chose k-means for clustering in agreement with the use of k-means in iCluster and euclidean distance in intNMF for clustering. As k-means clustering is stochastic, we performed clustering 1000 times and computed a consensus consisting in the most frequent associations between samples and clustersTo identify the clusters of samples starting from the jDR factor matrix, we applied k-means clustering to the factor matrix  and Adjusted Rand Index (ARI).JI=JI is a similarity coefficient between two finite sets A and B, defined by the size of the intersection of the sets, divided by the size of their union: n total elements, the Rand Index (RI) is defined as:n elements, a refers to the number of times a pair of elements belongs to a same cluster across the two partitions/clustering results b refers to the number of times a pair of elements belongs to different clusters across the two partitions/clustering results Given two partitions/clustering results The RI is always comprised between 0 and 1, with 1 corresponding to a perfect matching between two partitions/clustering results. However, for random partitions, the expected value of the RI is close to 1. To overcome this drawback, the Adjusted Rand Index (ARI) has been proposed. The ARI is the normalized difference between the RI and its expected value, according to the formula:The clinical annotations selected for benchmark testing are “age of patients”, “days to new tumor”, “gender” and “neo-adjuvant therapy administration”. This set of annotations is obtained after excluding redundant annotations , annotations having missing values for more than half of the samples, and annotations having no biological relevance . Four clinical annotations are available for nine or ten out of ten cancer types, while the others are only present for six or fewer cancer types (with most of them being available only for one or two cancer types).We define the selectivity as:PrerankedGSEA, implemented in the fgsea R package. In preranked GSEA, each metagene is considered as a ranking of genes, and the significance of the association of a biological annotation with the higher or lower part of the ranking is tested. We considered as biological annotations Reactome pathways, Gene Ontology (GO), and cancer Hallmarks, all obtained from MsigDB35.To test if metagenes are enriched in biological annotations, we used 36 an internal clustering evaluation index comparing the distance between intracluster points and interclusters points. It takes its values in  and should be minimum in an optimal clustering. Of note, to compute the C-index we do not need to perform a clustering on the jDR factor matrix, but only to compute the distance between cells known to belong to the same or different cell lines.To evaluate the quality of the 2D data distribution obtained from single-cell multiomics data, we employed the C-index measureInstead, in the computation of the ARI, we had first to define a clustering based on the jDR output, which was performed with k-means, and then compared to the cell lines of origin.Further information on research design is available in the Supplementary InformationPeer Review FileReporting Summary


Single-cell RNA-seq (scRNA-seq) enables the profiling of genome-wide gene expression at the single-cell level and in so doing facilitates insight into and information about cellular heterogeneity within a tissue. This is especially important in cancer, where tumor and tumor microenvironment heterogeneity directly impact development, maintenance, and progression of disease. While publicly available scRNA-seq cancer data sets offer unprecedented opportunity to better understand the mechanisms underlying tumor progression, metastasis, drug resistance, and immune evasion, much of the available information has been underutilized, in part, due to the lack of tools available for aggregating and analysing these data.We present CHARacterizing Tumor Subpopulations (CHARTS), a web application for exploring publicly available scRNA-seq cancer data sets in the NCBI’s Gene Expression Omnibus. More specifically, CHARTS enables the exploration of individual gene expression, cell type, malignancy-status, differentially expressed genes, and gene set enrichment results in subpopulations of cells across tumors and data sets. Along with the web application, we also make available the backend computational pipeline that was used to produce the analyses that are available for exploration in the web application.CHARTS is an easy to use, comprehensive platform for exploring single-cell subpopulations within tumors across the ever-growing collection of public scRNA-seq cancer data sets. CHARTS is freely available at charts.morgridge.org. Over the past three decades, the cancer research community has amassed large quantities of gene expression data from tumors. The premier example of such data was generated by The Cancer Genome Atlas , which gWhile a few web-based tools for analyzing scRNA-seq data are available, they are not designed specifically for cancer research or do not easily enable exploration of existing public data sets. For example, recent tools such as Alona  and GranTo address this gap, we present CHARacterizing Tumor Subpopulations (CHARTS), a web application and associated computational pipeline for analyzing and characterizing publicly available cancer scRNA-seq data sets. As described in detail below, for each tumor in its database, CHARTS identifies clusters and enables exploration via interactive dimension-reduction methods. Derived clusters are annotated with cell types from the Cell Ontology  via Celli.e., non-cell line and non-xenograft samples) that consist of cells from the full tumor microenvironment . For each study, we wrote a custom script for separating cells by their tumor and normalizing the data by estimating the transcripts per million (TPM) for each gene and then computing log(TPM + 1). We note that for droplet-based assays that sequence transcripts only from their end, such as Chromium 10x, an estimate of each gene’s TPM in a cell can be obtained by dividing each gene’s UMI count by the total UMI count in the cell and multiplying by one million. By estimating TPM, we measure gene expression using consistent units across assays. We exclude samples for which their corresponding GEO entry does not include enough information to estimate TPMs. Specifically, we exclude data sets originating from full length assays , such as Smart-seq2 [We used the curated database describing single-cell RNA-seq data sets by Svensson et al.  to identart-seq2 , if theiThese datasets were then processed with an offline computational pipeline. This pipeline implements a number of analyses in order to enable comprehensive characterization and comparison of tumor subpopulations within and between tumors Fig. . All anaA user may construct interactive dimension-reduction scatterplots using two or three-dimensional UMAP  or PHATEClustering is performed using the Leiden community detection algorithm , as implFor each clustering granularity, each cluster is annotated with cell types from the Cell Ontology  via CellEach cluster’s mean gene expression profile is scored for enrichment of gene sets describing molecular processes. Specifically, CHARTS uses GSVA  to scoreEach cell is assigned a malignancy score that describes the likelihood that the cell is malignant. The malignancy scoring approach builds upon the approaches used by Tirosh et al.  and CoutFor each cluster within each tumor, CHARTS uses a Wilcoxon rank-sum test, as implemented in Scanpy, to compute the set of genes differentially expressed in the given cluster versus cells outside the cluster within the given tumor. CHARTS presents the top 50 genes ranked according to their significance. In addition to differential expression analysis, CHARTS also presents boxplots and violin plots for comparing the distribution of gene expression for a user-selected gene between clusters.Two case studies demonstrate how CHARTS can be used, both to examine and generate new hypotheses.Investigators have recently reported a dysfunctional population of CD8 + T cells in lung cancer  and melaWe investigated the expression of MCT4, a prognostic biomarker of glioblastoma aggression , 30. UsiIn this work, we present CHARTS: a comprehensive framework for exploring single-cell subpopulations within tumors and the tumor microenvironment across ever-growing data sets. CHARTS can be used to develop and explore new hypotheses underlying tumor progression, drug resistance, and immune evasion. Specifically, CHARTS exposes the results of an analysis pipeline executed on publicly available data from GEO.There are a number of avenues that would benefit from further investigation. First, we note that CHARTS presents the results of a static analysis pipeline that was executed on diverse single cell datasets. Thus, individual datasets may benefit from tweaking the pipeline’s parameters on an individual basis . Future work will entail devising methods for fine-tuning the parameters used to process each dataset.We note that this analysis pipeline was executed on published gene expression matrices and did not involve processing the raw sequencing reads. Thus, there may be variability between data sets due to the varying methods employed by the data submitters for alignment, gene expression quantification, and quality control. Future work will seek to remove some of this variability by starting with the raw reads from each dataset, rather than the expression matrices, and uniformly quantifying expression and filtering cells across datasets.Project name: CHARTShttps://charts.morgridge.orgProject home page: Operating systems(s): The web application is platform independent. The backend pipeline has been tested only on Linux and MacOS.Programming language: PythonOther requirements: The backend pipeline requires Python 3.6 or higher.License: MITAny restrictions to use by non-academics: NoneAdditional file 1. Supplemental Methods. Description and benchmarking of malignancy score.


Statistical analysis of single cell RNA-sequencing (scRNA-seq) data is hindered by high levels of technical noise and inflated zero counts. One promising approach for addressing these challenges is gene set testing, or pathway analysis, which can mitigate sparsity and noise, and improve interpretation and power, by aggregating expression data to the pathway level. Unfortunately, methods optimized for bulk transcriptomics perform poorly on scRNA-seq data and progress on single cell-specific techniques has been limited. Importantly, no existing methods support cell-level gene set inference. To address this challenge, we developed a new gene set testing method, Variance-adjusted Mahalanobis (VAM), that integrates with the Seurat framework and can accommodate the technical noise, sparsity and large sample sizes characteristic of scRNA-seq data. The VAM method computes cell-specific pathway scores to transform a cell-by-gene matrix into a cell-by-pathway matrix that can be used for both data visualization and statistical enrichment analysis. Because the distribution of these scores under the null of uncorrelated technical noise has an accurate gamma approximation, both population and cell-level inference is supported. As demonstrated using simulated and real scRNA-seq data, the VAM method provides superior classification accuracy at a lower computation cost relative to existing single sample gene set testing approaches. Despite the diversity of cell types and states present in multicellular tissues, high-throughput genome-wide profiling has, until recently, been limited to assays performed on bulk tissue samples. For bulk tissue assays, the measured values reflects the average across a large number of cells and, when significant heterogeneity exists, only approximate the true biological state of the tissue. To address the shortcomings of bulk tissue analysis, researchers have developed a range of techniques for the genome-wide profiling of individual cells ,2 with sAlthough single cell data provides unprecedented insights into the structure and function of complex tissues and cell populations, technical and biological limitations make statistical analysis challenging . Single Although high-dimensional genomic data provides a molecular-level lens on biological systems, the gain in fidelity obtained by testing thousands of genomic variables comes at the price of impaired interpretation, loss of power due to multiple hypothesis correction and poor reproducibility . To helpH0 that none of the genes in the set has an association with the outcome or the H0 that the genes in the set are not more associated with the outcome than genes not in the set) and whether they test each gene set separately (uniset) or jointly evaluate all sets in a collection (multiset). In this paper, we focus on single sample gene set testing methods, i.e. those that compute a cell-specific statistic for each analyzed gene set to transform a cell-by-gene scRNA-seq matrix into a sample-by-pathway matrix. This class of techniques is of particular interest because the cell-level pathway scores can be leveraged for both exploratory data visualization, e.g. shading of cells in a reduced dimensional plot according to inferred pathway activity, as well as the full range of population-level statistical gene set tests, i.e. supervised or unsupervised tests of either the uniset or multiset flavor.Gene set testing methods can be categorized according to whether they support supervised or unsupervised analyzes , whether they provide results for each sample or for an entire population, whether they test a self-contained or competitive null hypothesis -based methods and z-scoring methods. Random walk methods  generatet al.z-scoring methods were all developed for the analysis of bulk gene expression data and were therefore optimized for, and evaluated on, non-sparse data with moderate levels of technical noise. Although AUCell, scSVA and Vision are all targeted at single cell expression data, they make no special provision for the statistical characteristics of single cell data such as sparsity and elevated noise. As we demonstrate through simulation studies later in the manuscript, these methods all have poor classification performance relative to the VAM technique on sparse and noisy data, i.e. they are not able to effectively identify cells whose transcriptomic profile is enriched for specific pathways. In contrast to the other existing single sample methods, PAGODA was designed for single cell analysis and specifically addresses the scRNA-seq features of sparsity and technical noise. In the case of PAGODA, however, the primary focus is an unsupervised and population-level analysis; the generation of sample-level scores is a secondary output which lacks inference support. Relative to the random walk and z-scoring approaches, the class of PCA-based methods, which includes PAGODA, is particularly poor at identifying cells with elevated expression of specific pathways in simulated data sets.GSVA, ssGSEA, PLAGE and the Lee z-scoring methods should have a standard normal distribution when the expression data follows an uncorrelated multivariate normal distribution, this distributional assumption does not hold for sparse scRNA-seq data. Neither the random walk nor the PCA-based method generate scores with a well characterized null distribution. While the lack of a null distribution does not prevent the cell-specific scores generated by these techniques from being used for visualization or as predictors in regression models, it does preclude cell-level inference and the use of scores as dependent variables in parametric models.Although the pathway scores generated by the Given experimental and cost constraints, most bulk gene expression data sets have sample sizes in the hundreds; bulk data sets with more than one thousand samples are rare. Single cell data sets, by contrast, typically profile thousands of cells and data sets containing tens-of-thousands to hundreds-of-thousands of cells are becoming increasingly common. These large sample sizes make computational cost an important factor, especially for techniques that are used in an exploratory and interactive context. Relative to the VAM approach, all of the existing single sample methods have significantly worse computational performance on even small  data sets. For very large scRNA-seq data sets (i.e. 100 000+ cells), the use of methods like GSVA and ssGSEA will be impractical for many users.http://www.dartmouth.edu/∼hrfrost/VAM.To support the gene set analysis of scRNA-seq data and address the limitations of existing gene set testing methods, we developed the VAM technique, which was specifically designed for the analysis of large, noisy and sparse transcriptomic data. In the remainder of the paper, we provide an overview of the VAM method, detail its statistical characteristics, and illustrate the comparative benefits of VAM via both simulation studies and real data analyses. An R package implementing the VAM method and several example vignettes are available at The VAM method generates cell-specific gene set scores from scRNA-seq data using a variation of the classic Mahalanobis multivariate distance measure . VAM takn × p matrix that holds the positive normalized counts for p genes in n cells as measured via scRNA-seq. As detailed in the VAM-Seurat integration Section below, VAM provides direct support for both Seurat  can alsm × p matrix that represents the annotation of p genes to m gene sets as defined by a collection from a repository like the Molecular Signatures Database (MSigDB) .(MSigDB)  . Similar variance decomposition approaches are supported by other scRNA-seq normalization pipelines (e.g. Scater (. Scater ). VAM caCompute modified Mahalanobis distances: Let n × m matrix of squared values of a modified Mahalanobis distance. Each column k of k, is calculated as:g is the size of gene set k, n × g matrix containing the g columns of k, g × g identity matrix, and g genes in set k.where Compute modified Mahalanobis distances on permuted : To capture the distribution of the squared modified Mahalanobis distances under the H0 that the normalized expression values in n × m matrix that holds the squared modified Mahalanobis distances computed on mentclass2pt{minimFit gamma distribution to each column of : A separate gamma distribution is fit using the method of maximum likelihood  to the mentclass2pt{minimUse gamma cumulative distribution function (CDF) to compute cell-specific scores: The cell-specific gene set scores are set to the gamma CDF value for each element of k of k, is calculated as:H0 of uncorrelated technical noise, valid P-values can be generated by subtracting the elements of Statistical properties of VAM Section explores the statistical properties of the elements of P-values generated via where The use of For the scenario represented by , the squmentclass2pt{minim\documentThe standard Mahalanobis distance uses the full sample covariance matrix whereas the modified Mahalanobis distance accounts for just the technical variance of each gene and ignores covariances.The standard Mahalanobis measure computes the distances from the multivariate mean whereas the modified Mahalanobis distance computes distances from the origin.A key feature of the VAM method, and the basis for the ‘variance-adjusted’ portion of the name, is the use of Use of the origin instead of the multivariate mean in  generate2 distribution with g degrees-of-freedom, where g is the size of gene set k. If 2 distribution with g degrees-of-freedom and non-centrality parameter If the values in ition in  can be aector in , the res2 distribution under the H0 of uncorrelated technical noise if the data in H0 of uncorrelated technical noise for sparsity values of both 0.5 and 0.8 , but overestimates the mean and significantly underestimates the variance of the squared distances when the sparsity increases to 0.8 (panel B).The modified squared Mahalanobis measure used by VAM and defined in  can alsomentclass2pt{minimmentclass2pt{minimmentclass2pt{minim2 distribution for realistic sparsity levels, we instead model the null distribution of elements in Type I error control and power Section below.Given the poor fit of a non-central χmentclass2pt{minimz-scoring methods, we used the technique of Lee et al.  was 0.048. To assess power, a random group of 50 genes were given inflated log-normal values for the first 50 cells with the mean value ranging from 0.7 to 1.7 . For each inflated mean value, 10 data sets were simulated and power was computed on the 50 non-null cells for a total of 500 hypothesis tests. The estimated power values ranged from 0.11 for an inflated mean of 0.7 to 0.99 for an inflated mean of 1.7  we envision VAM being used primarily as a means to rank order cells according to pathway activity rather than as a tool for cell-level statistical inference. Figure et al.  on scRNA-seq data sets simulated according to the procedure outlined in the SI Methods. Use of classification accuracy vs. statistical power for the comparative evaluation had two motivations: (i) VAM is the only method in the comparison group that generates valid e et al.  for z-sce et al.  for PCA-For each distinct combination of parameter values, 50 data sets were simulated according to the procedure outlined in the SI Methods and Figure Human PBMC analysis Section for detailed results on the PBMC data set), for the analysis of the 11.8k cell mouse brain scRNA-seq data set using the MSigDB Gene Ontology biological process (C5.BP) pathway collection , and for the analysis of the very large 242k cell Mouse Cell Atlas (MCA)   scRNA-sez-scoring and PCA-based methods could be evaluated due to memory constraints. It should be noted that execution times can be highly variable and, for techniques like GSVA, strongly dependent on how effectively the logic can leverage parallel processing in the underlying architecture so users may encounter a wide range of relative performance values in practice.Although the simulation results reflect the average across a large number of simulated data sets, the real data results represent a single execution on the relevant scRNA-seq data. The VAM method had a much faster average execution on the simulated data set relative to the other methods with the difference particularly dramatic for the two most popular single sample methods, GSVA and ssGSEA. Although the relative performance values dropped on the real scRNA-seq data sets, with notable scaling efficiency for the PCA-based method, the relative efficiency of VAM compared to GSVA and ssGSEA on these real data sets was still large with the absolute difference in execution time substantial given the longer time taken by VAM on large data sets. For the very large MCA data set, only the VAM, As detailed in the SI Methods, we applied the VAM method and comparison techniques to the 10× 2.7k human PBMC data set used in the Seurat Guided Clustering Tutorial (see the SI Methods for more details on this data set and the associated processing pipeline). Figure A important use for the cell-specific scores generated by VAM is the visualization of pathway activity across all cells profiled in a given scRNA-seq data set. Figure As detailed in the SI Methods, we applied the VAM method and comparison techniques to the 10x 11.8k mouse brain scRNA-seq data set. For this example, we used the SCTransform normalization technique instead of log-normalization and explored a much larger pathway collection  collection with 6097 gene sets after size-based filtering). Figure .BP colleWe highlight the results for cluster 4, which appears to represent glial cells including a population of astrocytes, a glial cell subtype. Table GLIAL-CELL-DIFFERENTIATION the top ranked set and both ASTROCYTE-DIFFERENTIATION and GLIAL-CELL-DEVELOPMENT also in the top five list. Figure GLIAL-CELL-FATE-COMMITMENT. None of these other methods identified an astrocyte-related gene set within the top five. Although it is not possible to say with certainty that cluster 4 captures the glial  sub-population in this scRNA-seq data, the top five most significantly up-regulated genes in cluster 4 according to a Wilcoxon test on the SCTransform-corrected counts all have a known association with astrocytes: Dbi  of 0.1 as computed using the Benjamini and Hochberg method  to be performed on the pathway-level rather than on the gene-level. Unfortunately, almost all existing single sample gene set testing methods were designed for the analysis of bulk tissue gene expression data, which is non-sparse and, compared to scRNA-seq data, has a small sample size and limited technical noise.To remedy the lack of effective single sample gene set testing methods for scRNA-seq data, we developed the variance-adjusted Mahalanobis (VAM) method, a novel modification of the standard Mahalanobis multivariate distance measure that generates cell-specific pathways scores which account for the inflated noise and sparsity of scRNA-seq data. Although we expect the scores generated by VAM to be primarily used in contexts that do not assume a specific statistical model, e.g. as predictor variables, the fact that the distribution of the VAM-generated scores has an accurate gamma approximation under the null of uncorrelated technical noise enables inference regarding pathway activity for individual cells. As demonstrated on both simulated and real scRNA-seq data, the VAM method provides superior classification performance at low computational cost relative to existing single sample techniques. The utility of VAM is also aided by direct integration with the popular Seurat framework, which makes it easy to incorporate VAM into existing scRNA-seq analysis pipelines. These features combine to make the VAM method an effective and practical tool for the visualization and statistical analysis of scRNA-seq data.gkaa582_Supplemental_FileClick here for additional data file.


Technological advances have enabled the profiling of multiple molecular layers at single-cell resolution, assaying cells from multiple samples or conditions. Consequently, there is a growing need for computational strategies to analyze data from complex experimental designs that include multiple data modalities and multiple groups of samples. We present Multi-Omics Factor Analysis v2 (MOFA+), a statistical framework for the comprehensive and scalable integration of single-cell multi-modal data. MOFA+ reconstructs a low-dimensional representation of the data using computationally efficient variational inference and supports flexible sparsity constraints, allowing to jointly model variation across multiple sample groups and data modalities. Single-cell methods have provided unprecedented opportunities to assay cellular heterogeneity. This is particularly important for studying complex biological processes, including the immune system, embryonic development, and cancer –4.Following the establishment of the first scalable methods for single-cell RNA sequencing (scRNA-seq), other molecular layers are increasingly receiving attention, including single-cell assays for DNA methylation –9 and chHowever, from a computational perspective, the integration of single-cell assays remains challenging owing to high degrees of missing data, inherent assay noise, and the scale of modern single-cell datasets, which can potentially span millions of cells. Previously, we introduced Multi-Omics Factor Analysis (MOFA) , a statiOther methods that have recently been proposed for integrating different data modalities include Seurat (v3) and LIGER, two strategies based on dimensionality reduction and manifold alignment , 31. BotIn a previous study, we introduced Multi-Omics Factor Analysis (MOFA), a statistical framework for the integrative analysis of multi-omics data from a common set of samples . BuildinWhile the model is applicable to single-cell assays, MOFA and related factor models have critical limitations, including their scalability and the lack of ability to account for side information about the structure between cells. In particular, these models do not provide a principled approach for integrating multiple sample groups and data modalities within the same inference framework.Here, we propose MOFA+, a model extension addressing these challenges by (i) developing a stochastic variational inference framework amenable to GPU computations, enabling the analysis of datasets with potentially millions of cells and (ii) incorporating priors for flexible, structure regularization, thus enabling joint modelling of multiple groups and data modalities.Briefly, the inputs to MOFA+ are multiple datasets where features have been aggregated into non-overlapping sets of modalities  and where cells have been aggregated into non-overlapping sets of groups Fig. a. Data mFor technical details and mathematical derivations, we refer the reader to “Initially, we validated the new features of MOFA+ using simulated data drawn from its generative model. We considered data representing a range of dataset sizes with differing numbers of data modalities and sample groups.First, to assess the utility of stochastic variational inference, we trained models either using conventional (deterministic) variational inference (VI) or using stochastic variational inference (SVI). Across a wide range of training hyperparameters see “”, we obsNext, we assessed the group-wise ARD priors, by assessing to what extent it facilitates the identification of factors with simultaneous differential activity between groups and data modalities. Indeed, when simulating data where factors explain differing amounts of variance across groups and across data modalities, MOFA+ was able to more accurately reconstruct the true factor activity patterns than MOFA v1 or conventional Bayesian Factor Analysis . In this dataset, individual embryos are expected to exhibit transcriptional differences at different stages and even between embryos from the same stage due to variation in the rate of the developmental progression. As a proof of principle, we used MOFA+ to disentangle stage-specific variation from variation that is shared across all stages. For this purpose, we considered the six batches of cells (two replicates for each of the three embryonic stages) as different groups in the MOFA+ model.Ttr and Apoa1 for ExE endoderm, Rhox5 and Bex3 for ExE ectoderm, and Mesp1 and Phlda2 for nascent mesoderm [MOFA+ identified 7 factors that explain at least 1% of variance, which collectively explain between 35 and 55% of the total transcriptional cell-to-cell variance per embryo . Nevertheless, the MOFA+ factors can also be used as input for other methods that infer non-linear manifolds that discriminate cell types Fig. e and enaAs a second use case, we applied MOFA+ to investigate variation in epigenetic signatures between populations of neurons. This use case illustrates how a multi-group and multi-modal structure can be defined from seemingly uni-modal data, which allows for testing specific biological hypotheses.We analyzed 3069 cells isolated from the frontal cortex of young adult mice, where DNA methylation was profiled using single-cell bisulfite sequencing . Recent Here we used MOFA+ to dissect the degree of coordination between mCH and mCG signatures in different regions of the brain. As input data we quantified mCH and mCG levels at gene bodies, promoters and putative enhancer elements “”. Each cNeurod6 and Nrgn [Vsig2, Taar3, and Cort , which is moderately correlated with differences in global mCG levels (R = 0.32) . RNAMOFA+ identified 10 factors that explain at least 1% of variation in gene expression , other cell types are associated with the emergence of local epigenetic patterns that are driven by specific genomic contexts .As single-cell technologies mature, they are applied to generate data sets with increasingly complex experimental designs , 47, 48.In this study, we introduced MOFA+, a generalization of the MOFA framework  that facAlthough MOFA+ represents an important step forward in the analysis of single-cell omics data, it also has limitations. First, it requires multi-modal measurements from the same set of cells. This contrasts with other integrative frameworks such as Seurat  or LIGERIn this study, we introduced MOFA+, a statistical framework aimed at the large-scale datasets with complex experimental designs that include multiple groups of features  and multiple groups of cells . We applied MOFA+ to single-cell data sets of different scales and designs. To facilitate adoption of the method, we deploy MOFA+ as open-source software with multiple tutorials and a web-based analysis workbench, enabling a user-friendly in-depth characterization of multi-modal single-cell data.The input to MOFA+ is a list of matrices, each matrix corresponding to specific group and data modality  and contains the low-dimensional representation of the samples from the gth group. The weight matrix Wm has dimensionality  and contains an association score for each feature with each factor. The noise matrix εgm contains the unexplained variance  for each feature in each group.The factor matrix The model is formulated in a probabilistic Bayesian setting. We introduce prior distributions on all unobserved variables of the model in order to induce specific regularization criteria, as described below in the section “The MOFA+ factors capture the global sources of variability in the data. Mathematically, each factor ordinates cells along a one-dimensional axis centered at zero. Samples with different signs manifest opposite phenotypes along the inferred axis of variation, with higher absolute value indicating a stronger effect. Note that the interpretation of factors is analogous to the interpretation of the principal components in PCA.The weight matrices provide a score for how strong each feature relates to each factor, hence allowing a biological interpretation of the MOFA+ factors. Features with no association with the factor have values close to zero, while genes with strong association with the factor have large absolute values. The sign of the weight indicates the direction of the effect: a positive weight indicates that the feature has higher levels in the cells with positive factor values, and vice versa.The regularization of the weights and the factors is critical to enable MOFA to perform inference with data sets that consists of multiple data modalities and/or groups of samples. In the original version of MOFA, hierarchical priors were applied to the weights to enable inference and interpretable outputs of multi-modal data sets. In MOFA+, we generalized this by introducing a symmetric regularization for both the factors and weights, hence accounting for structure in both the sample space and the feature space  prior to explicitly model differential activity of factors across data modalities and/or across sample groups. The second level consists of a spike-and-slab prior to simultaneously push individual weights and factor values to zero. The latter encourages sparse solutions where factors are  associated with a small number of active features and/or active within small subsets of samples.In MOFA, inference was performed using mean-field variational Bayes (VI) –55. WhilMathematically, the use of SVI is based on redefining the coordinate ascent optimization problem in VI in terms of a  gradient ascent problem that can be described by the following equation:x represents the variables to be inferred and F(x) is the objective function, in this case the Evidence Lower Bound (ELBO). In the stochastic inference framework, a fast approximation of the gradient is calculated using a random subset of the data (a batch). To ensure a smooth convergence, the step size ρ(t) is adjusted at each iteration using the following equation:where τ defines the starting learning rate and k controls its rate of decay (forgetting rate). Hence, the use of SVI comes at the cost of introducing additional hyperparameters: a batch size (as a percentage of the full data set), a starting learning rate and a forgetting rate. A trade-off exists where large batch sizes lead to a more precise estimate of the gradient, but they are more computationally expensive to calculate. While we find the hyperparameters to be relatively robust in simulated data  is sufficient. For other purposes, such as imputation, even small sources of variability are important to be captured and the threshold on variance explained should be lowered to retrieve a large number of factors.In MOFA+, we have implemented Automatic Relevance Determination priors  RNA-seq, we recommend size factor normalization followed by a variance stabilization transformation .We also advise the users to perform a feature selection step by subsetting highly variable features. The aim of this step is to reduce the feature imbalance between different views, simplify the model interpretation and speed up the training procedure.Finally, undesired technical sources of variation that should not be captured by the MOFA+ factors should be regressed out a priori. Typical examples are mitochondrial content or the number of expressed genes in scRNA-seq data. Alternatively, if the technical variation is driven by batch effects and the user is interested in exploring the heterogeneity between batches, we advise the users to use the batch label as grouping criteria.Groups are typically based on the experimental design , but the user can also explore data-driven groups. There is no “right” or “wrong” definition of groups, but some definitions will be more useful than others. Importantly, the aim of the multi-group framework is not to capture differential changes in mean levels between the groups . The aim is to find out which sources of variability are shared between the different groups and which ones are exclusive to a single group. To achieve this, the features are centered per group  before fitting the model.It is important to note that the size of the group can influence the reconstruction of factors. In general, the more samples per group, the more complexity there will exist in the dataset, which can manifest itself in retrieval of a higher number of factors.Data modalities typically correspond to different molecular layers, but the user can also explore data-driven modalities that do not necessarily correspond to different molecular readouts  versus the background set (the weights of features that do not belong to the set G). Resulting P values were adjusted for multiple testing for each factor using the Benjamini–Hochberg procedure [Gene set enrichment analysis was performed using the Reactome gene sets . For everocedure . Signifiscran [Cells were subset to stages E6.5, E7.0, and E7.25. Cells from stage E6.75 were not included in the analysis because they consist of a single biological replicate. Gene expression counts were normalized using scran , and thescran .DNA methylation was quantified over genomic features using a binomial model where the number of successes is the number of reads that support methylation (or accessibility) and the number of trials is the total number of reads. A CpG methylation rate was calculated for each genomic feature and cell using a maximum likelihood approach. The rates were subsequently transformed to M-values  and modeAs input to MOFA+, we filtered genomic features with low coverage (at least 3 CpG measurements or at least 10 CpH measurements) and we selected the intersection of the top 5000 most variable sites across the different genomic and sequence contexts  and the number of trials is the total number of reads. A CpG methylation or GpC accessibility rate for each genomic feature and cell was calculated by maximum likelihood. The rates were subsequently transformed to M-values  and modeAdditional file 1. Supplementary Figures S1-S21.Additional file 2. Supplementary Methods.Additional file 3. Supplementary Table 1, theoretical comparison with previous methods.Additional file 4. Review history.