Dimension reduction and  clustering is usually performed sequentially; however, the low-dimensional embeddings estimated in the dimension-reduction step may not be relevant to the class labels inferred in the clustering step. We therefore developed a computation method, Dimension-Reduction Spatial-Clustering (DR-SC), that can simultaneously perform dimension reduction and  clustering within a unified framework. Joint analysis by DR-SC produces accurate  clustering results and ensures the effective extraction of biologically informative low-dimensional features. DR-SC is applicable to spatial clustering in spatial transcriptomics that characterizes the spatial organization of the tissue by segregating it into multiple tissue structures. Here, DR-SC relies on a latent hidden Markov random field model to encourage the spatial smoothness of the detected spatial cluster boundaries. Underlying DR-SC is an efficient expectation-maximization algorithm based on an iterative conditional mode. As such, DR-SC is scalable to large sample sizes and can optimize the spatial smoothness parameter in a data-driven manner. With comprehensive simulations and real data applications, we show that DR-SC outperforms existing clustering and spatial clustering methods: it extracts more biologically relevant features than conventional dimension reduction methods, improves clustering performance, and offers improved trajectory inference and visualization for downstream trajectory inference analyses. Single-cell RNA sequencing (scRNA-seq) studies encompass a set of widely applied technologies that profile the transcriptome of individual cells on a large scale and can reveal cell subpopulations within a tissue ,2. SpatiIn the analysis of both scRNA-seq and spatial transcriptomics datasets, dimension reduction and  clustering are two key analytical steps that are critical for many downstream analyses such as cell lineage analysis and differential expression analysis. Specifically, due to the curse of dimensionality, dimension-reduction methods are usually applied to the transformation of the original noisy expression matrix in either scRNA-seq or spatial transcriptomics into a low-dimensional representation before performing  clustering analysis ,17–19. Tk-means  clustering analyses are then carried out. Clustering of scRNA-seq data aims to identify cell types and cluster cells into the distinct cell categories. Spatial clustering in spatial transcriptomics aims to use spatial transcriptomic information to cluster tissue locations into multiple spatial clusters, effectively segmenting the entire tissue into multiple tissue structures or domains. Cell-type clustering facilitates our understanding of the cellular composition of tissues with potentially heterogeneous cell types, whereas spatial clustering facilitates the characterization of the tissue structure and is a key step towards understanding the spatial and functional organization of tissue. Common clustering methods for scRNAseq analysis include k-means  and the k-means . Common k-means , the hidk-means , BayesSpk-means , SC-MEB k-means , and Spak-means , all of k-means , spatialk-means , and difk-means .The majority of existing methods for dimension reduction and  clustering have been used in a tandem analysis by first performing dimension reduction on expression matrix followed by  clustering analysis of the estimated low-dimensional embeddings , as showhttps://CRAN.R-project.org/package=DR.SC), with functions implemented for standalone analysis and Seurat based T for each spot, yi ∈ {1, ⋅⋅⋅, K}, and q-dimensional embeddings, zi’s, are unavailable. Without loss of generality, we assume that xi is centered and DR-SC models the centered log-normalized expression vector xi, with its latent low-dimensional feature, zi, and class label, yi, as1, ⋅⋅⋅, λp) is a diagonal matrix for residual variance, p-dimensional expression vector into q-dimensional embeddings, and kth class, respectively. Equation (xi) in p genes with a low-dimensional feature (zi) via a probabilistic PCA model while Equation  are available, we assume each latent class label, yi, is interconnected with the class labels of its neighborhoods via a Markov random field. To promote spatial smoothness within spot neighborhoods, we assume that the hidden Markov random field y = T takes the following Potts model (C(β) is a normalization constant that does not have a closed form, Ni is the neighborhood of spot i, and β is the smoothing parameter that controls the similarity among the neighboring labels, in other words, the degree of spatial smoothness. When this smoothing parameter β goes to zero, the spatial-clustering step in DR-SC, Equations \docuentclass1pt{minima\docuentczi’s, are estimated under the supervision of the estimated latent labels for each spot  clustering Figure . By combt Figure . Thus, tt Figure  without t Figure .stats; (2) WPCA  factorial k-means (FKM)  tSNE; (5) UMAP, in which tSNE and UMAP were implemented in the R package scater; (6) ZIFA implemented in the Python module ZIFA; (7) ZINB-WaVE implemented in the R package zinbwave; and (8) scVI implemented in the Python module scvi. As the last three methods, ZIFA, ZINB-WaVE, and scVI, can be applied to only raw count data, we compared their performance with that of DR-SC in Simulation 2 with the count matrix for expression levels and real datasets.We conducted comprehensive simulations and real data analysis by comparing the dimension reduction and clustering performance of DR-SC with those of existing methods. In detail, we considered the following eight dimension-reduction methods to compare the dimension-reduction performance: (1) PCA implemented in the R package (2) WPCA  implemenns (FKM)  implemenBayesSpace; (2) Giotto  SC-MEB  SpaGCN  Louvain  Leiden  GMM implemented in the R package mclust; (8) k-means implemented in the R package stats; (9) FKM  subspace clustering based on arbitrarily oriented projected cluster generation (ORCLUS)  BayesSpace  implemen) Giotto  implemen) SC-MEB  implemen) SpaGCN  implemen Louvain  implemen) Leiden  implemen (9) FKM  implemen(ORCLUS)  implemenyi, for each i = 1, ⋅⋅⋅, n in a rectangular 70 × 70 lattice from a K-state (K=7) Potts model with smoothing parameter β = 0 or 1 using function sampler.mrf in R package GiRaF. Then we generated latent low-dimensional features, zi, from the conditional Gaussian, such that zi|yi = k ∼ , where zi ∈ Rq with q = 10 and structures for μk and Σk are shown in j), j = 1, …, p. In the case of homoscedasticity, λj = 9, ∀j, while in the case of heteroscedasticity, We performed two sets of simulations as follows. Simulation 1 involved log-normalized gene expression data. In this simulation, we generated non-spatial/spatial log-normalized gene expressions. In detail, we generated the class label, yi, loading matrix, $w$, and latent features, zi’s, was the same as in Simulation 1, except that μk had a different value , j = 1, ⋅⋅⋅, p. To ensure a proper signal, we set λj = 1, ∀j, in the case of homoscedasticity and j and cell i for non-spatial settings and the raw gene expression j and spot i and spatial coordinates si for spot i.Simulation 2 involved raw gene expression data. In this simulation, we generated non-spatial/spatial raw gene expressions. The method used to generate the class label, alue see . The difhttps://github.com/LieberInstitute/spatialLIBD. This dataset is a collection of data from 12 human postmortem DLPFC tissue sections from three independent neurotypical adult donors and the raw data for each sample includes 33,538 genes. We first selected genes with spatial variation using SPARK  from ng SPARK  without 000 SVGs , then we000 SVGs . We furtprofiler  based onhttps://singlecell.broadinstitute.org/single_cell/data/public/SCP815/sensitive-spatial-genome-wide-expression-profiling-at-cellular-resolution#study-summary. We first selected the top 2,000 genes with spatial variation using SPARK  platform. This dataset contained 23,194 cells, 351 genes, and two-dimensional spatial coordinates. Cell types were annotated based on their nearest neighbors on an existing scRNA-seq atlas (Gastrulation atlas)  . Taking n atlas)  from eitdatabase . To visuhttps://zenodo.org/record/1443566#.XNV25Y5KhaR  values. We used DR-SC and FKM to simultaneously estimate the embeddings and class labels, while other methods were applied for tandem analyses. Using the estimated class labels for the 11 clusters from DR-SC, we performed DGE analysis using the R package BPSC  dataset from NCBI chnology  from twochnology . Followitte.html , we ignoage BPSC  on humanage BPSC . For eacary Text .We evaluated the performance of DR-SC from four aspects: feature extraction, clustering performance, selection of the number of clusters, and computational efficiency. Here, we briefly present the evaluation metrics for the feature extraction and clustering performance. For details on the other two aspects, please refer to the Supplementary Text.xi, and cell type label, yi, given the estimated latent features. Canonical correlation measures the similarity between two sets of random variables. Thus, a larger canonical correlation coefficient value suggests a better estimation of zi. For optimal performance, we aimed to obtain the estimated features, In the simulations, we used two metrics to assess the performance of the feature extraction including both the canonical correlations between the estimated features and the underlying true ones and the conditional correlation between gene expression, x and yx share. If x and y do not share information and are independent, then MI = 0. At the other extreme, if y = x, then MI = H(x), where H(x) is the marginal entropy of x. This indicates that MI does not take values between zero and one. Thus, some normalized versions have been proposed and we used one of these versions (NMI).To compare clustering performance, we evaluated both ARI  and normHere, we provide a brief overview of DR-SC, and further details are available in the Supplementary Text. The proposed method involves simultaneous dimension reduction and  clustering built on a hierarchical model with two layers, as shown in Figure In later sections, we show the improved clustering performance with spatial transcriptomics datasets from different platforms. In the Supplementary Text, we show how DR-SC improved the clustering performance for single-cell datasets. Aside from improving the  clustering performance, DR-SC estimates low-dimensional embeddings that can also be used in different types of downstream analyses Figure . First, j . Two simulation settings were considered. In Simulation 1, log-normalized and centered gene expression data were generated from Equation  and spatial β = 1) patterns and with both homogeneous and heterogeneous residual variance λ patternsEquation . In SimuEquation . In all k-means in the clustering step. Among these, SpaGCN software used the log-normalized expression matrix as the input and its internally embedded PCA algorithm to obtain PCs, and could only be applied to spatial clustering. The second group was a joint analysis with ORCLUS , Kyoto Encyclopedia of Genes and Genomes (KEGG), and Human Protein Atlas (HPA) were enriched with adjusted m spatial resolution , while the SpaGCN package took the normalized expression matrix as input and used its internally embedded PCA algorithm to obtain the PCs. In a joint method, DR-SC used the normalized expression matrix of SVGs as input. A spatial heatmap of the cluster assignments across the four methods is provided in Figure Using the cluster labels estimated in DR-SC, we performed DGE analysis to identify the marker genes for each cluster. The heatmap of differentially expressed genes for each cell type Figure  showed gP-values of less than 0.05. A bubble plot for this functional enrichment , while for SpaGCN and DR-SC, the normalized expression matrix was the input. SC-MEB-O and DR-SC shared similar spatial patterns, whereas BayesSpace-O assigned a large proportion of spots (78%) to a single cluster Figure . We alsoBased on the cluster labels estimated in DR-SC, we performed DGE analysis to identify marker genes for each cluster. A heatmap of the findings showed good separation of the differentially expressed genes across different cell types Figure . By checTtr had low expression levels at the early stage before substantially increasing later. In contrast, the expression levels of Nfib, Sox11, Nnat and Map1b changed from low to high, then back to low. Steele-Perkins et al.  as guides to check how these trajectories were distributed over brain segments. The cells and trajectories from each cluster overwhelmingly occupied different brain regions  family of transcription factors. A recent study  clustering sequentially, DR-SC unifies low-dimensional feature extraction with  clustering in the same, joint modeling framework, and provides an improved estimation for cell-type-relevant low-dimensional embeddings and enhanced clustering performance for both scRNA-seq and spatial transcriptomic data from different platforms. With simulation studies and benchmark dataset analyses, we demonstrated that DR-SC can improve clustering performance while effectively estimating low-dimensional embeddings.DR-SC relies on a hidden Markov random field model with a smoothing parameter to perform spatial clustering. The probabilistic framework of DR-SC allows us to adaptively update the spatial smoothing parameter that promotes similar cluster assignments for neighboring tissue locations in a data-driven manner. When the smoothness parameter is set to zero, DR-SC performs clustering for scRNA-seq data without spatial information. We developed an efficient EM algorithm based on iterative conditional mode and expectation-maximization (ICM-EM), making DR-SC computationally efficient and scalable to large sample sizes.Camk2b and Malat1. Küry et al. (Camk2b was important for learning and synaptic plasticity in mice, while Zhang et al. (Malat1 gene transcription in mice. Third, we applied DR-SC to analyze a seqFISH dataset and showcased its ability to infer cell lineages based on the reduced-dimensionality space estimated by DR-SC. Among the identified genes with interesting dynamic patterns over pseudotime, transcription factors Foxa1 and Foxa2 are crucial in maintaining key cellular and functional features of dopaminergic neurons in the brain (Fgfr2 and Fgfr3 play important roles during early neural development (In-depth analyses using scRNA-seq and spatial transcriptomic data from different platforms showed that the estimated clusters and embeddings from DR-SC effectively facilitated the downstream analysis. First, we applied DR-SC to a 10x Visium dataset for DLPFC to demonstrate the improved spatial clustering performance of DR-SC and further carried out conditional SVA to identify genes with pure spatial variations but not cell-type differences. The majority of genes identified in SVA without adjustment for cell-type-relevant covariates simply reflected cell-type differences. Functional enrichment analysis showed that the genes identified in SVA with adjustment for covariates were enriched in pathways related to the DLPFC tissue. For example, the most significant KEGG pathways including Huntington’s disease and Alzheimer’s disease were identified in all 12 LIBD samples. Second, we applied DR-SC to analyze two Slide-seqV2 datasets and found it outperformed both existing dimension reduction methods in terms of visualization and existing spatial clustering methods in terms of separation, as well as its usefulness in cell trajectory inference. In the mouse olfactory bulb data, we identified some genes with interesting dynamic expression patterns, such as y et al.  reportedg et al.  reportedhe brain  while Fgelopment ,77. FourThere are several potential extensions that can be applied to DR-SC. First, in the current study, we considered single transcriptional profiles. The framework of DR-SC could be naturally extended to perform joint-clustering analyses of multiple samples by properly removing their batch effects. Second, the fast-evolving technology of single-cell omics provides opportunities to integrate omics profiles from different modalities for the same individuals. Extending DR-SC by integrating multiple different omics techniques, such as through the canonical correlation analysis framework  which ishttps://github.com/feiyoung/DR-SC.Analysis. The source code is released under the GNU general public license. The 16 benchmark datasets with linear trajectory information are available at https://zenodo.org/record/1443566#.XNV25Y5KhaR. The cord blood mononuclear cells datasets are available at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE100866 via the accession number GSE100866. The human dorsolateral prefrontal cortex datasets on the 10x Visium platform are accessible at https://github.com/LieberInstitute/spatialLIBD. The mouse olfactory bulb data and mouse E15 neocortex data on the Slide-seqV2 platform are available at https://singlecell.broadinstitute.org/single_cell/data/public/SCP815/sensitive-spatial-genome-wide-expression-profiling-at-cellular-resolution#study-summary. The mouse embryo dataset on the seqFISH platform is accessible at https://content.cruk.cam.ac.uk/jmlab/SpatialMouseAtlas2020/.All codes in this paper are publicly available at gkac219_Supplemental_FilesClick here for additional data file.


Accurate identification and effective removal of unwanted variation is essential to derive meaningful biological results from RNA sequencing (RNA-seq) data, especially when the data come from large and complex studies. Using RNA-seq data from The Cancer Genome Atlas (TCGA), we examined several sources of unwanted variation and demonstrate here how these can significantly compromise various downstream analyses, including cancer subtype identification, association between gene expression and survival outcomes and gene co-expression analysis. We propose a strategy, called pseudo-replicates of pseudo-samples (PRPS), for deploying our recently developed normalization method, called removing unwanted variation III (RUV-III), to remove the variation caused by library size, tumor purity and batch effects in TCGA RNA-seq data. We illustrate the value of our approach by comparing it to the standard TCGA normalizations on several TCGA RNA-seq datasets. RUV-III with PRPS can be used to integrate and normalize other large transcriptomic datasets coming from multiple laboratories or platforms. Cancer RNA-seq data are normalized with respect to library size, tumor purity and batch effects. In cancer RNA-seq data, within-sample normalization should adjust for gene length, GC content and cellular compositions, whereas between-sample normalization should remove the impact of library size, tumor purity and batch effects on the data. Efficient removal of such variation from RNA-seq data is still a challenge. This variation can introduce artifactual or obscure true biological signals in the data and, consequently, lead to false or missed discoveries, resulting in misleading biological conclusions8.An essential step of RNA sequencing (RNA-seq) data analysis is normalization, whereby different sources of unwanted variation are removed to make gene expression measurements comparable within and between samples10. These normalizations simply divide all gene counts in each sample by a single scale factor. The implicit assumption underlying such methods is that all the gene-level counts are proportional to the scale factors and that it should be adequate to adjust them for library size in this way across samples. A current challenge for RNA-seq normalizations arises when the counts for a reasonable proportion of genes cannot be properly adjusted for library size by the use of a single scale factor, regardless of how it is computed. The bias between gene-level counts and library size has been discussed in single-cell RNA sequencing data12; however, this has not been recognized in RNA-seq data.Most RNA-seq normalizations adjust for library size variation using global scaling factors calculated based on either total counts or other statistical features of the raw count data, such as their upper quartiles16. Tumor purity could be considered as a source of unwanted variation in studies whose aims are restricted to tumor-specific expression. Variation in tumor purity can affect comparisons of a gene’s expression within and between samples, which can compromise downstream analyses in cancer RNA-seq studies19. Current RNA-seq normalizations and batch correction methods are unable to remove this kind of variation from the data. Adjusting counts for tumor purity variation using regression models risks removing biological signal if that signal is confounded with purity.Tumor purity—that is, the proportion of cancer cells in solid tumor tissues—is another major source of variation in cancer RNA-seq data. This variation has been viewed as an intrinsic characteristic of tumor samples and has been linked to several clinical outcomes in patients with various cancer types22. An implicit assumption underlying such methods is that the biological populations are evenly distributed within each batch—that is, that there is no association between batch and biological condition. However, if there is such an association (due to confounding), then correcting gene expression counts for batch effects using these methods risks removing biological signal along with the batch effects. Furthermore, batch effects usually influence subsets of genes in different ways8; sample-wise normalization, including normalizations that rely on global scaling factors, generally fail to remove this variation from the data.Batch effects are obvious sources of unwanted variation in large RNA-seq studies, where samples are necessarily processed across a range of conditions—for example, chemistry, protocol and facility. Most batch correction methods are based on linear regression. For individual gene expression, they fit a linear model with blocking terms for batch. Then, the coefficient for each blocking term is set to zero, and the corrected expression values are computed from the residuals8. The RUV-III method is a linear model through which the presence and impact of known and unknown unwanted factors can be inferred via technical replicates and negative control genes. However, RUV-III has two limitations. First, it is not designed to be used effectively in situations where technical replicates are not available or well-distributed across the sources of unwanted variation. Second, because a sample’s tumor purity will be essentially the same across all of its technical replicates, the original RUV-III is unable to estimate and remove this kind of variation using standard technical replicates.We previously developed a normalization method, called removing unwanted variation III (RUV-III), for gene expression studies with technical replicatesHere we propose an approach, called pseudo-replicates of pseudo-samples (PRPS), to deploy RUV-III to efficiently remove the impact of library size, tumor purity and batch effects from RNA-seq data. The PRPS approach overcomes the limitations of RUV-III in situations where suitable technical replicates are not available or where variation due to tumor purity is to be removed from cancer RNA-seq data. To use RUV-III with PRPS, we first need to identify the sources of unwanted variation and major expression-based biological populations in the data. We then create pseudo-samples, which are in silico samples derived from small groups of samples that are roughly homogeneous with respect to unwanted variation and biology. Two or more pseudo-samples with the same biology will be regarded as a pseudo-replicate set. The gene expression differences between such pseudo-samples will largely be unwanted variation. RUV-III makes use of such differences, together with negative control genes, to estimate and remove unwanted variation from the data.We make use of three RNA-seq datasets from The Cancer Genome Atlas (TCGA) studies to show that RUV-III with PRPS can effectively remove library size, tumor purity and batch effects and lead to meaningful biological results that are not compromised by this kind of variation. We will demonstrate that RUV-III with PRPS can be used to normalize multiple RNA-seq studies. We also present comprehensive strategies for revealing unwanted variation in large-scale RNA-seq studies, such as those of the TCGA project.The TCGA Research Network generated RNA-seq data from ~11,000 tumor and normal sample tissues obtained from 33 cancer types. To understand some potential sources of unwanted variation, fresh-frozen tissue samples were collected from tissue source sites (TSSs), allocated to 96-well sequencing plates  and processed at various times   reveal t23 for the raw count and TCGA normalized datasets datasets . In the Taken together, our results show that all the TCGA RNA-seq datasets, both raw and normalized, are greatly affected by the three major sources of unwanted variation. Next, we used the READ, COAD and BRCA RNA-seq datasets to illustrate the effects of unwanted variation on certain downstream analyses and show the performance and effectiveness of RUV-III with PRPS for these datasets. The details of each study are provided separately below.24—were identified using the R package CMScaller25  and BCLAF1 (Bcl-2-associated transcription factor 1) genes are ρ = 0.8 and ρ = 0.7 in the TCGA FPKM and FPKM.UQ normalized data, respectively. The role of the TMF1 gene has not been characterized in COAD, although the BCLAF1 gene shows a pro-tumorigenic role in this cancer type26. One might suggest that the TMF1 gene expression may have a role in tumorigenesis in colon cancer due to its high correlation with the BCLAF1 gene expression. However, we see no such correlation in the RUV-III normalized data, which is consistent with the correlation obtained from an independent platform, namely the TCGA READ microarray data  and EIF4H (eukaryotic translation initiation factor 4H) genes is ρ = −0.05, whereas they exhibit a high correlation within each key time interval in the TCGA normalized data  between all possible gene–gene Spearman correlations ρ using the TCGA READ microarray data and the FPKM.UQ and RUV-III normalized data.Figure RAB18 (Ras-related in brain 18) and FBX14 (F-box and leucine-rich repeat protein 14) genes are highly associated with overall survival outcome of patients in the data  in cancer, which indicates that FBXL14 could function as an EMT inhibitor to suppress metastasis in human cancers30. Other examples are PTPN14 and CSGALNACT2, whose associations with survival have been previously shown in colorectal cancer  gene is an example from this group  gene represents the behavior of genes in this group , whose expression levels are not associated with library size in the raw count data. Then, the FPKM and FPKM.UQ normalizations introduce the library size variation to the expression levels of genes in this group  whose expression levels relate to library size in a manner opposite to that motivating the use of global scaling factors. Applying scaling factors to such genes exacerbates, rather than removes, variation associated with library size  and ETS1 are both highly correlated with tumor purity. The ZEB2 gene is a one of the regulators of the EMT process that induces invasion of cancer cells34. ETS1 is member of a large family of transcription factors characterized by their ETS DNA‐binding domain. The gene appears to have dichotomous roles as an oncogene and a tumor suppressor gene in different cancer types36. The high correlation of ETS1 with ZEB2 in the TCGA BRCA RNA-seq data may confirm its oncogene role, but this is most likely a consequence of their correlations with tumor purity. The RUV-III normalized data and the breast cancer laser microdissection microarray data37 showed that the expression levels of these two genes are uncorrelated  gene, whose expression levels are associated with survival in several cancer types, including breast cancer42. However, this association was only evident in the present data after removing variation in tumor purity. We found many more examples of such genes using the RUV-III normalized data.Another example is the ESRRA  and MAP3K2 (mitogen-activated protein kinase kinase kinase 2) genes are positively correlated in this dataset; however, this correlation seems to be a consequence of the unwanted variation in the data  and CNOT1 (CCR4-NOT transcription complex subunit 1) genes is ρ = 0.1, and the average of the correlations of these genes within each of groups 1–4 of the unknown source of unwanted variation is ρ = 0.4 45. PCA plots, vector correlation between the first ten PCs cumulatively and the PAM50 subtypes, silhouette coefficients and ARI  gene expression data and found no normal-like subtype in the dataset. The results confirm previous studies that show that the normal-like subtype is due to the low tumor purity of samplesFOXA1 (forkhead box A1) is highly associated with tumor purity in the Her2, luminal A and luminal B subtypes in the FPKM.UQ normalized data . To simulate such situations, we used one of the CMS subtypes, CMS4, to create PRPS for RUV-III normalization of the TCGA READ RNA-seq data. Note that this subtype is not present across all the plates. Our results clearly show that RUV-III-P normalization led to very satisfactory normalization by removing the large library size differences and plate effects and also preserving the CMS clusters , batch effects and other technical variation in data.We proposed an approach, called PRPS, to deploy RUV-III for normalization of RNA-seq in situations where suitable technical replicates are not available. Our PRPS approach requires the presence of at least a homogenous biological population across sources of unwanted variation. Then, we create pseudo-samples by averaging gene expression of a group of samples that are roughly homogeneous regarding the unwanted variation and biology. The gene expression differences between pseudo-samples are mainly unwanted variation. These samples will be used by RUV-III as a set of pseudo-replicates to estimate one aspect of unwanted variation in the data.We made use of three TCGA RNA-seq studies to compare the performance of RUV-III with PRPS with the state-of-art normalizations proposed for RNA-seq data. RUV-III is not limited to TCGA data normalization, and we have also shown that the method can be used to normalize RNA-seq data, when the data come from multiple studies. Our comparisons are based on statistical summaries, biological positive and negative controls and concordance with the corresponding TCGA or independent microarray studies.We began by carefully identifying different known sources of unwanted variation in all the TCGA RNA-seq raw count, FPKM and FPKM.UQ normalized datasets. We illustrated that library size, tumor purity and plate or time effects are major sources of unwanted variation in these studies, and we showed how they can influence downstream analyses. These unwanted variations are likely to affect other downstream analyses not investigated in this study.In the TCGA READ RNA-seq study, noticeable library size differences between samples remained in the FPKM and FPKM.UQ normalized data due to the presence of genes whose raw counts showed weak or negative association with library size. In such situations, normalizations that rely on a global scale factor can introduce, rather than remove, library size variation. We found this issue in several TCGA cancer studies, even those that used a single plate for profiling. We took advantage of the gene-wise normalization ability of RUV-III to remove library size effects only from genes that are affected by this variation. Our results showed that RUV-III with PRPS effectively removed the library size effects from the TCGA READ RNA-seq data and led to better downstream analyses of gene–gene co-expression and association of gene expression with survival. Furthermore, the results showed that the variation due to tumor purity was highly similar for the TCGA and the RUV-III normalized datasets, as we did not attempt to remove the variation. This demonstrates the ability of RUV-III with PRPS approach to remove just the sources of unwanted variation that users aim to remove.We found that the TCGA COAD RNA-seq data are affected by the same sources of unwanted variation that were identified in the corresponding READ RNA-seq data, although their effects were less severe in the COAD data compared to the READ data. The first two PCs of the FPKM and FPKM.UQ dataset did not show that the library size effects have not been properly removed. This highlights the importance of gene-level examinations to assess the performance of normalizations.In the TCGA BRCA RNA-seq data, we designed our PRPS to remove variation in tumor purity as well as other sources of unwanted variation, including library size, flow cell chemistry and plate effects. We used LCM gene expression data to demonstrate the effects of tumor purity on the PAM50 subtype identification and gene co-expression analysis in the FPKM.UQ normalized data. We identified that the use of two flow cell chemistries introduced unwanted variation into the TCGA BRCA RNA-seq data. This introduced correlations between genes that were not truly associated and obscured the correlation between genes that were truly associated. We used the TCGA microarray data as an orthogonal platform to compare the gene expression patterns and their correlations in differently normalized datasets. The results of this comparison showed that the agreement between the RUV-III normalized data and the microarray data was much better than that found with the two TCGA normalized datasets.The performance of RUV-III with the PRPS approach relies on the identification of major gene-expression-based biological populations in the data. Our results clearly showed that a rough identification of such populations using inadequate normalizations is satisfactory for creating the PRPS. In the three TCGA RNA-seq studies used in this study, the major biological populations were identified using TCGA normalized data that included sources of unwanted variation. However, we observed an equally good performance of RUV-III with PRPS using the biological populations identified in the RUV-III normalized data as we did using TCGA normalized data. Furthermore, we demonstrated that RUV-III is reasonably robust to poorly chosen PRPS.Note that we could have created the PRPS using adjacent normal tissue, which is more homogeneous than cancer tissue, had these samples been more uniformly distributed across batches. However, we were not able to use such samples to create PRPS across the two flow cell chemistries for the BRCA study, as all 94 adjacent normal breast tissue samples were profiled using just one of the two chemistries. We found the same problem for the TCGA READ and COAD RNA-seq data. In these datasets, all normal adjacent tissues were profiled within a key time interval. It should be also noted that tumor purity is essentially the same across any set of technical replicates; thus, these sample are not useful to estimate and remove variation in tumor purity by the RUV-III normalization.In large-scale genomics studies such as TCGA, samples are inevitably profiled using different reagents and platforms at different times, which can introduce unwanted variation into the data. As such, we strongly recommend including technical replicates across any possible source of unwanted variation. These samples can be used by any technical-based normalizations and considered as positive control to assess of any normalizations. We also recommend distributing the biology of interests across batches, to the extent that this is possible, as this will assist the use of RUV-III with PRPS. However, as it is difficult to predict all sources of unwanted variation and appropriately design technical replicates across them, RUV-III with PRPS provides a tool to remove this unwanted variation from large-scale cancer and genomics studies.52. The TCGA RNA-seq data are publicly available in three formats: raw counts, FPKM and FPKM.UQ. All these formats for individual cancer types  were downloaded using the R/Bioconductor package TCGAbiolinks (version 2.16.1)53. The TCGA normalized microarray gene expression data were downloaded from the Broad GDAC Firehose repository (https://gdac.broadinstitute.org), data version 2016/01/28. TSSs and batches of sequencing plates were extracted from individual TCGA patient barcodes (https://docs.gdc.cancer.gov/Encyclopedia/pages/TCGA_Barcode/), and sample processing times were downloaded from the MD Anderson Cancer Center TCGA Batch Effects website: https://bioinformatics.mdanderson.org/public-software/tcga-batch-effects. Pathological features of patients with cancer were downloaded from the Broad GDAC Firehose repository (https://gdac.broadinstitute.org). The details of processing the TCGA BRCA RNA-seq samples using two flow cell chemistries were received by personal communication from Dr. Katherine Hoadley. The TCGA survival data reported by Liu et al.54 were used in this paper. The breast cancer LCM microarray dataset was downloaded using the GEOquery R/Bioconductor package (version 2.62.2) from the National Center of Biotechnology Information (NCBI) Gene Expression Omnibus (GSE78958 (ref. 37)). The two non-TCGA RNA-seq datasets were downloaded from the NCBI Gene Expression Omnibus with accession numbers GSE96058 and GSE81538 (refs. 50). The consensus measurement of purity estimation (CPE) was downloaded from the Aran et al. study17.The TCGA consortium aligned RNA-seq reads to the hg38 reference genome using STAR aligner and quantified the results at gene level using HTSeq and GENCODE version 22 gene annotation2 library sizes of three median absolute deviations lower than the median of all log2 library sizes were excluded from the data.We applied the following filtering steps to the individual TCGA RNA-seq datasets. Plates with fewer than three samples were removed. Samples with logThe R/Bioconductor package biomaRt (version 2.48.3) was used to annotate genes. All pseudo-genes and immunoglobulin genes were excluded. For the pan-cancer analyses, we retained genes with at least 15 raw read counts in at least 20% of samples. We considered numbers of samples in the biological subpopulations and sources of unwanted variation when removing lowly expressed genes in the TCGA READ, COAD and BRCA data. To do so, we kept genes that have at least 15 counts in the smallest biological subpopulations within each of the key time intervals in the datasets.55 and the R/Bioconductor package singscore version (1.12.0)32. The stromal–immune scores were transformed to 1–stromal–immune scores for downstream analyses. These measurements are called tumor purity scores in this study. The tumor purity scores showed high positive correlation  with the ESTIMATE measurements from the Aran et al. study17  of interest indexed by sample rather than assay. There are no constraints on p; indeed, X could be the m1 × m1 identity matrix. Our goal here is to remove unwanted variation, not to estimate regression parameters.Before we can describe the linear model underlying RUV-III, we need to introduce the m × n; the matrices X and M have just been defined; µ is the 1 × n row of gene means; β is p × n; the matrix W whose columns capture the unwanted variation is m × k; and α is k × n. 1 = 1m is the m × 1 column vector of 1s. Here, m = number of assays, n = number of genes and p is the dimension of the wanted variation X and k that of the unwanted variation W. Assume that The linear model we use is:nc of negative control genes whose m × nc submatrix Yc satisfies Ycβc = 0—that is, that there is no true association between these genes and the biology of interest.Also, we suppose that we have a subset of yij of Y by the simple average of the entries yi′j over all i′ for which i′ such that i′ and i label replicate assays of the same unique sample (or pseudo-sample) labeled h.The projection RM = I − PM for the corresponding residual projector. This is our source of information on the unwanted variation that we will remove. If the replication is technical at some level, then RMY mainly contains information about unwanted variation in the system after the technical replicates were created. Depending on the study details, technical replicates could be created immediately before the assay was run, in parallel with or immediately after sample was collected or somewhere in between. The earlier the creation of technical replicates, the more unwanted variation will be captured in their differences. The use of pseudo-replicates of suitable pseudo-samples enables us to start to deal with pre-technical unwanted variation.Write U is an m × m orthogonal matrix and D is an m × n diagonal matrix with entries ordered from largest to smallest eigenvalue. Let P1 be the orthogonal projection onto 1m.Write the spectral decomposition of k, 1 ≤ k ≤ m−m1,I.Uk)( is the first k columns of UDefine II.W by regressing the centered negative controls (I − P1)Yc on Estimate Finally, we:III.Y, Form the adjusted/normalized For a chosen 8. Ideally, technical replicates are placed across batches so that unwanted variation between any pair of batches is captured via differences of expression values between technical replicates. We previously showed the performance of RUV-III with pseudo-replicates in removing unwanted variation from gene expression data. Pseudo-replicates are samples from the same biological groups across batches. The idea of pseudo-replicates has also been used to remove batch effects in TCGA RNA-seq data with the unpublished algorithm EB++56.We used our recently developed normalization method, RUV-III, which makes essential use of technical replicates and negative control genes, to estimate unwanted variation and remove it from the dataAs there are no technical replicates in the TCGA RNA-seq datasets, we developed an approach, PRPS, to be able to use RUV-III to remove unwanted variation from the data. To use RUV-III with PRPS, we first need to find sources of unwanted variation that we aim to remove from the data and identify relatively homogenous biological subpopulations among the samples. Then, we build pseudo-samples, which are in silico samples derived from a group of samples that are roughly homogeneous with respect to the unwanted variation and biology. The pseudo-samples from each biological groups will be a set of pseudo-replicates.To make the process clear, we illustrate the PRPS for TCGA COAD and BRCA RNA-seq studies. For example, with the TCGA COAD samples, we regarded the 12 combinations of four CMSs  with three microsatellite instability (MSI) statuses  as defining biological subpopulations. The key time intervals 2010 and 2011–2014 contribute the main unwanted variation that we saw after preliminary exploration of the data. As these times are totally confounded with sequencing plates , we considered plates to be the batches when defining PRPS. In addition, we were able to remove plate effects within each key time interval. As a result, to remove unwanted variation from the COAD data without removing biology, we created sets of pseudo-samples as follows: (1) select those biological subpopulations out of the 12 mentioned above that have at least three samples in at least two plates while also ensuring that, in the end, there are samples from plates within and across the two key intervals; and (2) average the gene expression levels of the corresponding samples within the individual plates to create one pseudo-sample. Having done this for all 12 biological subpopulations, we suppose all the pseudo-samples created across plates for a particular biological subpopulation to form a pseudo-replicate set.For the BRCA data, we aimed to remove four different sources of unwanted variation—library size, tumor purity, flow cell chemistry and plate effects—from the data. Then, we needed to create distinct groups of PRPS for each source of unwanted variation. Note that we created a group of PRPS to remove the effects of plates and flow cell chemistries, as they are completely confounded with each other.We considered the PAM50 subtypes  to define the major biological subpopulations and then created several sets of PRPS for each source of unwanted variation. To create PRPS for library size, we selected plates that have at least 12 samples of a particular PAM50 subtype and then selected the samples with the three highest and the samples with the three lowest values of library size. Then, we created two pseudo-samples within each PAM50 subtype per plate by averaging the gene expression values across each set of three high library size samples and each set of three low library size samples. We adopted the same approach explained above for the COAD data to create PRPS for plate effects. For removing the effect of tumor purity in the BRCA data, we defined sets of PRPS for each PAM50 subtype in addition to those that we created for removing library size, flow cell chemistries and plate-to-plate variation. We performed this by selecting the samples with the three highest and the samples with the three lowest values of tumor purity within each PAM50 subtype. Then, we created two pseudo-samples within each PAM50 subtype by averaging the gene expression values across each set of three high-purity samples and each set of three low-purity samples. Finally, the two pseudo-samples (average high and average low purity) created for each PAM50 subtype were regarded as forming a pseudo-replicate set—that is, a pair of pseudo-duplicates.8 explained that our approach to negative controls is pragmatic: if regarding a set of genes as negative controls helps to remove unwanted variation using RUV-III, as evaluated by various metrics, then whether or not they are ideal negative control genes is not our concern. For the different cancer types discussed in this paper, we used different sets of negative control genes derived from either the literature  or the data itself . Candidate control genes have their effectiveness evaluated using various metrics after their use in RUV-III. It should be noted that unwanted variation mostly affects different subsets of genes in different ways.Negative control genes for RUV-III are genes that are not highly affected by the biological factors of interest but are affected by one or more forms of unwanted variation in the data. We previouslyK of unwanted variation needs to be determined. To find a suitable value, we repeated the analysis with a range of values of K and evaluated the quality of each analysis using different statistical metrics and prior biological knowledge. RUV-III is generally robust to overestimating K but not always.To use RUV-III, a dimension As described above, the 11 combinations (we do not have CMS4_MSI-H) of the four CMS subtypes identified by the R package CMScaller on the READ FPKM and FPKM.UQ RNA-seq data , and the three MSI statuses, were considered to be homogenous biological populations for the purpose of creating PRPS. Supplementary Fig. F-statistics were selected . PCA plots of the READ RNA-seq raw counts using the negative control genes showed that they capture the large library size differences between the key time intervals and do not capture CMS subtype differences  select genes (262 genes) that have lowest F-statistics (F-statistics < 20) from (1), the lowest correlations (ρ < 0.3) from (2) and the highest correlations (ρ > 0.9) from (3). PCA plots of the TCGA COAD RNA-seq raw count using negative control genes show that they capture the key time differences . We used these CMSs and MSIs to define homogenous biological populations for the purpose of creating the PRPS  from (1), the highest F-statistics (F-statistics > 100) from (2), the highest correlations (|ρ| > 0.7) from (3) and the highest correlations (ρ > 0.07) from (4). PCA plots of the TCGA BRCA RNA-seq raw count using the negative control genes show that these genes capture all sources of unwanted variation in the data  carry out an ANOVA on the FPKM.UQ normalized gene expression values with PAM50 subtype as the factor, within each flow cell chemistry; (2) carry out a similar ANOVA with flow cell chemistry as the factor; (3) calculate Spearman correlations between FPKM.UQ normalized gene expression and purity values within the PAM50 subtypes; (4) calculate similar Spearman correlations with library size but with the raw counts; (5) select genes  with the lowest 59, ComBat-seq22 and RUVg1 methods in our analysis as these are not specifically designed for normalization, although they can be helpful for that task when the unwanted variation is orthogonal to the biology, something that is rarely known in advance. The same applies to the RUVs method provided in the RUVseq package1. Although if there are true replicates (missing from TCGA and most large cancer RNA-seq studies), it can be used to normalize RNA-seq datasets5.We did not include the SVAseq4.The PCs  of the sample × transcript array of log counts are the linear combinations of the transcript measurements having the largest, second largest, third largest, etc., variation, standardized to be of unit length and orthogonal to the preceding components. Each will give a single value for each sample. In this paper, PCA plots are of the second PC values versus the first PC values and of the third PC versus the first PC. The calculations are done on mean-corrected transcript log counts, using the R code adopted from the R package EDAseq (version 2.26.1)23 are used to reveal trends, temporal clustering and other non-random patterns resulting from unwanted variation in gene expression data. To generate RLE plots, we first formed the log ratio log(yig/yg) of the raw count yig for gene g in the sample labeled i relative to the median value yg of the counts for gene g taken across all samples. We then generated a box plot from all the log ratios for sample i and plotted all such box plots along a line, where i varies in a meaningful order, usually sample processing date. An ideal RLE plot should have its medians centered around zero, and its box widths and their interquartile ranges (IQRs) should be similar in magnitude. Because of their sensitivity to unwanted variation, we also examined the relationships between RLE medians with potential sources of unwanted variation and individual gene expression levels in the datasets. In the absence of any influence of unwanted variation in the data, we should see no such associations.RLE plots60 to quantify the strength of (linear) relationships between two sets of variables, such as the first k PCs (that is 1 ≤ k ≤ 10) and dummy variables representing time, batches, plates and biological variables. Not only does this quantity summarize the full set of canonical correlations, but it also reduces to the familiar R2 from multiple regression (see below) when one of the variable sets contains just one element.We used the Rozeboom squared vector correlationR2 values of fitted linear models are used to quantity the strength of the (linear) relationships between a single quantitative source of unwanted variation, such as sample (log) library size or tumor purity, and global sample summary statistics, such as the first k PCs (1 ≤ k ≤ 10). The lm R function was used for this analysis.61. We computed the partial correlation between the expression levels of pairs of genes controlling for tumor purity using the pcor.test function from the R package ppcor (version 1.1)61.Partial correlation is used to estimate Pearson (linear) correlation between two variables while controlling for one variablesF-statistics to summarize the effects of a qualitative source of unwanted variation  on the expression levels of individual genes, where genes having large F-statistics are deemed to be affected by the unwanted variation. We also use ANOVA tests (the aov function in R) to assign P values to the association between tumor purity and molecular subtypes.ANOVA enables us to assess the effects of a given qualitative variable  on gene expression measurements across any set of groups (labeled by the levels of the factor) under study. We use ANOVA 62 and others that histograms of the raw  P values resulting from testing the same hypothesis  on thousands of genes can be powerful indicator of the presence of unwanted variation. When there is no such variation and the underlying statistical model is appropriate, such P value histograms should be uniform apart from a possible peak near zero corresponding to genes where the null should be rejected. When there is unwanted variation, the histograms typically look very far from uniform apart from a peak near zero.It has been shown by Leek and Storey63.We used silhouette coefficient analysis to assess the separation of biological populations and batch effects. The silhouette function uses Euclidean distance to calculate both the similarity between one patient and the other patients in each cluster and the separation between patients in different clusters. A better normalization method will lead to higher and lower silhouette coefficients for biological and batch labels, respectively. The silhouette coefficients were computed using the function silhouette from the R package cluster (version 2.1.2)64 is the corrected-for-chance version of the Rand index. The ARI measures the percentage of matches between two label lists. We used the ARI to assess the performance of normalization methods in terms of sample subtype separation and batch mixing. We first calculated PCs and used the first three PCs to perform ARI.The ARI2-transformed raw counts and normalized data65. To evaluate the effects of the different sources of unwanted variation on the data, DE analyses were performed across batches. In the absence of any batch effects, the histogram of the resulting unadjusted P values should be uniformly distributed. The wilcox.test R function was used for this analysis.DE analyses were performed using the Wilcoxon signed-rank test with logWe made use of both global and gene-level approaches to identify and quantify unwanted variation in RNA-seq datasets  are each of the first three PCs against each other, colored by known sources of unwanted variation—for example, time—or known biology—for example, cancer subtype. Linear regression is used to quantify the relationship between the first few PCs and continuous sources of unwanted variation, such as (log) library size. The R2 calculated from the linear regression analyses indicates how strongly the PCs capture unwanted variation in the data, and we perform these calculations cumulatively—that is, continuous source versus all of , for k = 1,…,5 or 10. Similarly to linear regression, we used vector correlation analysis to assess the effect on the data of discrete sources of unwanted variation, such as years or year intervals. Silhouette coefficients and ARIs were used to quantify how well experimental batches are mixed and known biology is separated. Finally, RLE plots23 were used to assess the performance of different normalizations in terms of removing unwanted variation from the data. We also explored the relationship between the medians and the IQRs of the RLE plots with sources of unwanted variation.Our global approaches involve the use of PCA plots, linear regression, vector correlation analyses, silhouette coefficients, ARIs and RLE plotsP value histograms and assessing the expression levels of negative control genes (see above), positive control genes (genes whose behavior we know), Spearman correlation and ANOVA between individual gene expression and sources of unwanted variation. These methods assess and quantify the effects of unwanted variation on individual gene expression levels in the RNA-seq datasets. See The gene-level approach includes DE analyses between experimental batches, looking at 25 was used to identify the CMSs in the TCGA READ and COAD RNA-seq data. The function provides classification based on pre-defined cancer-cell-intrinsic CMS templates.We identified gene-expression-based cancer subtypes to create PRPS for RUV-III normalization. The CMScaller function with default parameters from the R package CMScaller (version 2.0.1)66 on the estrogen receptor (ER) balanced data. The ER estimates = 1.4 were selected to divide samples into ER-positive and ER-negative groups, and then the calibration  factors were calculated.We used two approaches to identify the PAM50 subtypes in the TCGA BRCA RNA-seq data. We implemented an algorithm proposed by Picornell et al.https://github.com/bhklab/genefu) to calculate the correlation coefficient for individual PAM50 subtypes. Then, the individual sample is assigned to a particular PAM50 subtype based on its highest correlation coefficient.In addition, we also used the molecular.subtyping function with the PAM50 (single sample predictor) model from the R/Bioconductor package genefu (version 2.26.0) to identify the PAM50 subtypes. This method performs Spearman correlation between the expression of the PAM50 genes of each sample and PAM50 centroids (these data were downloaded here: We used Kaplan–Meier survival analysis to assess the prognostic values of different PAM50 identification approaches. The results showed that the PAM50 subtypes obtained by the genefu method are slightly more prognostic than those obtained by the other method (Supplementary Fig. We used two strategies to show spurious gene–gene correlations in the TCGA normalized data. First, we demonstrated how sources of unwanted variation, such as library size, tumor purity and batch effects, can introduce such correlations, which we did not see in the RUV-III normalized data. Second, we used the TCGA microarray gene expression data as orthogonal platform to explore and confirm these correlations. The TCGA microarray data contain gene expression data of subsets of samples that were profiled by RNA-seq platform. Our normalization assessment showed that the microarray data were not influenced by plates and time effects.To explore spurious gene–gene correlations introduced by tumor purity in the TCGA data, we used the LCM microarray data, as these contain only gene expression signals from cancer cells. Note that we assessed both purity variation and quality of the LCM data.Further information on research design is available in the Any methods, additional references, Nature Research reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at 10.1038/s41587-022-01440-w.Supplementary InformationSupplementary Text and Figs. 1–42Reporting SummarySupplementary Table 1Supplementary Table 1. Collated sample metadata for 11,023 TCGA RNA-seq samples, including date, plate identifier, center, tissue and purity


In the analysis of single-cell RNA-sequencing (scRNA-seq) data, how to effectively and accurately identify cell clusters from a large number of cell mixtures is still a challenge. Low-rank representation (LRR) method has achieved excellent results in subspace clustering. But in previous studies, most LRR-based methods usually choose the original data matrix as the dictionary. In addition, the methods based on LRR usually use spectral clustering algorithm to complete cell clustering. Therefore, there is a matching problem between the spectral clustering method and the affinity matrix, which is difficult to ensure the optimal effect of clustering. Considering the above two points, we propose the DLNLRR method to better identify the cell type. First, DLNLRR can update the dictionary during the optimization process instead of using the predefined fixed dictionary, so it can realize dictionary learning and LRR learning at the same time. Second, DLNLRR can realize subspace clustering without relying on spectral clustering algorithm, that is, we can perform clustering directly based on the low-rank matrix. Finally, we carry out a large number of experiments on real single-cell datasets and experimental results show that DLNLRR is superior to other scRNA-seq data analysis algorithms in cell type identification. The single-cell RNA-sequencing (scRNA-seq) technology is now a powerful tool that demonstrates unprecedented precision in exploring biological processes and disease mechanisms –3. The sIn recent years, a number of specific methods have been proposed to overcome the challenges posed by the inherent nature of scRNA-seq data. Xu and Su proposed a quasi-clique-based clustering algorithm called SNN-Cliq , which cXu et al. proposed Concept Factorization (CF), which attempts to find the data representation by using the linear combination of original data points to represent the cluster center . InspireBefore introducing the proposed model, we will review some related methods in this section, including LRR, manifold graph regularization and CF.A is a basis matrix, called a dictionary. H is the LRR matrix with respect to the dictionary A, and the column vector j in the subspace. The matrix E denotes the noise in the original data. A enables the LRR matrix H to reveal the true subspace structure of the data.Because of the good ability to grasp the global data structure and explore the low-dimensional subspace structure, LRR has attracted the attention of a large number researchers and achieved good results in the application of bioinformatics . In the mentclass2pt{minimX as vertices of the connected graph. Then, the symmetric weight matrix S is defined, S as follows:k nearest neighbors of t controls the width of the neighborhoods, which is 1 by default. Based on the local invariance assumption, the definition of graph regularization in low dimensional space is as follows:D is a diagonal matrix, L is the graph Laplacian matrix . In general, if the values of NMI and ARI are close to 1, it indicates that the clustering results are close to the real situation.Let  cluster . ARI is r of LRR matrix H becomes the key of this method, which will affect the accuracy of clustering results. We choose the Ting dataset as an example to further verify the influence of different dimensions r on the learning low-rank matrix. To more intuitively demonstrate the learning ability of low rank matrix to subspace structure, we show the heatmap of r in Fig. r is chosen appropriately, the number of presented blocks should be consistent with the number of the cell types. It can be seen from Fig. r is not appropriate, and its value is lower than the real number of clusters, resulting in considerable similarity between different types of cells. Obviously, it is inappropriate to set r. According to the previous discussion, subspace clustering can be realized directly based on H only when the value of r is consistent with the number of clustering clusters. We note that when r. So, we can infer that the number of clusters is 5. In order to achieve direct clustering, we choose the r value consistent with the number of diagonal blocks, that is, r is set to 5.According to section ‘DLNLRR Method’, each column vector of the LRR matrix entclass1pt{minimaentclass1pt{minimaThis subsection analyzes the impact of parameters in DLNLRR on the clustering performance. According to formula , there aCell clustering is one of the important tasks in the mining and analysis of scRNA-seq data, and its main purpose is to distinguish cell types. In this subsection, to validate the clustering performance of DLNLRR, DLNLRR and several most advanced single-cell data clustering methods are applied to cluster cells on eight real scRNA-seq datasets described in Table Firstly, it can be seen from Table Secondly,we can see from Fig. Visualizing scRNA-seq data in low-dimensional space is a powerful way to pre-identify cell subpopulations. Previous studies proposed an improved t-distributed Stochastic Neighbor Embedding (t-SNE) for dimensionality reduction and visualization of data to verify the performance of the learned similarity matrix . In thisGene marker prioritization has attracted extensive attention since it was proposed. Gene markers have rich in biological information, which helps to distinguish cell subpopulations and reveals the complexity of cells. In this subsection, we identify the gene markers for each cell type in the Darmanis dataset based on the learned low-rank matrix. First, Bootstrap Laplacian scoring  is perfoThe development of scRNA-seq and high-throughput technologies has facilitated the exploration of single-cell function and brings computational challenges to reveal the relationship between cell lineages. Cell clustering and extraction of gene markers are important research components of analyzing scRNA-seq data. In this paper, we propose the DLNLRR method for scRNA-seq data analysis. Inspired by the idea of CF, DLNLRR uses the linear combination of original data to construct the dictionary. Instead of using the predefined dictionary, DLNLRR can update the dictionary in the iterative solution process, which is helpful to obtain the mapping benchmark that can better represent the subspace, and then obtain the subspace structure of data accurately. In addition, in the DLNLRR model, we can cluster the samples directly based on the LRR matrix, which can avoid the influence of spectral clustering algorithms on the clustering results.A large number of experiments in this paper show that DLNLRR can capture local structures in the data, and can quickly and accurately obtain clustering results, which has advantages in cell type recognition.However, our method still has some limitations. More comprehensive experiments and analysis are needed. Our method has only been tested on real single cell sequencing datasets, and has not verified whether it is effective on other datasets or large-scale datasets. In addition, our method requires preset parameters, which may affect the performance of the method. The scRNA-seq data analysis still faces some challenges, such as the identification of cluster numbers, the inde-pendent selection of appropriate parameters, etc. In the next work, we will continue to explore adaptive parameter selection methods and the application of LRR on the scRNA-seq datasets, and pay attention to the development of ensemble clustering technology for scRNA-seq data analysis.


Dataset sizes in computational biology have been increased drastically with the help of improved data collection tools and increasing size of patient cohorts. Previous kernel-based machine learning algorithms proposed for increased interpretability started to fail with large sample sizes, owing to their lack of scalability. To overcome this problem, we proposed a fast and efficient multiple kernel learning (MKL) algorithm to be particularly used with large-scale data that integrates kernel approximation and group Lasso formulations into a conjoint model. Our method extracts significant and meaningful information from the genomic data while conjointly learning a model for out-of-sample prediction. It is scalable with increasing sample size by approximating instead of calculating distinct kernel matrices.To test our computational framework, namely, Multiple Approximate Kernel Learning (MAKL), we demonstrated our experiments on three cancer datasets and showed that MAKL is capable to outperform the baseline algorithm while using only a small fraction of the input features. We also reported selection frequencies of approximated kernel matrices associated with feature subsets (i.e. gene sets/pathways), which helps to see their relevance for the given classification task. Our fast and interpretable MKL algorithm producing sparse solutions is promising for computational biology applications considering its scalability and highly correlated structure of genomic datasets, and it can be used to discover new biomarkers and new therapeutic guidelines.https://github.com/begumbektas/makl together with the scripts that replicate the reported experiments. MAKL is also available as an R package at https://cran.r-project.org/web/packages/MAKL.MAKL is available at Bioinformatics online. Cancer is one of the most challenging healthcare problems of our era. With increasing size of patient cohorts and genomic characterizations collected from tumour biopsies, machine learning algorithms have been applied to different tasks in the hope of discovering molecular mechanisms related to the diagnosis, prognosis and treatment of cancer. Together with good predictive performance, building interpretable machine learning algorithms plays a crucial role in cancer research to discover new therapeutic biomarkers and to analyze the heterogeneity of tumours. Kernel-based machine learning algorithms have been widely used in computational biology, thanks to their capability to deal with highly correlated data and interpretability. However, the problem with these existing algorithms is their lack of scalability, which started to cause problems especially for single-cell cancer research, since traditional kernel-based methods are computationally prohibitive with large sample sizes.Kernel machines attract a significant amount of attention since they are able to approximate any function during the learning process when the size of the training data is large enough. Kernel machines are also capable of incorporating prior knowledge into learning process using the kernel function. However, machine learning methods that need to calculate a kernel matrix over the dataset scale poorly with increasing training sample size, since the size of the kernel matrix is quadratic in the number of training samples. One of the most widely known kernel-based machine learning method is the support vector machine algorithm . Some of the existing MKL algorithms assign equal weights to the input kernels, which may not be the best way of discovering the underlying dynamics. For instance, in computational biology, to discover the underlying molecular mechanisms from genomic data, instead of initially setting the kernel weights, making the MKL algorithm choose the weights assigned to each kernel in a data-dependent manner is more convenient. Additionally, combining kernels in a non-linear or data-dependent way is stated to seem more promising than linear combination in fusing information provided by simple linear kernels, whereas linear methods are stated as being more reasonable in case of combining complex Gaussian kernels .The main contribution of this study is to extend well-known MKL idea to large-scale problems using an approximation procedure for kernel matrices. The proposed algorithm was shown to have four important characteristics: (i) running in a scalable manner on large-scale genomic data, (ii) getting sparse solutions, (iii) integrating prior information during the learning process to increase the interpretability and (iv) maintaining or even increasing the predictive performance.In this study, we used three datasets formed using two data sources. First, we used data from The Cancer Genome Atlas (TCGA) project, to discover the molecular mechanisms related to early- and late-stage cancers and 2-year survival of patients. Second, we used single-cell RNA-Seq data provided by the Broad Institute Single Cell Portal to understand molecular signatures related to cell malignancy, which could be beneficial for understanding tumour plasticity and cancer progression mechanisms.https://portal.gdc.cancer.gov. TCGA shared the RNA-Seq measurements of the tumours from 33 cohorts and pre-processed them with a unified pipeline. We downloaded HTSeq-FPKM profiles of all primary tumours from the most recent data freeze  and finally got 9911 files in total. To obtain tumour grade levels and survival-related information, we used clinical annotation files of the patients.We gathered genomic data and clinical annotation files of over 10 000 cancer patients from 33 cancer cohorts in the Genomics Data Commons (GDC) data portal offered by TCGA consortium at For binary classification task of cancer stages, we considered patients clinically annotated as Stage I as having early-stage cancer; patients clinically annotated with Stages II, III or IV as having late-stage cancer. We included cohorts having at least 20 tumours from both categories. We combined data coming from all cohorts to form one big dataset of 19 814 features. As a result, the final dataset contains 5547 patients coming from 15 distinct cohorts.For 2-year survival classification, we considered patients that have vital_status, days_to_death, days_to_last_followup, days_to_last_known_alive information. We combined data from 33 cohorts by including patients with the required survival information. We included cohorts having at least 20 patients whose vital_status is dead and at least 100 patients in total. The resulting dataset contains 5168 patients coming from 20 distinct cohorts with 19 814 expression features. We labelled the patient as having ≥2-year survival (i) if vital_status is alive and days_to_last_followup or days_to_last_known_alive is ≥2 years; (ii) if vital_status is dead and days_to_death is ≥2 years. On the other hand, we labelled the patient as having <2-year survival if vital_status is dead and days_to_death is <2 years.https://singlecell.broadinstitute.org and found the melanoma immunotherapy dataset, which is appropriate for our binary classification task, provided by Analysis of single-cell RNA-Seq data holds promise to find new ways to treat cancer. To reveal the mechanisms related to tumour plasticity for targeted cancer therapies and to find new biomarkers, single-cell analysis is in the spotlight. Having this motivation, we searched for the datasets in the Broad Institute Single Cell Portal at We intersected the gene expression profiles of 7186 cells with malignancy information coming from 6738 clinical annotation files. For each cell, there were 23 686 gene expression measurements in the dataset. If a cell is annotated as B.cell, CAF, Endothelial.cell, Macrophage, NK, T.CD4, T.CD8 or T.cell, we labelled this cell as non-malignant. All other cells were labelled as malignant.We used two pathway/gene set collections from the Molecular Signatures Database (MSigDB) that are specifically curated for cancer research: Hallmark gene set collection  .D is given as an input to algorithm, and it can cope with large-scale data well, since instead of calculating kernel matrices which are of size N × N, we now need to calculate matrices of size N × D, where To accelerate the kernel matrix computation step of MKL, we used a modified version of random Fourier features mentioned by Once the appropriate low-dimensional approximation method for the kernel matrices is set, another problem arises at how to integrate this approximation matrices into the learning process.As a popular model selection and shrinkage estimation method, the Lasso estimator had been proposed by Despite giving interpretable results, the problem with all ordinary MKL methods is their scalability issue owing to their need to calculate distinct kernel matrices, which scale quadratically with the sample size. This issue could be problematic once they are applied to large-scale datasets in areas such as computational biology, computer vision etc.Theorem 1.A continuous kernel on is positive definite if and only if is the Fourier transform of a non-negative measure Rudin, .Owing to high computational complexity of using kernel trick in large-scale datasets, the questions of finding faster methods arises. Instead of computing the kernel matrices, to accelerate the learning process, there is a need to find a low-dimensional approximation for kernel matrix calculation. For this purpose, we approximate the kernel matrices using a modified version of random feature mapping originally has been introduced by Briefly, as a result of Bochner’s theorem, we are ensured that for any properly scaled shift-invariant kernel Z as the output from this algorithm, where we used the To calculate similarity between pairs of gene expression profiles, we used the Gaussian kernel on feature sets in Algorithm 1, which is a modified version of the random feature mapping algorithm proposed by The Gaussian kernel was previously reported in the literature as being reliable to be used with high-dimensional genomic data , λ is a positive regularization coefficient, pd refers to the size of feature set p and The group Lasso minimizes the following objective function with respect to model parameters Overall, our algorithm for MKL using any kernel approximation method is described in Algorithm 2 with details. Algorithm 2Fast integration of approximated kernels into group LassoInput: Training matrix P feature sets y of length N.Output: Classification area under the receiver operating characteristics curve (AUROC) value, pη for all P partitions.Split the training matrix into Compute low-dimensional approximation matrices Concatenate distinct approximation matrices Z as the input matrix.Solve the group Lasso formulation using pη for all Calculate pη is non-zero, count If To test our algorithm, we performed computational experiments for three binary classification tasks. To better compare the outcomes of the experiments, we benchmarked all algorithms over 100 independent replications. As the predictive performance metric for these binary classification tasks, we used AUROC, whose larger values correspond to a better classification performance.Owing to the serious scalability issues presented at earlier sections, we did not choose standard MKL algorithms for baseline comparison. We chose extreme gradient boosting XGBoost;  as the bFor all experiments, we split the corresponding dataset into two parts, in a way that 80% of the samples were included into the training set and the remaining 20% of the samples were included into the test set. We normalized each feature in the training set to have zero mean and unit SD. For the test set, we normalized each feature with the mean and SD calculated on the original training set. We performed 4-fold inner cross-validation on the training set for hyper-parameter tuning. We used two aforementioned pathway/gene set collections.For XGBoost algorithm that we used as the baseline algorithm, the hyper-parameter, maximum depth of a tree, was chosen using 4-fold inner cross-validation on the training set with a maximum number of iterations of 1000 and learning rate of 0.2. Since, in this article, we cover binary classification tasks, we used ‘binary: logistic’ option as the objective function argument to the algorithm. For XGBoost experiments, we used XGBoost R package .λ, for MAKL was chosen using four-fold cross-validation on the training set. The λ parameter set used for cross-validation was The hyper-parameter related to the regularization, D (i.e. the number of random features used to approximate distinct kernel matrices) was given as an input to the MAKL algorithm. We ran experiments using different D values, The dimensionality parameter rameters .D values for MAKL with Hallmark gene sets (i.e. different numbers of randomly chosen Fourier samples). We compared MAKL algorithm against the baseline algorithms, XGBoost (Subset) and XGBoost (Full). XGBoost (Subset) denotes the algorithm is trained using the genes from the Hallmark gene set collection, whereas XGBoost (Full) denotes the algorithm is trained using all available 19 814 genes.For this binary classification task, we performed sensitivity analysis for the AUROC values using different D = 150. It also outperformed XGBoost (Full) while using only 12.47% of the all available 19 814 genes on the average. Additionally, D values for MAKL with PID pathway collection and compares MAKL against the baseline algorithms, namely, XGBoost (Subset) and XGBoost (Full).D = 150. It also outperformed XGBoost (Full) while using only 4.15% of all available 19 814 genes on the average.For 2-year survival classification task, MAKL with PID pathway collection was able to outperform the baseline algorithms XGBoost (Subset) using only 5.96% (i.e. 11.69 out of 196) of the total feature sets (i.e. pathways) with D values for MAKL with PID pathway collection (i.e. different numbers of randomly chosen Fourier samples), and compares them against the baseline algorithms, XGBoost (Subset) and XGBoost (Full). XGBoost (Subset) denotes the algorithm is trained using the genes from the PID pathway collection, and XGBoost (Full) denotes the algorithm is trained using all available 19 814 genes. Additionally, D values of MAKL with Hallmark gene sets and compares the results with the baseline algorithms, XGBoost (Subset) and XGBoost (Full). XGBoost (Subset) denotes the algorithm is trained using the genes from the Hallmark gene set collection, and XGBoost (Full) denotes the algorithm is trained using all available 19 814 genes.As mentioned earlier, our algorithm is capable of extracting meaningful information while learning a predictive model. The weights associated with each pathway/gene set resulting from MAKL show the relevance of each pathway in the given classification task. Owing to the computational pre-processing and labelling techniques used in single-cell dataset that we used , we concentrated on the pathways/gene sets that play crucial role in cell malignancy classification task, rather than the classification performance. We demonstrated that the pathways/gene sets that our algorithm MAKL found important for this classification task are relevant in terms of finding new immunotherapy techniques, new ways to discover tumour heterogeneity since the pathways/gene sets extracted from our algorithm are in line with the most recent melanoma treatment trials, which are supported with the literature findings as detailed below.λ multipliers). If norm of weights associated with a feature set, resulting from MAKL is positive, we considered the corresponding feature set (i.e. pathway/gene set) as selected. λ multipliers). According to the average selection frequencies over 100 replications, the most selected pathway by MAKL with PID pathway collection was CXCR4 pathway. This result is relevant since targeting of CXCR4 pathway is under consideration in melanoma treatment trials. It is stated in literature that targeting melanoma by CXCR4 inhibition may be a good way to destroy the tumour cells and that CXCR4 regulates tumour immunity . MAKL is fast and appropriate for large-scale genomic data such as single-cell sequencing data having large sample-size with many features. Our method can integrate prior information in the form of gene sets/pathways into the learning process to increase the interpretability without sacrificing the predictive accuracy.To benchmark our algorithm, we used three datasets constructed from two data sources. As prior information sources, we used Hallmark and PID pathway/gene set collections particularly curated for cancer research. We processed genomic data from TCGA project to form the early- and late-stage cancer dataset together with 2-year cancer survival dataset. We also used single-cell RNA-Seq data provided by the Broad Institute Single Cell Portal to understand the molecular underpinnings of cell malignancy. Our algorithm MAKL was capable of outperforming the baseline algorithms based on XGBoost using only a small fraction of the pathway/gene sets available see ; thus, iAn interesting direction for future studies may be building a multi-class extension of MAKL to be used with multi-class problems. Also, integration of low-dimensional kernel approximation into other machine learning models is exciting considering the increasing need for fast data processing and interpretability. Our approach is promising in domains including but not limited to computational biology and computer vision thanks to its scalability, flexibility and also its ability to deal with highly correlated features.This work was supported by the Scientific and Technological Research Council of Turkey (TÜBİTAK) [EEEAG 117E181]. M.G. was supported by the Turkish Academy of Sciences  and the Science Academy of Turkey .Conflict of Interest: none declared.btac241_Supplementary_DataClick here for additional data file.


Deep learning has massive potential in predicting phenotype from different omics profiles. However, deep neural networks are viewed as black boxes, providing predictions without explanation. Therefore, the requirements for these models to become interpretable are increasing, especially in the medical field. Here we propose a computational framework that takes the gene expression profile of any primary cancer sample and predicts whether patients’ samples are primary  or metastasized to the brain, bone, lung, or liver based on deep learning architecture. Specifically, we first constructed an AutoEncoder framework to learn the non-linear relationship between genes, and then DeepLIFT was applied to calculate genes’ importance scores. Next, to mine the top essential genes that can distinguish the primary and metastasized tumors, we iteratively added ten top-ranked genes based upon their importance score to train a DNN model. Then we trained a final multi-class DNN that uses the output from the previous part as an input and predicts whether samples are primary or metastasized to the brain, bone, lung, or liver. The prediction performances ranged from AUC of 0.93–0.82. We further designed the model’s workflow to provide a second functionality beyond metastasis site prediction, i.e., to identify the biological functions that the DL model uses to perform the prediction. To our knowledge, this is the first multi-class DNN model developed for the generic prediction of metastasis to various sites. Precision medicine is a path that could profoundly change and improve medical practices. This idea proposes using genetic data of individual patients to enhance clinical decision-making, and “omics” technologies now provide a means to acquire such patient data, making precision medicine feasible. Clinical decision-making includes diagnosis, prognosis, choosing the most appropriate treatment, etc. One avenue pursued to support clinical decision-making is building classifiers using gene expression profiles that can function as forms of artificial intelligence (AI).Many machine learning methods, including support vector machines, random forest, and boosting, are among the primary tools currently being used to make biological discoveries from the vast amount of available gene expression data . HoweverThe main concern of DL used in medical applications is the lack of interpretability. The reason being, DL networks can be viewed as black boxes that form an input layer (wherein we place the gene expression profile of patients) and an output layer (offering predictions without interpretability). Suppose we do not meet this interpretability criterion at a good standard. In that case, physicians will not be able to trust the decision of the neural network, as they need interpretable data to ensure patients’ safety. Specifically, they need data about neurons, genes, and related biological processes involved in the prediction and the decision-making process to make informed decisions. Thus, researchers are now attempting to make the DL networks more interpretable.In this work, we attempted to develop an AI method that could translate into a tool that supports clinical decision-making with regard to identifying metastasis and pinpointing the metastasis site . In thisWe searched for gene expression datasets in Gene Expression Omnibus (GEO)  using thThe first part of our model’s framework comprises three key components, namely the AE , DeepLIFhttps://github.com/fchollet/keras) to implement an AE consisting of three fully connected hidden layers containing 500, 100, and 500 neurons. For each layer, we used “relu” as the activation function. Given m samples, each has a gene expression profile containing n genes; the input vector is reconstructed through a series of matrix transformations of multiple network layers. Training an AE involves finding parameters that minimize a specific loss function; we used mean absolute error (MAE) as the loss function. In addition, we added an L2 regularization penalty to control overfitting and used the early stopping technique. Finally, we trained the AE using the Adam  data the Adam  optimizaSecond, the DeepLIFT-based component is a feature scoring algorithm to calculate the contribution scores of each neuron. In our computing framework, we used DeepLIFT to calculate a contribution score for every gene of each input sample. The obtained contribution scores express the importance of the corresponding genes for the compression features of the low-dimensional representation (bottleneck) layer. Then, we ranked the genes based on their importance scores.Third, the DNN-based component is a neural network with three hidden layers with 64, 32, and 8 neurons, respectively, and uses “relu” as the activation function. We used the Python Keras library to design the DNN model to predict if a sample is primary or metastasized. Finally, we iteratively added ten top-ranked genes (based on the importance scores) to train the DNN model.https://www.python.org/), using the Keras deep learning and DeepLIFT frameworks  to determine each class’s most essential neurons in each layer.Finally, we ranked the neurons according to their average relevance scores and chose the most essential ones. Similar to , assuminL, its activation is back propagated using the Deep-LIFT approach to compute the relevance score of each input gene. We then identified the most critical inputs that have an impact on the activation of the neuron. Similar to identifying the essential neurons, we used a two-sided t-test to select the essential input genes.For a given important neuron in layer The second step is to connect each essential neuron to biological functions from GO, signature gene-set from MSigDB, and diseases from DisGeNET. Finally, we used an R interface to the Enrichr database EnricherR  to identIn the first part of our model’s framework, we used GEO samples to train an AE and applied DeepLIFT to calculate importance scores for each gene for ranking. Then, ten top-ranked genes (based on the importance scores) were iteratively fed to the DNN model to identify the gene set that provides maximum performance when determining if a sample is primary or metastasized. The DNN reaches its maximum performance when including 60, 80, 20, and 30 top-ranked genes in metastasis to bone, brain, lung, and liver data, respectively . For theAfter removing the duplicates, the 190 essential genes for all sites identified in the part of the model’s framework were reduced to 184 genes. The 184 genes were used as an input to the final multi-class DNN model. This model takes these 184 genes and predicts if the input samples are primary or metastasized to the bone, brain, lung, or liver site. The final multi-class DNN model achieved a good prediction performance; however, the prediction performance does not indicate the robustness of the final DNN model. Thus, we further evaluated the model’s performance using external testing data from the TCGA datasets  and usinNonetheless, the prediction performance using samples from a population-based cohort shows that the multi-class DNN model achieved good prediction performance based on area under the ROC curve (AUC of 0.72), when distinguishing between the primary and metastatic samples , and shoWe further designed the model’s workflow to provide a second functionality beyond metastasis site prediction, i.e., to identify the biological functions that the DL model uses to perform the prediction .We achieved this through the biological interpretation of the neural network predicting the metastasis. That is, for each class, the essential neurons of each layer are selected based on the mean of the relevance scores by using the method described in p-value < 0.05. For each essential neuron’s list of genes, we determined GO biological functions based on a To support and increase the biological insights extracted from the essential neurons’ list of genes, we also performed MSigDB enrichment. In this analysis, we only considered MSigDB enrichments significant to at least three metastasis sites. Only three enriched categories were significant to all four sites, namely “Epithelial Mesenchymal Transition,” “Apoptosis,” and “IL-2/STAT5 Signaling” . The apoWe also performed DisGenNET enrichment. In this analysis, we only considered DisGenNET disease enrichments significant to at least two metastasis sites. Only eight enriched disease categories were significantly associated with at least two metastasis sites, namely Autoimmune Diseases, Carcinoma breast stage IV, Cirrhosis, Dermatomyositis, Giant Cell Tumors, Leukemia, Metastatic malignant neoplasm to brain, and Rheumatoid Arthritis . Four diWe further determined the overlapping genes between the primary and metastasis samples for the four sites. This analysis includes only the genes used by the DL to perform the classification. If we only considered genes common to at least three sites, we found the products of two genes, HIP1 and LARP4, with expression levels downregulated in the primary samples but upregulated in the metastasis samples. HIP1 was used by the DL to predict metastasis to the bone, brain, and lung, while LARP4 was used to predict metastasis to the brain, lung, and liver. This is interesting as HIP1 is one of the essential proteins involved in clathrin-mediated endocytosis (CME) , and croMetastasis remains the leading cause of cancer-related deaths worldwide, and our inability to identify the tumor cells colonizing distant sites means that the physician cannot treat the metastasized tumors. Here, we developed a DL model that can be fed raw gene expression data to predict whether a sample is primary or metastasized to the brain, bone, lung, or liver. The final multi-class DNN model achieved more than acceptable prediction performance in all categories. We achieved the best prediction performance for the primary samples (AUC of 0.93), followed by the metastasis to bone and lung samples (AUC of 0.88). On the other hand, the metastasis to liver and brain samples achieved lower prediction performance with an AUC of 0.84 and 0.82, respectively. We observed the same trend when evaluating the prediction performance using external data, i.e., the highest prediction performance for the primary (AUC of 0.85) samples followed by the metastasis to lung (AUC of 0.78), bone (AUC of 0.72), liver (AUC of 0.61) and brain (AUC of 0.50) samples, respectively. However, the prediction performance for the metastasis to the brain and liver samples dropped by 32% and 23 %, respectively.Many factors may contribute to the result we obtained for the brain samples, as this data had the highest number of DEGs and required the highest amount of top-ranked genes to be included in the model, indicating biological complexity associated with the metastasis to the brain. Additionally, the brain samples had less than 50 primary samples. In contrast, we analyzed the metastasis to bone, lung, and liver samples using more than 200 primary samples . Beyond that, the brain only had two samples in the test set for the external data that exhibited the massive drop in prediction performance. Having this lower number of brain samples may also be contributing to the much lower prediction performance achieved with it. Thus, in the future, we will re-establish the gene set that provides maximum performance using a larger cohort of samples (when the data become available). Nonetheless, we further evaluated the prediction performance using samples from a population-based cohort to show that the multi-class DNN model achieved good prediction performance (AUC of 0.72) when distinguishing between the primary and metastatic samples, which shows the potential of our model.p-value < 0.05) with the neuron’s list of genes that differentiate the primary samples from the metastasized ones in the DL model. The critical neurons in each layer are grouped depending on the functions enriched. Thus, the first layer exhibits more general function categories, but more specific functions appear in subsequent neural network layers. Finally, we compared the enrichments retrieved through the DL model neuron interpretations with the MSigDB enrichment analysis. We found only a few functional categories common to both analyses but several inter-related categories. For example, the literature shows “Epithelial Mesenchymal Transition’ involves ‘Ion transport,” and “Extracellular matrix disassembly,” and it is linked to “Cell-cell adhesion,” “regulation of cell motility” and “apoptosis process” . The user needs to provide the raw gene expression for every sample.We also developed a web server that the scientific community can access. The web-based tool, MetastaSite