Importantly, pathwayMultiomics is computationally efficient and does not require matched samples in multi-omics data. We performed a comprehensive simulation study to show that pathwayMultiomics significantly outperformed currently available multi-omics tools with improved power and well-controlled false-positive rates. In addition, we also analyzed real multi-omics datasets to show that pathwayMultiomics was able to recover known biology by nominating biologically meaningful pathways in complex diseases such as Alzheimer’s disease.Recent advances in technology have made multi-omics datasets increasingly available to researchers. To leverage the wealth of information in multi-omics data, a number of integrative analysis strategies have been proposed recently. However, effectively extracting biological insights from these large, complex datasets remains challenging. In particular, matched samples with multiple types of omics data measured on each sample are often required for multi-omics analysis tools, which can significantly reduce the sample size. Another challenge is that analysis techniques such as dimension reductions, which extract association signals in high dimensional datasets by estimating a few variables that explain most of the variations in the samples, are typically applied to whole-genome data, which can be computationally demanding. Here we present pathwayMultiomics, a pathway-based approach for integrative analysis of multi-omics data with categorical, continuous, or survival outcome variables. The input of pathwayMultiomics is pathway  Recent advances in technology have made multi-omics datasets increasingly available to researchers. For example, The Cancer Genome Atlas (TCGA) and the Clinical Proteomic Tumor Analysis Consortium (CPTAC) have generated comprehensive molecular profiles including genomic, epigenomic, and proteomic expressions on matched samples for many types of human tumors. The underlying hypothesis is that multiple types of molecular profiles  might provide a more coherent and complete signature of the disease process.To leverage the wealth of information in multi-omics data, a number of integrative analysis strategies have been proposed  and compBecause of the complexities in multi-omics datasets, effectively extracting biological insights from these datasets remains challenging. A major challenge for multi-omics data analysis is that the samples are often measured on one or a few, but not all, omics data types. Therefore, multi-omics analysis tools that require matched samples  as input can significantly limit the sample size when several omics data types are considered. Another challenge is that analysis techniques such as dimension reduction techniques are typically applied to genome-wide data, which can be computationally demanding. Thus, to maximally leverage information from the multi-omics datasets, there is a critical need for developing additional integrative methods that are not restricted to only matched samples and/or shared genes in the input datasets.p-values for individual omics data types, which are then integrated using a novel statistic, the MiniMax statistic, to prioritize pathways dysregulated in multiple types of omics datasets. Because pathwayMultiomics only requires summary statistics  as input, it is computationally efficient. In addition, it is also flexible and can be used to analyze multi-omics datasets with categorical, continuous, or survival outcome variables. Importantly, using summary statistics as input allows pathwayMultiomics to maximally leverage information in multi-omics datasets by not restricting to only shared samples and/or genes. Using simulated datasets, we showed that pathwayMultiomics significantly outperforms currently available multi-omics methods with improved power and well-controlled false-positive rates. In addition, we also analyzed multi-omics datasets in Alzheimer’s disease to show that pathwayMultiomics was able to recover known biology by nominating biologically meaningful pathways.Here we present pathwayMultiomics, a pathway-based approach for integrative analysis of multi-omics data. Instead of testing individual genes, pathway analysis tests joint effects of multiple genes belonging to the same biological pathway, such as those defined in the KEGG  databasehttps://github.com/TransBioInfoLab/pathwayMultiomics.p-values) can be computed. This flexibility enables pathwayMultiomics to take advantage of different pathway analysis software to model and account for special characteristics in different types of omics datasets. For example, for pathway analysis of DNAm data, the missMethyl method , copy number alterations (CNAs), or gene expressions. Of particular interest are dysregulated pathways at multiple molecular levels, for example, those with changes in both DNA methylation and gene expressions. Importantly, pathwayMultiomics is flexible; the samples can be either matched (multiple types of molecular traits are measured on the same set of samples), or un-matched (distinct samples from the same disease are measured with different types of omics technology). Moreover, because the units of analyses for pathwayMultiomics are pathways , different omics datasets can also include different genes, as long as pathway-level association statistics that relate each type of omics profiles to the phenotype , , , with maximums 0.03, 0.05, and 0.05 respectively. The MiniMax statistic is the smallest value of these maximums, which is 0.03. Intuitively, the MiniMax statistic provides a way to identify pathways with differential changes  in at least two types of omics data. Note that in this case, the MiniMax statistic is equivalent to taking the second smallest p-value among all p-values; that is, the second-order statistic, P(2), of the pathway p-values. Instead of considering pairs of p-values, the MiniMax statistic can also be computed for triplets or quadruplets of p-values from three, four, or more types of omics data similarly to identify pathways with differential changes  in more than two types of omics data.Given pathway p-values for the MiniMax statistic, pathwayMultiomics has two modes: 1) by approximation or 2) by simulation. More specifically, the “approximation” approach is based on the theory that when different types of omics data are independent, the rth order statistic p(r) of the p-values follows a Beta distribution, that is, To compute ics data . Therefop-values. These resulting p-values are our empirical null p-values. To account for non-independence in the different data types, instead of using the above formula, we estimate values for p-values. In practice, we have found that the more correlated the p-values are across the multi-omics platforms, the smaller On the other hand, in the “simulation” approach, we simulate the distribution of MiniMax statistics under the null hypothesis, that is, when there is no association between phenotype and the pathway in each type of omics data. More specifically, we generate random phenotype labels for each sample and then re-compute pathway p-values in multiple omics data types, the MiniMax statistic and significance level for each pathway, and the omics data types that were contributing to the MiniMax statistic. For example, in the apoptosis pathway example we described above, the MiniMax statistic was 0.03, its p-value  was created. Next, within each data type, data for each gene were centered and scaled. Finally, a modified Ward’s method (method = “ward.D” in hclust function) was then used to partition the genes into 50 clusters or 50 synthetic pathways. The number of genes in the resulting pathways ranged from 9 to 74, with an average of 34 genes.p = 20, 40, 60, 80%) within each pathway in each of the multi-omics datasets for samples in subtype A group. This process was then repeated 100 times to create 100 simulated multi-omics datasets, each including 50 pathways, among which 5 pathways are true positive pathways. Overall, we generated datasets for a total of 20 simulation scenarios . This benchmark dataset (available at https://zenodo.org/record/5683002#.YZF5SGDMKUk), which was systematically modified from real multi-omics data, can be used for reproducing analyses in this study as well as benchmarking future multi-omics data analysis methods.Next, we simulated treated  and un-treated  pathways. First, we randomly assigned each of the 623 subjects to one of two cancer subtypes: A or B. Next, among the 50 synthetic pathways, we selected five pathways to be our true positive pathways, and treatment effects at different levels  were added to a subset of genes . Multi-omics data was created for a total of 5,000 pathways by generating random sample labels 100 times for the 50 synthetic pathways. The false-positive rate  for each method was then estimated by the percentage of pathways p-value, test statistic, or score returned by a method), constructed ROC curves, and estimated AUC for each method.Given the known status of the pathways, we next computed the area under the ROC curve (AUC) for each method. The receiver operating characteristic (ROC) curves is a plot of sensitivity versus 1-specificity as the cutoff for declaring significant pathways is varied. AUC assesses the overall discriminative ability of the methods to determine whether a given pathway is significantly associated with the phenotype  over all possible significance cutoffs. More specifically, for each of the simulation scenarios, we recorded the rankings of the 50 pathways from most to least extreme  , MFA Dr, iProFunp-values for single omics data, we used pathwayPCA R package , optimal weights and penalties were identified by the MultiCCA.permute function.Sparse Canonical Correlation Analysis (sCCA) is a matrix factorization method that uses penalized multivariate analysis for identifying linear combinations of two groups of variables that are highly correlated. p genes are measured on all omics data types on potentially different subjects. Therefore, the statistical accuracy and power of MFA to detect multi-omics changes will not be affected by the number of matched samples, but will decrease as the number of shared features decreases, because features not shared across all data sets will be discarded. In the first step, MFA reshapes data by stacking the multi-omics datasets, each with samples as rows and the same p genes as columns. Next, MFA performs a weighted principal components analysis, where the weights from each data set are inversely related to the principal eigenvalue of the data set . Then, genes are given a score measuring its concordance across the datasets for different omics types, where the distribution of these scores follows N  where p is the number of genes measured on all omics data types. Finally, genes with upper-sided p-values The MFA method is also a matrix factorization technique, but it differs from sparse mCCA in that it only requires data to be matched on features rather than samples. For MFA analysis of multi-omics data, the main requirement is that the same set of G omics data types, and subsets the multi-omics datasets to include only these p-values or t-statistics) that associate each gene with the phenotype. This step produces a p × G data matrix . Therefore, the statistical accuracy and power of mitch to detect multi-omics changes will not be affected by the number of matched samples, but will decrease as the number of shared features decreases, because features not shared across all data sets will be discarded. Finally, for each pathway, mitch performs a one-way MANOVA to test if gene scores across the G omics data types are different for genes within the pathways compared to background genes. We compared the mitch algorithm, computed using the mitch_calc routine from the mitch R package with priority = “effect”, with two alternative gene-wise summary statistics: the gene-specific t-statistic obtained after fitting a linear model that associated each gene with subtype group effect . Note that using the t-statistic accounted for different directions of associations among genes while using the p-value did not.The mitch method is very similar to the proposed MiniMax statistic because it also computes pathway-level enrichment scores from summary statistics rather than using the data itself. There are several steps in the mitch algorithm: first, users identify the set of p-values of the predictor (CNA or DNAm) in each of the three linear models, and genes with at least one significant predictor are selected. Finally, Fisher’s Exact Test is used to identify pathways enriched with selected genes. Notably, iProFun allows more flexibility in the input dataset and can take advantage of samples not completely measured on all omics types. Specifically, iProFun requires samples to be measured by at least one genomic  trait and at least one transcriptomic  or proteomic  trait, but it does not require samples to be measured by more than one genomic trait or more than one transcriptomic/proteomic traits. In the simulation study, the number of shared samples analyzed by iProFun were 216 (copy number and RNAseq) and 88 (copy number and proteomics). The statistical accuracy and power of sparse iProFun to detect multi-omics changes will decrease as the number of these shared samples  decreases, because samples not shared by at least two data sets will be discarded. In our simulation study, we used the iProFun_permutate function in the iProFun package to independently predict synthetic gene expressions and proteomics data from simulated copy number aberrations. Default parameter values, as shown in package examples, were used for all functions.The iProFun method  aims to p-values for single omics data. Therefore, we first performed pathway analysis for genetic variants, DNAm, and gene expressions using the mixed model approach . Next, we performed GWAS pathway analysis using the mixed model approach   compared to the rest of the genes.For the analysis of RNASeq data, we analyzed 640 samples of RNAseq data measured on postmortem prefrontal cortex brain samples in the ROSMAP AD study. Normalized FPKM (Fragments Per Kilobase of transcript per Million mapped reads) gene expression values generated by the ROSMAP AD study were downloaded from the AMP-AD Knowledge Portal (Synapse ID: syn3388564). For each gene, we assessed the association between gene expression and Braak stage. More specifically, for each gene, we fitted the linear model log2  ∼ Braak stage + ageAtDeath + sex + markers for cell types. The last term, “markers for cell types,” included multiple covariate variables to adjust for the multiple types of cells in the brain samples. Specifically, we estimated expression levels of genes that are specific for the five main cell types present in the CNS: ENO2 for neurons, GFAP for astrocytes, CD68 for microglia, OLIG2 for oligodendrocytes, and CD34 for endothelial cells, and included these as variables in the above linear regression model, as was done in a previous large study of AD samples . This li (fgsea)  softwarep-values in individual omics data types are very small, at ρ = 0.0045 , −0.0263 , and 0.0432 . In pathwayMultiomics, we used the approximation approach, supported by the relatively low pairwise correlations in pathway p-values of individual omics data types.The pairwise correlations of p-values for different types of omics data. For the GWAS meta-analysis results described in , we next fit a generalized additive model using the R package gam: Yi is - log10 transformation of the smallest p-value for gene i, n. linksi is the number of SNPs associated with gene i, and f is a spline function. We assumed gamma distribution for Yi, as under the null hypothesis of no association, Yi follows the chi-square distribution . The spline model allows us to model linear and nonlinear associations between the number of SNPs mapped to a gene and the strength of significance for the gene as previously described  to adjust for the multiple types of cells in the brain samples. The -log10 transformation of the p-values for the Braak stage in the above model was then used as input for RNASeq data in mitch.For the analysis of RNAseq data, we used the R package fgsea . For eachttps://github.com/TransBioInfoLab/pathwayMultiomics_manuscript_supplement.All analyses were performed using the R software (version 4.0) and SAS software (version 9.4). We used the venny tool -2015. Top-values, either by approximation using formula or by simulation. Our results showed the estimated parameters α and β for Beta distribution based on simulation are α = 1.85 and β = 1.9, which are very similar to the theoretical values of α = 2 and β = 2 used in the approximation approach. The results in As discussed in Methods, pathwayMultiomics has two approaches for computing p-values performed better than ranking genes by t-statistic in most simulation scenarios, except the ones with weak association signals . The iProFun method also performed well in the simulated pathways that included a high proportion  of genes with large association signals . On the other hand, the sparse mCCA and MFA methods lacked power, probably because these matrix factorization techniques lost information by requiring matched samples or genes across all platforms, and their unsupervised framework also ignored phenotype information. Because sparse mCCA lacked power even in the last simulation scenario with the strongest signal (80% genes in a true positive pathway are treated with an effect size of 0.5), we only included AUC for sparse mCCA in the last simulation scenario.Among all methods, the pathwayMultiomics method performed best with the highest AUCs across all 20 simulation scenarios . The secWe next applied the two methods that performed best in our simulation study, pathwayMultiomics and mitch, to analyze a collection of real multi-omics datasets in Alzheimer’s disease, which included summary statistics for genetic variants and DNA methylation from two recent large-scale meta-analysis studies , as wellWe analyzed 2,833 canonical pathways (C2:CP collection) in MSigDB  that incp-values from individual omics, that is min{ max , max , max  }. For these significant pathways, we next examined which two omics data types contributed to the MiniMax statistics. Among the 74 pathways, the significance of the pathwayMultiomics p-value (for MiniMax statistic) was driven by pathway p-values for DNAm and RNA in the majority of pathways , followed by pathway p-values for SNP and RNA , recapitulating the prominent gene regulatory role of DNAm in AD  out of the 74 significant pathways, consistent with the relatively independent contributions of genetic variants and DNA methylations in influencing AD susceptibility , there was only modest overlap . A totalp-values for individual omics data types as input, thus making it possible to take advantage of pathway analysis tools that are specially designed for each omics data type. In addition, pathwayMultiomics is computationally efficient, does not require matched samples from multi-omics data, and is applicable in situations when raw omics data are not available, such as when aggregating summary statistics from meta-analyses related to the same disease. PathwayMultiomics is also informative; the individual omics data type that contributed to pathwayMultiomics significance can be used to distinguish pathways with potentially different underlying regulatory mechanisms, such as the pathways for which gene expressions are regulated by DNA methylation versus pathways for which gene expressions are mainly regulated by genetic variants.To identify pathways dysregulated in multiple types of omics datasets, we developed the pathwayMultiomics R package. PathwayMultiomics is flexible and only requires pathway We performed a comprehensive simulation study to assess the statistical properties of our method. To emulate correlation patterns in real omics datasets, we generated simulation datasets using real TCGA multi-omics datasets as input. We showed that pathwayMultiomics significantly outperforms currently available multi-omics methods with improved power and well-controlled false-positive rates. A challenge with analyzing multi-omics datasets is that many of the samples with data recorded for one molecular type did not have matching data from other data types. Therefore, methods that require matched samples across all data types  would only analyze a subset of the samples, which would result in reduced statistical power. Also, often only a subset of genes is measured by multiple omics platforms. Therefore, methods that require the same set of genes measured on all omics data types  may also exclude important biological signals, leading to reduced power. Finally, unsupervised methods  might also lose power because they do not leverage information in the phenotypes. In contrast, pathwayMultiomics gains power by leveraging information in all samples (including the un-matched samples), and all features  mapped to the pathways, as well as phenotype information along with multi-omics data.https://github.com/TransBioInfoLab/pathwayMultiomics.To further assess the performance of pathwayMultiomics on real datasets, we also compared it with two alternative approaches using the Venn diagram and mitch. When multiple types of omics data are available, a commonly used strategy is to test for marginal associations between each type of omics data with phenotype first, and then use Venn diagram to intersect significant pathways or genes that overlap in different omics data types. Although a good visualization tool, Venn diagrams do not provide prioritization or any statistical assessment for pathways. In addition, it might be overly stringent because when several types of omics data are considered, often few (if any) pathways pass the threshold of statistical significance in all omics data types. In contrast, pathwayMultiomics provides prioritization and statistical assessment for pathways with moderate to strong association signals in multiple omics data types. In our analysis of multi-omics AD datasets, at 5% FDR, pathwayMultiomics identified 67 pathways in addition to the seven FDR-significant pathways in more than one type of omics data as identified by the Venn diagram method. The discrepancy in multi-omics analysis results by pathwayMultiomics and mitch is not unexpected. In addition to the differences in underlying algorithms, an important reason might also be the different hypotheses these methods test. While mitch tests the competitive null hypothesis that the genes in a pathway show the same magnitude of associations with the disease phenotype compared with genes in the rest of the genome, pathwayMultiomics tests the self-contained null hypothesis that the genes in a pathway are not associated with the disease phenotype . TherefoIn summary, we have presented the pathwayMultiomics method, which can be used to analyze multi-omics data with any type of outcome variables . We have shown that pathwayMultiomics significantly outperforms currently available multi-omics methods with improved power and well-controlled false-positive rates. In addition, we also analyzed multi-omics datasets in Alzheimer’s disease to show that pathwayMultiomics was able to recover known biology, as well as nominate novel biologically meaningful pathways. We expect pathwayMultiomics to be a useful tool for integrative analysis of multiple types of omics data.


B‐Raf V600E mutations account for about half of all skin cutaneous melanoma cases, and patients with this mutation are sensitive to BRAF inhibitors. However, aberrations in other genes in the MAPK/ERK pathway may cascade a similar effect as B‐Raf V600E mutations, rendering those patients sensitive to BRAF inhibitors. We rationalized that defining a signature based on B‐Raf pathway activity may be more informative for prognosis and drug sensitivity prediction than a binary indicator such as mutation status.In this study, we defined a B‐Raf signature score using RNA‐seq data from TCGA. A higher score is shown to not only predict B‐Raf mutation status, but also predict other aberrations that could similarly activate the MAPK/ERK pathway, such as B‐Raf amplification, RAS mutation, and EGFR amplification.We showed that patients dichotomized by the median B‐Raf score is more significantly stratified than by other metrics of measuring B‐Raf aberration, such as mutation status, gene expression, and protein expression. We also demonstrated that high B‐Raf score predicts higher sensitivity to B‐Raf inhibitors SB590885 and PLX4720, as expected, but also correlated with sensitivity to drugs targeting other relevant oncogenic pathways.The BRAF signature may better help guide targeted therapy for melanoma, and such a framework can be applied to other cancers and mutations to provide more information than mutation status alone. Our BRAF score that captures BRAF V600E mutation pathway activity correlates with sensitivity to BRAF inhibitors, as well as other targeted therapies. The score is also able to predict prognosis and response to immunotherapy. We validated the efficacy of the signature in predicting BRAF V600E mutation, as well as other common genomic aberrations in the MAPK/ERK pathway. We then demonstrated the ability of the signature in predicting prognosis compared to BRAF mutation, expression, and protein levels. Lastly, we explored the correlation of the signature with various drugs and proposed mechanisms for drug sensitivity and resistance.22.1https://gdac.broadinstitute.org/). This dataset consisted of RSEM normalized gene expression data for 20,501 genes from 473 skin cutaneous melanoma samples. Of the SKCM samples, 103 are from primary tumors and 368 are from metastatic tumor samples. Besides gene expression, BRAF V600E mutation, copy number variation (CNV) data, and reverse phase protein assay (RPPA) data were used in this study. The mutations in 24,058 genes for each TCGA SKCM patient were downloaded as Mutation Annotation Format (MAF) files from Firehose. The TCGA CNV dataset was downloaded as a segmented copy number alteration (sCNA) file and contains CNV information for 23,311 genes, while the RPPA dataset was downloaded as raw protein expression data containing protein expression levels for 208 proteins and 355 samples. Both of these files were also downloaded through Firehose.The Cancer Genome Atlas (TCGA) RNA‐seq dataset for skin cutaneous melanoma (SKCM) was downloaded from Firehose  database (https://www.cancerrxgene.org) and the Cancer Cell Line Encyclopedia (CCLE) database . Mutation status information was available for the 41 skin cancer cell lines in the GDSC dataset and 62 skin cancer cell lines in the CCLE dataset. Additional information for these datasets can be found in Table An additional four external datasets were used as validation. Of the four validation datasets, one was primary bulk tumor sample data, one was single cell tumor sample data, and two were cell line data. The bulk tumor sample data were downloaded from Gene Expression Omnibus (GEO) under accession ID 2.2n = 27), and if the patient did not exhibit any mutations in the BRAF gene (including other possible mutations in the BRAF gene besides V600E), they are labeled as wild type (n = 26). To define the signature, we first log2 transformed the RSEM expression values of the genes to avoid extreme values. Each gene in the transformed gene expression profile is then used to predict the BRAF V600E mutation status  of the patient using a logistic regression model (Equation 1).We defined a weighted gene signature using primary SKCM RNA‐seq and BRAF V600E mutation status data from TCGA. Metastatic samples were left for the evaluation of its effectiveness. The BRAF mutation status was defined from the MAF files as follows: if the patient exhibited the V600E mutation in the BRAF gene, they are labeled as mutant (1 > 0) to form the BRAF signature. The weights of signature genes were determined based on the p‐value for β1 using the following formula: upw = min, 10)/10. Note the log transformed p‐values were trimmed at 10 to avoid extreme values. We have evaluated the ability of gene signatures of different sizes  for classifying BRAF‐mutant versus wild‐type melanoma samples based on cross‐validation. Our results indicated that the signature based on the top 200 most significant genes achieved the highest and most stable classification power, and was thus utilized through the rest of the study. These genes are listed in Table Note that the effect of clinical variables such as age and gender is also considered in the logistic model. Based on the model, we identified the top n genes that are most significantly upregulated  and background b(i) cumulative functions are applied to the data for both the upregulated and downregulated weights.Utilizing the top 200 most significant upregulated gene weights defined in the previous section, we calculated sample‐specific BRAF scores for patients in external gene expression datasets using a previously defined method named BASE (binding association with sorted expression).The foreground function captures the distribution influenced by highly informative genes, while the background function captures random distribution. The maximum difference between the two functions (denoted BRAF pre‐score) captures the biased distribution of BRAF signature genes in the sorted gene expression profiles for a tumor sample, similar to the enrichment score calculated in the gene set enrichment score (GSEA) analysis.Next, the BRAF pre‐score is normalized by dividing by the null distribution BRAF pre‐score, determined based on 1000 permutations. In each permutation, the same process for calculating BRAF pre‐score was performed as described above, but the foreground and background functions are calculated using a random ordering of the patient gene expression profile. The normalized score, denoted BRAF score, summarizes the relative expression levels of BRAF signature genes and therefore indicates the BRAF pathway activity in the sample. Samples with high scores are expected to have a gene expression profile similar to patients with mutated BRAF V600E, capturing alterations in the MAPK/ERK pathway that is affected by the mutation. The result is a score that quantifies BRAF V600E mutation activity in the MAPK/ERK pathway.2.4To determine whether a sample exhibited amplification (AMP), we first halved and log2 transformed the raw TCGA CNV data downloaded from Firehose. Wild‐type copy number would then be transformed to log2(2/2) =0, and anything greater would indicate an amplification of copy number. We classified patients with CNV greater than log2(2.8/2) and Ch7 CNV segment length less than 151 Mb as amplification, and all other patients as wild type. Of note, 2.8 rather than 3 is used as the threshold for gain of additional copies to protect against tumor purity. Although some driver genes like MYCN in neuroblastoma and HER2 in breast cancer are amplified with large numbers of extra copies in the genome, most genes do not have such an extreme amplification. In the TCGA SKCM dataset, the few most amplified patients have a fold change of about 1.5, which is about 4 copy gains Figure . With ou2.5We studied the relationship of various different BRAF mutations , BRAF amplification, NRAS mutation, and EGFR amplification with BRAF score in metastatic SKCM patients in the TCGA dataset. NRAS mutation status was determined from the TCGA MAF files, where 81 patients have at least one mutation in the NRAS gene (denoted as NRAS‐mutant) and 57 have no mutations in the NRAS gene (denoted as NRAS wild type). To determine the association of these aberrations with the BRAF score, we separated the samples into two groups: samples with the genomic aberrations and samples without the genomic aberrations. We then used a one‐sided Wilcoxon test to determine whether the group of patients with the genomic aberration had higher BRAF scores than the group of patients without the aberration. R package ggplot2 and pROC were used to generate the boxplots and ROC curves, respectively.2.6Data for overall survival and time to event are contained in the TCGA dataset. We constructed univariate Cox regression models to determine the association of overall survival (OS) of the patient with the following factors: BRAF score, BRAF mutation status, BRAF expression levels, and BRAF protein expression, independently. The BRAF mutation status (“Mutant” or “Wild‐type”) could be directly used to divide patients into the BRAF‐mutant group and the BRAF wild‐type group. The BRAF score, BRAF expression levels, and BRAF protein expression are continuous variables, so we used the median of each of these variables as the threshold for dividing patients into two equal‐sized groups (high and low). The Kaplan–Meier method was used to plot the survival curves. The log‐rank tests were used to determine the difference between the two survival curves and statistical significance. All survival analyses were performed with the R library “survival,” and the samples were limited to the metastatic TCGA patients.2.7Data for drug sensitivity were downloaded from GDSC database,2.8Data used to calculate tumor purity for skin cutaneous melanoma in the TCGA dataset were downloaded from Firehose. Tumor purity was then calculated using the ESTIMATE algorithm.2.9Data for immunotherapy were downloaded from dbGaP. The accession code for this clinical study is phs000452.v3.p1. This dataset contains 42 patient samples each treated with checkpoint blockade immunotherapy.33.1(p1) Additionally, KEGG pathway enrichment analysis revealed that the gene set contained genes highly enriched in melanoma and RAS signaling pathway. Furthermore, we observed that genes in the cAMP signaling pathway were enriched in our gene set, which could be explained by the well‐established cross talk between the MAPK/ERK pathway and cAMP pathway.We first defined a gene signature based on the top 200 most significantly upregulated genes for BRAF V600E mutations in primary TCGA melanoma samples. These 200 genes are listed in Table p = 4e‐13) and metastatic (p = 3e‐11) samples, indicating increased pathway expression in patients with the BRAF V600E mutation. Interestingly, BRAF scores in metastatic tumor samples appear to be overall lower than BRAF scores in primary tumor samples even though there is higher BRAF expression in metastatic mutant samples. A potential mechanism for this difference in scores is that metastatic samples tend to have lower tumor purity compared to primary samples  samples. As shown in Figure s Figure . The difp = 0.01, Figure p = 0.003, Figure p = 0.02, Figure p = 0.1). We also showed that NRAS mutants generally produced higher BRAF scores than NRAS wild type in BRAF wild‐type samples . Since these genomic aberrations activate the MAPK/ERK pathway, it is expected that they exhibit higher BRAF scores.Beyond BRAF V600E mutation, we explored the association of BRAF score with additional genomic aberrations, specifically BRAF amplification Figure , NRAS muFinally, as shown in Figure 3.2p = 8e‐04, Figure p = 0.01, Figure p > 0.1, Figure Having shown the ability of the BRAF score to recapitulate pathway activity, we next demonstrated the prognostic value of the BRAF score. TCGA melanoma samples were dichotomized into high and low BRAF score groups using the median BRAF score as the threshold. Utilizing the dichotomized scores, we constructed a univariate Cox regression model and generated survival curves. Patients with high scores exhibited significantly better prognosis compared to patients with low scores . Similar to the previous results, higher BRAF scores are significantly associated with BRAF‐mutant samples in GDSC (p = 0.001) and CCLE (p = 0.002) samples . To determine the capability of BRAF score to differentiate between BRAF mutant, NRAS mutant, and wild‐type samples, we segregated samples in the GSE81383 dataset into samples with BRAF mutation and wild‐type NRAS, samples with NRAS mutation and wild‐type BRAF, and samples with both wild‐type BRAF and NRAS , as expected. Furthermore, samples that contain NRAS mutations (but BRAF wild type) have lower BRAF scores than BRAF mutants (p = 5e‐10) but significantly higher BRAF scores than wild‐type only samples (p = 1e‐6). This observation demonstrates that both BRAF V600E mutations and NRAS mutations activate downstream pathways, but the extent in which each mutation influences activity differs. Such a result may be explained by the different effects of mutations in different NRAS loci. In fact, Murphy et al. have reported that NRAS Q61R, Q61K, and Q61L mutations cause melanoma in mice but not G12D, G13D, G13R, Q61H, or Q61P.We further demonstrated BRAF score's ability to differentiate between BRAF V600E mutants and wild‐type samples by comparing the BRAF scores between the two groups in two cell line datasets (GDSC and CCLE) and two external tumor datasets  and PLX4720  in Figure Given the availability of BRAF inhibitors and other drugs used for targeted therapy, we explored the relationship of BRAF score with various inhibitors targeting the MAPK pathway. To determine the possible mechanisms for drug sensitivity, GDSC melanoma samples were segregated based on their BRAF and RAS  mutation status. The BRAF scores of these samples were then correlated with the IC50 of the BRAF inhibitors available in the GDSC database. As expected, there was a negative correlation between BRAF score and the IC50 of BRAF inhibitors SB590885 , gemcitabine (p = 0.02), and bosutinib (p = 0.02), indicating cell line resistance to the drugs . To further examine BRAF scores ability to differentiate between CTLA4‐sensitive and insensitive patients, we constructed a ROC curve to demonstrate that BRAF score can predict patient response  by binding GDP to GTP, but NRAS normally remains GDP bound and inactive. However, mutations in NRAS causes loss of enzymatic function, resulting in GTP being constitutively bound to NRAS and perpetually activating RAF.We also explored the correlation of the BRAF score with various drugs in the GDSC dataset. The observation that higher BRAF score correlates with better sensitivity to SB590885 and PLX4720 is expected: higher BRAF score is correlated with the presence of the BRAF V600E mutation, and BRAF inhibitors are known to be effective in patients with the BRAF V600E mutation.We further found that higher BRAF score predicted worser sensitivity for tipifarnib, gemcitabine, and bosutinib, which are understood as follows. First, tipifarnib is an RAS inhibitor.Lastly, the BRAF score's clinical utility and applicability to patients in clinical trials were demonstrated in the phs000452 dataset. A higher BRAF score was found to be correlated with a worser response rate to CTLA4 immunotherapy. BRAF mutations are known to promote cancer evasiveness and enhance oncogenic activity, providing a possible mechanism of resistance to PD‐1/PD‐L1 immunotherapy.5In summary, we developed a gene signature based on BRAF V600E mutation that captures pathway activity, predicts prognosis, and correlates with sensitivity to BRAF inhibitors and other targeted therapies. Although we were able to study drug response in cell lines, this study was limited by the unavailability of open‐source BRAF inhibitor response data in clinical trials with patients. Once such data becomes available, the ability of BRAF score to predict patient response can be studied. The methodology described in this study may help guide clinical decisions in prescribing BRAF inhibitors and can be generalizable and applied to other cancers and targeted therapies.CC is responsible for the study concept and design. CC, KY, and EZ are responsible for the acquisition of data. KY and EZ are responsible for the analysis and interpretation of data. KY and EZ are responsible for the drafting and revising of the manuscript. CC is responsible for the study supervision. All authors have approved the final version of the manuscript.The authors have declared that no conflict of interest exists.This work was supported by the Cancer Prevention Research Institute of Texas (CPRIT) (RR180061 to CC) and the National Cancer Institute of the National Institutes of Health (1R21CA227996 to CC). CC is a CPRIT Scholar in Cancer Research.Fig S1‐S4Click here for additional data file.Table S1Click here for additional data file.Table S2Click here for additional data file.Table S3Click here for additional data file.Table S4Click here for additional data file.Table S5Click here for additional data file.


Currently, there is no principled way to decide whether a cluster of cells contains meaningful subpopulations that should be further resolved. Here, we present phiclust (ϕThe online version contains supplementary material available at 10.1186/s13059-021-02590-x. Unsupervised clustering methods –4 are inTo assess the quality of a clustering quantitatively and help choose optimal parameters, some measures of clustering quality and clusterability have been proposed , 7, mostclust), a new clusterability measure for scRNA-seq data that addresses some of the shortcomings of existing methods. This measure is based on the angle ϕ between vectors representing the noise-free signal and the measured, noisy signal, respectively. We consider clusterability to be the theoretically achievable agreement with the unknown ground truth clustering, for a given signal-to-noise ratio.  Importantly, our measure can estimate the level of achievable agreement without knowledge of the ground truth. High clusterability (phiclust close to 1) means that multiple phenotypic subpopulations are present and that clustering algorithms should be able to distinguish them. Low clusterability (phiclust close to 0) means that the noise is too strong for even the best possible clustering algorithm to find any clusters accurately. If phiclust equals 0, the observed variability within a cluster is consistent with random noise. Any subclusters of such a cluster still have a phiclust of 0, which prevents over-clustering of random noise. Instead of assuming a certain noise distribution or relying on a selection of informative genes, our measure can be applied to arbitrary types of random noise and includes all genes in the analysis. This is made possible by certain universal properties of random matrix theory (RMT) [Here, we present phiclust (ϕry (RMT) , which hry (RMT) , physicsry (RMT)  and recery (RMT) .Below, we will use results of RMT on the singular value decomposition (SVD) of a single-cell gene expression matrix, where rows correspond to genes and columns correspond to cells. To get an intuitive understanding of RMT, it is useful to first consider the cell-cell correlation matrix, calculated from the gene expression profiles. We start from the null hypothesis that the data does not contain any structure and is produced by a random process. In the context of single-cell transcriptomics, “structure” means multiple, distinguishable clusters of cells, or phenotypes. RMT can predict, what the correlation matrix looks like, if the entries of the gene expression matrix are samples of random variables that are independent and identically distributed. Trivially, the diagonal elements of this correlation matrix are all equal to 1. The off-diagonal elements are not exactly 0, however, despite the absence of any meaningful structure in the data. Only in the limit of measuring an infinite number of (random) genes would the off-diagonal elements become identically 0, and the correlation matrix would become the identity matrix. In that case, the only eigenvalue of the correlation matrix is 1. RMT describes the properties of a correlation matrix for a finite ratio of cells and genes. These correlation matrices are, in a sense, distributed “around” the identity matrix, which corresponds to an eigenvalue spectrum distributed around 1. Although the individual entries of the correlation matrix fluctuate from realization to realization, RMT shows that the eigenvalue spectrum is robust  and an analytical expression for it can be obtained . LikewisAs explained above, the presence of structure manifests itself as singular values above the MP distribution . Qualitatively, the magnitude of those outlying singular values corresponds to the magnitude of the differences between clusters. We can understand this relationship, if we assume that the measured gene expression matrix is the sum of a random matrix (the “noise”) and a matrix of noise-free gene expression profiles ; see Fig. phiclust is meant to help identify non-random (or deterministic) structure. At the level of a complete data set, for example of a complex tissue, clusters are typically easily discernible. However, if we zoom in on a single cluster, it is much more difficult to decide, whether the variability within that cluster corresponds to meaningful sub-structure (such as the presence of multiple phenotypes) or is consistent with random noise. Below, we will precisely define a notion of clusterability, based on the adjusted rand index, and show that it strongly correlates with phiclust. Furthermore, we will demonstrate that our measure compares favorably to the silhouette coefficient and ROGUE on simulated data and experimental data sets with known ground truth.   as a welTake, for instance, the simplest possible case of two cell types, A and B. Without any noise , expression profiles within a cell type are identical and the data can be clustered perfectly. Correspondingly, the singular vector of the expression matrix has only two different entries  and known as Bayes error rate  in the cSo far, we have assumed additive noise . Low-rank perturbation theory also makes a prediction for multiplicative noise . In that case, phiclust still scales approximately linearly with the tARI, but its dynamic range depends somewhat on the cell-to-gene ratio  . We chosTo further characterize the performance of phiclust on experimental data sets with known ground truth, we used a measurement of purified RNA from 3 cell types, mixed at different ratios  Fig. 1e1e. We no2 (coefficient of determination) of the linear regression. Since the squared singular values can also be interpreted as the amount of variance explained, we correct them by multiplying with 1 − the adjusted R2 found in the confounder regression. . Depending on the biological question, it might in fact be desirable to cluster differentiation paths, for example, to separate a stem cell state from a differentiated cell type. For other applications, it could be preferable to treat a differentiation path as one cluster. In that case, we can use pseudotime approaches  to inferk = 2, which might prompt a user to conclude (wrongly) that there are 2 sub-clusters present. Phiclust, which does not require any further partitioning of the cluster, was 0, indicating correctly that the observed variability was consistent with random noise. All in all, these comparisons indicate that phiclust is a sensitive measure, which detects differences between highly similar phenotypes.After correction for unwanted variability, we compared the performance of phiclust with other clusterability measures in the RNA mixture data set Fig. . PhiclusIn full analogy to the reasoning outlined so far, our approach can also be used to characterize variability in gene space, for which we defined the conjugate measure g-phiclust  , we calcTo confirm the presence of relevant substructure, we subclustered the two original clusters with the highest phiclust  , revealed a subset of cells with markers of urothelial cells UPKA, KRT7 UPKA, KRT7 . They could now be identified within the SSBpr cluster  Fig. , togetheFurther analysis of a cluster of interstitial cells (ICa) revealed multiple subpopulations  Therefore, regressing out uninformative, deterministic factors, is an important part of the method.The approach underlying phiclust also allows us to identify the genes that are most relevant for the biological interpretation of the observed variability. We found these genes to overlap strongly with differentially expressed genes identified after sub-clustering. The g-phiclust measure, a conjugate to phiclust, quantifies how distinguishable the expression profiles of different phenotypes are in the presence of noise.The most important application of phiclust is the nomination of clusters for sub-clustering and subsequent experimental validation. All clusters that were nominated in the fetal kidney data set turned out to have subpopulations that could be validated by experiments: rare urothelial cells, which differ from nearby clusters in only a few genes; PECs and subtypes of interstitial cells, which had distinct spatial distributions.There are several other methods that attempt to detect the presence of meaningful information in single-cell RNA-seq data. Below, we will compare phiclust to some of the most popular examples: the silhouette coefficient, ROGUE, robust PCA, the dip test, and ZINB-WaVE.The silhouette coefficient is a popular tool to assess clustering quality. In contrast to phiclust, this coefficient requires a (sub-)clustering, and it cannot be used to decide, whether a cluster contains meaningful variability and should be sub-clustered further. As demonstrated, using the silhouette coefficient can lead to over-clustering of random noise as well as missing the presence of subtly different phenotypes. Likewise, phiclust appeared to be more sensitive than ROGUE, an entropy-based clusterability measure. Both ROGUE and the silhouette coefficient do not scale linearly with the tARI, which we introduced as an upper limit to the achievable agreement of an empirical clustering with the ground truth.Robust PCA , 32 decoThe dip test , a methoZINB-WaVE  performsWe hope that this manuscript will bring renewed awareness to random noise as a factor that imposes hard limits on clustering and identification of differentially expressed genes. We hope that quantitative measures of clusterability, such as phiclust, can play an important role in making single-cell RNA-seq analysis more reproducible and robust.Before applying the method to simulated or measured single-cell RNA-seq data sets, several preprocessing steps are necessary. The raw counts are first normalized and log-transformed. Next, the expression matrix is standardized, first gene-wise, then cell-wise. These steps assure the proper agreement of the bulk of the singular value distribution with the MP distribution  have identical expression profiles in the signal matrix We apply SVD to obtain the singular values, as well as the right and left singular vectors of P a perturbation to the random matrix X, we can apply results from both random matrix theory and low-rank perturbation theory. Random matrix theory [X is a Marchenko-Pastur (MP) distribution [P. Since the agreement with the MP distribution holds strictly only for infinite matrices, we use two additional concepts to identify relevant singular values exceeding the range defined by the MP distribution. The Tracy-Widom [Considering the signal matrix x theory , 34 predribution , 18, 35,ribution –13 of \dcy-Widom , 35 (TW)θi) of P from the outlying singular values (γi) of the measured expression matrix We apply low-rank perturbation theory  to calcuc is the cell-to-gene ratio, i.e., the total number of cells divided by the total number of genes.where θi are then used to obtain the angles αi between the singular vectors of P. These angles are conveniently expressed in terms of their squared cosine asThe values of The leading singular vector of the measured expression matrix, which has the largest singular value, has the smallest angle to its unperturbed counterpart. The squared cosine of this smallest angle is then used as a measure of clusterability:For a detailed derivation of phiclust, see Additional File P and add new realizations of the noise matrix by sampling from a random distribution. The uncertainty is then obtained from the values phiclust calculated for this ensemble of sampled matrices. First, we decompose a simulated or measured expression matrix Xr and a matrix Xs that contains deterministic structure. Xs is constructed from the relevant singular vectors, which were identified as described in the previous section. Note that Xs contains noise and is thus different from the signal matrix P. To create an approximation Ps of the signal matrix P, we replace the singular values γi used in the construction of Xs with the singular values θi of P, calculated using low-rank perturbation theory as shown in the previous section. The entries of the noise matrix Xr have a mean of 0 and a standard deviation of 1, as a result of preprocessing. Since the results of RMT are valid irrespective of the particular noise distribution, we can create additional realizations of the noise matrix by sampling from a normal distribution with mean 0 and standard deviation 1. By adding sampled noise matrices to the approximated signal matrix Ps, we can create an ensemble of matrices with the same singular value spectrum as the original measured expression matrix but different realizations of the noise. The uncertainty for positive and negative deviations from the mean is then calculated as the standard deviation for at least 50 sampled matrices. See Additional file The uncertainties for the values phiclust are estimated using a sampling approach. The basic idea is to approximate the signal matrix entclass1pt{minimap values are adjusted for multiple hypothesis testing with the Benjamini-Hochberg procedure [To validate the use of the MP distribution, we test whether the bulk of the measured singular value distribution deviates significantly. Singular values are considered to be part of the bulk, if they are located below the MP upper bound and not associated with the transcriptome mode. We sample 1000 values from the MP distribution using the RMTstat R package (V 0.3) and subsequently test for similarity with the Kolmogorov-Smirnov test . The resrocedure .R2 (coefficient of determination) of this linear regression. To relate the regression result to the singular values, we consider the squared singular values  which correspond to the variance explained by the corresponding singular vectors/eigenvectors. Squared singular values are corrected by multiplication with (1 − adjusted R2) to retrieve the fraction of variance not explained by nuisance parameters. The square root of the result is the corrected singular value See also Additional file scRNA-seq data contains various confounding factors that drive uninformative variability. These either emerge from technical issues  or biological factors . To account for these factors, a regression step, inspired by current gene expression normalization methods , 25, is X with the same dimensions as the measured expression matrix P whose number of rows or columns is equal to the number of measured genes. The measured expression matrix To model multiplicative noise, we use a rectangular random noise matrix I denotes the identity matrix. Importantly, the bulk of the singular vector distribution of the measured expression matrix P are calculated from the outlying singular values of where with As before, the largest of these values is taken to be the clusterablity measure. More information on multiplicative perturbation can be found in .To construct a Bayes classifier , which aWe also tested the silhouette coefficient  as a potSeurat R package with 10 principal components (PCs) and 20 nearest neighbors. Three different resolution parameters were used: 0.1, 0.6, and 1.6. Scanpy clustering [scanpy python package with 10 PCs and 20 nearest neighbors. Three different resolution parameters were used: 0.1, 0.6, and 1.6. Hierarchical clustering [For the validation of the tARI and tSIL, several clustering methods were used on simulated data with two clusters. Seurat clustering  was perfustering  was perfustering  was perfustering  was calck = 10 . In Additional file 1: Fig. S2f, g a rank 1 signal matrix was created similarly as described above. The cell-singular vector with a number of entries matching the number of cells in the cluster was constructed as before. The gene-singular vector was drawn from a normal distribution and subsequently normalized to unit length. The rank 1 signal matrix was then added to the preprocessed expression matrix of the indicated cluster.The simulated data sets in Additional file splatter [R package (V 1.10.1). The parameters used for the simulation are shown in Table SThe remaining simulated data sets were produced with the splatter  R packagackage V .10.1. Thhttps://cf.10xgenomics.com/samples/cell/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz). For the calculation of the tARI, clustering with Scanpy, TSCAN, k-means, and hierarchical clustering, preprocessing was performed with the scanpy python package (V 1.4.6) following the provided pipeline  for the filtering of cells and genes, normalization, and log-transformation as well as cluster annotation. For the clustering with Seurat, the provided Seurat pipeline was used  for preprocessing, such as cell and gene filtering, normalization, log-transformation, and cluster annotation using the Seurat R package (V 3.1.5). CD8 T cells and B cells were extracted from the data, and each cluster was standardized gene-wise and cell-wise before the calculation of the SVD. To remove any sub-structure in these clusters and before the reconstruction of the matrices from the SVD, singular values above the MP distribution were moved into the bulk, and the transcriptome mode  ∙ cCD8 T. For cluster 2, the weights were flipped: c2 = (1 − α) ∙ cB + α ∙ cCD8 T. α was chosen in a range from 0 to 1. α close to 0.5 produced highly similar clusters, while α close to 0 or 1 produced maximally different clusters . For each value of α, the procedure was repeated 50 times, each with a different seed for selecting 300 cells per cell type, and the results were averaged.PBMC data  was downRNA-mix data  was downR package SeuratData . Normalization and calculation of the G2M score [Seurat R package (V 3.1.5). Confounder regression was performed for the log-transformed total number of transcripts, cell cycle score, and average expression of: mitochondrial genes and ribosomal genes (list obtained from the HGNC website).The bone marrow mononuclear cell data set (BMNC)  was down2M score  were perSeurat R package. Confounder regression was performed for the log-transformed total number of transcripts, G2M scores, and the average expression of: mitochondrial genes, ribosomal genes, and stress-related genes [For the fetal kidney data set , the samed genes .Seurat R package using 2000 highly variable genes (hvg), d = 50, k = 50, min.dist = 0.6, and metric = cosine. For the fetal kidney data set, a force-directed graph layout was calculated using the scanpy python package. The graph was constructed using 100 nearest neighbors, 50 PCs, and the ForceAtlas2 layout for visualization.Uniform Manifold Approximation and Projections  (UMAPs) findMarkers of the scran R package on log-transformed normalized counts. Genes with a false discovery rate below 0.05 were selected and then sorted by log2 fold change. In Figures S9e and S10e, genes with the top 20 highest/lowest values in the gene singular vectors are listed and colored blue if they correspond to the top 20 DE genes.Differentially expressed genes within the sub-clusters found in Additional file A human fetal kidney  at week 15 of gestation was used for immunofluorescence using the same procedure as reported previously . The folAdditional file 1: Supplementary figures: Fig. S1 to Fig. S11.Additional file 2: Contains further details on the mathematical derivation of phiclust.Additional file 3: Contains a more detailed interpretation of the sub-clustering results for the human fetal kidney [l kidney , 47.Additional file 4: Review history.Additional file 5: Table S1 Information on all scRNA-seq data sets and parameters for simulated data sets.Additional file 6: Table S2 Tables with information on nuisance parameters.Additional file 7: Table S3 Results of differential expression tests for BMNC data.Additional file 8: Table S4 Results of differential expression tests for kidney data.


We introduce DeepProg, a novel ensemble framework of deep-learning and machine-learning approaches that robustly predicts patient survival subtypes using multi-omics data. It identifies two optimal survival subtypes in most cancers and yields significantly better risk-stratification than other multi-omics integration methods. DeepProg is highly predictive, exemplified by two liver cancer (C-index 0.73–0.80) and five breast cancer datasets (C-index 0.68–0.73). Pan-cancer analysis associates common genomic signatures in poor survival subtypes with extracellular matrix modeling, immune deregulation, and mitosis processes. DeepProg is freely available at The online version contains supplementary material available at 10.1186/s13073-021-00930-x. Most survival-based molecular signatures are based on one single type of omics data . Since eAmong the multi-omics data integration methods, most of them do not model patient survival as the objective; rather, the survival differences associated with molecular subtypes are evaluated in a post hoc fashion . MoreoveIdentifying disease subtypes is clinically very significant. For example, it is generally accepted that cancer of a specific organ has multiple subtypes. Using molecular signatures to identify cancer subtypes allows tumor classification beyond tumor stage, grade, or tissue of origin . Cancer Here we propose a unique computational modeling framework called DeepProg, different from all methods mentioned above. It explicitly models patient survival as the objective and is predictive of new patient survival risks. DeepProg constructs a flexible ensemble of hybrid-models (a combination of deep-learning and machine learning models) and integrates their outputs following the ensemble learning paradigm. We applied DeepProg on RNA-Seq, Methylation, and miRNA data from 32 cancers in The Cancer Genome Atlas (TCGA), with a total of around 10,000 samples. DeepProg shows better predictive accuracies when compared to SNF based multi-omics integration method and the baseline Cox-PH method. The gene expression in the worst survival subtype of all cancers shares common signatures involved in biological functions such as mitotic enhancement, extracellular-matrix destabilization, or immune deregulation. Moreover, DeepProg can successfully predict the outcomes for samples from one cancer using the models built upon other cancers. In short, DeepProg is a powerful, generic, machine-learning and deep-learning based method that can be used to predict the survival subtype of an individual patient.https://tcga-data.nci.nih.gov/tcga/). We use the package TCGA-Assembler (versions 2.0.5) and write custom scripts to download RNA-Seq (UNC IlluminaHiSeq RNASeqV2), miRNA Sequencing , and DNA methylation (JHU-USC HumanMethylation450) data from the TCGA website on November 4–14, 2017. We also obtain the survival information from the portal: https://portal.gdc.cancer.gov/. We use the same preprocessing steps as detailed in our previous study [DownloadRNASeqData, DownloadmiRNASeqData, and DownloadMethylationData from TCGA-Assembler, respectively. Then, we process the data with the functions ProcessRNASeqData, ProcessmiRNASeqData, and ProcessMethylation450Data. In addition, we process the methylation data with the function CalculateSingleValueMethylationData. Finally, for each omic data type, we create a gene-by-sample data matrix in the Tabular Separated Value (TSV) format using a custom script.We obtain the 32 cancer multi-omic datasets from NCBI using TCGA portal  were downloaded from Gene Expression Omnibus (GEO). Their accession IDs are GSE4922 , GSE1456hallenge . The methallenge .s, the survival data have two features: the observation time t and the observed event (death) e. The pipeline is composed of the following unsupervised and supervised learning modules (the detail of each step is described in the subsequent paragraphs). Module 1: unsupervised subtype inference: each input matrix is processed with (a) normalization, (b) transformation using an autoencoder for each omics data type, and (c) selection of the survival-associated latent-space features from the bottle neck layer of autoencoders. The selected survival-associated latent-space features from all the omics are then combined for clustering analysis. Module 2: supervised prediction of a new sample; this module is composed of the following steps: (a) construction of a classifier using the training set, (b) selection and normalization of the common features with the new sample, and (c) prediction. For both unsupervised and supervised inferences, we use an ensemble of DeepProg models through boosting approach: each model is constructed with a random subset (80%) of the training dataset. The clustering and the prediction results are combined according to the relevance of each model.DeepProg is a semi-supervised flexible hybrid machine-learning framework that takes multiple omics data matrices and survival information as the input. For each sample n, the number of samples. For each sample, we use the sample-sample distances as new features and normalize them using the rank as well. As a result, each sample has n features with the score of the first feature equal to 1.0 and the last feature equal to 0.As default, DeepProg first selects the top 100 features from the training set that have the highest variance. Then, for each sample, we inversely rank the features and divide them by 100, so that the score is normalized between 0 and 1. Next, we compute the sample-sample Pearson correlation matrix of size To normalize a new sample (in the model prediction stage), we first select the set of common features between the new sample and the training set. We then perform the same steps as described above: (a) selection of top 100 features, (b) rank-based normalization, (c) distance computation with the samples from the training set, and (d) rank normalization.f(v) = v′’ that reconstructs the original input vector v composed of m features through multiple nonlinear transformations (size(v) = size(v′) = m). For each omic data type, we create one autoencoder with one hidden layer of size h (default 100) that corresponds to the following equation:An autoencoder is a function W’, W are two weight matrices of size h by m and m by h, and b, b’ are two bias vectors of size h and h’. tanh is a nonlinear, element-wise activation function defined asW*, W`*, b*, and b’* that minimizes the log-loss function.To train our autoencoders, we search the optimal W*, W`*, b*, and b’*. We train our autoencoder on 10 epochs and introduce 50% of dropout  at each training iteration.We use python (2.7) Keras package (1.2.2) with theano as tensor library, to build our autoencoders. We use the Adam optimization algorithm to identify scikit-optimize (https://scikit-optimize.github.io/stable/) and the tune (https://docs.ray.io/en/latest/tune.html) python libraries. The computation of the ensemble of models and/or the hyperparameters grid-search can optionally be distributed on multiple nodes and external supercomputers using the python ray framework (https://docs.ray.io/en/latest/).To help selecting the best set of hyperparameters , DeepProg has an optional hyperparameter tuning module based on Gaussian optimization and it relies on the lifelines (https://github.com/CamDavidsonPilon/lifelines) and identify those with log-rank p values (Wilcoxon test) < 0.01. We then extract all the significant new latent features from all autoencoders and combine them as a new matrix Z.For each of the transformed feature in the hidden layer, we build a univariate Cox-PH model using the python package GaussianMixture function from the scikit-learn package with 1000 iterations, 100 initiations, and a diagonal covariance matrix. The resulting clusters are sorted according to their median survival levels: the cluster labeled as “0” has the overall lowest median survival, while the last cluster “N” has the highest survival overall. Other clustering methods, K-means and dichotomized Lasso Cox-PH model, can replace the default Gaussian mixture method.The default clustering method to identify subtypes is the Gaussian mixture model-based clustering. We use the sklearn package to construct SVM models and infer the class probability with the predict_proba function, by fitting a logistic regression model on the SVM scores [We use the cluster labels obtained from the above Gaussian mixture model to build several supervised machine learning models that can classify any new sample, under the condition that they have at least a subset of features in common with those input features from the training set. First, we compute a Kruskal-Wallis test for each omic type and each feature, in order to detect the most discriminative features with respect to the cluster labels. Then, we select the 50 most discriminative features for each omic type and combine them to form a new training matrix M. We apply Support Vector Machine (SVM) algorithm to construct a predictive model using M as the input and the cluster labels as classes. To find the best hyper-parameters of the classifier, we perform a grid-search using a 5-fold cross-validation on M, with the objective to minimize the errors of the test fold. The algorithm constructs at first a classifier using all the omic types from the training samples. If a new sample shares only a subset of omics data types and a subset of the features with the training samples , then DeepProg constructs a classifier using only this subset of omics data type and features, before applying it to the new sample. We use the python M scores .To obtain a more robust model, we aggregate multiple DeepProg models constructed on a random subset of the training samples. As the default, we use 10 models with 80% of original training samples to construct all the cancer models, except for LUSC and PRAD which we use 20 models since they are more difficult to train. The aggregation of these models (per cancer) works as the following: after fitting, we eliminate those models without any new features linked to survival or having no cluster labels significantly associated with survival . For a given sample, the probability of belonging to a particular cancer subtype is the average of the probabilities given by all the remaining models. We use the class probability of the worst survival subtype to assign the final label.(a) for each pair of models, we compute the adjusted Rand Index between the two set of cluster labels (ARI) [(b) we then calculate the mean of all the pair-wise rand indexes. For each cancer model, we test different initial number of clusters . We then select the K presenting the best overall results based on silhouette score. Furthermore, we also select carefully the K that minimizes the crossovers on the Kaplan-Meier (KM) plots, when plotting the stratified patient survival groups according to the cluster labels.When fitting a model, DeepProg computes several quality metrics: the log-rank p value for a Cox-PH model using the cluster labels from all or only the hold-out samples as described above, the concordance index (C-index) , and thels (ARI) , and (b)To identify the input omics features differentially expressed between the worst survival subtype and other(s), we perform two-group  Wilcoxon rank-sum test for each feature, using the Scipy.stats package. We then select features significantly over- or under-expressed with p values< 0.001. Next, we rank the differentially expressed features among the 32 cancers. For this purpose, we construct a Cox-PH model for each cancer and each significant feature and rank the features according to their -log10 . We then normalize the ranks among these significant features between 0 and 1, where 1 is attributed to the feature with the lowest Cox-PH log-rank p value and 0 is assigned to the feature with the highest Cox-PH log-rank p value in the set. We then sum the ranks of each feature among the 32 cancers to obtain its final score.statsmodels python library model, fit_regularized function from the logit class with alpha = 1.0 for each gene with and without the tissue composition as cofounders and using the cluster labels as outcome. Prior to the regression, we scale the features to have mean = 0 and std = 1 for each features using the RobustScaler from scikit-learn. We rank the significant  coefficient for the two types of models and compared their overall similarities using the Kendall-Tau correlation measurement similar to before [We use xCell web interface  to infero before .ExecuteSNF` function from the CancerSubtypes R library (v1.16) with the default parameters and use the same number of clusters (k) for DeepProg. We also substitute the autoencoder step of the DeepProg configuration with two other matrix factorization methods: MOFA+ and MAUI, using TCGA HCC and BRCA datasets. In each alternative approach, we transform the multi-omic matrices into 100 new components, followed by the same remaining steps in DeepProg . For MOFA+ method (package MOFA), we obtain 100 features using the following parameters iterations = 500, convergence_mode=’slow’, startELBO = 1, freqELBO = 1. For MAUI , we obtain 100 latent features using the following parameters learning rate = 0.0001, epochs = 500, one hidden layer of 1100 nodes. However, none of the 100 features is significantly (P < 0.05) associated with survival in Cox-PH regression step of DeepProg workflow. Finally, we also substitute the autoencoder step by standard PCA using the scikit-learn python library.To infer clusters from SNF, we use rpy2 to call SNF from python with the `arboreto. Specifically, we use the following steps: (a) aggregating the nodes, node weights, edges, and edge weights of each cancer network into a consensus graph; (b) selecting the top 200 genes and construct its corresponding subgraph; (c) performing edge pruning on each gene by removing all but the top 10 edges, according to their weights; and (d) applying a community detection algorithm on the graph using the random-walk algorithm from the python library igraph and visualizing the graph using Gephi [For each cancer, we first identify (at most) the top 1000 RNA-Seq genes enriched in the worst subtype according to their Wilcoxon rank test p value. We then use these genes to construct a Gene Regulatory Network. For each pair of genes (nodes), we obtain an interaction score based on their correlations, and assign it to the edge between them. For each network, we normalize the scores by dividing them with the maximal value. We then score each gene with the sum of its edge scores. We combine the network obtained for the 32 cancers into a global pan-cancer co-expression network, using the GRNBoost2 algorithm from the python package ng Gephi .https://github.com/lanagarmire/DeepProg. The workflow is written in Python3 and tested under Linux, OSX, and Windows. The package contains instructions for installation and usage and the different requirements. A docker image containing all the dependencies installed is also freely available at: https://hub.docker.com/repository/docker/opoirion/deepprog_docker.The source code and documentation for the DeepProg framework is free for non-commercial use under GPL v3 license at: DeepProg is a general hybrid and flexible computational framework to predict patient survival based on one or more omics data types, such as mRNA transcriptomics, DNA methylation, and microRNA expression Fig. . The firWe applied DeepProg to analyze the multi-omics data  of 32 cancers in TCGA  algorithm , a stateWe also substituted the autoencoder step of the DeepProg configuration with a simple PCA decomposition and two matrix factorization methods including MAUI and MOFA+, using TCGA HCC and BRCA datasets. In each alternative approach, we transformed the multi-omic matrices into 100 new components, followed by the same remaining steps in DeepProg . While none of the 100 features from MAUI has a p value < 0.05 in Cox-PH filtering step, PCA and MOFA+ both have much worse performances compared to the default dimension reduction step in DeepProg Fig. A, B, comLastly, we compared DeepProg with a baseline model, where the Z-score normalized features are directly fit by the Cox-PH model with Lasso penalization without the autoencoder step. The samples are dichotomized into the same number of clusters as in DeepProg, subjective to the same parameterization whenever applicable. On the same hold-out samples across 32 cancers, DeepProg shows significantly better log-rank p values  than the baseline Cox-PH model . Upon normalizing these ranks between 0 and 1, we obtained a pan-cancer rank by summing over all 32 cancer types see the “” sectionThe RNA-Seq analysis shows some emerging patterns of over-represented genes within the poorest survival group Fig. A. CDC20 Among the genes that are under-expressed in the poorest survival groups, CBX7 and EZH1 are the top 2 genes . Two clusters (groups 4 and 5) that include multiple collagens are enriched with extracellular matrix (ECM) organization and receptors function   and smooth muscle contraction , respectively. Similar to signature gene results  and machine-learning algorithms, specifically for survival prediction. We have demonstrated several characteristics of DeepProg, including its superior predictive accuracy over other state-of-the-art methods, its robustness at predicting other HCC and BRCA population cohorts’ patient survival, as well as its suitability as a transfer learning tool trained from a relevant cancer to predict another cancer.out-of-bag samples and/or external datasets. In this study, we demonstrated the value of DeepProg in integrating 3 types of omics data: RNA-Seq, microRNA-Seq, and DNA methylation. Other specialized deep-learning models to handle mutation or pathology image data can be developed as individual modules and added to DeepProg. DeepProg performs significantly better overall than unsupervised methods such as SNF at predicting patient survival. One major reason is that SNF does not include survival information when performing integration; rather, it only relies on the patterns from multiple types of genomics data. Moreover, unsupervised methods such as SNF cannot predict the prognosis of new sample(s) like DeepProg.A few unique mechanistic features of DeepProg contribute to its accuracy. First, it uses boosting procedures that increase the robustness of the final model, by agglomerating weaker models from different subsets of the original samples. This design is well adapted to distributed computing architectures and can be scaled-up easily. Secondly, it employs a modularized design for each omic data type and can be extended to other omics and data types. In its default configuration, DeepProg first processes each omic data set individually with autoencoders, and then merges the hidden layer features under a unified Cox-PH fitting – clustering – supervised classification workflow. The autoencoder structure transforms the initial input features of various omics types into new features. Another key aspect of DeepProg is the validation of the inferred labels using either internal We further used DeepProg to identify global signatures of tumor aggressiveness among 32 types of cancers. Although previously several pan-cancer studies used one or different omic types to understand pan-cancer molecular hallmarks , 45, 46,DeepProg is a novel ensemble framework of deep-learning and machine-learning approaches that robustly predicts patient survival subtypes using multi-omics data. We anticipate that DeepProg models are informative to predict patient survival risks, in diseases such as cancers.Project name: DeepProghttps://github.com/lanagarmire/DeepProgProject home page: Operating system(s): Linux, OSX, Windows, DockerProgramming language: Python, ROther requirements: Python 3.8, Tensorflow, scikit-learn, scikit-survival, lifelines, KerasLicense: PolyForm Perimeter License 1.0.0Any restrictions to use by non-academics: license neededAdditional file 1: Table S1. DeepProg performances when using a number of models ranging from 1 to 30. The Cox-PH log-rank p value  and the C-index (CI) are calculated for two datasets: HCC and BRCA. Two validation datasets are used for HCC: LIRI and GSE, and four datasets for BRCA, named Anna, Patiwan, Miller, and Metabric datasets respectively.Additional file 2: Table S2. Summary of 32 TCGA cancer types.Additional File 3: Table S3. Cox-PH log-rank p value, Silhouette score, and clustering stability score obtained using DeepProg on the 32 cancers and with a number of input clusters from 2 to 5.Additional File 4: : Additional information including details of DeepProg model and its comparison with other methods. Figure S1. Autoencoder hidden-layer features significantly associated with survival for each omic data type and cancer type. Figure S2. Compare the performance of DeepProg vs. a simple model based on stage stratification (stage I + II vs. Stage III + IV) on 32 cancer types, on their distributions of the –log10 of stratified survival curves differences. And compare the performance of DeepProg vs. a simple model based on late stage (Stage III / IV) STAD (B-C) and COAD (D-E): DeepProg survival plots; : survival plots using stage stratification . Figure S3. Kaplan-Meier plots for each cancer type, where the survival risk group stratification is determined by SNF, with the same datasets used from the DeepProg analysis in Figure Figure S4. DeepProg has significantly better (more significant) log-rank p values compared to the baseline Cox-PH model. Figure S5. COAD and READ mutual predictability analysis using RNA as features. A) Subtypes inferred with DeepProg for COAD and used to predict READ subtypes. B) READ subtypes inferred with DeepProg and used to predict COAD subtypes.Additional File 5: Table S4. Comparison of DeepProg results using the same benchmark datasets as those obtained from the original SNF paper.Additional File 6: Table S5. Significances of survival-clusters inferred using MAUI, MOFA+, and PCA as alternatives to the autoencoder module in DeepProg, on TCGA HCC and BRCA datasets.Additional File 7. Additional analysis of microRNA and methylation signatures between the worst survival subtype vs. the remaining samples in each cancer.Additional File 8: Table S6. Top 100 features significantly linked to the lowest survival subtypes among the 32 cancers and ranked using their individual Cox-PH p value in the different datasets. The features are separated among RNA, MIR and METH features, by over- and under-expressed features.


A new logistic regression-based method to distinguish between cancerous and noncancerous RNA genomic data is developed and tested with 100% precision on 595 healthy and cancerous prostate samples. A logistic regression system is developed and trained using whole-exome sequencing data at a high-level, i.e., normalized quantification of RNAs obtained from 495 prostate cancer samples from The Cancer Genome Atlas and 100 healthy samples from the Genotype-Tissue Expression project. We could show that both sensitivity and specificity of the method in the classification of cancerous and noncancerous cells are perfectly 100%. According to the US cancer statistics report for 2020, there are estimated 191,930 new cases of prostate cancer and 33,330 deaths because of it, and the importance of early diagnosis has repeatedly been emphasized . In thisComputational techniques and tools are rapidly opening their positions in medical and pharmaceutical sciences too . Differe2In this project, we have used LGR algorithm from Sci-Kit Learn on 495 samples from The Cancer Genome Atlas (TCGA) research network and 100 samples of the Genotype-Tissue Expression (GTEx) project portal and directly fed the genome data to the machine to do heavy statistical calculations on our high dimensional data. The different parts of the method are clarified below. We use all the available data at the time of accessing the databases and have not ignored any sample.2.1The LGR is a group of statistical techniques that aim to test hypotheses or causal relationships when the dependent variable is nominal.vice versa. Generally, this value is 0.5, although it can be increased or decreased to manage the number of false positives or false negatives [Despite its name, it is not an algorithm applied in regression problems, in which continuous values are dealt with, but it is a method for classification problems, in which a binary value, i.e., either 0 or 1 is obtained. For example, a classification problem is to identify if a given tumor is malignant or benign. With the LGR, the relationship between the dependent variable, i.e., the statement to be predicted, with one or more independent variables, i.e., the set of features available for the model is determined. To do this, it uses a logistic function that determines the probability of the dependent variable. As previously mentioned, what is sought in these problems is a classification, so the probability must be translated into binary values for which a threshold value is used. If the probability values were above the threshold value, the statement is true and egatives .In supervised classification methods the input data, usually seen as p points, are viewed as a p-dimensional vector (an array or ordered list of p numbers). Then the classifiers are more or less based on similar criteria, e.g., in the Bayesian classifiers, the classifier looks for a hyper surface that maximizes the likelihood of drawing the sample, or in SVMs, it looks for a hyperplane that optimally separates the points of one class from the other, which eventually could have been previously projected to a higher dimensional space. The LGR is a generalization of logits to distinguish samples that belong to one of the two different classes; hence, it is usually called binary LGR.2.2There are wrong perceptions in the computer science community about life science data that have prevented potential achievements, for instance, one is about the number of features  such as 2.3Condition positive (P): the number of real positive cases in the dataCondition negative (N): the number of real negative cases in the dataTrue positive (TP) or hitTrue negative (TN) or correct rejectionFalse positive (FP), false alarm, or type I errorFalse negative (FN), miss, or type II errorWe have used LGR classifier also known as Logit or MaxEnt classifier from Scikit-Learn 0.23.1 with its default settings. Model evaluation produces measures to approximate a classifier’s reliability. To distinguish between cancerous and noncancerous cells, as it is a binary classification, we use accuracy, precision, specificity, sensitivity, f1 score, several averaging techniques, and receiver operating characteristic curve to evaluate the model. We, indeed, use Sci-kit Learn Metrics Classification Report that returns precision, recall, and f1 score for each of two classes. In binary classification, recall of the positive class is called “sensitivity,” and recall of the negative class is “specificity.” In what follows, the terms and derivations from confusion matrix such as accuracy, specificity, sensitivity, and f1 score are given to review and compare:Sensitivity, recall, hit rate, or true-positive rate (TPR):Specificity, selectivity, or true-negative rate (TNR):Precision or positive predictive value (PPV) is the ratio of the correctly labeled samples by our program to all labeled ones in reality.Precision can be calculated only for the positive class, i.e., class 1 that shows cancer or can be evaluated for each one of the two classes independently treating each class as it is the positive class at time, and the latter is done in Sci-kit Learn Metrics Classification Report as shown in Negative predictive value (NPV):Miss rate or false-negative rate (FNR):Fall-out or false-positive rate (FPR):False discovery rate (FDR):False omission rate (FOR):Accuracy (ACC):The harmonic mean of precision and sensitivity or f1 score (F1):As we are using Sci-kit Learn Metrics Classification Report to show the results as shown in Micro-average of recall (MIAR):Micro-average of f-score (MIAF) would be the harmonic mean of the two numbers above.Macro-average of precision (MAAP):Macro-average of recall (MAAR):Macro-average of f-score (MAAF) would be the harmonic mean of the two numbers above.Macro-average method is suitable to know how the system performs overall across different sets of data but should not be considered in any specific decision-making because it calculates metrics for each label and finds their unweighted mean, i.e., it does not take label imbalance into account, while in our case, the labels are highly imbalanced, i.e., 495 vs 100. On the contrary, micro-average is a useful tool and returns measures for decision-making especially when datasets vary in size because it calculates metrics globally by counting the total true-positives, false-negatives, and false-positives. Finally, weighted-average, according to Sci-kit Learn documentation on f1-score metrics, calculates metrics for each label and finds their average weighted by support (the number of true instances for each label). This alters “macro” to account for label imbalance; consequently, it can result in an F-score that is not between precision and recall.3Genomic variation files of 595 samples including healthy people  and cancer patients  were obtained from the GTEx Project and the TCGA online database. The binary classification results of cancerous and noncancerous samples were great because the system can detect all cancerous and noncancerous samples correctly and as seen in the classification report shown in 4The classifier did its task perfectly with no error, at least on our available data. There are yet some aspects to reflect on. Although most TCGA prostate cancer (PRAD) comprise white men’s samples, they have considered human variations to contain samples of different races and groups as well to represent the US demographic information fairly. As our method classifies all cancerous and non-cancer samples correctly using the information available in genomic variation, it can mean that the genetic signatures of cancer are detected universally without the need to consider racial or sexual differences.Our work provided a new approach in application of computers using medical data that resulted in excellent classification between cancerous and noncancerous cells of the prostate. In this work, we did not reduce the dimension of input data and left all the statistical analysis to the computer, and it could do its job very well and distinguished the cancerous samples from healthy cells almost perfectly. We even did not need to balance the number of samples of each class and it shows that the difference between two classes is so much that providing hundreds of samples enables the machine to distinguish between two categories containing 495 and 100 samples perfectly. It is also useful to consider the fact that TCGA and GTEx data are not perfect and there are several rows of missing data for some of gene quantities in some samples, yet the data provided by these two projects are fairly clean and reliable and it was enough for our classifier to be able to do its classification 100% correctly. This system is trained now to receive any new person’s RNA-seq data and recognize if the patient’s prostate is cancerous or not. The limitation of our model is that it needs future collaboration with both hospitals and well-equipped laboratories and also needs the whole genome data of samples from the organ, and the involving labs should follow the same protocols to obtain the transcriptomics data. Therefore, we cannot add training data from other sources and databases to include as many samples as we want. Fortunately, we do not need to do it because our data have been enough to train the system and achieve perfect classification ability. Furthermore, an advantage of our approach is that we have used a classic interpretable method that is based on statistics, unlike other works such as Sun et al.  who have