To identify features that are associated with prognosis is essential to precision medicine by providing clues for target identification, drug discovery. Here, we developed a tumor online prognostic analysis platform (ToPP) which integrated eight multi-omics features and clinical data from 68 cancer projects. It provides multiple approaches for customized prognostic studies, including 1) Prognostic analysis based on multi-omics features and clinical characteristics; 2) Automatic construction of prognostic model; 3) Pancancer prognostic analysis in multi-omics data; 4) Explore the impact of different levels of feature combinations on patient prognosis; 5) More sophisticated prognostic analysis according to regulatory network. ToPP provides a comprehensive source and easy-to-use interface for tumor prognosis research, with one-stop service of multi-omics, subtyping, and online prognostic modeling. The web server is freely available at   •ToPP platform integrated eight multi-omics and clinical data from 68 cancer projects•ToPP provides multi-omics combination and subgroup selection for prognostic analysis•ToPP provides automatic construction of prognostic model for public and custom data•Users can perform prognostic analysis based on regulatory network or pathways in ToPP Oncology; Public health; Microbiome Cancer is among the top fatality diseases which cause approximately 10.0 million deaths worldwide each year . PrognosRecently, with the efforts from large consortia, such as The Cancer Genome Atlas (TCGA) , InternaTo bridge the gap between the high demand for versatile prognosis analyses from clinical researchers and the big multi-omics datasets, here, we developed the tumor online prognostic analysis platform (ToPP), which provides prognostic analysis using multi-omics  data and clinical data in both the univariate and multivariate modules. Furthermore, users can build prognostic models or perform pan-cancer prognostic analysis on this platform by submitting their own data. For promoting the application in precision medicine, we also provide in-depth investigation of regulatory relationship or molecular mechanism among prognostic features, as illustrated by four sophisticated case studies on a more precise tumor subgroup stratification.The overview of ToPP is illustrated in The univariate module focuses on the impact of a single variable  on prognosis. Genetic features contain gene or protein expression changes, DNA variation, and epigenome changes. Users can select the datasets, data types, and variables to their interest. If the variable is continuous, the default patient stratification strategy is to use the median value for this variable  across all samples selected as a cutoff. In clinical research, especially in the demand of precision medicine, researchers may focus on the performance of prognostic factors in a particular subgroup, we therefore developed conditional screening tool to meet the subgroup screening. The conditional screening is based on gene mutation, methylation changes, or clinical classification. The univariate module provides a KM curve, the log-rank p-value, and hazard ratio of the two groups of patients. Meanwhile, users can make adjustments to the KM curve visualization, such as risk table, group color, unit time, median survival, and hazard ratio (HR). Furthermore, boxplot for the gene expression in tumor and normal groups can be drawn if the user sets the radio named differential expression to “yes”.Multivariate module provides prognosis analysis for multiple variables, such as for a given gene list. The result for multivariate module contains a table with five columns: gene list, univariate analysis log-rank p-value and hazard ratio, and multivariate analysis log-rank p-value and hazard ratio for each gene, so that users can screen out the independent prognostic factors. In addition, it also provides the function of prognostic modeling, which includes model building, model evaluation, model diagnosis, and model validation. With the input of a gene list, the selection of training set and verification set, ToPP will automatically build the model by default parameters which are set by common processes. After checking the performance of the model in the training set and validation set  and the results of the model diagnosis , users can adjust the model accordingly by further variable selection, nonlinear transformation. Users can also consider the interaction of covariates or time-dependent covariates.Pan-cancer module was designed to investigate the prognostic effects of factors in a variety of tumors. All data were acquired from the Pan-Cancer Atlas  with uniCombination module allows users to input multiple levels of data for two genes separately. It allows for more precise stratification of patients. For example, given a condition of mutation of one gene, researchers may want to know whether the mutation or expression of its regulated gene will have an impact on the patient's survival. Through the combination module, the samples would be subdivided into four subgroups; each subgroup has more clear molecular characteristics. By analyzing whether there are significant differences in survival among the four subgroups, users can carry out molecular subtyping of patients.Data upload module allows users to upload their own data with survival time and status to conduct all kinds of analysis in ToPP. Also, users are advised to set the permission limit for their own data and use an e-mail for contact if necessary.Our previous study has shown lipid metabolism in cancer shows significant prognosis effects . PPAR siPoly (ADP-ribose) polymerase 3 (PARP3) is the third member of the PARP family and it can accelerate the repair of chromosomal DNA single-strand breaks . ResearcOwing to the complexity and heterogeneity of diseases, it remains a challenge to accurately predict the prognosis of all patients using a single marker. Therefore, it is desirable to combine different signatures to construct a versatile prognostic model to predict outcome. However, building a stable prognostic model requires sophisticated statistical skills and profound medical knowledge which may be cumbersome for researchers. Here, we illustrate a case study to build a prognostic model automatically with a set of gene signatures to predict prognosis for one cancer type using ToPP. Let’s take liver cancer as an example. ToPP provides two independent liver cancer cohorts with gene expression data and complete clinical follow-up data (OS), one cohort with 365 samples obtained from TCGA, and the other cohort with 232 samples from the LIRI-JP project downloaded from ICGC Data Portal . We usedTo prove the effect of our automatically generated prognosis model, we validated the model by the following analyses. The C-index with 1000 bootstrap replications in the training cohort (TCGA-LIHC) was 0.724 and 0.752 in validation cohort (LIRI-JP). We stratified the samples into two groups (high-risk or low-risk) according to the median value of the < 0.0001 A. Beside< 0.0001 B. A nomo< 0.0001 D and theThe featured combination module in ToPP can help us to have better stratification of patients and understand the prognostic effects. As an example, here we used the mutation of isocitrate dehydrogenase 1(IDH1) as a condition for patient stratification and prognosis analysis. IDH1 is one of the most commonly mutated genes in diffuse lower-grade gliomas (LGGs), and it is significantly related to the prognosis of LGG . IDH1/2 ToPP is an interactive web application for survival analysis for multi-omics data  of 68 cancer projects from the TCGA, CPTAC, and ICGC Data Portal. It collects comprehensive, standardized data. Therefore, it is at the first place in an affluent data repository for versatile cancer prognostic features. Genes or proteins can be searched in ToPP to check on its prognostic value in multiple published datasets.ToPP provides flexible and comprehensive prognosis analysis modules, such as univariate, multivariate, combined, patient subgroup stratification survival analysis, self-designed prognostic model construction, and pan-cancer prognosis analysis module. Also, users can upload their own data for prognostic modeling and analysis.All the data resources and analysis functions are provided in a friendly and simple interface for the convenient application to experimental cancer researchers. For more advanced computational biologists, the scripts in our GitHub account with full resources are also provided.There are also some limitations in ToPP. It only contains relatively common types of tumors, which may not be suitable for some specific types of tumors or other types of diseases. The clinical subtype annotation information of some tumors is not comprehensive enough, such as liver cancer or colon adenocarcinoma and so on. The molecular subgroup does not include the mutation sites, alternative splicing, or integrated signatures such as tumor mutation burden, which we notice are being increasingly studied in cancer prognosis. The analytical conclusions obtained by our platform are all data-driven which may lack more biological explanations and experiments to validate the results. We will keep expanding the data repository and function panels of ToPP to make it a sustainable resource for cancer researchers.luxiex2017@outlook.com).Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Lu Xie (This study did not generate new unique reagents.•http://www.biostatistics.online/topp/index.php).All the public data including multi-omics data and clinical data for all the samples can be obtained from the ToPP web server  and ICGC data portal while gene fusion datasets were obtained from TumorFusions and clinical data for multiple tumor types from TCGA, ICGA and CPTAC projects were integrated in the ToPP platform  and S1. orFusions. The TraorFusions. ProteomorFusions, Colon aorFusions, OvarianorFusions and RectorFusions were extorFusions. EpigenoorFusions vital storFusions . Here we only kept the information of CNV for type gain and loss. Somatic mutation data were normalized into 1 and 0, where 1 represents the gene with at least one site mutated, and 0 represents no mutation in this gene. Similarly, gene fusion data was normalized to 1 or 0. Here, 1 indicates the observation of a fusion event in this gene and 0 is the opposite. The methylation value of a gene was determined by the mean beta value of all CpG sites in this gene, and we defined beta values greater than 0.8 as hypermethylation and lower than 0.2 as hypomethylation by default. Users can also customize the thresholds of beta values by selecting hypermethylation or hypomethylation sites in survival analysis. For all of the above data types, we only kept tumor samples. For other data types, such as expression data, we kept both tumor and normal samples.With the advance of precision medicine, developing drugs that target a subset of patients with particular features contributing to poor survival can be beneficial to these patients. The function of “subgroup selection” in ToPP is aimed to meet this demand. It mainly includes molecular subgrouping and clinical subgrouping. Molecular subgrouping function allows users to classify mutation subgroups, CNV subgroups and methylation subgroups according to one or more genes to the user’s interest. Clinical subgrouping includes not only the traditional classification such as gender, race or tumor stage, but also the molecular label subgroups, which have been widely applied in clinic, such as the HER2 positive or negative subgroup in breast cancer. Also, the distribution of the clinical features for each molecular stratified group will be show to help identify any potential bias due to some clinical confounding variables.Univariate analysis was performed with log rank test. For continuous variables such as gene or protein expression, patients can be divided into two groups according to the median value of the variable, quantile or the ‘best-cut’ value. Here, we calculated the log rank p values for the filtering of each sample and kept at least 10% of samples for each group, and the cut point with the lowest p value was represented as the “best-cut”. For categorical variables such as somatic mutation, CNV or gene fusion, patients were divided into two groups by the binarized classification. Kaplan-Meier (KM) survival curves were theCombination analysis is to analyze the prognosis effects by considering two features. These two features can be in the same level of data such as the expression of two genes, or they can be in different levels. For instance, if a researcher wants to know the prognostic impact of gene A expression in combination with gene B mutant. Then, all the patients were divided into four groups  according to the threshold of two features, and then log-rank test was performed to test whether there was significant difference between any of two groups, separately. Co-inactivation of one gene pair which results in better survival may provide evidence for synthetic lethality interaction prediction.Cox regression  is used for multivariate analysis. When coIn order to reduce the impact of tissue specificity on the data, we used a specially standardized dataset named the Pan-Cancer Atlas, which hThe ToPP web interface was developed using HTML5 and PHP script (version 7.0.12) in the Bootstrap framework. JavaScript and jQuery were also used to perform dynamic web services with js library such as DataTables.js and select2.js. The database was implemented in MySQL (version 5.7.17) and deployed in Apache web server running on the CentOS 6.5 system. Data preprocessing and data analyses were performed by the R scripts (version 3.5.0) with several R packages . Wilcoxon rank sum test was used to compare two groups, and p < 0.05 was considered as significant difference. Significance levels p < 0.05, p < 0.01 and p < 0.001 are noted using asterisks ∗, ∗∗, and ∗∗∗, respectively.


Spatially resolved transcriptomics (SRT) technology enables us to gain novel insights into tissue architecture and cell development, especially in tumors. However, lacking computational exploitation of biological contexts and multi-view features severely hinders the elucidation of tissue heterogeneity. Here, we propose stMVC, a multi-view graph collaborative-learning model that integrates histology, gene expression, spatial location, and biological contexts in analyzing SRT data by attention. Specifically, stMVC adopting semi-supervised graph attention autoencoder separately learns view-specific representations of histological-similarity-graph or spatial-location-graph, and then simultaneously integrates two-view graphs for robust representations through attention under semi-supervision of biological contexts. stMVC outperforms other tools in detecting tissue structure, inferring trajectory relationships, and denoising on benchmark slices of human cortex. Particularly, stMVC identifies disease-related cell-states and their transition cell-states in breast cancer study, which are further validated by the functional and survival analysis of independent clinical data. Those results demonstrate clinical and prognostic applications from SRT data. Multi-view graph approaches could enhance the analysis of tissue heterogeneity in spatial transcriptomics. Here, the authors develop the Spatial Transcriptomics data analysis by Multiple View Collaborative-learning - stMVC - framework, and apply it to detect spatial domains and cell states in brain and tumor tissues. These resulting multiple types of profiles: histology, spatial location, and gene expression, provide novel insights into the organization of cells and developmental biology, especially for the evolution of the tumor7. However, SRT data analysis for biology discovery remains challenging due to its low throughput, little sensitive, much sparse, and noisy9.The recent technological innovation of SRT platform, including sequencing-based technology  and imaging-based technology 8. For example, Giotto uses a similar processing strategy to single-cell RNA-seq (scRNA-seq), for feature selection, dimension reduction, and unsupervised clustering10. BayesSpace utilizes a fully Bayesian statistical method to enhance the spatial measurement via spatial neighborhood structure for clustering analysis9. SpaGCN adopting a graph convolutional network (GCN) approach integrates gene expression, spatial location, and histology to identify spatial domains and spatially variable genes (SVGs)11. stLearn integrates features of histology with spatial location to normalize gene expression data, and followed by clustering12. Squidpy brings together omics and image analysis tools to enable scalable description of spatial transcriptomics and proteomics data13. ClusterMap incorporates physical location and gene identity of RNAs to identify biologically meaningful structures from image-based in situ transcriptomics data14. DR-SC15 and SC-MEB16 utilizing latent hidden Markov random field model integrates gene expression and spatial location for spatial clustering. STAGATE17 combines gene expression and spatial information to detect spatial domains via graph attention auto-encoder framework. While these methods have many interesting findings, the lack of visual features that can be effectively and globally extracted from histology, efficient multi-view information fusion, and the biological contexts such as global positional information within a tissue, limits their disentangling capabilities in developmental biology.Recently, several computational methods have been designed to analyze SRT data18 and SRT data 11, however, these methods usually study networks with a single type of proximity between nodes, namely single-view network. Although SpaGCN proposes an RGB color space averaging strategy to convert histological data into the same measurement space as 2D space in the tissue slice as a third dimension before calculating the similarity between any two spots, to a certain extent, this strategy discards the texture features in each spot, i.e., the strategy extracts the color features from color space without fully utilizing the spatial distribution of gray tone variations within a specified area namely texture features19. Besides, in SRT studies, 21. More importantly, the quality of the information in different views may be different, hence, it would be preferable that one novel model can learn the representations for each view by GAT, and meanwhile collaboratively integrate multiple networks to learn robust representations by automatically training the weights of different views22.On the other hand, GCN-based models have appeared as powerful tools to learn the representations of scRNA-seq data 8; (ii) the determination of each cell type (or cell-state) to which each cell belongs, is related to its size, shape, and arrangement , hence the texture data of histology has rich information to characterize cell type or cell-state23; and (iii) the colors of the antibodies on the immunofluorescence staining of the tumor sample can roughly mark the tumor position in the tissue, yielding region segmentation indicating biological contexts related to the tumor development.We reason that (i) cells belonging to the same cell type but distributed in different areas and interacting with different cell types in the tissue may have different cell-statesSpatial Transcriptomics data analysis by Multiple View Collaborative-learning), a framework that integrates four-layer information to elucidate tissue heterogeneity by attention-based multi-view graph collaborative learning, i.e., histology, gene expression data, spatial location, and region segmentation  indicating biological contexts. The features of stMVC are as follows: (i) for each spot, globally learning efficient visual features while removing artifacts from histology by data augmentations and contrastive learning; (ii) learning robust representations for each spot by training the weights of multi-view graphs including histological-similarity-graph (HSG) by visual features and spatial-location-graph (SLG) by physical coordinates, via attention-based collaborative learning strategy, under semi-supervision of region segmentation; (iii) for samples of human ovarian endometrial adenocarcinoma (OEAD) and breast invasive ductal carcinoma (IDC), identifying cancer-related cell-states  missed by competing methods, and also transition cell-states, which are further validated by clinical data from other independent studies, demonstrating potential clinical and prognostic applications from SRT data; and (iv) for sample of mouse primary visual cortex, enabling us to detect layer-specific inhibitory neurons. In particular, such a multi-view graph collaborative learning method is a flexible framework that is able to integrate not only SRT data from multi-sources or spatial multi-omics data but also spatial epigenomics or proteomics data.In this work, we introduce stMVC (24 (an efficient computer vision framework) that was trained by maximizing the agreement between differently augmented views of the same histological spot image, and then constructed HSG based on stMVC collaboratively integrates histological image (AE) see , and theion Fig. . Hence, sis Fig. .Fig. 1Ov25) was used to evaluate the quality of the visual features by our ResNet-50 model trained by histological images; and (iv) the low-dimensional representations encoded from gene expression data, via AE or semi-AE extended from AE through adding a classifier on bottleneck layer  the mean of all view-specific representations was used as a comparison to assess the efficiency of the attention-based multi-view integration strategy enabling the network to focus on the key features for characterizing tissue structure, which is named stMVC-M; (ii) the single-view graph representations by semi-supervised graph attention autoencoder (SGATE) for either SLG (SGATE-SLG) or HSG (SGATE-HSG) were utilized to assess the effectiveness of the robust representations by multi-view graphs; (iii) the naïve HSG (SGATE-HSG-N) constructed based on the visual features by the ResNet-50 model 29 by calculating the similarity of the transcriptome and multi-view joint features between spots within each predicted cell cluster, respectively, and then visualized these low-dimensional features by mapping them into two uniform manifold approximation and projection (UMAP) spaces.The most important feature of the stMVC model is that the learned low-dimensional representations can reveal the trajectory relationship between different spatial domains through semi-supervised biological contexts. To assess the performance of stMVC, we applied it to process 12 slices of the human dorsolateral prefrontal cortex (DLPFC) dataset, each of which was manually annotated with six layers and white matter (WM), and the spatial adjacency and chronological order among these layers are WM → Layer6 → Layer5 → Layer4 → Layer3 → Layer2 → Layer1 Fig. 26,27. ByIn summary Fig. –4, we foPCP4, PVALB, ENC1, CCK, KRT17, and MOBP26 are more specifically expressed in their corresponding spatial location on slice 15,1673, compared with those by BayesSpace and STAGATE  scores based on how far the expression distribution of layer-specific genes deviated from a totally equal distribution, where the higher the GI score, the more layer-specific the distribution of gene expression . Overall, we observed that stMVC, stMVC-M, and SGATE-SLG have comparable performance, and those three methods perform better than BayesSpace, STAGATE and the two HSG-based models, while stLearn performs worst. For instance, some known layer marker genes such as 9  and breast cancer  publicly published by 10X Genomics. We separately annotated 18 and 16 regions for ovarian and breast cancer based on our segmentation strategy detailed in Manual region segmentation, where for breast cancer, 15 tumor regions were classified into three different types: invasive carcinoma, carcinoma in situ, and benign hyperplasia by the previous study9 Fig.  and 8a. We found that stMVC facilitates detecting more domains enriched for the cancer regions, compared to those by all other computational methods Fig. , c and 930, by evaluating whether or not the average expression levels of the signature genes between two domains are different or not correlated 34, and NRF2 pathway for tumor adaptation to microenvironmental hypoxia35; domain 12 cells show overexpression of matrix metalloproteinases genes36, and TGF-beta secreted by infiltrating macrophages37, for promoting ovarian cancer metastasis and migration; and domain 13 cells are mediated by VEGFA/VEGFR2 signaling pathway serving a vital function in the angiogenesis of ovarian cancer;38 and (iii) the expression of signature genes of domains 10  the domains 11 and 12 cells are enriched in different cell populations that are not clearly separated in the scRNA-seq data of 4081 epithelial cells of one additional ovarian cancer sample from previous researchCD14), dendritic cell (HLA-DRB1), and T cell (CD4 and CD8A); domains 11, 15, and 16 are mostly surrounded by plasma B cell (IGCL2 and IGHG3) and two types of endothelial cell  defined by our previous study;41 and domain 15 cells are also surrounded by myofibroblast (MYLK) and follicular B cell (MS4A1).In addition to infiltrating stromal and immune cells, we found that distinct cell-states are surrounded by different microenvironments. For example, among the five domains in the region of carcinoma in situ of breast cancer Fig. , we noteTP53 and BRCA2)42 and lower-expression of oncogenes 43 are mainly enriched in domains 4, 5, 8, and 9, and normal and cancer cells are separated at the UMAP space . For example, in ovarian cancer, we observed that normal cells with over-expression of suppressor genes  predicts a possible trajectory between three domains, i.e., domain 6 → domain 10 → domain 3, where the trajectory is also detected by DR-SC, Squidpy and stLearn, and the inferred trajectories by those four computational methods can be further validated by trajectory algorithm PAGA44; and (iii) similar to stMVC, STAGATE can detect domain 10 cells, however, the trajectory detected by it is very distinct from that by other methods, i.e., there is no direct trajectory between domains 10 and 3 2 and estrogen receptor (ER) throughout the tumor regions and minimal expression of progesterone receptor (PR) for the data from stMVC, which is in line with the clinical report of Luminal B  there is a trajectory between three clusters identified from independent scRNA-seq data of 2352 epithelial cells of CID4067  four domains have different functions, i.e., domain 6 cells show the high expression levels of genes such as s50 Fig. a and 13;d 3 Fig. ; and  from raw fluorescence data based on our segmentation strategy  1020-gene dataset from the STARmap platformegy Fig. , randomlSlc17a7) and inhibitory (Gad1) neurons are distributed in the L2/3, L4, L5, and L6 canonical layers  each cortex layer is assigned almost by different feature embeddings from stMVC, compared to ClusterMap, and the excitatory  stly3 Fig. . Howeverers Fig. , which iIn this work, we proposed stMVC for analyzing SRT data to disentangle the heterogeneity of tissue, especially for the tumor, which integrates four-layer profiles: histological image data, spatial location, gene expression, and region segmentation  indicating biological contexts, by attention-based multi-view graph collaborative learning. Such tumor position information in the tissue structure used by stMVC can help us to elucidate intratumoral heterogeneity. Different from previous methods that integrate histology and spatial location data by a user-defined weight, for example, SpaGCN manually adjusts the weight of histology in gene expression smoothing, stMVC adopts an attention-based strategy to automatically learn weights of different views for robust representations. Besides, our feature extraction framework from the ResNet-50 model trained for histological image data by data augmentations and contrastive learning did help stMVC to learn more efficient visual features, compared with those from the pre-trained ResNet-50 mode by ImageNet, which was used by stLearn  regarding histology without rich texture information, such as DLPFC and ovarian cancer samples, stMVC-M is more vulnerable to the noise signals from the histological visual features while stMVC is easier to capture the finer structure by automatically learning the weight of each of multiple graphs; (ii) regarding histology with rich tissue anatomical structure, like breast cancer sample, both models have a similar result. Hence, we also implemented the mean-based strategy in stMVC model as an option for users to select.55. However, we believed that (i) the visual feature extraction model by data augmentations and contrastive learning provides a solution to construct the association of spots between different samples; and (ii) the multi-view graph collaborative learning model can provide a novel perspective to integrate multiple SRT datasets by combining multi-layer profiles data. Besides, stMVC is easily scalable to process spatially resolved chromatin accessibility (ATAC-seq) or proteomics data57, by replacing the feature matrix from gene expression data with that from ATAC-seq or proteomics data. Furthermore, with the advance of spatial multi-omics technology58, stMVC can be easily adjusted to adapt by either adding more graphs created by different omics data or substituting the feature matrix from single-omics data with that from multi-omics data fusion60. Similar to single-view GAT models such as STAGATE, stMVC can be applied to analyze the SRT data of other sequencing-based technologies such as Slide-seq61 and Slide-seqV262. Additionally, by exploiting spatiotemporal information derived from stMVC, we can calculate spatial (dynamic) network biomarkers66 or metabolic states68 for accurately and reliably quantifying biological systems and further predicting their complex dynamics/behaviors.As of now, developing models for integrating SRT data from multiple samples are facing several challenges, for example, the batch effect of gene expression data from multiple sources, as well as their sparsity and noise; constructing the association between spots from different physical metric spaces; and removing the artifacts of histology while creating the relationship between multiple samplesWe benchmarked the running time of stMVC on the simulated datasets by subsampling spots from the human DLPFC datasets. We observed that stMVC is fast, and takes 38 min to process the SRT dataset with 20 K spots. In particular, the running time is approximately linearly related to the number of input spots  compared with the visual features extraction framework from the ResNet-50 model pre-trained by the ImageNet, the preprocessed step for training SimCLR needs more computational resources and times; and (ii) region segmentation for tumor position is manually annotated based on the staining density of antibodies. With the exploration of deep learning frameworks of generalizable segmentation tools71, and then constructed HSG based on the similarity of the learned visual features  Fig. ; (2) sepion Fig. .71.For each image 24 as a base encoder We adopted the ResNet-50 frameworkWe used a Multilayer Perceptron (MLP) with one hidden layer to obtain latent features Given a set To remove the noise from histology staining and extract morphological information from histological image data, we adopted SimCLR model to efficiently learn visual features for each spot by maximizing agreement between differently augmented views of the same spot image via a contrastive loss in the latent space Fig. . For a sFor each SRT data, we randomly selected 85% of the spots as the training sets to train the SimCLR model, and the remaining 15% of the spots as the testing set to test it. Adam optimizer with both a 72, and meanwhile collaboratively integrated them to learn robust representations for each spot  optimized the parameters of each SGATE model to learn the view-specific representations based on Eq. , as well the Eq. , and fol via Eq. ; and  ovarian cancer sample with matching gene expression (3493 spots) and immunofluorescence staining with an anti-human CD45 antibody and DAPI; and (iii) breast cancer sample with paired gene expression data (4727 spots) and immunofluorescence staining with an anti-human CD3 antibody and DAPI. In addition, a mouse primary visual cortex (V1) 1020-gene sample with matching gene expression (1365 cells predicted by ClusterMap) and raw immunofluorescence staining with DAPI was available from STARmap platform3.In our study, human DLPFC, ovarian, and breast cancer datasets with paired gene expression and histology were publicly available from the 10X Genomics website. Specifically, (i) DLPFC dataset with 12 slices each is manually annotated with the six cortical layers and WM by the previous study, where the number of spots ranging from 3460 to 4789 with a median of 384430. Besides, 24,489 epithelial cells were retrieved from 20 breast cancer patients from previous research, including five luminal B, three Luminal A, three HER2+, seven triple-negative breast cancers (TNBCs), one HER2+ and ER+, and one normal75, to support our findings, and the subtypes are determined by their clinical reports and predictions. Furthermore, the bulk RNA-seq and clinical data of breast cancer in TCGA database were downloaded from the Xena platform for survival analysis76, and the PAM50 subtype classifications of breast cancer of TCGA were downloaded from previous research77.To validate our predictions for the sample of ovarian cancer, we extracted 4081 epithelial cells from one ovarian cancer patient from a previous study78 for each gene expression data from the Visium platform, and all 1020 genes of the mouse primary visual cortex dataset from the STARmap platform, were used to comprehensively compare each computational method. Besides, to efficiently capture the information within gene expression data, we mapped each data into 50-dimensional latent features based on our designed autoencoder-based framework  to which each spot  the colors of the antibodies on the immunofluorescence staining of the tumor histology can roughly define the tumor region; (2) cells belonging to the same cell type but are separated by other cells such as immune or other mesenchymal cells may have different cell-states; and (3) the majority of cells in the tumor region are tumors while a few are normal epithelial and infiltrating stromal or immune cells, and the tumor purity was larger than 70%, which was estimated by over 9300 tumors of 21 cancer types from the TCGA databaseBy setting In addition, we further extended the protocol to process imaging-based SRT data. Specifically, we predicted RNA clusters for each cell by ClusterMap; for each cell pted Eq.  to predipted Eq. .60 to quantify the quality of denoised gene expression data by estimating a degree of inequality in the distribution of known layer-specific gene expression levels. Specifically, for each domain of the ground truth, the average expression of each marker gene was calculated; and then the GI for each gene calculated by the ‘gini’ function from reldist package was used to evaluate the specific level of the marker gene82. The higher the GI score, the better the denoised data. The marker genes used in DLPFC dataset were downloaded from the previous study26 .To clarify that stMVC is able to distinguish normal cells from cancer cells by integrating the histological features, we further designed a statistical-based measure to detect normal cells from SRT data. Specifically, we reasoned that (1) normal cells tend to be with a higher expression of tumor suppressor genes while cancer cells tend to have a higher expression of oncogenes; and (2) the expression levels between the tumor suppressor genes and the oncogenes are different or not correlated, which were calculated by Fisher’s exact test see . Hence, After applying stMVC in analyzing SRT data, we learned the accurate low-dimensional representations (We utilized the UMAP algorithm to map the low-dimensional features from each computational method to two-dimension, visualized the distance of cell embeddings between different cell populations by ‘Dimplot’ function, and visualized the clusterings and gene expression patterns at the spatial level by ‘SpatialDimPlot’ and ‘SpatialFeaturePlot’ function, respectively.28, an entropy-based statistic to quantify the homogeneity of identified cell clusters based on transcriptome similarity between spots within each cluster, and the higher the value, the better the clustering; silhouette width29, a measure of how similar a spot is to its predicted cluster compared to other clusters, and the higher the value, the spot well belongs to its cluster, which is calculated as follows:We adopted two different metrics to evaluate the clustering by calculating the similarity of features between spots within the predicted cell clusters. Specifically, ROGUEWe constructed KNN graph for each spot based on the learned low-dimensional representations (Further information on research design is available in the Supplementary InformationReporting Summary


Modern oncology offers a wide range of treatments and therefore choosing the best option for particular patient is very important for optimal outcome. Multi-omics profiling in combination with AI-based predictive models have great potential for streamlining these treatment decisions. However, these encouraging developments continue to be hampered by very high dimensionality of the datasets in combination with insufficiently large numbers of annotated samples. Here we proposed a novel deep learning-based method to predict patient-specific anticancer drug response from three types of multi-omics data. The proposed DeepInsight-3D approach relies on structured data-to-image conversion that then allows use of convolutional neural networks, which are particularly robust to high dimensionality of the inputs while retaining capabilities to model highly complex relationships between variables. Of particular note, we demonstrate that in this formalism additional channels of an image can be effectively used to accommodate data from different omics layers while implicitly encoding the connection between them. DeepInsight-3D was able to outperform other state-of-the-art methods applied to this task. The proposed improvements can facilitate the development of better personalized treatment strategies for different cancers in the future. The risks of side effects can be reduced by improving the prediction rate of drug response from targeted therapy which would undoubtedly improve patients’ treatment. In this respect, drug prediction on in vivo datasets is a step towards clinical applicability. However, since in vivo data like The Cancer Genome Atlas (TCGA) repository (https://www.cancer.gov/tcga) has a scarcity of patient records and drug responses, it is difficult to train a model on in vivo datasets. Considering this challenge, Sharifi-Noghabi et al.2 trained their computational model on in vitro data and obtained predictability on in vivo data. In vitro projects have compiled datasets such as Genomics of Drug Sensitivity in Cancer (GDSC)3 and Cancer Cell Line Encyclopedia (CCLE)4. These datasets consist of multi-omics profiles such as gene expression, copy number alteration (CNA) and somatic mutations. Although gene expression datasets have shown to be very useful5, adding more omic layers could improve the predictability of pan-cancer models2. Nevertheless, the platforms of these datasets are different. Furthermore, there exists a scarcity of related samples in GDSC datasets. This augments the problem of adequately estimating a model on the training set of the data.Precision oncology is rapidly developing. However, only a very small percentage of patients can currently take advantage of it6. To position this work among previous efforts, the field of anticancer drug response prediction can be viewed from the perspective of either the types of data used or the methodology applied. From the data-centric perspective, our method attempts to leverage patient multi-omics data, and as such, is closest to the following two approaches. In2, the MOLI method is proposed and learned on datasets from GDSC. It predicts response to a drug from TCGA and patient-derived xenograft (PDX)7 datasets. These collated datasets have multi-omics profiles: gene expression, CNA, and somatic mutation. The MOLI method reported the area under the curve (AUC) on seven datasets from TCGA and PDX. Their average AUC over the seven datasets was 0.63. Park et al.8 also constructed training and test dataset from GDSC and TCGA/PDX resulting in more than seven datasets and proposed the Super.FELT method. The performance of their method also included the seven datasets used in2, though some of the resulting test samples slightly differed due to methodology. Nonetheless, their average AUC over the seven datasets was reported to be 0.68. When we reimplemented the Super.FELT method on the MOLI test sets, an average AUC of 0.65 was obtained.The application of modern machine learning (ML) methods to problems in biomarker discovery has allowed better prediction of response to treatment. Full scope of these advancements is too extensive to cover here, and the following recent review provides a more complete overview of proposed approaches9. However, OmicsMapNet method was limited in that it relied on pathway maps for creating images. This means that the genes not mapped to particular pathways would be excluded. Another application of CNNs to drug response prediction was DeepIC5010, and this one-dimensional (1D) CNN method achieved the best result on the evaluated gastric cancer-specific dataset. As the idea of tabular image transformation gained popularity, more specialized methods for optimizing this transformation were developed, notably IGTD11, which was also applied to the task of drug response prediction. In contrast with these previous approaches that predominantly used a single type of data to create an image, this work specifically evaluates the potential of incorporating multiple, connected ‘omics layers within the same image through the use of multiple color channels to capture these relationships. Additionally, we illustrate how application of class-activation maps (CAMs) feature extraction can be used to facilitate interpretation of constructed CNN models. The theoretical basis of these proposed improvements is briefly explained below.From the methodological angle, our approach is distinguished by the idea of applying convolutional neural networks to tabular data, more specifically by converting it to images. Some of the drug response prediction applications with similar methodology are described as follows. Omics data as images was used with convolutional neural networks (CNNs) in OmicsMapNetTypically, for tabular data, traditional ML methods are used for feature extraction, feature selection and classification problems. Essentially, a column vector of size 19. Contrary to traditional ML methods, CNNs undertake feature extraction and classification through their convolutional layers using an input image  transformation using the dominant layer, where the most informative layer dictates the pixel locations and the values of other layers are mapped to these pixel locations. (2) The transformation by the equal contribution of layers. This approach finds the pixel locations in two stages; i.e., the element arrangement step is performed two times. In this way, the proposed DeepInsight-3D model, first converts multi-omic or multi-layered data into corresponding images that are organized and colored, and then applies a convolutional neural network (CNN) with automatic feature extraction ability. Moreover, the element or feature selection of the transformed data  through the application of CNNs. Two ways of image construction are introduced, 1) by mapping elements to the pixel locations of the dominant layer , artificial neural network after feature selection (ANNF), AutoBoruta Random Forest (AutoBorutaRF)41 and SVM42 as reported in 8. All these methods were compared with many preceding algorithms and showed superior performance. The test set configurations (in terms of the number of samples) were kept the same for a fairer comparison. However, the test samples may slightly differ for the methods used in the Super.FELT work. The AUCs were computed for all possible drugs-method combinations and are given in Table Two recently developed methods, MOLI and Super.FELT, were used as benchmark methods. Other than these methods, we have also compared with non-negative matrix factorization (NMF), feed-forward net, and Geeleher et al.It can be observed from Table DeepInsight-3D can also perform feature selection via class-activation maps (CAMs) to identify genes of interest for each dataset. Since the data dimensionality is very large compared to the number of samples available, there is a high chance of producing an unstable model estimate. Furthermore, not all genes can be well represented in a limited pixel-framework. Appropriate feature selection would reveal background scientific mechanisms. Therefore, we applied an iterative way of conducting feature selection. Gene selection can be performed in 3 ways, 1) considering CAM values for every training sample, 2) taking an average of CAM over training samples, and (3) class-based CAM (described in the Methods section) where the average over a particular class is considered. In this work, class-based CAM has been applied for gene selection. Table S5  knowledgebase44, as described in the methods section. This analysis has revealed that there were both a unique as well as a shared component that was recovered as important for all drugs. The overlap between drug-specific gene sets . One drug, gemcitabine, was profiled in both of these datasets. As described in detail in the methods section, our model interpretation approach allows recovery of class-specific important feature sets, therefore in total fourteen sets were generated and analyzed—i.e., separate gene sets were produced for responders and non-responders for each drug/data source combination. Note that for the purposes of interpretation, feature sets identified for the responder and non-responder classes are of equal interest and relevant gene combinations may be allocated to either or even both of the sets. Profiled drugs have covered a range of commonly exploited mechanisms of action: growth factor blockers, microtubule-binding and different forms of DNA damage.The initial analysis that was based on the Reactome pathway definitions did not identify substantial commonalities between significantly enriched pathways across all of the drugs  p = 0.017; Erlotinib p = 0.022; Paclitaxel p = 0.01) and non-responder (Gemcitabine (TCGA) p = 0.02; Docetaxel p = 0.04; Gemcitabine (PDX) p = 0.0008; Erlotinib p = 0.011; Paclitaxel p = 0.049) gene sets. STAT3 gene is often highly expressed in various cancers and can lead to therapy resistance by upregulating expression of anti-apoptotic proteins, stimulating DNA repair and cell proliferation45. Functionally closely related pathways “PI3K/AKT Signaling”  p = 0.017; Erlotinib p = 0.022; Paclitaxel p = 0.011; non-responders: Gemcitabine(TCGA) p = 0.004; Docetaxel p = 6.5 × 10–5; Gemcitabine(PDX) p = 0.048; Erlotinib p = 0.004; Cetuximab p = 0.021) and “JAK/STAT Signaling” (responders: Gemcitabine(TCGA) p = 8.6 × 10–6; Gemcitabine(PDX) p = 2 × 10–4; Cetuximab p = 2.3 × 10–4; Paclitaxel p = 0.008; non-responders: Docetaxel p = 0.008) were also found important in multiple sets and are likewise well-known to be involved in multidrug resistance47. All of these pathways are well-known to play a key role in cancer cellular growth, survival, and resistance to treatments. Likewise, three variants of Rho GTPase family signaling pathways were recovered multiple times among the non-responder gene sets, which are an important mechanism for the upstream activation of JAK/STAT pathway48 and cancer progression49. These results are in line with current understanding of the pivotal role these signaling pathways play in various drug resistance mechanisms50.Metabolic pathway enrichment analysis using IPA software has identified 269 (responders) and 342 (non-responders) non-redundant canonical pathways that were significant in at least one of the drug-specific sets. None of the pathways were found to be significantly enriched in all 7 of the drug-specific sets in either responder or non-responder categories , autophagy  p = 3.4 × 10–6; Cetuximab p = 0.006) and lysosome regulation pathways “CLEAR Signaling Pathway” (responders: Gemcitabine (PDX) p = 0.009; Cetuximab p = 7.2 × 10–4; non-responders: Gemcitabine (TCGA) p = 0.044; Docetaxel p = 0.017; Gemcitabine (PDX) p = 4.4 × 10–4; Cetuximab p = 2.6 × 10–4; Paclitaxel p = 9.3 × 10–5) were found multiple times among the non-responder gene sets. Ubiquitination marks proteins for proteolysis and this mechanism is usually preserved largely intact in cancer cells, suggesting functional essentiality51. Both lysosome52 and ubiquitination-meditated proteasome53 are believed to be involved in multidrug resistance. These processes are part of a broader set of autophagy processes that in turn lead to cellular rejuvenation, stress resistance and medication of anti-drug responses on a cellular level54.Another common theme was the potential importance of protein degradation and recycling mechanisms for cancer drug resistance. Functionally-linked ubiquitination  p = 2.5 × 10–6; Erlotinib p = 0.015; Cetuximab p = 8.1 × 10–4), “Sertoli Cell-Sertoli Cell Junction Signaling” (responders: Gemcitabine (TCGA) p = 0.002; Docetaxel p = 0.026; Erlotinib p = 2 × 10–6; non-responders: Gemcitabine (TCGA) p = 0.026; Docetaxel p = 0.003; Erlotinib p = 1.2 × 10–4; Cetuximab p = 0.016) and “Tight Junction Signaling”  p = 0.005; Gemcitabine (PDX) p = 0.020; Erlotinib p = 0.002; Cetuximab p = 4.7 × 10–5). Interestingly, multiple pathways in this broader functional category were identified in both Reactome and IPA-based enrichment analyses. These processes are important for formation of tumor microenvironment, interactions with the immune system as well as cancer cell dissimilation/metastasis. Some of the identified pathways were previously reported to be involved in drug resistance, for example paxillin was reported to be involved in tyrosine kinase inhibition in lung cancer55 and was significantly enriched for both of the tyrosine inhibitor drugs sets (Cetuximab and Erlotinib) in our analysis. More generally, tight junctions play a role in drug efflux56 and are important for anoikis and metastasis57. Extracellular matrix can confer a cell adhesion-mediated drug resistance in some cancers, like glioblastoma58 and small cell lung cancer59. However, the potential broader roles in anti-cancer drug resistance are currently not very well defined and these results hinting at prominent involvement highlight a need for further study.The last commonly represented theme was the presence of several pathways to do with extracellular structures and cell adhesion. Some specific examples in this category included “Epithelial Adherens Junction Signaling” (responders: Gemcitabine (PDX) p = 0.028; non-responders: Gemcitabine (TCGA) 61, and better understating of its interplay with established treatments may facilitate these developments. Another pathway of interest was “Clathrin-mediated Endocytosis Signaling”, which was significant for the same set of drugs. Endocytosis is a process by which molecules on the surface of a cell, like nutrients and signaling molecules, are internalized, mostly for subsequent post-processing or degradation. As a number of cancer driver and drug target proteins are located on the cell surface, it plays an important but very complex role in cancer62, and clathrin is one of the key components of this process63. Likewise, the role of this pathway in mediating response to these particular drugs still appears to be unclear. Of course, more detailed follow up of these links will be necessary to verify the observed effects and understand their roles further.Among the pathways found to be important and significant across several drugs two appear to be relatively unexplored with respect to selected drugs. The first example is “Tryptophan Degradation X”, which was significant for Gemcitabine (TCGA), Erlotinib (PDX) and Cetuximab (PDX). This result is of interest because tryptophan metabolism was proposed as a source of potential targets for future anti-cancer drugs64, which can interact to give rise to yet more variation across all other ‘omics. This inevitably leads to very high degree of tumor heterogeneity across patients due to their genomic characteristics65, which in turn leads to often vast differences in efficacy of specific treatments. Better understanding of these factors is essential for improvements in drug response prediction that will not only allow more patients to benefit through better targeting of available therapies1 but also avoid potential adverse effects67. In order to manage the complexity, increase scale and minimize risks to patients, drug response is often studied on large in vitro datasets4. The discovered mechanisms are then taken forward to pre-clinical models and eventually to clinical trials. One of the key aspects, therefore, is translatability of these results to actual better outcomes in cancer patients68. Specifically in the case of computational predictive models, the performance achieved on the in vitro training data must be comparable to that on the real patient data. Importantly, although in vitro high-throughput screening is inevitably required, the number of possible genomic factors and drug combinations make it impossible to exhaustively test all possible options69. Therefore, statistical and machine learning models are increasingly necessary to mine these huge datasets and discover novel clinically usable patterns70. In turn, these types of analyses are also used to streamline experimental work and therefore reduce the amounts of data that needs to be collected. As a possible additional benefit, more advanced machine learning methods can help to identify combinations of factors that contributed to making a particular prediction in each case and in that way facilitate the identification of pharmacological functions of these targets71.Response to cancer treatment is highly complex as it is underlain by both genomic and somatic variationsIn this context, DeepInsight-3D offers two important capabilities. First one is an ability to make accurate predictions of drug response from the high-dimensional ‘omics data. As the cost of collecting such samples continues to decrease, this information is now increasingly used by doctors to make better treatment decisions, creating a need for relevant tools. Potentially, ‘omic-based predictions can be combined with other clinical data as part of the decision process. The second capability is the discovery of underlying features of importance that may then facilitate identification of relevant mechanisms. Deep learning methods have an inherent ability to discover non-linear and interacting features when sufficient data is available—and therefore can expand the range of characterizable features beyond what is possible using less complex methods. DeepInsight-3D extends the versatility of applying CNN to multi-layered tabular data. In this work, DeepInsight-3D provided very encouraging results on drug response multi-omics data. DeepInsight-3D was able to produce an average AUC of 0.72 over seven drug response datasets which is encouraging compared to competing methods in the literature.The promising performance of DeepInsight-3D could be attributed to the following: (1) The inclusion of neighborhood information through the placement of similar elements together and dissimilar ones apart. (2) Three different layers can be combined into colored images, keeping all the possible information as kernels. (3) The interconnectivity of deeper networks (CNNs) taps into the non-linear connectivity between elements which shallow or traditional machine learning methods could miss. However, at the same time as one of limitations, a limited number of samples affects the confidence of the model, which could lead to unsatisfactory performance. Therefore, it is generally opted to have ample training samples for better estimation of the model.20, and a parsimonious memory footprint. These properties of CNN can be integrated with the inception of DeepInsight-3D for non-image tabular data with multi layers.Deep learning nets, such as CNN, have many merits, such as automatic feature extraction, finding hidden structures from hyper-dimensional data, finding higher-order statistics of image and non-linear correlations, economical use of neurons for large input sizes allowing much deeper networks are plausible with fewer parameters21. DeepInsight-3D further extends this property to multi-layered data. Moreover, the application of DeepFeature is extended. DeepFeature enables a powerful means for the identification of biologically relevant gene sets and provides methodological basement for "explainable AI"30. This has been integrated with DeepInsight-3D to simultaneously identify elements for multi-layered data.In machine learning techniques for tabular data, any two features are considered mutually independent. However, DeepInsight tries to establish a relationship through the element arrangement step by positioning similar elements together and dissimilar ones apartFunctional annotation of gene sets found to be important for particular classifications was largely in line with pathways and mechanisms previously reported in the literature. Notably, despite very small training sets, there was remarkable degree of consistency across discovered functional categories among different classifiers. These functions also corresponded to well-known multidrug resistance mechanisms as well as core cancer pathways that are linked to general “fitness” of cancerous cells. Since drug response is highly complex and determinants are highly drug-specific these processes would not be expected to be fully predictive of drug response on their own. However, an ability to discover this relevant subset of functions across multiple datasets suggests that DeepInsight transformation may prove particularly useful in comparative analysis across multi-omics datasets and lead to discovery of common mechanisms. Another notable pattern in these results was that relatively few functions were completely unique to particular drugs. This is likely due to the way relatively small sample size of the datasets interacts with convolution done when data is presented as image, which would favor more consistently present features in such cases.In this work, DeepInsight-3D is used for multi-omics datasets. However, the proposed method is not limited to omics data. It can handle different kinds of multi-layered tabular data (as long as the elements and samples of diverse layers are arranged in the same order). This method does not require any specific biological information such as chromosome locations and visualizes non-image data through multi-layered mappings.Although the results were promising, the severe scarcity of training and test samples hindered getting a reasonable model estimate. The same was true for MOLI and Super.FELT methods, as their results were sensitive to parameter tuning. In general, CNN works very well when the samples are sufficiently large. However, this was not the case in the work. Nonetheless, all these methods provided a good platform in this direction. DeepInsight-3D can perform sufficiently well when the sample size is sufficient such as in the case of single-cell analysis. This would be our future direction of work.The proposed method, DeepInsight-3D, demonstrates how the data-to-image approach for analysis of biological data can effectively incorporate different types of omics data and preserve the explicit connections between these layers by placing them in the same positions but in different channels of an input image. As was demonstrated in our previous work, once converted to image form data becomes suitable for use with image-specific convolutional neural network architectures. This study is the first to use this type of omics integration and likewise the first to apply this type of approach to the problem of personalized cancer drug response prediction. Our results have shown that DeepInsight-3D can outperform previously proposed methods and can also be very powerful way to discover underlying important genes, which can then be interpreted to understand the decisions made by the classifier and also identify key biological processes of potential interest. In future work, we would extend this approach by using a regression layer in place of a classification layer in the CNN net to enable us to utilize continuous output values. We would also look at integrating the transformer model for our future work. Presently, transformer models are predominantly used in NLP areas where input signals are time-based data. We will also look if this can be taken further to time-independent data.This section covers the proposed DeepInsight-3D methodology. The model consists of the following constituents 1) image transformation by DeepInsight-3D, (2) ResNet-50 model of CNN architecture, (3) class-based CAM to find activation maps, and 4) element decoder to decode genes . However, the Bayesian optimization technique has been employed for Cisplatin to tune the hyperparameters. Further description is given in Supplement File In this work, ResNet-50 has been used for CNN. For feature selection, we have incorporated class-activation maps (CAMs)mentclass2pt{minimmentclass2pt{minimCAMs are computed for each image sample from the training set For class-based CAMs, instead of taking from Eq. . Under td by Eq. ), would Let 2, where training sets were collated from GDSC cell lines resource3. The test sets were collated from TCGA patients with the drug response74 and PDX encyclopedia resource7.We used the same setup of datasets as done inhttps://zenodo.org/record/4036592) and correlated into the seven testing and training using R.The data was downloaded from the Zenodo repository  is 43, Gemcitabine (PDX) is 25, Cetuximab (PDX) is 60, Erlotinib (PDX) is 21, Docetaxel (TCGA) is 16, Cisplatin (TCGA) is 66 and Gemcitabine (TCGA) is 57. The training sets were subdivided into 90:10 ratio for training the model and validating the trained model. The separate test set was never used during the training and/or validation process. It has been used at the end to provide the evaluation parameters from the trained model. All the three layers, expression, CNA and mutation, are used to transform into colored images by the Deepinsight-3D model and ResNet-50 is used to find the classification performance.Various parameters such as t-SNE distance, different types of CNN nets and maximum epochs were estimated using the validation set of a drug dataset  and responders (R). The test set labels are exactly the same asTable S6  with 24 CPU cores and 2 parallel NVIDIA A100 PCIe GPUs (CUDA cores: 6912 with 40 GB GPU memory on each A100 GPU). The operating system used was Linux (Ubuntu Desktop version 20.04).75. To counter this, we have used guilt-by-association principle to propagate the likely impact of mutations by using protein–protein interaction network. Briefly, the goal of this approach was to assign some part of an “impact” for each actual mutation to proximal genes in the network, as these are likely to be involved in similar biological functions. In this way, some meaningful value is assigned to each gene in all situations, while the information about actual mutations is still preserved by assigning to them the highest possible score. Note that the fine calibration of the impact score is not necessary for this use-case, as neural network is able to discover its own optimal weighting as long as the generated distribution is consistent across all of the training set.Cancer mutation data is most often extremely sparse, meaning that only a small number of different genes have consequential mutations in each sample. This presents a unique challenge when using it with a CNN classifier—as most inputs in this channel would be zero, it can result in inefficient use of information in that layer due to “dead” artificial neurons76. A diffusion state distance matrix was calculated for the network based on the original definition of this distance metric77. Then, each node was assigned a score equal to the normalized inverse distance value of the closest mutated gene. In this way, the approach has facilitated the identification of possible functionally equivalent mutations as well as mutation hotspots, which have also been demonstrated to be an important network-based feature potentially predictive of clinical outcomes78.This was done by mapping all of the genes in the dataset to the corresponding proteins of the protein interaction network obtained from STRING database v11.0https://alok-ai-lab.github.io/DeepInsight/) for this method with a few variations  were subdivided into two sets with a 90:10 ratio. The larger set was employed to estimate the model and the smaller set was applied to validate it. The AUC was computed on the test set. In general, the default parameters of DeepInsight (version 2) were employed . To address this overlapping issue up to some extent, DeepInsight-3D can be run iteratively to gradually select the elements. The initial iteration will identify a subset of elements that can be utilized as input in subsequent iterations to find a smaller subset of genes or elements.The analysis described above resulted in two gene lists (one each for responder and non-responder class) from each trained model that contained the genes identified as important for classifying training samples into a respective category. Functional interpretation of the recovered gene subsets was done individually, by mapping them onto metabolic and signaling pathways as defined by Reactome. This was followed by gene set enrichment analysis done using Fisher’s exact test with a Benjamini–Hochberg false discovery rate correction. The analysis was further extended using Ingenuity Pathway Analysis software from QIAGEN Digital Insights, which is backed by an extensive manually-curated knowledgebase. Owing to its more extensive collection of data and more detailed pathway definitions, IPA has allowed identification of a much larger set of additional significant pathways. This analysis was done both for the metabolic and phosphorylation pathways that were filtered to only include direct links with experimental evidence.Supplementary Information 1.Supplementary Information 2.Supplementary Information 3.