The ability to accurately predict non-small cell lung cancer (NSCLC) patient survival is crucial for informing physician decision-making, and the increasing availability of multi-omics data offers the promise of enhancing prognosis predictions. We present a multimodal integration approach that leverages microRNA, mRNA, DNA methylation, long non-coding RNA (lncRNA) and clinical data to predict NSCLC survival and identify patient subtypes, utilizing denoising autoencoders for data compression and integration. Survival performance for patients with lung adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) was compared across modality combinations and data integration methods. Using The Cancer Genome Atlas data, our results demonstrate that survival prediction models combining multiple modalities outperform single modality models. The highest performance was achieved with a combination of only two modalities, lncRNA and clinical, at concordance indices (C-indices) of 0.69 ± 0.03 for LUAD and 0.62 ± 0.03 for LUSC. Models utilizing all five modalities achieved mean C-indices of 0.67 ± 0.04 and 0.63 ± 0.02 for LUAD and LUSC, respectively, while the best individual modality performance reached C-indices of 0.64 ± 0.03 for LUAD and 0.59 ± 0.03 for LUSC. Analysis of biological differences revealed two distinct survival subtypes with over 900 differentially expressed transcripts. Non-small cell lung cancer (NSCLC) makes up 85% of all cases, most of which fall into two major variations: lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC), which comprise 40% and 30% of all lung cancers, respectively.Lung cancer is the leading cause of cancer-related mortality worldwide, accounting for 18.4% of total cancer deaths in 20203 or gene expression data5, to predict survival. However, recent advances in high throughput “omics” technologies have led to publicly accessible descriptions of the genetic, epigenetic, and transcriptional profiles of cancer cells. Due to the high heterogeneity of lung cancer cells, the integration of multiple data types can offer improved survival analysis over unimodal approaches by providing more context for predictions. Although these multimodal datasets provide the potential to build more precise models of lung cancer progression, multimodal data integration techniques are still in their infancy6.The ability to accurately project NSCLC prognosis is crucial to inform physician decision-making, but it remains a difficult prediction task. Traditional approaches tend to use unimodal data sources, most commonly clinical8. Multimodal approaches for NSCLC prediction specifically tend to focus on either identifying clinically meaningful subtypes of patients, predicting patient survival9, or both10. For example, Lai et al.9 implemented a neural network that combined gene expression and clinical data to predict 5-year NSCLC survival. Another approach combined messenger RNA (mRNA), microRNA (miRNA), and clinical data to subtype LUAD patients before performing survival analysis and identifying survival-associated biological pathways11.Many current multimodal approaches to predicting cancer progression are geared toward “pan-cancer” prediction with the use of deep learning in processing the data13. The relative effectiveness of each approach depends heavily on the prediction task and the amount of data available14. However, early integration is more commonly used due to its potential to identify additional survival-related markers by combining multiple omics types13.While emerging multimodal approaches are promising, they encounter a few common issues, including how to most effectively integrate heterogenous data modalities. Early integration, defined as omics concatenation prior to feature extraction, allows for the modeling of biological interactions between different omics types as the modalities are processed together. Conversely, late integration provides more precise representations of individual modalities, as each modality is extracted separately6. Solutions include reducing the dimensionality of the feature space by using supervised and unsupervised methods. Previously used feature extraction algorithms include principal component analysis (PCA)15, deep highway networks7, and autoencoders16. In particular, autoencoders, which are fully connected neural networks with “encoding” and “decoding” network paths17, have shown promise as a dimensionality reduction technique, including in application to cancer datasets18. They do so by reducing data dimensionality to a bottleneck in a hidden layer, while the decoding path uses the bottleneck representation to recreate the original input by minimizing a chosen loss function. Autoencoders have been found to improve accuracy in predicting kidney cancer survival compared to PCA by creating nonlinear feature combinations16.The most significant impediment to multimodal approaches remains the high dimensionality of sequencing data combined with the relatively small number of patients, causing models to overfit17. Recent survival prediction studies have experimented with denoising autoencoders19, which add excess noise to the input data to improve generalization to new data20. As an unsupervised method, autoencoders are not sensitive to features most relevant to survival prediction, which is problematic given that only a relatively small subset of genomic features contributes to survival prediction21. One solution, which we implement here, is to use feature selection before denoising autoencoder compression to filter for the most relevant features.Despite their demonstrated effective performance, autoencoders are highly prone to overfittingIn this study, we explored the benefits of multimodal data integration as applied to NSCLC survival analysis. This included the development of robust early-integration patient survival models that used a combination of feature selection and denoising autoencoders for dimensionality reduction. Using mRNA, miRNA, DNA methylation, long non-coding RNA (lncRNA), and clinical data, we aimed to demonstrate the advantage of combining multiple modalities of data in predicting NSCLC patient survival. To the best of our knowledge, this combination of modalities, and in particular the lncRNA modality in a multimodal context, have not been investigated for NSCLC survival prediction. We evaluated multiple data integration and parameter estimation methods, as well as different autoencoder variations to improve NSCLC survival prediction and to identify predictive biomarkers associated with patient survival.). For multimodal models, results are reported using a single denoising zeros autoencoder (early integration), and an elastic net model trained on both LUAD and LUSC data. Results of variations of the pipeline discussed in the Methods are presented as Supplementary tables and figures.The predictive survival performance for models using each individual modality and their combination is shown in Fig. 8. As expected, the clinical data for both LUAD and LUSC were strongly prognostic. Yet, models based on multimodal data outperformed models based on clinical data alone and any other individual omics modality. Specifically, models utilizing early integrated multimodal data achieved mean C-indices of 0.67 ± 0.04 and 0.63 ± 0.02 for LUAD and LUSC data, respectively, compared to C-indices of 0.64 ± 0.03 and 0.59 ± 0.03 for clinical data alone. However, this difference was not statistically significant.The performance of the models predicting LUAD patients’ survival was superior to that of LUSC patients across individual modalities and in the multimodal setting, which agrees with previous findingsThe biological data seemed particularly informative for LUAD, as all modalities except methylation achieved a C-index of at least 0.60. This is in contrast to the LUSC data, where all biological modalities except lncRNA scored at or below an average 0.55 C-index. Across both NSCLC subtypes, methylation, and miRNA showed the lowest average performance and appeared to be the least predictive modalities. Gene expression data were moderately predictive and performed best on LUAD data, scoring a mean C-index of 0.63 ± 0.03. Notably, lncRNA narrowly scored the highest of any individual biological modality for both LUAD and LUSC data, with mean C-indices of 0.64 ± 0.03 and 0.56 ± 0.02, respectively. Ultimately, two-sample T-testing between the mean C-indices of the individual modalities compared to the multimodal modality (both LUAD and LUSC) revealed varied significance with p-values of 0.04, 0.05, 0.08, 0.15 and 0.55 for the comparison of mean C-index of DNA methylation, miRNA, mRNA, lncRNA and clinical, respectively, to multimodal.We evaluated combinations of modalities, as well as the relationships between modalities, to examine which among them were necessary for strong survival performance . Clusters were better distinguished by survival outcome than by NSCLC type (both not participating in the clustering process), as there was a relatively even split of LUAD and LUSC cases in each cluster . Patients from Cluster 1 exhibited consistently lower survival times, which explains its slightly larger proportion of LUSC cases. Genes commonly linked to NSCLC survival, including EGFR , MET , and ERBB2 , were differentially expressed in Cluster 1, but the KRAS gene was not .K-means clustering was performed on the multimodal feature space using the multimodal non-zero features from the elastic net. The Silhouette methodWe used these two groups to identify differentially expressed transcripts among all four biological modalities. In Cluster 1, which was associated with poorer survival, we identified 507 differentially expressed genes , 110 differentially expressed lncRNAs , and 7 differentially expressed miRNAs , 33.For differentially methylated genes, The denoising zeros autoencoder recorded the highest mean survival performance with the sigmoid activation function and with the addition of zeros to 30% of the matrix  across all modalities  with 0.02 increases in the multimodal and methylation modalities. Otherwise, performance appeared similar other than a 0.02 decrease for the miRNA modality . Notably, lncRNAs were included in three and four of the five top-performing modalities combinations for LUSC and LUAD. This could partially be explained by the low correlation between lncRNA and other modalities. Further, although the investigation of lncRNAs in cancer is relatively new, lncRNAs are commonly dysregulated in NSCLC19. Another study combined histopathology whole-slide images (WSIs) with clinical data for a 0.62 C-index36. In an examination of previous TCGA LUAD dataset performance, one study used mRNA data to achieve a C-index of 0.65637, and a second integrated mRNA and clinical data for a C-index of 0.68938. Finally, a LUAD study used an autoencoder to combine mRNA, miRNA, methylation, and copy-number variation data to achieve a 0.65 C-index39. As compared to the findings above, our proposed methodology demonstrates comparable or stronger performance using similar data and provides biologically actionable results. A higher performance achieved by Cheerla & Gevaert7 can potentially be explained by an increase in the training data size when using pan-cancer TCGA data (20 different cancer types), which is in line with our finding of improved performance when using both LUAD and LUSC data for training.Overall, the highest survival performance was achieved with lncRNA and clinical data  for LUAD (mean C-index = 0.69 ± 0.03) and using all modalities for LUSC (mean C-index = 0.63 ± 0.02). It is challenging to directly compare survival performance with other studies because of differences in performance evaluation methods and the dataset utilized. A notable TCGA LUSC study implemented a multimodal approach by using basic autoencoders and four biological data types to achieve a C-index of 0.597In addition to improved predictive power, our approach has demonstrated that multimodal analysis provides a unique opportunity to build robust survival subgroups that take multiple aspects of a patient’s biology and clinical status into consideration. Our differential analysis based on those survival subgroups identified a combination of differentially expressed mRNAs, miRNAs, and lncRNAs, as well as differentially methylated genes associated with improved or shortened survival. Top genes in these categories have been previously linked to shown to be associated with NSCLC and/or cancer prognosis  model.Biomarker identification: K-means clustering was conducted on the survival-associated variables (as identified by the elastic net model) of the multimodal feature space to identify survival subtypes. Differential expression analysis was carried out between the survival subtypes to identify biological transcripts associated with survival outcomes.Our proposed data analysis approach is summarized in Fig. 40. In addition, 12,440 lncRNAs were extracted from the mRNA modality according to the lncRNA list published by Lin et al.41. Each patient has a time-to-death in days right-censored up to 11,000 days post-diagnosis and a binary survival status variable.Data for this study were obtained from The Cancer Genome Atlas (TCGA) dataset, which contains multiple modalities of lung cancer data, including 1881 miRNAs, mRNA expression data for 44,162 genes, DNA methylation data from 485,512 probes, and clinical data  survived (or were censored) and 283 (38.7%) died during the study period.After all data modalities were combined, there were 544,006 features in the initial feature space. For clinical data, we used 11 features provided by TCGA Table . The “tu10. Individual mRNA, miRNA, and lncRNA features were removed if they had zero or missing values in more than 20% of patients in the dataset. Patients with follow-up times of 0 or 1 day were also removed. The features of each modality were Z-scored (i.e. each feature had zero mean and unit variance). Imputation was performed for missing values using the median. These computations were performed separately for training and testing data.For the initial preprocessing steps, the LUAD and LUSC datasets were combined as has been reported previously39. Finally, before LFS, the top 25,000 probes with the highest beta value variance were filtered to extract more informative features. This led to a reduction of lncRNA features from 12,440 to 5411, mRNA features from 44,162 to 19,837, miRNA features from 1881 to 432, and methylation features from 485,512 to 25,000 . In addition, only the 131,106 probes found in CpG islands within 1500 base pairs upstream of the transcriptional start site were used, a commonly used methylation processing approachP values were selected for each modality, proportional to the initial number of features in each modality. Feature selection was performed using training data only, with validation schema presented in the ‘Prior to autoencoder-based data integration and lower dimensionality projection, each feature in each modality was separately evaluated through an LFS process to ensure feature relevance to the prediction task. Specifically, Wald significance tests based on univariate Cox PH models were employed to measure the association of each feature with patient survival time. The top 500 features for mRNA and methylation and the top 300 features for lncRNA and miRNA with the lowest Wald 42. The denoising zeros autoencoder takes the concatenated biological data (1600 features) and reduces the data to a 160-dimensional vector . Performance of these autoencoder models and hyperparameter combinations were assessed by averaging C-index across each individual modality as well as the multimodal modality , tanh, and sigmoid activation functions were compared. For the denoising zeros autoencoder, the proportion of matrix observations replaced with zeros was modulated  and the SD of the distribution of noise added  was adjusted for the denoising Gaussian model.43. More specifically, elastic net uses both Lasso (L1) and Ridge (L2) regularization together to penalize complex Cox PH models and decrease coefficient values. The alpha value (between 0 and 1) determines how much weight is given to the L1 and L2 penalties, and an alpha value of 0.5 was chosen to give equal weight to each. Training data was used to find the optimal lambda value, a coefficient that determines the degree of coefficient shrinkage.For this study, we used an elastic net as the survival model. This model is a regularized form of the Cox PH model that deals effectively with high-dimensional data by preventing overfittingIn addition to combining all the modalities in survival prediction models, we evaluated several two, three, and four modality combinations. Both early and late data integration were evaluated for each combination.22.To gain additional insight into the features most affecting patient survival, we implemented K-means clustering using features significantly associated with survival by the elastic net model. By using one training run, features with coefficients driven to zero by the elastic net model were removed to keep only survival-associated features. These survival-related features were used in unsupervised K-means clustering to identify distinct survival subtypes. The optimal number of clusters was determined by silhouette index P value of < 0.05 and absolute fold change of > 1.5.Using these identified survival subtypes, we conducted differential expression analysis to find differences in mRNA, miRNA, and lncRNA expression between groups. Differential expression was defined as a false-discovery rate adjusted 44. Methylation probes were mapped to genes by averaging M-values across each gene. An analysis-of-variance test identified differentially methylated genes based on the mean M-value, as described previously (P < 0.05)39.For the methylation data, beta values were mapped to M-values through a logistic transformationWe used a robust training/validation and independent test set schema for reporting the results. Training/validation data (80% of the data) were used for parameter estimation through all preprocessing and modeling steps. Results are reported on the independent test set (20% of the data). To further assess the stability of the results and to report confidence intervals, a bootstrapping schema was introduced. The results are reported across these 5 independent runs.45. The C-index is used to evaluate the probability that a model will correctly predict which of two randomly selected patients will die first. In other words, it measures the ability to correctly order patients in terms of survival time, ranging from 0.5 (random chance) to 1 (perfect prediction).To estimate prediction error, we used the Harrell concordance index (C-index), the most commonly used metric for survival analysis7. However, due to the prevalence of LUAD over LUSC cases in TCGA, the survival model may prioritize features that are geared more toward LUAD, compromising LUSC performance if training is done on both LUAD and LUSC. We therefore compared models’ performance trained using LUAD or LUSC data only versus their combination.Earlier studies showed improved performance of survival models when training using pan-cancer dataThe effect of reducing the data dimensionality of each data type separately (late integration) was compared with that of using one denoising autoencoder to compress all of the biological data into a 160-dimensional vector together (early integration). For late integration, each autoencoder was structured with layers of size 500, 50, and 500 for mRNA and methylation data or 300, 30, and 300 for miRNA and lncRNA data. Survival performance was compared by using the C-index of each integration scheme on multimodal data and all possible combinations of omics data.The performance of this combined feature selection and autoencoder feature reduction approach was compared with three alternative data dimensionality reduction methods. These alternatives were a denoising autoencoder with no feature selection, PCA, and LFS with no autoencoder. We compared the survival performance of all four techniques on multimodal data across both NSCLC type and early and late modality integration.All data analyses for this study were completed with R, version 4.2.0. The Keras package in R with a Tensorflow back end was employed to implement the denoising autoencoders. Differential expression analysis was implemented with the DESeq2 R package, and the lumi R package was used to map beta values to M-values for differential methylation analysis.Supplementary Information.


Hierarchical classification offers a more specific categorization of data and breaks down large classification problems into subproblems, providing improved prediction accuracy and predictive power for undefined categories, while also mitigating the impact of poor-quality data. Despite these advantages, its application in predicting primary cancer is rare. To leverage the similarity of cancers and the specificity of methylation patterns among them, we developed the Cancer Hierarchy Classification Tool (CHCT) using the idea of hierarchical classification, with methylation data from 30 cancer types and 8239 methylome samples downloaded from publicly available databases (The Cancer Genome Atlas (TCGA) and the Gene Expression Omnibus (GEO)). We used unsupervised clustering to divide the classification subproblems and screened differentially methylated sites using Analysis of variance (ANOVA) test, Tukey-kramer test, and Boruta algorithms to construct models for each classifier module. After validation, CHCT accurately classified 1568 out of 1660 cases in the test set, with an average accuracy of 94.46%. We further curated an independent validation cohort of 677 cancer samples from GEO and assigned a diagnosis using CHCT, which showed high diagnostic potential with generally high accuracies (an average accuracy of 91.40%). Moreover, CHCT demonstrates predictive capability for additional cancer types beyond its original classifier scope as demonstrated in the medulloblastoma and pituitary tumor datasets. In summary, CHCT can hierarchically classify primary cancer by methylation profile, by splitting a large-scale classification of 30 cancer types into ten smaller classification problems. These results indicate that cancer hierarchical classification has the potential to be an accurate and robust cancer classification method.The online version contains supplementary material available at 10.1186/s12859-023-05529-0. The aggressive and unpredictable spread pattern of malignant tumors always poses a considerable challenge to patients and medical personnel. Traditional cancer detection techniques are usually invasive, single-target, and difficult to detect cancer at early stages. However, with the development of genetic testing technology, molecular diagnostic technologies are playing an increasingly important role in cancer detection and early diagnosis. As a method for molecular tumor diagnosis, DNA tumor detection is divided into genetic mutation detection and methylation detection. Nevertheless, normal human cells undergo continual mutations, some of which involve cancer-related genes. However, only a limited number of these mutations actually contribute to tumor development. This situation can result in false positives and may suggest the challenge of distinguishing the tissue origin of tumors from various organs solely based on genetic mutations . In contDNA methylation is a fundamental epigenetic mark that governs cell identity and gene expression changing occurs in early carcinogenesis . Tumor sIn this regard, various studies have focused on identifying tumor based on DNA methylation patterns. Koelsche et al. constructed a random forest model to classify soft tissue and bone tumors using a dataset of 1077 methylation profiles. Their findings demonstrated the potential of DNA methylation-based sarcoma classification for research and future diagnostic applications . Hao et The classification method used in these studies, as mentioned above, is flat classification, which requires all features that are informative enough to distinguish all involved cancer types to be found at once. However, (i) in flat classification, the increase in the number of categories to be recognized may lead to a decrease in classification accuracy, especially in the case of relatively homogeneous category classifications. (ii) There is also no way for flat classification to provide predictive power in the face of cancer types not included in the classifier, even if such cancer is somewhat associated with certain cancer in the classifier. In fact, the natural hierarchy of the data could have high classification values. Many important real-world classification problems are naturally reduced to hierarchical classification problems in which the classes to be predicted are organized into a hierarchy of classes-typically lung cancer, squamous cell lung carcinoma and lung adenocarcinoma, ignoring those parent–child class relationships could lose some vital information. (iii) In flat classification, poor data quality in some categories may affect all other categories, which in turn affects the performance of the overall models .In addition, methylation data, as a complex dataset, can reflect the similarities and origins of different tumors. It can be used to organize cancer entities into a tree structure and hierarchically classify them based on their similarities, thereby achieving a more complex classification.In this study, instead of flattening out and ignoring those inner hierarchies, we considered the similarity of the cancers and developed a Cancer Hierarchy Classification Tool (CHCT) using the idea of hierarchical classification that splits a large-scale classification problem of 30 cancer types into a set of small classification problems to classify primary cancer by the methylation profile hierarchically. To carry out our study, we utilized 8239 tissue samples spanning 30 common primary cancers from the open-access database TCGA and GEO (Gene Expression Omnibus), split into training set and test set with a 4:1 ratio. UPGMA (unweighted pair group method with arithmetic mean), an unsupervised algorithm for grouping data based on overall similarity, was used to determine the architecture of CHCT, which divided 30 cancer types into 12 groups. Thus, the large problem of classifying 30 types of cancer was divided into ten subproblems. The Random Forest algorithm has excellent robustness, outstanding performance in handling high-dimensional data, overfitting prevention capabilities, and effectiveness in handling imbalanced data. Thus, we used the Random Forest algorithm to develop our models with the training set samples to solve these ten subproblems. Each classification problem corresponds to a classification model. We assess the model’s performance on a held-out test cohort. Finally, we evaluated our models by testing them on an independent test cohort consisting of multiple tumor types from the GEO database.CHCT can be viewed as a tool with a two-tier architecture. The data to be predicted is first predicted by the first layer of CHCT, and the result of this layer is used to further mobilize the next layer of the prediction model to get the final prediction results (For OV (Ovarian Cancer), THYM (Thymoma), and LIHC (Liver Cancer), the first layer prediction model gives the prediction directly). The overall procedure of CHCT comprises five main steps: (i) sample acquisition from TCGA and GEO, and data preprocessing; (ii) clustering samples using the UPGMA algorithm; (iii) screening biomarkers using ANOVA  test, Tukey-Kramer test, and Boruta algorithm; (iv) training random forest models to classify cancer types and constructing CHCT; and (v) validating CHCT. An overview of the study workflow is illustrated in Fig. In this study, raw primary cancer methylation data from the Illumina Human Methylation 450 BeadChip (450K) methylation platforms were obtained from TCGA and GEO. The UCSC Xena browser  was usedWe took several steps to preprocess our data effectively. First, we excluded subtypes that either had fewer than five samples or contributed less than 10% to the total number of samples for a given cancer type. Additionally, we excluded subtypes with ambiguous information. To reduce the dimensionality of our data and make it more amenable to result visualization without significant information loss, we calculated the mean methylation fraction for each subtype within each cancer type, resulting in clustered data. Subsequently, we divided the raw data into 103 distinct groups based on clinical subtypes. To identify the most informative CpG sites, we selected the top 1% of CpG sites that exhibited the highest variability across all samples. Next, we used the UPGMA  clusterip-value adjusted for multiple comparisons < 0.01 and the absolute mean methylation difference between the compared groups of > 0.2  were analyzed to screen the group-specific probes for the first layer. We firstly applied ANOVA test to find any statistical differences in probes between the means of the 12 groups of cancer types, probes with p-value less than 0.5 were screened. Then, the Tukey-kramer test was performed for pairwise comparison of 12 groups aiming to screen probes with differences from the all the other 11 groups using the cut-off of the Tukey-corrected We first randomly split each type of cancer samples into training set that was participated in model training and test set that was not involved in model construction throughout with a 4:1 ratio. Then, we reduced the dimension of these train data  to ensure that the model is trained with a modest number of features. The Random Forest model , where cTo access the robustness of the classifier algorithm, we use the external independent validation cohort obtained from the GEO database. The model was evaluated using the top-k differential diagnosis accuracy, which measures how accurately the ground truth label is found in the k highest confidence predictions of the model. This allows us to potentially use the top predictions of CHCT for a given sample to narrow down the origin of the tumor to a handful of possible types. CHCT defaults to top-2² accuracy as the output accuracy: CHCT would select the top-2 of the first layer classifier firstly. If the top-2 of the first layer contains multi-classification groups such as group-5, group-6, group10, group11, or group12, CHCT would further select the top-2 of these groups in the second layer. Overall, the result of CHCT can be anywhere from 2 to 4. To visualize heterogeneity between samples, we used Uniform Manifold Approximation and Projection (UMAP) , which iAfter completing data collection and processing, we first clustered the methylation data consisting of 30 cancer types from TCGA and GEO. The raw data was divided into 103 groups based on the clinical subtypes. We further used an unsupervised agglomerative algorithm (UPGMA) to cluster the 103 subtypes of 30 kinds of cancers by the top 1% most variable CpG site among them. These cancer classes were assigned to 12 categories relating to their Pearson similarity.For each cancer, its subtypes cluster in close proximity to each other, reflecting the homogeneity of different subtypes of cancer. Since ESCA adenocarcinomas and CESC adenocarcinomas are distinct from ESCA squamous cell carcinomas and CESC squamous cell carcinomas respectively, we separated the ESCA dataset and CESC dataset into four subtypes: ESCA adenocarcinoma ESCA11), ESCA squamous carcinoma ESCA12), CESC adenocarcinoma (CESC11) and CESC squamous carcinoma (CESC12). The result of the cluster is shown in Fig. , CESC ad1, ESCA sWe conducted a differential methylation analysis on the 12 classes defined above to identify the most informative CpG sites for cancer-type classification. To achieve this, we used the ANOVA test, Tukey-Kramer test, and Boruta algorithm. Firstly, we used the ANOVA test to select probes with a significant difference. We then applied the Tukey-Kramer test to retain CpGs that could best distinguish one cancer group from all other cancer groups. From the 12 groups, we identified 13,103 probes that were statistically significant. We then used the Boruta algorithm to screen for more informative CpG sites, which resulted in the retention of 8061 probes. The heatmap shown in Fig. We further selected the features of the group-2, group-4, group-5, group-6, group-7, group-8, group-11, and group-12 of the second layer using the same procedure, resulting in the identification of 284, 238, 986, 1078, 339, 456, 874, and 1481 probes, respectively. The above selected markers are sufficiently discriminatory for each cancer type  from the GEO cohort. We used top-entclass1pt{minimaWe performed UMAP clustering on the same cancer data from different databases. Data from TCGA were mainly used for model construction and testing, and data from GEO were mainly used to verify model generalizability. We found a high degree of heterogeneity between the same type samples from TCGA and GEO. This suggests that different processing methods may have been used between different databases. However, our tool performs relatively well in different data processing methods. In brief, our models are capable of generalizing to diverse data sources and processing methods not encountered during model training or probe normalization.We utilized CHCT to predict outcomes for two representative datasets: the medulloblastoma dataset (GSE75153) and the pituitary tumor dataset (GSE147548). As expected, 100% of the predictions for medulloblastoma were concentrated in the group-6 category, in spite of medulloblastoma being a clinical subtype of glioma not covered by the TCGA dataset GBM and LGG. In addition, the classification of pituitary tumors in the CHCT dataset was also all clustered in group-6. Since pituitary tumors, like ACC, are endocrine adenomas composed of neuronal cells, it is logical for them to cluster in group-6. Hierarchical classifiers have the potential to offer predictive power for new tumors that were previously unknown.Considering the similarities between tumors, we used the strategy of hierarchical classification that splits a complete multi-class problem into a set of smaller classification problems to build CHCT and achieved robust predictive performance.In contrast to CHCT, we built a flat classification model using the same scenario, resulting in 3612 markers after screening. A random forest model was then built and achieved an accuracy of 93.23%, with a 5-fold cross-validation score of 0.932. And we also compare the CHCT with the 2 classifier that are used for primary cancer classification , 32. In Apart from its performance advantages, hierarchical classifiers also offer some predictive power for new or unknown tumors. While it may not be possible to classify these new tumors, the classifiers can still predict the upper classes, which can be clinically relevant in determining the appropriate treatment. For instance, similar androgen reduction treatments, such as testosterone antagonists, can be used to treat both prostate and breast cancer. The CHCT provides predictive results that suggest specific cancer foci or associated features, enabling individualized treatment recommendations.The hierarchical classification can better adapt to dynamic changes. Later on, as the number of cancer types increases, the classification structure can be dynamically adjusted according to the similarity between the newly added cancers and the original ones.Furthermore, the advantages of hierarchical classification are further magnified in cases where data quality in some categories may not be comparable to other categories, as poor data quality in a category in a flat classification model may affect all other categories. In contrast, hierarchical taxonomy ensures that taxonomically distinct classes are not susceptible to this phenomenon.Although the multi-level classification model may lead to error propagation when the upper-level classifier produces an error result, errors are most likely confined to its class or near classes. Hence, the k highest confidence predictions designed to narrow down potential cancer types could be used as the output of CHCT.There are still some samples in the independent validation cohort that cannot be correctly classified by the current version of CHCT. There are several reasons: (i) the train data used to build models does not cover the entire spectrum of all subtypes, and an increase in the number of subtypes could help identify more methylation subclasses. (ii) Batch effects are also an essential factor affecting the validation results. Batch effects are sub-groups of measurements that have qualitatively different behavior across conditions and are unrelated to the biological or scientific variables in a study. Different reagent dosages, chips, experiment instruments, operation personnel, etc., result in batch effect . (iii) DIn summary, we have developed the CHCT, a powerful tool for cancer classification that serves as proof-of-concept for hierarchical classification models in methylation analysis, advancing the application of hierarchical classification in translational medicine. We propose that hierarchical classification has the potential to be an accurate and robust cancer classification method with advantages that flat classification does not possess. Moreover, different feature selection algorithm can be used to build models in future. It makes sense that different algorithms and features for different subproblems can provide better results. Further research may be necessary to optimize model performance.Additional file 1: Contains heatmaps of methylation site markers, ROC curve plots of models, a table of sample information, and classification performance reports of models.Additional file 2: Provides the ID of TCGA Samples.


The application of next-generation sequencing (NGS) has transformed cancer research. As costs have decreased, NGS has increasingly been applied to generate multiple layers of molecular data from the same samples, covering genomics, transcriptomics, and methylomics. Integrating these types of multi-omics data in a combined analysis is now becoming a common issue with no obvious solution, often handled on an ad hoc basis, with multi-omics data arriving in a tabular format and analyzed using computationally intensive statistical methods. These methods particularly ignore the spatial orientation of the genome and often apply stringent p-value corrections that likely result in the loss of true positive associations. Here, we present GENIUS , a framework for integrating multi-omics data using deep learning models developed for advanced image analysis. The GENIUS framework is able to transform multi-omics data into images with genes displayed as spatially connected pixels and successfully extract relevant information with respect to the desired output. We demonstrate the utility of GENIUS by applying the framework to multi-omics datasets from the Cancer Genome Atlas. Our results are focused on predicting the development of metastatic cancer from primary tumors, and demonstrate how through model inference, we are able to extract the genes which are driving the model prediction and are likely associated with metastatic disease progression. We anticipate our framework to be a starting point and strong proof of concept for multi-omics data transformation and analysis without the need for statistical correction. The recent advent of next-generation sequencing (NGS) has revolutionized research and has been applied extensively to investigate complex biological questions. As the cost of sequencing continues to drop, it has become increasingly common to apply NGS technology to investigate complementary aspects of the biological processes on the same samples, particularly through analysis of DNA to resolve genomic architecture and single-nucleotide variants, RNA to investigate gene expression, and methylation to explore gene regulation and chromatin structure. Such multi-omics data provides opportunities to perform integrated analysis, which investigates multiple layers of biological data together. Over the years, this has resulted in the generation of an incredible amount of very rich data derived from the genome itself, either directly or indirectly. The genome is spatially organized, with genes positioned on chromosomes sequentially and accessed by biological processes in blocks based on chromatin organization . HoweverUsing artificial intelligence methods may help overcome this problem. Over the past decade, the development of artificial intelligence methods, particularly within deep learning architectures, has thoroughly revolutionized several technical fields, such as computer vision, voice recognition, advertising, and finance. Within the medical field, the roll-out of AI-based technologies has been slower, hampered in part by considerable regulatory hurdles that have proven difficult for machine-learning applications where the systems may accurately classify patients or samples by some parameter, but the logical reason behind this is unclear . NeverthIn medicine, the interpretability of models is crucial since there is a need for discovering new biomarkers as well as identifying underlying biological processes . In addiHere, we present a framework for multi-omics analysis based on a convolutional deep learning network to find hidden, nonlinear patterns in spatially connected feature-rich multi-layered data. The spatial connection of the data is made by transforming the data into a multi-channel image in such a way that spatial connections between genes are captured and analyzed using convolutional layers. Using spatial connections between the data showed superior performance when compared to non-spatially data transformations. Furthermore, the trained model is combined with IG, which allows us to evaluate the relative contribution of individual features and thus decipher the underlying biology that drives the classification provided by the deep learning models. IG is a non-parametric approach that evaluates the trained model relative to input data and output label, resulting in attribution scores for each input with respect to the output label. In other words, IG represent the integral of gradients with respect to inputs along the path from a given baseline. By using IG, we provide an alternative solution to the problem posed by performing multiple independent statistical tests. Here, instead of performing multiple tests, a single analysis is performed by transforming multi-omics data into genome images, training a model, and inspecting it with IG. IG will output an attribution score for every gene included in the genome image. These can be ranked in order to retrieve a subset of the most associated genes relative to the output variable. We named the framework GENIUS , and the methodology may be split into two parts, classification and interpretation. First, the key feature of GENIUS is that for classification, multi-omics data is transformed into multi-channel images where each gene is presented as a pixel in an image that covers the whole genome . We thenAll predictions are based on multi-omics input through the GENIUS framework. Users may train their own or publicly sourced multi-omics data against a specified endpoint tailored to the user’s choice. The GENIUS framework thus overcomes the issue of multiple hypothesis testing and may provide new insights into the biology behind classification by deep learning models. The GENIUS framework is made available as a GitHub repository and may be used without restrictions to develop stratification models and inform about genome biology using multi-omics input.We designed a four-part convolutional neural network with the purpose of extracting the features from multi-dimensional data while minimizing the impact of noise in the data . The netThe first part of the network is called the encoder, as its purpose is to encode the entire image to a vector of size 128, representing the latent representation of the input data, ‘L’. Next, the original image is reconstructed from L into its original size using a decoder module in the network. In this step, since we are not using the reconstruction loss, the network reconstructs the image of a genome which is optimal for information extraction. This is followed by the extractor module containing convolution and max-pooling layers aiming to extract relevant information from the reconstructed image. The final part of the network flattens the learned features obtained from previous layers, concatenates them with the L vector, and forwards it to a fully connected dense feed-forward network where the final prediction is made . During All models were trained with Adagrad optimizer with the following hyperparameters: starting learning rate = 9.9e−05 (including learning rate scheduler and early stopping), learning rate decay and weight decay = 1e−6, batch size = 256, except for memory-intensive chromosome images where the batch size of 240 was used. Adding chromosome interaction information to the data transformation showed improvement during training; next question was whether we should penalize the reconstruction of genome image during the training process. After multiple training scenarios and hyperparameter exploration, we concluded that by forcing the network to reconstruct genome images in the process of learning, we are limiting network performance. Instead, we used the appropriate loss function for prediction and allowed the network to reconstruct genome images that are optimal for making predictions.To evaluate the performance of GENIUS with an image-based transformation of input omics data, we tested four different image layouts of the genome. For each layout, we created a set of images where each sample is represented by one multi-channel image and each channel represented a specific type of omics data  . Each daSquare image (198 × 198 pixels), each gene represented by one pixel ordered by chromosome position. Chromosomes are ordered by interaction coefficient based on Hi-C sequencing .Square image (198 × 198 pixels), each gene is represented by one pixel located on the image in random order; thus, the 2D location carries no information.Rectangular image (24 × 3760 pixels), each gene represented by one pixel ordered by chromosome position. Chromosomes are ordered by interaction coefficient based on Hi-C sequencing .A flat, one-dimensional vector containing all features from the five data sources in random order.By using different image layouts, we wanted to investigate the spatial dependency of observations. Images were created by making a matrix for each source of data where each cell was represented by a single gene  and 2. TR value by dividing the total copy number with ploidy and taking the log2 value. The weighted genome integrity index (wGII) . These were picked to filter out cancer types with less than 400 samples. Next, cancer types with an extremely high or extremely low proportion of metastatic samples (0.85 < Proportion > 0.15) were removed, resulting in ovarian serous cystadenocarcinoma (OV), colon adenocarcinoma (COAD), uterine corpus endometrial carcinoma (UCEC), kidney renal clear cell carcinoma (KIRC), urothelial bladder carcinoma (BLCA), and stomach adenocarcinoma (STAD) . RNAseq x (wGII)  was calcTwo independent cohorts of bladder cancer patients were used for validation. The UROMOL cohort  containsFor both cohorts, RNAseq data was aligned against hg38 using STAR  version R segmented copy number data was analyzed as deletion and amplification separately. Copy number deletion was defined as log R scores <log2 of 0.5/2, copy number amplification was defined as log R scores >log2 of 5/2. All data types were defined on the gene level. For copy number alterations, we defined genes as amplified if the entirety of the gene was found within the amplified DNA segment. Genes were defined as deleted if they were partially or wholly within the deleted DNA segment , defined as stage IV versus stages I–III.TP53 mutation (binary classification), where the TP53 mutation was removed from the input data and used only as a binary outcome label.The tissue of origin (multi-class classification).Age (continuous variable).wGII , a chromRandomized tissue of origin (multi-class variable). By randomizing the tissue of origin labels, a negative control was created. The purpose of this negative control was to confirm the model would fail to predict a pattern when none existed.F1 score, a standard measure for multi-class classification that combines the sensitivity and specificity scores and is defined as the harmonic mean of its precision and recall. To evaluate model performance against the binary outcome, ROC analysis was performed, and the area under the curve (AUC) was used as the performance metric.In order to adapt the network for predicting different variables, we simply changed the output layer and loss function for training. Binary classifications and the multi-class classification used softmax as the output layer and the cross entropy loss function. When predicting continuous values, the model used the output from the activation function with the mean squared error loss function. When predicting multi-class labels, the performance measure was defined by the The purpose of latent vectors is to capture the most significant information from the entire genome data and compress it into a vector of size 128. This vector was later appended into a feed-forward network when making the final prediction. This way, the model had access to extracted information before and after image reconstruction. After successful model training, we extracted the latent representations of each genome and performed the Uniform Manifold Approximation and Projection (UMAP) of the data for the purpose of visual inspection of a model . The UMAOnce the model was trained on the data, the appropriate loss function and output layer, including the model weights, were stored in a .pb file. The model and final weights were analyzed using the IG method implemented by Capture . IG is ahttps://github.com/mxs3203/GENIUS; copy archived at All code is available on the public GitHub repository  SquF1 score of 0.81. Chromosome Image followed with an F1 score of 0.74, and the flat vector of features performed similarly to the random square image, reaching F1 scores of 0.66 and 0.71, respectively , which provides the opportunity to inspect model performance . The L vTo explore the utility of the GENIUS framework to classify tumors from multi-omics data and to interpret the biological drivers behind the classification, we further investigated the GENIUS model trained against metastatic disease using the TCGA datasets . This anAnalyzing raw attribution scores we concluded the most informative data type overall regarding the development of metastatic disease was methylation . To idenTo investigate if the genes with the highest attribution score in the TCGA bladder cancer analysis were indeed associated with metastatic bladder cancer, we utilized an immunotherapy-treated predominantly late-stage (mainly stage III and IV) bladder cancer cohort with gene expression data available for 348 tumors . For thiTo investigate if the metastasis-associated genes found through the GENIUS framework also plays a role in the development of aggressive features in early-stage bladder cancer, we acquired the UROMOL dataset , which iIn this work, we explored multiple options on how to transform multi-omics data into an image, leading to the utilization of deep learning models, which are often described as ‘black box’ models. The model architecture was evaluated in six different training scenarios, with a focus on validating the prediction of metastatic cancer. In this process, we also evaluated four different image layouts, concluding that of these, projecting the genome into a 198 × 198 square image with genes organized based on chromosome interaction  performeThe main purpose behind the study was to demonstrate the feasibility of leveraging the power of deep learning techniques optimized for image analysis to interpret genome-derived multi-omics data. A key element of this approach includes the transformation of genomic data into images with genes arranged as pixels organized by chromosomal location. Beyond the readout from multi-omics data, this approach provides spatial information to the deep learning framework, which significantly improves the performance of the models . To the For every cancer type included in the dataset, we listed the top 10 genes driving metastatic disease and investigated in detail genes associated with BLCA metastasis and aggressiveness . For thiTaken together, we provide here the GENIUS framework along with analysis demonstrating the utility in multi-omics analysis. While we have focused on cancer analysis here, we believe GENIUS may find utility in a diverse range of genome-based multi-omics analyses. We have provided a git-hub repository that can be used to transform data into images and train the same model predicting variables of user’s interest and inferring the importance of input with respect to the desired output. valuable manuscript presents a new approach to transform multi-omics datasets into images and to exploit Deep Learning methods for image analysis of the transformed datasets. As an example, the method is applied to multi-omics datasets on different cancers. While the evidence in this specific case is solid, whether the method is working as advertised in other settings is not yet known.This  This study by Sokač et al. entitled "GENIUS: GEnome traNsformatIon and spatial representation of mUltiomicS data" presents an integrative multi-omics approach which maps several genomic data sources onto an image structure on which established deep-learning methods are trained with the purpose of classifying samples by their metastatic disease progression signatures. Using published samples from the Cancer Genome Atlas the authors characterize the classification performance of their method which only seems to yield results when mapped onto one out of four tested image-layouts.A few remaining issues are unclear to me:1. While the authors have now extended the documentation of the analysis script they refer to as GENIUS, I assume that the following files are not part of the script anymore, since they still contain hard-coded file paths or hard-coded gene counts:https://github.com/mxs3203/GENIUS/blob/master/GenomeImage/make_images_by_chr.pyhttps://github.com/mxs3203/GENIUS/blob/master/GenomeImage/randomize_normal_imgs.pyhttps://github.com/mxs3203/GENIUS/blob/master/GenomeImage/utils.pyIf these files are indeed not part of the script anymore, then I would recommend removing them from the GitHub repo to avoid confusion. If, however, they are still part of the script, the authors failed to remove all hard-coded file paths and the software will fail when users attempt to use their own datasets.2. The authors leave most of the data formatting to the user when attempting to use datasets other than their own presented for this study:--clinical_data: Path to CSV file that must contain ID and label column we will use for prediction--ascat_data: Path to output matrix of ASCAT tool. Check the example input for required columns--all_genes_included: Path to the CSV file that contains the order of the genes which will be used to create Genome Image--mutation_data: Path CSV file representing mutation data. This file should contain Polyphen2 score and HugoSymbol--gene_exp_data: Path to the csv file representing gene expression data where columns=sample_ids and there should be a column named "gene" representing the HugoSymbol of the gene--gene_methyl_data: Path to the csv file representing gene methylation data wherecolumns=sample_ids and there should be a column named "gene1" representing the HugoSymbol of the geneWhile this suggests that users will have a difficult time adjusting this analysis script to their own data, this issue is exacerbated by the fact that their analysis script has almost no internal checks whether data format standards were met. Thus, the user will be left with cryptic error messages and will likely give up soon after. I therefore strongly recommend adding internal data format checks and helpful error or warning messages to their script to guide users in the input data adoption process. In this manuscript, Birkbak and colleagues use a novel approach to transform multi-omics datasets in images and apply Deep Learning methods for image analysis. Interestingly they find that the spatial representation of genes on chromosomes and the order of chromosomes based on 3D contacts leads to best performance. This supports that both 1D proximity and 3D proximity could be important for predicting different phenotypes. I appreciate that the code is made available as a github repository. The authors use their method to investigate different cancers and identify novel genes potentially involved in these cancers. Overall, I found this study important for the field.In the original submission there were several major points with this manuscript could be grouped in three parts:1. While the authors have provided validation for their model, it is not always clear that best approaches have been used. This has now been addressed in the revised version of the manuscript.2. Potential improvement to the methodIt is very encouraging the use of HiC data, but the authors used a very coarse approach to integrate it (by computing the chromosome order based on interaction score). We know that genes that are located far away on the same chromosome can interact more in 3D space than genes that are relatively close in 1D space. Did the authors consider this aspect? Why not group genes based on them being located in the same TAD? In the revised version of the manuscript, the authors discussed this possibility but did not do any new additional analysis.https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02728-5. In the revised version of the manuscript, the authors addressed fully this comment.Authors claim that "given that methylation negatively correlates with gene expression, these were considered together". This is clearly not always the case. See for example 3. Interesting results that were not explained.In Figure 3A methylation seems to be most important omics data, but in 3B, mutations and expression are dominating. The authors need to explain why this is the case. In the revised version of the manuscript, the authors have clarified this. The following is the authors’ response to the original reviews.Reviewer #1 (Public Review):This study by Sokač et al. entitled "GENIUS: GEnome traNsformatIon and spatial representation of mUltiomicS data" presents an integrative multi-omics approach which maps several genomic data sources onto an image structure on which established deep-learning methods are trained with the purpose of classifying samples by their metastatic disease progression signatures. Using published samples from the Cancer Genome Atlas the authors characterize the classification performance of their method which only seems to yield results when mapped onto one out of four tested image-layouts.Major recommendations:In its current form, GENIUS analysis is neither computationally reproducible nor are the presented scripts on GitHub generic enough for varied applications with other data. The GENIUS GitHub repository provides a collection of analysis scripts and not a finished software solution (e.g. command line tool or other user interface) (the presented scripts do not even suffice for a software prototype). In detail, the README on their GitHub repository is largely incomplete and reads analogous to an incomplete and poorly documented analysis script and is far from serving as a manual for a generic software solution (this claim was made in the manuscript).https://github.com/mxs3203/GENIUS) that contains a small set of example data and all the necessary scripts to run GENIUS. The README file guides the user through each step of the GENIUS framework but it also contains a bash script that runs all the steps at once. When a user would like to use it on their own data, they need to replace the input data with their data but in the same format as the example input data. This is now fully documented in the README file. All scripts have arguments that can be used to point to custom data. The entire pipeline using example data can be run using run_genius.sh script. This script will produce CSV files and PNG files inside the ExtractWithIG folder containing attribution scores for every cancer type tested.We apologize for this oversight, and we have now invested considerable resources into making the documentation more detailed and accurate. We have created a new GitHub repository  from the cited databases and how such data should then be curated alongside the input genome to generically create the "genomic image".https://xenabrowser.net/). However, the data formats are generic, and similar data types  can be obtained from multiple sources. We have added example data to demonstrate the layout, and we have a script included that creates the layout from standard mutation, expression, methylation and copy number data formats. We have substantially improved the annotations, including detailed descriptions of the data layout along with examples, and we have, as part of our validation, had an independent person test run the scripts using TCGA example data we provided on the new GitHub page.Data for analysis can be sourced from multiple locations, what we have used in our examples and for development was based on data from the TCGA. It can be retrieved from the official TCGA data hub or through Xena Browser . Overall, due to the lack of documentation and hard-coded local-machine folder paths it was impossible to computationally reproduce this study or run GENIUS in general.Apologies, we have completely reworked the code base, and extensively annotated the code. We have also made highly detailed step-by-step instructions that should enable any user to run GENIUS on their own or public data.In the Introduction the authors write: "To correct for such multiple hypothesis testing, drastic adjustments of p-values are often applied which ultimately leads to the rejection of all but the most significant results, likely eliminating a large number of weaker but true associations.". While this is surely true for any method attempting to separate noise from signal, their argument fails to substantiate how their data transformation will solve this issue. Data transformation and projection onto an image for deep-learning processing will only shift the noise-to-signal evaluation process to the postprocessing steps and won't "magically" solve it during training.The data transformation does not solve the problem of multiple hypothesis testing but it facilitates the use of computer vision algorithms and frameworks on rich multi-omics data. Importantly, transforming the data into genome images, training the model, and inspecting it with integrated gradients can be interpreted as running a single test on all of the data.Analyzing multiomics data using classical statistical methods typically means that we perform extensive filtering of the data, removing genes with poor expression/methylation/mutation scores, and then e.g. perform logistic regression against a desired outcome, or alternatively, perform multiple statistical tests comparing each genomic feature independently against a desired outcome. Either way, information is lost during initial filtering and we must correct the analysis for each statistical test performed. While this increases confidence in whichever observation remains significant, it also undoubtedly means that we discard true positives. Additionally, classical statistical methods such as those mentioned here do not assume a spatial connection between data points, thus any relevant information relating to spatial organization is lost.Instead, we propose the use of the GENIUS framework for multiomics analysis. The GENIUS framework is based on deep neural nets and relies on Convolutions and their ability to extract interactions between the data points. This particularly considers spatial information, which is not possible using classical statistical methods such as logistic regression where the most similar approach to this would include creating many models with many interactions.Furthermore, integrated gradients is a non-parametric approach that simply evaluates the trained model relative to input data and output label, resulting in attribution for each input with respect to the output label. In other words, integrated gradients represent the integral of gradients with respect to inputs along the path from a given baseline to input. The integral is described in Author response image 1:https://captum.ai/docs/introduction) or in original paper https://arxiv.org/abs/1703.01365.More about integrated gradients can be read on the Captum webpage(Since we transformed the data into a data structure (genome image) that assumes a spatial connection between genes, trained the model using convolutional neural networks and analyzed the model using integrated gradients, we can treat the results without any parametric assumption. As a particular novelty, we can sort the list based on attribution score and take top N genes as our candidate biomarkers for the variable of interest and proceed with downstream analysis or potentially functional validation in an in vitro setting. In this manner, the reviewer is correct that the signal-to-noise evaluation is shifted to the post-processing steps. However, the benefit of the GENIUS framework is particularly that it enables integration of multiple data sources without any filtering, and with constructing a novel data structure that facilitates investigation of spatial dependency between data points, thus potentially revealing novel genes or biomarkers that were previously removed through filtering steps. However, further downstream validation of these hits remains critical.We added the following paragraph to make this more clear"Integrated Gradients is a non-parametric approach that evaluates the trained model relative to input data and output label, resulting in attribution scores for each input with respect to the output label. In other words, Integrated Gradients represent the integral of gradients with respect to inputs along the path from a given baseline. By using Integrated Gradients, we provide an alternative solution to the problem posed by performing multiple independent statistical tests. Here, instead of performing multiple tests, a single analysis is performed by transforming multiomics data into genome images, training a model, and inspecting it with Integrated Gradients. Integrated Gradients will output an attribution score for every gene included in the genome image and those can be ranked in order to retrieve a subset of the most associated genes relative to the output variable."In addition, multiple-testing correction is usually done based on one particular data source (e.g.expression data), while their approach claims to integrate five very different genomic data sources with different levels and structures of technical noise. How are these applications comparable and how is the training procedure able to account for these different structures of technical noise? Please provide sufficient evidence for making this claim .The reviewer is correct that there will be different technical noise for each data source. However, each data source is already processed by standardized pipelines used for interpreting sequence-level data into gene expression, mutations, copy number alterations and methylation levels. Thus, sequence-level technical noise is not evaluated as part of the GENIUS analysis. Nevertheless, the reviewer is correct that sample-level technical noise, such as low tumor purity or poor quality sequencing, undoubtedly can affect the GENIUS predictions, as is true for all types of sequence analysis. As part of GENIUS, an initial data preprocessing step , is that each data source is normalized within that source and linearly scaled in range zero to one . This normalization step means that the impact of different events within and between data sources are comparable since the largest/smallest value from one data source will be comparable to the largest/smallest value from another data source.Additionally, deep neural networks, particularly convolutional networks, have been shown to be very robust to different levels of technical noise . In the manuscript we show the attribution scores for different cancer types in figure 3B of the paper. Here, the top genes include established cancer genes such as P53, VHL, PTEN, APC and PIK3CA, indicating that the attribution scores based on GENIUS analysis is a valid tool to identify potential genes of interest. Furthermore, when focusing the analysis on predicting metastatic bladder cancer, we were able to show that of the top 10 genes with the highest attribution scores, 7 showed significant association with poor outcome in an independent validation cohort of mostly metastatic patients (shown in figure 4).I didn't find any computational benchmark of GENIUS. What are the computational run times, hardware requirements (e.g. memory usage) etc that a user will have to deal with when running an analogous experiment, but with different input data sources? What kind of hardware is required GPUs/CPUs/Cluster?We apologize for not including this information in the manuscript. We added the following section in to the manuscript:"Computational RequirementsIn order to train the model, we used the following hardware configuration: Nvidia RTX3090 GPU, AMD Ryzen 9 5950X 16 core CPU, and 32Gb of RAM memory. In our study, we used a batch size of 256, which occupied around 60% of GPU memory. Training of the model was dependent on the output variable. For metastatic disease prediction, we trained the model for approximately 4 hours. This could be changed since we used early stopping in order to prevent overfitting. By reducing the batch size to smaller numbers, the technical requirements are reduced making it possible to run GENIUS on most modern laptops."A general comment about the Methods section: Models, training, and validation are very vaguely described and the source code on GitHub is very poorly documented so that parameter choices, model validation, test and validation frameworks and parameter choices are neither clear nor reproducible.Apologies, we have updated the methods section with more details on models, training and validation. Additionally, we have moved the section on evaluating model performance from the methods section to the results section, with more details on how training was performed.We also agree that the GitHub page is not sufficiently detailed and well structured. To remedy this, we have made a new GitHub page that only has the code needed for analysis, example input data, example runs, and environment file with all library versions. The GitHub repository is also updated in the manuscript.https://github.com/mxs3203/GENIUSThe new GitHub page can be found on: Please provide a sufficient mathematical definition of the models, thresholds, training and testing frameworks.We sincerely apologize, but we do not entirely follow the reviewers request on this regard. The mathematical definitions of deep neural networks are extensive and not commonly included in research publications utilizing deep learning. We have used PyTorch to implement the deep neural net, a commonly used platform, which is now referenced in the methods. The design of the deep learning network used for GENIUS is described in figure 1, and the relevant parameters are described in methods. The hyper parameters are described in the methods section, and are as follows:"All models were trained with Adagrad optimizer with the following hyperparameters: starting learning rate = 9.9e-05 (including learning rate scheduler and early stopping), learning rate decay and weight decay = 1e-6, batch size = 256, except for memory-intensive chromosome images where the batch size of 240 was used."In chapter "Latent representation of genome" the authors write: "After successful model training, we extracted the latent representations of each genome and performed the Uniform Manifold Approximation and Projection (UMAP) of the data. The UMAP projected latent representations into two dimensions which could then be visualized. In order to avoid modeling noise, this step was used to address model accuracy and inspect if the model is distinguishing between variables of interest.". In the recent light of criticism when using the first two dimensions of UMAP projections with omics data, what is the evidence in support of the author's claim that model accuracy can be quantified with such a 2D UMAP projection? How is 'model accuracy' objectively quantified in this visual projection?We apologize for not clarifying this. The UMAP was done on L, the latent vector, which by assumption should capture the most important information from the “genome image”. In order to confirm this, we plotted the first two dimensions of UMAP transformation and colored the points by the output variable. If the model was capturing noise, there should not be any patterns on the plot (randomized cancer-type panel). Since, in most cases, we do see an association between the first two UMAP dimensions and the output variable, we were confident that the model was not modeling (extracting) noise.To clarify this, we changed the sentence in the manuscript so it is more clear that this is not an estimation of accuracy but only an initial inspection of the models:The UMAP projected latent representations into two dimensions which could then be visualized. In order to avoid modeling noise, this step was used to inspect if the model is distinguishing between variables of interest.In the same paragraph "Latent representation of genome" the authors write: "We observed that all training scenarios successfully utilized genome images to make predictions with the exception of Age and randomized cancer type (negative control), where the model performed poorly .". Did I understand correctly that all negative controls performed poorly? How can the authors make any claims if the controls fail? In general, I was missing sufficient controls for any of their claims, but openly stating that even the most rudimentary controls fail to deliver sufficient signals raises substantial issues with their approach. A clarification would substantially improve this chapter combined with further controls.We apologize for not stating this more clearly. Randomized cancer type was used as a negative control since we expect that model would not be able to make sense of the data if predicting randomized cancer type. As expected, the model failed to predict the randomized cancer types. This can be seen in Figure 2C, where UMAP representations  are made for each output variable. Not seeing any patterns in UMAP shows that, as expected, the model does not know how to extract useful information from “genome image” when predicting randomized cancer type (as when randomly shuffling the labels there is no genomic information to decipher). Similar patterns were observed for Age, indicating that patient age cannot be determined from the multi-omics data. Conversely, when GENIUS was trained against wGII, TP53, metastatic status, and cancer type, we observed that samples clustered according to the output label.Reviewer #2 (Public Review):In this manuscript, Birkbak and colleagues use a novel approach to transform multi-omics datasets in images and apply Deep Learning methods for image analysis. Interestingly they find that the spatial representation of genes on chromosomes and the order of chromosomes based on 3D contacts leads to best performance. This supports that both 1D proximity and 3D proximity could be important for predicting different phenotypes. I appreciate that the code is made available as a github repository. The authors use their method to investigate different cancers and identify novel genes potentially involved in these cancers. Overall, I found this study important for the field.The major points of this manuscript could be grouped in three parts:1. While the authors have provided validation for their model, it is not always clear that best approaches have been used.a) In the methods there is no mention of a validation dataset. I would like to see the authors training on a cancer from one cohort and predict on the same cancer from a different cohort. This will convince the reader that their model can generalise. They do something along those lines for the bladder cancer, but no performance is reported. At the very least they should withhold a percentage of the data for validation. Maybe train on 100 and validate on the remaining 300 samples. They might have already done something along these lines, but it was not clear from the methods.Apologize for not being sufficiently clear in the manuscript. We did indeed validate the performance within the TCGA cohort, using holdout cross validation. Here, we trained the network on 75% of the cohort samples (N = 3825), and tested on the remaining 25% (N = 1276).To make this more clear, we have rewritten section “GENIUS classification identifies tumors likely to become metastatic” as such:"The omics data types included somatic mutations, gene expression, methylation, copy number gain and copy number loss. Using holdout type cross-validation, where we split the data into training (75%) and validation (25%), we observed a generally high performance of GENIUS, with a validation AUC of 0.83 for predicting metastatic disease ."We also added the following sentence in the legend of Figure 2:"The x-axis represents epochs and y-axis represents AUC score of fixed 25% data we used for accuracy assessment within TCGA cohort."The accuracy of GENIUS could not be validated on the other two bladder cohorts since they do not contain all the data for the creation of five-dimensional genome images. However, we were able to investigate if the genes with the highest attribution scores towards metastatic bladder cancer obtained based on the TCGA samples also showed a significant association with poor outcome in the two independent bladder cancer cohorts. Here, we observed that of the top 10 genes with the highest attribution scores, 5 were associated with poor outcome in the early stage bladder cancer cohort, and 7 were associated with poor outcome in the late stage/metastatic bladder cancer cohort.b) It was not clear how they used "randomised cancer types as the negative control". Why not use normal tissue data or matched controls?In the study, we built six models, one for each variable of interest. One of them was cancer type which performed quite well. In order to assess the model on randomized data, we randomized the labels of cancer type and tried predicting that. This served as “negative control” since we expected the model to perform poorly in this scenario. To make this more clear in the manuscript, we have expanded the description in the main text. We have also added the description of this to each supplementary plot to clarify this further.While normal tissue and matched controls would have been an optimal solution, unfortunately, such data is not available.c) If Figure 2B, the authors claim they have used cross validation. Maybe I missed it, but what sort of cross validation did they use?We apologize for not being sufficiently clear. As described above, we used holdout cross-validation to train and evaluate the model. We clarified this in the text:"Using holdout type cross-validation, where we split the data into training (80%) and validation(20%), we observed a generally high performance of GENIUS, with a mean validation AUC of0.83 "1. Potential improvement to the methoda) It is very encouraging the use of HiC data, but the authors used a very coarse approach to integrate it (by computing the chromosome order based on interaction score). We know that genes that are located far away on the same chromosome can interact more in 3D space than genes that are relatively close in 1D space. Did the authors consider this aspect? Why not group genes based on them being located in the same TAD?We thank the reviewer for this suggestion and we will start looking into how to use TAD information to create another genome representation. In this study, we tried several genome transformations, which proved to be superior compared to a flat vector of features (no transformation). We are aware that squared genome transformation might not be optimal, so we designed the network that reconstructs the genome image during the training. This way, the genome image is optimized for the output variable of choice by the network itself. However, we note that the order of the genes themselves, while currently based on HiC, can be changed by the user. The order is determined by a simple input file which can be changed by the user with the argument “all_genes_included”. Thus, different orderings can be tested within the overall square layout. This is now detailed in the instructions on the new GitHub page.The convolutional neural network uses a kernel size of 3x3, which captures the patterns of genes positioned close to each other but also genes that are far away from each other . Once convolutions extract patterns from the image, the captured features are used in a feed-forward neural network that makes a final prediction using all extracted features/patterns regardless of their location in the genome image.We also inserted the following sentence in discussion:"Given that spatial organization improved the prediction, we recognize that there may exist a more optimal representation of multi-omics data which should be explored further in future work. Potential methods for organizing gene orientation in a 2D image could consider integrating topologically associating domains[39] along with the spatial information from HiC. This is already possible to explore with the current implementation of GENIUS, where gene layout can be set manually by the user."https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02728-5. What would happen if they were not considered together?b) Authors claim that "given that methylation negatively correlates with gene expression, these were considered together". This is clearly not always the case. See for example We thank the reviewer for this insightful comment. We agree with the reviewer that methylation does not always result in lower expression, although methylation levels in most cases should correlate negatively to RNA expression, but with a gene-specific factor. Indeed, there are tools developed that infer RNA expression based on methylation, making use of gene-specific correction factors. E.g. Mattesen et al .However, upon reflection we agree with the reviewer that we cannot assume for all genes that methylation equals low expression. Therefore, we have performed an analysis where we compared the methylation level to gene expression levels for all tested genes within bladder cancer. We computed Pearson’s correlation of 16,456 genes that have both methylation and expression scores. Of these, 8528 showed a negative correlation. After p-value correction, this resulted in 4774 genes where methylation was significantly negatively associated with expression. For these genes we performed the subsequent analysis in bladder cancer, where methylation and expression were considered together. This updated analysis has been included in supplementary figure 10, and the results section has been amended to reflect this. Overall, this analysis resulted in 4 of 10 genes being replaced in the downstream analysis. However, we note that the final results did not materially change, nor did the conclusions.1. Interesting results that were not explained.a) In Figure 3A methylation seems to be the most important omics data, but in 3B, mutations and expression are dominating. The authors need to explain why this is the case.We apologize for not explaining this in more detail. Figure 3B shows the attribution scores scaled within the cancer type, where Figure 3A shows raw attribution scores for each data source included. The reason for this is that methylation and expression have in general, smaller attribution scores but more events where a single mutation often is characterized with large attribution scores and the rest of them with very small attribution. In order to make those numbers comparable and take into account biological differences between the cancer type, we scaled the scores within each cancer type.To make this more clear we modified the first sentence in “Interpreting the GENIUS model classifying metastatic cancer biology” section:"Analysing raw attribution scores we concluded the most informative data type overall regarding the development of metastatic disease was methylation .…We also noticed that mutation data often had a single mutation with large attribution score where expression and methylation showed multiple genes with high attribution scores… … The normalization step is crucial to make results comparable as underlying biology is different in each cancer type included in the study."Reviewer #1 (Recommendations For The Authors):While I appreciate the creative acronym of the presented software solution (GENIUS), it may easily be confused with the prominent software Geneious | Bioinformatics Software for Sequence Data Analysis which is often employed in molecular life science research. I would suggest renaming the tool.We appreciate the comment but prefer to keep the name. Given that the abbreviation is not exactly the same and the utility is different, we are confident that there will be no accidental mixup between these tools.A huge red flag is the evaluation of the input image design which clearly shows that classification power after training is insufficient for three out of four image layouts (and even for the fourth AUC is between 0.70-0.84 depending on the pipeline step and application). Could the authors please clarify why this isn't cherry-picking (we use the one layout that gave some form of results)? In light of the poor transformation capacity of this multi-omics data onto images, why weren't other image layouts tried and their classification performance assessed? Why should a user assume that this image layout that worked for this particular input dataset will also work with other datasets if image transformation is performing poorly in most cases?We apologize for not describing this further in the manuscript. We wrote in the manuscript that we could not know what genome representation is optimal as it is difficult to know. A flat vector represents a simple (or no) transformation since we simply take all of the genes from all of the data sources and append them into a single list. Chromosome image and square image are two transformations we tried, and we focused on the square image since in our hands it showed superior performance relative to other transformations.Reviewer #2 (Recommendations For The Authors):Minor points:1. Legends of supplementary Figures are missing.We thank the reviewer for this comment and apologize for missing it. All legends have been added now.1. For some tests the authors use F1 score while for other AUC, they should be consistent. Report all metrics for all comparisons or report one and justify why that only metric.We apologize for not being sufficiently clear. AUC is a standard score used for binary classification, while the F1 score is used for multiclass classification. We have now described this in the methods section, and hope this is now sufficiently clear."When predicting continuous values, the model used the output from the activation function with the mean squared error loss function. When predicting multi-class labels, the performance measure was defined by the F1 score, a standard measure for multiclass classification that combines the sensitivity and specificity scores and is defined as the harmonic mean of its precision and recall. To evaluate model performance against the binary outcome, ROC analysis was performed, and the area under the curve (AUC) was used as the performance metric."1. not sure how representation using UMAP in Figure 2C is helping understand the performance.Apologies for the poor wording in the results section. The purpose of the UMAP representation was to visually inspect if the model was distinguishing between variables of interest, not to estimate model performance. We have rephrased the text in the methods section to make this clear:"After successful model training, we extracted the latent representations of each genome and performed the Uniform Manifold Approximation and Projection (UMAP) of the data for the purpose of visual inspection of a model."And"In order to avoid modeling noise, this step was used to inspect if the model is distinguishing between variables of interest."And also in the results section:"In order to visually inspect patterns captured by the model, we extracted the latent representations of each genome and performed the Uniform Manifold Approximation and Projection (UMAP) of the data to project it into two dimensions."1. Instead of pie chart in 3A, the authors should plot stacked barplots (to 100%) so it would be easier to compare between the different cancer types.We thank the reviewer for the suggestion; however, since we wanted to compare the relative impact of each data source with each other, we used pie charts. Piecharts are often better for describing relative values, whereas bar plots are better for absolute values.ReferencesDu, Ruishan, Wenhao Liu, Xiaofei Fu, Lingdong Meng, and Zhigang Liu. 2022. “Random Noise Attenuation via Convolutional Neural Network in Seismic Datasets.” Alexandria Engineering Journal 61 (12): 9901–9.Jang, Hojin, Devin McCormack, and Frank Tong. 2021. “Noise-Trained Deep Neural Networks Effectively Predict Human Vision and Its Neural Responses to Challenging Images.” PLoS Biology 19 (12): e3001418.Mattesen, Trine B., Claus L. Andersen, and Jesper B. Bramsen. 2021. “MethCORR Infers GeneExpression from DNA Methylation and Allows Molecular Analysis of Ten Common Cancer Types Using Fresh-Frozen and Formalin-Fixed Paraffin-Embedded Tumor Samples.” Clinical Epigenetics 13 (1): 20.


Recent advances in multimodal single-cell omics technologies enable multiple modalities of molecular attributes, such as gene expression, chromatin accessibility, and protein abundance, to be profiled simultaneously at a global level in individual cells. While the increasing availability of multiple data modalities is expected to provide a more accurate clustering and characterization of cells, the development of computational methods that are capable of extracting information embedded across data modalities is still in its infancy.We propose SnapCCESS for clustering cells by integrating data modalities in multimodal single-cell omics data using an unsupervised ensemble deep learning framework. By creating snapshots of embeddings of multimodality using variational autoencoders, SnapCCESS can be coupled with various clustering algorithms for generating consensus clustering of cells. We applied SnapCCESS with several clustering algorithms to various datasets generated from popular multimodal single-cell omics technologies. Our results demonstrate that SnapCCESS is effective and more efficient than conventional ensemble deep learning-based clustering methods and outperforms other state-of-the-art multimodal embedding generation methods in integrating data modalities for clustering cells. The improved clustering of cells from SnapCCESS will pave the way for more accurate characterization of cell identity and types, an essential step for various downstream analyses of multimodal single-cell omics data.https://github.com/PYangLab/SnapCCESS under the open-source license of GPL-3. The data used in this study are publicly available (see section ‘Data availability’).SnapCCESS is implemented as a Python package and is freely available from  The development of novel single-cell technologies, such as cellular indexing of transcriptomes and epitopes by sequencing (CITE-seq)  which implements MOFA+ , pre-normalized datasets were first used to create the mofa object using create_mofa function. The data were then used as input for the modal training using run_mofa function with default parameters. We use the “get factors” function with factors = “all” to obtain the embeddings for each input dataset.Jvis-learn: Jvis-learn (v0.0.12) (v0.0.12)  was empltotalVI: totalVI is designed for anayzing CITE-seq data , the raw count matrices of RNA and ADT were first normalized using the normalize_total and log1p functions and then top 4000 most variable genes were selected using the highly_variable_genes function. The data were input for model training using scvi.model.TOTALVI.setup_anndata, scvi.model.TOTALVI, and train functions in scvi-tools. The latent space of RNA and ADT modalities was generated using the get_latent_representation function.MultiVI: MultiVI is a sibling of totalVI and specifically designed for anayzing data with RNA and ATAC modalites , the raw count matrices of RNA and gene activity score matrices from ATAC and the paired matrix of RNA and ATAC were utilized as input. These data were first concatenated using the organize_multiome_anndatas function in scvi-tools and then used for model training using scvi.model.MULTIVI.setup_anndata, scvi.model.MULTIVI and train functions in scvi-tools. The latent space of RNA and ATAC modalities was generated using the get_latent_representation function.Ramaswamy CITE-seq dataset  and 20 cell types for the subsequent analyses.Ma SHARE-seq dataset  and 23 cell types for the subsequent analyses.Swanson TEA-seq dataset .Each data modality was first normalized by a S be a set of N cells, then a clustering U on S is a way of partitioning S into non-overlap subset R cell types, C clusters generated by a clustering. Pair counting-based measures can be used for counting pairs of items on which the partition U and V agree or disagree. Specifically, the S can be classified into one of the four types: (i) N11: the number of pairs that are in the same partition in both U and V; (ii) N00: the number of pairs that are in different partitions in both U and V; (iii) N01: the number of pairs that are in the same partition in U but in different partitions in V; (iv) N10: the number of pairs that are in different partitions in U but in the same partition in V. Following this, ARIand NMI can be defined as follows:where U and V, defined asand For evaluating clustering performance, we used adjusted Rand index (ARI) and normalized mutual information (NMI) to evaluate the clustering concordance with respect to pre-defined cell-type annotations from their original studies . We then assessed each set for its concordance of clustering output and the pre-defined cell-type labels. We found in all six datasets a higher clustering performance of SnapCCESS (epoch = 1) with an ensemble of 50 compared to those without using the ensemble in both the major and minor partitions of the cell types . Lastly,k-means clustering algorithm on either the integrated embedding of multimodality or embeddings from each unimodality. We found that in general clustering of cells using integrated embeddings of multimodality leads to significantly better results than from using any unimodality alone  by coupling it with a spectral clustering algorithm and SIMLR, a kernel-based clustering algorithm. Note that the two large CITE-seq datasets, Stephenson CITE-seq and Hao CITE-seq, were excluded due to the exponential growth of computational complexity with the number of cells in a dataset for the spectral clustering algorithm. Overall, we observed a clear increase in clustering performance with the increasing ensemble size, regardless of the types of clustering algorithms and concordance evaluation metrics . Nonethek-means clustering, the improvement from SIMLR and spectral clustering also peaked around the ensemble size of 50  with TotalVI on the CITE-seq datasets, MultiVI on SHARE-seq and SNARE-seq datasets, and MOFA+ and Jvis-learn on all six datasets with two or three modalities. In particular, we used the multimodality-integrated embeddings generated from each of these methods as input to sing ARI  and NMI sing ARI .k-means and spectral clustering algorithms performed reasonably well, the use of SIMLR on Jvis-learn generated embeddings leads to poor results in many cases. These results may indicate the varying degree of generalizability of the multimodality embedding generation methods on different clustering algorithms. To this end, multimodality embeddings generated from the other four methods  worked well regardless of the used clustering algorithm.We found that SnapCCESS performed competitively to other multimodality embedding generation methods. In most cases, its performance is significantly better than totalVI when applied to CITE-seq datasets and MultiVI when applied to SHARE-seq and SNARE-seq datasets . While MA main challenge in single-cell omics data analysis is in handling the high-dimensionality of the feature space (A key innovation in SnapCCESS is the adaptation of the snapshot ensemble learning technique (The utility of the embeddings generated from multimodal single-cell omics data is much wider than cell clustering. While clustering is one application that can make use of embeddings learned from such data, other tasks such as supervised cell-type classification (While the current study evaluates the SnapCCESS framework for clustering cells into discrete groups, the cell-type structures from many biological systems are hierarchical, with subpopulations of cells existing in each major cell type (In summary, our previous work demonstrated that ensemble learning of embeddings provides an effective approach for improving downstream clustering analyses by providing a multi-view of the input data (btad382_Supplementary_DataClick here for additional data file.