Recent advances in machine learning research, combined with the reduced sequencing costs enabled by modern next-generation sequencing, paved the way to the implementation of precision medicine through routine multi-omics molecular profiling of tumours. Thus, there is an emerging need of reliable models exploiting such data to retrieve clinically useful information. Here, we introduce an original consensus clustering approach, overcoming the intrinsic instability of common clustering methods based on molecular data. This approach is applied to the case of non-small cell lung cancer (NSCLC), integrating data of an ongoing clinical study (PROMOLE) with those made available by The Cancer Genome Atlas, to define a molecular-based stratification of the patients beyond, but still preserving, histological subtyping. The resulting subgroups are biologically characterized by well-defined mutational and gene-expression profiles and are significantly related to disease-free survival (DFS). Interestingly, it was observed that (1) cluster B, characterized by a short DFS, is enriched in KEAP1 and SKP2 mutations, that makes it an ideal candidate for further studies with inhibitors, and (2) over- and under-representation of inflammation and immune systems pathways in squamous-cell carcinomas subgroups could be potentially exploited to stratify patients treated with immunotherapy. While clinical and imaging features can help the clinician distinguish NSCLC from SCLC, histopathologic features and immunohistochemical markers are required to make further subtyping2. In recent years, the renewed interest in agents able to relieve the immunosuppression state observed in many human cancers, namely the immune-checkpoint inhibitors4, has paved the way to further improvements. Despite these achievable survival benefits, the outlook for the majority of patients is still unsatisfactory and more collaborative and comprehensive approaches are needed.Lung cancer, and in particular NSCLC, has benefited from the clinical implementation of precision medicine, since it has become clear that the introduction of routine molecular testing, which might lead to targeted therapies, in clinical practice has significantly contributed to an initial decrease of patients’ mortality, with 5 year-survival rates near to 22% nowadays, as compared to 17% few years ago6. Understanding this progression is the unifying goal of much cancer research which aims at improving patients’ management strategies9.Tumour evolution is the result of genomic instability caused by somatic mutations, chromosomal rearrangements, copy number alterations and epigenetic changes, resulting in loss of tumour suppressor genes and activation of oncogenes that disengage the affected cells from their regulatory cycles10, combined with cheaper sequencing costs enabled by modern next-generation sequencing (NGS) technology, brought to a new era of bioinformatics. In this renovated context, scientists are challenged to analyse huge amounts of data to unveil the molecular mechanisms underlying complex pathologies such as cancer. One of the major aims of such analyses is represented by disease subtyping, which consists in stratifying patients into different clusters based on similarities and differences between their molecular profiles. Several clustering models designed for single- and multi-omics data have been developed and can be roughly divided in three main categories: variance-based, similarity-based, and network-based models. All models must face the dimensionality issue deriving from the high number of genomic features, which are typically orders of magnitude larger than the number of samples in the dataset11.The recent advances in artificial intelligence (AI) and machine learning (ML) research., which dimensions catch most of the variance of original data. Among these models, it is worth mentioning iCluster12 and MOFA13, with their subsequent enhancements iClusterPlus14, iClusterBayes15 and MOFA+ 16 respectively, JIVE17 and intNMF18. Similarity-based models solve the dimensionality issue by applying kernels to the original data, generating a similarity matrix between patients through well-known methods such as spectral clustering19. The most known of such models are SNF20, ANF21 and SRF22. Finally, network-based models build the network or multi-network of interactions between genes which is then exploited to get meaningful features for clustering. This is for example the case of NBS23 and SALMON24.Variance-based models implement an initial step of dimensionality reduction in which raw data is projected in a lower-dimensional subspace; i.e25, such as the Silhouette index, which are also not conclusive, since each CVI is biased toward models, utilising similar metrics of the adopted CVI26. All these features are particularly critical in the case of non-small cell lung cancer (NSCLC), which is a highly heterogenous tumour27, with molecular profiles characterised by a large biological variability leading to poor performance of the prognostic models29.A major and common issue to all the models is that they rely on many parameters which are hard to be tuned when solving an unsupervised learning task, whereas they may strongly affect the resulting clustering. Moreover, each of the above-mentioned categories are focused on different aspects of the data, depending on the structure and the metrics of the model itself, and focusing on a single aspect may be limiting for finding general subgroups. The final choice of the model and parameters is often guided using cluster validity indexes (CVI)30, which already proved excellent performance with other multi-omics datasets31 and it is here applied for the first time to NSCLC, revealing well characterised molecular subtypes with prognostic implications. Such approach consists in training several diverse and independent clustering models (learners) and integrating their results through a voting process, deriving a final clustering where labels are those with the most support from the community of independent learners. It should be noted that the proposed approach has a general validity and might be easily customised and adapted to other cancer types.To address all these issues, we adopted a cutting-edge ensemble learning approach, known as consensus clustering32 (which is part of a wider research project named DEFLeCT) designed to correlate follow-up, clinical and pathological information with molecular features including mutations, translocations, copy number variations and gene expression profiles respectively collected by exome and bulk-RNA sequencing  sub-group of NSCLC patients recruited in the prospective observational clinical PROMOLE studyIn summary, relying on a cutting-edge ensemble learning approach and applied to well-established (TCGA) as well as newly acquired (PROMOLE) data, the present study increases the current understanding of NSCLC, providing a novel and robust molecular-based stratification of such disease, with well-defined molecular profiles that represent a potential signature against which to test new patients and with prospective implications in immunotherapy.The gene expression profiles of PROMOLE samples n = 81) were jointly normalised and integrated with TCGA data (n = 889) see “”. The re were joiUsing the above-mentioned gene-expression profiles, we adopted an ensemble learning approach to investigate whether novel biological subgroups could be found by consensus clustering see “”. Such ap-value of p < 0.001). Overall, survival analysis proved that the identified subgroups were characterised by distinct clinical outcomes in terms of DFS. To provide a clinical characterisation of the subgroups, we tested the relative abundances of the main clinical variables for each emerging cluster with χ2 test  of each identified subgroup Fig. a. DFS wadel Fig. b to evaldel Fig. ”. Intereentclass1pt{minima34 for each of the four clusters separately, versus all the remaining samples. Results are reported in Fig. To investigate the biological meaning of the identified molecular subtypes, we applied a gene-set enrichment analysis (GSEA) to investigate possible enriched functions associated to each of the four consensus clusters see “”. In so Overall, cluster C appeared to be the most enriched in functions compared to the rest of the samples . Details of the most representative enriched pathways can be found on Supplementary Fig. . signal transduction, protein expression and translation control, regulation of protein activity; see MEMBRANE_TRANSPORT, EPITHELIUM and MIGRATION related terms resulted to be commonly modulated in all the four cluster, even if positively in cluster C or negatively in clusters A, B and D. In addition, also the functional macro-categories DNA_and_RNA_PROCESSING and CELL_PROLIFERATION were among the top-enriched ones, suggesting a common dysregulation of proliferative genes in all the samples we profiled. CYTOSKELETON, VASCULAR, MAPK pathways genes and general IMMUNE_SYSTEMS/INFLAMMATION gene sets are instead significantly linked only with cluster B and D. Several METABOLISM-associated gene sets resulted significantly negatively enriched only in cluster A, B and D. Gene sets correlated with WNT_PATHWAY were specifically associated only to cluster B and C. The MOLECULAR_PROCESSES macro-category groups several gene sets associated with the major cellular functions  were significantly overexpressed in LUSC compared to LUADWe also characterised the mutational landscape of the identified molecular subgroups by quantifying over- and under-represented mutated genes of each cluster through Z-scores analysis Fig. . To incrKRAS mutations41 but also showed well distinct features, namely: cluster A was characterised by over-represented mutation of EGFR oncogene and interestingly a significative (p < 0.001) under-representation of KEAP1 and NFE2L2 mutations, which has already been found to be mutually exclusive with EGFR42. KEAP1 mutations, which has been identified as a potential prognostic factor in resected NSCLC43, were instead over-represented in cluster B, which we found to be as the one with the poorest outcome in terms of DFS  mutations of well-known BRAF and SMAD4 genes. Cluster B was characterized by over-represented mutations in established or candidate tumour suppressor genes such as STK1144 and RASSF845 and oncogenes such as SKP246 and FAM83B, which participates in the EGFR signalling pathway and may activate both the EGFR itself and downstream RAS/MAPK and PI3K/AKT/TOR signalling cascade47. Moreover, cluster B was characterised by genetic alterations of ERBB4, and DNA-repair genes such as ATM and PANK1, where the first is co-regulated by the tumour suppressor TP53, while the second functions as a regulator of a wide variety of downstream proteins, including tumour suppressor proteins p53 and BRCA1, checkpoint kinase CHK2, checkpoint proteins RAD17 and RAD9, and DNA repair protein NBS1. Cluster C and D shared major features of LUSC such as PIK3CA and tumour suppressor TP53, PTEN and CDKN2A mutations, which were instead significatively under-represented in cluster A and B. Cluster D was well characterised by significant (p < 0.001) over- and under- representation of NFE2L2 and EGFR mutually exclusive mutations respectively and NRAS over-representation while cluster C exhibited unique over-representation of mutated RASA1, which SH2 domain has been identified as a probably oncogenic48, and CIB1 and FANCC, which are involved in post-replication DNA repair49. It is important to note that other relevant mutations typically associated with lung cancer  have been found with frequencies compatible with previously published data, but without a significant difference between subgroups.From a mutational perspective, the identified subgroups presented well-defined and recognisable genetic alterations. First of all, clusters A and B shared major features widely associated with LUAD, such as 50.In clinical practice, histological classification of lung cancer remains one of the cornerstones of the diagnostic work-up coupled with the assessment of the extent of the disease that is strictly related to survival probability31. Bioinformatics approaches are clearly advocated to address this complexity and the clinical value of identifying similar characteristics across different omics is at the root of the development of new computational methods implementing different strategies to analyse several omics datasets simultaneously.As the mechanisms of tumour onset and progression are not restricted to a single molecular level, but rely on a huge number of molecular networks, which in turn sustain intercellular circuits, the detection of groups of patients showing similar characteristics across different omics is a key issue to enable personalized medicine, which aims to offer patients a treatment adapted to the characteristics of their tumours. noise, bias and variance) with respect to traditional single-model approaches, and have the major advantages of resulting in more accurate and especially more stable and robust outcomes. This is crucial in the case of such a multivariate problem as omics data analysis, typically characterised by the so-called curse of dimensionality, i.e. datasets with a number of features (genes) way above the number of samples (patients). Moreover, ensemble learning may be used to capture both linear and non-linear emerging patterns within data by exploiting different classes of models.In this context, the main novelty and strength of our approach lies in the effective application of an ensemble learning algorithm to clustering of molecular profiles, that combines multiple state-of-the-art and well-accepted models (such as iClusterPlus and SNF), which are then aggregated through a begging-like majority voting strategy. Ensemble methods, in fact, have been specifically designed to reduce the three main sources of error in machine learning  algorithms as proposed by Sabbatini and Manganaro52. Some gene expression signatures to predict NSCLC outcome have been identified55, but poorly characterized in relationships to the histological subtypes LUAD and LUSC. In this context, the PROMOLE study and the proposed approach represents a further step, well-integrated with what has already been explored in MATCH56 or BATTLE-1 and 257 or in the European Proof-of-Concept Therapeutic Stratification Trial of Molecular Anomalies in Relapsed or Refractory Tumors for paediatric tumours (ESMART) trial platform58 in which tumour biopsy specimens from adult or paediatric patients are sequenced and used to match patients to relevant treatment arms under defined study protocols.In the last 20 years the detection of specific gene mutations/rearrangements/amplification in NSCLC have been associated with specific targeted therapies which significantly impacted on survival expectations in any stage of lung cancerOur findings have identified four distinct NSCLC subtypes, each associated with unique clinical and molecular features that represent a potential signature against which to test new patients. Such grouping is well supported by a clear relation with DFS, which results in significantly separated survival curves, and which is effectively proved by Cox regression analysis. In fact, it has to be emphasised that the contribution of histology in such analysis is totally masked by stronger impact of the identified molecular subgroups, that overwrite the histological signal. However, it is worth clarifying that the data presented here has to be considered an exploratory cohort that need validation through a confirmatory cohort matched for the major clinical and pathological characteristics to make robust our preliminary observation. As such, while it is unlikely the use of the four clusters as stratification factor for the next generation of clinical trials, they could predict the response to specific treatment such as immunotherapy or targeted therapies.4. Moreover, it should be noted that the differences between cluster C and D, both belonging to LUSC group, are particularly impressing. Cluster D shows a much better DFS score compared to cluster C and it is characterized by completely different regulation of gene pathways. For instance, inflammation and immune systems gene pathways are over-represented in cluster C compared to D. Although we do not have any experimental evidence linking patient DFS with expression of genes related to immune defence, we can speculate that our consensus cluster analysis might be able to separate LUSC with high level of inflammatory cytokines and much more infiltrated by immune cells, characterized by better DFS, and LUSC less immunoreactive (i.e. cold) with a worse prognosis. If this hypothesis will be verified it could be exploited to stratify patients treated with immunotherapy. Additionally, other insights from our classification could open new therapeutic perspectives. In particular, cluster B, characterized by a short DFS, is enriched in KEAP1 and SKP2 mutations. Interestingly, it has been recently reported that small molecule inhibitors of NRF2 pathway sensitize cancer cells carrying KEAP1 mutants to chemotherapy59 and that inhibition of SKP2 sensitizes lung cancer cells to paclitaxel61. Furthermore, LUAD and LUSC classification based on gene expression indicates differential biological mechanisms of cancer onset and progression that in further studies could be mirrored in clinical aspects of NSCLC tumours that stem from abrogated evolutionary signalling and its downstream outcomes.The clinical correlate of the data presented indicates that the four different identified clusters are summarising what was already known in terms of prognostic information of specific gene mutations and it is questing for the inclusion of molecular information in the next edition of the TNM classification. It should be noted that none of the PROMOLE patients received specific targeted treatments and, consequently, the prognostic impact on DFS is prevalent. With the widespread advent of immunotherapies, it is progressively evident that some genomic correlates, independently from the histological subtypes, are negatively modulating the therapeutic effect of such novel therapies, such for instance STK11 mutation. Beyond shaping the morphological classification of NSCLC by specific genomic alteration in generating a deeper distinction between LUAD and LUSC, there were also potential useful information derived by different clustering of sets of genes leading to a candidate DNA damage-related gene signature associated with an enrichment for immune-related genes in cluster C, making more likely a better response to immunotherapies as already seen in earlier trials with nivolumabThe definition of four clusters, characterized by different mutational and transcriptomic profiles, opens the question of what are the molecular determinants and pathways that determine the difference between the groups in terms of DFS, that we partly tried to address with the present preliminary study. However, even within the limits of current preclinical models such as cell lines or patient-derived xenografts that cannot recapitulate the role of the microenvironment, the comparative evaluation between the clusters defined in this study and the transcriptomes of preclinical models of NSCLC and the subsequent genetic or pharmacological manipulation with approaches of gain- or loss-of-function could help to identify molecular pathways drivers and new genetic vulnerabilities that explain the different clinical outcome. Depending on the resources available, these experiments will be part of our future research plan.Normal and tumours tissues were collected from 81 resectable patients having histologically confirmed stage IB-IIIA NSCLC and enrolled in a prospective observational clinical trial (PROMOLE) approved by the Ethical Committee of the hospital AOU San Luigi Gonzaga   and conducted in accordance with the Declaration of Helsinki. Written informed consent was collected for each participant. At the time of surgical resection, a portion of the tumoral and normal tissue isolated at least 2 cm far from the macroscopical border of the cancer lesion were collected and stored in RNAlater (Thermo) at − 80 °C for subsequent DNA and RNA extraction. Genomic DNA and RNA extractions were respectively performed by Maxwell-RSC Tissue DNA and Maxwell-RSC simply RNA tissue kits (Promega) following manufacturer’s instruction. From RNA samples residual DNA was removed by treatment with RNase-Free DNase Set (QIAGEN). Quality and quantification of nucleic acids was assessed by 2100 Bioanalyzer (Agilent) and Qubit assays (Thermo).Exome-capture was performed using the SureSelectXT Human All Exon Kit, according to the manufacturer’s instructions (Agilent Technologies). Whole-exome sequencing was performed with the Illumina NovaSeq 6000 (Illumina Inc.) platform with a depth of 35 M and 70 M paired-end reads for normal and tumour samples respectively. In order to identify tumour-specific mutations, we compared each tumour sample to the corresponding normal DNA from normal tissue.Library was generated with TruSeq® Stranded mRNALibrary (Illumina) and sequenced on a NextSeq 500/550 HighOutput Kit v2.5 flow cell (Illumina), obtaining a mean of 20 million 35–151 bps single-end reads per sample.63, NGS sequencing of PROMOLE exomes have been processed through a bioinformatic pipeline in order to determine the mutational landscape of NSCLC patients. In the present study, we adopted the workflow of the IDEA pipeline64 which allows to identify: single nucleotide variants (SNV), insertions or deletions (INDEL) and gene copy-number alterations (CNA). The parameters of the pipeline have been tuned in order to minimise the batch-effect with respect to TCGA data and ensure integrability of the two datasets. The results of somatic variant calling have been benchmarked against that of the well-established Genome Analysis ToolKit (GATK) Mutect265 and found to be coherent. Annotation of SNV and INDELS is based on the Catalogue Of Somatic Mutations In Cancer (COSMIC) (https://cancer.sanger.ac.uk/cosmic).After initial quality check63 and processing with Cutadapt66, RNA-seq data have been processed to calculate the gene expression levels following the workflow described by Korpelainen67. Reads alignment to hg38 reference and assembly has been implemented with STAR68 and RSEM69 (the same tool used in the legacy workflow of TCGA transcriptomes70). The transcripts per million (TPM) abundance have been compared and found coherent with the median TPM of the 100 most expressed genes in lung tissues reported in the Genotype-Tissue Expression (GTEx) .After initial quality checkhttps://portal.gdc.cancer.gov/legacy-archive/search/f), selecting all the samples related to TCGA-LUAD and TCGA-LUSC projects from the Pan-Cancer Atlas cohort. Following the same procedure adopted for PROMOLE samples, we filtered the subset of the protein coding genes common to the two datasets and transformed gene expression data into TPM, using the gene lengths reported in the GRCh38.p12 version of the reference genome. TCGA data also includes 33 cases of stage IV NSCLC.TCGA tumour and normal raw counts data have been downloaded from the Genomic Data Common (GDC) portal  is 2.7%, which can be considered negligible different clustering methods are run independently on the same data and the resulting cluster associations are obtained for each patient and method;clustering associations are compared and a voting process is established, in order to detect the most supported label for each patient.We implemented an ensemble learning model based on consensus clustering applied to log2-scaled gene expression data. The method is deployed in two phases  or Kohonen’s Map77 before clustering. In fact, SOMs are used to produce a low-dimensional representation of a higher dimensional dataset still preserving its topological structure and already proved to be effective on gene-expression data78, in combination with K-Means, hierarchical and consensus clustering, for different types of tumour subtyping tasks80. SOMs were not applied on iClusterPlus and SNF since these methods were already designed to deal with high-dimensional genomic matrices.In our study, the ML clustering methods employed for consensus clustering cover most of the state-of-the-art and well-established algorithms. Specifically, we employed K-Means clusteringEach method was employed with three different seeds, in order for it to be tested with three different initialization points. Moreover, four maximum number of clusters were employed  have been compared among the four clusters. The relative abundances of such variables for each emerging cluster have been tested using χsee Fig.  test andNull models have been built via bootstrap resampling of the consensus clustering labels and provide the statistically expected abundance of each variable in each cluster, if no correlation is assumed, given the distribution of the variable itself over the whole dataset and the numerosity of samples in each cluster. In particular, the consensus labels have been resampled 1000 times, finally computing the average abundance of each variable for each cluster and its standard deviation. The Z-score is built as the difference between observed and relative abundance in units of standard deviation.82 was defined as the time from the initial treatment to recurrence of tumor. The null hypothesis that there is no difference among clusters in the probability of cancer recurrence has been tested with a log-rank test83.The DFS84 was fitted to identify significant regressors among clinical features and clustering. The following clinical variables have been tested: histological subtype (LUSC or LUAD), age at diagnosis , sex, tumour stage, smoking history  and clustering. Only the significant regressors have been tested in the final model to calculate the Hazard Ratios (HR). Cox models, HR and Kaplan–Meier curves have been computed and plotted using the R survival85 and survminer86 packages. Please note that, since information about treatment received was missing for the vast majority of the patients (~ 70%), we did not distinguish between groups receiving different treatments. However, we observed that the percentage of patients receiving the same treatment was approximately uniform across the four clusters.A Cox regression model87 desktop application from the BROAD Institute (https://software.broadinstitute.org/cancer/software/gsea/wiki/index.php/Main_Page), with standard parameters. The gene set databases “GeneOntology–BiologicalProcesses”75, “KEGG pathways” and “HALLMARK of Cancer” from the Molecular Signatures Database34  were used as reference for the annotation. In all the cases, we loaded the entire RNA-seq dataset in the GSEA application and we performed a comparison between each of the four clusters versus the remaining samples. Only results associated with an FDR q-value < 0.1 were retained for further analysis. Each macro-category was reported in Fig. Functional annotation of the four main transcriptional clusters was based on the GSEAWe performed an over-representation analysis of the mutations characterising each subgroup by testing the observed number of mutations for each gene and each emerging cluster with Z-tests against their respective null models. Null models have been built following the same procedure described in the case of clinical features.The prospective observational clinical trial PROMOLE has been approved by the Ethical Committee of the hospital AOU San Luigi Gonzaga  . Written informed consent has been prospectively collected for each participant.Supplementary Information 1.Supplementary Information 2.


Despite advancements in the available radiological investigation techniques, not all cases of Distant Metastasis (DM) are diagnosed at initial clinical presentation. Also, there are currently no standard biomarkers of metastasis. Early, accurate diagnosis of DM is however crucial for clinical decision making, and planning of appropriate management strategies. Previous works have achieved little success in attempts to predict DM from either clinical, genomic, radiology, or histopathology data. In this work we attempt a multimodal approach to predict the presence of DM in cancer patients by combining gene expression data, clinical data and histopathology images. We tested a novel combination of Random Forest (RF) algorithm with an optimization technique for gene selection, and investigated if gene expression pattern in the primary tissues of three cancer types  with DM are similar or different. Gene expression biomarkers of DM identified by our proposed method outperformed Differentially Expressed Genes (DEGs) identified by the DESeq2 software package in the task of predicting presence or absence of DM. Genes involved in DM tend to be more cancer type specific rather than general across all cancers. Our results also indicate that multimodal data is more predictive of metastasis than either of the three unimodal data tested, and genomic data provides the highest contribution by a wide margin. The results re-emphasize the importance for availability of sufficient image data when a weakly supervised training technique is used. Code is made available at:  Proper management planning for carcinoma patients requires accurate diagnosis, and prognosis prediction. A major prognostic factor, metastasis, however is not always diagnosed at initial patient presentation. Metastasis refers to the dissemination of cancer cells away from the initial site of origin to form colonies at distant organs. This single hallmark of malignancies is responsible for the highest proportion (approximately 90%) of cancer related mortalities . In HNSCTP53 was significantly different between primary and metastatic samples in seven cancer types, while PTEN mutation level was different in five cancer types. Copy number variations also differ significantly in all 15 cancer types examined. TP53, PTEN, CDKN2A, and MYC as significantly mutated genes in the metastasis of various subsets of cancer types.Past studies have investigated the presence or absence of generalizable links between genes involved in the metastasis of different cancer types, either on the basis of mutation, or expression level. However, metastasis has proven to be a complex molecular and biochemical process. While similar mutation rate, and expression pattern have been reported in selected genes across groups of metastatic cancer, there are multiple claims that there are in fact no specific cancer metastatic genes. In a study carried out by In specific cancer types, designated differentially expressed signature genes are the basis of some of the past attempts to stratify cancer patients based on the risk of DM . A recenThe wealth of morphological information contained in the tumor microenvironment is routinely exploited by pathologists for making definitive diagnosis of cancer and predicting patient prognosis. However, aside from being time consuming, this tedious process has also been associated with inter- and intra-observer variability that sometimes lead to unresolved diagnosis or worse, errors in diagnosis. With the use of Convolutional Neural Network (CNN), histopathology images have proven to be good predictors of malignancy status, important molecular biomarkers of various clinical and research relevance, as well as other cellular and extracellular processes . While mRecent trends of use of multimodal data for prediction of diagnosis and outcomes in cancer patients have reported mostly improved metrics compared to use of singular modes of data . In the - We identify genomic markers of DM in three different carcinoma types using a novel combination of the random forest algorithm with an optimized feature selection approach described in . Genes s- Using various machine learning techniques, we investigate and substantiate claims that genes involved in DM tend to be more cancer type specific rather than general across all cancer types, and that there are no specific cancer metastasis genes.- We built separate models to predict DM from gene expression data, clinical data, or histopathology images, as well as multimodal combinations of the three data types. Models metrics show that multimodal data provides an edge for prediction of DM over genomic, clinical or histopathology data. However, the genomic data has the highest contribution by a wide margin.- Unlike with genomic data in which features tend to be more cancer type specific, models built from histopathology image dataset of all three cancer types in total had better metrics than those built from a single cancer type image dataset, emphasizing the importance of sufficient data to build a robust model when a weakly supervised technique is used.In this work, we attempt to predict DM using gene expression data, clinical data, and histopathology images from primary tumors of three carcinoma types - Pancreatic Adenocarcinoma (PAAD), Bladder Carcinoma (BLCA), and Head and Neck Squamous cell Carcinoma (HNSC). The contributions of this research include:Barcode of patients with DM in the TCGA-HNSC, TCGA-BLCA, and TCGA-PAAD projects were retrieved from Genomic Data Commons (GDC) Data Portal, and gene expression data, clinical data, and histopathology images of these patients were downloaded. The few number of available cases of DM is a common challenge in most studies on metastasis. BLCA, and PAAD were selected based on the number of cases of DM in these cancer types that are available on The Cancer Genome Atlas (TCGA), and the number of cases with available corresponding genomic, image, and clinical data. A few cancer types were dropped during this selection process due to the absence of some variables in their records. For example, Skin Cell Carcinoma (SKCM) was eliminated due to missing “number_of_lymph_nodes_positive_by_HE” variable which is present in BLCA, PAAD, and HNSC datasets. HNSC was added based on the interest of one of the authors. See We designed a study that utilized machine learning techniques to simultaneously identify biomarkers of DM in each of BLCA, PAAD, and HNSC, and investigate if genes involved in DM are similar across the three cancer types. Each cancer type dataset was pre-processed separately. Equal numbers of complementary gene expression data of samples without DM  were downloaded for each of the cancer types, and we ensured that there were no overlapping samples between the groups with DM, and those without DM. After extracting Transcript Per Million (TPM)) normalized values of protein-coding genes from the individual records, all samples in a group were merged to create a table, and NaN values were replaced with zero. Other initial preprocessing steps include log to base ten transformation of data to adjust for the wide and non-linear values, removal of genes with a value of zero in greater than 80% of the samples, and selection of only top 10000 genes with highest variance across all samples.To identify important genes that are involved in DM, we utilized the Random Forest (RF) algorithm with an optimization technique described in . RF is aTo assess the strength of the selected biomarkers for prediction of DM, and to investigate if genes involved in DM are similar across the three cancer types, first, we used only the selected genes in each of the groups as features to train multiple ML models for the task of DM prediction. Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and RF models were trained on datasets from each of the four  groups , and theTo further assess the strength of genes selected by the proposed method for predicting DM, under the same study design, we carried out an analysis to identify Differentially Expressed Genes (DEGs) between samples with DM and those without DM in each of the four groups. Instead of the TPM normalized values, raw counts of unstranded RNASeq were extracted from the TCGA RNASeq records. A sample information table was also generated. Differential Gene Expression (DGE) analysis was performed in R using the DESeq2 Bioconductor package. DESeq2 detects DEGs by normalizing raw count values of genes in the experimental groups, fitting negative binomial generalized linear models for each gene, and detecting significance by Wald test . ThreshoWe downloaded diagnostic pathology Whole Slide Histopathology Images (WSIs) of patients with DM in the TCGA projects of BLCA, HNSC, and PAAD from GDC Data Portal. The number of samples in each group reduced slightly after collating a list of only patients with available histopathology images, and RNASeq data, and clinical data. Number of samples with DM in BLCA, PAAD, and HNSC groups are 50, 44, and 18 respectively. Again, we downloaded random complementary WSIs  of patients without DM for each of the cancer types, and ensured no overlap between samples . Total nOne WSI was preprocessed per patient. Due to the typical large size of WSIs which hovers around 100000 pixels in both horizontal and vertical axis, we isolated representative regions within each WSI for analysis. First, a maximum of 500 random non-overlapping patches of size 512 * 512 pixels were extracted from each image at 20 × magnification. These were reduced to a maximum of 200 patches after checking for a tissue area of at least 80% in each patch. To ensure selection of tumor representative regions we imitated the concept of high cellularity described in  which as. Loss function was binary cross-entropy, and optimizer, Adam with a learning rate of 0.0001. Training epoch was set at 40.A DenseNet121 model, pretrained on Imagenet data, and KimiaNet  were cho. Loss fFor prediction on the test sets, patient level features were derived by passing all patches for each patient through the trained feature extractor models, and averaging all patch features from a slide. We trained traditional machine learning algorithm- SVM and a single layer MLP on the patient level features to classify samples as DM positive or DM negative . ConsideMultimodal concatenation of patient level features derived from the CNN models to log transformed values of TPM normalized RNASeq data of selected DM associated genes from above was carried out. For BLCA, and PAAD groups, the 50 genes selected using the (RF + Optimization) technique were combined with 32 features derived from histopathology images of each patient to train classifiers, and predict presence or absence of DM . 150 genLikely due to the effect of a larger training dataset given the weakly supervised training technique that was employed, it was observed from early results that image only models trained on data from the ALL3 group generally performed better in their predictions on the test sets than other groups. Similarly, multimodal models of genomic and histopathology image were generally better than unimodal model of either data type in the ALL3 dataset, while metrics of multimodal models are generally lower than either genomic or image only models in other individual cancer dataset . Therefore further experiments were carried out on the ALL3 dataset. We trained fully connected layers on KimiaNet, a DenseNet121 architecture model finetuned on about 250,000 histopathology images to extract features from the images in the ALL3 dataset. Metrics of multimodal models built from these features, and genomic data were compared to those from DenseNet121.Four variables—Age, T stage, N stage, and Number of lymph nodes positive by HE—were selected from the clinical data, and concatenated with genomic, and histopathology features to predict presence or absence of DM in the ALL3 dataset . As partFKBP6, ASIC5, MAPK8IP1, F11R, and PABPC5, while CTSV, BIRC5, SERPINA7, CST2, KLHL3 are the highest ranked genes in PAAD. In HNSC, TM4SF1, C19orf18, EXPH5, FKBP2, PTPRZ1 are the highest ranked genes, and ALL3 group had CHRNG, CPT1B, CGREF1, GPR31, SPTBN5. Past publications have confirmed the activities of some of these genes in the metastasis of various cancers, either as oncogene or as tumor suppressors , and simultaneous gene selection (top 50) in each of the groups . While Rpressors . There aactivity .Dataset in each group was randomly split into train and test set in a 75:25 ratio, and SVM with linear kernel, KNN, and RF models were trained on the highest ranked 30, 25, 20, 15, 10, 5, 3, 2, and 1 gene from list of selected top 50 genes. Outcomes of single instance predictions, and five-fold cross-validation on the test sets were recorded .p-value of 0.05, the number of DEGs in BLCA, PAAD, HNSC, and ALL3 are 229, 2142, 1100, and 658 respectively. Of these 5, 39, 19, and 13 overlap with the 50 genes selected using the ML method in each of the respective groups. Presence or absence of DM was predicted in the four groups using DEGs with the lowest 30, 25, 20, 15, 10, 5, 3, 2, and 1 adjusted p-value as variables. In almost all cases, genes selected using the ML (RF + optimization) techniques had higher evaluation metrics  than those selected via DESeq2 DGE analysis in the task of DM prediction . Highest mean AUROC score achieved over a five-fold cross validation was 0.87, 0.92, 0.97, 0.79 in BLCA, PAAD, HNSC, and ALL3 groups respectively, compared to 0.65, 0.80, 0.92, 0.62 with DESeq2 DGE analysis, when models were trained on top 15 selected genes (To test the strength of genes selected based on the (RF + Optimization) method as predictors of DM, a DGE analysis was carried out with the DESeq2 software package to identify DEGs between the DM and non-DM samples. With a threshold adjusted diction . Highesed genes .CGREF1, SPTBN5, TAS1R3, FAM241B, EL5, CSPG5, and MAPK8IP1), and 6 genes  overlap with those selected in BLCA, and PAAD respectively   models on BLCA, PAAD, and HNSC datasets, but with genes selected from the combined dataset of the three cancer types (3 group) . These m3 group) . To furt3 group) .DenseNet121 was trained on histopathology images from BLCA, PAAD, and ALL3 groups, after the HNSC group was dropped due to small sample size. One WSI was preprocessed per patient as described in the methods section. For each group, classification models—SVM, and MLP were trained on patient level features extracted by the trained CNN models, and prediction was carried out on the test set features . The imaEarly results showed that metrics of multimodal (Image + Genomic) models built from the ALL3 dataset generally improved on image, or genomic unimodal models by 1–3 percent margin, with genomic data contributing more to the multimodal metrics by a large margin. Following a five-fold Monte Carlo cross-validation, a multimodal (Image + Genomic) SVM classifier produced a mean AUROC score of 0.77, while the corresponding image and genomic unimodal models produced a score of 0.58, and 0.74 respectively. The mean AUROC scores from MLP classifiers are . In either BLCA or PAAD dataset, results of multimodal (Image + Genomic) models mostly improved on those of the image only models, however they were generally below that of genomic only models. The highest mean AUROC derived from a BLCA dataset multimodal (Image + Genomic) model is 0.62, and the corresponding values from image only, and genomic only model are 0.57, and 0.80 respectively. Similar pattern was seen in the PAAD dataset . These Based on these early results, we carried out further experiments with the ALL3 dataset. To evaluate the effect of using a CNN model with lower layers pre-finetuned on histopathology domain images as features extractor, fully connected layers were trained on KimiaNet with the ALL3 dataset histopathology images, and features derived from the model were combined with genomic data to predict DM in the test set. There was generally no additional advantage observed in the results of KimiaNet over those from DenseNet121. Highest mean AUROC from the MLP multimodal model (KimiaNet Image + genomic data) is 0.77. Unimodal image and genomic models from the same dataset produced AUROC scores of 0.55, and 0.80 respectively. The same pattern was seen with the SVM classifier  .Lastly, four clinical variables—Age, T stage, N stage, and Number of lymph nodes positive by HE were concatenated with genomic, and histopathology features to train SVM, and MLP classifiers. This led to an improvement of the mean AUROC score of the Image + Genomic SVM model from 0.77 to a score of 0.79. Mean accuracy, and F1 scores also improved from 0.67 and 0.67 to 0.71 and 0.71 respectively. MLP classifier however produced a mean AUROC score of 0.80 against the score of 0.81 achieved with Genomic + Histopathology data . CombiniZNF and FBKP family of genes which is present in the list of selected genes in the BLCA, and HNSC datasets have been severally associated with cancer progression, and metastasis in multiple cancers  outperform those built from genes selected by DESeq2 DGE analysis . While it is important to note that the datasets in our study are focused specifically on DM, similar studies on related tasks have been reported. Using just RF algorithm for gene selection in a breast cancer study , achieveThe reported AUC scores from our study were derived from the highest ranked 15 genes out of the initially selected 50. Perhaps a protein-protein interaction analysis including the initial 50 genes could lead to selection of fewer genes with higher relevance than the ranking we have employed here, which may in turn lead to improved prediction metrics. The biomarkers discovered in each cancer type using the strict ML approach we have employed should be further investigated for their potential as diagnostic indicators, and as basis for development of new therapies against cancer metastasis.Our results indicate that multimodal data  provides a predictive advantage over either of the unimodal data types in the task of DM detection, however the contribution of only gene expression data is highest by a wide margin. Similar studies have acknowledged the superiority of multimodal data for prediction of medical diagnosis and various outcomes in cancer patients , howeverThe highest mean AUC score of 0.79 derived from the SVM classifier in our study was from multimodal combination of gene expression, clinical, histopathology data, while the score of 0.81 derived from the MLP classifier was obtained from a genomic + histopathology image model. Overall, our results indicate that combining clinical, genomic, and histopathology image data increases the prediction metrics for DM, however genomic data alone is a strong contributor.


We previously developed a new reference-based approach, OptiFit, that fits new sequence data to existing de novo OTUs without changing the composition of the original OTUs. While OptiFit produces OTUs that are as high quality as de novo OTUs, it is unclear whether this method for fitting new sequence data into existing OTUs will impact the performance of classification models relative to models trained and tested only using de novo OTUs. We used OptiFit to cluster sequences into existing OTUs and evaluated model performance in classifying a dataset containing samples from patients with and without colonic screen relevant neoplasia (SRN). We compared the performance of this model to standard methods including de novo and database-reference-based clustering. We found that using OptiFit performed as well or better in classifying SRNs. OptiFit can streamline the process of classifying new samples by avoiding the need to retrain models using reclustered sequences.The ability to use 16S rRNA gene sequence data to train machine learning classification models offers the opportunity to diagnose patients based on the composition of their microbiome. In some applications, the taxonomic resolution that provides the best models may require the use of de novo operational taxonomic unit (OTU)-based classification models is that 16S rRNA gene sequences are often assigned to OTUs based on similarity to other sequences in the dataset. If data are generated from new patients, the old and new sequences must be reclustered to OTUs and the classification model retrained. Yet there is a desire to have a single, validated model that can be widely deployed. To overcome this obstacle, we applied the OptiFit clustering algorithm to fit new sequence data to existing OTUs allowing for reuse of the model. A random forest model implemented using OptiFit performed as well as the traditional reassign and retrain approach. This result shows that it is possible to train and apply machine learning models based on OTU relative abundance data that do not require retraining or the use of a reference database.There is great potential for using microbiome data to aid in diagnosis. A challenge with  There is increasing interest in training machine learning models to diagnose diseases such as Crohn’s disease and colorectal cancer using the relative abundance of clusters of similar 16S rRNA gene sequences . These mde novo approaches cluster sequences based on their similarity to other sequences in the dataset and can change when new data are added. Although it would be preferable to select an approach that generates stable clusters, there may be cases where OTUs generated by a de novo approach outperform those of the other taxonomic levels. In fact, we recently trained machine learning models for classifying patients with and without screen relevant neoplasias (SRNs) in their colons and found that OTUs generated de novo using the OptiClust algorithm performed better than those generated using ASVs or at higher taxonomic levels  are defined without consideration of sequences in other samples, phylotypes are defined by clustering sequences that have the same taxonomy  when classified using a taxonomy database, and closed reference operational taxonomic units are defined by mapping sequences to a collection of reference OTUs. In contrast, c levels .It could be possible to construct reference OTUs and map new sequences to those OTUs to attain similar performance as was seen with the OptiClust-generated OTUs. The traditional approach to reference-based clustering of sequences to OTUs has multiple drawbacks and does not produce clusters as good as those generated using OptiClust . Sovacoode novo and reference-based clustering approaches, we used a publicly available dataset of 16S rRNA gene sequences from stool samples of healthy subjects (n = 226) as well as subjects with screen relevant neoplasia consisting of advanced adenoma and carcinoma (n = 229)  and OptiFit self-reference workflows . Consistent with prior findings, the reference-based methods produced lower MCC scores  than the de novo methods  . MCC is an objective metric used to measure OTU cluster quality based on the similarity of all pairs of sequences and whether they are appropriately clustered or not . We expede novo models were not significantly different . The trained model was then applied to the test data classifying samples as either control or SRN. The VSEARCH Greengenes method performed slightly worse than the OptiClust de novo method . However, the performance on the test data for the OptiClust de novo, OptiFit Greengenes, OptiFit self-reference, and VSEARCH de novo approaches was not significantly different  for each data split. During cross-validation (CV) training, the performance of the OptiFit self-reference and OptiClust de novo OTU assignments. A potential problem with reference-based clustering methods is that sequences that do not map to the reference OTUs are discarded, resulting in a possible loss of information. However, we demonstrated that the training samples represented the most important OTUs for classifying samples. Missing important OTUs is more of a risk when using a database-reference-based method since not all environments are well represented in public databases. Despite this and the lower quality OTUs, the database-reference-based approach performed as well as the models generated using OptiFit. This likely indicates that the sequences that were important to the model were well characterized by the Greengenes reference OTUs. However, a less well-studied system may not be as well characterized by a reference database that would make the ability to utilize one’s own data as a reference an exciting possibility. Our results highlight that OptiFit overcomes a significant limitation with machine learning models trained using de novo OTUs. This is an important result for those applications where models trained using de novo OTUs outperform models generated using methods that produce clusters that do not depend on which sequences are included in the dataset.Random forest machine learning models trained using OptiClust-generated OTUs and tested using OptiFit-generated OTUs performed as well as a model trained using entirely Raw 16S rRNA gene sequence data from the V4 region were previously generated from human stool samples. Sequences were downloaded from the NCBI Sequence Read Archive (accession no. SRP062005) (https://mothur.org/wiki/miseq_sop/). One hundred splits of the 490 samples were generated where 80% of the samples (392 samples) were randomly assigned to the training set and the remaining 20% (98 samples) were assigned to the test set. Using 100 splits of the data accounts for the variation that may be observed depending on the samples that are in the training or test sets. Each sample was in the training set an average of 80 times (standard deviation = 4.1) and the test set an average of 20 times (standard deviation = 4.1).The full dataset was preprocessed with mothur (v1.47)  to join OptiFit Self: The preprocessed data were split into the training and testing sets. The training set was clustered into OTUs using OptiClust, and then the test set was fit to the OTUs of the training set using the OptiFit algorithm . The Optde novo with OptiClust   to prediescribed . The aveThe analysis workflow was implemented in Snakemake . Scripts


Overall, asmbPLS-DA achieved competitive performance in terms of feature selection and classification. We believe that asmbPLS-DA can be a valuable tool for multi-omics research.With the growing use of high-throughput technologies, multi-omics data containing various types of high-dimensional omics data is increasingly being generated to explore the association between the molecular mechanism of the host and diseases. In this study, we present an adaptive sparse multi-block partial least square discriminant analysis (asmbPLS-DA), an extension of our previous work, asmbPLS. This integrative approach identifies the most relevant features across different types of omics data while discriminating multiple disease outcome groups. We used simulation data with various scenarios and a real dataset from the TCGA project to demonstrate that asmbPLS-DA can identify key biomarkers from each type of omics data with better biological relevance than existing competitive methods. Moreover, asmbPLS-DA showed comparable performance in the classification of subjects in terms of disease status or phenotypes using integrated multi-omics molecular profiles, especially when combined with other classification algorithms, such as linear discriminant analysis and random forest. We have made the R package called  The improvements in high-throughput technology have enabled the collection of different types of omics data such as transcriptomics, epigenomics, proteomics, and metabolomics quickly and cost-effectively for the same subject. Each type of omics data, on its own, may contribute unique information. By identifying biomarkers within each type of omics data, we can gain insight as to which biological pathways or processes are different between different phenotypic groups. However, when analyzing only one type of omics data, the correlation between different omics layers is not considered, preventing us from understanding the complex relationship (molecular mechanism) between the multi-omics data and the host systematically and holistically. Therefore, in order to draw a more comprehensive view of systemic biological processes, omics data from the different platforms have to be integrated and analyzed . Multi-omics data are often structured in blocks and characterized by hundreds or thousands of variables with only a small sample size. Meanwhile, they often imply a high degree of correlation within and between different types of omics data. With a large number of variables, the interpretation of their collective contribution to the process can be a hard task. Therefore, there is a need to perform variable selection to obtain the most relevant and predictive features for each block.In the past decade, many approaches have been proposed for identifying predictive biomarkers that are relevant to the patient’s phenotypes. For some methods, such as L1-regularized logistic regression, L1-regularized multinomial logistic regression , and spaIn this study, we present adaptive sparse multi-block partial least square discriminant analysis (asmbPLS-DA), an extension of our previously proposed method, asmbPLS , which iasmbPLS, which is publicly available on GitHub (https://github.com/RunzhiZ/asmbPLS). The package also includes visualization functions that can assist with result interpretation.Our study demonstrates that asmbPLS-DA performs well in identifying predictive biomarkers and achieves comparable classification performance for both binary and multiclass outcomes. We validated the method using both simulated and real data. To facilitate its use, we have developed an R package called In this section, we introduce adaptive sparse partial least square discriminant analysis (asmbPLS-DA), which is based on adaptive sparse partial least square (asmbPLS)  for disc, indicating the relevance of variables, and In asmbPLS, the goal is to construct a set of orthogonal PLS components that maximize the covariance between the two latent variables The problem can be formally expressed as follows:X and Y are deflated, and then the deflated X and Y are used for the calculation of the second PLS component, and so on. Once all the parameters are calculated for the first PLS component, Once the asmbPLS model is built, we can then predict the outcome for the original and the new samples. Specifically, given the new data For asmbPLS-DA, we focus on the categorical outcome instead of the continuous outcome. The extension of asmbPLS to asmbPLS-DA is straightforward. For the binary outcome, a matrix with one column dummy variable is used as the outcome matrix  is one of the most used measures, which is defined as:nce (MD)  is anothWe let l–1)th PLS component(s).In the K-fold CV, the samples are randomly placed into In our study, we choose Using the results of the CV, we can determine the number of PLS components used for classification as well. To avoid the over-fitting issue, the strategy for selecting the optimal number of PLS components is: (1) let the initial number of components be In addition to the different decision rules, we have included several vote algorithms to help the discrimination. (1) The unweighted vote, where each decision rule used will be assigned equal weight, and the final classification is based on the results of the methods. (2) The weighted vote, where each decision rule is assigned a weight based on the BA with the optimal quantile combination, i.e., In addition to the above-mentioned original asmbPLS-DA classification algorithms, we have combined asmbPLS-DA with other methods for classification. Once the asmbPLS-DA model is built, the super score of different PLS components or the selected features can be used as predictors for other classification algorithms, such as RF and LDA. By doing this, we aim to test if this two-stage strategy will increase the classification capability of asmbPLS-DA.All the implementations were conducted using R 4.2.1  in the “TCGAbiolinks (2.25.3) [GDCquery function was used to query the required GDC data, where the project option was set to be “TCGA-BRCA”. For gene data, we set data.category = “Transcriptome Profiling”, data.type = “Gene Expression Quantification”, and workflow.type = “STAR-Counts”; for miRNA data, we set data.category = “Transcriptome Profiling”, data.type = “miRNA Expression Quantification”, and workflow.type = “BCGSC miRNA Profiling”; for protein data, we set data.category = “Proteome Profiling” and data.type = “Protein Expression Quantification”.The real data downloading and pre-processing were implemented using the R package (2.25.3) . The GDCsfsmisc (1.1-14) [glmnet (4.1-7) [mixOmics (6.22.0) [ipflasso (1.1) [randomForest (4.7-1.1) [MASS (7.3-58.3) [The R package 1.1-14)  was used  was use(6.22.0) ; and IPF4.7-1.1) . LDA was.3-58.3) . The useasmbPLS-DA employs different decision rules and vote functions that can impact its performance in various scenarios. Thus, the identification of the best decision rules and vote functions is crucial for selecting the optimal model. Moreover, asmbPLS-DA can be combined with other classification methods, such as random forest (RF)  and lineThe objective of the simulation studies is to investigate the performance of asmbPLS-DA and compare it with other classification methods. To accomplish this, we consider two correlated high-dimensional predictor blocks that vary in the total number of variables and the number of highly correlated variables in each block. In addition, we examine two phenotypic group scenarios, including binary and multiclass (three groups) outcomes. By conducting simulation studies with varying parameter settings, we aim to comprehensively evaluate the performance of asmbPLS-DA and other methods for feature selection and classification. n samples, q variables in the first block p variables in the second block q + p variables, ith and jth features, respectively, and posdefify function from the R package sfsmisc to make the variance–covariance matrix positive definite. We simulate Once For binary outcome:For multiclass outcome:Since only the first ten variables in glmnet to represent L1-regularized logistic regression and L1-regularized multinomial logistic regression in the figures for binary and multiclass outcomes, respectively. For asmbPLS-DA, we conducted the CV with varying quantile combinations for each scenario. In scenarios with To evaluate the performance of asmbPLS-DA and compare it with other classification methods, we conducted a comprehensive analysis of feature selection, classification accuracy, and computation efficiency. We compared asmbPLS-DA with L1-regularized logistic regression and L1-regularized multinomial logistic regression for binary and multiclass outcomes, respectively. In addition, we examined sPLS-DA and DIABLO for both binary and multiclass outcomes and IPF-Lasso for binary outcomes. To perform the comparisons, we used 5-fold cross-validation (CV) with 10 repetitions for asmbPLS-DA, sPLS-DA, IPF-Lasso, and DIABLO. We used the corresponding R package name In our simulation setting, only the first ten variables in For the multiclass outcome, the results of feature selection are presented in The classification performance of all the methods was measured in terms of overall accuracy with models fitted on the training set and then applied to the test set.For the binary outcome, Based on the comparison among the asmbPLS-DA methods, we have selected a subset of asmbPLS-DA methods to compare with other methods for simplicity . The selFor the multiclass outcome, the comparison among all asmbPLS-DA methods shows that RF using features selected by asmbPLS-DA performs the best in scenarios with structure = cor . For strAccording to https://portal.gdc.cancer.gov/repository. We used the tissue type  as the binary outcome to assess the methods’ performance. Additionally, we removed samples without pathologic stage information from the 854 tumor samples, resulting in 127 samples with stage I breast cancer, 491 samples with stage II breast cancer, and 221 samples with stage III or stage IV breast cancer. We used the pathologic stage as the multiclass outcome (stage I vs. stage II vs. stage III + stage IV) for evaluation. The pre-processed procedures, including outlier removal, data normalization, and filtering, were implemented for the two omics data, resulting in a final dataset of 45,371 genes and 1410 miRNAs, with a feature ratio of approximately 32:1. For the pathologic stage outcome, we have also included one more omics data, i.e., protein expression, to further test the performance of methods. After removing samples and features with missing data, we obtained 45,371 genes and 1410 miRNAs, as well as 457 proteins for 822 tumor samples. Of these samples, 125 were in the stage I group, 480 were in the stage II group, and 217 were in the stage III or stage IV group.We evaluated the performance of asmbPLS-DA and the other competitive methods using the TCGA breast cancer dataset, which included two types of omics data: gene expression and miRNA expression. The dataset consisted of 949 samples, including 854 tumor samples and 95 normal samples. The data were downloaded from We included three PLS components for PLS-based methods and performed 5-fold cross-validation with 10 repetitions for asmbPLS-DA, sPLS-DA, and IPF-Lasso for parameter tuning. DIABLO was not included due to its long computation time. For asmbPLS-DA, we used pre-determined quantile combinations of First, we used tissue type as the outcome to test the performance of different methods on the binary outcome. For asmbPLS-DA methods, for simplicity, we only used asmbPLS-DA with the decision rule of fixed cutoff for both feature selection and classification. Additionally, we utilized the features selected by asmbPLS-DA in both LDA and RF for the purpose of classification. Regarding feature selection, forty-six genes and eight miRNAs were selected by asmbPLS-DA; thirty genes and zero miRNAs were selected by sPLS-DA; thirty-six genes and zero miRNAs were selected by L1-regularized logistic regression; twenty-six genes and fifteen miRNAs were selected by IPF-Lasso. Since sPLS-DA and L1-regularized logistic regression do not consider group structure, no miRNA was collected by these methods, which is expected given the relatively small number of miRNAs compared to genes. In addition, due to the limitation of the Lasso method, the features selected by L1-regularized logistic regression and IPF-Lasso show relatively lower correlations compared to the features selected by asmbPLS-DA and sPLS-DA , indicatTo further evaluate the methods, we used the pathologic stage as our multiclass outcome with more balanced group sizes of 127, 491, and 227 samples in stage I, stage II, and stage III + stage IV, respectively. Similarly, we applied asmbPLS-DA with the decision rule of Max Y for both feature selection and classification and used the selected features in LDA and RF for classification.The results showed that asmbPLS-DA selected forty-six genes and fifteen miRNAs, sPLS-DA selected one hundred genes but no miRNA, and L1-regularized logistic regression selected forty-seven genes and one miRNA. The top five features from both blocks selected by each method are listed in The classification results using the pathologic stage as the outcome are listed in To test the performance of the methods with more than two blocks, we added protein expression data to the analysis for the pathologic stage outcome. asmbPLS-DA identified twenty-eight genes, eight miRNAs, and fourteen proteins as the most important features. sPLS-DA identified 70 genes as the most important features but did not select any miRNAs or proteins. L1-regularized logistic regression selected sixty genes, two miRNAs, and four proteins. Notably, asmbPLS-DA selected nearly the same top genes and miRNAs as when using only two omics data (as shown in asmbPLS available on our GitHub (https://github.com/RunzhiZ/asmbPLS), and it will soon be uploaded to CRAN.In this paper, we extended our previous work, asmbPLS, to a new method called asmbPLS-DA. This new method considers both binary and multiclass categorical outcomes. The aim of asmbPLS-DA is to identify the most predictive features in diverse types of omics data and then use the selected features for classification. This helps to improve our understanding of the molecular mechanisms behind diseases. To achieve this goal, we have included different decision rules and several vote functions. Additionally, we have combined asmbPLS-DA with other classification algorithms for further testing. The proposed method is implemented in the R package Simulation studies have shown that asmbPLS-DA has very satisfactory performances regarding feature selection, especially for multiclass outcomes and cases where different omics data vary greatly in the number of features. In methods that do not consider the group structure, such as sPLS-DA, L1-regularized logistic regression, and L1-regularized multinomial logistic regression, the feature selection can be dominated by the block with the greatest number of features, ignoring information from smaller blocks. However, due to the nature of asmbPLS-DA, it can select the predictive features from all the blocks included, even if these features are highly correlated. The feature selection performance of asmbPLS-DA is stable regardless of the scenario setting. In real data, the top features selected by asmbPLS-DA have been validated by the literature, further indicating the power of feature selection by asmbPLS-DA. In the simulation studies, asmbPLS-DA performs worse overall for classification compared to the other methods, except for DIABLO. However, RF with the features selected by asmbPLS-DA shows similar performance to the best in the scenarios where all the relevant features are highly correlated, which is quite common for multi-omics data. This indicates the significance of the selected features. Moreover, as the noise increases, the performances of asmbPLS-DA methods get closer to the best. In the real data, we found that asmbPLS-DA with LDA shows the best classification when considering both the overall accuracy and the balanced accuracy, while asmbPLS-DA with RF shows good performance regarding the overall accuracy only.The current version of asmbPLS-DA does not include the pre-processing of missing values. Additionally, if there are no relevant features in the predictor blocks, classification can be problematic for the method since asmbPLS-DA will always select some features for each block, resulting in non-relevant features being selected.In summary, we have shown that asmbPLS-DA is a useful tool for multi-omics analysis. We believe that asmbPLS-DA provides a valuable alternative for both feature selection and classification in binary and multiclass problems.


Current biotechnologies can simultaneously measure multiple high-dimensional modalities  from the same cells. A combination of different analytical tasks  is required to comprehensively understand such data, inferring how gene regulation drives biological diversity and functions. However, current analytical methods are designed to perform a single task, only providing a partial picture of the multi-modal data. Here, we present UnitedNet, an explainable multi-task deep neural network capable of integrating different tasks to analyze single-cell multi-modality data. Applied to various multi-modality datasets , UnitedNet demonstrates similar or better accuracy in multi-modal integration and cross-modal prediction compared with state-of-the-art methods. Moreover, by dissecting the trained UnitedNet with the explainable machine learning algorithm, we can directly quantify the relationship between gene expression and other modalities with cell-type specificity. UnitedNet is a comprehensive end-to-end framework that could be broadly applicable to single-cell multi-modality biology. This framework has the potential to facilitate the discovery of cell-type-specific regulation kinetics across transcriptomics and other modalities. Multimodal biological data is challenging to analyze. Here, the authors develop UnitedNet, an explainable deep neural network for analyzing single-cell multimodal biological data and estimating relationships between gene expression and other modalities with cell-type specificity. For example, the patch-seq technique simultaneously measures cell gene expression and intracellular electrical activity4, and the multiome ATAC + gene expression technique jointly measures cell gene expression and DNA accessibility5. Such multimodal omics data provide direct and comprehensive views of cellular transcriptional and functional processes simultaneously. However, methods developed for analyzing single-modality biological data cannot be directly applied to multi-modality data6. Compared with single-modality analysis, recent studies have identified more tasks for multi-modality analysis7 such as (i) identification of biologically meaningful groups from different modalities, enabling a deeper biological understanding of cellular identities and functions for different biological systems, and (ii) cross-modal prediction among different modalities, inferring the information of cells that cannot be easily or simultaneously measured. Moreover, the multi-modality data generated for the same type of cells provides the opportunity to discover cell-type-specific relationships between gene expression and other modalities, helping to uncover the regulatory mechanisms underlying the biological condition of interest. A method that can simultaneously address these different tasks and automatically quantify cross-modal relevance is needed to fully utilize the potential of multi-modality datasets.Recent advances in single-cell biotechnology make it possible to simultaneously measure gene expression along with other high-dimensional modalities for the same cells8, which are then used for unsupervised or supervised classification to identify cell types and states or tissue regions10. For the cross-modal prediction task, autoencoder-based neural networks have been developed to predict across different modalities16. For the relevance discovery across modalities, Schema represents state-of-the-art multi-modal integration methods that can identify the features in a user-defined primary modality that are important to other modalities17. More recently, GrID-NET has also been proposed to identify genomic loci that mediate the regulation of specific genes in multiome ATAC + gene expression datasets18.Several methods of multi-modality analysis have been developed to address each task separately or to identify the cross-modal feature-to-feature relevance relationships. For the joint group identification task, multi-modality data integration methods have been developed to fuse different modality measurements into joint representations19. Still, combining multiple tasks into a single framework can be challenging for the following two reasons. First, each modality measurement has unique statistical characteristics , requiring different statistical assumptions. While there have been several statistical models developed for different modalities , a method that can accommodate unknown distributions of simultaneously measured modalities is still lacking. Second, joint group identification and cross-modal prediction typically represent separate objectives. Specifically, the objective of joint group identification is to penalize wrong group assignments of cells, whereas that of cross-modal prediction is to minimize the gap between predicted reconstruction and ground truth measurement. Thus, a strategy to integrate the different objectives needs to be designed to avoid performance degradation. Moreover, when no prior knowledge is given, it also remains a major challenge to find relevance relationships between gene expression and other modalities in certain cell types. If simply iterating all possible combinations of feature sets, the identification and quantification of the co-varying set of features will be computationally intractable for high-dimensional data. An efficient method is desired to first identify the set of features from multiple modalities that are important for a specific biological condition of interest  and then, quantify the relationship across these features.Compared with the above methods, an approach that can address all tasks within a unified framework, quantify the cell-type-specific, cross-modal relevance, and do so without prior knowledge can streamline data analyses, potentially improve each task performance, and help gain biological insights from single-cell multimodality data25. We have found that alternating training between joint group identification and cross-modal prediction maintains or improves the performance for both tasks. In addition, we applied explainable machine learning to dissect the trained network and quantify the cell-type-specific, cross-modal feature-to-feature relevance. We have applied this network to various multi-modality datasets  simultaneously measured transcriptomics and intracellular electrophysiology29 (multi-sensing data), (iii) simultaneously profiled transcriptomics and DNA accessibility7 (multi-omics data), and (iv) spatially resolved transcriptomics and proteomics31 . The results show higher performances in both tasks, achieving similar or better unsupervised and supervised joint group identification and cross-modal prediction compared with other state-of-the-art methods. Moreover, we show that this approach recapitulates several previously published cell-type-specific feature-to-feature relevance relationships.Here, we introduce an explainable multi-task deep neural network to address the above challenges for multi-modality data analysis. This network has an encoder-decoder-discriminator structure and is trained by alternating between two tasks: joint group identification and cross-modal prediction. Specifically, this encoder-decoder-discriminator structure does not presume that the data distributions are known and instead implicitly approximates the statistical characteristics of each modalityets Fig. , includi32 (see “Methods”). The model then assigns the group labels, such as cell types, to each cell through unsupervised or supervised group identification networks, where the latter task is also known as annotation/label transfer10  and then fuses these codes into shared latent codes using an adaptive weighting schemer10 Fig. . For cror10 Fig. . The dis33, (ii) a contrastive loss that aligns the different modality-specific latent codes of the same cell and further separates the latent codes from other cells of different clusters34, (iii) a reconstruction loss that compares the reconstruction from the encoders and decoders with the original data so that the latent code better represents the cell15, (iv) a prediction loss that measures the performance of cross-modal predictions15, (v) a discriminator loss that distinguishes the original and reconstructed data in the target modality25, and (vi) a generator loss that pushes the decoded data to resemble the original data25  an unsupervised clustering loss or a supervised classification loss that separates and closely packs shared latent codes in different clusters to better assign group labelsa25 Fig. . During a25 Fig. .35 algorithm , commonly used to interpret deep learning models, to dissect the trained UnitedNet. During the explainable learning, we can identify features that show higher relevance to specific groups 36, totalVI16, and Weighted Nearest Neighbor (WNN)8. We applied the Leiden clustering37 method to cluster the integrated joint representations from these methods and used single-modality Leiden clustering as the performance baseline. As a result, UnitedNet consistently exhibits similar or better unsupervised joint group identification accuracy compared with the single-modality Leiden clustering and other state-of-the-art methods  with their ground truth labels from Dyngen, a multi-omics biological process simulatorWe then studied why multi-task learning can improve the performance of both tasks. Based on the previous designation of shared-latent space in multi-modal and multi-task learning Fig. , we hypo32 to automatically assign lower weights to the modalities with more noise, reducing their impact on the group identification results , electrophysiology (E), and transcriptomics (T) in the same neuronsely Fig. .Fig. 3Mu38. Compared with other state-of-the-art cell typing methods including MOFA, totalVI, Schema, WNN, and Leiden clustering using single modality data, the performance of cell typing results by UnitedNet demonstrates improvement in terms of cell type separability and identification accuracy  datasets. By directly fusing the three modalities together and assigning the label for each cell, UnitedNet identified the cell MET-types with a high degree of congruence (ARI = 0.82) and a roughly diagonal correspondence ARI = 0.41) between the major MET-types and subtle MET-types compared with the previously reported results Fig. . Further1 between11. In addition, we found that the morphology modality was also aligned with the transcriptomics and electrophysiology modalities, with a relatively less level of alignment for the Pvalb neurons . The results  the inherent randomness of the SHAP method and (ii) the randomness introduced by training the UnitedNet model on different data, which are two major sources of randomness in the explainable learning outcomes. First, to validate the robustness of SHAP with different hyperparameters, we used the fixed Patch-seq GABAergic neuron dataset to train a UnitedNet model. Then, we applied 10 folds cross-validation to the Patch-seq GABAergic neuron dataset to calculate Shapley values. We then counted the frequency of the top n features identified by Shapley values in each cross-validation replication  channels, which are critical to neuronal firing dynamics and neurotransmitter release41. These results suggested that UnitedNet could be potentially used to facilitate the identification of cell-type-specific gene-to-function relevance for Patch-seq data.Then, using the Pvalb neuron type as an example, we qualitatively validated the SHAP-selected relevance Fig. . For theThe above experiments showed the robustness and effectiveness of the Shapley value-based feature relevance identification. Based on the ranking and robustness of the Shapley value for each feature  has been used to profile diverse types of immune cells7. One challenge in analyzing these datasets is using previously annotated multi-modality datasets as references to analyze new measurements from a different batch of the same biological system. Another challenge is the batch effect, which refers to differences between cells caused by inter-sample variations. This batch effect makes it difficult to use labeled multiome ATAC + gene expression datasets as a reference atlas to annotate other new datasets. We found that UnitedNet can address these challenges in two ways. First, it can perform a supervised group identification task (termed annotation transfer) that automatically identifies cell types in new, unlabeled test samples based on previously labeled training samples, while simultaneously enabling cross-modal prediction. In addition, we found that UnitedNet can effectively reduce batch effects among different biological samples  in 13 batches from different tissue sites and donorspes Fig. . We traipes Fig. . We alsopes Fig. .To understand how UnitedNet reduces batch effect, we conducted additional ablation studies  and DNA accessibility sites  that were differentially expressed for the CD8+ T major cell type  as an example Fig. , our resype Fig. . Within ons Fig. . The res48. However, spatial information is often not fully utilized when analyzing spatial omics data for group identification tasks. UnitedNet is flexible to integrate different modalities as inputs, including spatial information. As demonstrated below, UnitedNet can utilize cell niche information (neighborhood gene expression information of each cell) as an additional modality to identify biologically meaningful groups and potentiate cross-modal prediction  dataset that spatially maps gene expression and H&E staining on DLPFC brain slices from 12 batchesIn addition, we explored whether UnitedNet can reduce the batch effect in the spatial DLPFC dataset, in a similar way as the analyses for the multiome ATAC + gene expression BMMC dataset. The results showed that UnitedNet maintained both good separability in the latent space and the ability to reduce the batch effect, enabling the higher or comparable performance in the group identification task compared to other ablation studies  from continuous measurements of other modalities . The enhanced performance of UnitedNet could enable the construction of a reliable predictive model, allowing for the use of continuous prediction to generate biological insights. Last but not least, UnitedNet could be used to suggest the most likely relevance relationships from presented multi-modal data, providing useful guidance for downstream wet-lab validation and potentially reducing time-consuming experiments.We envision several directions where UnitedNet could be used for data-driven scientific discoveries. In the group identification task, UnitedNet can adaptively and effectively integrate multiple modalities measured from the same cells, potentially improving current cell typing systems to discover cell types that may not be identified using single-modality methods and helping to infer cell-type-specific phenotypes and functions19, how to reduce randomness in the neural network training, how to design other loss functions to integrate more tasks , and how to design wet lab experiments to validate the indicated cross-modal relevance. We envision that future mechanistic studies, including ablation experiments and theoretical analysis, could help address these questions.Although the UnitedNet performed well for different multi-modality datasets, a number of improvements may be explored in the future, such as automatic searching for the optimal network configuration  and hyperparameter  for deep neural networks. Moreover, there are some remaining questions such as why multi-task learning can improve multi-modal data analysisUnitedNet features both cross-modality prediction and joint group identification. The above two tasks are accomplished based on learned joint low-dimensional representations of different modalities, which contain the essential information of the cells Fig. . SupposeUnitedNet consists of encoders, decoders, discriminators, and a group identification module. It is trained based on a within-modality prediction loss, a cross-modality prediction loss, a generator loss, a discriminator loss, a contrastive loss, a clustering loss (for unsupervised group identification), and a classification loss (for supervised group identification). The details about the components and losses of UnitedNet are as follows.For each modality The low-dimensional representations from different modalities are required to have the same number of components.The decoder Discriminators assist the training of the generator that consists of the encoders and decoders. The discriminator Denote the number of groups to be 33. They ensure that the obtained groups are separable and compact. The third component is based on the self entropy51. It avoids trivial solutions where most of the cells are assigned to a small portion of the total number of groups.The clustering loss consists of three components. The first two components are adopted from the Deep Divergence-based Clustering (DDC)Component 1 reduces the two-by-two correlations between the cluster probability assignments (soft assignments) of different groups. It increases the group separability since those correlations are positively related to the similarity between different groups. Define the matrix The component 2 pushes the soft assignment values of different groups to distinct corners of the simplexes in It enforces the orthogonality between Component 3 aims to avoid the trivial solution where most of the cells are assigned to a small proportion of the total groups. Let 13, we replace equations 10 and 11 with binary cross-entropy loss.The within-modality prediction loss is defined by25. We assign the within-modality predicted features The generators and discriminators are trained by the least-squares loss34 to align the latent codes from different modalities. Define the cosine similarity byWe apply the contrastive lossgi = nk(i)/n where nk  is the number of cells with cell type label Define vector group identification update step and prediction update step. In the group identification update step, the encoder outputs the modality-specific codes and feeds them to the group identification module. Then, the group identification module fuses the modality-specific codes as shared latent codes and obtains the prediction update step, the encoders output the low-dimensional representations from different modalities, feed them to the decoders, and obtain the cross-modality predicted features We first present the procedure where UnitedNet is trained without cell labels for cross-modality prediction and unsupervised group identification. It is trained iteratively with two steps: The above two training steps are summarized in Algorithms 1 and 2, respectively. We acknowledge that the inherent stochasticity in the training of artificial neural networks, such as GPU computation, may result in subtle differences in the outcomes.To train UnitedNet for supervised classification, we modify the group identification loss by:53, which is commonly used to interpret machine learning models. The idea of this approach is to approximate the influence of a feature with respect to the output by a linear function while fixing the other features, and the function’s coefficient corresponds to the Shapley value. The advantages of SHAP include its (1) theoretically based interpretability, (2) wide application scope, and (3) calculation procedure without the need to perturb the model or data, which is required by many other feature importance methods. We will first present the calculation procedure of the Shapley value and then explain about how it is applied to UnitedNet.To provide insights into the importance of different features, we applied SHAP (SHapley Additive exPlanations)Suppose that we want to assess the importance of a feature 35 with the following recursive formula:For neural network Namely, we can compute the Sharpley value of n features with the highest averaged values are interpreted as having high relevance with the group (n = 7 for the Patch-seq GABAergic neuron dataset and n = 20 for the multiome ATAC + Gene expression BMMCs dataset). To quantify the cross-modal feature-to-feature relevance within groups, we considered the high-relevance features of this group selected in the previous step. Next, we calculated the Sharpley value of each feature with respect to each other features from another modality. Then, the absolute values of the Sharpley values from different cells were aggregated by averaging. The features with relatively large values were viewed as important. The relevance relationship was visualized using a chord diagram with a python package MNE-Connectivity54.To identify features with high relevance to a specific group, for each cell, we calculated the Sharpley value of each input feature with respect to the soft assignment of that group. Next, all Sharpley values from the cells that are classified as this group are taken as absolute values and used to calculate an average value. The top We tested the robustness of SHAP with respect to two sources of randomness in the explainable learning outcomes: the inherent randomness of the SHAP method and the randomness introduced by training the UnitedNet model on different data.k folds cross-validation to the dataset to calculate the Shapley values from the trained UnitedNet. The calculation of Shapley values  requires a background dataset to determine the reference output of the model and calculation data to generate the Shapley value by comparing it to the background data35. In the k folds cross-validation, we took each fold in turn as the calculation data and the remaining k−1 folds as the background data . We then counted the frequency of the top n features identified by Shapley values in each cross-validation replication. If the top n features are consistently identified across different cross-validation replications (with high frequency), we conclude that SHAP is robust with respect to its inherent randomness. For extensive data  that require vast computing power, we used the average of each cell type as background data and a subset of data as the calculation data. We also tested the robustness of SHAP with respect to both its inherent randomness and the training of UnitedNet. The procedure is similar to the above one, except we retrained UnitedNet on the k−1 folds of the data.For the inherent randomness, we first trained UnitedNet on a fixed dataset to eliminate the randomness from model training. We then used scikit-learn55. The details about these metrics are as follows.The unsupervised and supervised group identification performance is evaluated by the adjusted rand index. The prediction performance is evaluated by the coefficient of determination and area under the ROC curve. The relationship of two tasks is evaluated by Pearson’s correlation. All metrics are calculated by The adjusted rand index compares the clusters obtained from the model with the one from the cell type labels. Let ARI is close to one when the clustering result from the model is close to the one from the observed cell type labels and close to zero for a random guess.Let It compares the mean square error (MSE) of the prediction Let It takes a value in 13. Let For modeling the multiome ATAC + gene expression dataset, we have binarized the DNA accessibility data. Thus, to evaluate the prediction for those data, we adopt the area under the ROC curve, which was also used in the previous studyThe input of UnitedNet can be various multi-modality dataset. We conclude that four major categories of such datasets include multi-modal simulation datasets, multi-modal sensing datasets, multi-omics datasets, and spatial omics datasets. The experimental details are specified in the following sections.Dyngen26 to simulate the four-modality dataset. Specifically, we generate 500 cells with simulated DNA, pre-mRNA, mRNA, and protein modalities, each modality containing 100 dimensions of features. Meanwhile, the ground truth cell-type annotations are generated along with the dataset. For the parameter of the Dyngen simulator, we use the default setting of a linear backbone model in the tutorial of Dyngen with the functions including backbone_linear, initialize_model, and generate_dataset.We use MUSE27 to simulate two-modality inputs to assess the robustness of UnitedNet with one low-quality modality. We simulate 11 two-modality datasets with 1000 cells and 10 cell types. Each modality contains 500 modality-specific features. For each of the 11 datasets, one of the modalities is simulated with a controllable decay coefficient. We use 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1 as different decay coefficients when benchmarking with other methods.We apply the simulator in 29. We use the same dataset after quality control of previous research11, in which 3395 neurons remain for E-T analysis and 448 neurons remain for M-E-T analysis. We standardize the input matrices of each modality to make the mean value and the standard deviation of all features in each cell to be 0 and 1, respectively.We use a Patch-seq dataset that simultaneously characterizes the morphological (M), electrophysiological (E), and transcriptomic (T) features obtained from GABAergic interneurons in the mouse visual cortex7. In addition to the quality control in the previous study, we use the standard preprocessing procedure for the multiome ATAC + gene expression BMMCs datasets6. For the preprocessing of the gene expression modality, we use median normalization and the log1p transform and standardization and select the top 4000 most variable genes through Scanpy56. For the preprocessing of the DNA accessibility modality, we binarize the data by replacing all nonzero values with a value of 1 and select the top 13,634 most variable DNA accessibility features through Scanpy. We use the ChIPseeker57 and scanpy.var_names_make_unique to annotate the DNA-accessibility peaks.We use a multiome ATAC + gene expression dataset that simultaneously combines gene expression and genome-wide DNA accessibility obtained in BMMC tissue from 10 donors and 4 tissue sitesi, we compute i, and Using the measured expression of RNAs of cells or spots, we incorporate the spatial information of each cell or spot and generate a weighted average expression of RNAs. With two-dimensional spatial coordinates  and testing dataset  for the prediction task.We use the DBiT-seq embryo dataset30. We use the following three modalities: mRNA expression, morphological features extracted from the H&E staining images, and niche mRNA. For modality of mRNA expression, we normalize the raw count matrix and select the top 2365 differentially expressed genes. We use a pre-trained convolutional neural network58 to extract morphological features from the H&E staining images implemented by stLearn59. A 50-dimensional morphological feature is used as the second modality for each spot. For the supervised group identification task, we use 11 batches with their tissue region annotations to train a UnitedNet model. Then we apply the trained model to the remaining batch to identify tissue region annotation and perform the cross-modal prediction between the H&E image features and mRNA expression. We compare the identification performance of UnitedNet with SpatialDE PCA and pseudobulk PCA from the original DLPFC paper30. After the identification task, we applied a refinement step for the clustering result following SpaGCN with a number of 35 spots in the nearest neighbors60.We use the human adult dorsolateral prefrontal cortex (DLPFC) datasets with 12 batches48. No data were excluded from the analyses. Statistical comparisons were performed using Python 3.7 and Scipy 1.7.3 with appropriate inferential methods, as indicated in the figure legends. Graphs were created using Python 3.7, matplotlib 3.5.1, and seaborn 0.11.2. Statistical results in the figures are presented as exact P value. Cross-validation was used to verify the performance on each dataset. Each fold of the cross validation was randomly splitted from the dataset or used as the existing biological group identities. The conclusions were drawn from the analysis of multiple experiments. Blinding was not relevant to the multi-modal data analyses because the datasets were not divided into control and experimental groups.In total, experiments and analyses were conducted on 7 different publicly available multi-modality datasets. No statistical methods were used to pre-determine dataset number and sizes, but they are similar to those reported in previous publicationsFurther information on research design is available in the Supplementary informationPeer Review FileReporting Summary