Cancer is a heterogeneous disease, and patients with tumors from different organs can share similar epigenetic and genetic alterations. Therefore, it is crucial to identify the novel subgroups of patients with similar molecular characteristics. It is possible to propose a better treatment strategy when the heterogeneity of the patient is accounted for during subgroup identification, irrespective of the tissue of origin. This work proposes a machine learning (ML) based pipeline for subgroup identification in pan-cancer. Here, mRNA, miRNA, DNA methylation, and protein expression features from pan-cancer samples were concatenated and non-linearly projected to a lower dimension using an ML algorithm. This data was then clustered to identify multi-omics-based novel subgroups. The clinical characterization of these ML subgroups indicated significant differences in overall survival (OS) and disease-free survival (DFS) . The subgroups formed by the patients from different tumors shared similar molecular alterations in terms of immune microenvironment, mutation profile, and enriched pathways. Further, decision-level and feature-level fused classification models were built to identify the novel subgroups for unseen samples. Additionally, the classification models were used to obtain the class labels for the validation samples, and the molecular characteristics were verified. To summarize, this work identified novel ML subgroups using multi-omics data and showed that the patients with different tumor types could be similar molecularly. We also proposed and validated the classification models for subgroup identification. The proposed classification models can be used to identify the novel multi-omics subgroups, and the molecular characteristics of each subgroup can be used to design appropriate treatment regimen. The recent advances in therapy protocols have increased the five-year prognosis of many cancer types, yet, cancer remains the second most common cause of death globally . ClassifThe advancements in next generation sequencing technologies have contributed to the generation of a massive amount of data representing various genomic and epigenomic changes. This laid the foundation for the establishment of The Cancer Genome Atlas (TCGA), a multiplatform cancer database housing information of more than 11,000 samples from 33 cancer types. This database instigates us to understand the association of various molecular features with the phenotype. As each molecular level of evidence (omic level or datatype) provides a different set of biomarkers and insights about the aberrations, integrative analysis of multiple datatypes accounting for the non-linear interactions will result in the identification of better subgroups , 8. The Samples from different organs-of-origin and different grades can also be similar molecularly. The first attempt to identify the subgroups in pan-cancer data was made by Hoadley et al., using the data from 12 tumor types to identify 11 subgroups . ThroughAll these pan-cancer studies are based on statistical methods. Statistical models can be used to understand the relation between the variables and draw inferences. However, they cannot be used to identify and generalize the hidden patterns, particularly in the multi-modal heterogeneous data . As the K-means clustering was then carried out using this representation to identify the ML-based molecular subgroups. Pairwise statistical tests were carried out in the identified subgroups using all the features. Further, the features specific to each subgroup were identified by filtering them based on the q-value and fold-change (FC). The chosen features were then used to train the classification models to identify the multi-omics-based molecular subgroup for a new sample. Three widely used ML-based classification models, support vector machines (SVM), random forest (RF), and feed-forward neural network (FFNN) were trained, and their prediction probabilities were combined to obtain the decision-level fused classification models as the decision-level fused models are more robust and stable to outliers. Also, as each datatype is known to convey complementary information, we trained the classification models by combining the features from different data levels to obtain feature-level fused classification models.In this work, different data types, including mRNA, miRNA, methylation, and protein expression, were integrated. To model the non-linear interaction among the molecular features from multiple data types, an autoencoder (AE) was trained, and a reduced dimension representation was obtained. Consensus The steps followed in this work for the identification of novel ML-based subgroups are outlined in n) should be approximately equal to the number of dimensions (p)  Reviewers' comments:Reviewer's Responses to Questions  Comments to the Author1. Is the manuscript technically sound, and do the data support the conclusions?  The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: YesReviewer #2: No**********  2. Has the statistical analysis been performed appropriately and rigorously?   Reviewer #1: I Don't KnowReviewer #2: Yes**********  3. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.  The Reviewer #1: YesReviewer #2: No**********  4. Is the manuscript presented in an intelligible fashion and written in standard English?  PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: YesReviewer #2: Yes**********  5. Review Comments to the Author  Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: In this paper, the authors are proposed “Integration of Pan-cancer Multi-omics Data for Novel Mixed Subgroup Identification using Machine Learning Methods”The strengths of the paper are that it is well structured, the description of the related work is well done and that results are extensively compared to results of the similar research.Minor revisions:1. Authors should draw a graphical abstract of the proposed approach2. Authors should justify the proposed approach3. Authors write the comparison with other models developed for this cancer4. Proofread the entire manuscript.Reviewer #2: In this study the author have tried to integrate multi-omics studies to identify novel subgroup identification from different cancers. However the manuscript has multiple issues and there is overinterpretation of the data. Below are some of the points:1. It is not clear how many samples were used for training and validation in the three methods SVM: support vectormachine, RF: random forest, FFNN: feed-forward neural network from each dataset and author need to mention this in the methods.2. The authors say that the proposed framework can be used to identify 12 subgroups for new patients and help in personalized clinical decision making. Then this should performed on the another set of new samples as validation set.3. Authors need to write in the discussion how these subgroups can be used to design appropriate treatment regimen for different cancers.4. The authors have found GO process in each ML cluster and the genes identified in each GO process or pathway can be compared with The Drug Gene Interaction Database to identify clinically actionable gene targets identified in these clusters5. The authors need to provide the list of genes identified in each GO process or pathway in these cluster as Supplementary table.**********  6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.If you choose “no”, your identity will remain anonymous but your review may still be made public.Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.  Reviewer #1: NoReviewer #2: No**********https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,  20 Mar 2023We would like to thank the editor and reviewers for their time and input. Please find the point-by-point response to all queries below.Reviewer #1: 1. Authors should draw a graphical abstract of the proposed approachResponse: We would like to thank the reviewer for the input. We have now included the graphical abstract of the proposed work (Reproduced here for the convenience of the reviewer). Rebuttal Figure 1: Graphical abstract of the proposed work is uploaded with the manuscript. 2. Authors should justify the proposed approachResponse: We would like to thank the reviewer for the question and would like to apologize as this was not made clear in the manuscript.Distinct phenotypes are the result of the underlying molecular variations in the genome. In cancer, the response to therapy relies hugely on the molecular alterations in the tumor. Hence, the stratification of patients based on the molecular characteristics of individuals has a significant role to play in tailoring treatment strategies to reduce adverse effects , Vargas AJ et al. (2016)). Most of the existing studies proposed for subgroup identification are based on statistical techniques which might not be able to model the non-linear interactions occurring in the genome appropriately , Mo Q et al. (2013), Hoadley KA et al. (2014), Wu D et al. (2015), Hoadley KA et al. (2018), Nguyen H et al. (2019), González-Reymúndez A et al. (2020)). However, the pipelines that are based on machine learning (ML) techniques have the ability to model non-linear interactions, but, the studies are limited to single tumor types , Baek B et al. (2020)). Tumors originating from different organs can have similar genetic and epigenetic variations. In order to identify the subgroups independent of the organ of origin, pan-cancer analysis which includes samples from all the tumor types, is carried out in this work. To obtain a holistic view of a heterogenous disease like cancer, there is an inherent need to model it using multiple levels of evidence, as they carry complementary information. To integrate and capture the non-linear interaction between the different levels of evidence, in this work, autoencoder (AE), an ML-based non-linear dimensionality reduction algorithm, is used. This reduced dimensional data is then clustered using consensus clustering to group the samples with similar characteristics together. To obtain further insights into the molecular features contributing to the similarity of the patients within the ML clusters, and the differences between the patients of different ML clusters, various statistical tests were carried out using different levels of evidence. To identify the pathways that can be targeted in each ML cluster, pathway analysis was carried out using the tools, GSEA, and metascape.To identify the subgroup for a new sample, multiple classification models were built. The models were trained using each level of evidence separately, as most of the existing studies generate data by quantifying only one level of evidence. To propose a robust classification model, the prediction probabilities from these classifiers were combined to obtain decision-level fused classification models. Baek B, Lee H. Prediction of survival and recurrence in patients with pancreatic cancer by integrating multi-omics data. Scientific reports. 2020 Nov 3;10(1):18951.Brigham & Women’s Hospital & Harvard Medical School Chin Lynda 9 11 Park Peter J. 12 Kucherlapati Raju 13, et al. "Comprehensive molecular portraits of human breast tumours." Nature 490.7418 (2012): 61-70.Chaudhary K, Poirion OB, Lu L, Garmire LX. Deep learning–based multi-omics integration robustly predicts survival in liver Cancer using deep learning to predict liver cancer prognosis. Clinical Cancer Research. 2018 Mar 15;24(6):1248-59.González-Reymúndez A, Vázquez AI. Multi-omic signatures identify pan-cancer classes of tumors beyond tissue of origin. Scientific reports. 2020 May 20;10(1):8341.Hoadley KA, Yau C, Wolf DM, Cherniack AD, Tamborero D, Ng S, Leiserson MD, Niu B, McLellan MD, Uzunangelov V, Zhang J. Multiplatform analysis of 12 cancer types reveals molecular classification within and across tissues of origin. Cell. 2014 Aug 14;158(4):929-44.Hoadley KA, Yau C, Hinoue T, Wolf DM, Lazar AJ, Drill E, Shen R, Taylor AM, Cherniack AD, Thorsson V, Akbani R. Cell-of-origin patterns dominate the molecular classification of 10,000 tumors from 33 types of cancer. Cell. 2018 Apr 5;173(2):291-304.Mo Q, Wang S, Seshan VE, Olshen AB, Schultz N, Sander C, Powers RS, Ladanyi M, Shen R. Pattern discovery and cancer gene identification in integrated cancer genomic data. Proceedings of the National Academy of Sciences. 2013 Mar 12;110(11):4245-50.Nguyen H, Shrestha S, Draghici S, Nguyen T. PINSPlus: a tool for tumor subtype discovery in integrated genomic data. Bioinformatics. 2019 Aug 15;35(16):2843-6.Saria S, Goldenberg A. Subtyping: What it is and its role in precision medicine. IEEE Intelligent Systems. 2015 Jul 14;30(4):70-5.Vargas AJ, Harris CC. Biomarker development in the precision medicine era: lung cancer as a case study. Nature Reviews Cancer. 2016 Aug;16(8):525-37.Wu D, Wang D, Zhang MQ, Gu J. Fast dimension reduction and integrative clustering of multi-omics data using low-rank approximation: application to cancer molecular classification. BMC genomics. 2015 Dec;16(1):1-0.3. Authors write the comparison with other models developed for this cancerResponse: We would like to thank the reviewer for the input.Existing works on subgroup identification using pan-cancer data are based on statistical techniques. One of them is based on iClusterPlus ), and the other is on sparse singular value decomposition (sSVD) ). iClusterPlus is one of the widely used techniques to obtain subgroups using multi-omics data ). It is based on generalized linear regression. Here the authors model both discrete and continuous variables using appropriate random variables. A joint parametric model is then used to project this data to a latent space and identify the clusters. Hoadley et al. applied this technique to pan-cancer samples and identified 27 subgroups using methylation, mRNA expression, miRNA expression, and copy number variation ).González-Reymúndez A et al. proposed a statistical technique based on sSVD to identify eight clusters using pan-cancer data ). The authors applied singular value decomposition (SVD) to identify the major axis of variation. This was penalized using the elastic net penalty to induce sparsity. The data projected along the major axis of variation was further reduced using t - Stochastic Neighbor Embedding (tSNE). This reduced dimension representation was clustered using Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to identify the clusters. In contrast to the existing works, which are based on statistical techniques, the current work is based on ML algorithms for the identification of subgroups using multi-omics data. In the proposed work, data from mRNA expression, miRNA expression, DNA methylation, and protein expression are used. The multi-omics data from these levels of evidence is pre-processed, integrated, and projected to a lower dimension using AE, a non-linear dimensionality reduction technique. Consensus clustering is then carried out using this lower dimensional representation to identify the multi-omics subgroups. The clusters obtained using different pipelines are compared using two cluster evaluation techniques, silhouette coefficient, and Calinski-Harabasz index. The closer the value of the silhouette coefficient to one and the higher the Calinski-Harabasz index, the better the clustering. iClusterPlus was applied to the multi-omics data matrix obtained by concatenation of data from different data types. The parameters were tuned using tune.iClusterPlus as recommended by the authors , Mo Q et al. (2023)), and the pan-cancer patients were divided into 13 subgroups. Both the silhouette coefficient and the Calinski-Harabasz index indicated that the clusters obtained using the proposed algorithm were better separated than those obtained using the iClusterPlus . To compare the proposed ML-based pipeline with the work of González-Reymúndez A et al., all the parameters were tuned as recommended by the authors ). Each data type was centered and scaled and then concatenated with other data types to obtain the multi-omics representation of the data. This was then projected to lower dimensional space representing 50 major axes of variation by sSVD. This data was further reduced to 2-dimensional space by tSNE and was followed by clustering using the DBSCAN algorithm. The resulting 15 clusters compared using the Silhouette coefficient and the Calinski-Harabasz index indicated better clusters using the proposed pipeline . The pipelines based on statistical techniques will model the linear relationship appropriately but might not be able to capture the non-linear relationships. However, we know that the interaction between the different features in the genome is non-linear. We believe AE being a non-linear dimensionality reduction technique, was able to better model and capture this variation. Hence, resulted in better clusters as compared to other techniques . Rebuttal Table 1: Comparing the clustering scores between the clusters obtained using iClusterPlus and sSVD followed by DBSCAN with the proposed technique applied to the multi-omics data.Pipeline K Silhouette width Calinski-Harabasz indexiClusterPlus ) 13 0.06 414.03sSVD + DBSCAN ) 15 -0.461 168.851Proposed technique 13 0.24 565.31González-Reymúndez A, Vázquez AI. Multi-omic signatures identify pan-cancer classes of tumors beyond tissue of origin. Scientific reports. 2020 May 20;10(1):8341.Hoadley KA, Yau C, Hinoue T, Wolf DM, Lazar AJ, Drill E, Shen R, Taylor AM, Cherniack AD, Thorsson V, Akbani R. Cell-of-origin patterns dominate the molecular classification of 10,000 tumors from 33 types of cancer. Cell. 2018 Apr 5;173(2):291-304.Mo Q, Wang S, Seshan VE, Olshen AB, Schultz N, Sander C, Powers RS, Ladanyi M, Shen R. Pattern discovery and cancer gene identification in integrated cancer genomic data. Proceedings of the National Academy of Sciences. 2013 Mar 12;110(11):4245-50.Mo Q, Shen R (2023). iClusterPlus: Integrative clustering of multi-type genomic data. R package version 1.34.3.4. Proofread the entire manuscript.Response: We would like to apologize for the inconvenience caused. We have now thoroughly proofread the entire manuscript and made the appropriate corrections.Reviewer #2: 1. It is not clear how many samples were used for training and validation in the three methods SVM: support vector machine, RF: random forest, FFNN: feed-forward neural network from each dataset and author need to mention this in the methods.Response: We would like to apologize for the inconvenience caused. The total number of samples used to carry out the analysis was N = 5703. The train-test split of 90%-10% was used to obtain the train (N = 5132) and test (N = 571) data. The training data was further split (90%-10%) to obtain the train (N = 4619) and validation (N = 513) set. We have trained three ML models, SVM, RF, and FFNN (L0). To obtain a robust classifier, the prediction probabilities from these classifiers were combined, and a decision-level fused classifier (L1) was obtained. We have trained two types of decision-level fused classifiers, one without a hold-out dataset and the other one with a hold-out dataset. In the decision-level fused classifier without hold-out, both L0 and L1 levels of classifiers were trained using all the training samples (N = 4619). For the decision-level fused classifier with hold-out, a hold-out dataset is created by splitting the training dataset in the ratio of 60%-40%. The L0 model is trained using 60% (N = 2771) and L1 using 40% (N = 1848) of the data from the training set. We have now added these details to the manuscript. 2. The authors say that the proposed framework can be used to identify 12 subgroups for new patients and help in personalized clinical decision making. Then this should performed on the another set of new samples as validation set.Response: We would like to thank the reviewer for the input. We understand that the proposed pipeline must be validated in an independent dataset. We are actively looking for a dataset with information from all levels of evidence and clinical data to validate the proposed pipeline. However, we would like to mention that the model was validated using the TCGA samples, which had only methylation values and were not used to obtain the cluster labels or to build the classification models . We do understand this does not answer the question adequately, and validation of the proposed model in an independent dataset will be considered as part of future work.3. Authors need to write in the discussion how these subgroups can be used to design appropriate treatment regimen for different cancers.Response: We would like to thank the reviewer for the input. Using the conventional treatment strategies, all the patients from the same tumor type would be treated using a similar approach. Following the proposed pipeline, samples from the same tumor type were split across multiple clusters, and various pathways enriched in each cluster can be used to target the samples. The pathways enriched in different clusters were identified using metascape analysis  and GSEA analysis .LUAD (Lung adenocarcinoma) samples were divided into six different clusters  and S3 Fig). The GO pathways identified in each of these clusters indicated that they could be targeted using the inhibitors for microtubule bundle formation (C2 - GYN cluster), epithelial cell differentiation (C7 - GI cluster), epithelial cell differentiation (C8 - Bladder cluster), epidermis development (C9 - Squamous cluster), regulation of blood pressure (C10 - Low CNV), and lymphocyte activation (C13 - Mixed).CESC  and ESCA  samples were split across four clusters C2 (GYN cluster), C7 (GI cluster), C9 (Squamous cluster), and C13 (Mixed)  and S3 Fig). The GO pathways that can be inhibited here include microtubule bundle formation (C2), epithelial cell differentiation (C7), epidermis development (C9), and lymphocyte activation (C13).ACC , MESO (Mesothelioma), and PCPG (Pheochromocytoma and Paraganglioma) samples were split into two clusters C11 (M2 phenotype) and C13 (Mixed), which can be targeted using the inhibitors for the GO pathways phenol-containing compound biosynthetic process (C11), and lymphocyte activation (C13).C9 (Squamous cluster) had patients from 13 different tumor types  and S3 Fig). The samples here can be treated using the drugs targeting the positively enriched GSEA pathways, including, MTORC1 signaling, G2M checkpoint, E2F targets, MYC targets, or P53 pathway as summarised in S10 Table.C13 (Mixed) was comprised of samples from 27 different tumor types  and S3 Fig). The positively enriched GSEA pathways that could be targeted here include allograft rejection, IL6 JAK STAT3 signaling, inflammatory response, interferon-gamma response, IL2 STAT5 signaling, TNFA signaling via NFKB, interferon-alpha response, KRAS signaling. These details are now updated in the discussion section of the manuscript.4. The authors have found GO process in each ML cluster and the genes identified in each GO process or pathway can be compared with The Drug Gene Interaction Database to identify clinically actionable gene targets identified in these clustersResponse: We would like to thank the reviewer for the input. We had summarized the top five GO pathways associated with each cluster in S6 Table. We have now used the gene list obtained from each GO pathway to identify the drug-gene interaction and druggable genes in each GO process ). These results are now added as part of the supplementary table (S8 Table). We have also carried out the drug-gene interaction analysis using the DE genes in each cluster. These results are now included in S9 Table.Freshour, Sharon L., et al. "Integration of the Drug–Gene Interaction Database (DGIdb 4.0) with open crowdsource efforts." Nucleic acids research 49.D1 (2021): D1144-D1151.5. The authors need to provide the list of genes identified in each GO process or pathway in these cluster as Supplementary table.Response: We would like to apologize for the inconvenience caused. The differentially expressed (DE) protein-coding genes (PcGs) in each cluster were used to carry out the metascape analysis. The resulting top five GO pathways were summarized in S6 Table. We have now included the gene list associated with each GO pathway as the supplementary table (S7 Table).AttachmentPlosOne_pancancer_rebuttal.docxSubmitted filename: Click here for additional data file. 1 Jun 2023Integration of Pan-cancer Multi-omics Data for Novel Mixed Subgroup Identification using Machine Learning MethodsPONE-D-22-28355R1Dear Dr. Shukla,We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at onepress@plos.org.If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact Kind regards,Amy McCart ReedAcademic EditorPLOS ONEAdditional Editor Comments :Thank you for addressing all the comments made by the reviewers.Reviewers' comments:Reviewer's Responses to Questions  Comments to the Author  1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.Reviewer #2: All comments have been addressedReviewer #3: All comments have been addressed**********  2. Is the manuscript technically sound, and do the data support the conclusions?  The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: YesReviewer #3: Yes**********  3. Has the statistical analysis been performed appropriately and rigorously?   Reviewer #2: YesReviewer #3: I Don't Know**********  4. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.  The Reviewer #2: YesReviewer #3: Yes**********  5. Is the manuscript presented in an intelligible fashion and written in standard English?  PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #2: YesReviewer #3: Yes**********  6. Review Comments to the Author  Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #2: (No Response)Reviewer #3: This is a useful and interesting study to combine pan-cancer cohorts across multiple omics dimensions and then co-clustering after dimensionality reduction to identify pan-cancer clusters. The tumours within each of the clusters, despite coming from different cancer types, have common molecular characteristics. This will be useful to design basket trials for drugs based on molecular composition rather that just cancer type.I do not have any further comments. The authors have addressed the comments from the earlier reviewers satisfactorily.**********  7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.If you choose “no”, your identity will remain anonymous but your review may still be made public.Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.  Reviewer #2: NoReviewer #3: No********** 12 Jun 2023PONE-D-22-28355R1 Integration of Pan-cancer Multi-omics Data for Novel Mixed Subgroup Identification using Machine Learning Methods Dear Dr. Shukla:I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. onepress@plos.org.If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact plosone@plos.org. If we can help with anything else, please email us at Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staffon behalf ofDr. Amy McCart Reed Academic EditorPLOS ONE


To fulfill this, specific high-throughput experimental technologies have been developed to capture TSSs in a genome-wide manner, and various computational tools have also been developed for in silico prediction of TSSs solely based on genomic sequences. Most of these computational tools cast the problem as a binary classification task on a balanced dataset, thus resulting in drastic false positive predictions when applied on the genome scale. Here, we present DeeReCT-TSS, a deep learning-based method that is capable of identifying TSSs across the whole genome based on both DNA sequence and conventional RNA sequencing data. We show that by effectively incorporating these two sources of information, DeeReCT-TSS significantly outperforms other solely sequence-based methods on the precise annotation of TSSs used in different cell types. Furthermore, we develop a meta-learning-based extension for simultaneous TSS annotations on 10 cell types, which enables the identification of cell type-specific TSSs. Finally, we demonstrate the high precision of DeeReCT-TSS on two independent datasets by correlating our predicted TSSs with experimentally defined TSS chromatin states. The source code for DeeReCT-TSS is available at https://github.com/JoshuaChou2018/DeeReCT-TSS_release and https://ngdc.cncb.ac.cn/biocode/tools/BT007316.The accurate annotation of  As mature transcripts produced by RNA polymerase II (RNAPII) have a specific cap structure at their 5′ ends, a so-called cap analysis of gene expression (CAGE) method has been developed to capture the TSSs In addition to the experimental approach, many computational methods have been developed to predict TSSs by learning features in the flanking regions of the annotated TSSs. Two major strategies have been employed in promoter/TSS prediction.cis-elements, and improves the performance by extending the predefined feature sets. For example, TSSG and TSSW cis-element preferences, and potential trans-factor binding sites One focuses on manual extraction of TSSs and promoters related The second strategy applies machine learning models to automatically extract features to predict promoters/TSSs. Promoter2.0 is the seminal work that uses neural networks in the promoter identification More importantly, even though the DNA sequence of a promoter is identical across different types of cells, it might be activated in only certain cell types, which shapes a cell type-specific TSS landscape. However, all of the previous computational methods only consider the DNA sequence information and therefore are apparently unable to determine whether a TSS is active or inactive in a given cell type. Although experimental CAGE-seq or chromatin state-based methods like ChIP-seq are able to identify active TSSs by profiling multiple histone modification markers, they are rather laborious and costly to perform on a grand scale. As a result, other than FANTOM, ENCODE, and a couple of epigenome roadmap projects, the CAGE-seq or histone modification data with ChIP-seq have only been sparsely collected. In contrast, conventional RNA sequencing (RNA-seq) has been routinely used to characterize transcriptional profiles across tremendous amounts of primary tissues and cell lines. Thus, for computational methods to predict active TSSs in a biological sample, it would be convenient to integrate the conventional RNA-seq data with the DNA sequence information to further improve the performance in predicting TSSs precisely. On one hand, RNA-seq data would provide a distinct coverage pattern at a TSS flanking region. Theoretically, across a TSS, there should manifest a sharp increase of RNA-seq coverage. On the other hand, since only expressed regions need to be scanned, by integrating RNA-seq data, the total number of sites for genome-wide TSS prediction could be dramatically reduced.Here, we introduce DeeReCT-TSS, a novel deep learning-based method for the accurate prediction of TSSs by incorporating both DNA sequence and RNA-seq coverage information as a new member in the Deep Regulatory Code and Tools (DeeReCT) family We designed a deep neural network to extract information from both the DNA sequence and the RNA-seq coverage, and the details of data preparation are shown in File S1A. Both inputs went through two independent CNNs that had a convolution layer with 64 filters of width 10 as the first layer to extract motif features and the trend of coverage changes. Following the convolution layer, a rectified linear unit (ReLU) was applied as the activation function followed by the max pooling layer. Two feature vectors were concatenated together and fed into the fully connected (FC) layer. The last softmax layer gave the prediction between 0 and 1. Weight decay and dropout were applied to improve the generalization capability of our method. We used TensorFlow 1.14.0 as the framework of our model, trained the model, and applied the model for prediction with one Tesla V100 GPU on average 2 h for 10,000 epochs and around 30 min for one human sample.Following the work in PromID, we chose 1001 as the length of both the DNA sequence and the corresponding RNA-seq coverage inputs, in order to maintain a balance between retaining enough information and reducing computing resource requirements as longer input means more potential features and greater memory and time requirements. The DNA sequence was one-hot coded with the dimension of 1 × 1001 × 4, in which A was encoded as (1 0 0 0), T was encoded as (0 1 0 0), C was encoded as (0 0 1 0), and G was encoded as (0 0 0 1). The coverage information was strand-sensitively counted based on the RNA-seq data with SAMtools Binary classification is built on balanced datasets, whereas the data in genome scanning are highly unbalanced. Thus, we introduced the iterative negative data enhancement as a negative data argumentation method to our model to reduce the false positive rate (FPR) as shown in Algorithm 1 (File S1B). In brief, for each repetition, we trained the model and selected the best one by evaluating with multiple metrics, including recall, false discovery rate (FDR), and F1-score. Then we randomly selected 100 true TSS positions (+1) from the training dataset and applied the best model on the scanning task in the corresponding scanning window (−5000 bp to +5000 bp). For each scanning window, we divided it into 2 partitions. One was the positive region around the true TSS site , and all others outside the positive region were considered as negative regions. Any site predicted as a TSS in the positive region will be considered as true positive (TP), otherwise false positive (FP). Then half of the negative data in the training dataset was replaced by the randomly selected cases from FPs generated from the scanning task, and the whole process was repeated again for the next repetition. In this way, the results of the initial round were irreverent to the circular training, so we could compare the results of the initial round with subsequent rounds as an ablation study for applying the circular training. Since the positive and negative samples in the genome are extremely unbalanced, the selection of the same number of negative samples as the ground truth (negative TSSs) in the initial round could be not representative enough, which might generate a large number of FPs during genome scanning. Therefore, by using the circular training, the trained model could be able to perform a more accurate TSS classification and reduce the FPR in genome scanning because continuously adding the misclassified data to the training dataset makes the model remember more complicated features from the negative samples.To obtain a more generalized model across different cell types, we incorporated the state-of-the-art meta-learning algorithm and trained the meta-model from 10 cell lines. We integrated the Reptile To ensure that all models had a fair chance of learning a useful representation, we trained multiple instances of each model on the binary classification task using manually generated hyperparameter settings, including learning rate, number of epochs, number of hidden layers, number of filters, batch size, momentum, initial weight, weight decay, and keep probability. Then, we selected model instances based on their training performance and fixed the hyperparameters for all following models as below for easier reproduction of the training process: number of epochs = 10,000, batch size = 4000, learning rate = 1E−4, momentum = 0.98, weight decay = 4E−4, and keep probability = 0.5.To evaluate our method and to objectively compare predictions by our model and other methods for TSS identification, we measured the performance using accuracy, recall, FPR, FDR, and F1-score, which are defined as:In binary classification, one region in our dataset can only be positive or negative and all metrics work without confusion. To better define the TP and FP during scanning along a certain region, we firstly searched for the strongest signal predicted by our model and counted signals with distance ≤ tolerant distance (100 bp) to the nearest real TSS as TP, otherwise FP. In addition, those real TSSs without a strong signal (> 0.5) as a neighbor were counted as FN.Since we randomly shifted the paired DNA sequence and coverage within 50 bp from the CAGE peak to allow a certain level of freedom for data augmentation during the training phase, the sites at the vicinity of true TSSs could also get relatively high prediction scores. Therefore, to reduce FDR by directly using outputs from the deep learning model, we developed a clustering-based method by grouping any sites with a prediction score (outputted from the model and the range is from 0 to 1) larger than 0.5 and within 10 bp into a cluster with merge function from BEDTools Homo sapiens in JASPAR P < 0.001 were selected and a total number of 157 unique TFs were identified. We further measured the expression levels of those TFs in 10 cell types. For each TF, we counted the profile from the fine-tuned models of 10 cell types  and calculated the Pearson correlation between the profile and the expression level in 10 cell types. 42 TFs out of 157 TFs with concordant expression patterns (Pearson correlation coefficient > 0.2) were selected and visualized.To understand sequence features learned by our model during the fine-tuning stage of meta-learning, we paired each filter in the convolution layer of the meta-model and the fine-tuned model of each cell type, and calculated the gain of motif information (GMI), defined as the difference of two positional weighted matrixes, among which each GMI was compared with all transcription factor (TF) binding profiles of For TSS prediction, we built a deep learning model by taking both DNA sequence and RNA-seq coverage information as inputs. Both inputs went through two independent CNN models that had a convolution layer with 64 filters to extract motif features and the trend of coverage changes, respectively. Following the convolution layer, a ReLU was applied as the activation function followed by the max pooling layer. Two feature vectors were then concatenated together and fed into the FC layer. The last softmax layer gave the prediction value between 0 and 1. To improve the generalization capability, weight decay and dropout were applied A.Figure https://fantom.gsc.riken.jp/5/), and then calculated their usage in three cell lines, including a colon carcinoma (COLO-320), a renal carcinoma (OS-RC-2), and a T cell leukemia cell lines (ATN-1), based on the respective CAGE-seq data . The active TSSs from protein-coding genes were used as the positive dataset, with their corresponding DNA sequences and RNA-seq coverages as inputs. Then we randomly picked the same number of regions with a distance between 500 bp to 1000 bp from the nearest TSS peak as the initial negative datasets. Since we selected the activate TSSs as the positive dataset, the number of ground truth TSSs in different cell types was different. An average of ∼ 20,000 active TSSs from on average 11,000 protein-coding genes were obtained in each cell line (Table S1). By integrating RNA-seq data into the model, we could capture the change of the sequencing coverage around the TSS  from FANTOM , while much fewer sites were predicted as TSSs (FP + TP), the FPR derived from FP divided by the sum of FP and TN is extremely low. Therefore, we defined a metric as the FPs in every 1000 bp to evaluate the performance of each method. In all of these three datasets, compared to PromID, our method could achieve a higher recall with a much lower number of FPs in every 1000 bp A. HoweveIn this way, we could reduce FDR dramatically, achieving ∼ 77% recall with ∼ 25% FDR in the colon carcinoma cell line A and B. In comparison, after the clustering procedure, to get a 70% recall, FDR of the PromID method remains as high as 50% A. MoreovTSSs might be only used in certain cell types, and computational methods that are solely based on DNA sequences are unlikely to determine such cell type-specific TSS usage. In comparison, by incorporating RNA-seq information, our method should be able to address this issue. To assess the performance on this perspective, we split the predicted TSSs into three groups: the ones only predicted by our method (Group 1), the ones only predicted by PromID (Group 2), and the ones predicted by both methods (Group 3), and compared them with true TSSs in each cell line. By analyzing the expression levels of these TSSs in the respective cells, we found that the TSSs in Groups 1 and 3 were expressed based on CAGE-seq and the expression levels were only slightly lower than that of the true TSSs, whereas the TSSs only predicted by PromID (Group 2) were almost not expressed at all. Moreover, the expression of TSSs in Group 1 was significantly higher than that in Group 2, suggesting that a much higher proportion of TSSs predicted by our method are truly expressed compared to that predicted by PromID C.We also performed an ablation study by only using DNA sequence information (the sequence only model) or the integration of RNA-seq data and DNA sequence information (the integrated model) to check the contribution of the RNA-seq data in TSS prediction with DeeReCT-TSS. Among 201,802 TSSs annotated in FANTOM, 17,725, 19,721, and 22,171 were used in colon, renal, and T cell with sufficient expression based on the CAGE-seq data and the rest were considered as unused. The integrated model detected slightly more used TSSs than the sequence only model in each cell line. More importantly, the number of FPs, including the unused TSSs and predictions without overlapping with any of the 201,802 TSSs in FANTOM, was much higher in the sequence only model than the integrated model (Table S3).EIF4G3, while only the proximal TSS was used in the colon carcinoma cell line (represented by CAGE-seq), which could be distinguished by our method , 100 M, 50 M, 10 M, and 5 M (where M stands for millions of sequencing reads). We then applied DeeReCT-TSS on each dataset and found that the performance was indeed affected by the sequencing depth. However, the identification of TSSs was quite robust once the sequencing depth was above 50 M (Table S4).We also checked whether DeeReCT-TSS model trained in one species could be applied to another species. For this purpose, we applied DeeReCT-TSS on an RNA-seq data from mouse with a pre-trained model based on the data from human. As shown in Table S5, although the performance of the model on mouse was not as good as that of human, still, the model was able to detect more than half of the truly expressed TSSs. To be noted, unlike datasets from human that had the matched RNA-seq and CAGE-seq data from FANTOM in each cell type, RNA-seq and CAGE-seq data of mouse were from two independent sources. This might also introduce some batch effects for the predictions. Overall, these results showed high robustness of DeeReCT-TSS on RNA-seq data with reasonable sequencing depth and its transferability across multiple species.Since DeeReCT-TSS is able to identify active TSSs in each cell type, we would like to further explore the commonality among different cell types and, at the same time, build a model that can be easily generalized to handle the difference among them. For this purpose, the state-of-the-art meta-learning strategy, which takes advantage of an abstract model trained with data from multiple tasks and is capable of good adaptation to a specific task with a mini fine-tuning session, exactly meets our needs. Here, the more specific aim was to train a generalized model across multiple cell types, which is also capable of being fine-tuned for any specific cell type. To check the feasibility of integrating the meta-learning strategy in DeeReCT-TSS, we extended our analysis to 10 cell lines and a total of 53,969 TSSs that are active in at least one cell type were used as the positive dataset. The number of TSSs that are active across different numbers of the cell lines shows a typical bimodal distribution, where the two peaks represent those only expressed in one cell line and those expressed in all 10 cell types, respectively A. Next, SYTL2, which is active in the colon carcinoma, the gastrointestinal carcinoma, the natural killer T cell leukemia, and the renal carcinoma cell lines. Such usage pattern was successfully predicted by our method. Overall, these results suggest that our model could be generalized to identify active TSSs in different cell types.We further applied the fine-tuned model on the genome scanning task in each cell line, and eventually, we identified TSSs with a recall around 73% and an FDR around 28%. Among these false predicted TSSs, half of them are annotated, but their expression levels do not pass the threshold, and only 10% of the total predicted TSSs are neither annotated nor supported by CAGE-seq . We alsocis-elements extracted from deep learning model in different cell lines are shown in One major advantage of our meta-learning model is that its abstract model can capture common features of TSSs among all cell lines, while the fine-tuned model to each cell line has a preference to identify the active TSSs in the corresponding cell line by enhancing the usage of the cell type-specific features. To this end, we visualized the GMI from the convolutional layers by comparing the difference between paired filters of the meta-model and the fine-tuned model of each cell type and matched those GMI to the motifs of TF-binding sites (see Method). In total, 42 putative TFs with concordant expression patterns (Pearson correlation coefficient ≥ 0.5) were identified across 10 cell lines A and B. As the information for the active TSSs could also be encoded at the chromatin level, to further evaluate the generalization ability on TSS identification of our meta-learning extended model to unseen datasets, we applied it to two cell lines HepG2 and K562, for which ENCODE project collected both RNA-seq and different histone modification data. In total, we predicted 14,569 and 13,465 putative TSSs in HepG2 and K562, respectively. Next, we associated the prediction with chromatin states of these two cell lines, which were identified by using a series of histone modification data and TF ChIP-seq data in the ENCODE project, including Tss, TSS flanking region (TssF), and PromP (inactive promoter). The results showed that our predictions were highly consistent with Tss chromatin state in these two cell lines. Importantly, the overlap between the predicted TSSs in HepG2 and the Tss chromatin state from K562, as well as the overlap between the predicted TSSs in K562 with the Tss chromatin state from HepG2, were much lower than that within the same cell type, suggesting the high cell type specificity of our prediction A.Figure As regions of the ENCODE chromatin state are still very wide (∼ 500 bp), to inspect our predicted TSS position more precisely, we performed the meta-gene analysis of the predicted TSSs with H3K4m1, H3K4m2, H3K4m3, H3K9ac, and H3K27ac ChIP-seq signals in single base-pair resolution. Surprisingly, our predicted TSSs were exactly located within the NDR flanked by the modified histones, where the transcription was likely initiated. Again, this pattern could only be observed by overlapping results from the same cell line B and C, cis-elements residing around the TSS. One shortage limiting their practical applications is that the majority of them can only do binary classification between sequences with TSSs and the ones without TSSs, on a balanced dataset. However, in the human genome, these positive and negative sequences are highly unbalanced, while the number of negative sequences is 15,000 times more than the positive ones. In DeeReCT-TSS, we addressed this challenge by several means. First, we iteratively enhanced the negative dataset, which dramatically decreases the amount of potential FPs by forcing the model to distinguish between active TSSs and inactive TSSs with similar features. In addition, we incorporated RNA-seq data information, which significantly improves the accuracy of the predicted TSS positions. Also, as we filtered out a huge number of query positions, this also helped to decrease the FDR when performing the genome scanning task. Moreover, we introduced a clustering-based method on the prediction scores from the model, which further greatly reduces the FDR. Finally, meta-learning technology worked together with those strategies to ensure the robust and generalizable performance of DeeReCT-TSS in the genome-scale TSS identification task.Computational methods for TSS prediction have been used to annotate TSSs and to study regulatory mechanisms via In spite of the fact that the DNA sequence of a promoter is identical in different cell types and tissues, their transcription profiles are highly variable, resulting in distinct landscapes of active TSSs. In the last decades, the FANTOM project annotated more than 200,000 TSSs in the human genome by combining CAGE-seq data from dozens of cell types, while only a handful of them (∼ 10%) could be active in a single cell type or tissue. Discriminating whether a TSS is active or inactive in each cell type or tissue could be equally important as predicting a novel TSS in studying gene regulation across cell types/tissues and between different physiology conditions. To our knowledge, DeeReCT-TSS is the only computational method that is capable of accurately identifying active TSSs in a cell type by incorporating RNA-seq data and meta-learning. By applying our method, we expect to identify more than 70% of the active TSSs and about 10% of our predicted TSSs might be true FPs. As our method takes both DNA sequence and RNA-seq coverage as inputs, some issues of RNA-seq, such as genes with low expression and misalignment of the RNA-seq reads, might affect the performance of our method. Indeed, we found that our method is more powerful to identify TSSs from genes with higher expression compared to that from genes with lower expression D. This cDeeReCT-TSS also has some limitations in comprehensively identifying TSSs. Since more than 90% of the human genome are non-coding and many non-coding genes may play important roles in gene regulation, the feasibility of applying DeeReCT-TSS on detecting TSSs for non-coding genes remains need to be explored. It might be used for several species of non-coding genes and fail for the rest, depending on the characteristics of non-coding genes, such as the expression level and whether they are independent transcriptional units. In addition, DeeReCT-TSS was trained by using the data from FANTOM5, which has the matched data (RNA-seq and CAGE-seq) for different cell lines but without data for histone modifications and chromatin states. Further integrating epigenetic markers could potentially improve the performance as more information would be taken for TSS prediction. However, to date it is very challenging to find available datasets with these three types of sequencing data simultaneously. Still, we also plan to further improve DeeReCT-TSS and take more sources of information into consideration in the future version, once we can get high-quality data with RNA-seq, CAGE-seq, and CHIP-seq simultaneously.Finally, as demonstrated in https://github.com/JoshuaChou2018/DeeReCT-TSS_release and https://ngdc.cncb.ac.cn/biocode/tools/BT007316.DeeReCT-TSS, which is implemented as an python software and accompanied by a user guide, is available at Juexiao Zhou: Conceptualization, Methodology, Formal analysis, Software, Validation, Data curation, Visualization, Writing - original draft, Writing - review & editing. Bin Zhang: Conceptualization, Methodology, Formal analysis, Software, Validation, Data curation, Visualization, Writing - original draft, Writing - review & editing. Haoyang Li: Formal analysis, Writing - review & editing. Longxi Zhou: Formal analysis, Writing - review & editing. Zhongxiao Li: Formal analysis, Writing - review & editing. Yongkang Long: Formal analysis, Writing - review & editing. Wenkai Han: Writing - review & editing. Mengran Wang: Validation, Writing - review & editing. Huanhuan Cui: Writing - review & editing. Jingjing Li: Software, Validation. Wei Chen: Conceptualization, Investigation, Writing - review & editing, Supervision, Project administration, Funding acquisition. Xin Gao: Conceptualization, Investigation, Writing - review & editing, Supervision, Project administration, Funding acquisition. All authors have read and approved the final manuscript.The authors have declared no competing interests.


We re-analyzed the data from a recent large-scale study that reported strong correlations between DNA signatures of microbial organisms and 33 different cancer types and that created machine-learning predictors with near-perfect accuracy at distinguishing among cancers. We found at least two fundamental flaws in the reported data and in the methods: (i) errors in the genome database and the associated computational methods led to millions of false-positive findings of bacterial reads across all samples, largely because most of the sequences identified as bacteria were instead human; and (ii) errors in the transformation of the raw data created an artificial signature, even for microbes with no reads detected, tagging each tumor type with a distinct signal that the machine-learning programs then used to create an apparently accurate classifier. Each of these problems invalidates the results, leading to the conclusion that the microbiome-based classifiers for identifying cancer presented in the study are entirely wrong. These flaws have subsequently affected more than a dozen additional published studies that used the same data and whose results are likely invalid as well.Recent reports showing that human cancers have a distinctive microbiome have led to a flurry of papers describing microbial signatures of different cancer types. Many of these reports are based on flawed data that, upon re-analysis, completely overturns the original findings. The re-analysis conducted here shows that most of the microbes originally reported as associated with cancer were not present at all in the samples. The original report of a cancer microbiome and more than a dozen follow-up studies are, therefore, likely to be invalid. Helicobacter pylori for stomach cancer  with large numbers of mislabeled sequences. The biggest problem, as reported in one study , is thatThis contamination problem is of particular concern when using a metagenomics analysis method to classify reads that are derived from a human sample and that have a relatively small number of microbial reads –9. RecWe re-analyzed all of the raw and normalized taxonomic classification data from the Poore et al. study , which iAs described by Poore et al. , their aTo illustrate, we re-aligned all of the initially unmapped reads from the 1,255 BLCA, HNSC, and BRCA samples shown in the last column of Thus, in each of these data sets, the strategy of relying on the mapping information in the raw BAM files, as Poore et al. did, leaves on average 1.4–3.3 million human reads in each sample.The presence of millions of human reads in the samples means that in the primary analysis step in Poore et al., where they matched all of these reads to a microbial database, any microbial genome that contained short regions of human sequences could generate large numbers of false-positive matches; i.e., reads that were reported to match a bacterial genome when in fact the reads were from human DNA. As mentioned above, thousands of draft genomes do indeed contain small amounts of human DNA sequences that are erroneously labeled as bacterial .The TCGA read data were analyzed with the Kraken program , a very We re-analyzed 156 bladder cancer samples  by matching them against a curated, publicly available Kraken database  that conStreptococcus, Mycobacterium, and Staphylococcus, with average read counts per sample of 560,000, 411,000, and 241,000, respectively. In our re-analysis of the same samples, we found far fewer reads in these genera: an average of 36, 6, and 266 reads, respectively, values that are 16,000, 67,000, and 900 times smaller. Table S2 shows the top 20 genera found in our analysis, which had abundances ranging from 10 to 447 reads per sample.As shown in the figure, the top genera from Poore et al. were As we describe below, the vast majority of the excessive counts in the Poore et al. study were apparently due to human reads in the filtered data that were incorrectly labeled as bacterial. Because filtering the raw reads only against GRCh37 or GRCh38 did not remove all human reads, the input to their metagenomics pipeline included 1.4–3.3 million human reads per sample, and these reads explain the dramatic over-counts shown on the left side of Campylobacter). Nearly half of the top-weighted genera had average read counts below 10 in the Poore et al. data as well, although several had counts in the thousands. Below we explain how genera with raw counts near zero were selected by the machine-learning models.We also compared the read counts for the genera that were given the highest weights in the machine-learning models created by Poore et al. Tables S3 and S4 show the average read counts for the 20 top-weighted genera in models that classified bladder cancer tumors vs other tumor types and tumors vs normal tissue. In our analysis, nearly all counts for these genera averaged between 0 and 1, with a maximum value of just 18 reads . As with the BLCA samples, we filtered to remove reads matching the CHM13 human genome and then used the Kraken program to match all reads against a curated database of microbes, common vectors, and the human genome. Note that even after filtering by alignment to the human genome, an average of 227,272 (5%) and 34,806 (1%) of the reads in the HNSC and BRCA samples, respectively, were classified as human by Kraken.Figure S1 and S2, and Tables S5 and S6 show the read counts for the 20 most abundant genera in the HNSC data  and BRCA data  as computed by Poore et al. contrasted with our read counts for the same genera in the same samples. As with the bladder cancer results shown in We then looked more broadly at all of the read counts, for all genera, for the BLCA, HNSC, and BRCA whole-genome samples, focusing on the non-zero counts reported by Poore et al. Note that in the original raw data matrix, 21,074,259 (60%) of the entries were zero and 14,071,985 (40%) were non-zero. Each cell in the matrix represents a sample/genus pair, i.e., the count of the number of reads from a given sample that were assigned to a given genus.As shown in the table, in the BLCA samples the number of reads reported by Poore et al. was at least 10 times larger than our results for 98.5% of the data entries. If we looked only at samples and genera where Poore et al. found ≥100 reads, their value was more than 10 times too large in 99.4% of all cases. The results were similar for HNSC, where 92.9% of values were at least 10 times too high, and in BRCA where 97.9% of the read counts were inflated at least 10-fold.We also computed how many of the non-zero read counts were at least approximately the same as the value as determined in our re-analysis. In the BLCA samples, only 90 out of 37,258 (0.2%) were within 50% of the counts that we found in the same samples. Equivalently, fewer than 1 in 400 non-zero values in the bladder cancer data were within 50% of the value found upon re-analysis. The HNSC and BRCA read counts were only marginally better, with just 1.0% and 0.2%, respectively, within 50% of the correct value. Thus, the vast majority of the non-zero data in Poore et al.’s original data matrix—the data upon which all of their results were based—appears to be wrong, by very large amounts.The likely reason for these vast over-counts is that human reads were erroneously categorized as bacterial by Poore et al. The number of human reads matching bacteria was unrelated to the actual presence of bacteria in the tumor sample; instead, it was determined by the database itself, in which many draft bacterial genomes contained mislabeled human sequences.Streptococcus and Waddlia, in one primary tumor sample, s2707 (case ID TCGA-DK-A1AB), from the BLCA data. We chose these genera because they were reported to be among the most abundant in Poore et al., as shown in Streptococcus reads and 20,673 Waddlia reads. When we aligned s2707 to our Kraken database, which only contains complete bacterial genomes, we found just one read labeled as Streptococcus and none labeled as Waddlia.To illustrate how such high read counts can appear when few or even no reads from a bacterial genus are present, we did a deep analysis of two genera, Streptococcus genomes (including draft genomes) available in GenBank as of 2016.  that are publicly available. We then ran KrakenUniq , to find large numbers of reads from each of these genera when aligning against a database built entirely from bacterial genomes, as long as that database does not contain the human genome.We then extracted all reads from s2707 that did not match the human genome, which comprised 11,997,726 unmapped reads. Next, we built a custom Kraken database containing all 10,270 une 2016 .) We buiakenUniq  to map tStreptococcus or Waddlia in the Kraken analyses above and aligned them to the CHM13 human genome using Bowtie2 . All of the ACC cancer samples had raw read counts of zero for this virus, but during normalization, 71 of the 79 samples (90%) were assigned the value 3.078874655 by Poore et al. Out of all 17,625 samples across all cancer types , only 77 other samples had a value equal to or smaller than this value in the normalized data. In the raw data, however, 17,624 samples had zero Hepandensovirus reads, and one sample had two reads.First, consider the values for Hepandensovirus value HN, then if the model splits the samples using the simple rule HN > 3.078874655, it will label 71/79 (90%) of the positive samples correctly, and only make 77/17,625 (0.4%) errors (Hepandensovirus was the highest-weighted feature for the machine-learning model distinguishing ACC from other cancers in the most stringent decontamination (MSD) data set, despite the fact that only 1/17,624 samples had any reads at all matching this virus.As illustrated in ) errors . This exThiorhodospira, for the Kidney chromophobe (KICH) tumor samples  data set]. The TCGA-KICH data contained 51 tumor samples and 41 normal tissue samples, and in the raw data, 85 samples had read counts of zero, and 7 samples  had counts of 1 for Thiorhodospira, meaning that it had almost no utility as a discriminating feature. In the normalized data, though, the cancer samples were assigned an almost perfectly disjoint set of values from the normal tissues, as shown in We observed a similar pattern in the normalized values of another genus,  samples . ThiorhoNitrospira, which was a highly weighted genus for the machine-learning models in 13 different cancer types in Poore et al., including lung squamous cell cancer (LUSC) where it was the top-ranked genus. Nitrospira reads, after Voom-SNM normalization, in the LUSC samples compared to all other cancer types. In the figure, the frequencies of Nitrospira in LUSC are shifted to the right, i.e., they have larger values on average than other cancers. This explains why the machine-learning model gave Nitrospira the highest weight; however, in the raw data, there is no such shift to the right. Thus, the Voom-SNM normalization process created a signature of lung cancer even though no such signature was present in the original read counts.Another example is Mulikevirus. This genus had the highest weight both for distinguishing tumor from normal tissue and for distinguishing HNSC from all other cancers. All 906 HNSC samples, including tumor, blood, and normal tissue, had zero reads in the raw data for Mulikevirus, making this virus useless at discriminating between tumor and normal samples.We observed this phenomenon again in HNSC, where the genus with the highest weight in the MSD data set was Mulikevirus, despite the fact that all the raw read counts were zero.However, in the Voom-SNM normalized data, values for the 70 normal samples were set to lower values than any of the tumor samples, as shown in Hepandensovirus were erroneously tagged with tumor-type specific values, we wanted to explore how this tagging would affect the performance of machine-learning classifiers on a larger selection of tumor types and taxa. To investigate this question, we extracted a completely empty microbial-sample matrix  from the raw Kraken classification data provided in the Poore et al. study. To obtain the empty matrix, we filtered the data to retain only genera present in fewer than 50 samples, and then removed any samples with non-zero values for any genus. This produced a matrix containing 16,567 samples and 170 genera in which all values were zero. No machine-learning classifier can use such data to discriminate among cancer types, because every entry in the matrix is identical.Given that individual genera such as N = 12,803) so that we could attempt to build classifiers discriminating each cancer type from all others.We then populated each cell in the empty matrix with its corresponding value from the Voom-SNM normalized data. For this experiment, we used the Voom-SNM data from the “MSD” data set, which included only 66 of the 170 taxa in our initial empty matrix. We then filtered to retain only primary tumor samples , which are unlikely to be present in human samples.A 2023 study by Parida et al.  that in the raw data had zero discriminative power. This error seems to have involved every tumor type and many genera that had zero or near-zero read counts across all of the human samples.Either of these two errors suffices to invalidate the conclusions of the Poore et al. study and of the other studies that relied upon the same data. The original data matrix of raw read counts contained millions of wildly inaccurate values, and the normalized data compounded this error by tagging the cancer types with distinctive normalized values. Our conclusion after re-analysis is that the near-perfect association between microbes and cancer types reported in the study is, simply put, a fiction.gdc.cancer.gov) for three types of cancer from the TCGA project: bladder cancer, head and neck cancer, and breast cancer. These were chosen randomly from the 15 cancer types in Poore et al. for which both cancer and normal tissue samples were available. For TCGA-BLCA, we downloaded read data from 683 samples, which included 277 WGS samples and 406 RNA-seq samples , and the GRCh38 human genome. This 384 GB KrakenUniq database is available for download from https://benlangmead.github.io/aws-indexes/k2. Files containing lists of all species, genera, and NCBI accession numbers in this database are available as Data files 1 to 3. All supplemental files and tables from this study are available at https://github.com/yge15/Cancer_Microbiome_Reanalyzed.Using these two-pass filtered files, for all samples in the BLCA, HNSC, and BRCA data, we ran KrakenUniq  against http://cancermicrobiome.ucsd.edu/CancerMicrobiome_DataBrowser. These include all the classifiers for the APCR and MSD data sets (separate classifiers were created for each data set) and for those data sets, the table includes the top genera used for classifying one cancer type vs all others and for distinguishing tumor from normal tissue.


The true accuracy of a machine-learning model is a population-level statistic that cannot be observed directly. In practice, predictor performance is estimated against one or more test datasets, and the accuracy of this estimate strongly depends on how well the test sets represent all possible unseen datasets. Here we describe paired evaluation as a simple, robust approach for evaluating performance of machine-learning models in small-sample biological and clinical studies. We use the method to evaluate predictors of drug response in breast cancer cell lines and of disease severity in patients with Alzheimer’s disease, demonstrating that the choice of test data can cause estimates of performance to vary by as much as 20%. We show that paired evaluation makes it possible to identify outliers, improve the accuracy of performance estimates in the presence of known confounders, and assign statistical significance when comparing machine-learning models. •Paired evaluation enables detection of outliers and statistical comparison of models•The method can evaluate the impact of confounders without model retraining•The method scales efficiently to millions of data points Limited data availability, outliers, and confounding variables are ubiquitous in biological and clinical datasets and present challenges for evaluation and comparison of machine-learning models. We present a method that uses simple pairing of data samples to overcome those challenges and reveal how individual outliers and confounders impact performance estimates without the need to train multiple models. We also propose an efficient implementation of the method, allowing it to be applied at scale. A simple method based on pairing of data samples allows for effective evaluation of performance in machine-learning models. The method helps reveal how outliers and confounders affect performance estimates and model interpretation. An effective implementation of the method enables its application in modern large-scale datasets. Effectively evaluating the performance of predictive computational models is a crucial aspect of machine learning. Knowing when a model is accurate allows for reliable predictions on new data and provides valuable insights about which features in the training data carry predictive information. However, the true accuracy of a model is a population-level statistic that is generally unknown, because it is impossible to consider all—potentially infinitely many—datasets to which a model will be applied. Model performance must therefore be estimated by appropriately sampling available data, and reliable estimates require a sufficient number of points to be adequately representative of the population. The presence of systematic biases and confounding variables can lead to incorrect accuracy estimates and inflated confidence in machine-learning models that are subsequently found to perform poorly in deployment.,k-fold; leave-one-out; Monte Carlo methods, in which a fixed proportion of data are repeatedly sampled and withheld for evaluation; and bootstrap methods, where the withheld portion is automatically defined by the data not sampled for training.When data are limited (as they commonly are in biomedicine), model performance is routinely evaluated using cross-validation, which involves withholding a portion of the available data and using the remainder to train a model, which is then evaluated against the withheld portion.4 for genome-scale data) from a relatively small number of samples (10–100). While conducting more experiments to increase sample size is sometimes possible, the small-sample issue is insurmountable in other cases due to the limited availability of biological material  and the significant cost associated with molecular profiling. For example, cell culture studies focused on breast cancer are generally limited to the ∼75 commercially available breast cancer cell lines. While deriving new cell lines is possible, it is time consuming and expensive.The limitation of data availability is particularly prominent in -omics datasets, which commonly contain many molecular measurements .δ, which sets a minimum required distance of separation in the label space for a pair of samples to be considered rankable . In practice, clinical subtype determines how a cancer will be treated. Breast cancers are also classified based on gene expression profiles into four intrinsic molecular subtypes: luminal A, luminal B, basal, and HER2 enriched.AOC)—from the baseline mRNA expression of a set of pre-selected genes. Random forests were selected to model potential non-linear relationships in the data and because more complex models, e.g., neural networks, are overparameterized for the small number of data points (cell lines) in the dataset.We considered a dataset recently collected in our laboratory that characterizes the sensitivity of 63 breast cancer cell lines of different subtypes to 72 small molecule drugs, with a focus on kinase inhibitors. The dataset comprises growth rate-corrected measures of drug sensitivity  was greater than experimental error. For a given drug-cell line combination, the experimental error was taken to be the standard deviation of GRAOC across three or four technical replicates. For any pair of cell lines, the larger of the two standard deviations was then used as the value for δ to determine if that pair was rankable. For most rankable pairs, this corresponded to a difference in GRAOC of δ < 0.3 , with the total number of rankable pairs on the order of hundreds for each drug  inhibitor,ERRB2 (HER2) increased in importance . Several recent studies—those from the Accelerating Medicines Partnership – Alzheimer’s Disease (AMP-AD) program, for example,,Here, we consider the task of predicting disease severity from mRNA expression. We make use of the data collected by two joint longitudinal cohort studies, the Religious Orders Study (ROS) and the Memory and Aging Project (MAP), that comprise over 200 bulk RNA-seq profiles of postmortem brain specimens, along with matching pathology annotations.AD progression takes place on a timescale of years, and disease severity is strongly correlated with a patient’s age of death (AOD) A. An impAOC), and the confounder was represented by a discrete variable . The opposite is true of ROSMAP data; the labels are discrete, resulting in a straightforward definition of rankable pairs: two brain specimens are rankable if they have distinct Braak stage ranges. Conversely, age is a continuous variable that is censored at 90 years old (y.o.). The censoring provides a natural inflection point for determining whether two data points are matched in age, giving rise to two evaluation scenarios  implementation of paired evaluation using inversion counting  could be based on variance across biological or technical replicates .Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Artem Sokolov  satisfyf is a distance function and δi,j is the minimum necessary threshold of label separation. In classification, f was set to be an indicator function returning 0 if the arguments are identical and 1 otherwise, while δi,j was set to 0.5 for all . In linear regression, f was the L1-norm distance |yi − yj| in the label space, and δi,j = max was taken to be the expected measurement error, estimated by the standard deviations σi and σj computed across technical replicates.We consider a pair of samples R, AUC is estimated bypij is an indicator variable that takes on the value of 1 when the pair of samples i,j is correctly ranked by the predictor, and 0 otherwise.Given the space of rankable pairs To determine if a rankable pair is ranked correctly by a model, paired evaluation requires that the corresponding predictor assigns scores to both samples in the pair. The scores can be probabilities that the input sample belongs to the positive class in binary classification, ranks in a recommender system, or predictions of real-valued measurements. Since pairwise comparison of scores by paired evaluation is a separate task, the scores from multiple test partitions can be pooled together. This allows paired evaluation to be applied in any cross-validation setting, including leave-one-out cross-validation, as long as pooled prediction scores can be ordered by their corresponding representation . If cross-validation produces multiple scores for a single sample, e.g., through repeated random partitioning into 80% train/20% test splits, we use the score average for that sample in paired evaluation. An exception to the above rule is LPOCV, where each rankable pair can be evaluated directly, without the need for score averaging.y1 ≤ y2 ≤ … ≤ yn that designate the desired ranking of samples, paired evaluation counts the number of pairs that are actually ranked correctly when the samples are sorted based on the scores produced by a given machine-learning model. This is a well-known problem in computer science called inversion counting and an O(n log n) implementation is given by the following modified merge sort algorithm:1.Sort the samples based on the scores produced by a model2.i designate the label of the sample in the ith positionLet z3.count, where l and r are the left and right endpoints of an interval along the sorted list. Inside this function,a.inv = 0Initialize b.r < lTerminate recursion if c.m = floor((l + r) / 2)Compute the midpoint of the interval: d.Recurse on each half: inv = inv + count + counte.i.zi≤ zjDuring the traversal, no inversions are counted whenever ii.i and the midpoint is an inversion relative to j: inv = inv + m − i +1Otherwise, everything in the left half of the interval between the current pointer Count the inversions using the standard merge sort loop, by initializing pointers i = l, j = m + 1 and traversing the two halves of the interval, while i ≤ m and j ≤ r is satisfiedf.inv from the functionReturn the overall tally We define a function 4.)/|R|, where R is the set of all rankable pairs as before.The total number of correctly ranked pairs is the number of rankable pairs minus the total number of inversions. Thus, AUC is given by Given a test dataset of n samples with the corresponding labels k-th sample as,We define the sample-specific AUC for the Rk ⊂ R is the subset of all rankable pairs that include the sample k, and pi,j has the same interpretation as in k than the overall AUC were considered to be potential outliers and inspected in more detail to decide whether they warrant an exclusion from the study. As with method comparison , the values were matched exactly. For continuous variables, we selected a single rankable pair per sample, such that the difference between the two values of the confounder was minimized. A possible unexplored alternative was to consider all samples that fell within a certain predefined "match" window for a given index sample. In all cases, we refer to resulting subsets of rankable pairs as confounder matched and the remaining rankable pairs as confounder mismatched.If AUC estimated on confounder-matched pairs was significantly lower than its equivalent derived from confounder-mismatched pairs, then this was interpreted as a strong indication that the corresponding predictor has learned to recognize the confounder instead of the variable of interest. Statistical significance was again assessed with a Fisher’s exact test applied to a two-by-two contingency table cataloging whether rankable pairs were more likely to be ranked correctly if they are confounder matched or confounder mismatched.