The advance of mass spectrometry-based technologies enabled the profiling of the phosphoproteomes of a multitude of cell and tissue types. However, current research primarily focused on investigating the phosphorylation dynamics in specific cell types and experimental conditions, whereas the phosphorylation events that are common across cell/tissue types and stable regardless of experimental conditions are, so far, mostly ignored.Here, we developed a statistical framework to identify the stable phosphoproteome across 53 human phosphoproteomics datasets, covering 40 cell/tissue types and 194 conditions/treatments. We demonstrate that the stably phosphorylated sites (SPSs) identified from our statistical framework are evolutionarily conserved, functionally important and enriched in a range of core signaling and gene pathways. Particularly, we show that SPSs are highly enriched in the RNA splicing pathway, an essential cellular process in mammalian cells, and frequently disrupted by cancer mutations, suggesting a link between the dysregulation of RNA splicing and cancer development through mutations on SPSs.https://github.com/PYangLab/SPSs under the open-source license of GPL-3. The data used in this study are publicly available (see Section 2.8).The source code for data analysis in this study is available from Github repository Bioinformatics online. Protein phosphorylation regulates diverse protein function, including the catalytic activity and stability of proteins, dictates their subcellular localization and controls the dynamics of protein–protein interaction (PPI) . WorkingRecent advances in mass spectrometry-based technologies  and the Here, we hypothesize that, akin to stably expressed genes such as housekeeping genes (http://qphos.cancerbio.info) (http://qptm.omicsbio.info) , and undergoing minimal changes of phosphorylation level across different biological processes or under a wide range of perturbations (i.e. phosphorylation changes), we developed a statistical framework to identify SPSs that are characterized by these two stability features from 53 high coverage human phosphoproteomic datasets ≥5000 phosphosites and ≥2 conditions relative to control) curated from the qPhos database (io.info) . SpecifiX1 and X2 for recurrence and phosphorylation change, and derived the P-values from the upper-tail of the model fitted to X1 (i.e. recurrence) and lower-tail of the model fitted to X2 (i.e. phosphorylation change), respectively:To obtain a statistical significance for the phosphosites with respect to their stability, for each phosphosite, we first fitted a gamma distribution to the values of each of the two stability features, denoted as P-values for each phosphosite using Fisher’s method to derive a single statistical significance:P is a vector of the two P-values obtained from the two stability features for each phosphosite.We then combined the two P < 0.01 were defined as SPSs , and sites that are not defined as SPSs (i.e. non-SPS) in terms of their maximum absolute log2 fold change in each of the three datasets. Then, for each of the three datasets, we performed principal component analysis (PCA) and hierarchical clustering using either data subset by SPSs or all sites. We quantified the concordance of clustering output with pre-defined labels (time points or conditions) in each dataset using five metrics, including adjusted Rand index (ARI), Fowlkes–Mallows index (FMI), normalized mutual information (NMI), purity and Jaccard index, with the expectation that data subset by SPSs will have significantly lower concordance given they are stably (unchanged) phosphorylated irrespective to time points or conditions   databases, and also against cancer gene neighborhoods (CGN) collected from MSigDB  , ReactomThe relative enrichment of cancer mutations in spliceosome-associated SPSs was performed using odds ratio test. The significance of odds ratios and confidence intervals were estimated based on approximation, followed by null-hypothesis , as implemented in the fmsb R package .http://qphos.cancerbio.info) and its updated version (http://qptm.omicsbio.info).The phosphoproteomic data described in this study are publicly available. In particular, the mouse ESC dataset , we obtained a total of 326 phosphosites that were considered as highly SPSs  .Characteristically, SPSs by nature of their stability are expected to demonstrate a low capacity to discriminate between samples or timepoints. We would therefore expect that if SPSs are used to discriminate phosphoproteomic data covering distinct samples in replicates, they would do poorly to recapitulate the similarity within replicates of the same samples and variability among different samples. To investigate this, we applied PCA and hierarchical clustering on the mESC differentiation dataset using either SPSs, a size-matched random set or all the phosphosites. Using all the phosphosites or the random set, we observed a strong clustering of biological replicates as well as a clear ordering of the time points that were in line with the expected findings. In contrast, neither the clustering of biological replicates nor the trajectory of the differentiation was observed using SPSs . We obseTo comprehensively characterize SPSs, we assessed its various features on both phosphosite- and protein-level. We first derived the functional scores for each phosphosites from a previous study which examined phosphosites functionality  Investigator [1173469 to P.Y.], Children’s Medical Research Institute Postgraduate Scholarships to D.X. and H.J.K., and an Australian Research Council (ARC) Postgraduate Research Scholarship to H.J.K. and Luminesce Alliance—Innovation for Children’s Health established with the support by the NSW Government. Conflict of Interest: none declared. btac015_Supplementary_DataClick here for additional data file.


Large-scale multiparameter screening has become increasingly feasible and straightforward to perform thanks to developments in technologies such as high-content microscopy and high-throughput flow cytometry. The automated toolkits for analyzing similarities and differences between large numbers of tested conditions have not kept pace with these technological developments. Thus, effective analysis of multiparameter screening datasets becomes a bottleneck and a limiting factor in unbiased interpretation of results. Here we introduce compaRe, a toolkit for large-scale multiparameter data analysis, which integrates quality control, data bias correction, and data visualization methods with a mass-aware gridding algorithm-based similarity analysis providing a much faster and more robust analyses than existing methods. Using mass and flow cytometry data from acute myeloid leukemia and myelodysplastic syndrome patients, we show that compaRe can reveal interpatient heterogeneity and recognizable phenotypic profiles. By applying compaRe to high-throughput flow cytometry drug response data in AML models, we robustly identified multiple types of both deep and subtle phenotypic response patterns, highlighting how this analysis could be used for therapeutic discoveries. In conclusion, compaRe is a toolkit that uniquely allows for automated, rapid, and precise comparisons of large-scale multiparameter datasets, including high-throughput screens. Biology has seen huge advances in technology in recent years. This has led to state-of-the-art techniques which can test hundreds of conditions simultaneously, such as how cancer cells respond to different drugs. In addition to this, each of the tens of thousands of cells studied can be screened for multiple variables, such as certain proteins or genes. This generates massive datasets with large numbers of parameters, which researchers can use to find similarities and differences between the tested conditions.Analyzing these ‘high-throughput’ experiments, however, is no easy task, as the data is often contaminated with meaningless information, or ‘background noise’, as well as sources of bias, such as non-biological variations between experiments. As a result, most analysis methods can only probe one parameter at a time, or are unautomated and require manual interpretation of the data.Here, Chalabi Hajkarim et al. have developed a new toolkit that can analyze multiparameter datasets faster and more robustly than current methods. The kit, which was named ‘compaRe’, combines a range of computational tools that automatically ‘clean’ the data of background noise or bias: the different conditions are then compared and any similarities are visually displayed using a graphical interface that is easy to explore.Chalabi Hajkarim et al. used their new method to study data from patients with acute myeloid leukemia (AML) and myelodysplastic syndrome, two forms of cancer that disrupt the production of functional immune cells. The toolkit was able to identify subtle differences between the patients and categorize them into groups based on the proteins present on immune cells.Chalabi Hajkarim et al. also applied compaRe to high-throughput data on cells from patients and mouse models with AML that had been treated with large numbers of specific drugs. This revealed that different cell types in the samples responded to the treatments in distinct ways.These findings suggest that the toolkit created by Chalabi Hajkarim et al. can automatically, rapidly and precisely compare large multiparameter datasets collected using high-throughput screens. In the future, compaRe could be used to identify drugs that illicit a specific response, or to predict how newly developed treatments impact different cell types in the body. Technological developments have accelerated the generation of large-scale multiparameter screening data through methodologies such as high-content microscopy and high-throughput flow cytometry . These tThere have been efforts to cluster samples from large-scale multiparameter  screening data. A simple approach is to use a representative value for each cell marker such as median fluorescence intensity (MFI) for clustering samples . HoweverAvailable computational toolkits  mostly acompaRe performed robustly in the presence of background noise and batch effects even where these input data artifacts could not be corrected. compaRe analyses of multiparameter mass and flow cytometric data from acute myeloid leukemia (AML) and myelodysplastic syndrome (MDS) patient samples revealed interpatient heterogeneity and recognizable phenotypic profiles. When applied to high-throughput flow cytometry of the dose response of AML samples treated with various drugs, compaRe successfully corrected for various sources of bias and clustered the samples based on their response to treatment, allowing for detection of both drastic and subtle phenotypic responses.compaRe is designed to analyze the data from small to large-scale multiparameter screening assays such as high-throughput flow cytometry, high-content microscopy, mass cytometry, and standard flow cytometry. The toolkit comprises several modules for quality control, bias correction, clustering, and visualization. At the core of the compaRe toolkit is a module for pairwise comparisons of samples. It measures the similarity between two samples using a dynamic mass-aware gridding algorithm  and meta-clustering with PhenoGraph (for simplicity just meta-clustering) . We analEven with the added noise, the comparison module correctly identified similar samples . ConversNotably, compaRe took only 25 min to analyze the 21 samples (210 pairwise comparisons), without subsampling or dimension reduction. Meanwhile, meta-clustering and JSD took 39 hr and 10 hr respectively. For the feasibility of JSD, we subsampled each sample to 100,000 cells  markers . Taking Non-AML myeloid neoplasias such as MDS can evolve to become AML. Over time, about one-third of all MDS cases develop into AML . The risUnlike the backbone markers, the characterization markers are divided into each tube exclusively. This design was made so that characterization markers from different tubes can be inferred on the same backbone marker subpopulations, but the design makes it impossible to form a multiparameter dataset which is required for clustering methods. However, as compaRe’s comparison module can compare cell population morphologies even in subspaces, we were able to use it to measure similarities between patient samples.We analyzed 25 bone marrow mononuclear cell samples collected from 16 MDS patients and 9 AML patients . The comWe further investigated how different the three groups of the AML samples were . AML samIn conclusion, compaRe’s comparison module can be used to optimize true cytometric n-dimensional immunophenotypic characterization of patient samples. Interpretation can then be performed in a conventional manner assisted by lower-dimensional projection tools such as PCA and UMAP that promptly provide a phenotypic profile of the patient samples.We applied compaRe to high-throughput flow cytometry data to identify cell subtype-specific responses evoked by antineoplastic agents in leukemic spleen cells from an AML mouse model. Splenic cells were sorted for c-Kit cell surface expression, allowing for the enrichment of stem/progenitor-type leukemic cells. On ex vivo expansion, these cells continuously expand and differentiate in a similar way as in vivo with a clear stem cell/progenitor population and partial differentiation towards CD11b/Gr-1 or CD16/CD32-expressing myeloid cells. After ex vivo expansion, the leukemic cells were plated onto multi-well plates containing a library of 116 antineoplastic agents including surface and nuclear receptor inhibitors and activators, enzyme inhibitors and, cytotoxic chemotherapy in a five-point concentration range, as well as 20 negative control wells . After 7compaRe corrected the intraplate signal drift, sources of bias in cell numbers, as well as inter-plate sources of bias . After cTo get an overview of the assay, we generated a dispersion map of the clusters . Most ofIn high concentration, molibresib and birabresib, inhibitors of BET proteins BRD2, BRD3, and BRD4, caused a reduction in live cell counts but also a reduction of MFI in all the measured markers, which corresponds to the loss of differentiation marker positive cells  . The BRDIn this cell model, the leukemic stem-like cells are expected to be present within the differentiation marker negative population. These cells are potential targets for treatments against leukemia. We observed response group 2  had a hiTaken together, compaRe analysis of the high-throughput flow cytometry screening data allowed rapid identification of several distinct phenotypic responses in this mouse AML model, as well as the cellular signals that drive them. Drugs of different mechanism of action can still cluster together if the cellular processes they affect converge in a specific model. Drug response in association with genetic alterations can be one of the applications of compaRe. The genetic alteration could be visualized in the clusters that compaRe identifies.We further applied compaRe to the drug screening data from an AML patient sample. Primary AML bone marrow mononuclear cells were dispensed into a 384-multiwell plate containing a library of 40 drugs and drug combinations in seven-point concentration ranges . After 7compaRe analysis identified several distinct response groups . ResponsResponse group 2 consisted of two drugs: birabresib and lenalidomide in different concentrations. These induced a decrease in the MFI of CD45-RA and CD45 channels . In the compaRe also detected response group three as distinct from the controls. This group includes treatment with tretinoin , navitoclax, and mitoxantrone (low dose). Further validation showed the phenotypic response in group three is subtle but with a distinct increase in CD34+ cells . This reTechnological advancements in multiparameter high-throughput screening have enabled testing thousands of biological conditions in a short amount of time. This requires algorithmic development to analyze the large amount of data generated by such technologies. We developed an automated comprehensive toolkit, compaRe, for robust analysis of small- to large-scale multidimensional screening data with several modules for quality control, bias correction, comparison, clustering, and visualization.The toolkit is unique in many ways. Its quality control and bias correction modules can correct for signal and cell viability drifts in large-scale fluorescence-based screening assays using regression analysis. Its comparison module utilizes a dynamic mass-aware gridding algorithm, which substantially reduces the computing cost and provides robustness to signal shift . Alternative approaches such as meta-clustering and JSD require both sub-sampling of the data, with the possible loss of valuable subpopulations, and considerably more computing time.We tested the robustness of the comparison module to batch effect and noise through simulation. The module effectively circumvented the batch effect while JSD and meta-clustering significantly suffered from it. The poor accuracy of meta-clustering demonstrates the drawback of using cluster centroids for similarity comparison across samples while the poor performance of JSD indicates that this approach can work well only in the absence of signal shift. It is of particular note that compaRe does not need subsampling or dimension reduction of the input data.Multiparameter cytometric analysis of immunophenotypes of AML and MDS patient samples by the comparison module coupled with the EuroFlow AML/MDS antibody panel revealed interpatient heterogeneity and recognizable phenotypic profiles. Even though EuroFlow markers are divided into several discrete tubes, compaRe’s comparison module can compare the cell population distribution to measure similarities between patient samples.We investigated several types of responses evoked by different doses of antineoplastic agents in two high-throughput flow cytometry screening assays of an AML mouse model and an AML human patient. We could identify subtle but distinct phenotypic drug-induced changes. We also identified drugs with different mechanism of action but similar responses. In general, we showed that drugs will cluster together if the cellular processes they affect converge in a specific model.The quality control and bias correction modules could successfully correct for signal and cell viability drifts in these studies. In our explored assays, signal drift was obviously associated with the order in which wells were read. It was caused by the time differences in antibody incubation across the plate as the high-throughput flow cytometer requires more than one hour to sample all wells in a 384-well plate. For high-density assay plate formats with large numbers of wells, this can cause gradual incremental influences in intensity and cell viability. Therefore, when aligning wells along the order that the flow cytometer sampled the wells, we found a linear trend in MFIs. We benefited from regression analysis to remove the effect of signal shifts.During the analyses, the compaRe toolkit made it easy to explore and compare highly complex datasets in a substantially reduced timeline. It is equipped with multithreading and can run through command-line interface on a computer server or GUI on a desktop. The GUI provides the investigator with numerous interactive visualization tools including cell staining, graphical representation, and gating. In sum, it provides a total package for fast, accurate, and readily interpretable multiparameter screening data analysis.44185. There are 378 FCS files in this experiment with one FCS file for each of 21 patients for each of 17 conditions . All FCS files from a single patient had been pooled then clustered with the PhenoGraph algorithm. Each file includes a column named PhenoGraph that specifies the PhenoGraph cluster to which each event was assigned as an integer. A value of 0 indicates no cluster was assigned because the cells were identified as outliers during some stage of analysis. Using the PhenoGraph column, we determined centroids of cell clusters, and used PhenoGraph to meta-cluster them as described in n to four for this assay .Mass cytometry dataset for 21 samples labeled with 16 surface markers collected from 16 pediatric AML patients obtained at diagnosis and five healthy adult donors  were dow+/cA . The cells were treated with a library of 40 chemotherapy and targeted antineoplastic agents in a seven-point concentration range  for 72 hn to five for this assay.Clinical flow cytometry data using a slightly modified AML panel as described by the Euroflow Consortium  from 25 Multiwell plate heatmaps of medians come in handy in QC to reveal issues such as signal and cell viability drifts occurring during screening. However, as a typical heatmap has an equally spaced color palette, small but significant differences between wells may be obscured and not visible. Therefore, we normalized the color palette by the distribution of the medians. Also, before clustering, we removed outliers in the negative controls that were different from the others in terms of similarity values measured by compaRe.Depending on the protocol by which wells are processed, time may become a major concern so that some specific wells may have lower or higher values than expected. To correct for these sources of bias, we employed a two-step correction: intra-plate shift  correction and inter-plate shift (batch effect) correction. For a given plate, we first fit a linear regression model and then vertically translate points  with respect to the learned line as it rotates to the slope zero. After correcting for the intra-plate bias, the inter-plate bias is corrected by aligning medians of the plates, that is, translating to a common baseline.n subsets for each dataset individually so that a dataset with d dimensions (markers) will be gridded into at most dn spatial units called hypercubes. compaRe grids only the part of the space encompassing data points, avoiding empty regions. It then measures the proportion of data points for either dataset within each of the corresponding hypercubes. The difference between the two proportions is indicative of the similarity within that relative spatial position represented by each hypercube. The similarity in the exclusive hypercubes is considered 0. We employed local outlier factor . This way, two technical replicates analyzed by two different instruments or configurations suffering from signal shift will still have the highest similarity. To generate a similarity matrix of multiple input samples, compaRe runs in parallel. The similarity matrix could then be used for identifying clusters of samples such as drugs with similar dose responses.To cluster samples, we developed a graphical clustering algorithm in which initially all nodes (samples) are connected forming a weighted complete graph wherein edges represent similarity between nodes. This graph is then pruned to remove potential false positive edges for a given cutoff inferred from negative controls. The optimal cutoff turns out to be the minimum weight in the maximum spanning tree of negative control nodes. After pruning, some samples may end up being connected to the negative controls  and some disconnected (active agents). After constructing this graph, clustering is tantamount to finding maximal cliques among potent agents. In addition to maximal cliques, it also reports communities (a clique is a subset of a community). Communities can be seen as loose clusters. In a community, unlike a clique, similarity is not necessarily transitive meaning that if A is similar to B and B is similar to C, A is not necessarily similar to C. If these were three drugs within a community, concluding they had an equal response was not necessarily right unless they would form a clique.compaRe visualizes the similarity of samples in the form of a dispersion graph by constructing their maximum spanning tree  feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.Our editorial process produces two outputs: i) Decision letter after peer review:eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Aleksandra Walczak as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Aik Choon Tan (Reviewer #2).Thank you for submitting your article "Comprehensive and unbiased multiparameter high-throughput screening by compaRe finds effective and subtle drug responses in AML models" for consideration by The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.Essential revisions:Please address all the reviewers comments. The reviewers agree it is a strong paper that is well written but certain improvements can still be made. Especially please give the reader more context in the introduction and compare your method with other existing methods.Reviewer #1:Comprehensive and unbiased multiparameter high-throughput screening by compaRe finds effective and subtle drug responses in AML models by Hajkariim et al. introduces a pipeline for pre-processing and analyzing data from multiplex flow cytometry and other technologies. Preprocessing steps include algorithms for correcting common sources of bias in such data. Another key feature is a robust approach to measuring cell similarity across samples. Among the strengths are that the manuscript is well-written, the analysis pipeline is well-motivated, and illustrated with apt examples. The similarity measure is very interesting as well.There are a few weaknesses as well. It is not completely clear to me how this pipeline agrees and disagrees with common practice in the field. References 1-3, cited to document ongoing analytic challenges, are all at least 5 years old. Comparisons to other approaches, including the use Jensen-Shannon Divergence for similarity, make a convincing case that the proposed method is both effective and computationally efficient, but it is not clear if the comparators represent true standard of practice, or mere straw men. Methodologies are complex and can be difficult to follow, especially the similarity measure.I would like to see the current state of the field described more clearly in the introduction, as context for the current effort. What makes this unique and important today? This type of clustering is common in single cell sequencing analysis, is it also commonly done in multiplex flow etc, maybe with less computationally demanding tools than JSD? If not, why not?I think I understand how the similarity measure here works, though its hard to follow the details. Its very interesting, but to be honest, I can't decide if I think its a good solution or needlessly complex. The key, as I understand it, is the binning into "relative" expression groups. This is, after all, how 2-dimensional flow data is commonly interpreted – with plots treated as a visual 2 X 2 table, with row and column boundaries existing largely in the eye of the beholder. The methods need to be clearer throughout, esp. this part. It might help to add another author, a third party who has to understand the method from scratch, and who, having not lived with the details as long, might be expected to provide a more user friendly sense of the overall approach.Reviewer #2:In this manuscript, Hajkarim et al. developed compaRe, a user friendly software suite (written in R) for analyzing high-throughput, multi-parameter screening data. There are several modules included in the compaRe toolkit, which can be individually invoked to perform specific tasks, such as quality control, bias correction, pairwise comparisons, clustering and data visualization. All of these modules are available as command-line version and a GUI version for users to use in data analysis, visualization and results interpretation. The authors showed the utility of their toolkit in analyzing multiparameter mass and flow cytometric data from AML and MDS patient samples. Through this analysis using compaRe, the authors showed that they can identify patient heterogeneity and drug response profiles. Overall, this is a well organized and written manuscript describing the development of the new compaRe toolkit. The method is clearly described, and the user manual/tutorial is easy to follow. It seems like compaRe will be a useful toolkit for the research community, which is eager for a one-stop pipeline for analyzing high-throughout multiparameter screening data.et al.Strengths:1. All of these modules are available as command-line version and a GUI version for users to use in data analysis, visualization and results interpretation.2. The authors showed the utility of their toolkit in analyzing multiparameter mass and flow cytometric data from AML and MDS patient samples. Through this analysis using compaRe, the authors showed that they can identify patient heterogeneity and drug response profiles.3. Overall, this is a well organized and written manuscript describing the development of the new compaRe toolkit. The method is clearly described, and the user manual/tutorial is easy to follow.4. It seems like compaRe will be a useful toolkit for the research community, which is eager for a one-stop pipeline for analyzing high-throughout multiparameter screening data.Weaknesses:1. The current manuscript didn't compare with some other existing programs/software in analyzing flow and mass cytometry data. It will be important to compare compaRe with existing tools, to show the strengths and weaknesses of compaRe with other tools.2. The authors could think about adding an additional module to integrate other "omics" data , this could be useful for doing the clustering step or to identify patients having the same mutational profiles.Reviewer #3:Hajkarim et al. implement an algorithm in their presented toolkit compaRe to compare samples based on the similarities of samples, distinct from the more commonly used meta-clustering approaches, such as PhenoGraph, or dimensional reduction with Jenssen-Shannon Divergence analysis. Similarities among samples are calculated based on the proportions of cells within a sample belonging to an n-dimensional "hypercubes"  that are stratified by expression levels for n number of markers. The authors demonstrate that this method is much more time-efficient, obviates subsampling, and is robust to batch effects. This method is particularly appropriate for large-scale datasets, facilitating the comparison of numerous samples which would be helpful in screening efforts. The manuscript is written and presented well.Major strengths:1. The study demonstrates sufficiently strong support for the toolkit's ability to determine similarity across samples and its computing efficiency with Figure 2, an important advantage of this tool.2. Compared to other approaches, the method is advantageous for identifying groups of samples that may be similar in a very large-scale dataset. CompaRe does not require (or make use of) manual expert annotation of meta-clusters. The workflow is efficient and unbiased.Major weakness:A major weakness of the current presentation of the study is that it has not clearly demonstrated the toolkit's utility in exploring specific phenotypes in-depth within a high-parameter dataset. The following are two examples in which this limitation is relevant, and the authors may address this to strengthen this paper, if in fact detailed phenotyping is considered by the authors as an important feature of the toolkit. If not, the authors should revise the manuscript as such.First, the authors stated that their approach can be used to "optimize true cytometric n-dimensional immunophenotypic characterization" even in the setting of multi-panel workflow. However, the demonstration was based on samples that seem to have predominant phenotypes that are almost mutually exclusive. It is unlikely that this toolkit would be useful for reliable phenotypic characterization in a largely heterogeneous population of cell types, e.g. even in normal peripheral blood, unless a high number of parameters was concurrently acquired within the dataset. This is an inherent limitation. Nonetheless, a revision to demonstrate how compaRe can evaluate specific clusters of phenotypes with biological significance from a high-parameter dataset (20-30 marker cytometry) would be very helpful.Second, the authors refer to the method's ability to include all rare cell subsets in the analysis, i.e. the ability to forego any subsampling. The work does not, however, demonstrate clearly how the presence of a rare cell subset in a given sample influences its similarity to other samples. Thus, the toolkit's value of being able to include such a rare subset in the analysis remains unestablished. It would be beneficial to include such a test to see whether the algorithm is sensitive to such changes. Essential revisions:Please address all the reviewers’ comments. The reviewers agree it is a strong paper that is well written but certain improvements can still be made. Especially please give the reader more context in the introduction and compare your method with other existing methods.Reviewer #1:[…] There are a few weaknesses as well. It is not completely clear to me how this pipeline agrees and disagrees with common practice in the field. References 1-3, cited to document ongoing analytic challenges, are all at least 5 years old. Comparisons to other approaches, including the use Jensen-Shannon Divergence for similarity, make a convincing case that the proposed method is both effective and computationally efficient, but it is not clear if the comparators represent true standard of practice, or mere straw men. Methodologies are complex and can be difficult to follow, especially the similarity measure.I would like to see the current state of the field described more clearly in the introduction, as context for the current effort. What makes this unique and important today? This type of clustering is common in single cell sequencing analysis, is it also commonly done in multiplex flow etc, maybe with less computationally demanding tools than JSD? If not, why not?Thank you for your comments. We believe compaRe is a unique tool in that it is designed specifically to cluster samples. Other methods that are commonly used for the analysis of mass/flow cytometry data are primarily designed to cluster cells within samples. The obtained cell clustering information is subsequently used for assessing similarity between samples by meta-clustering. Also, using clustering-free algorithms is often computationally expensive and not as robust as one may wish. For example, JSD was employed to compare samples, but it is not efficient for comparing a large number of samples, and, as we showed in the results, is computationally expensive.We had previously highlighted these limitations by comparing the outcome of a compaRe analysis to meta-clustering and JSD. We have now provided the comparisons with other well-established and commonly used cell clustering approaches for mass/flow cytometry: FlowSOM and SPADE (main text lines 113-115 and Appendix 1-figure 31). To make these points clearer, we have modified the Introduction. Specifically, we added that:i. Main text lines 33-35: ‘These technologies can test hundreds of samples (such as drug treatments) each with tens of thousands of events  labeled for numerous biomarkers (such as cytoplasmic or membrane markers).’ii. Main text lines 43-48: ‘Meta-clustering with single-cell clustering algorithms has been suggested to cluster samples based on the similarity of the centroids of cell subpopulations identified in the individual samples (5-8). While these algorithms are widely used in single-cell data analysis for clustering cells, they are not efficient for clustering of samples. This is because centroid-based analysis can be misleading when subclusters are not sufficiently distinct or the number of sub-clusters varies.’iii. Main text 56-58: ‘Available computational toolkits mostly allow for single-parameter or unautomated analyses of large-scale screening data using the aforementioned methods. In these toolkits, each well should be first represented by a single parameter such as cell counts or centroids or they require manual intervention.’I think I understand how the similarity measure here works, though it’s hard to follow the details. It’s very interesting, but to be honest, I can't decide if I think it’s a good solution or needlessly complex. The key, as I understand it, is the binning into "relative" expression groups. This is, after all, how 2-dimensional flow data is commonly interpreted – with plots treated as a visual 2 X 2 table, with row and column boundaries existing largely in the eye of the beholder. The methods need to be clearer throughout, esp. this part. It might help to add another author, a third party who has to understand the method from scratch, and who, having not lived with the details as long, might be expected to provide a more user friendly sense of the overall approach.Dynamic programming is key for reducing processing power. We try to explain the concept more clearly as follows . As you pointed out, the goal is to bin/grid data into relative expression groups (hypercubes). Gridding can be implemented by a simple algorithm dividing each dimension in each iteration. However, after a couple of rounds, this naïve algorithm turns out to be infeasible. This can be shown through visualizing the gridding algorithm by a tree graph (Appendix 1-figure 29). The tree clearly shows that the comparison complexity of the gridding algorithm is exponential since the tree grows exponentially. Therefore, we were forced to use a more efficient algorithm for implementing gridding. Dynamic programming  turned out to be quite effective. What makes dynamic programming very effective is its ability to memorize the values computed in the previous iterations avoiding recomputing potentially expensive algebraic operations (Appendix 1-equation 3). Moreover, considering that any signal drift correction is essentially an approximate method, the robustness of this algorithm to noise creates a synergistic effect. Therefore, we believe that the current description about similarity measure is required.Reviewer #2:[…] 1. The current manuscript didn't compare with some other existing programs/software in analyzing flow and mass cytometry data. It will be important to compare compaRe with existing tools, to show the strengths and weaknesses of compaRe with other tools.Thank you for your comments. We previously benchmarked compaRe’s similarity module against meta-clustering with several cell clustering methods and selected widely used PhenoGraph which performed better than others . We also used JSD as a clustering-free approach. In response to the request from the reviewer, we have now added the comparisons with SPADE and FlowSOM, two other commonly used algorithms for the analysis of flow and mass cytometry data (main text lines 113-115 and Appendix 1-figure 31). However, please note that available automated and commonly used tools are primarily cell clustering/automatic gating approaches. These include methods such as PhenoGraph, SPADE, and FlowSOM. A key reason why compaRe outperforms these and other methods is that they are designed to cluster cells rather than samples, while compaRe clusters samples.2. The authors could think about adding an additional module to integrate other "omics" data , this could be useful for doing the clustering step or to identify patients having the same mutational profiles.Drug response in association with genetic alterations is one of the applications of compaRe. The genetic alteration can be visualized in the clusters that compaRe identifies. However, at the moment, we do not have relevant data to apply this type of analysis for visualization. To highlight that the functionality for this type of analyses exists within compRe, we have now explained this in the main text lines 212-213.Reviewer #3:[…] A major weakness of the current presentation of the study is that it has not clearly demonstrated the toolkit's utility in exploring specific phenotypes in-depth within a high-parameter dataset. The following are two examples in which this limitation is relevant, and the authors may address this to strengthen this paper, if in fact detailed phenotyping is considered by the authors as an important feature of the toolkit. If not, the authors should revise the manuscript as such.First, the authors stated that their approach can be used to "optimize true cytometric n-dimensional immunophenotypic characterization" even in the setting of multi-panel worklow. However, the demonstration was based on samples that seem to have predominant phenotypes that are almost mutually exclusive. It is unlikely that this toolkit would be useful for reliable phenotypic characterization in a largely heterogeneous population of cell types, e.g. even in normal peripheral blood, unless a high number of parameters was concurrently acquired within the dataset. This is an inherent limitation. Nonetheless, a revision to demonstrate how compaRe can evaluate specific clusters of phenotypes with biological significance from a high-parameter dataset (20-30 marker cytometry) would be very helpful.Thank you for your comments. To address this comment, we generated Appendix 1-figure 32 and added the result to the main text lines 124-131. As compaRe measures the similarity between cluster morphologies, we carried out an analysis in which cells from a cluster (an immunophenotypic cell population) were gradually removed to contort its configuration. We used a dataset of 3 healthy and 2 pediatric AML bone marrow mononuclear cell samples from the data provided in the 6th reference. Samples were stained with 29  markers. Taking H1 as reference, we gradually removed 25%, 50%, 75% and 100% (phenotypic changes) of cells from a target cluster identified by PhenoGraph. As the UMAP projections show, the similarity decreased concurrently and more drastically when phenotypic changes were detected.Second, the authors refer to the method's ability to include all rare cell subsets in the analysis, i.e. the ability to forego any subsampling. The work does not, however, demonstrate clearly how the presence of a rare cell subset in a given sample influences its similarity to other samples. Thus, the toolkit's value of being able to include such a rare subset in the analysis remains unestablished. It would be beneficial to include such a test to see whether the algorithm is sensitive to such changes.We previously discussed in the manuscript that ‘the poor performance of JSD indicates that this approach can work well only in the absence of signal shift. It is of particular note that compaRe does not need subsampling or dimension reduction of the input data, avoiding the risk of losing rare subpopulations.’ The use of full data rather than subsampled data allows for more robust analyses, in part, simply, owing to using more data and the ability to establish a more refined overall cell cluster morphology of each sample.Additionally, in the analysis explained above (Appendix 1-figure 32), the 75% sample potentially resembles a rare cell population, a cell cluster with a small number of cells. The analysis demonstrated compaRe was sensitive enough to maintain a certain similarity in the presence of a rare cell population, as the similarity score was reduced when this population was gone in the 100% sample. Nonetheless, since subsampling does not necessarily lose rare subpopulations and to avoid confusion, we removed the phrase ‘avoiding the risk of losing rare subpopulations’ from the discussion.


The use of high-throughput omics technologies is becoming increasingly popular in all facets of biomedical science. The mRNA sequencing (RNA-seq) method reports quantitative measures of more than tens of thousands of biological features. It provides a more comprehensive molecular perspective of studied cancer mechanisms compared to traditional approaches. Graph-based learning models have been proposed to learn important hidden representations from gene expression data and network structure to improve cancer outcome prediction, patient stratification, and cell clustering. However, these graph-based methods cannot rank the importance of the different neighbors for a particular sample in the downstream cancer subtype analyses. In this study, we introduce omicsGAT, a graph attention network (GAT) model to integrate graph-based learning with an attention mechanism for RNA-seq data analysis. The multi-head attention mechanism in omicsGAT can more effectively secure information of a particular sample by assigning different attention coefficients to its neighbors. Comprehensive experiments on The Cancer Genome Atlas (TCGA) breast cancer and bladder cancer bulk RNA-seq data and two single-cell RNA-seq datasets validate that (1) the proposed model can effectively integrate neighborhood information of a sample and learn an embedding vector to improve disease phenotype prediction, cancer patient stratification, and cell clustering of the sample and (2) the attention matrix generated from the multi-head attention coefficients provides more useful information compared to the sample correlation-based adjacency matrix. From the results, we can conclude that some neighbors play a more important role than others in cancer subtype analyses of a particular sample based on the attention coefficient. Cancer is a complex and heterogeneous disease with hundreds of types and subtypes spanning across different organs and tissues, originating in various cell types ,2. For eThe high-throughput RNA-seq datasets show quantitative measures of more than tens of thousands of mRNA isoforms for a cohort of hundreds or thousands of samples . However, due to the unavoidable sample heterogeneity or experimental noise in the data, extracting valuable biological information and discovering the underlying patterns from the data is becoming a serious challenge to computational biologists . While hInspired by the GAT model for capturing node dependencies in a wide range of domains, we propose the omicsGAT model and apply it to cancer samples with RNA-seq data. First, we test omicsGAT on The Cancer Genome Atlas (TCGA) breast invasive carcinoma (BRCA) data collections  and urotWe carry out experiments on TCGA RNA-seq datasets and scRNA-seq datasets to evaluate the performance of omicsGAT in this section. In the first part, we perform experiments with omicsGAT for cancer outcome prediction on TCGA breast cancer dataset and cancer patient stratification on TCGA bladder cancer dataset . In the The proposed framework, omicsGAT, is tested on TCGA breast invasive carcinoma (BRCA)  and urotioPortal . The BRCWe design three tasks on TCGA BRCA mRNA expression data to evaluate the performance of the omicsGAT Classifier  on cancep-values clearly demonstrate a strong additional prediction power of the learned embeddings beyond the gene expression.To evaluate the performance of omicsGAT in greater depth, the patient’s overall survival time and disease-free time are predicted on the breast cancer dataset. The Cox proportional hazards model with elastic net penalty  evaluatesklearn.decomposition.PCA package. The top 400 PCA components are then used as input in the omicsGAT pipeline, and the generated embeddings are fed to the second step of omicsGAT Clustering, a hierarchical clustering model which would cluster the samples into different groups based on their embeddings.To evaluate the generalization of our embedding mechanism, we employ omicsGAT Clustering  to stratBefore clustering the samples into different groups, we first cluster the embeddings generated by omicsGAT, as illustrated in Next, we compare the performance of omicsGAT Clustering with the five baselines  for clusseaborn [To visualize the clustering performance, tSNE plots (Python seaborn  package)Single-cell RNA-seq (scRNA-seq) data reveals heterogeneity at the cell level and offers a more significant number of samples  compared to bulk RNA-seq data . We apply omicsGAT Clustering on scRNA-seq data and cluster cells to evaluate the generalization of our proposed model.scRNA-seq data from six primary H3K27M-gliomas (H3 lysine27-to-methionine mutations) are used in the following experiment. This type of gliomas  primarily arises in the midline of the central nervous system of young children . Early dThe same omicsGAT Clustering method  is folloomicsGAT can successfully integrate the structural information within gene expression data into sample embeddings, enabling better classification and clustering performance compared to the original dataset. The self-attention mechanism in omicsGAT contributes to the stronger predictive ability of the embeddings. A binary adjacency matrix is applied to define neighborhoods in omicsGAT that includes self-connections to ensure that the information of a sample itself is also considered in the embedding. The performance is reduced when we run the same tasks with just the adjacency matrix. The adjacency matrix is calculated using correlation only, which keeps track of the pairwise linear relations between samples. However, using the attention mechanism, omicsGAT can capture complex nonlinear relations by accounting for the importance of neighboring samples on the classification or clustering of a target sample. The captured relations among samples are represented in the generated embeddings, enabling the model to perform better on classification and clustering tasks.h heads assigns different attention coefficients to its neighbors, and only the highest among the h attention coefficients is considered for each neighbor to represent its relation with the target sample. The same procedure is repeated to generate the full attention matrix. This process is applied to build the attention matrix for both BLCA and cell clustering tasks described in xavier normal library function of Pytorch [In order to verify the effect of the multi-head attention mechanism, a  Ben-Hur . For a t Pytorch .For the clustering tasks, the NMI and ARI scores of the baselines are relatively low with hierarchical clustering, which can be observed in To evaluate the clustering performance of omicsGAT in greater depth, it is also applied on the latest SARS-CoV-2 antibodies scRNA-seq data , consistIn this section, we first introduce our proposed framework, omicsGAT, which generates embeddings from gene expression data to be used in downstream classification and clustering. We extend the GAT model  to betteThe omicsGAT model architecture builds on the concept of the self-attention mechanism. In omicsGAT, embedding is generated from the gene expression data, assuming that the samples  with similar features (gene expressions) are expected to have similar disease outcomes or cell types, and therefore, are connected to each other. Hence, network information is injected into the model using the adjacency matrix to consider these connections. However, all connected neighbors of a target sample should not get equal attention in generating the embedding for that sample. A particular neighbor of a target sample can contribute more to its subsequent prediction or clustering, which similarity metrics cannot accurately apprehend. Therefore, to capture the importance of each neighbor on a sample, the omicsGAT model automatically assigns different attentions to the neighbors of that sample for a singular head while generating the embedding. Moreover, to consider the impact of different types of information secured from the neighbors and stabilize the learning process, the above procedure is repeated multiple times in parallel employing several heads (independent attention mechanisms) in a multi-head framework.n be the number of samples  and m be the number of features  representing each sample. The input feature matrix is given by p is the embedding size, and LeakyReLU activation function. Then, the structural information of the network is introduced by masking the attention values using the adjacency matrix. Only the attention values where a connection is present between the nodes (samples) in the adjacency matrix Softmax function, which follows the equation below:The mathematical notations used to explain omicsGAT are summarized in The attention coefficients calculated for all of the neighbors of Equation  are leveh is the number of heads. The output projected in the embedding space is represented by In a multi-head attention network, each head has a separate attention mechanism with its own weight matrix i et al.  in self-Softmax function. Let the number of classes for the classification task be c. The FC layer transforms omicsGAT Classifier is a unified model that passes the embedding n samples be Let the ground truth labels for ReLU, and no activation function is used in the final layer.For clustering, we propose a two-step omicsGAT Clustering framework. The first step is an autoencoder that generates the gene expression embedding in an unsupervised approach, and the second step is a hierarchical clustering model. omicsGAT described in scikit-learn package [The output, projected back to the input feature space by the decoder, is given by  package . It straScikit-learn [PyTorch [PyTorch Geometric [Support Vector Machine (SVM), Random Forest (RF), Deep Neural Network (DNN), Graph Convolutional Network (GCN) , and Grait-learn , PyTorch[PyTorch , and PyTeometric .ReLU activation function. For better evaluation of our model by comparing it to similar graph-based deep learning models, we follow the GCN proposed by Kipf and Welling [SVM and RF are two of the most widely used machine learning models. In this study, ‘rbf’ kernel is applied for SVM. Hyperparameters for RF, including the number of trees, split criterion, maximum depth of the tree, and maximum number of features considered for split, are also tuned. The Deep Neural Network model consists of three fully connected linear layers, with the first two of them followed by the  Welling  and Grap Welling . Both moThe performance of embeddings generated by omicsGAT for the downstream clustering task is evaluated against embeddings generated by a DNN-based autoencoder and a GCN-based autoencoder. The encoder part in the autoencoders consists of the respective model, and the decoder part comprises three FC layers. Hierarchical or k-means clustering is employed on the embeddings generated by the trained encoders of the baselines. Furthermore, to get a better understanding of the improvements made by omicsGAT, clustering of the raw features (gene expression), their PCA components, and adjacency matrix are compared with omicsGAT-generated embeddings. In addition to the aforementioned models, an efficient clustering method targeted for scRNA-seq data named SC3s  is used scikit-learn library of Python. The Area Under the Receiver Operating Characteristic Curve (AUROC) is used for the comparison of the classification models. It is defined as the area under the curve plotted using True Positive Rate (precision) along the y-axis and False Positive Rate (1-specificity) along the x-axis. The Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI) are applied to evaluate the clustering methods, both ranging from 0 to 1, where 1 means perfect clustering and 0 means totally random.In this section, we define three evaluation metrics used in this study implemented using the Powered by high-throughput genomic technologies, the RNA-seq method is capable of measuring transcriptome-wide mRNA expressions and molecular activities in cancer cells. Hundreds of computational methods have been developed for cancer outcome prediction, patient stratification, and cancer cell clustering. Some of these methods consider sample-sample similarities in the analysis, and some of them do not. These sample similarity-based methods cannot distinguish the importance of the neighbors for a particular sample in the downstream prediction or clustering tasks. Therefore, we introduce omicsGAT in this study, which leverages a self-attention mechanism consisting of multiple heads to assign proper attention weights to the neighbors of a sample in the network. Experiments on cancer subtype analyses show the superior performance of the model in every aspect compared to the baseline methods. We also show the generalization of omicsGAT’s performance on both bulk RNA-seq and scRNA-seq data. As a future objective, we would like to extend omicsGAT to include metapath selection, which would consider the best paths in a network to perform a certain task on a particular sample.


Human microbiome research is moving from characterization and association studies to translational applications in medical research, clinical diagnostics, and others. One of these applications is the prediction of human traits, where machine learning (ML) methods are often employed, but face practical challenges. Class imbalance in available microbiome data is one of the major problems, which, if unaccounted for, leads to spurious prediction accuracies and limits the classifier's generalization. Here, we investigated the predictability of smoking habits from class-imbalanced saliva microbiome data by combining data augmentation techniques to account for class imbalance with ML methods for prediction. We collected publicly available saliva 16S rRNA gene sequencing data and smoking habit metadata demonstrating a serious class imbalance problem, i.e., 175 current vs. 1,070 non-current smokers. Three data augmentation techniques  were applied together with seven ML methods: logistic regression, k-nearest neighbors, support vector machine with linear and radial kernels, decision trees, random forest, and extreme gradient boosting. K-fold nested cross-validation was used with the different augmented data types and baseline non-augmented data to validate the prediction outcome. Combining data augmentation with ML generally outperformed baseline methods in our dataset. The final prediction model combined tree-based associative data augmentation and support vector machine with linear kernel, and achieved a classification performance expressed as Matthews correlation coefficient of 0.36 and AUC of 0.81. Our method successfully addresses the problem of class imbalance in microbiome data for reliable prediction of smoking habits. In recent years, human microbiome research has elucidated the importance of microbes in the host's wellbeing and their interplay with different phenotypes   optimization and validation of different data augmentation techniques and ML methods using nested cross-validation, (ii) identifying the best-performing approach for predicting smoking habits by taking class imbalance in the underlying microbiome data into account, and (iii) applying the best-performing approach for prediction modeling of human smoking habits from saliva microbiome data despite the underlying class-imbalanced data. The data and the code used are made publicly available.N = 543)   . The first study  (v.15.2) . Moreover, microbiome datasets are normally sparse and characterized by a zero-inflated distribution, where most taxa are not shared among the majority of the samples. This is magnified in cross-study applications with study-specific taxa, which can limit the generalizability of the applications. Based on this finding, we merged the two ASV tables from the two analyzed studies and filtered out study-specific taxa, keeping 124 species from 30 families that were common between the two datasets for downstream analyses .The data from the two selected studies were processed separately. Primer sequences were obtained from the original studies and were removed from the raw sequencing reads using cutadapt (v.1.15) Martin,  by settianalyses   were obtained with default 999 permutations.The overall differences in the saliva microbial communities between the smoking classes were calculated in QIIME 2 v.2019.10)  and the majority class  , we aimed to employ a binary machine learning (ML) classifier. For that, data imbalance was a marked issue in our dataset with a ratio of about 1:6 between the minority class (smokers) . The prosmokers)  and the smokers) , as wellsmokers) .k of its nearest neighbors are found. A line is drawn between the original sample and one randomly chosen neighbor, where a new synthetic sample is generated at a random point along the line (linear combination of samples). The process is repeated generating the same number of synthetic samples for each original minority sample until a specific ratio between the minority and majority classes is reached or to equal the majority class. On the other hand, ADASYN adds random small values to the neighbor samples; hence, they are not linearly correlated to the original sample. By this, ADASYN considers a density distribution between the original sample and its neighborhood, which acts as the criterion to set the number of synthetic samples to be generated from each original sample. On another point, with the under-sampling approach, random majority class samples are dropped out until a specific ratio between the classes is reached. Both SMOTE and ADASYN techniques were implemented using the imbalanced-learn Python toolbox (v.0.6.1) , under-sampling (decrease the majority class), or a combination of the two. Particularly, SMOTE and ADASYN techniques differ in the generation of synthetic samples in the minority class (over-sampling). For that, SMOTE over-sampling pinpoints the samples belonging to the minority class in a Euclidean space, and a random sample is first chosen for which Cmin is the number of the minority class samples, Cmax is the number of the majority class samples, and t is the absolute value of the difference between Cmin and Cmax.where We also used the over-sampling approach alone, indicated as SMOTE-2 and ADASYN-2 in this study, by which the number of the samples in the minority class was equaled to the majority class.ConsensusSequence function in DECIPHER (v.2.12.0) , otherwise it is 1 (non-smoker). KNN is a non-parametric model and as such supports non-linear solutions. It finds the Euclidean distances between a query sample and a k number of its closest samples (nearest neighbors) in the feature space and identifies their most frequent class label. SVM models take the data points and find a separating hyperplane between the two classes. SVML is a linear method that looks for linear dependencies among the input features to separate classes. SVMR is a non-linear method that adds an extra dimension to the data (kernel), so they become linearly separable and then project back the decision boundary to the original dimension using the dot product of two vectors in the feature space known as the kernel function. DT is a tree-based ML algorithm that mimics a decision diagram. Each input feature constitutes a node in the tree, where based upon a certain condition or rule splits into sub-nodes and extends until the leaf node that represents the classification decision (0 for smoker or 1 for non-smoker). Finally, RF and XGBoost are tree-based ensemble models that combine several models to improve their outcome predictions. RF generates a large number of decision trees on different subsamples and combines their outputs using averages at the end of the learning process. On the contrary, XGBoost combines the decision trees during the learning process for which it uses a gradient descent algorithm. By this, the mistakes done in a previous model are learned and improved in the subsequent model until no further improvement can be achieved. Hyperparameter optimization for all the ML models was performed using nested k-fold cross-validation  included in the scikit-learn (v.0.23.2) Python package   for model hyperparameter tuning (k test fold (k × n) nCV where each of the first splits is named outer-fold (k) and each of the inside splits for hyperparameter tuning as inner-fold (n). Hyperparameter optimization for the seven ML models was performed using the RandomizedSearchCV function in scikit-learn.Nested cross-validation (nCV) is a resampling procedure that enables both model optimization and evaluation  , each k r tuning . The optest fold . We emplSince most ML algorithms operate trying to maximize the classification accuracy, spurious high classification occurs in imbalanced datasets by correctly classifying all or almost all the samples from the majority class at the cost of misclassifying many samples from the minority class. Hence, performance metrics, such as accuracy, or F1 score alone can lead to misleading results in imbalanced datasets  and the ML method (m = 7), we used the MCC metric. However, we also reported the AUC metric to show potential misleading results in those models with baseline non-augmented data . Comparisons among the different data types with a given ML method were performed in the R environment (v.3.6.1) using the Kruskal–Wallis and Wilcoxon tests. Significant p-values were determined with a cut-off value of 0.05 following Benjamini-Hochberg (BH) correction.For a high MCC score, the classifier has to correctly predict a high percentage of the samples in both the majority and the minority classes, independent of their ratios in the overall dataset, and hence is independent of data imbalance. MCC gives a score ranging from [−1 to +1], where 1 means perfect prediction, 0 random prediction, and −1 perfect inverse prediction. Based on this finding, in order to compare all the possible combinations of input data types, including original non-augmented data , d = 6 aL method , m = 7, N = 175 smokers and N = 1,070 non-smokers) was split into a training set (80%) and a holdout test set (20%) maintaining the sample ratio between the classes (d = 6), including the original non-augmented and the five augmented data types (m = 7) using an nCV approach as explained before (i = 10) (k) over 10 times (i = 10)  resulting values for each metric)  , aiming  metric) . The bes metric) , and the metric)  was vali metric) .Datasets in the METHODS section for more details). Filtering samples for quality-controlled metadata, de-noising of sequencing reads, and sequencing depth filtering resulted in a total of 1,245 samples . In the whole dataset, class imbalance in smoking habits was large with 512 (44.1%) never smokers, 558 (44.8%) former smokers, and only 175 (14.1%) current smokers. Female samples accounted for 41.5% of the total sample, and the average age (±standard deviation) was 65.2 (±11.0) years. European ancestry of the saliva sample donors was overrepresented (87.3%), as typically encountered in human microbiome data publicly available thus far. The selected characteristics of the two datasets are described in further detail in The data comprised saliva 16S rRNA gene amplicon sequencing data and associated metadata from two different studies referred to here as dataset S1  and abundance filtering resulted in 200 species from 33 families in dataset S1 and 168 species from 35 families in dataset S2. Both datasets were dominated by a few species that comprised more than 75% of the total microbial composition (21 species in S1 and 15 species in S2). These species belonged to different genera, including Streptococcus, Rothia, Haemophilus, Prevotella, Veillonella, and Actinomyces. Dataset S1 was dominated by Streptococcus oralis , followed by S. salivarius (0.09), Rothia mucilaginosa (0.06), S. parasanguinis (0.05), and Haemophilus parainfluenzae (0.05). Dataset S2 was also dominated by S. oralis (0.24), followed by S. parasanguinis (0.06), S. salivarius (0.06), Prevotella melaninogenica (0.06), and R. mucilaginosa (0.06)  (v.15.2)  . These ta (0.06) . For dowa (0.06) . These cR = 0.04, q = 0.03; PERMANOVA pseudo-F = 11.37, q = 0.002), and current and former smokers , but not between never and former smokers . Therefore, we grouped the never and former smokers into a single category of non-current smokers, which when compared with the current smokers showed statistically significant differences in the overall microbial communities . Based on these results, we used two classes of non-current and current smokers in all downstream analyses.The overall saliva microbial communities differed with statistical significance between current and never smokers (ANOSIM d = 6), including augmented data and baseline non-augmented data, and each ML model (m = 7), the resulting classifiers' performance metrics are expressed as Matthews correlation coefficient (MCC) and area under the receiver operating characteristic curve (AUC), which are summarized in A step-by-step overview of our analytical setup can be found in p = 9.93E−20)   . However= 0.882) . From thFor the training  and the In this study, coming with the available 16S rRNA gene amplicon microbiome sequencing data, we deal with the common issue of data imbalance in human microbiome binary classification, with the aim of unlocking the prediction of human host's traits from saliva microbiome. As a data source, we focused on studies targeting the saliva microbiome and did not use data from studies targeting other niches in the oral cavity due to known diverse microbial assemblies on different oral sites , but significantly differed between never and current , and between former and current smokers . One of the two studies .Our observations in the overall microbial composition of saliva were in agreement with the two original studies  using an RF approach and conventional non-nested k-fold CV and obtained an AUC of 0.75 from the test set. This prediction was improved to AUC = 0.80–0.93 when using single-nucleotide variants of single species as input data instead of relative abundances of all species. More recently, Carrieri et al.  using the XGboost method and conventional non-nested k-fold CV and reported the F1 performance metric in the CV folds (F1 = 0.72 ± 0.12), training set (F1 = 0.98), and test set (F1 = 0.85). The noted differences in the F1 scores might be an indication of introduced variation by the different data partitions and bias toward model selection, which can be overcome using an nCV approach as proposed by us and others. Notably, the methods applied in both of these previous studies did not take the class imbalance problem in the used data into account. Therefore, and because of the small sample size in one of these studies at least, the previously reported prediction accuracies are not expected to be reliable, in contrast to the results from our study.With the best data augmentation and ML approach chosen, we predicted individuals' smoking habits from saliva 16S rRNA gene microbiome data in the final holdout test set with MCC of 0.36 and AUC of 0.81. Previously, Sato et al.  predictei et al.  predicteHowever, in our dataset, we acknowledge some metadata-related characteristics that might limit the prediction of microbiome-based smoking habits, even when the data imbalance issue was accounted for by our approach. Precise phenotype descriptions were available in only one of the two studies , which iTo conclude, by testing different data augmentation techniques and ML methods on class-imbalanced microbiome data, we established a best-practice approach for reliable prediction of individuals' smoking habits from the saliva microbiome that takes the underlying data imbalance into account. We found that combining data augmentation with ML generally outperformed baseline methods in our dataset for our purpose, as other researchers have also suggested before (Knights et al., https://github.com/dmontielg/smokingmicrobiome.The original contributions presented in the study are included in the article/CD and AV conceptualized the idea. CD, DM, and AV designed the study with contributions by MK. DM implemented and performed the data analysis. CD contributed to initial data collection and curation, performed some of the statistical analyses, and prepared the display items. MK provided resources. CD, AV, and MK wrote the manuscript with contributions by DM. All authors approved the final manuscript.The work of all authors was supported by Erasmus MC University Medical Center Rotterdam. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


In this modern era of precision medicine, molecular signatures identified from advanced omics technologies hold great promise to better guide clinical decisions. However, current approaches are often location-specific due to the inherent differences between platforms and across multiple centres, thus limiting the transferability of molecular signatures. We present Cross-Platform Omics Prediction (CPOP), a penalised regression model that can use omics data to predict patient outcomes in a platform-independent manner and across time and experiments. CPOP improves on the traditional prediction framework of using gene-based features by selecting ratio-based features with similar estimated effect sizes. These components gave CPOP the ability to have a stable performance across datasets of similar biology, minimising the effect of technical noise often generated by omics platforms. We present a comprehensive evaluation using melanoma transcriptomics data to demonstrate its potential to be used as a critical part of a clinical screening framework for precision medicine. Additional assessment of generalisation was demonstrated with ovarian cancer and inflammatory bowel disease studies. These high-resolution molecular snapshots hold great promise in today’s era of precision medicine, in which an individual’s molecular measurements can be used to diagnose disease, direct care, guide therapy, and better support patient and clinician decision-making, thus improving over conventional protocols. Despite the large body of literature on published “molecular signatures”, with a few notable exceptions, many of these signatures have limited reproducibility in independent datasets and critical challenges still exist for the effective clinical deployment of omics signatures. Using melanoma cancer as an example, it is apparent that while recent progress in molecular biomarker discovery has found various signatures10. Here, we describe not only the reproducibility of discovered biomarkers but also the importance of having a model that requires no feature scale adjustment for any incoming new data. In practice, a successful risk model must overcome the challenge of transferability:11 it must be able to make predictions on future data without having to re-normalise the data nor to re-train the model. Most omics measurements exhibit high variation between datasets due to batch effects, intra-platform protocol changes, or other site-specific factors; this makes the prediction challenge much harder than when using clinical variables that have clearly defined measurement units such as age, height and weight. This extra “noise” affects the risk model parameters constructed from the training data and creates inaccurate predictions for new and independent data.Widespread adoption of predictive models could be achieved if omics signatures were measured consistently across different platforms, e.g. when measuring gene expression using anything from microarray to RNA-sequencing. We refer to the ability to extract reliable predictions across heterogeneous samples as “model transferability”, extending the notion of “platform-independent markers”, which was recently explored in selected diseases13 and a newly generated dataset (n = 46) based on an independent cohort. Furthermore, CPOP’s utility is further illustrated on two other complex diseases, ovarian cancer and inflammatory bowel disease (IBD).Here, we address the transferability challenge for practical implementation by developing a workflow for identifying features that are comparable across multiple datasets and multiple platforms, and for traditional microarray to modern sequencing data. Our Cross-Platform Omics Prediction (CPOP) procedure is a new end-to-end framework that accurately predicts clinical outcomes. We developed CPOP in the context of identifying a gene expression signature that can predict prognosis outcomes in stage III melanoma across different platforms. CPOP meets the transferability challenge through three distinct innovations. First, ratio-based features are constructed, resulting in features that are robust to inter-platform scale differences. Second, the feature selection process incorporates weights that measure the stability of features across multiple datasets. Third, selecting features that have consistent estimated effects across multiple datasets in the presence of noise. The model constructed for stage III melanoma is validated on two published melanoma studieshttp://shiny.maths.usyd.edu.au/CPOP/) has been constructed to illustrate how gene expression data can be uploaded and predictions of outcomes for stage III melanoma patients can be obtained in real-time. Together, these innovations and methodologies deliver a molecular risk prediction platform with statistical modelling improvements, thereby producing a coherent framework that can be adopted in multi-centre and prospective settings.An interactive web portal  procedure, a five-step approach to select transferable biomarkers  cohort, and six housekeeping genes. The full list of probes is presented in Supplementary Table n = 45) using the new NanoString platform and the resulting dataset is referred to as “MIA-NanoString”.Informative datasets for the CPOP procedure can either be identified in public literature or constructed from experiments. Here, we constructed a clinical-ready molecular assay using the NanoString nCounter™ platform (NanoString Technologies Inc.). This technology was chosen for its low per-assay cost and wide deployment. We constructed a gene set panel Fig.  consistir = 0.9) with those originally measured in MIA-Microarray  on the same MIA discovery cohort shows similar analytical results. We confirmed that both the gene expression  reflects the prediction outcome across platforms without renormalisation. We compare the prediction findings to an ideal situation with no scale differences, where we build a model from dataset B and apply it again to dataset B. We refer to this result as “within-data prediction outcome” (y-axis). A transferable model  as the within-data prediction outcome (y-axis). That is, on a scatter plot we expect the results from these two models to cluster around the identity line (i.e. y = x).We apply the CPOP procedure to the MIA-Microarray and MIA-NanoString data to identify a molecular signature and construct a corresponding prognostic model. We demonstrate transferable properties as shown in the schematic drawing in Fig. del Fig.  will pro15  with some features entirely not captured. Such a situation with entire missing features is unlike in classical missing value problems where the missingness occurs in the training data. Here, we examine the situation where we have missingness in the incoming sample. CPOP handles this situation by using additional regression models on the reference training data to impute the missing feature in the new/test data. We created two simulations and show our model is robust to this type of missing data  plots show significant differences in survival probability between the good and poor predicted classes from the CPOP model in TCGA  and in the Sweden data . In contrast, KM-plots that were generated for the Lasso model in Supplementary Fig. p = 0.096 for TCGA and p = 0.062 for the Sweden data). As the training data (MIA-Microarray and MIA-NanoString) and the validation data (TCGA and Sweden) were generated from distinct gene expression platforms and distinct samples, we have demonstrated that the CPOP procedure is able to select features that are stable between platforms and sample selection. We then establish the robustness of our results by repeating the cross-data validation using different training-testing pairs. Supplementary Fig. p < 0.05. Furthermore, Supplementary Fig. 17 normalisation on the log-ratio features.To further assess “model transferability” of the CPOP model on external data, we applied the trained melanoma model on the TCGA19, CCL820, TSPAN10, PLEK, CLEC10A21, SLAMF1, CLECL1, MAGEA10, CXCL13 and LYZ22, which all have been identified as potential biomarkers in previous studies and show enrichment in the inflammatory response.The CPOP procedure identified the most stable relationships between gene expression values in diverse datasets, which may enhance biological interpretation. The CPOP melanoma signature is based on relative expression levels between genes and after applying the CPOP procedure onto the MIA-Microarray and MIA-NanoString data, the final CPOP signature consisted of 24 log-ratio features derived from 40 genes as presented in Fig. n = 46, see Methods) in 2020. These samples were obtained from Melanoma Institute Australia and were accrued since 2010 at the same centre, as described in Table p = 0.037). CPOP demonstrated a consistent performance . Here, we are addressing a translational gap where renormalisation for incoming samples together with an existing cohort is not possible due to constraints with consent, data security, or other practical reasons. Therefore, data normalisation alone is not enough to address the various challenges in implementing precision medicine utilising omics data with additional considerations relating to the model transferability and reproducibility also being necessary.Importantly, the concept of transferability is not the same as data integration, data harmonisation or data normalisation. To date, the most popular method for adjusting data scales between different datasets is through the use of normalisation, often divided into two broad categories of z-score standardisation and between-data normalisation. We refer to z-score standardisation as a pre-processing step where each omics predictor is centred at zero and its sample variance is scaled to one. This procedure has been widely used, for example for investigating ovarian cancer28).It is important to highlight that while the calculation of (log)-ratios is not a new idea, the work presented in this paper challenges the practice of using only genes as features but not functions of sets of genes. Specifically, in our framework, we propose using (log-)ratios of all gene pairs as potential features when working with transcriptomics data. Traditionally, most model prediction frameworks begin with value adjustments on genes through the use of either endogenous or artificial “control genes”. However, in practice, it often means that the values of these control genes become dominant in deriving any prediction score, sometimes even more dominant than the statistical prediction model itself. This is problematic especially given the controls can be dataset-dependent. The calculation of these (log-)ratios is intended to obtain a quantitative measure for the relative differences between the gene expression values, which we have found to be better preserved across different patient cohorts and data generation platforms. As this (log-)ratio value is calculated within a sample, we are assuming that the relative interactions between the omics features are dataset-independent and cross-platform, provided that all the omics platforms in question can unbiasedly estimate the relative expression level of features. The prefix of ‘log’ simply reflects the prevalence of the log transformation in dealing with omics data. While this all-pairwise feature construction is not suitable to be applied to the whole genome (e.g. whole genome RNA-sequencing) directly due to the large dimensionality, this approach is achievable on targeted omics assays. These targeted omics assays typically provide a higher signal-to-noise ratio for candidate features that are of higher clinical relevance and are in wide use in clinical validation, translational work and in the implementation phase of precision medicine. It is critical to note that this concept of ‘relative difference’ is not limited to transcriptomics  that does not state explicitly how decisions are made is not considered as such a model would obscure clinical interpretability. We ultimately opted for a regression-based modelling approach because of its relative simplicity and prevalence in the literature. In the CPOP procedure, the difference in the scaling of features is further mitigated by using weights in the penalised regression model to preferentially select features with stable distributions and estimates across multiple datasets. While in principle it is possible to derive platform and batch agnostic features or weighted statistics for other machine learning methods as the training sample size increases, the resulting procedure will still need to be carefully adapted and tested to ensure that the increase in model complexity is also matched by the relative gain in accuracy.http://shiny.maths.usyd.edu.au/CPOP/ for prognostic prediction in stage III melanoma patients. This interactive tool illustrates how gene expression data can be uploaded and predictions to develop stage III melanoma can be obtained in real-time. For creating CPOP models for other data and diseases, an R package is made available at https://sydneybiox.github.io/CPOP/. Using the CPOP package, we demonstrate the generalisability of the procedure to two other diseases: an ovarian cancer data collection (nine datasets) and a large IBD dataset with 983 samples. In conclusion, we deliver a molecular (omics) risk prediction platform with substantial improvements in reproducibility and stability that can be adopted in multi-centre and prospective settings.In summary, in order to enable multi-centre implementation of gene expression signatures of prognosis in stage III melanoma specifically and omics signatures more widely in clinical practice, we have developed a new Cross-Platform Omics Prediction (CPOP) procedure that closes an important gap between molecular biomarker discovery and wider clinical use. The CPOP approach accounts for differences in feature scaling in omics data by performing weighted feature selection and estimation that preferentially selects stable features across multiple datasets. Our application of the CPOP method on multiple melanoma studies and validation work using the NanoString technology serve as important components towards the future implementation of this methodology into clinical practice. We construct a web portal that is available online at Tumour samples were obtained from the Melanoma Institute Australia (MIA) Biospecimen Bank, a prospective collection of fresh-frozen tumours accrued with written informed patient consent and Institutional Review Board approval  Zone) Protocol No. X08-0155/HREC 08/RPAH/262, No. X11-0023/HREC 11/RPAH/32, and No. X07-0202/HREC/07/RPAH/30) since 1996 through MIA, formerly the Sydney Melanoma Unit. Both the data, MIA-Nanostring and MIA-Validation were generated (see ‘Melanoma data collection’ section below for details) from this biospecimen bank with non-intersecting samples.14. Of the 186 probes, 46 were found to be differentially expressed genes between poor and good prognosis patients , and 180 probes were identified in a model built to predict prognosis7. Between the two studies, 39 of these probes overlapped. In instances where there was more than one probe per gene, the most informative probe based on an inclusion frequency of 30% for that particular gene was included in the NanoString panel. Six housekeeping genes were selected from a list of previously reported housekeeping genes29 and had variances in the lowest quintile (20%) of the data presented in two previous studies14, a differential expression p value greater than 0.5. These housekeeping genes were selected to cover a range of high expression , medium  and low expression (UQCR). A complete list of the customised probe set is given in Supplementary Table Gene expression profiling was carried out using the NanoString nCounter® platform . NanoString designed and manufactured customised probes corresponding to 192 probes. 186 of these were identified in our previously reported studies7 RNA purity and concentration were assessed using the Agilent TapeStation system. The nCounter® gene expression assay  was performed according to the manufacturer’s instructions using 100 ng of total RNA. For each assay, a high-density scan (encompassing 600 fields of view) was performed.Tumour RNA was extracted as previously described by Mann et al.A thermal cycler is preheated to 67 °C. Tagset and RNA samples are removed from a freezer and thawed at room temperature. The tubes are inverted to mix and then briefly spun down. A hybridisation master mix is created by adding 70 μL of hybridisation buffer and 7 μL of the Probe A working pool directly to the tube containing the Tagset. The solution is inverted several times to mix and spin down. 7 μL of the Probe B working pool is added to the master mix. 8 μL of the master mix is added to each of the 12 strip tubes then 7 μL of RNA sample is added to each tube. Strip tubes are inverted several times and flicked to increase mixing. The tubes are briefly spun and immediately placed in the preheated 67 °C thermal cycler, incubated for 16 h and then ramped down to 4 °C.30 using the NanoStringQCPro package31. For the purpose of illustrating cross-platform noise, we performed a simple log2-transformation on the raw counts. This allows us to assess if the panel can facilitate prospective experiments without model modification. Of all the samples measured in the “MIA-NanoString” data, 45 samples overlap with previous microarray studies14, named “MIA-Microarray”. This allows us to correlate genes between the two data in Supplementary Fig. NanoString data was read into R7 and Jayawardana et al.14 is updated in 2018 and we define recurrence-free survival (RFS) as the time difference between the date of the first recurrence after tissue banking and the date of tumour banked. Based on RFS, we further defined:“Good” prognosis group being RFS greater than four years and alive with no sign of recurrence.“Poor” prognosis group being RFS less than one year and died due to melanoma.Clinical follow-up time for samples presented in Mann et al.32 to compute moderated t-statistics for all 192 overlapped genes.This resulted in 19 samples in the good prognosis group and 26 samples in the poor prognosis group. Using these RFS-defined prognosis classes for these 45 samples, we use limmaClassifyR33 package to compute 100 repeats of 5-fold cross-validation with the limma32 feature selection and support vector machine classifier. This is specified via the parameters ResubstituteParams, performanceType = “balanced error”, better = “lower”) before executing the function runTests. All analyses are performed in R30 version 3.6.2.We validate the prognostic assessment for the MIA-NanoString datasets by comparing its performance accuracy for both overall survival (OS) and RFS against the same samples in the MIA-Microarray data. Performance accuracy is obtained by using the Melanoma data collection: We collected five melanoma datasets mostly consisting of late-stage samples, see Supplementary Table 23 and Yoshihara et al.24, with a few modifications (detailed in Supplementary Information). We focus on the 126-genes signature reported in Yoshihara et al.24 and select genes that are also present in all nine datasets , IBD3 (n = 295) and IBD4 (n = 385) from chemical reagent changes26.Inflammatory bowel disease data: A study of the inflammatory bowel disease (IBD) with 983 samples and 712 genes. Raw NanoString data is downloaded from Gene Expression Omnibus with accession GSE73094. The data contains three batches, IBD2 , and restrict usage of the term test set or validation set to situations with known patient outcomes, i.e. to situations where we are assessing or comparing the performance of CPOP. We use the term test sample or validation sample when the unknown subjects are to be predicted.Cross-Platform Omics Prediction (CPOP) is a procedure that enables sample prediction across gene expression datasets with different scales (e.g. different sample means). We will use the generic phrase of “scale difference” to encompass all situations where multiple gene expression data exhibit different scales in the data due to, for example, the use of different experimental instruments/platforms or drifts in measurements in a prospective setting. We use the term “biomarker” and “feature” interchangeably. We will use the term CPOP uses (log)-ratios of genes as biomarkers (features), which are more stable than using individual gene expression values (Step 2 of CPOP).34 to perform feature selection using weights proportional to the stability of features across more than one dataset. This allows the selection of common predictive markers (Step 3 of CPOP).CPOP uses the Elastic Net modelCPOP selects for features with high similarity in their between-data estimated effects (Step 4 of CPOP).A major consideration in developing CPOP is to make predictions on a single sample without normalisation or combining it with additional data. The CPOP procedure has the following three key characteristics:X, of size n × p, where n is the number of samples and p is the number of omics features (e.g. genes on a gene expression platform). We define the “log-ratio matrix” as a matrix, Z, of dimension n × q, where Z is the pairwise difference between two log-transformed columns in X. Formally, each column of Z is given by enumerating all log-ratio features xl and xm being columns of X.Suppose we have an omics data matrix, Data selection: the first step of data selection is dependent on the research questions to be addressed and one should select data with similar and appropriate clinical outcomes of interest. For example, the selected cohort can consist of independent samples at the same cancer stage. In the rest of the procedure, we assume we have two gene expression data and the CPOP model training will aim to find features consistently predictive in both data.Z1 with size n1 × q and Z2 with size n2 × q. Here, n1 and n2 are the sample sizes for the two datasets, respectively. We do not impose the restriction of paired samples across the two data; however, we assume the two data measure the same q log-ratio features, or we restrict our modelling to the common q log-ratio features between the two datasets. For both data, we also have a clinical outcome measurement, denoted as y1 and y2 associated with data 1 and 2 respectively.Log-ratio matrices construction: The associated log-ratio matrices for the two gene expression data are constructed as above and denoted as Z1 and Z2 to obtain estimated regression coefficients Z1 and Z2 respectively. The superscript denotes that this is the first step of the CPOP feature selection. In fitting the WEN models, we propose to use weights that measure the similarities between log-ratio features in the two data. In this paper, we primarily use the differences between column-means as weights: Selecting common predictive features: Fit a Weighted Elastic Net (WEN) model  for both datasets S(1) and obtain second feature selection step where we select features with coefficients that are similar between both data. Mathematically, this set can be written as Selecting features with between-data stability: Next, we fit an unweighted ridge regression model to the two matrices including only features present in S(2), which are features that are common between two datasets and with similar coefficients. We will refer to these models as Final model estimation: The final CPOP models are the unweighted ridge regression models fitted on the subset of the data including only features present in In the Main Fig. https://sydneybiox.github.io/CPOP.Additional methodological details, including statements of data collection and curation, mathematical formulations and method evaluations and discussions are available in Supplementary Materials. The CPOP package is available at Further information on research design is available in the Supplementary materials + Supplementary table 1 and 2Supplementary table 3Supplementary table 4Reporting Summary