The workflow/platform is developed upon benchmarking four kinds of causal discovery methods and is examined by analyzing multiple single-cell RNA-sequencing (scRNA-seq) datasets. Our results suggest that different situations need different methods and the constraint-based PC algorithm with kernel-based conditional independence tests work best in most situations. Related issues are discussed and tips for best practices are given. Inferred causal interactions in single cells provide valuable clues for investigating molecular interactions and gene regulations, identifying critical diagnostic and therapeutic targets, and designing experimental and clinical interventions.Correlation between objects is prone to occur coincidentally, and exploring correlation or association in most situations does not answer scientific questions rich in causality. Causal discovery  infers causal interactions between objects from observational data. Reported causal discovery methods and single-cell datasets make applying causal discovery to single cells a promising direction. However, evaluating and choosing causal discovery methods and developing and performing proper workflow remain challenges. We report the workflow and platform CausalCell ( RNA-sequencing (RNA-seq) has been used to detect gene expression in a lump of cells for years. Many statistical methods have been developed to explore correlation/association between transcripts in RNA-seq data, including the ‘weighted gene co-expression network analysis’ that infers networks of correlated genes . Since astatistics alone cannot tell which is the cause and which is the effect’ (Single-cell RNA-sequencing (scRNA-seq) has been widely used to detect gene expression in single cells, providing large samples for analyzing cell-specific gene expression and regulation. On statistical data analysis, it is argued that ‘ effect’ . Corresp effect’ , and the effect’ , from th effect’ , ‘causal effect’ , ‘networ effect’ , and ‘ge effect’ .Tools and Resources article addresses the above issues by benchmarking multiple causal discovery methods and CI tests, applying causal discovery to multiple scRNA-seq datasets, developing a causal discovery workflow/platform , and summarizing tips for best practices. Specifically, the workflow combines feature selection and causal discovery. The benchmarking includes 11 causal discovery methods, 10 CI tests, and 9 feature selection algorithms. In addition, measures for estimating and ensuring the reliability of causal discovery are developed. Our results indicate that when relationships between variables are free of missing variables and missing values, continuous optimization-based methods perform well. Otherwise, the PC algorithm with kernel-based CI tests can better tolerate incomplete models and missing values. Inferred relationships between gene products help researchers draw causal hypotheses and design experimental studies. The remaining sections describe the workflow/platform and data analysis examples, discuss specific issues, and present tips for best practices. The details of methods and algorithms, benchmarking results, and data analysis results are described in appendix files.Kernel-based CI tests are highly time-consuming and thus infeasible for transcriptome-wide causal discovery. Recently other causal discovery methods are reported, especially continuous optimization-based methods . Thus, iCausal discovery cannot be performed transcriptome-wide due to time consumption and the power of methods. A way to choose a subset of genes based on one or several genes of interest is feature selection. A feature selection algorithm combines a search technique and an evaluation measure and works upon one or several response variables (i.e. genes of interest). After obtaining a measure between the response variable(s) and each feature , a subset of features most related to the response variable(s) are extracted from the whole dataset. Using simulated data and real scRNA-seq data , we bencMany causal discovery methods have been proposed. Constraint-based causal discovery identifies causal relationships between a set of variables in two steps: skeleton estimation  and orientation . The PC algorithm is a classic and widely recognized algorithm . Causal The results of benchmarking the 11 causal discovery methods and 10 CI tests show that when causal discovery is without the problems of incomplete models (i.e. ones that miss nodes or edges from the data-generating model) and missing values, nonlinear versions of continuous optimization-based methods  perform better than others . When caThe CausalCell workflow/platform is implemented using the Docker technique and Shiny language and consists of feature selection, causal discovery, and several auxiliary functions . A paralz-score data and can turn raw data into either of the two forms. A dataset (i.e. the ‘case’) can be analyzed with or without a control dataset (i.e. the ‘control’). Researchers often identify and analyze special genes, such as highly expressed or differentially expressed genes. For each gene in a case and control, three attributes  are computed. Fold changes of gene expression are also computed (using the FindMarkers function in the Seurat package) if a control is uploaded. Genes can be ordered upon any attribute and filtered upon a combination of five conditions . Since performing feature selection transcriptome-wide is unreliable due to too many genes, filtering genes before feature selection is necessary, and different filtering conditions generate different candidates for feature selection.scRNA-seq and proteomics data generated by different protocols or methods  can be analyzed. CausalCell accepts log2-transformed data and Batch effects may influence identifying differentially expressed genes. Since removing batch effects should be performed with raw data before integrating batches and there are varied batch effect removal methods , it shouFeature selection selects a set of genes (i.e. features) from the candidate genes upon one or multiple genes of interest (i.e. response variables). As above-mentioned, candidate genes are extracted from the whole dataset upon specific conditions because performing feature selection transcriptome-wide is unreliable. Based on the accuracy, time consumption, and scalability of the nine feature selection algorithms , BAHSIC The PC and GSP algorithms can work with the 10 CI tests to provide varied options for causal discovery. In the inferred causal networks, direction of edges is determined by the meek rules , and eacThere are two ways to construct a consensus network that is statistically more reliable. One way is to run multiple algorithms (i.e. multiple CI tests) and take the intersection of some or all inferred networks as the consensus network . The othIf a scRNA-seq dataset is large, a subset of cells should be sampled to avoid excessive time consumption. We suggest that 300 and 600 cells are suitable for reliable inference if the input is Smart-seq2 and 10x Genomics data, respectively, the input contains about 50 genes, and genes are expressed in >50% cells. Here, reliable inference means that key interactions (those with high CI test significance) are inferred . More cells are needed if the input genes are expressed in fewer cells and if the input contains >50 genes. Larger sample sizes (more cells) may make more interactions be inferred, but the key interactions are stable . As HSICFour parameters influence causal discovery. First, ‘set the alpha level’ determines the statistical significance cut-off of the CI test, and large and small values make more and fewer interactions be inferred. Second, ‘select the number of cells’ controls sample size, and selecting more cells makes the inference more reliable but also more time-consuming. Third, ‘select how a subset of cells is sampled’ determines how a subset of cells is sampled. If a subset is sampled randomly, the inferred network is not exactly reproducible  cell line H2228 and the normal lung alveolar cells (as the case and control) . CompariWe used the five NSCLC cell lines , the normal alveolar cells, and genes in specific pathways to validate network inference by PC+DCC.gamma . First, https://string-db.org/) can also be used to evaluate inferred interactions  as the response variables  and selected 50 feature genes  from all genes expressed in >50% cells. Then, we applied the nine causal discovery algorithms to the 50 genes in 300 cells sampled from each of the datasets. The two DCC algorithms performed the best when processing the H2228 cells and lung alveolar epithelial cells  and experimental findings (NQO1 include that it determines cellular sensitivity to the antitumor agent napabucasin in many cancer cell lines (NQO1 are associated with susceptibility to various forms of cancer. Previous findings on PSMC4 include that high levels of PSMC4 (and other PSMC) transcripts were positively correlated with poor breast cancer survival  and analysis of multiple datasets generate several conclusions. First, although kernel-based CI tests are time-consuming , applyinSingle-cell causal discovery also faces several challenges. First, causal discovery assumes there are no unmeasured common causes , but in real data latent and unobserved variables are common and hard to identify. Specifically, inferring interactions between highly expressed or differentially expressed genes is a case of causal discovery with incomplete models (i.e. models with missing variables from the data-generating model). In this situation, what are inferred are indirect relationships instead of direct interactions between gene products. Second, constraint-based methods cannot differentiate networks belonging to a Markov equivalent class . This can be solved partly by combined use of PC and DAGMA-nonlinear (which can better determine the direction of edges). Third, the following examples indicate that the lack of relevant information makes judging inferred interactions and relationships difficult. (a) TOX is reported to activate PDCD1 in exhausted CD8 T cells in mice , but wheThe time consumption of kernel-based CI tests disallows inferring large networks, and how this challenge can be solved remains unsolved. C codes may be developed to replace the most time-consuming parts of the R functions, but this has not been done.First, exploring different biological modules or processes needs careful selection of genes . When it This manuscript presents an important tool for causal inference intended for the analysis of single cell datasets but possibly with broader applications. It compares several algorithms and incorporates a number of them in the platform and offers convincing evidence of its usefulness. With the rapid expansion of large datasets, this tool is beneficial in offering several causal inference analysis options and expediting the interpretation of data. public reviews designed to be posted alongside the preprint for the benefit of readers; (ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.Our editorial process produces two outputs: (i) Decision letter after peer review:eLife. Your article has been reviewed by 2 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Anna Akhmanova as the Senior Editor.Thank you for submitting your article "CausalCell: applying causal discovery to single-cell analyses" for consideration by The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.Essential revisions:This manuscript presents a tool for causal inference intended to be used for the analysis of single-cell datasets. The tool, named CausalCell, attempts to address a quite important question in the network biology field. That is how to infer a directed gene network to reveal causal relationships among genes. Given scRNA-seq data and a set of genes of interest, CausalCell is potentially useful for inferring cell type-specific intracellular causal networks among genes. Although in principle this can be a very helpful tool, the evidence that the incorporated algorithms are the most suitable for the proposed applications is inadequate. Data preprocessing, result illustration, and validation should be substantially strengthened.1) The authors need to either justify their choice of algorithms in the paper or expand the work to include other important algorithms.2) Throughout the paper, the authors need to be more explicit about the description of preprocessing as well as the quantification of stated claims.3) Stronger validation is needed to support and clarify the usefulness of the proposed tool.Reviewer #1 (Recommendations for the authors):1. There seem to be three general categories of causal inference algorithms: constraint-based, score-based, and hybrid .This paper appears to focus only on different implementations of constraint-based algorithms; however, it appears that score-based algorithms in general produce better results for complex biologically-driven datasets. I have checked this with experts since this is a little outside of my field. In my opinion, this is a major shortcoming that significantly reduces the impact of the current manuscript. In other words, the authors need to justify that the algorithm  they have picked is the best option for single-cell analyses . Alternatively , they should incorporate other options into the tool.2. Some statements in the comparison section need clarification. For example, in Table 1, the authors mention that "Both ExtraTrees and RandomForest perform well, but XGBoost does not." I believe it is necessary to state more details about what the context/condition is, in order to justify such claims.3. Figure 2: The comparison shows a relatively poor similarity between different algorithms. What is the reason for this?4. In Discussions, the authors state that kernel-based CI tests perform better than faster methods that make additional assumptions. I am not sure if I see the evidence for this. Especially when different kernel-based algorithms poorly agree with each other, it is unclear how a case can be made. Ideally, this issue can be addressed with a "mock" dataset in which the causal links are already known. In the absence of a known "ground truth" or a "gold standard", their arguments about accuracy and performance are in general less convincing.5. Line 353: The authors make a statement about the dataset being "large enough". I'd suggest including a formal treatment, in which the causal inference results are compared as the number of data points increases (for example by sub-sampling one of the existing datasets). I believe this will provide more convincing evidence for how many data points are required and what to expect as that number increases/decreases.Reviewer #2 (Recommendations for the authors):1) Reactome and KEGG databases curated directed links between genes or proteins, which is suitable for result evaluation.2) 50 feature genes represent a small fraction of the whole genome. This means CausalCell cannot fully take advantage of the high-throughput feature of current scRNA-seq technologies. Any suggestions to deal with this concern?3) What techniques do the authors use to generate the consensus network?4) On lines 68-70, what are the differences among regulatory networks, causal networks, (ordinary) networks, and gene networks? Readers could be confused by such an introduction.5) Figure 4 and other causal network plots should have a legend to indicate the color scheme for the upregulation or downregulation of gene expression.6) Volume and page information of Reference [49] are missing.eLife. Your revised article has been evaluated by Anna Akhmanova (Senior Editor) and a Reviewing Editor.Thank you for resubmitting your work entitled "Applying causal discovery to single-cell analyses using CausalCell" for further consideration by The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:1. The need to include more recent advances in causal discovery, such as continuous optimization-based, neural network-based methods is brought up by the reviewers again. There is a need to either include these approaches or offer concrete evidence that the PC algorithm works better  for the type of biological questions the tool is addressing. This is perhaps best done by a direct comparison of results between the methods included in CausalCell and some other representatives such as Notears, Golem, DAG-GNN, lingam, and Dlingam.2. A more thorough comparison of independence/CI tests with more recent methods such as regression-based, ranking-based, and deep neural network-based CI tests  needs to be included.Additionally, please address the more detailed issues raised by Reviewer #3 below.Reviewer #3 (Recommendations for the authors):This work developed a workflow and platform for effectively performing causal discovery from scRNA‐seq data. The workflow/platform is developed upon the benchmark of 9 feature selection algorithms, 3 causal discovery methods, 9 CI tests, and the analyses of multiple datasets. The authors suggest that kernel-based conditional independence tests generate reliable results. Some key issues are discussed, and tips for best practices are provided. In my opinion, this work has the potential to help biologists discover some causal relationships among single-cell data, but the main drawback of this approach is the lack of new technologies on causal discovery as well as CI tests, and the biological significance of the work is not quite clear. Following are some of my concerns or questions about this work:1. In recent years, continuous optimization-based methods have become the most popular method for causal discovery, which yield much better performance than the PC algorithm, I think this work should discuss and take some representative continuous optimization-based methods into account, such as Notears, Golem, DAG-GNN. There are also some causal functional model-based methods such as lingam and Dlingam should be discussed.2. The 9 independence/CI tests might not stand for the state-of-the-art, more kinds of, and more recent methods should be taken into account, such as regression-based, ranking-based, and deep neural network-based CI tests .3. Theoretically, not all causal directions can be discovered by the orientation step of the PC algorithm, how to address the Markov equivalence classes in this paper? And there are also some other constraint-based methods, why choose PC?4. I suggest the author present the time complexity of each method not just 'time consumption', as it seems a little bit confusing. For example, HSIC.gmma should work much faster than HSIC.perm, but they are both '*' at 'time consumption'.5. How to perform PC with HSIC? only 0-order CI test?6. See "……However, because most algorithms have been designed for handling limited variables and few algorithms have been evaluated using real data, applying causal discovery to single-cell data remains challenging……". I don't quite agree with this statement, because there are lots of constraint-based methods (or combining with feature selections as dimension reduction step) with different CI tests for causal discovery on RNA-sq/Microarray, and this work also does a similar or same job.7. See "……These features of CI tests enable causation between any genes and molecules to be inferred……". It should be noted that not all causation between them can be inferred by CI tests.8. " ……the time consumption of kernel-based CI tests disallows large-scale network inference……", how about the parallel PC? And sometimes one can limit the size of the conditional set.9. Actually, I wonder whether this workflow/platform can find some interesting biological results (say biomarkers) from the data. This is related to the biological significance of the work. Computational results in the paper do not provide convincing support for this point. Essential revisions:This manuscript presents a tool for causal inference intended to be used for the analysis of single-cell datasets. The tool, named CausalCell, attempts to address a quite important question in the network biology field. That is how to infer a directed gene network to reveal causal relationships among genes. Given scRNA-seq data and a set of genes of interest, CausalCell is potentially useful for inferring cell type-specific intracellular causal networks among genes. Although in principle this can be a very helpful tool, the evidence that the incorporated algorithms are the most suitable for the proposed applications is inadequate. Data preprocessing, result illustration, and validation should be substantially strengthened.1) The authors need to either justify their choice of algorithms in the paper or expand the work to include other important algorithms.(1) We have examined the representative score-based method GES (Chickering 2003) and the representative hybrid method GSP   . We used genes in the WikiPathway pathway "Metabolic reprogramming in colon cancer" (WP4290) and the data of the five lung cancer cell lines to compare the PC, GES, and GSP methods. The results suggest that GES is somewhat less accurate, but faster, than PC+DCC.γ and GSP+DCC.γ. At the same significance level, the networks of GES have more isolated nodes and more unreasonable interactions (see Appendix file 3). We think this is partly because GES is a parametric approach and assumes Gaussian distribution of data, and nonlinear relationships, missing values, and latent variables may impair the performance of GES. On the other hand, both PC+DCC.γ and GSP+DCC.γ capture the common features of reprogrammed glucose metabolism in cancer cells, but GSP+DCC.γ infers more interactions than PC+DCC.γ, even with a more stringent significance level (α=0.05 rather than 0.1) (Appendix file 3).(2) We have incorporated GES and GSP into the CausalCell platform. The original GSP package includes just two CI tests ; we now enable it to run with the 9 CI tests to provide varied options for users.2) Throughout the paper, the authors need to be more explicit about the description of preprocessing as well as the quantification of stated claims.(1) On data preprocessing. We have greatly revised the section "3. Data input and preprocessing" and made the following points clear. (a) CausalCell accepts log2-transformed data and z-score data and can turn raw data into either of the two forms. (b) Data can be sorted upon any attribute . (c) The user can filter genes upon multiple conditions  so as to, for example, select differentially expressed genes or genes with high variance. (d) Filtering genes upon specific conditions is an important preprocessing step for feature selection, because applying feature selection to genes genome-wide may impair the accuracy of feature selection.(2) We have added the option "TF=Yes/No" as a new condition for filtering genes, which makes all selected feature genes be transcription factors. More details are given in the responses to Reviewer 2's comments.(3) On removing batch effects (raised by Review 2): our situations. (a) We analyzed the exhausted and non-exhausted CD8 cells. All samples were sequenced by the same protocol, each dataset was from the same kind of tumor, and data were preprocessed and integrated by the original authors (see related citations). (b) We analyzed the mouse CD4 data. We note that batch effects are very limited, as all samples were sequenced by the same protocol and cells did not cluster topologically by depth of sequencing, experimental batch, or individual mouse. Also, we manually organized the inputs without detecting differentially expressed genes and performing feature selection. (c) We analyzed the five non-small cell lung cancer cell lines and lung alveolar cells. According to the information in the paper and GEO website, these five cell lines were sequenced in the same batch and preprocessed carefully for benchmarking scRNA-seq analysis pipelines . We do not have the raw data of the lung alveolar cells and did not remove batch effects. By using the five cell lines to make the 5-to-1 comparisons, we think the revealed key differences between the lung cancer cell lines and alveolar cells are reliable. (d) When using the five lung cancer cell lines and lung alveolar cells to strengthen the validation, we used genes in KEGG/WikiPathway pathways as inputs without detecting differentially expressed genes and performing feature selection.(4) On removing batch effects: the user's situations. If the user selects feature genes from differentially expressed genes, batch effects may have an influence. We make it clear in the revised manuscript that it is the user's responsibility to remove batch effects. This is because (a) varied methods have been developed to remove batch effects with varied performance , and (b) removing batch effects should be performed before integrating batches, and we do not know how users' data are generated and whether the uploaded data are integrated or not.(5) On the quantification of stated claims. We think the claims on algorithms' performance and on sample size are important. (a) We revised Table 1 and Table 2. A note is added to Table 1 and Table 2, respectively, explaining how the performance indicators are estimated and how advantage/disadvantage is made. (b) We recommend 300 and 600 cells for Smart-seq and 10X Genomics data in the main text. The new claim is "Our application and validation examples suggest that 300 and 600 cells are recommended for reliable inference if the input is Smart-seq2 and 10X Genomics data, the input contains about 50 genes, and genes are expressed in >50% cells. Here, reliable inference means that key  interactions are inferred , and 300 and 600 are estimated rather than accurate. More cells are needed if input genes are expressed in less cells and if the input contains more genes. While more cells make more interactions  be inferred, the key interactions are stable". (c) We also explain the recommendation of 300 and 600 cells in the section "Tips for best practices".(6) In the in the original Appendix file 3, we wrote "Third, to evaluate the impact of sample size on algorithms' performance, we ran the 9 algorithms using 200 (instead of 300) H2228 cells. The results of 200 cells were poorer than the results of 300 cells (compared with the consensus network in Main-figure 2 and Appendix file 3-figure 2), and the two DCC algorithms performed the best and were less sensitive to the decreased sample size than the two HSIC algorithms". On the newly added claim "While more cells make more interactions  be inferred, the key interactions are stable", we used genes in the WP4290 pathway and the dataset of H838 to compare networks inferred using different sub-samples . The results are given in Appendix file 3 and indicate that more cells cause more interactions be inferred but the key interactions are quite stable.3) Stronger validation is needed to support and clarify the usefulness of the proposed tool.(1) As described in the original manuscript (mostly in Appendix files), (a) we examined many real datasets  to evaluate these algorithms, (b) we developed three methods  to help validate network inference, (c) in the first example of the section "APPLICATIONS", we purposefully focused on inferring interactions between HLA genes, because these interactions have been well studied, and our results agree with experimental reports.(2) During the revision, we have taken a systematic approach using the five non-small cell lung cancer cell lines and lung alveolar cells. First, we identified differentially expressed genes in each cell line against the alveolar cells upon different conditions. A typical set of conditions was (a) gene expression value >0.1, (b) gene expression > 50% cells, (c) fold change >0.3. Second, we applied GO analysis to the differentially expressed genes in each cell line using g:Profiler (https://biit.cs.ut.ee/gprofiler/gost) . We found that the WikiPathway and KEGG pathways "Metabolic reprogramming in colon cancer" (WP4290), "Pyrimidine metabolism" (WP4022), and "Nucleotide metabolism" (hsa01232) were commonly enriched in all cancer cell lines (Appendix file 4 gives the result of the cancer cell lines A549). We also performed GO analysis using the GSEA  package, which identified the KEGG pathway "Non-small cell lung cancer" (hsa05223) as an enriched pathway. Third, we used the PC+DCC.γ to infer interactions among genes in the three pathways WP4290, hsa01232, and hsa05223 in the five cancer cell lines. Fourth, we also used GSP+DCC.γ and GES to infer interactions among genes in the WP4290 pathway.(3) Numerous studies report that glucose metabolism is reprogrammed and nucleotide synthesis is increased in cancer cells. The increased glucose uptake and fermentation of glucose to lactate are known as the Warburg Effect and have been documented for over 90 years. Thus, we first examined and compared the WP4290 pathway in the five lung cancer cell lines and lung alveolar cells. The key features of the reprogrammed glucose metabolism in the cancer cell lines are (a) glucose intake is increased, (b) the glycolysis/TCA cycle intermediates are used for synthesizing nucleotide, (c) lactate generation is increased. The inferred networks capture these features (Appendix file 4). (a) Activations of SLC2A1 (which encodes a major glucose transporter and controls glucose intake), PGD (which promotes glucose metabolism into the pentose phosphate shunt), PSAT1 , and LDHA  are inferred in all cancer cell lines but not in alveolar cells. (b) Activations of genes by downstream genes are inferred; this sort of feedback is an important feature of metabolism. (c) In contrast, none of these features occur in the alveolar cells . These inferred results are literature-supported (there are many Warburg Effect-related papers) and biologically reasonable, even though the absence of metabolites in the data flaws network inference (Appendix file 4).(4) We next examined the "Pyrimidine metabolism" (WP4022) pathway. We used genes in hsa00240 ("Pyrimidine metabolism" defined in KEGG) to perform the inference (because WP4022 contains too many POLR gene families) and used the more readable WP4022 pathway to illustrate the results. Compared with glucose metabolism, pyrimidine metabolism has many reversible reactions, making interactions vary more significantly in cells (compared with WP4290). The following genes and reactions are notable. (a) TYMS catalyzes dUMP->dTMP unidirectionally toward DNA synthesis. (b) Tk1/2 catalyze Thymidine->dTMP and Deoxyuridine->dUMP toward DNA synthesis (while NT5C/E/M do the opposite). (c) DUT catalyzes dUTP->dUMP (note that dUMP is the substrate for TYMS). (d) TYMP catalyzes Thymidine->Thymine unidirectionally away from DNA synthesis. (e) ENTPD1/3 catalyze dTTP->dTDP->dTMP, UTP->UDP->UMP, and CTP->CDP->CMP away from DNA and RNA synthesis (but AK9/NME reverse these reactions). (f) NT5C/E/M catalyze dCMP->Deoxycytidine, dUMP-> Deoxyuridine, and dTMP->Thymidine away from DNA synthesis. In accordance with these, the following interactions were inferred from cancer cell lines. (a) TYMS (the key gene that promotes DNA synthesis) is activated (and is not repressed by any gene) in all cancer cell lines but not in alveolar cells. (b) Tk1/2 are activated in cancer cells and alveolar cells. (c) DUT is activated in all cancer cell lines but is not expressed in alveolar cells. (d) activations of TYMP  by multiple others are inferred in alveolar cells. (e) ENTPD1/3 (genes making reactions away from DNA synthesis) are activated only in alveolar cells. (f) NT5C/E/M are repressed in all cancer cell lines but are not expressed in alveolar cells. The most notable may be DUT->Tk1 and DUT->TYMS in all cancer cell lines, indicating feedforward or coordinated regulations that promote DNA synthesis. These features are biologically reasonable, even though the absence of metabolites in the data flaws network inference and there are many reversible reactions (Appendix file 4).(5) We further examined the "Non-small cell lung cancer" (hsa05223) pathway. We used the "graphite" R package to turn hsa05223 into an adjacency matrix and mapped the inferred causal network to the matrix. An inferred interaction is assumed to be mapped to hsa05223 if it can be mapped to an edge or a path . About 30% of inferred interactions can be mapped to hsa05223. In addition, hsa05223 contains sub-pathways such as p53 signaling pathway and PI3K-AKT pathway, thus containing considerable unannotated epistatic interactions (Appendix file 4). Synergistic interactions are also unannotated in hsa05223, and many of epistatic and synergistic interactions are literature-supported . We found that many inferred interactions can be mapped to epistatic and synergistic interactions in hsa05223. Taken together, in each cell line, about 50% of inferred interactions can be mapped to hsa05223 (Appendix file 4). These percentages are underestimated, because there are considerable feedback regulations  in hsa05223 which are unannotated and we do not handle these interactions . If these interactions are annotated and handled, higher percentages of inferred interactions can be mapped to hsa05223. It is also worth noting that the annotated hsa05223 has defects because it is not detected as an enriched pathway by g:Profiler.(6) We also used the GES and GSP+DCC.γ to repeat the analysis of the WP4290 pathway in the five lung cancer cell lines, and the same features of reprogrammed glucose metabolism were obtained (Appendix file 3).(7) The above results are described briefly in the main text and in detail in the revised Appendix file 3 and 4. The reader may notice that inferred interactions vary to different extent in different cell lines. These cell lines have different genetic backgrounds, including different numbers of genes in WP4290 are expressed in different cell lines, and it has been recently found that cancer cells show great transcriptional plasticity .Reviewer #1 (Recommendations for the authors):1. There seem to be three general categories of causal inference algorithms: constraint-based, score-based, and hybrid .This paper appears to focus only on different implementations of constraint-based algorithms; however, it appears that score-based algorithms in general produce better results for complex biologically-driven datasets. I have checked this with experts since this is a little outside of my field. In my opinion, this is a major shortcoming that significantly reduces the impact of the current manuscript. In other words, the authors need to justify that the algorithm  they have picked is the best option for single-cell analyses . Alternatively , they should incorporate other options into the tool.(1) The NIPS-2017 paper  reports a hybrid algorithm (IGSP), which features in (a) combining a score-based method and CI tests, (b) based on GSP and being able to handle interventional data, (c) outperforming the representative score-based method GIES. We have examined the representative score-based method GES/GIES and the representative hybrid method GSP/IGSP  using small-scale simulated and real data . Comparisons of GES, GSP+DCC.γ, and PC+DCC.γ based on these data suggest that GSP+DCC.γ and PC+DCC.γ have comparable performance and somewhat outperform GES, probably because GES assumes Gaussian distribution of data but single-cell data are complex.(2) We have incorporated GES and GSP into our platform and briefly described the benchmarking of GES, GSP, and PC (Appendix file 3). The GSP package originally contains only two CI tests ; we now enable it to run with the 9 CI tests.(3) Perturbations can be applied to single cells on the protein level , on the genome level (using the CRISPR/Cas9 system), and on the transcriptome level (using the CRISPR/dCas9 system). The latter two combine pooled CRISPR screen with scRNA-seq . A revised method is direct-capture Perturb-seq, a versatile screening approach in which expressed sgRNAs are sequenced alongside single-cell transcriptomes . We do not include GIES and IGSP in the current version of CausalCell because of unsolved issues, including (a) individual cells receive different gRNAs and are perturbed according to the gRNAs received by the cell, thus the numbers of cells targeted by different gRNAs vary substantially in an experiment, and (b) the data  are much sparser  than normal 10X Genomics data.(4) Please see also the responses to Essential revisions.2. Some statements in the comparison section need clarification. For example, in Table 1, the authors mention that "Both ExtraTrees and RandomForest perform well, but XGBoost does not." I believe it is necessary to state more details about what the context/condition is, in order to justify such claims.(1) We revised Table 1 and Table 2. A note is added to Table 1 and Table 2, respectively, explaining how the performance indicators are estimated and how advantage/disadvantage is determined. An important claim is the recommendation of 300 and 600 cells for Smart-seq and 10X Genomics data. We have explained why the two numbers are recommended in the main text and details are also given in Appendix file 3 and 4.(2) Please see also the responses to Essential Revisions.3. Figure 2: The comparison shows a relatively poor similarity between different algorithms. What is the reason for this?(1) We now make it clear that Figure 2 shows that (a) kernel-based CI tests are better than non-kernel-based methods, (b) DCC.perm, DCC.γ, HSIC.perm, and HSIC.γ are more accurate than the other three kernel-based methods, (c) RCIT and RCoT are approximated versions of KCIT, and that is why they are poorer than DCC- and HSIC-based algorithms. Thus, it is not surprising that the comparison shows a relatively poor similarity between different algorithms.(2) We have greatly revised the legends of Figure 2 and Appendix file 3-figure 2. We note that RCIT and RCoT are much faster than DCC- and HSIC-based algorithms and thus suitable for large-scale inference.4. In Discussions, the authors state that kernel-based CI tests perform better than faster methods that make additional assumptions. I am not sure if I see the evidence for this. Especially when different kernel-based algorithms poorly agree with each other, it is unclear how a case can be made. Ideally, this issue can be addressed with a "mock" dataset in which the causal links are already known. In the absence of a known "ground truth" or a "gold standard", their arguments about accuracy and performance are in general less convincing.how a case can be made, we explain in the revised legend of Figure 2 that the accuracy of PC+9 CI tests was evaluated with four steps. First, 9 causal networks were inferred using the 9 CI tests. Second, pairwise structural Hamming distances (SHD) between these networks were computed, which indicate that the networks of DCC.γ, DCC.perm, HSIC.γ, and HSIC.perm share the highest similarity. Third, a consensus network was built using the networks of the above four CI tests and was assumed to be closer to the ground truth than any network inferred by a single algorithm. Fourth, each of the 9 networks was compared with the consensus network.(1) Figure 2 and Appendix file 3-figure 2 give the evidence. On (2) As mentioned above, RCIT and RCoT are poorer than DCC- and HSIC-based algorithms because the two are approximated versions of KCIT.(3) We used multiple synthetic datasets to examine these algorithms but found that performance evaluation upon synthetic datasets is unreliable, probably because the defined relationships between variables poorly catch the real relationships between varied genes in varied cells. Appendix file 3-figure 1BD shows an example. That is why we chose the five lung cancer cell lines and alveolar cells to make 5-to-1 comparisons, and think this may be the best way one can use to estimate algorithms' performance.(4) We have made more analyses during the revision using the five lung cancer cell lines and genes in specific pathways. Key features of interactions in these genes are well annotated.5. Line 353: The authors make a statement about the dataset being "large enough". I'd suggest including a formal treatment, in which the causal inference results are compared as the number of data points increases (for example by sub-sampling one of the existing datasets). I believe this will provide more convincing evidence for how many data points are required and what to expect as that number increases/decreases.We evaluated the impact of sample size on algorithms' performance by running the 9 algorithms (CI tests) using 200 and 300 H2228 cells. The results indicate that 200 cells generate poorer networks than 300 cells . In both situations, complete kernel-based CI tests performed better than other CI tests. We also inferred interactions between genes in the "Metabolic reprogramming in colon cancer" (WP4290) pathway using 200, 400, 600, and 800 H838 cells (Appendix file 3-figure 3). The results indicate that more cells make more interactions be inferred, but the interactions with high significance are stable. Upon these results, we explain in the main text  that 300 and 600 cells for Smart-seq and 10X Genomics data are suitable for typical situations, that the required sample size depends on multiple factors, and that performing inference using large datasets is highly time-consuming.Reviewer #2 (Recommendations for the authors):1) Reactome and KEGG databases curated directed links between genes or proteins, which is suitable for result evaluation.Substantial analyses have been made using the data of the five lung cancer cell lines and alveolar cells and the genes in multiple pathways in the KEGG and WikiPathway databases. Please see the responses to Essential Revisions.2) 50 feature genes represent a small fraction of the whole genome. This means CausalCell cannot fully take advantage of the high-throughput feature of current scRNA-seq technologies. Any suggestions to deal with this concern?(1) Trade-offs between time consumption, network accuracy, and network size are inevitable. We also note that statisticians are working on improving the time consumption of kernel-based CI tests  and that a researcher usually cares only about specific genes linked to a phenotype or disease.(2) In most situations, the genes linked to a phenotype or disease are unknown. Thus, a tool should take advantage of the high-throughput feature of current scRNA-seq to identify such genes. Instead, inferring networks genome-wide is often not required.(3) In the section “Tips for best practices”, we wrote, “a solution to fully taking advantage of the high-throughput feature of scRNA-seq data is to infer multiple networks with shared genes and then try to merge these networks”.3) What techniques do the authors use to generate the consensus network?We have added in the section "5. Causal discovery" that there are two ways to build a consensus network: using shared interactions inferred by different algorithms or using shared interactions inferred by running an algorithm for multiple times.4) On lines 68-70, what are the differences among regulatory networks, causal networks, (ordinary) networks, and gene networks? Readers could be confused by such an introduction.These methods were historically named and may confuse a reader. We feel this manuscript is not a proper place to explain the differences in detail and refer the reader to these reviews. We also use the quotation mark to cite them in the revised manuscript and briefly explain the difference between signaling networks and regulatory networks in the Discussion.5) Figure 4 and other causal network plots should have a legend to indicate the color scheme for the upregulation or downregulation of gene expression.A color bar is added.6) Volume and page information of Reference [49] are missing.The mistake is corrected.References1. Boch et al. High-content CRISPR screening. Nat Rev. Methods Primers. 2022, 2, 8.2. Chickering, D.M. Optimal structure identification with greedy search. Journal of Machine Learning Research 2003, 3, 507-554.3. Dong et al. EGFR and c-MET cooperate to enhance resistance to PARP inhibitors in hepatocellular carcinoma. Cancer Res. 2918, 79, 819–829.4. Househam et al. Phenotypic plasticity and genetic control in colorectal cancer evolution. Nature 2022, 611, 744-753.5. Replogle et al. Combinatorial single-cell CRISPR screens by direct guide RNA capture and targeted sequencing. Nature Biotechnology 2020, 38, 954-961.6. Sachs et al. Causal protein-signaling networks derived from multiparameter single-Cell Data. Science 2005, 308, 523-529.7. Solus et al. Consistency guarantees for greedy permutation-based causal inference algorithms. Biometrika 2021, 108, 795–814.8. Tran et al. A benchmark of batch-effect correction methods for single-cell RNA sequencing. Genome Biology 2020, 21, 12.9. Wang et al. BAX and CDKN1A polymorphisms correlated with clinical outcomes of gastric cancer patients treated with postoperative chemotherapy. Medical Oncology 2014, 31, 249.10. Wang et al. Permutation-based causal inference algorithms with interventions. NIPS 2017.The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:1. The need to include more recent advances in causal discovery, such as continuous optimization-based, neural network-based methods is brought up by the reviewers again. There is a need to either include these approaches or offer concrete evidence that the PC algorithm works better  for the type of biological questions the tool is addressing. This is perhaps best done by a direct comparison of results between the methods included in CausalCell and some other representatives such as Notears, Golem, DAG-GNN, lingam, and Dlingam.(1) Among the various causal discovery methods, we think that PC + kernel-based CI tests may best suit various situations, including missing values in data, different distributions of data, noise in data, and complex relationships between genes. Therefore, we initially focused on identifying what kernel-based CI tests perform best. Agreeing with theoretical studies (see the PhD dissertation of Patra Verbyla and other related papers), we found that DCC and HSIC-based CI tests outperform others. Now, we have benchmarked eight new methods (including the continuous optimization-based NOTEARS and its follow-up methods), together with two new CI tests, using simulated and real data. In the benchmarking, these methods are called NOTEARS-linear , NOTEARS-nonlinear , GOLEM , DAGMA-linear , DAGMA-nonlinear , ICALiNGAM , DirectLiNGAM , and DAG-GNN , and the CI tests are called GCM  and KRESIT . Simulated data are generated using the methods described in NOTEARS-linear and NOTEARS-nonlinear. Real data are the scRNA-seq data of five non-small cell lung cancer (NSCLC) cell lines , which have been used widely to evaluate scRNA-seq data analysis algorithms.(2) Here we describe seven cases of benchmarking. First, we used the method in NOTEARS-linear to simulate 600 samples of 10 variables. In this case, continuous optimization-based methods  perform very well . Since tSecond, we benchmarked these methods using simulated 600 samples of 10 variables in nonlinear relationships. Data were simulated using the method in NOTEARS-nonlinear. In this case, nonlinear versions of continuous optimization-based methods (NOTEARS-nonlinear and DAGMA-nonlinear) perform better than others. But, no method reaches TPR=1.0 .Third, we evaluated these methods by inferring the network of 49 genes in the "non-small cell lung cancer" (hsa05223) KEGG pathway using 600 A549 cells . We used the graphite package in Bioconductor to derive the DAG of the human hsa05223 pathway and used the DAG as the ground truth to evaluate the accuracy of inferred networks . In thisThe poor performance of continuous optimization-based methods on real data may be due to (a) missing values in scRNA-seq data, (b) inference with missing variables , (c) complex relationships between genes, and (d) noise in data. Note that many relationships between the 49 hsa05223 genes should be indirect because some genes remain unannotated in the pathway and not all annotated genes are highly expressed in these cells.We next evaluated the power of these methods to infer indirect relationships between genes. To perform the fourth case of benchmarking, we first performed pathway enrichment analysis using the g:Profiler program to identify enriched pathway genes in NSCLC cells. A small and well annotated pathway is the pentose phosphate pathway WP134 (“WP” indicates pathways defined in the WikiPathways database). Branching from glycolysis at the first committed step of glucose metabolism, the pentose phosphate pathway is critical for cancer cells because it generates pentose phosphates to supply their high rate of nucleic acid synthesis and provides NADPH that is required for both the synthesis of fatty acids and cell survival under stress . This pathway is a potential target for cancer therapy . Since the five enriched WP134 genes G6PD, PGD, TKT, TALDO1, PGLS in A549 cells do not encode transcription factors, we used BAHSIC as the method and the five genes as the response variables to perform feature selection to identify these genes' potential transcription factors. The top two transcription factors are ENO1 and NPM1. We then used the seven genes and 600 A549 cells to infer the regulatory network. The result is notable in that (a) some continuous optimization-based methods  can infer edges this time , (b) theTo examine the conjecture continuous optimization-based methods perform better when there are direct interactions between genes, we performed the fifth case of benchmarking to examine only the enriched hsa05223 genes. We used the GSEA program to identify 14 highly and differentially expressed hsa05223 genes in H2228 cells compared with the lung alveolar cells (see Appendix 1 table 1) . Then, we used the ten methods and 600 H2228 cells to infer the networks of the 14 genes. This time, all continuous optimization-based methods perform worse (worse than the case of 49 hsa05223 genes). However, PC+DCC.γ still performs well , as the A challenge when benchmarking methods with scRNA-seq data is identifying or building a reasonable ground truth. Working also on causal hypothesis testing, we find that systematically analyzing accepted and rejected causal hypotheses can help build approximations to the ground truth of causal relationships between genes. To further confirm that the poor performance of continuous optimization methods on real data is due greatly to missing variables (inference with an incomplete model), we simulated nonlinear relationships between 20 variables using the method of NOTEARS-nonlinear and inferred the relationships between 18 variables. In the first round of inference X8 and X20 were removed from the list of variables, and in the second round of inference X4 and X7 were removed from the list of variables. The results indicate that PC+DCC.γ identifies more indirect relationships between the 18 variables than DAGMA-nonlinear in both rounds .Finally, we examined whether these methods are sensitive to missing values in data. We randomly removed 10% and 20% of values per variable from the simulated data that describe nonlinear relationships between 10 variables see  and perf(3) The above results suggest that continuous optimization-based methods  perform very well if missing values and missing variables are not serious. We have incorporated DAGMA-nonlinear into CausalCell.2. A more thorough comparison of independence/CI tests with more recent methods such as regression-based, ranking-based, and deep neural network-based CI tests  needs to be included.(1) (a) We have examined multiple other CI tests, including the regression-based tests SCIT and KRESIT , the GAN-based test GCIT , and the generalized covariance measure-based GCM  . (b) We ran PC+SCIT using the authors' MATLAB code and found that the inferred networks are skeletons . Also, since SCIT is much faster than KCIT, but Type I error is comparable to KCIT, SCIT may not outperform HSIC and DCC. (c) We compared PC+KRESIT and PC+GCM with PC+DCC.γ and PC+GaussCItest, together with the other eight new causal discovery methods. We found that PC+KRESIT is 69 times slower than PC+DCC.γ and underperforms PC+DCC.γ in terms of accuracy . PC+GCM (2) (a) Based on the paper of Azadkia and Chatterjee , Shi and Han indicate that CI test using the Azadkia–Chatterjee coefficient is inefficient , and Lin and Han propose a revised method to overcome the rate-inefficiency . But, Lin and Han do not give an implementation of their revised method. (b) Both Figure 2 in Bellot and van der Schaar (2019) and our running of GCIT suggest that the set of conditional variables should not be empty. Thus, the current GCIT cannot work with PC, as it cannot handle the 0-order situation. (c) We used Google to search the internet, with the keywords "multilayer perceptron conditional independence" and "mlp conditional independence". The top and most relevant papers are "Shen Y et a. Conditional independence in Testing Bayesian Networks. Proc. Machine Learning Res. 2019; Shrivastava H et al. A deep learning approach to recover conditional independence graphs. NeurIPS 2022; Duong B, Nguyen T. Conditional independence testing via latent representation learning. arXiv 2022". Shen Y et al. examined CI in a specific class of Bayesian network (Testing Bayesian Network). The uGLAD test reported by Shrivastava H et al. assumes that the input comes from an underlying multivariate Gaussian distribution. The LCIT test reported by Duong B and Nguyen T also demands that the set of conditional variables should not be empty. Therefore, we did not examine these CI tests.Summary:(a) Using conditional independence of variables to infer the underlying causal network does not require specific assumptions about the functional forms of causal relationships or distribution of data. But, this approach only infers a set of equivalent networks. Other methods make some assumptions, for example, a differentiable score function. The new methods (including the continuous optimization-based) recently reported raise two questions – which ones perform well when being applied to scRNA-seq data, and whether the inferred networks are reliable. The above results suggest that continuous optimization-based methods  perform well if the inference is without missing variables and the data has no missing values, but otherwise not, and that PC + kernel-based CI tests  can tolerate inference with missing variables and missing values in scRNA-seq data. (b) Missing values in scRNA-seq have drawn researchers’ attention, but inference with missing variables may present a more serious problem for causal discovery. (c) The benchmarking results suggest that GCM is also a good CI test. (d) As inferred causal networks help the researcher better draw causal hypotheses, it is sensible to use causal hypothesis testing to evaluate inferred networks. We develop a method using causal hypothesis testing to build approximate ground truths of causal relationships between variables. By comparing inferred networks with two kinds of approximate ground truth – the KEGG pathways and the approximate ground truth built upon causal hypothesis testing, one can better ensure the reliability of causal discovery . (e) We have incorporated DAGMA-nonlinear and GCM into CausalCell. (f) The above results allow us to look at causal discovery methods from a more balanced perspective, and we have revised the main text and Appendix 3 significantly.Additionally, please address the more detailed issues raised by Reviewer #3 below.Reviewer #3 (Recommendations for the authors):This work developed a workflow and platform for effectively performing causal discovery from scRNA‐seq data. The workflow/platform is developed upon the benchmark of 9 feature selection algorithms, 3 causal discovery methods, 9 CI tests, and the analyses of multiple datasets. The authors suggest that kernel-based conditional independence tests generate reliable results. Some key issues are discussed, and tips for best practices are provided. In my opinion, this work has the potential to help biologists discover some causal relationships among single-cell data, but the main drawback of this approach is the lack of new technologies on causal discovery as well as CI tests, and the biological significance of the work is not quite clear. Following are some of my concerns or questions about this work:1. In recent years, continuous optimization-based methods have become the most popular method for causal discovery, which yield much better performance than the PC algorithm, I think this work should discuss and take some representative continuous optimization-based methods into account, such as Notears, Golem, DAG-GNN. There are also some causal functional model-based methods such as lingam and Dlingam should be discussed.(1) Many thanks for the valuable comments. We have used both simulated (using the methods described in NOTEARS-linear and NOTEARS-nonlinear) and real data  to evaluate the accuracy and time consumption of multiple methods. The results are presented in the responses to the editor's first suggestion. Evaluating the accuracy of networks inferred using scRNA-seq data is challenging because of the problems of missing values and missing variables and the lack of ground truth. Upon our recent work on causal hypothesis testing, we develop a method to reasonably build an approximation to the ground truth of a set of genes in specific cells. We use two kinds of approximate ground truth – KEGG pathways and the approximation to ground truth built upon causal hypothesis testing – to evaluate inferred networks. We also use gene annotations and reported gene functions to evaluate inferred networks.(2) We examined the representative continuous optimization-based method NOTEARS  and its follow-up methods, including NOTEARS-nonlinear , GOLEM , DAGMA-linear , and DAGMA-nonlinear . We also examined the LiNGAM-based methods, including ICALiNGAM  and DirectLiNGAM . Some of these methods perform very well on simulated data but poorly on real data. We identified that missing variables (inference with incomplete models) and missing values in data are two causes of poor performance of continuous optimization-based methods.2. The 9 independence/CI tests might not stand for the state-of-the-art, more kinds of, and more recent methods should be taken into account, such as regression-based, ranking-based, and deep neural network-based CI tests .We have examined more CI tests (please see the responses to the editor's second suggestion). The results suggest that GCM is also a good one and we have added the GCM test into the platform.3. Theoretically, not all causal directions can be discovered by the orientation step of the PC algorithm, how to address the Markov equivalence classes in this paper? And there are also some other constraint-based methods, why choose PC?(1) We chose PC for three reasons. (a) It is a well-recognized constraint-based method. (b) We actually examined FCI in the study, and found that, as FCI infers more kinds of interactions than PC does, the inferred causality is too complex for biologists to serve as reliable leads for subsequent experimental studies.(2) There is a trade-off between performing inference without assumptions about the functional forms of causal relationships or distribution of data and distinguishing networks of the same Markov equivalence class. As the above benchmarking results show , 4, 6, P(3) We have discussed the two issues in the revised Discussion.4. I suggest the author present the time complexity of each method not just 'time consumption', as it seems a little bit confusing. For example, HSIC.gmma should work much faster than HSIC.perm, but they are both '*' at 'time consumption'.(1) Indeed, time complexity and time consumption are different concepts. We did not describe the accurate time complexity because (a) the time complexity of some algorithms may be too complex for ordinary biologists, (b) the time complexity of some algorithms is empirical  instead of theoretical, (c) time complexity does not accurately indicate time consumption because the latter also depends on sample size and parameters .(2) We have added time complexity in Appendix 3.5. How to perform PC with HSIC? only 0-order CI test?(1) HSIC is a measure of dependency between two variables. Petras Verbyla developed a method to apply HSIC to CI test. Performing two extra transformations can determine if X and Y are conditionally independent given the conditioning set Z: first, performing nonlinear regressions for X and Z and for Y and Z, respectively, to generate the residuals 6. See "……However, because most algorithms have been designed for handling limited variables and few algorithms have been evaluated using real data, applying causal discovery to single-cell data remains challenging……". I don't quite agree with this statement, because there are lots of constraint-based methods (or combining with feature selections as dimension reduction step) with different CI tests for causal discovery on RNA-sq/Microarray, and this work also does a similar or same job.(1) The sample sizes of almost all microarray and RNA-seq experiments are <100 per tissue. In addition, a tissue sample  contains varied cells. Thus, the reliability of inferring causal networks from microarray data and RNA-seq data is questionable. One specific study reporting very large RNA-seq samples was performed by Maathuis M et al. . This study was specific in that RNA-seq was applied to the single-cell organism yeast. Basically, only scRNA-seq can accurately detect gene expression in hundreds of isolated single cells and generate data for reliable causal discovery.(2) We have deleted this sentence.7. See "……These features of CI tests enable causation between any genes and molecules to be inferred……". It should be noted that not all causation between them can be inferred by CI tests.This sentence has been revised.8. " ……the time consumption of kernel-based CI tests disallows large-scale network inference……", how about the parallel PC? And sometimes one can limit the size of the conditional set.(1) We indeed use the parallel PC, which allows multiple tasks to be run simultaneously.(2) Limiting the size of the conditional set is a good point. We have now set the size of the conditional set as a parameter.9. Actually, I wonder whether this workflow/platform can find some interesting biological results (say biomarkers) from the data. This is related to the biological significance of the work. Computational results in the paper do not provide convincing support for this point.(1) We analyzed many scRNA-seq datasets (see Appendix files). We previously found that some inferred interactions are difficult to interpret. As we use two kinds of approximate ground truth to evaluate inferred networks in this round of revision, we find that  many inferred relationships  are quite reliable but are indirect causal relationships between genes.(2) Instead of stressing the novelty and significance of our findings, we hope the tips for best practices are helpful. To our knowledge, no such tips have been reported.References1. Azadkia M, Chatterjee S. A simple measure of conditional dependence. Ann. Statist 2021.2. Bello K et al. DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterization. NeurIPS 2022.3. Bellot A, van der Schaar M. Conditional Independence Testing using Generative Adversarial Networks. NeurIPS 2019.4. Cho ES et al. The Pentose Phosphate Pathway as a Potential Target for Cancer Therapy. Biomol Ther (Seoul) 2018.5. Dragoj M et al. Association of CCND1 overexpression with KRAS and PTEN alterations in specific subtypes of non-small cell lung carcinoma and its influence on patients' outcome. Tumour Biol. 2015.6. Ghanem N et al. The Pentose Phosphate Pathway in Cancer: Regulation and Therapeutic Opportunities. Chemotherapy 2021.7. Ikeuchi T et al. Python package for causal discovery based on LiNGAM. Journal of Machine Learning Research 2023.8. Lin Z, Han F. On boosting the power of Chatterjee's rank correlation. arXiv 2021.9. Maathuis MH et al. Predicting causal effects in large-scale systems from observational data. Nature Methods 2010.10. Ng I et al. On the Role of Sparsity and DAG Constraints for Learning Linear DAGs. arXiv 2021.11. Patra KC, Hay N. The pentose phosphate pathway and cancer. Trends Biochem Sci. 2014.12. Rao G et al. Inhibition of AKT1 signaling promotes invasion and metastasis of non-small cell lung cancer cells with K-RAS or EGFR mutations. Sci. Rep. 2017.13. Sasaki H et al. CCND1 messenger RNA expression is correlated with EGFR mutation status in lung cancer. Clin Lung Cancer. 2007.14. Shah R, Peters J. The hardness of conditional independence testing and the generalized covariance measure. Annals of Statistics 201915. Shi H et al. On Azadkia–Chatterjee’s conditional dependence coefficient. arXiv 2022.16. Shimizu S et al. DirectLiNGAM: A direct method for learning a linear non-Gaussian structural equation model. Journal of Machine Learning Research 2011.17. Shimizu S et al. A linear nonGaussian acyclic model for causal discovery. Journal of Machine Learning Research 2006.18. Verbyla P et al. Exploiting general independence criteria for network inference. bioRxiv 2017.19. Yu Y et al. DAG-GNN: DAG Structure Learning with Graph Neural Networks. arXiv 2019.20. Zhang R et al. RXRα provokes tumor suppression through p53/p21/p16 and PI3K-AKT signaling pathways during stem cell differentiation and in cancer cells. Cell Death Dis. 2018.21. Zhang H et al. Residual Similarity Based Conditional Independence Test and Its Application in Causal Discovery. AAAI 2022.22. Zhang Q et al. Feature-to-feature regression for a two-step conditional independence test. UAI 2017.23. Zheng X et al. DAGs with NO TEARS: Continuous Optimization for Structure Learning. arXiv 2018.24. Zheng X et al. Learning Sparse Nonparametric DAGs. arXiv 2020.Zheng X et al. Learning Sparse Nonparametric DAGs. arXiv 2020.


Kinases are key players in cancer-relevant pathways and are the targets of many successful precision cancer therapies. Phosphoproteomics is a powerful approach to study kinase activity and has been used increasingly for the characterization of tumor samples leading to the identification of novel chemotherapeutic targets and biomarkers. Finding co-regulated phosphorylation sites which represent potential kinase-substrate sets or members of the same signaling pathway allows us to harness these data to identify clinically relevant and targetable alterations in signaling cascades. Unfortunately, studies have found that databases of co-regulated phosphorylation sites are only experimentally supported in a small number of substrate sets. To address the inherent challenge of defining co-regulated phosphorylation modules relevant to a given dataset, we developed PhosphoDisco, a toolkit for determining co-regulated phosphorylation modules. We applied this approach to tandem mass spectrometry based phosphoproteomic data for breast and non-small cell lung cancer and identified canonical as well as putative new phosphorylation site modules. Our analysis identified several interesting modules in each cohort. Among these was a new cell cycle checkpoint module enriched in basal breast cancer samples and a module of PRKC isozymes putatively co-regulated by CDK12 in lung cancer. We demonstrate that modules defined by PhosphoDisco can be used to further personalized cancer treatment strategies by establishing active signaling pathways in a given patient tumor or set of tumors, and in providing new ways to classify tumors based on signaling activity.  •PhosphoDisco is a computational approach to define co-regulated phosphosites.•Implementation finds signaling modules in breast and non-small cell lung cancer.•Identified a cell cycle checkpoint module in enriched basal breast tumors.•PRKC-associated and proliferative enriched modules found in lung tumors. et al., demonstrate the use of PhosphoDisco, a newly developed tookit for the identification of co-regulated phosphorylation sites from tandem mass spectrometry–based phosphoproteomic data. Using data from breast and non-small cell lung cancers they identify several clinically relevant phosphorylation modules including a cell cycle checkpoint module enriched in basal breast cancer samples and a module of PRKC isozymes putatively co-regulated by CDK12 in lung cancer.Schraink, Blumenberg, and Hussey  Protein phosphorylation results in conformational changes, leading to changes in protein activity, substrate affinity, and degradation. This process is regulated by enzymatic kinases and phosphatases that catalyze the transfer of phosphate between their substrates. Subsequently, the activity of these enzymes results in the activation or deactivation of signaling pathways that drive different cellular processes such as cell growth, apoptosis, and differentiation. Improper regulation of these pathways can lead to severe disease states such as cancer. Although protein kinase genes account for only 2% of human genes, up to 30% of all human proteins can be modified by kinase activity , 2. TherQuantitation of the phosphoproteome by mass spectrometry (MS) provides a particularly useful perspective on signaling patterns and vulnerabilities in cancer and can provide a personalized view of aberrations in potentially targetable pathways. However, comprehensively discerning the activity of phosphorylation signaling pathways is challenging because phosphorylation is often context-dependent and significantly modified in cancer. To address this challenge, the Clinical Proteomics Tumor Analysis Consortium (CPTAC) has conducted proteogenomic characterization of several cancer types by performing DNA and RNA sequencing, as well as MS-based proteomic and phosphoproteomic analysis , 16, 17.To better define these co-regulated modules in patient samples, we developed a computational toolkit, PhosphoDisco, for the analysis of tumor phosphoproteomic and proteomic data. We applied PhosphoDisco to data from a cohort of breast cancer (BRCA) tumors, and a combined data set of lung squamous cell carcinoma (LSCC) and lung adenocarcinoma (LUAD) tumors with their respective matched normal samples to define co-regulated phosphorylation modules within and across data sets. We show that putative modules can be used to nominate biomarkers for disease-specific treatments, as well as targets for novel treatment strategies, and highlight the strength of pan-cancer phosphoproteomics for cancer discovery.The dataset we chose to demonstrate the utility of PhosphoDisco was collected and processed as part of the CPTAC consortium. Comprehensive characterization of the BRCA , LSCC 1, and LUABriefly, tumor samples were snap-frozen less than 30 min after collection, after which genomic and transcriptomic sequencing was completed. Samples also underwent higher-energy C-trap dissociation (HCD) liquid chromatography (LC)-MS/MS analysis of tandem mass tag (TMT)-labeled samples for proteomic and phosphoproteomic characterization as previously described .https://www.broadinstitute.org/proteomics) was used for MS data analysis. Protein identification was performed by searching the MS/MS spectra against the protein sequence database obtained using the UCSC Table Browser (https://genome.ucsc.edu/cgi-bin/hgTables) on September 14, 2016, which contains 37,579 proteins mapped to the human reference genome (hg19), adding common contaminants, mitochondrial proteins, and non-canonical small open reading frames. The searches were performed allowing ±20 ppm mass tolerance for precursor and product ions, allowing for common modification. Peptide spectrum matches (PSMs) were filtered for 30% minimum matched peak intensity and target-decoy-based false discovery rate (FDR) estimates at the PSM level, and for proteins protein level for each TMT-plex for all TMT-plexes for a tumor type, and for phosphorylation at the site levels. Normalization of each peptide was performed using the common reference, and a two-component Gaussian mixture model-based normalization was used to nullify the effect of differential protein loading and/or systematic MS variation.The Spectrum Mill software package v7.0 pre-release  co-developed by Karl Clauser of the Carr laboratory . All relevant code and documentation can be found here: https://github.com/ruggleslab/phosphodisco.PhosphoDisco comes with a Snakemake  pipeline2(relative abundance). Input data should be structured as samples as columns and proteins/phosphopeptides as rows . A function that can perform different normalization procedures is included   . We do nnormalize_phospho_by_protein method which accounts for variation in abundance of the parent protein, and extracts variation due to changes in phosphorylation. The main caveat of this approach is that it will over-correct for peptides that are auto-phosphorylated or auto-dephosphorylated on kinases and phosphatases, respectively. In these cases, protein abundance and phosphorylation are interconnected, and therefore normalizing by the former will cancel out the latter. For this reason, PhosphoDisco also identifies putative regulator sites on kinases and phosphatases which can identify these special cases , resulting in normalized phosphopeptide abundance (ProteomicsData.normed_phospho). During normalization, only peptides that share at least as many non-missing values as the CV fold are retained; this step acts as a missing values filter. Regularization is important in the case of low correlation between a phosphopeptide and its parent protein  (A). Regularization values and CV fold are changeable parameters, and by default, the regularization values 0.00032, 0.0016, 0.008, 0.04, 0.2, 1, 5, 25, 125, 625, and 3 CV folds are used.To complete protein normalization for each phosphopeptide, we train a model using regularized linear regression, using cross-validation (CV) to choose the regularization parameter (it-learn ). We trait-learn A, result exists) A. Regulae.g., different cancer types or tumor and normal samples), we recommend first performing this normalization for each subgroup, so in the case of two cancer types A and B, and their matched normals, we would normalize A-normals, B-normals, A-tumors, and B-tumors independently. This helps ameliorate non-linear scale differences between datasets.When combining different datasets  we recommend performing this filtering step within each subgroup separately. For example, in the case of two cancer types A and B, and their matched normals, we would apply these filters to the groups of A-normals, B-normals, A-tumors, B-tumors, A, B, as well as A and B combined . In the examples below, we chose the best modules by finding the top parameter sets based on the highest adjusted Rand index as suggested to identify the most robust modules. An example table with modules can be found in calculate_module_scores method by taking the mean of all members of the module per sample. Next users can use the impute_missing_values function to impute values missing in the normed_phospho attribute of the ProteomicsData object. This can be useful for plotting purposes for data sets with few missing values but can lead to artifacts with larger missingness percentages. For this sklearn.impute is used with a KNNImputer by default.For small datasets, the nakemake  pipelinenakemake B. In thevisualize_modules function that generates a clustered heatmap of protein-normalized phosphopeptides per sample for each co-regulated module (B). User-supplied sample annotations can be visualized alongside the heatmap to show relationships between module phosphosite levels and clinical features. This visualization is a key step for users to filter and interpret results. It is also important to visualize modules because clustering algorithms will often generate spurious modules, which can be manually removed by users from downstream analyses.PhosphoDisco includes a d module B. User-scalculate_annotation_association method. These associations are useful for finding modules that may be relevant to diagnostic subgroups of tumors. An example of such an annotation table can be found in add_annotations method. When add_annotations is applied, users must supply a list, defining which columns of the DataFrame are continuous or categorical variables, as categorical columns are split into binarized groups. Next, using the calculate_annotation_association function, users can calculate p-values for module enrichment per group per categorical variables, or correlation for continuous variables. The p-values are then corrected for multiple hypothesis testing .Users can also assess whether module scores are associated with user-supplied sample annotations such as tumor subtype or survival using the rocedure ) from throcedure  , by taking the mean values per sample. To prevent collapsing features, users can set the corr_threshold variable to a value above 1. After correlated features are collapsed, the remaining missing values are imputed with a sklearn.impute object (default is KNNImputer). The collected data is stored in the ProteomicsData.possible_regulators_data attribute. For this analysis, raw phosphopeptide data for each possible regulator is used, rather than protein-normalized data, to allow for the identification of auto-phosphorylation or auto-dephosphorylation, as discussed previously.Proteins regulating the phosphorylation status of multiple phosphopeptides within a module are of particular interest as potential upstream targets for module inhibition and treatment. We therefore include methods to assist with nominating regulators for modules. Users can provide a list of gene names of putative regulators, matching identifiers in the top level of the index for the protein and phosphopeptide tables. The function  consolidcalculate_regulator_association method finds the association between the vector of module scores and features in the possible_regulators_data attribute (D) for each pair of a regulator and a module. The method can be run in two modes: (a) it simply returns the correlation (default is Spearman), or (b) it builds a regularized linear model using possible_regulator_data as features and the modules scores at targets. There is also an option to transform the module scores so that the relationship being quantified between the regulators and modules scores is a sigmoid curve, as that is often the relationship between log(kinase abundance) and substrate concentration (A).Once regulator data is collected, the ttribute D for eacntration , 29  and one column with a comma separated ordinal value of the modified amino acid(s) along with a corresponding protein fasta file. Users can employ the collect_aa_sequences method  (E) of amino acids at each site. The function visualize_aa_similarity (B). This allows users to identify multiple groups of similar peptides within a module.The method richment  then finp-value) E of aminmilarity  can thenmilarity B. This acollect_aa_sequences to collect sequences, the ptm_ssgsea function can be used to find enriched peptide sets. In addition, users can apply gene ontology enrichment tests on gene sets from each module, using gseapy and Enrichr  to specifically check for enrichment of protein complexes within modules.In addition to motif discovery, flanking amino acids can also be used to compare phosphorylation modules to known modified peptide sets from PTM-ssGSEA , which c Enrichr , 34. EitFig. 1F, . In addiThe strength of the signal for a given module will depend, to some degree, on the number of samples. Therefore, to provide guidance on sample size and associated limitations, we completed a sample size sensitivity analysis based on input size. Specifically, we assessed PhosphoDisco’s ability to produce consistent results when applied to smaller cohorts by running the pipeline on 15 randomly-selected patient cohorts of size n = 25, n = 50, n = 75 or n = 100, resulting in 60 individual PhosphoDisco runs. This allowed us to determine if these smaller subsampled cohorts would consistently recapitulate our original findings. Using the breast cancer dataset and specifically module BRCA-63 as a test case, we determined the ability of PhosphoDisco to resolve the same phosphosites within each cluster at different cohort sizes based the adjusted rand index and percent of phosphosite overlap in BRCA-63 across the 15 runs. Patient samples were subsampled from the full CPTAC BRCA tumor dataset (n = 122) using the ShuffleSplit module from the python scikit-learn package to ensure phosphosite variability was as evenly distributed as possible.To demonstrate the utility of PhosphoDisco to analyze proteomic and phosphoproteomic data, we provide two case studies, the first from a breast cancer cohort   and the A). There were several sets of labels from the hdbscan algorithm . To identify the most reproducible modules across hyperparameter settings, we determined how similar each set of labels is to each other, as measured by the adjusted rand index, which is a measure of the similarity between sets of labels A. There lgorithm  that werFig. 2A, .i.e., triple-negative breast cancer, TNBC) have the worst prognosis and the most aggressive suggested treatment , progesterone receptor (PR), and the human epidermal growth factor receptor (HER2). Patients with tumors presenting these receptors can receive therapy that prevents receptor signaling and downstream proliferation. Tumors lacking all three receptors  (A).The majority of TNBCs are in the Basal subtype (50–75%), and the majority of Basal tumors are TNBC (>90%). Target discovery is especially important for patients with Basal subtype tumors, as they usually have poor prognosis and dearth of viable targets. As such, we first set out to prioritize modules that may represent key biological pathways active in this highest risk subtypes. To do this, we calculated an adjusted h module  identify 1.8e-7) A.Fig. 3MB). Furthermore, the phosphosites in this module significantly overlapped with known CDK1 and CDK2 targets  (C), though the majority of the sites were not contained in the CDK1 or CDK2 substrate lists from PhosphoSitePlus (E). When looking for regulators of the module, we identified a strong correlation between the module score and CDK1 abundance (D), as well as other mitotic kinases such as TTK, AURKB and MASTL (D). Moreover, this module contains compelling novel targets of CDK1 and/or CDK2, such as serine 183 on EIF3H, a site whose phosphorylation has been shown to be functionally necessary for oncogenic proliferation  , 44  C, thoughSitePlus E. When lbundance D, as welnd MASTL D. Moreovferation , potentiA and supplemental Fig. S5A), the cell cycle checkpoint module is secondarily high in Her2e samples, while the proliferation module is also high in LumB samples (B and C). This discrepancy indicates that proliferation and cell cycle checkpoints are separable pathways at the phosphorylation signaling level. The cell cycle checkpoint module is of particular interest, because CDK4/6 inhibitors (CDK4/6i) are currently being used to treat ER+ breast cancer  is separate from a module containing several KI67 phosphosites (module BRCA-48) , which iFig. 3t cancer , but thet cancer , 52. Sint cancer , the phoA) between the original BRCA-63 module and its most similar cluster in each of the runs.To assess PhosphoDisco’s ability to produce consistent modules with smaller sample size inputs, we ran 15 randomly selected patient cohorts at four different cohort sizes , resulting in 60 total runs. We then compared the modules identified by this subsampling to determine if smaller patient cohorts were able recapitulate the same modules that we found in our 122-patient cohort set, with a particular focus on BRCA-63 as a test case. Using the HBDSCAN-min-4 clustering method, the same method applied in our original analysis, we found that PhosphoDisco performed exceptionally well regardless of sample size. Comparing phosphosite overlap for module BRCA-63 and modules identified in our subsampled analysis, we found an average phosphosite overlap of 86.65%, 96.20%, 97.99%, and 98.44% for n = 25, 50, 75, and 100 patient cohorts, respectively A betweenB). This suggests that n = 25 patients may be sufficient to find the majority of the important phosphosites for key modules within your data, however additional patients can substantially increase consistency across runs. We suggest that, when feasible, users use a similar bootstrapping method to test the robustness of their modules based on a random down sampling approach.Although the high performance for the n = 25 patient cohorts suggests that we can confidently identify modules from cohorts of 25 patients or more, we did note higher module variability as sample size decreased. Using an adjusted rand index for which BRCA-63 phosphosites were clustered across the 15 runs at each cohort size level we found that n = 25 had the lowest adjusted rand index at 0.06. This jumps considerably to 0.80 for n = 50 patients and is a perfect score of 1.0 for n = 75 and n = 100 patients B. This sTo demonstrate PhosphoDisco’s functionality in the analysis of combined datasets, we combined tumor and normal samples from a lung squamous cell carcinoma (LSCC)  and lungPhosphoproteomics data from LSCC tumors, LSCC normals, LUAD tumors, and LUAD normals were normalized separately and combined following the procedure outlined in B). As seen previously (A), there were several sets of labels from the hdbscan algorithm .To find the most reproducible modules across hyperparameter settings, we determined how similar each set of labels are to each other, as measured by the adjusted rand index B. As seeeviously A, there lgorithm  that werA). These phosphosites included PRKC isoforms B, D, E, Q, all of which have been associated with reduced drug sensitivity, increased cell survival, proliferation, invasion, migration, evasion of apoptosis, anchorage-independent growth, progression, chemotaxis, as well as cell cycle progression  (B and C), where the CDK12 site is part of the module, and the CDK13 site is not. Additional potential regulators included PDGFA, PRKCD, PRKCZ, and PRKRA, indicating a potential role for ceramide metabolism (B and C). Further, we found that the Lung-3 module scores were significantly correlated with a mutation in the platelet-derived growth factor receptor alpha gene PDGFRA (adj-p = 0.036) (C). This gene has been shown to play a role in wound healing and tumor progression in a variety of cancers , which would be consistent with these phosphosites being phosphorylated by the same kinase or family of kinases. Interestingly, we found a highly similar module (p < 2e-10) in our breast cohort (BRCA-38), containing phosphosites in PRKCB and PRKCD as well as CDK12 and CDK13 (C and D).Expression of PRKC isoforms has been associated with poor prognosis in non-small cell lung cancer (NSCLC) . In our Fig. 4gression  making ttabolism  in this = 0.036) C. This g cancers . In summdentical B, which nd CDK13 , C and Dp-values < 0.0002) . Lung-5 module scores were found to have high correlation with CDK1 and CDK2 expression (B and C) and were found to contain a significant number of known CDK1 and CDK2 phosphosites, as determined by PTM-SEA . Further evidence for CDK1 and CDK2 as regulators for this module, is the S/T-P-x-K peptide motif enriched in the phosphosites of this module , which is consistent with the known CDK1 motif    in five protein complexes  based on the NURSA protein complexes database, a dataset made up of protein-protein complex associations that were identified by protein in complex recovered using Immunoprecipitation-MS . RANBP2,p < 6e-14) with the breast cancer module BRCA-63  further highlighting its ability to assists in processing and prioritizing phosphoproteomics data to generate hypotheses about biomarkers and novel therapeutic targets.While certain signaling pathways are well-established targets for cancer treatment, determining relevant players in a particular disease is challenging. We show that PhosphoDisco can help users identify and interpret co-regulated signaling modules in phosphoproteomic datasets. Applying PhosphoDisco to a breast cancer data set, we identified a putative cell cycle checkpoint module, which nominated novel targets of cell cycle kinases, drawing a potential link between cell cycle checkpoints and translation initiation control. In addition, the members of this module are good candidates to be tested as biomarkers of resistance to CDK4/6 inhibition. Lastly, PhosphoDisco was able to identify modules associated with mutation status and other relevant clinical features in lung cancer C furtherResults from a preliminary pan-cancer PhosphoDisco analysis hint that phosphorylation modules found across cancers are more similar to each other than to current databases of kinase-substrate sets. The reproducibility of some of these modules increases the confidence in the approach taken by PhosphoDisco. The signaling pathways repeatedly found in CPTAC phosphoproteomics data could redefine key phosphorylation cascades, narrowing down the scope of downstream validation experiments across cancer biology. In particular, PhosphoDisco can be used for the design of combination kinase inhibitor therapy, where it can identify kinases likely to be in the same or different pathways. This can guide experiments testing whether inhibition of two kinases would be additive or redundant, or simply be effective in different populations , 62, 63.Since individual phosphosites are typically only quantified based on the measurement of one peptide, we expect variation in measurements to be larger for phosphosite level analysis compared with protein quantification, where often several peptides can be used. Further, phosphorylation data is very high in its level of missingness. These inherent limitations require tools specific to this data structure, and PhosphoDisco was designed to support both the larger quantitative error rate and the missingness in phosphorylation data. The sparsity of this data type also complicates our ability to comprehensively assess ‘gold standard’ kinase-substrate sets in our data but we expect that this will improve as mass spectrometry-based proteomics technology continues to advance.Each function provided by PhosphoDisco could be useful as steps in other custom analysis pipelines. We particularly anticipate the protein normalized phosphopeptide abundance to be of broad use for other analyses, where determining whether differences in abundance are due to phosphorylation or parent protein expression changes. While this tool is currently most suited to data sets from CPTAC and similar consortia, there are increasing numbers of research programs that are using phospho-enriched shotgun mass spectrometry to study different systems . Phosphohttps://gdc.cancer.gov/about-gdc/contributed-genomic-data-cancer-research/clinical-proteomic-tumor-analysis-consortium-cptac).All data is available in public CPTAC repositories (This article contains The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.