Background: The commercially available 10x Genomics protocol to generate droplet-based single cell RNA-seq (scRNA-seq) data is enjoying growing popularity among researchers. Fundamental to the analysis of such scRNA-seq data is the ability to cluster similar or same cells into non-overlapping groups. Many competing methods have been proposed for this task, but there is currently little guidance with regards to which method to use.Methods: Here we use one gold standard 10x Genomics dataset, generated from the mixture of three cell lines, as well as multiple silver standard 10x Genomics datasets generated from peripheral blood mononuclear cells to examine not only the accuracy but also running time and robustness of a dozen methods.Results: We found that Seurat outperformed other methods, although performance seems to be dependent on many factors, including the complexity of the studied system. Furthermore, we found that solutions produced by different methods have little in common with each other.Conclusions: In light of this we conclude that the choice of clustering tool crucially determines interpretation of scRNA-seq data generated by 10x Genomics. Hence practitioners and consumers should remain vigilant about the outcome of 10x Genomics scRNA-seq analysis. This is achieved using computational methods for scRNA-seq that characterize cells into novel and known cell types. Characterization consists of two steps: (i) unsupervised or semi-supervised clustering of same or similar cells into non-overlapping groups, and (ii) labeling clusters, i.e. determining the cell type, or related cell types, represented by the cluster. Here, we focus on the first step of this process.Single-cell RNA-sequencing (scRNA-seq) studies have opened the way for new data-driven definitions of cell identity and function. No longer is a cell’s type determined by arbitrary hierarchies and their respective predefined markers. Instead, a cell’s transcriptional and epigenomic profile can now be used2. Due to the relative youth of the field, there are currently no rules guiding the application of these clustering algorithms. If tools’ performances have been tested outside synthetic scenarios, testing seems to be confined to scenarios with limited biological variability. Furthermore, most tools were developed and consequently tested only on the Fluidigm C1 protocol, despite considerable differences in throughput capabilities and sensitivities3 in the different scRNA-seq platforms. Here we focus solely on clustering performance on medium-sized scRNA-seq data generated by 10x Genomics as it is currently the most widely used platform. Commercially available scRNA-seq platforms, like 10x Genomics’ Chromium, are being widely adopted due to their ease of use and relatively low cost per cell4. The 10x Genomics protocol uses a droplet-based system to isolate single cells. Each droplet contains all the necessary reagents for cell lysis, barcoding, reverse transcription and molecular tagging. This is followed by pooled PCR amplification and 3’ library preparation, after which standard Illumina short-read sequencing can be applied5. Unlike other commercially available scRNA-seq protocols, like Fluidigm C1, 10x Genomics allows for sequencing of thousands of cells albeit at much shallower read depths per cell, and without allowing the use of fluorescence markers to establish cell identity. As such the 10x Genomics platform is particularly suited to detailed characterization of heterogeneous tissues.Research into clustering has produced many algorithms for the task, including over 90 tools specifically designed for scRNA-seqCell Ranger. Since many methods are still being actively developed, we include assessment of program versions available in October 2017 and April 2018. Our evaluation comprised four core aspects: (i) accuracy of clustering solutions compared to a gold standard , (ii) performance of clustering methods using silver standard data , (iii) stability of clustering solutions, and (iv) miscellaneous characteristics, such as time and practicality.In this study, we performed comprehensive evaluation of a dozen clustering methods . We focuGold standard. Three human lung adenocarcinoma cell lines, HCC827, H1975 and H2228, were cultured separately6. The cell lines were obtained from ATCC and cultured in Roswell Park Memorial Institute 1640 medium with 10% fetal bovine serum  and 1% penicillin-streptomycin. The cells were grown independently at 37°C with 5% carbon dioxide until near 100% confluence. Before mixing cell lines, cells were dissociated into single-cell suspensions in FACS buffer , catalog number: 14190-144; Thermo Fisher Gibco) with 5% FBS , stained with propidium iodide  and 120,000 live cells were sorted for each cell line by FACS  to acquire an accurate equal mixture of live cells from the three cell lines. The resulting mixture was then processed by the Chromium Controller (10x Genomics) using single Cell 3’ Reagent Kit v2  (seeics) see. AfterwaSilver standard. We consider five fresh human peripheral blood mononuclear cells (PBMCs) scRNA-seq datasets to be the silver standard . Datasets 3 and 5 were generated with the latest instrument, the 10x Genomics Chromium Controller .tandard . All datFor Dataset 1, PBMCs were isolated from whole blood obtained through the Australian Red Cross Blood Service in the following manner. First, 50ml of blood was diluted using 50ml of PBS . We then added 30ml of Ficoll-Paque medium . We then centrifuged at room temperature for 20 minutes at 400 g and carefully removed the interface layer containing PBMCs, located between the top plasma layer and middle layer . To remove the supernatant, we further centrifuged at 400 g for 10 minutes at room temperature. This process was repeated to remove the contaminating Ficoll medium or platelets. Finally, cells were resuspended in 20ml of cell culture media with 5% FBS  and counted . The resulting mixture was then processed by the Chromium Controller (10x Genomics) using single Cell 3’ Reagent Kit v2 . Afterwards the library was sequenced using HiSeq2500 (Illumina) and V4 chemistry  with 101bp paired end reads. RTA  was used for base calling.Cell Ranger to align to the GRCh38 (version 90) genome annotation, de-duplicate, filter barcodes and quantify genes. Note that, Cell Ranger filters any barcode that contains less than 10% of the 99th percentile of total UMI counts per barcode, as these are considered to be barcodes associated with empty droplets. The barcode by design can take one of 737,000 different sequences that comprise a whitelist. This feature allows the performance of error correction when the observed barcode does not match any barcode on the whitelist due to sequencing error. Using the Bioconductor packagescater18 (version 1.6.3), we then removed low quality data from cells with low library size or low number of expressed gene transcripts. We also removed cells with a high mitochondrial read proportion as this can indicate apoptosis, also known as programmed cell death. Stressed cells undergoing apoptosis have an aberrant transcriptome profile in comparison to a living cell and have previously been acknowledged to adversely influence transcriptome studies14.For Datasets 1-3, we used the 10x Genomics software version 2.0.0,TENxPBMCData. However, preprocessing was conducted with a CellRanger modified version of GRCh38 (version90) genome annotation resulting in slightly different versions for Dataset 2 and 3, referred to as Dataset 2a and Dataset 3a.Preprocessed versions of Datasets 2-5 were available in the R packagewww.scRNA-tools.org2 in October 2017. We only considered methods with an R package that had sufficient documentation to enable easy installation and execution and had at least one preprint or publication associated with it. Note that for some of the R packages the primary focus is not clustering, but the package authors explicitly describe how their packages can be applied to achieve clustering of the scRNA-seq data. We also excluded any methods that required extensive prior information not provided in the package. We also excluded any methods that continually failed to run . This resulted in the evaluation of 12 methods  and silver standard datasets in evaluation 2 (R version 3.5.0), respectively. Marker genes were required for the analysis withscran, which we obtained by performing differential expression analyses on GSE86337 and an in-house dataset of isolated cell types in PBMCs21 for the gold standard and silver standard datasets, respectively. Furthermore, we also followed upstream data handling, such as filtering of genes and normalization, as described in the documentation of the respective clustering method. We concede that it is possible that more care in the upstream data handling and selection of parameters could result in different results. However, confronted with the extremely large number of parameter choices, we believe that this evaluation suffices to identify strengths and weaknesses of each method.The aim of this study is to provide guidance for the use of clustering methods to non-experts. Hence, we used all clustering methods with their default parameters as this represents the most common use case. In the case of22 and the normalized mutual information (NMI)23, two metrics routinely applied in the field of clustering, to assess the similarity of clustering solutions or their similarity to a known truth. Both metrics can take values from 0 to 1, with 0 signifying no overlap between two groupings and 1 signifying complete overlap. These metrics are also applicable in the absence of known cluster labels. Furthermore, they share the following advantages: bounded ranges, no assumptions regarding cluster structures and symmetry.To evaluate the similarity of different clustering solutions, we rely on two different metrics. We use the adjusted Rand index (ARI)24. The homogeneity score takes the value 1 when all of its clusters contain only data points that are members of a single known group. Values of this score closer to 0 indicate that clusters contain mixed known groups. Unlike ARI, this score does not penalize members of a single group being split into several clusters and thus serves as a complimentary score to the ARI. Furthermore, bounded ranges and no assumptions regarding cluster structures are properties of both the ARI with regards to ground truth and the homogeneity score.To evaluate the performance of the different clustering methods with regards to an underlying truth, we use the ARI as well as a homogeneity scoreX be a finite set of sizen. A clustering solutionC is a setC1, . . . ,kC of non-empty disjoint subsets ofX such that their union equalsX. LetM = (ijm ) of the pair of setsC,C′ is ak ×l matrix whosei,j-th entry equals the number of elements in the intersection of clustersiC andLetARIt1 = whereNMIH(C ) =I is the entropy ofC. Note thatC andC‘.Homogeneity. Now let us assumeC′ is the known and correct grouping of the cells. Then,We evaluated accuracy, robustness and running time for all methods  in publicly available bulk RNA-seq of the same cell lines (GSE86337)25. Drawing on these SNVs, we then applydemuxlet26 (version 0.0.1), which harnesses the natural genetic variation between the cell lines to determine the most likely identity of each cell. We observe almost complete concordance between the result from demuxlet and clustering of cells seen in dimension reduction visualizations of the data  was performed with Datasets 1-3. The second evaluation (R version 3.5.0) was performed on Datasets 2-5, as these were available in the R packageTo test the robustness of different clustering methods we pursued a sampling strategy in terms cells. We also investigated the robustness of different methods with regards to different stringency of gene filtering. Finally, the impact of different aligners and preprocessing was assessed using all possible combinations of programs (i.e. some clustering methods did not run with scPipe output).Cells. In the first evaluation (R version 3.4.3) we used Dataset 3 for the robustness evaluation with regards to cells. We randomly sampled 3,000 cells in Dataset 3 , generating five (non-independent) datasets. For every combination of two datasets  we then investigated for each clustering method separately how often cells contained in all five sampled datasets were assigned to the same cluster using the ARI_comp. In the second evaluation (R version 3.5.0) we used Dataset 5. Here, we randomly sampled 4,000 cells , generating five (non-independent) datasets. We then repeated the evaluation procedure described above. We also investigated the variability of ARI_truth for all methods in both evaluations.Genes. Impact of gene filtering was only investigated for methods available in R version 3.5.0 during the second evaluation. We analyzed Dataset 4, as it had the most detected genes, with 10%, 20%, 30%, 40% and 50% of the most expressed genes . We investigated both the ARI_comp with regards to the clustering solution produced on a version of the dataset with no gene filtering, as well as the ARI_truth.Aligners and preprocessing pipelines. In order to assess the effect of using different preprocessing pipelines on the data, we applied the Bioconductor packagescPipe29 (version 1.0.6) to the raw data. Like Cell Ranger, scPipe can be used to align, de-duplicate, filter barcodes and quantify genes. Since scPipe is modular, we tried it with both theSTAR30 (version 020201) andSubread31 (version 1.5.2) aligners. In order to ensure comparability we aligned reads to the same GRCh38 genome annotation and repeated quality control with scater. We investigated the similarity of clustering solutions applied to the differently preprocessed and aligned versions of the same dataset by ARI_comp. Note that this was only done for the evaluation with methods available in R version 3.4.3.Each execution of a method on a dataset was performed in a separate R session. Each task was allocated as many CPU cores of a 24 core Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz as specified by the default parameters, but less than 10 cores. The base::set.seed was set for all steps involving stochasticity (i.e. dimension reduction and clustering). Timings for each method include any preprocessing steps.R2 of these models was used to assess which properties influenced the clustering solutions. Properties investigated included: (i) the total number of detected genes, (ii) the total read count, and (iii) the percentages of reads aligning respectively to ribosomal proteins, mitochondrial genes and ribosomal RNA (only Datasets 1–3).We also investigated what properties of each cell’s data were driving the clustering solutions produced by the different methods as well as the inferred cell labels. Properties of a cell’s data refer to features such as the number of total reads that included the cell’s barcode, the total number of detected genes found for this cell, etc. To this end, we used linear mixed models where cell data properties were predicted using the indicators for cluster membership. We predicted cell data properties and not cluster membership for modeling ease. The adjustedGold standard dataset. For the gold standard dataset consisting of three cell types, half of the tested clustering methods overestimated the true number of different cell types in the data. Methods with cluster number estimations close to the correct number of different cell types included methods with prior information, such asSIMLR,countClust andscran, as well asascend,Cell Ranger,RaceID andCIDR  filtering of cellsRaceID andRaceID2 did not appear very robust. Due to its reliance on reference profilesRCA is extremely robust, achieving ARI_comp above 0.9 consistently in both evaluations. In contrast, changes to gene filtering seemed to result in method specific effects, probably owing to individual filtering and normalization procedures. The performance ofSeurat improved dramatically with the inclusion of more genes, whereas it deteriorated forRaceID. In contrast, bothCell Ranger andSC3 exhibited stable performance when the percentage of highly expressed genes was varied.29 with Subread31, or STAR30. We found that different aligners largely result in the same gene counts, but with some notable exceptions for processed pseudogenes  than in evaluation 1 (R version 3.4.3) were reported forSeurat andSC3 . In combination with ribosomal protein genes being less affected by dropout due to their relatively high expression, it is perhaps unsurprising that clustering solutions of all methods foremost reflect differences in the amount of ribosomal protein genes between cells.The variation in the percentage of reads aligning to ribosomal protein genes strongly predicted all clustering solutions as well as the inferred cell labels see. ExpressTSCAN was particularly affected , but forRaceID2 similar effects were observed. It can be speculated that this strong influence of total number of features and total number of counts on their clustering solutions points to a failure to appropriately normalize the data.Most methods’ solutions were much more driven by the total number of detected genes and total number of counts than the inferred solution.Seurat provides the best clustering solutions for 10x Genomics scRNA-seq data in terms of running time, robustness and accuracy. The next best performing methods wereRCA,SC3,Cell Ranger andCIDR. However, it should be noted thatRCA performed particularly poorly on the gold standard dataset. This highlights thatRCA’s performance hinges on the studied cell types being represented in the reference used during the supervised clustering approach. These results closely mimic benchmarking results observed by Duòet al.33 on independent silver standard and simulated datasets across multiple single cell technologies.We also summarized the performance of each method across all evaluations see. This suet al.34, which systematically studied the effect of different similarity metrics on performance of scRNA-seq clustering methods, found that correlation-based similarity metrics outperformed distance-based metrics.We also investigated whether properties of the clustering method correlated with their performance. We found that neither the type of clustering method used nor the similarity metric used seemed to correlate with the performance. However, our ability to identify patterns might have been impacted by the small sample size. A recent paper by Kimwww.omicstools.org andwww.scRNA-tools.org2, many of the tools developed for scRNA-seq specifically focus on clustering. With so many choices, it is thus important to evaluate their performance for droplet based protocols, such as 10x Genomics, specifically.Most biological conclusions obtained from droplet-based scRNA-seq data crucially rely on accurate clustering of cells into homogeneous groups. Indeed, one can argue that it is the very act of clustering that unlocks the technology’s potential for discovery. Therefore it is not surprising that according to several repositories, such as• Inclusion of methods limited to R packages and methods published before October 2017• Parameter selection limited to defaults• No assessment of robustness to noise and parameter changes• No assessment of ability to discover rare cell populations• Evaluation of more silver standard datasets from systems other than PBMCs• No evaluation of ability to deal with batch effects or other more complex designs• No evaluation of quality of code and documentation• No assessment of scalability of methodsIn this study, we presented our evaluation of a dozen clustering method on scRNA-seq 10x Genomics data. The results of our investigations will be useful for method users, as we provide practical guidelines. Nonetheless, our evaluation has several limitations:Seurat performed slightly better than the next best methods, in our opinion, the choice of clustering method should be informed by the user’s familiarity with statistical concepts and R programming. Many methods, includingSeurat, require the user to make informed parameter choices and occasionally troubleshoot code. Methods requiring no parameter choices, likeCell Ranger, may offer a better choice for non-experts.WhileIn general, we recommend that practitioners and consumers of results generated from 10x Genomics scRNA-seq data alike remain vigilant about the outcome of their analysis, and acknowledge the variability and likelihood of undesired influences. The choice of clustering tool for scRNA-seq data generated by the 10x Genomics platform crucially determines interpretation. Hence, we suggest using several clustering methods ideally with multiple parameter choices on 10x Genomics scRNA-seq data in order to ensure that biological results are not artifacts of method or parameter choice. This should help guard against subjective interpretation of the data and thus increase robustness of and confidence in results.Repository: Gold Standard Dataset. Single cell profiling of 3 Human Lung Adenocarcinoma cell lines, GSE111108 Repository: Silver Standard Dataset 1. Single cell profiling of peripheral blood mononuclear cells from healthy human donor, GSE115189https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.0.0/pbmc3k, Version 1.1.0:https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3kRepository: Silver Standard Dataset 2. 3k PBMCs from a Healthy Donor, Version 1.0.0:https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.2.0/pbmc4k, Version 2.1.0https://support.10xgenomics.com/single-cell-gene-expression/datasets/2.1.0/pbmc4kRepository: Silver Standard Dataset 3. 4k PBMCs from a Healthy Donor, Version 1.2.0https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc6kRepository: Silver Standard Dataset 4. 6k PBMCs from a Healthy Donor,https://support.10xgenomics.com/single-cell-gene-expression/datasets/2.1.0/pbmc8kRepository: Silver Standard Dataset 5. 8k PBMCs from a Healthy Donor,https://github.com/bahlolab/cluster_benchmark_dataWe also provide versions in the R Single-CellExperiment format of all datasets atAll code is available for download at:https://github.com/SaskiaFreytag/cluster_benchmarking_code.Archived code at time of publication: 10.5281/zenodo.2008645License: MIT LicenseWritten informed consent for publication of the participant’s transcriptomic information was obtained . The authors have provided excellent responses to reviewer comments, leading to a more comprehensive and useful manuscript.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thank you to the authors for their thoughtful responses. I appreciated the detail in version 2 of the manuscript.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Freytag et al. have produced a nice research article on assessing methods for clustering scRNA-seq data from the 10x Genomics platform. I was excited to read the article to learn about what they recommend using. I have made some suggestions below for improvements that are mostly related to providing more intuition and higher-level summaries. This is mostly because as a user of these methods, at the end of the paper, I still felt a little confused about which method the authors would recommend using. I hope the authors can update the article with some of the suggestions: While the authors have provided detailed comparisons , the biggest suggestion would be that the authors provide a higher-level summary of what the authors would suggest a user use to cluster his/her data. At the end of reading this paper, I felt a little overwhelmed at the amount of comparisons across various datasets. It's hard to look at Figs 1-7 and get an overall summary of which method to use. The authors do state in the abstract "We found that some methods, including Seurat and Cell Ranger, outperform other methods, although performance seems to be dependent on the complexity of the studied system", but it would be great if the authors could somehow provide a visual high-level summary of how they came to that conclusion, or elaborate in the discussion on that.  For the "gold standard" data, what was the percent of each human lung cell lines  that were mixed together? Equal proportions? Was the reason you needed to use demuxlet was because the cell lines were mixed up for sequencing? It would be great if the authors could elaborate on the experimental design.  Is the "gold standard" data available with the SNVs called for each cell. It would be useful to have this count matrix and corresponding phenotypic information about each cell in a SingleCellExperiment object for others to have access to.  It would be great if the authors could include another example dataset with a batch effect in it or something with a slightly less clean design, given most datasets are not quite this "clean". Also, maybe different clustering methods would perform better / worse depending on they data contained rare vs common cell types or included more or less diversity.  There is a TENxPBMCsData package (https://github.com/kasperdanielhansen/TENxPBMCData) that has been submitted to Bioconductor (similar to the TENxBrainData). This includes all PBMC 10X datasets currently listed on their site and loads in a SingleCellExperiment object into R. For the Silver Standard Datasets, you might incorporate this into your workflow.  How did you (or Cell Ranger) deal with empty droplets or swapped barcodes on the 10x platform? This seems relevant for discovering cell types using some form of clustering. Supplemental Table 1 could use a caption and a label at the top saying "Supplemental Table 1". I had many tabs open with different supplemental figures and tables, and was getting confused about which was which one.  Why did Linnorm and Monocle "continually failed to run"? Did the authors contact the original authors of Linnorm and Monocle to determine if there was a problem with the actual software or if it was a problem with the implementation of the software? It would be great if the authors could elaborate.  I agree with this statement: " We concede that it is possible that more care in the upstream data handling and selection of parameters could result in different results." This is true for almost all benchmarking papers. Given the authors are working within the R/Bioconductor framework, it would be great if the authors could use something like SummarizedBenchmark (http://bioconductor.org/packages/release/bioc/vignettes/SummarizedBenchmark/inst/doc/SummarizedBenchmark.html) to keep track of these parameters.  Could the authors elaborate on how they decided which performance metrics to use?  What does this mean: "The impact of different aligners and preprocessing was assessed using all appropriate combinations of programs"? Could the authors be more specific? I'm a little concerned about how much the solutions differ between methods and parameter choices. I understand the point of this paper is to make comparisons between already published methods, but as the authors are now very familiar with these methods, it would be great if they could provide some more practical guidance. What would the authors suggest using?  Fig 1 -- Could the authors hypothesize on why Seurat, TSCAN, RCA, SC3, RaceID, RaceID2 are estimating so many clusters? Also, why does countClust tend to underestimate the number of clusters? It would be great if the authors could provide some intuition.  Fig 3 -- If I'm understanding, ascend and countClust produce clusters that are very different than the rest?  Thank you to the authors for making their code publicly available!I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We would like to thank the reviewer for reviewing our manuscript and for their constructive comments. Below are point-by-point responses to the individual comments. While the authors have provided detailed comparisons , the biggest suggestion would be that the authors provide a higher-level summary of what the authors would suggest a user use to cluster his/her data. At the end of reading this paper, I felt a little overwhelmed at the amount of comparisons across various datasets. It's hard to look at Figs 1-7 and get an overall summary of which method to use. The authors do state in the abstract "We found that some methods, including Seurat and Cell Ranger, outperform other methods, although performance seems to be dependent on the complexity of the studied system", but it would be great if the authors could somehow provide a visual high-level summary of how they came to that conclusion, or elaborate in the discussion on that. We have added a discussion section in which we summarize the results across all evaluations. This discussion section includes a visual high-level summary . For the "gold standard" data, what was the percent of each human lung cell lines  that were mixed together? Equal proportions? Was the reason you needed to use demuxlet was because the cell lines were mixed up for sequencing? It would be great if the authors could elaborate on the experimental design. We mixed the cell lines in equal proportions. Due to using 10x Genomics technology, the cell lines were mixed up in the process but could be deconvoluted using demuxlet (ref?). We have elaborated on this further in the manuscript to clarify the experimental design. Is the "gold standard" data available with the SNVs called for each cell. It would be useful to have this count matrix and corresponding phenotypic information about each cell in a SingleCellExperiment object for others to have access to. We have made all datasets as SingleCellExperiment objects, including their phenotypic information, available on Github at https://github.com/bahlolab/cluster_benchmark_data . We have added information regarding the availability of all processed datasets to the manuscript. It would be great if the authors could include another example dataset with a batch effect in it or something with a slightly less clean design, given most datasets are not quite this "clean". Also, maybe different clustering methods would perform better / worse depending on they data contained rare vs common cell types or included more or less diversity. We agree that investigating the performance of clustering approaches on ”messy” scRNA-seq designs would be very interesting. However, this is beyond the scope of this paper, as it requires the application of sophisticated batch correction methods. Such methods should generally be performed by experts rather than beginners, who were the target audience of this paper. We have added a discussion to this effect.Finally, in order to investigate whether methods perform better or worse in more or less diverse situations, one requires either simulations or mixture experiments. These were beyond the scope of this paper. However, we now refer the readers of our manuscript to the recent benchmarking study of scRNA-seq clustering methods by Duó et al, which investigates just such a scenario. For most methods they did not observe overt differences. There is a TENxPBMCsData package (https://github.com/kasperdanielhansen/TENxPBMCData) that has been submitted to Bioconductor (similar to the TENxBrainData). This includes all PBMC 10X datasets currently listed on their site and loads in a SingleCellExperiment object into R. For the Silver Standard Datasets, you might incorporate this into your workflow. We decided to incorporate all moderately large fresh PBMC samples included in the TENxPBMCs into our workflow. This also provided us with an opportunity to update the package versions for the individual clustering tools for our silver standard benchmarking and stability analyses.  How did you (or Cell Ranger) deal with empty droplets or swapped barcodes on the 10x platform? This seems relevant for discovering cell types using some form of clustering.We added the following explanation: “Cell Ranger filters any barcode that contains less than 10% of the 99th percentile of total UMI counts per barcode, as these are considered to be barcodes associated with empty droplets. The barcode by design can take one of 737,000 different sequences that comprise a whitelist. This feature allows the performance of error correction when the observed barcode does not match any barcode on the whitelist due to sequencing error.” Supplemental Table 1 could use a caption and a label at the top saying "Supplemental Table 1". I had many tabs open with different supplemental figures and tables, and was getting confused about which was which one. We added a caption on the top of all Supplemental Tables. Why did Linnorm and Monocle "continually failed to run"? Did the authors contact the original authors of Linnorm and Monocle to determine if there was a problem with the actual software or if it was a problem with the implementation of the software? It would be great if the authors could elaborate. Linnorm failed because its calculations would time out. Monocle failed because the dispersion could not be calculated. However, neither of the programs was tried using their newer package versions corresponding to R version 3.5.0 nor were any of the packages’ authors contacted. We have included a statement in the manuscript to this regard. I agree with this statement: " We concede that it is possible that more care in the upstream data handling and selection of parameters could result in different results." This is true for almost all benchmarking papers. Given the authors are working within the R/Bioconductor framework, it would be great if the authors could use something like SummarizedBenchmark (http://bioconductor.org/packages/release/bioc/vignettes/SummarizedBenchmark/inst/doc/SummarizedBenchmark.html) to keep track of these parameters. We did take a look at the SummarizedBenchmark package, but did not find it suitable for our needs. However, we understand the need to provide all parameters (including defaults) used in the individual analyses and thus have added additional files providing this information to the GitHub respository.  Could the authors elaborate on how they decided which performance metrics to use? We used performance metrics commonly used in the clustering literature. We also made sure that the selected metrics were applicable in the absence of known cluster labels. Furthermore, they share the advantages of bounded ranges and no assumptions regarding cluster structures. Additionally they offer complementary insights. We have added this explanation to the manuscript. What does this mean: "The impact of different aligners and preprocessing was assessed using all appropriate combinations of programs"? Could the authors be more specific?We meant to say that we assessed the impact of combinations of different aligners and preprocessing (i.e. CellRanger or scPipe) for all possible clustering methods. Some clustering methods, like ascend, failed to run for scPipe generated output and it was too challenging to run the CellRanger clustering approach on scPipe generated output. I'm a little concerned about how much the solutions differ between methods and parameter choices. I understand the point of this paper is to make comparisons between already published methods, but as the authors are now very familiar with these methods, it would be great if they could provide some more practical guidance. What would the authors suggest using? We suggest using several clustering methods ideally with multiple parameter choices in order to ensure that biological results are not artifacts of method or parameter choice. Unfortunately, we do not feel in a position to give specific practical advice for the specific use of individual methods, as optimal parameter choices depend on many different factors including the type of biological system studied. Fig 1 -- Could the authors hypothesize on why Seurat, TSCAN, RCA, SC3, RaceID, RaceID2 are estimating so many clusters? Also, why does countClust tend to underestimate the number of clusters? It would be great if the authors could provide some intuition. We believe that many methods tended to overestimate the number of clusters in the gold standard dataset, because the cell lines may be heterogeneous with regards to other biological factors, such as cell state. Consequently, in such a scenario methods may split cells of the same population but in different cell states into multiple clusters.We have no intuition as to why countClust underestimates the number of clusters. Fig 3 -- If I'm understanding, ascend and countClust produce clusters that are very different than the rest? Yes that is correct. Freytag and colleagues provide a comprehensive comparison of clustering methods - specifically designed for scRNA-Seq data - on data collected using the popular droplet-based 10x Genomics platform. A total of four datasets, comprising a Gold standard mixture of cell lines as well as three Silver standard PBMC datasets, were compared in terms of accuracy, stability as well as other metrics like runtime and ease of use. Freytag et al also perform an analysis to try to determine the factors influencing the resulting clusterings for the Silver standard datasets. It is a very challenging task to perform a comprehensive characterization and comparison of clustering methods on such types of high-dimensional data, due to the sheer number of choices that need to be made, the difficulty in establishing ideal performance, and the relative lack of ground truth. Freytag et al do a great job of addressing these challenges and working towards providing an overall recommendation of clustering methods for non-expert practitioners, while stressing the need for careful interpretation of such results.  With this in mind, I have some comments/suggestions, as well as a number of minor comments/suggestions, as follows: **Comments to authors** Linnorm and Monocle failed - expand on why? I understand that this is indeed a limitation especially for a non-expert practitioner, but it would be good to have an understanding towards what the issue might have been. Could use a flowchart to summarise the study and various comparisons, as well which methods could no longer be compared (e.g. methods that could not work within the scPipe framework). Different upstream data handling was performed for each clustering method. How much of a difference was observed just due to this preprocessing, as opposed to the actual clustering step? I understand that each method provides their own preprocessing as *part* of the method, but at least some of these methods would have been developed with plate-based and/or non-UMI-based scRNA-Seq in mind, so may not be intended for the context of 10x Genomics data. Again I understand that you're comparing methods 'out of the box' but it would be insightful to see what differences there are. I suggest a figure like an upsetR plot for the genes/cells filtered and a correlation heatmap of the expression values themselves. Could you summarise the distance metrics used in the clustering and if there is a general flavour to the clustering algorithm? e.g. hierarchical, k-means, density-based etc. How do these relate in terms of overall accuracy, stability and other metrics? Stability assessment - mentions that half of the 58,302 genes were randomly selected, but Table 1 says 24,654 total genes detected. There's a big discrepancy between these two so please clarify; if half of the 58,302 genes were selected then a large proportion of genes would have identically zero rows. Also Table 1 shows Dataset 3 had the highest number of 'total genes detected', so how was Dataset 1 the one with "most number of non-zero genes after filtering"? Run time section - What do you mean by 'overridden'? And for which aspects of the analysis steps was this done? Figure 4 - These boxplots show ARI among multiple clustering solutions, so a method that gives a consistently bad result is still high (e.g. in this case the RCA method). Suggest an analogous set of boxplots but with ARI_truth, is there a similar variability observed, as seen in these boxplots? Gene-wise stability analysis - I'm actually unsure how realistic this particular comparison is. It would be insightful to assess clusterings depending on different levels of gene filtering stringency , or stringency on selection of features based on various criteria like highly variable genes. Figure 7 - Please clarify how 'total number of features' is a cell-specific quantity. Do you mean total number of non-zero features? Was this analysis also performed on the Gold Dataset and what overall similarities could be observed? Factors influencing clustering solutions - It would be interesting to consider the factors associated with 'correct' cluster assignment for cells. Optionally suggest to perform this for either the Gold Dataset or the Silver datasets and perform a logistic regression with the response being success/failure of a cell to belong to the cluster most associated with the 'true' cell type group. There is an added subtlety as far as matching clusters with cell type groups goes, but I think there are a few reasonable ways to perform this . Performing this kind of analysis could shed light on properties of cells that don't tend to cluster correctly, and if there is consistency in this across multiple disparate datasets. **Minor comments** Table 1 - countClust 'version' formatted with verbatim. Table 1 - I would suggest the 'properties' column could be better presented in a checklist format, with ticks/crosses for fulfilling various criteria listed. Section beginning "silver standard" - 10x is capitalised. Supplementary Figure 1 - legend fallen off panel a), needs a higher resolution or larger points NMI definition - trailing parenthesis in denominator typo - assess the effect** Figure 2a - I found this quite busy, hard to interpret. Suggest to add shading that covers the points for same method or to facet by dataset. I don't believe the ARI values are particularly comparable between datasets so I would prefer facetting by dataset. Figure 3 - rows/columns are ordered differently between panels, what's driving this difference? Supplementary Figure 3 was not mentioned in the main text Supplementary Figure 4 is a two page pdf, with the first page blank Figure 6 - Figure caption says Dataset 1 but reports 29,151 genes. Do you mean the Gold Dataset and 29,451 genes? If not, please clarify which data and how many genes. Discussion - One instance of "Seurat" is missing verbatim formatI have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We would like to thank the reviewer for reviewing our manuscript and for their constructive comments. Below are point-by-point responses to the individual comments. Linnorm and Monocle failed - expand on why? I understand that this is indeed a limitation especially for a non-expert practitioner, but it would be good to have an understanding towards what the issue might have been.Linnorm failed because its calculations would time out. Monocle failed because the dispersion could not be calculated. However, neither of the programs was tried using their newer package versions corresponding to R version 3.5.0. We have included a statement in the manuscript to this regard. Could use a flowchart to summarize the study and various comparisons, as well which methods could no longer be compared (e.g. methods that could not work within the scPipe framework).While we were unable to summarize our study design effectively in a flowchart, we have summarized it in a table (see Supplementary Table 2). We hope that this will clarify the various assessments performed in this paper. Different upstream data handling was performed for each clustering method. How much of a difference was observed just due to this preprocessing, as opposed to the actual clustering step? I understand that each method provides their own preprocessing as *part* of the method, but at least some of these methods would have been developed with plate-based and/or non-UMI-based scRNA-Seq in mind, so may not be intended for the context of 10x Genomics data. Again I understand that you're comparing methods 'out of the box' but it would be insightful to see what differences there are. I suggest a figure like an upsetR plot for the genes/cells filtered and a correlation heatmap of the expression values themselves.We agree that different data handling influences the performance of each clustering method, which were indeed designed with different single cell technologies in mind, and that the effect of this would be interesting to further investigate. However, the `black-box` nature of some of the investigated methods means that even recording these differences is challenging. Take Seurat as an example it is unclear whether to report the number of genes passing the filtering step or the number of genes that are used in the clustering.  Instead, we would like to refer you to the recent benchmarking study of clustering methods for scRNA-seq by Duó et al, where the authors investigated the effects of different gene filtering on clustering solutions. Could you summarise the distance metrics used in the clustering and if there is a general flavour to the clustering algorithm? e.g. hierarchical, k-means, density-based etc. How do these relate in terms of overall accuracy, stability and other metrics?Thank you for the suggestion. We have updated the table summarizing the properties of the different clustering methods and added a discussion regarding how different flavors of clustering methods relate to overall performance (see Table 1 and Discussion). Stability assessment - mentions that half of the 58,302 genes were randomly selected, but Table 1 says 24,654 total genes detected. There's a big discrepancy between these two so please clarify; if half of the 58,302 genes were selected then a large proportion of genes would have identically zero rows. Also Table 1 shows Dataset 3 had the highest number of 'total genes detected', so how was Dataset 1 the one with "most number of non-zero genes after filtering"?You are correct. We randomly selected half of 58,302 genes of which many were zero. We have since replaced this analysis, as per your suggestion, with an analysis that assesses stability when keeping only the top 10th, 20th, 30th, 40th, and 50th percentile of all genes including the ones not detected. With regards to the number of detected genes in dataset 1 and dataset 3, indeed dataset 3 had more detected genes. Thank your for correcting this. Run time section - What do you mean by 'overridden'? And for which aspects of the analysis steps was this done?We meant to say that a seed had been set to provide reproducibility of all parts of the analysis that involve randomness. This has been corrected in the manuscript. Figure 4 - These boxplots show ARI among multiple clustering solutions, so a method that gives a consistently bad result is still high (e.g. in this case the RCA method). Suggest an analogous set of boxplots but with ARI_truth, is there a similar variability observed, as seen in these boxplots?Thank you for the suggestion, we have included a boxplot with ARI_truth. Gene-wise stability analysis - I'm actually unsure how realistic this particular comparison is. It would be insightful to assess clusterings depending on different levels of gene filtering stringency , or stringency on selection of features based on various criteria like highly variable genes.We have replaced the gene-wise stability analysis with an assessment of the performance when keeping only the top 10th, 20th, 30th, 40th, and 50th percentile of all genes . We think that this is more insightful as it is closer to filtering performed during analysis. Figure 7 - Please clarify how 'total number of features' is a cell-specific quantity. Do you mean total number of non-zero features? Was this analysis also performed on the Gold Dataset and what overall similarities could be observed?Indeed we do mean the number of non-zero genes and we have replaced this in the figure with “number of detected genes”. We also include the same analysis on the gold standard dataset in the Supplementary . Factors influencing clustering solutions - It would be interesting to consider the factors associated with 'correct' cluster assignment for cells. Optionally suggest to perform this for either the Gold Dataset or the Silver datasets and perform a logistic regression with the response being success/failure of a cell to belong to the cluster most associated with the 'true' cell type group. There is an added subtlety as far as matching clusters with cell type groups goes, but I think there are a few reasonable ways to perform this . Performing this kind of analysis could shed light on properties of cells that don't tend to cluster correctly, and if there is consistency in this across multiple disparate datasets.We did perform the suggested analysis. However, results from this analysis did not give any insights beyond the already conducted analysis (see https://github.com/SaskiaFreytag/cluster_benchmarking_code/tree/master/revision_figure). Hence, we chose not to include this in the manuscript.  Table 1 - countClust 'version' formatted with verbatim.Thank you for noticing, this has been corrected. Table 1 - I would suggest the 'properties' column could be better presented in a checklist format, with ticks/crosses for fulfilling various criteria listed.Table 1 is now Supplementary Table 1. Unfortunately, properties differ too much to adequately represent these in a checklist.  Section beginning "silver standard" - 10x is capitalised.Thank you for noticing, this has been corrected. Supplementary Figure 1 - legend fallen off panel a), needs a higher resolution or larger pointsWe have increased the resolution. NMI definition - trailing parenthesis in denominatorThank you for noticing, this has been corrected. typo - assess the effect**Thank you for noticing, this has been corrected. Figure 2a - I found this quite busy, hard to interpret. Suggest to add shading that covers the points for same method or to facet by dataset. I don't believe the ARI values are particularly comparable between datasets so I would prefer facetting by dataset.We agree with the reviewer and now use faceting. Figure 3 - rows/columns are ordered differently between panels, what's driving this difference?The difference by clustering on the similarity across methods, i.e. more similar methods are closer to each other. We have included a statement explaining this in the figure description. Supplementary Figure 3 was not mentioned in the main textWe now mention this Supplementary Figure. Supplementary Figure 4 is a two page pdf, with the first page blankWe have corrected this error. Figure 6 - Figure caption says Dataset 1 but reports 29,151 genes. Do you mean the Gold Dataset and 29,451 genes? If not, please clarify which data and how many genes.Note that this figure has been replaced. We indeed meant Dataset 1, but with only half the genes. Discussion - One instance of "Seurat" is missing verbatim formatThank you for noticing, this has been corrected. This paper presents a well-designed and comprehensive evaluation of widely used clustering algorithms for medium-sized 10x Genomics scRNA-seq data. Clustering is a highly active area of research in scRNA-seq data analysis. With so many published clustering tools available, it is often difficult to choose the most appropriate tool. This paper attempts to address this problem by systematically comparing the performance of 12 commonly used clustering tools. The evaluation results should serve as an important guide to bioinformatics practitioners. This paper is a very useful contribution to the field.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.


Drug-induced liver injury (DILI) is a serious concern during drug development and the treatment of human disease. The ability to accurately predict DILI risk could yield significant improvements in drug attrition rates during drug development, in drug withdrawal rates, and in treatment outcomes. In this paper, we outline our approach to predicting DILI risk using gene-expression data from Build 02 of the Connectivity Map (CMap) as part of the 2018 Critical Assessment of Massive Data Analysis CMap Drug Safety Challenge.First, we used seven classification algorithms independently to predict DILI based on gene-expression values for two cell lines. Similar to what other challenge participants observed, none of these algorithms predicted liver injury on a consistent basis with high accuracy. In an attempt to improve accuracy, we aggregated predictions for six of the algorithms  using a soft-voting method. This approach also failed to generalize well to the test set. We investigated alternative approaches—including a multi-sample normalization method, dimensionality-reduction techniques, a class-weighting scheme, and expanding the number of hyperparameter combinations used as inputs to the soft-voting method. We met limited success with each of these solutions.We conclude that alternative methods and/or datasets will be necessary to effectively predict DILI in patients based on RNA expression levels in cell lines.This article was reviewed by Paweł P Labaj and Aleksandra Gruca (both nominated by David P Kreil). Drug-induced liver injury (DILI) is a serious concern during both drug development and the treatment of human disease. DILI is characterized by elevated levels of alanine aminotransferase; in serious cases, it can ultimately result in acute liver failure and patient death . ReactivThe 2018 Critical Assessment of Massive Data Analysis (CAMDA) Connectivity Map (CMap) Drug Safety Challenge was held in conjunction with the Intelligent Systems for Molecular Biology conference in Chicago, Illinois. The challenge organizers instructed participants to train predictive models on gene-expression data from Build 02 of CMap . CMap waIn Phase I, we evaluated seven classification algorithms on the training data Fig. . In addiAfter submitting our predictions to the challenge organizers, we learned that our predictions performed worse than random-chance expectations. Thus, during the second phase of the challenge, we explored various options for improving classification accuracy, including different preprocessing methods, feature-selection and feature-transformation approaches, class weighting, and multiple hyperparameter combinations Fig. .During Phase I, we used cross validation to evaluate seven classification algorithms, multiple hyperparameter combinations for each of these algorithms, and a voting-based classifier that aggregated these individual classifiers. Table In Phase I, none of our solutions produced consistently accurate predictions on the test set Fig. . AccuracIn addition to providing class labels for the test set, the CAMDA organizers provided us with suggestions from reviewers. These suggestions gave us ideas for improving classification performance, which we evaluated in Phase II. Because we did not have an additional, independent dataset, our Phase II evaluations were only exploratory in nature. We explored four types of techniques for improving performance: a multi-sample normalization method and batch correction, feature scaling/selection/reduction techniques, custom class weights, and scaling of the voting-based ensemble method. To quantify the effects of these alternative approaches, we compared the performance of our classifiers with and without each change, averaged across all classification algorithms—with the exception of adjusting the class weights, which was only possible for a subset of the algorithms see . Figure In Phase I, we preprocessed the microarray array using the SCAN algorithm, a single-sample normalization method. We hypothesized that preprocessing the data using the FARMS algorithm  would result in improved performance by reducing technical variability across the samples via quantile normalization. In addition, because the CMap data had been processed in many batches, we hypothesized that correcting for batch effects using the ComBat algorithm would increase classification performance. In some cases, these changes improved predictive performance slightly, whereas in other cases the performance was reduced, irrespective of whether we used SCAN, FARMS, and/or batch adjustment Fig. a.Although microarray normalization methods help to remove technical biases and multi-sample corrections can remove inter-sample variations, some classification algorithms assume that each feature has been scaled to have the same mean and standard deviation. Accordingly, in Phase II, we used scikit-learn’s RobustScaler functionality to scale the expression data for each gene; this method also adjusts for any outliers that may exist. Secondly, we reduced the feature space via feature selection  and dimensionality reduction . These adjustments did not improve performance consistently Fig. b.In an attempt to mitigate the effects of class imbalance, we adjusted weights assigned to the class labels. By default, classification algorithms in scikit-learn place an equal weight on each class label, but many algorithms provide an option to adjust these weights. We attempted many different weight ratios, even placing 50 times more weight on the minority class than the majority class. These adjustments often improved sensitivity or specificity, but none of these changes resulted in a higher MCC value Fig. c.Finally, we made various attempts at improving the voting-based classifier. We used hard voting rather than soft voting. With this approach, the predictions for the individual classifiers are treated as discrete rather than probabilistic values, which may improve ensemble predictions in situations where probabilistic predictions are poorly calibrated. In addition, we increased the number of individual classifiers used for voting. We retained the same classification algorithms, but we included predictions for multiple hyperparameter combinations per algorithm. We suspected that a larger and more diverse set of predictions would improve voting performance. None of these approaches resulted in consistent improvements for any of the metrics except specificity Fig. d; these Our goal was to make progress toward accurately predicting DILI based on gene-expression profiles of cell lines. The ability to predict these outcomes could reduce patient injury, lower costs associated with drug development, and optimize treatment selection. As a step toward these objectives, we analyzed gene-expression levels from cancer cell lines that had been treated with small molecules; we used machine-learning classification to predict DILI. Our study design relied on the assumption that drugs causing liver injury induce transcriptional changes that are common across many or all of these drugs and that these transcriptional changes might also occur in liver tissue in vivo.In Phase I, we employed seven classification algorithms as well as a soft-voting ensemble classifier that aggregated predictions from six of the seven individual algorithms. On the training data, we observed relatively high performance for the Random Forests and Logistic Regression algorithms, which coincides to an extent with prior findings . HoweverWe emphasize the importance of considering multiple, diverse performance metrics when evaluating classification results. Even though our classification algorithms sometimes attained higher levels of accuracy on the test set than the training set Fig. a, these Additionally, we attempted to improve classification performance using a multi-sample normalization method, adjusting for batch effects, scaling features, selecting features, reducing data dimensionality, and using multiple hyperparameter combinations as input to the voting-based classifier. Although these techniques might have resulted in improvements in other classification scenarios, they resulted in minimal improvements, if any, in predictive ability in our analysis. The batch-effect correction method that we used (ComBat) requires the researcher to assign batch labels to each biological sample. Alternative tools such as PEER  and SVA The original goal of this CAMDA challenge was to predict hepatic injury from mRNA expression profiles. Our findings suggest that some or all of the following factors may explain our limited success in predicting these outcomes: 1) gene-expression microarray measurements are often noisy, 2) mRNA expression levels in cell lines may be inadequate surrogates for in vivo responses in this setting, 3) larger datasets may be needed, and 4) more sophisticated analytic techniques may be needed.https://portals.broadinstitute.org/cmap/). As part of the normalization process, we used BrainArray annotations to discard faulty probes and to summarize the values at the gene level (using Entrez Gene identifiers) [https://python.org) to summarize the data and execute analytical steps. The scripts we used to normalize and prepare the data can be found here: https://osf.io/v3qyg/.The training set was a subset of CMap consisting of gene-expression data and known DILI status for 190 small molecules (130 of which had been found to cause DILI in patients). The test set consisted of an additional 86 small molecules. The CMap gene-expression data were generated using Affymetrix gene-expression microarrays. In Phase I, we used the Single Channel Array Normalization (SCAN) algorithm —a singletifiers) . We wrotFor each treatment on each cell line, CMap provides gene-expression data for multiple biological replicates of vehicle-treated cells. For simplicity, we averaged gene-expression values across the multiple vehicle files. We then subtracted these values from the corresponding gene expression values for the compounds of interest. Finally, we merged the vehicle-adjusted data into separate files for MCF7 and PC3, respectively.The SCAN algorithm is designed for precision-medicine workflows in which biological samples may arrive serially and thus may need to be processed one sample at a time . This apInitially in Phase I, we used a variance-based approach for feature selection . We calculated the variance of the expression values for each gene across all samples; then we selected different quantities of genes that had the highest variance and used those as inputs to classification. However, in performing 10-fold cross validation on the training set, we observed no improvement in classification performance regardless of the number of high-variance genes that we used, so we decided not to use feature selection for our Phase I predictions. To perform cross-validation, we wrote custom Python code that utilizes the scikit-learn module (version 0.19.2), .In Phase II, we used the following scaling and feature-selection methods in an attempt to improve performance: robust scaling, feature selection based on the ANOVA F-value, and principal component analysis. We used scikit-learn implementations of these methods and used default hyperparameters .We performed classification using the following algorithms from the scikit-learn library: Gradient Boosting , LogistiIn Phase I, we sought to select optimal hyperparameter values for each algorithm via cross validation on the training set. We evaluated a variety of parameter combinations (combinations of model settings) for each algorithm, assessing each combination based on its effect on accuracy, specificity, sensitivity, and the Matthews Correlation Coefficient (MCC) . Each ofIn an attempt to optimize the performance of the voting-based classifier, we devised a weighting scheme, which assigned higher weights to individual algorithms that performed relatively well during cross validation; we also experimented with excluding individual classifiers from the voting-based classifier. The only approach that appeared to have a consistently positive effect on performance was to exclude the Gaussian Naïve Bayes algorithm, which had also performed poorly in isolation. Our final voting-based model in Phase I excluded Gaussian Naïve Bayes and assigned an equal weight to each individual classifier.n = 5), Support Vector Machines (n = 4), Logistic Regression (n = 2), Random Forests (n = 5), K-nearest Neighbor (n = 5), and Gradient Boosting classifiers (n = 3). We also investigated whether assigning weights to each class label would help overcome the effects of class imbalance and improve classification performance. Four of the classifiers from Phase I—Random Forests, Support Vector Machine, Logistic Regression, and the soft-voting ensemble method—support a class_weight hyperparameter, which allowed us to apply custom weights to each class label . Adjusting the class_weight hyperparameter required providing a weight for the non-DILI (weight_1) and DILI observations (weight_2), indicated here as weight_1:weight_2. We used class weights of 50:1, 25:1, 10:1, 5:1, 2:1, 1:1, and 1:2.In Phase II, we attempted to improve the voting-based classifier in multiple ways. First, rather than selecting a single hyperparameter combination for each algorithm and using those as input to the voting-based classifier, we used multiple hyperparameter combinations for each classification algorithm (except Gaussian Naïve Bayes). For this approach, we incorporated the following classification algorithms (with the number of distinct hyperparameter combinations): Multilayer Perceptron .Overall the manuscript is well written, however, reader can loose a track in both methods and results. Better structure complemented with a figure outlining the analysis procedure would improve readability and by this improve the quality of the manuscript.What is missing in the manuscript is deeper description of Ensemble approach with all pros and cons. This approach could be easily tricked if a few used methods have similar bases / are from close families of solution. Here it is not a case but should be pointed out and described. Connected to this is selection of used methods, just saying that these ones are available ‘scikit-learn library’ is not enough.Authors, in one of the improvements, have used ComBat for batch correction, but this will work only for known confounders. It would be interesting to see, or at least, comment the application of solutions which could detect also hidden confounders, like PEER or SVA.Figure presenting the overview of the analysis and all additions should be provided to improve readability. The additional comment to second point is that CMap is created when cell line has been treated with a specific dose, while DILI is based on meta-analysis of real patients data. One could expect that an important factor for DILI is whether the therapy was short time or prolonged as in the other even small toxicity can accumulate and lead to DILI. Of course the necessary data were not provided here, but it could be that therapy type factor could be detected as hidden confounder.The manuscript by G. Rex Sumsion et al. presents ‘An Ensemble Approach to Predicting Drug-induced Liver Injury Using Gene-Expression Profiles’. DILI prediction with use of single source of data (like expression microarrays) is extremely challenging what has been presented in the course of CAMDA conferences. Sometimes it is very valuable to obtain information what will not work and why. In this manuscript a multiple approaches has been tested as well as some ‘improvements’ suggested by CAMDA reviewers, but none is providing really good results. The proposed Ensemble approach is a good idea in such cases, however, I would expect better explanation when Ensemble approach might not work (more specific comments in next point).We thank the reviewer for taking the time to review our manuscript and for providing these comments.We have revised the text in the Methods and Results sections to make the manuscript easier to read. We have also revised the sub-section headings to facilitate better organization. In addition, we have added a figure that illustrates our workflow across the two phases of the CAMDA challenge.We modified the wording in the 3rd paragraph of the Introduction section to say the following: “Generally, voting approaches are most effective when they incorporate individual classifiers that perform reasonably well in isolation and when the component classifiers use diverse methodological approaches and thus are more likely to have deficiencies in different areas of the input space, often allowing for improved performance in aggregate. We hoped that this would hold true for predicting DILI in this study because the individual algorithms that we used represent diverse methodological approaches.” We also modified the Discussion section as follows: “The soft-voting approach yielded better performance than the individual algorithms at times, but this pattern was inconsistent. Voting-based approaches often outperform single-classifier approaches because they combine diverse algorithmic techniques—where one algorithm fails, other(s) may succeed. However, they rely on a diverse range of inputs; using algorithms from a narrow range of methodologies will generally be less performant.” In addition, we have provided an expanded table that shows which parameters we used for each algorithm.We added the following statement to the last paragraph of the Discussion section: “The batch-effect correction method that we used (ComBat) requires the researcher to assign batch labels to each biological sample. Alternative tools such as PEER and SVA can be used in situations where batch labels are unknown or more generally to detect other types of hidden variation.”In complement to the previous point, we have modified the Discussion to add the point that the reviewer mentioned: “… hidden factors—perhaps due to treatment duration and physiological complexity—–may have confounded this study. DILI was determined based on a meta-analysis of patient data, whereas our predictions were derived from treatments administered to cell lines over the course of only a few hours or days.”Authors’ response: Aleksandra Gruca, Silesian University of Technology .The authors analysed dataset from CAMDA 2018 DILI contest. The main goal of the contest is to accurately predict DILI risk of particular drug based on cell lines gene expression data. To achieve this, the authors try different parameter settings for data preprocessing and apply seven classification algorithms that are finally combined in an ensemble approach. The presented work is of a limited novelty. In general, data processing workflow is designed correctly and the analytical steps performed by the authors are typical for such kind of problems. I do not find any flaws in the proposed approach, although I also do not see any novelty in it. On the positive side I notice that the authors have tried several different combinations of methods and parameters in searching for the best outcome. However, none of the applied techniques was able to significantly improve performance of the classifiers which may be due to the fact that DILI dataset from CAMDA 2018 contest is very difficult to analyse as it is characterised by a weak signal.The analysed dataset in described very briefly in the paper. The paper is a separate piece of scientific work, therefore authors should not assume that the reader is familiar with CAMDA contest and the dataset, and they should provide more detailed description of analysed data. For example: how many drugs were measured, what is the distribution of objects between DILI and non-DILI class.I suggest adding the figure representing proposed workflow. It would also clarify if the preprocessing steps were performed separately or as a single workflowI notice the following sentence (2nd paragraph of page 8 of the manuscript): “Naive Bayes algorithm, which had performed quite poorly in isolation Fig. ”. HoweveIn the description of Fig. In the Fig. I have the following comments:We thank the reviewer for taking the time to review our manuscript and for providing these comments.We now provide information about sample sizes and class imbalance in the Data preprocessing section of Methods.We have added a workflow diagram that illustrates the key components of Phases I and II.We thank the reviewer for catching this. We have removed the part in parenthesis from the manuscript.We have thoroughly revised this figure caption (as well as the others) to improve clarity.We have updated this figure according to the reviewer’s suggestion .Authors’ response:


Gene expression profiling of tumours is an important source of information for cancer patient stratification. Detecting subtle alterations of gene expression remains a challenge, however. Here, we propose a novel tool for high-sensitivity detection of differential pathway activity in tumours. For a pathway defined by a collection of genes, the samples are projected onto a low-dimensional manifold in the subspace spanned by those genes. For each sample, a score is next found by calculating the distance between each projected sample and the projection of a subgroup of reference samples. Depending on the aim of the analysis and the available data, the reference samples may represent e.g. normal tissue or tumour samples with a particular genotype or phenotype. The proposed tool, PathTracer, is demonstrated on gene expression data from 1952 invasive breast cancer samples, 10 DCIS, 9 benign samples and 144 tumour adjacent normal breast tissue samples. PathTracer scores are shown to predict survival, clinical subtypes, cellular proliferation and genomic instability. Furthermore, predictions are shown to outperform those obtained with other comparable methods. However, within the set of tumours being subject to deregulation of a particular pathway, the specific genomic and transcriptomic alterations can vary substantially. As a consequence, it can be challenging to link cancer progression to aberrations of a particular gene. Estimating the activity of pathways rather than single genes seeks to alleviate this problem by accumulating evidence of deregulation across all genes in a pathway, increasing the sensitivity compared to single-gene analyses. The identification of altered pathways may also facilitate development of personalised cancer treatments by identifying the main driving mechanisms in each tumour3. Notably, in recent years, cancer combination therapies have emerged that target specific pathways in parallel or multiple targets in the same pathway5. Carcinogenic alterations may occur at different molecular levels affecting the same genes and pathways. Analysis at the pathway level may identify pathways that are affected by alterations at different levels and in different genes in the same pathways, eg: DNA mutations and aberrations in some genes, silencing by methylation or miRNA-regulation of others. Gene sets or pathways have been analysed in various ways6. Gene set enrichment and gene ontology analyses identify genes that are significantly over-represented (enriched) in a gene list. The list is defined by an arbitrary threshold that may exclude significant genes. Construction of de novo networks of mutated/altered genes and their neighbours discards previous knowledge about biological function, but may identify known genetic associations. Finally, network-based modelling uses prior knowledge about networks/pathways to identify alterations in cancer.Cancer initiation and progression involves deregulation of multiple signalling and metabolic pathwaysPARADIGM7 was one of the first methods trying to meet the challenge of personalised pathway analysis, by describing the deregulation in a particular individual, as opposed to characterising a pathway’s activity for an entire sample set. Another such pathway-based method was Pathifier2. Pathifier calculates for every pathway a so-called Pathway Deregulation Score (PDS) representing the extent to which the pathway is deregulated in every individual sample. Pathifier utilises a non-linear analogy to principal components called principal curve8. Specifically, each tumour sample is projected onto the principal curve before calculating the distance to the centroid of the normal samples. A potential challenge using principal curves to represent the underlying data is that not all data can be represented by a single non-linear principal component. In that case, the behaviour of the principal curve approach will be unpredictable. Samples with high deregulation of a particular pathway will be located far from the normal samples in the multidimensional expression space, and we expect them to be located at the extreme end of the curve. However, if the principal curve does not represent the data well, this may not be the case . In such cases, the most deregulated samples may be located far from the extreme ends of the curve and then the distance from the tumour sample along the curve to the normal samples does not reflect the degree of deregulation. We propose to avoid this problem by using instead the Euclidean distance between two points on the principal curve, one representing the centroid of the normal samples and the other the projection of the tumour sample.The data input for the PathTracer method is a genome-wide gene expression profile (e.g from RNA-sequencing or microarray data) for a collection of samples of interest (e.g tumour samples), as well as for a collection of reference samples . First, expression values are gene-centered by subtraction of the mean and gene-scaled by division with the standard deviation. This standardisation is performed to ensure that all genes are on the same scale, otherwise the highly expressed genes will dominate in the principal components that will be constructed in the subsequent analysis. We assume the data to be normalised between samples prior to the analysis.n samples for the genes in the pathway. For a pathway consisting of p genes, we thus obtain an n × p matrix E of gene expression values, and for the i th row (sample) we have a label Li indicating whether the row represents a sample of interest or a reference sample. We here assume, without loss of generality, that the samples of interest are tumour samples and reference samples are from normal tissue . We will only consider pathways above a given size, as pathways with very few genes are likely to give unstable results. We first derive a low-dimensional feature vector zi ∈ Rm for the i th sample, where m ≪ p. The components of the vector zi are the first m scores from a principal component analysis (PCA) derived from the gene expression matrix E. The number of components m may be fixed in advance or estimated from the data. We now have a reduced data matrix n × m where rows represent samples and columns represent features. The next step reduces the dimension even further from the m- dimensional feature vectors zi to one-dimensional scores. To do this, a principal curve8 is fitted to the sample feature vectors zi. The first component in classical PCA is the line minimising the sum of squared distances from the data points to their orthogonal projections onto the line. A principal curve is analogous, except that the line is replaced by a smooth curve. Fitting a principal curve involves a trade-off between smoothness (degree of deviation from a straight line) and goodness-of-fit to the data. This trade-off is governed by a model parameter λ > 0 which is determined by cross-validation.The method considers one pathway at a time, extracting gene expression data in all m-dimensional feature vector zi has an orthogonal projection P(zi) = pi onto the principal curve . Let k be the index of the normal sample for which dk = median(di) where the median is calculated across all normal samples . Let cN = P(zk) = pk be the projection onto the principal curve of this normal sample. For the i th sample, define the score ei = ||pi − cN||, i.e the Euclidean distance between pi and cN  must also be chosen. The optimal m will handle the bias-variance trade-off by including enough PCs to avoid underfitting, but at the same time avoiding overfitting by including enough PCs. We evaluated the performance in terms of survival prediction on training and test sets, and found that m = 4 gave the best results on the test set, but that results were quite stable in the range of m ∈ {3, …, 10}. See Supplementary Fig. pmin must also be chosen. In our application we have used pmin = 10, i.e about twice the number of principal components.The number of principal components 9. The METABRIC study collected samples from existing biobanks and the study was approved by the ethics committees in Cambridge and Vancouver ensuring that the methods were carried out in accordance with relevant guidelines and regulations. The expression data are available at the European Genome-phenome Archive (http://ebi.ac.uk/ega/) at the European Bioinformatics Institute (accession number EGAS00000000083). The Illumina HT-12 v3 platform was used for whole genome expression profiling9. This study has only used data available in the original publication. The cohort consists of a total of 2115 samples including 1952 invasive breast cancer samples, 10 DCIS, 9 benign samples and 144 tumour adjacent normal breast tissue samples.The breast cancer dataset used for the analyses has previously published as the Metabric dataset and the samples were collected in five different hospitals in the UK and Canada as previously described9. Pathways were downloaded from the Reactome database, extracting all Homo sapiens pathways (n = 1288) using the reactome.db R/Bioconductor package11.The PathTracer algorithm was applied to whole genome gene expression data from the Metabric breast cancer cohort including 1952 breast cancer samples 10 DCIS, 9 benign samples and 144 adjacent normal breast tissue samplesThe file loops.html  shows a selected set of pathways where the original points are plotted together with the principal curve in rotatable 3d figures. For many of the pathways we see that the principal curve is quite far from many of the original points. We also see, as for example for the Defensin pathway, that the principal curve “back-flips”, and almost forms a loop formation. We note that even though the normal samples are quite close in terms of Euclidean distance, they are very from each other in terms of distance along the principal curve, and would thus get a very high PDS value. We also note that for this pathway the correlation between the PDS and the PTS is very low (Pearson correlation = −0.08). A low correlation between PDS and PTS will in general be indicative of such loop-like principal curve formation. We therefore calculated the Pearson correlation between PDS and PTS for all pathways , ER-status (P = 2.2E − 100), the expression based PAM50 breast cancer subtypes (P = 3.4E − 288), and the previously published Integrated Clusters subtypes12 (P = 9.6E − 272)(chi squared tests)  group13, genomic instability index (GII), proliferation score and HER2-status   separating significantly between tumour and normal samples. In order to identify these pathways, we calculated the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC). In a ROC curve, true positive rate or sensitivity (y-axis) is plotted against false positive rate or 1-specificity (x-axis) for the whole range of specificities. Here, tumour and normal samples were regarded as positives and negatives, respectively, and true positive rate and false positive rate for a given PTS threshold value was defined as the number of tumour samples and normal samples, respectively, with PTS over the threshold value. Based on the PTS of the pathways with AUC ≥ 0.98 (n = 322), we identified a total of six sample clusters highly associated with tissue type (ts) Fig. . The sixtus Fig. . Breast st) Fig. .Figure 2TP53 mutation. For sample cluster 2 (green) most samples belong to IC4ER+ and IC3 and normal-like or luminal A subtypes, while sample cluster 3 (blue) contains mainly samples of IC3 and luminal A subtypes. Both the gene-expression based PAM50 subtypes and the IntClust (based both on gene expression and genomic data) are highly associated with the clustering observed based on pathway deregulation  and Genome Instability Index (GII) across pathways  was significantly higher than the correlation between the Pathifier deregulation scores PDS (PDS) and GII   is higher for PTS than for PDS for a range of different significance thresholds  Fig.  and overTP53 mutated and wild-type samples for tumours in the Metabric dataset using wild-type samples as reference group. Figure TP53 wild-type ER positive samples; cluster 2 (yellow) and cluster 5 (blue). Both are dominated by luminal A and IC3 samples and have a good prognosis , we have identified PTSs comparing sis Fig. . The yelAnalysis of pathway deregulation in cancer captures alterations at different molecular levels as well as in different genes in the pathway and may give an overview of biological alterations in carcinogenesis and characteristics particular to a specific tumour. Identifying pathway deregulation allows identification of combined alterations that affect the cancer biology more than identifying single mutations. This is important for a broader understanding of the biology, but also for the development of new treatment strategies and for understanding treatment failure.PathTracer identifies deregulation of a user-defined list of input pathways in cancer samples compared with a specified set of normal (reference) samples. Use of the method may be exploratory by including a wide range of pathways to identify which are most deregulated. The method may also be used for studying the behaviour of a specific set of pathways related to one or a few biological functions.2, in that it utilizes principal curves to obtain deregulation scores, but differ in many ways, affecting both robustness of results, computational time, and output visualisation. Most importantly, we have implemented a novel distance measure method to address the problem of pathways where the principal curve goes in a loop. Pathifier will in such situations estimate the deregulation score along the curve and assign a high deregulation score to a sample located at the tail of the curve but close to the reference samples. The biological meaning of such a score is questionable. By using the Euclidean distance from the projection of the sample onto the principal curve to the reference point cN, the samples will get a deregulation score that reflects the distance from the sample to the reference samples in a multidimensional space and which is more likely to reflect the underlying biology. In addition to the problem with principal curves going in loops, one may get unreliable results if the curves are very ragged (non-smooth), because such curves will generate long distances along the curves (and very high deregulation score), even though the Euclidean distances to the reference point may not be long. To further improve robustness, PathTracer has an optional pre-filtering step where pathways with fewer than k genes are omitted from the analysis . The rationale behind this pre-filtering is that pathways consisting of very few genes are less likely to benefit from the dimension reduction step in PathTracer. There is a trade-off involved here, however, as one may risk missing key pathways by such filtering. We would therefore recommend that use of pre-filtering is used with caution if one suspects that important pathways may thus be discarded. Further, as pathway analyses tend to be extremely time consuming, substantial changes to decrease computational time has been implemented. The most time-consuming part in the Pathifier method is a cross-validation procedure to find the optimal number of principal components (PCs). By instead locking the number of PCs, we reduce the computational time about 10-fold. In addition, we have added the possibility of performing parallelization, which enables the user to run the analysis on several cores simultaneously, drastically reducing the calculation time. Several features enabling data presentation have been incorporated in the PathTracer method, including generation of a heatmap with clustering of samples and deregulation scores including significance testing between the input categories for each pathway. A principal curve plot for each pathway may help interpret the results and help the researcher understand the biology underlying the figures produced by the algorithm.The motivation for developing PathTracer was the worry that previous methods would estimate an unjustified high deregulation score to certain pathways. The presented method is similar to the Pathifier methodThe PathTracer method was applied to a set of breast cancer samples to illustrate its use. Survival and the biological features genome instability index and proliferation score show increased correlation with PTS (PathTracer) compared with PDS (Pathifier). Clustering based on PTS identified clusters with varying degree of pathway deregulation and a high correlation with clinical and biological features.Supplementary figure 1Supplementary figure 2