Clustering is indispensable in the quest for robust vegetation classification schemes that aim to partition, summarise and communicate patterns. However, clustering solutions are sensitive to methods and data and are therefore unstable, a feature that is usually attributed to noise. Viewed through a central‐tendency lens, noise is defined as the degree of departure from type, which is problematic since vegetation types are abstractions of continua, and so noise can only be quantified relative to the particular solution at hand. Graph theory models the structure of vegetation data based on the interconnectivity of samples. Through a graph‐theoretic lens, the causes of instability can be quantified in absolute terms via the degree of connectivity among objects. We simulated incremental increases in sampling intensity in a dataset over five iterations and assessed classification stability across successive solutions derived using algorithms implementing, respectively, models of central‐tendency and interconnectivity. We used logistic regression to model the likelihood of a sample changing groups between iterations as a function of distance to the centroid and degree of interconnectivity. Our results show that the degree to which samples are interconnected is a more powerful predictor of instability than the degree to which they deviate from their nearest centroid. The removal of weakly interconnected samples resulted in more stable classifications, although solutions with many clusters were apparently inherently less stable than those with few clusters, and improvements in stability flowing from the removal of outliers declined as the number of clusters increased. Our results reinforce the fact that clusters abstracted from continuous data are inherently unstable and that the quest for stable, fine‐scale classifications from large regional datasets is illusory. Nevertheless, our results show that using models better suited to the analysis of continuous data may yield more stable classifications of the available data. Clustering is indispensable to the task of building robust vegetation classification systems, and yet the attribution of samples to clusters is notoriously unstable. The likelihood of a sample relocating to a new cluster (instability) upon the addition of new data is inversely proportional to the degree to which it is interconnected with other samples (A&C). Conversely, proximity to the cluster centroid is a poor predictor of instability because cluster centroids are mobile (B&D). Noise clustering  as an alternative approach that avoids some of the limitations of central‐tendency approaches. Graphs are mathematical structures that model pairwise relationships between samples (referred to from here on as interconnectivity) as opposed to relationships between group members and a theoretical average state . In the context of vegetation science, a graphical model comprises a set of vertices, one for each plot sample (releve) contained in the dataset. Edges (junctions) between pairs of vertices represent relationships between samples and can be either binary (e.g. pairwise similarity exceeds a specified threshold) or weighted . Thresholds or weights may be used as criteria to create a sparse graph, so‐called because it contains many fewer edges than the maximum possible  is a better indicator of instability  than the degree of deviation from the centroid ;classification stability over multiple classification iterations is higher when undertaken using a graph model compared to a central‐tendency model;the removal of outliers leads to more stable classifications from one iteration to the next; andclassification stability increases in successive iterations of the classification as the volume of data increases under either model .In this paper, we assess the extent to which classification instability can be attributed to either deviation from the centroid  or interconnectivity (which quantifies noise when vegetation data are modelled as graph networks). We simulated incremental changes in a classification arising from the addition of new data and hypothesised that:In posing these hypotheses, we reasoned that a useful measure of the utility of any measure of noise is how well it forecasts instability in future iterations of the classification as new data are added, and that since the graph model is theoretically better suited to continuous data, it should produce more stable clustering solutions (hypotheses 1 and 2). We reasoned that if instability is a function of noisy samples continually changing groups, then the removal of those samples should increase classification stability (hypothesis 3). Finally, since our approach (described below) involved simulation of the development of a classification as progressively more data were added, we evaluated the evidence for the common assumption that clustering solutions become more stable with increasing sampling because ‘types’ become clear when represented by a sufficient number of samples (hypothesis 4).22.12 located in southeast Australia. Our initial clustering solution was derived using a random selection of 50% of the available samples. This number approximated the number of samples available three decades ago , although our selection was almost certainly more evenly distributed in relation to environmental gradients than those available historically. We added a further random selection of 10% of the data (i.e. 20% of the remaining 50%) over five successive iterations in order to simulate the circumstances under which a classification is revised following the addition of new data. No fixed groups were involved; instead, new classes were allowed to form at each iteration. At each iteration, we categorised the classification of a sample as stable if members of its cluster in the first iteration formed a relative majority in its second . We analysed solutions across a range of cluster numbers (25–250) to determine the generality of our conclusions for applications requiring different levels of classification detail  at 0.1 and 0.2. That is, a vertex represented in fewer than (x*nnbrs) of the neighbourhood sets of its nearest neighbours was eliminated, where nnbrs = the number of nearest neighbours specified as a starting parameter. Similarly, an edge in the sparse graph was eliminated if fewer than (x*nnbrs) were common to the neighbour sets of both vertices.Cluto's scluster function allows for vertices with low interconnectivity to be eliminated from the graph prior to partitioning by specifying the minimum representation of the vertex in the member sets of its nearest neighbours. The reasoning is that those vertices not well represented in the neighbourhood sets of their nearest neighbours are likely to be outliers Karypis, . In addi2.4(x*nnbrs). This measure is intrinsic for each sample in each data subset (i.e. independent of any clustering solution). We range‐standardised both metrics to facilitate the comparison of model coefficients (i.e. change in the log‐odds ratio for every unit change in deviation from centroid or interconnectivity metrics). Ten models were fitted, representing five clustering iterations for each of the two models. Model coefficients (mean model with logit link) were assessed using z‐tests.We used logistic regression Table , H1 to eWe used beta regression  the deviance accounted for by degree of deviation from the centroid and with much smaller confidence bounds on the estimated coefficients Table . Coeffic–18 times3.2No significant differences in classification stability were detected between k‐means clustering and graph partitioning when data were pooled across iterations Table . Both mo3.3Classification stability increased when vertex and edge pruning were applied in combination, with a higher threshold resulting in higher stability Table . Stabili3.4There was no evidence of increasing classification stability as the volume of data increased with each iteration Table . A signi44.1Our results support our hypothesis that the degree of interconnectivity is a better indicator of classification stability than the degree of deviation from the centroid. Poorly interconnected samples had a much higher probability of changing classes between iterations, and conversely, well‐interconnected samples had a higher probability of remaining within their respective classification groups from one iteration to the next. In contrast, the degree to which samples deviated from centroids in one iteration had relatively little bearing on whether or not they changed groups, with the exception of a very small number of samples that either very strongly or very weakly resembled the centroid. As a consequence, interconnectivity explained a much higher proportion of deviance in the modelled outcome (sample changed class or not) compared to the degree of deviation from the centroid.Our results are consistent with the concept of cluster mobility: distances to centroids are properties of clustering solutions rather than properties inherent in any data. Furthermore, they highlight the danger of overweighting the inferences drawn from clustering solutions and reinforcing them against later evidence that may not support them. As currently formulated, noise clustering promotes a clearer picture of patterns in vegetation data by differentiating between samples with strong resemblance to type and those with very little resemblance  biases in the distribution of samples that result in under‐sampling of parts of environmental gradients (gaps); (ii) rare or extreme environments; or (iii) depleted or degraded vegetation types. Such samples may appear as outliers but nonetheless warrant a place in classifications.Gaps can potentially be identified by analysing environmental gradients associated with species turnover and quantifying the distribution of samples relative to the expression of those environmental gradients in geographic space  should have become more strongly interconnected as samples with similar compositions were added back in. In fact, although the distribution of samples was similar in the data used in each of our iterations, the full dataset showed a slightly larger skew in the number of weakly interconnected samples than any of the subsets, and there was a weak trend of increasing sample interconnectivity with a decreasing number of samples. Further investigations are required to determine whether this trend arose due to the chance of partitioning our data into subsamples or if it reflects some systematic properties of the data. We acknowledge that our simulations probably do not reflect the way in which our dataset was compiled through time, which is likely to have commenced opportunistically (targeting accessible areas) switching to gap‐filling over time. In theory, this could imbue a developing classification with greater stability if patterns in accessible areas were well sampled prior to switching to gap‐filling. However, the difficulties inherent in characterising a ‘typical’ pattern of data accumulation suggest any generalities emerging from this line of investigation may be limited. We also note that the intensity of sampling necessary to reliably elucidate vegetation patterns is unknown. Our samples comprise only 0.003% of the area of our study area, and it remains to be seen whether the patterns we observed in our simulations are sustained following increases in sampling over several orders of magnitude. Nevertheless, the contemporary distribution of samples in relation to degree of interconnectivity appears to support Mucina's  contentiFinally, the general trends in our results were consistent for solutions across the range of cluster numbers. However, both classification stability and the degree of improvement in stability achieved by the removal of outliers declined under both graph and central‐tendency models as the number of clusters in the solution increased Figure . This pa5Despite the consensus that vegetation is continuous but can be classified by convention Keddy, , there hMark G. Tozer: Conceptualization ; data curation (lead); formal analysis (lead); methodology (lead); writing – original draft (lead); writing – review and editing (lead). David A. Keith: Conceptualization ; data curation (supporting); formal analysis (supporting); funding acquisition (lead); investigation ; methodology (supporting); project administration (lead); supervision ; writing – original draft (supporting); writing – review and editing .This research was supported by an Australian Government Research Training Program scholarship.The authors declare no conflicts of interest exist in the presentation of this work.Appendix S1.Click here for additional data file.


The recent development of spatially resolved transcriptomics (SRT) technologies has facilitated research on gene expression in the spatial context. Annotating cell types is one crucial step for downstream analysis. However, many existing algorithms use an unsupervised strategy to assign cell types for SRT data. They first conduct clustering analysis and then aggregate cluster-level expression based on the clustering results. This workflow fails to leverage the marker gene information efficiently. On the other hand, other cell annotation methods designed for single-cell RNA-seq data utilize the cell-type marker genes information but fail to use spatial information in SRT data.spatial transcriptomics cell assignment model, SPAN, to annotate clusters of cells or spots into known types in SRT data with prior knowledge of predefined marker genes and spatial information. The SPAN model annotates cells or spots from SRT data using predefined overexpressed marker genes and combines a mixture model with a hidden Markov random field to model the spatial dependency between neighboring spots. We demonstrate the effectiveness of SPAN against spatial and nonspatial clustering algorithms through extensive simulation and real data experiments.We introduce a statistical https://github.com/ChengZ352/SPAN. Many statistical and machine learning models have been proposed for the clustering analysis of scRNA-seq data, including . HoweverTo utilize the prior information of cell-type specific marker genes, CellAssign  that assigns cells or spots into known types in the SRT data with prior knowledge of predefined marker genes and spatial information. The SPAN model combines a mixture model with an HMRF to model spatial dependency between neighboring spots and annotates cells or spots from SRT data using predefined overexpressed marker genes. The discrete counts of SRT data are characterized by the negative binomial (NB) distribution. Other experimental or technical covariates, such as batch or individual information, can also be incorporated into the model. We evaluate SPAN on extensive simulations and real data experiments and show that SPAN outperforms existing SRT and single-cell clustering methods in various analyses.To address these issues, we propose a statistical The framework of SPAN consists of two modules: a mixture NB distribution module and an HMRF module, as illustrated in Y represents a spot-by-gene count matrix with N spots and G marker genes. We assume that these N spots can be divided into K cluster types. The proposed method models the count matrix Y by the likelihood of NB distribution. The expression of gene g at spot i for a latent clustering k can be obtained by:Let igkμ and dispersion i belongs to.where the NB distribution is parameterized with mean igkμ as:We follow CellAssign . The constraint with the constraint that i, the conditional probability of the spot having type iz = k, given the type of all its neighbors isFor a specific spot i having clustering type l, and where Then, the parameters i.where SPAN model is formed by integrating the two modules. We introduce the HMRF as the prior on the mixture NB model to build the SPAN model.iz isLet I is the indicator function.where The log-likelihood of parameter z.We estimate the model parameters and infer the clustering assignment ls (ICM)  to estimAlgorithm 1. SPAN model training processInput: gene expression matirx Y; covariate matrix X ; marker gene—clustering type indicator matrix DOuput: an clustering assignment vector z1: Initialize 2: Initialize t = 0;3: while  marker genes and (ii) selected high-variance genes (HVGs). We first selected the top 2000 genes (1000 genes for the simulated datasets) by using the mean-variance relationship , and perFollowing CellAssign, we mapped the unsupervised clustering results to the ground-truth groups. First, we applied analytic Pearson residuals normalization  to evaluate the performance of different methods. To generate the macro F1 score value, we first calculated the F1 score for each cluster with the One-vs-Rest strategy and computed the mean of all F1 scores.g was selected as a marker for group k, if gkDEFacGroup > 1.5. We repeated all experiments ten times under the same setting with different random seeds. The detailed simulation settings are summarized in To evaluate the model performance, we designed several simulations for different biological scenarios. We extracted the spatial information and ground truth spot type assignment from the sample 151 673 in the dorsolateral prefrontal cortex (DLPFC) , but other marker genes are assigned to the corresponding groups. The second case assumes that the simulated data does not have nonmarker genes but some markers are assigned to incorrect clusters.In the first case, we randomly replaced some marker genes with other nonmarker genes and randomly assigned a group type to each fake marker gene. We fixed the sigma and varied the ratio of nonmarker genes from 0% to 20%. A ratio equal to 0 means that we do not introduce any fake marker genes, and the larger the ratio, the more fake markers. We compared the performance of marker gene-based approaches, SPAN and CellAssign, and two spatial clustering methods, Bayesspace and stLearn, using only marker genes as input. As shown in In SRT, expression patterns are often correlated, and adjacent locations are more likely to belong to the same group. While we expect this cell type smoothness assumption to hold well in most SRT, there are also cases where this is decidedly not the case and more diverse spatial dependency patterns are observed. Therefore, we conducted the following experiments to see how the performance of SPAN persists under different spatial dependencies.Firstly, we introduced spatial noise by randomly switching the gene expression and related spot type assignments of spots between different groups. We fixed the sigma and varied the switch ratio from 0% to 20%. A ratio equal to 0 means that we do not switch the spot position, and the larger the ratio, the greater the signal noise, and the less smooth the group assignment. We tested the performance of three spatial clustering methods, SPAN, Bayesspace and stLearn, under different levels of spatial noise. Secondly, to further demonstrate the performance of the proposed method when the smoothness is not strong, we directly simulate a scenario where different cell types are mixed. Specifically, spatial information is extracted from a Slide-seq cerebellum dataset from the RCTD . These sThirdly, we also evaluated the model performance using the spatial information extracted from another ST platform. This melanoma dataset , thanks K, and the marker-cell type indicator matrix, ρ. It is common that our prior knowledge about cell types may not be perfect, with one cell type not provided (corresponding to K − 1), or additional cell types provided but not existing in the given sample. It is interesting to test whether SPAN can assign the cells with missing cell type information to NAs (“to be determined”), and if SPAN can decline to assign any cells to additional cell types when they are provided but not actually present in the given sample. Thus, we next evaluated the model’s robustness with imperfect prior cell type information that reflects real-world scenarios  dataset, which is provided by the 10× scRNA-seq platform (k = 15. We then applied SPAN to the osmFISH (mouse cortex) dataset  by taking the batch ID as input. We have demonstrated the performance over other unsupervised clustering methods and cell annotation algorithms on different simulation scenarios and real datasets.This study finds that the prior information of canonical marker genes described in the literature or from curated datasets can improve the accuracy of cell type assignments. Other studies have also demonstrated the utility of marker gene information in clustering cells (Although SPAN achieves good performance for SRT data, there are still some limitations that can be improved. First, we used the k-nearest neighbors approach to determine the neighbors based on the Euclidean distance between cells. However, alternative methods for generating neighbors can be further investigated. Second, in SRT, a spot may contain different types of cells, such that the cell-type markers cannot reflect the entire gene expression of the spot. When two or more types of cells are evenly distributed in a spot, it may greatly impact the model performance. This is likely to occur at the border of two groups. Implementing some downstream analysis may provide more accurate predictions. Third, our model only applies the first-order Markov model to simulate the correlated expression pattern in the adjacent spots in SRT data. However, it is possible to explore more complex spatial patterns by incorporating second or higher-order neighbors. Thus, we may apply a second or higher-order Markov model to enhance the performance of SPAN. Fourth, since we apply the ICM to estimate the model parameters during the model training process, we need to assign each cell/spot to a known cluster type at each iteration. In other words, our model cannot handle unknown cluster types. We will solve this matter in our future studies.btad641_Supplementary_DataClick here for additional data file.


Many computational tools are developed for this purpose. However, the robustness of existing CCC methods remains underexplored. We develop a user-friendly tool, RobustCCC, to facilitate the robustness evaluation of CCC methods with respect to three perspectives, including replicated data, transcriptomic data noise and prior knowledge noise. RobustCCC currently integrates 14 state-of-the-art CCC methods and 6 simulated single-cell transcriptomics datasets to generate robustness evaluation reports in tabular form for easy interpretation. We find that these methods exhibit substantially different robustness performances using different simulation datasets, implying a strong impact of the input data on resulting CCC patterns. In summary, RobustCCC represents a scalable tool that can easily integrate more CCC methods, more single-cell datasets from different species  to provide guidance in selecting methods for identification of consistent and stable CCC patterns in tissue microenvironments. RobustCCC is freely available at  CellPhoneDB is applied to the research of human gonadal development (Cell-cell communication (CCC) is a biological process where cells transmit signals to other cells through biological or chemical molecules, playing an important role in tissue formation and disease development . With thelopment . Jin et A large number of CCC methods have been developed, raising the question of how to systematically evaluate these methods. https://github.com/GaoLabXDU/RobustCCC). In summary, RobustCCC represents a scalable tool that can easily integrate more CCC methods and more single-cell datasets, easily perform CCC methods and systematically evaluate their robustness, providing guidance in selecting methods for identification of consistent and stable CCC patterns in tissue microenvironments.To answer this question, we develop a user-friendly tool, RobustCCC, which aim to evaluate the robustness of CCC methods from three perspectives: replicated data, transcriptomic data noise and prior knowledge noise. RobustCCC currently integrates 14 state-of-the-art CCC methods and 6 simulated single-cell transcriptomics datasets, generating robustness evaluation reports in tabular form for easy interpretation. By using RobustCCC to evaluate the robustness of 14 CCC methods, we observe significant variations across different simulation datasets. As a result, no single method emerge as the most robust across all simulation datasets. In addition, RobustCCC is implemented as an R package and is freely available at GitHub  The inputs of the methods include scRNA-seq data, annotations of two cell types, and known ligand-receptor pairs.2) The outputs of the methods are communication scores or significances of ligand-receptor pair.We collect CCC methods according to the following criteria:p-value<0.05). Network-based methods consider the upstream and downstream pathway of ligand-receptor pairs and identify ligand-receptor pairs or network modules involved in communication from biological networks or co-expression networks. The details of these methods are shown in In total, we collect 14 CCC methods, comprising 11 statistics-based methods and 3 network-based methods. The statistics-based methods refer to calculating communication scores of each ligand-receptor pair based on their gene expression. To determine the significance of communication scores, the cell labels are shuffled to obtain a null distribution. Finally, the significant ligand-receptor pairs involved in communication are inferred by filtering based on a predefined significance threshold  available on the Single Cell Portal (ID: SCP795) . This daWe create 12 distinct pairs of cell types by combining the four cell types mentioned earlier . Considering the four mouse scRNA-seq datasets, we have obtained a total of 48 scRNA-seq datasets for each cell type pair (4 mouse multiplied by 12 cell type pairs). These datasets will serve as the basis for constructing simulation dataset and evaluating the robustness of CCC methods.The following 6 simulation datasets are constructed, including biological replicates, simulated replicates, Gaussian noise, dropout, cell type permutation, ligand-receptor permutation. Specifically, biological replicates and simulated replicates are two replicated data. Gaussian noise and dropout are two transcriptomic data noise. Cell type permutation and ligand-receptor permutation are two prior knowledge noise.We collect four mouse scRNA-seq data, including two biological replicates from male mouse and two biological replicates from female mouse.We randomly sample cells from the original scRNA-seq datasets to simulate batch effects. For each cell type pair and cell sampling proportion , we perform three random samplings from each scRNA-seq dataset, resulting in a total of 144 simulated replicates (3 times multiplied by 48 cell type pairs).We add Gaussian noise to the original scRNA-seq datasets. We calculate the mean and variance of expression for each gene in all cells and construct a Gaussian noise distribution based on these statistics. We then generate Gaussian noise data for each gene and merge it with the original data using a certain proportion . For each cell type pair and noising proportion , we generate three Gaussian noise from each scRNA-seq dataset, resulting in a total of 144 data with Gaussian noise (3 times multiplied by 48 cell type pairs).We randomly set expression values to 0 in the original scRNA-seq datasets. For each cell type pair and noising proportion , we generate three dropout events from the each scRNA-seq dataset, resulting in a total of 144 data with dropout events (3 times multiplied by 48 cell type pairs).We randomly permuted the cell labels in the scRNA-seq datasets. This permutation obscures the assignment of sender and receiver cells in the CCC methods. For each cell type pair and permuting proportion , we perform three permutations on each scRNA-seq data, resulting in a total of 144 data with cell type permutation (3 times multiplied by 48 cell type pairs).To generate inaccurate ligand-receptor pairs, we randomly permute the gene symbols in the scRNA-seq datasets, since most of the ligand-receptor datasets are integrated in the tools and difficult to extract and import. This permutation increase the false positive rate of CCC methods. For each cell type pair and permuting proportion , we perform three permutations on each scRNA-seq data, resulting in a total of 144 data with ligand-receptor permutation (3 times multiplied by 48 cell type pairs).Robustness is the capability of a method to handle noises and produce consistent results. The evaluation of the robustness of CCC methods aims to assess the stability of the inferred ligand-receptor pairs under different simulation datasets, including replicated data, transcriptomic data noise and prior knowledge noise. The inferred ligand-receptor pairs can be considered as a set, and the stability of the inferred ligand-receptor pairs can be quantified by measuring the similarity between sets. The Jaccard coefficient is a widely used metric for measuring set similarity. It calculates the size of the intersection divided by the size of the union of two sets. In the context of evaluating robustness, we utilize the Jaccard coefficient as an indicator to quantify the similarity between the sets of inferred ligand-receptor pairs under different simulation datasets.For replicated data, we calculate Jaccard coefficient of inferred ligand-receptor pairs between batch effect data. The formula is as follows:Where For transcriptomic data noise and prior knowledge noise, we calculate Jaccard coefficient of the inferred ligand-receptor pairs between original data and noising data. The formula is as follows:Where Then, the average Jaccard coefficient is calculated under different simulation datasets to characterize the overall performance of the CCC methods. The formula is as follows:OrBy comparing the Jaccard coefficients and average Jaccard coefficient under multiple simulation datasets, we can evaluate the robustness of CCC methods and identify variations in their performance.Replicated data are multiple datasets generated under nearly identical conditions, providing evidence for generalizability of biological results . In otheRegarding the robustness evaluation based on biological replicates, we analyze the average Jaccard coefficient of inferred ligand-receptor pairs between biological replicates. Among all the methods, iTalk ranks first with an overall average Jaccard coefficient of 0.796, followed by CytoTalk with an average Jaccard coefficient of 0.721. Kumar, NATMI, and SingleCellSignalR have similar overall average Jaccard coefficients of 0.668, 0.664, and 0.663, respectively. We also observe the distribution of Jaccard coefficients between the inferred ligand-receptor pairs in biological replicates. 6 out of the 14 methods have a distribution standard deviation of no more than 0.1, namely, CytoTalk, Kumar, NATMI, SingleCellSignalR, iTalk, and scConnect, which are all ranked in the top five based on the average Jaccard coefficient. It is worth noting that there is a large difference between Zhou’s median and mean Jaccard coefficient, mainly due to the distribution having many 0 s (in 7 out of 24 data) and 1 s (in 13 out of 24 data) .Regarding the robustness evaluation based on simulated resplicates, we also analyze the average Jaccard coefficient of inferred ligand-receptor pairs between each pair of simulated resplicates. Skelly ranks first with an overall average Jaccard coefficient of 0.959 and performs particularly well at different sampling proportions, ranking first at 90% and 85% proportions. iTalk, NATMI and Kumar ranks second with an overall average Jaccard coefficient of 0.949. At the 95% sampling proportion, the average Jaccard coefficient for NATMI and Kumar (both are 0.973) is higher than that of CellChat (0.968). SingleCellSignalR and CytoTalk rank fifth and sixth, respectively. The boxplots illustrate that the median Jaccard coefficient of most methods decreases as the sampling proportion decreases, except CellChat and Zhou whose median Jaccard coefficient remain at 1.0. Among the 14 CCC methods, CellPhoneDB shows a more moderate decrease in the median Jaccard coefficient, from 0.963 at a 95% proportion to 0.958 at a 90% proportion, and then to 0.948 at a 85% proportion, resulting in a total decrease of 0.015. Furthermore, the standard deviation of Jaccard coefficient distribution of NATMI (0.0131) and CytoTalk (0.0133) is smaller than other methods at 95% proportion .Transcriptomic data noise is primarily caused by expression divergence and dropout events. These noises mask the real communication pattern during the execution of CCC methods. To evaluate the robustness of CCC methods, we consider whether CCC methods can infer same communication patterns in data with Gaussian noise and dropout as in the original data.In the robustness evaluation based on data with Gaussian noise, the heatmap of the average Jaccard coefficient of ligand-receptor pairs between the data with Gaussian noise and the original data show that CytoTalk ranks first among all methods, with an overall average Jaccard coefficient of 0.988. Additionally, CytoTalk also achieves the highest average Jaccard coefficients at noising proportions of 5%, 10%, and 15%, with values of 0.994, 0.989, and 0.982, respectively. iTalk ranks second with an overall average Jaccard coefficient of 0.818, while Kumar ranks third with 0.729, and NATMI ranks fourth with 0.723. NicheNet ranks fifth with an overall average Jaccard coefficient of 0.722, surpassing Kumar and NATMI in average Jaccard coefficient at 5% and 10% noising proportions. The boxplots of the Jaccard coefficient reveal that Gaussian noise has different effects on different methods. Some methods, such as CytoTalk, iTalk, and NicheNet, show a decrease in median Jaccard coefficient with increasing noising proportion . Other methods, like NATMI, Kumar, and SingleCellSignalR, have a low median Jaccard coefficient at 5% noise and show little sensitivity to increases in the noising proportion  .In the robustness evaluation based on data with dropout, we calculate the average Jaccard coefficient of ligand-receptor pairs between the data with dropout and the original data. Most methods perform similarly on data with dropout as on original data, with overall average Jaccard coefficients exceeding 0.95 . The standard deviation is also within an acceptable range, not exceeding 0.05 (except for ICELLNET at 15% with a standard deviation of 0.056). However, we still observe a decrease in the median Jaccard coefficient as the proportion of random set 0 increases. Although the decrease is not significant for the top-performing methods, such as Kumar, SingleCellSignalR, NATMI, CytoTalk, iTALK, ICELLNET, and scConnect, with decreases ranging from 0.026 to 0.051  .Cell type annotations and ligand-receptor data are necessary when inferring CCC. The different cell type annotations can bias CCC methods in the selection of sender cells and receiver cells, affecting the inference of communication pattern. The unvalidated ligand-receptor pairs could increase the false positive rate of CCC methods. To evaluate the robustness of CCC methods, we permute cell type annotations and ligand-receptor data, considering whether CCC methods can infer same communication patterns in data with these two prior knowledge noises as in the original data.For the robustness evaluation based on cell type permutation, we consider the Jaccard coefficient between the permuted scRNA-seq data and the original data. CytoTalk achieves the highest Jaccard coefficient of 0.882, and it also ranks first in average Jaccard coefficients at permuting proportions of 5%, 10%, and 15%, with values of 0.922, 0.882, and 0.843, respectively. Kumar and NATMI both rank second with an overall average Jaccard coefficient of 0.844, with NATMI performing better at 5% proportion and Kumar performing better at 10% proportion. SingleCellSignalR, scConnect, and ICELLNET rank fourth, fifth, and sixth, respectively, with overall average Jaccard coefficients of 0.841, 0.806, and 0.755. The boxplots demonstrate that the median Jaccard coefficient for each method decreases as the permuting proportion permuting increases, indicating that permuting cell type annotations affects the performance of CCC methods. Among these methods, CytoTalk, Kumar, NATMI, and SingleCellSignalR are less affected by shuffled cell type annotations, with median similarities above 0.8, a decrease of no more than 0.1 (from 5% to 15% proportion), and a Jaccard coefficient variance at each proportion not exceeding 0.1 .For the robustness evaluation based on ligand-receptor permutation, The heatmap of the average Jaccard coefficient of ligand-receptor pairs between the permutation data and the original data show that CytoTalk achieves the highest Jaccard coefficient of 0.915. It also ranks first in average Jaccard coefficients at permuting proportions of 5%, 10%, and 15%, with values of 0.962, 0.906, and 0.878, respectively. Kumar ranks second with an overall average Jaccard coefficient of 0.898, and NATMI performs similarly, ranking third with an overall average Jaccard coefficient of 0.897. SingleCellSignalR, scConnect, and ICELLNET rank fourth, fifth, and sixth, respectively, with overall average Jaccard coefficients of 0.889, 0.846, and 0.845. We also observe the median and standard deviation of the Jaccard distribution. The median decreases with an increase in the permuting proportion, indicating that gene permutation has an impact on the CCC methods. CytoTalk, Kumar, NATMI, and SingleCellSignalR are less affected by ligand-receptor permutation, with median Jaccard distribution values above 0.8 and a decrease of no more than 0.1 (from 5% to 15% proportion) .We conduct a comprehensive ranking of cell-cell communication methods based on their overall average Jaccard coefficients across different simulation datasets, including biological replicates, simulated replicates, Gaussian noise, dropout, cell type permutation, ligand-receptor permutation .The evaluation results reveal significant variations in the robustness of CCC methods across different simulation datasets. Among the 14 methods evaluated, CytoTalk emerges as the most robust method, achieving the highest overall ranking and ranking within the top 5 in most simulation datasets. It notably performs well in Gaussian noise, cell type permutation and ligand-receptor permutation, where it secures the first position. Kumar secures the second position in the overall ranking and stands out as the top performer in dropout. NATMI secures the third position in the overall ranking and consistently ranks within the top 5 in all perturbed data. It performs well in dropout, cell type permutation, ligand-receptor permutation, where it secures the third position. iTALK attains the fourth position in the overall ranking and exhibits the best performance in biological replicates. Skelly ranks first in simulated replicates.These findings emphasize the varying degrees of robustness exhibited by different CCC methods under different simulation datasets. CytoTalk demonstrates the best robustness across multiple perturbed data types. Kumar, NATMI, iTALK and Skelly also demonstrate strong performance in specific simulated data.We develop RobustCCC, a user-friendly tool to evaluate robustness of CCC methods. RobustCCC incorporates 14 state-of-the-art CCC methods and 6 simulated single-cell transcriptomics datasets to generate comprehensive robustness evaluation reports presented in tabular form for easy interpretation. In summary, RobustCCC serves as a scalable tool that can easily integrate more CCC methods and more single-cell datasets from different species  to provide guidance in selecting methods for identification of consistent and stable CCC patterns in tissue microenvironments.During the robustness evaluation, we notice that some methods performed differently than others. For instance, in the case of Zhou , we obsede novo construction of an inter-cellular regulatory network, resulting in a denser network with more edges. These additional edges can be considered as redundancy or backup, providing better protection against noises. In contrast, NicheNet and scMLnet rely on sparse real biological networks, making them relatively weaker against noises.Regarding the comparison between network-based and statistical-based methods, we do not observe a clear difference in robustness. However, within the network-based methods [CytoTalk , NicheNeFurther, we compare all methods in all datasets by calculating Jaccard of top 30 ligand-receptor pairs ranked by interaction scores . The result exhibited in In addition, we compare the results of 14 CCC methods with each other. The average Jaccard coefficient of results between one CCC method to the other CCC methods show that the consistency of these methods is not high. The mean pairwise Jaccard coefficient ranged from 0.003 to 0.183 across datasets . DimitroIn our evaluation framework, we use 48 scRNA-seq datasets of mouse cerebellar cortex (4 mouse multiplied by 12 cell type pairs) to evaluate robustness of CCC method. Since the performance of different methods may change with respect to different datasets, more datasets from different organizations and different species enable more systematic evaluation CCC methods.We also notice that many methods for inferring CCC based on single-cell spatial transcriptome data have been designed in recent years, such as COMMOT , SpaTalk


The data is supplemented with geological samples collected from five raw deposits in southeastern Missouri (n = 70). Data was acquired through Neutron Activation Analysis to assess provenance of all archaeological samples. Following the irradiation of samples, data was normalized for iron content before statistical analysis. A variety of multivariate statistical routines, including principal component and cluster analyses were then employed to assess possible origin locales for all archaeological samples. This data article also includes maps, tables, and figures to assist in understanding the analysis conducted.Hematite, and more broadly ochre, have long been used by humans throughout history for a variety of applications. In prehistoric North America the use of hematite is as old as its first migrants. This data article includes data related to the analysis of archaeological hematite in the American Bottom region in Missouri and Illinois, U.S.A. The data include archaeological samples dating from the Late Archaic Period (3000 – 1000 BCE) to the Middle Woodland Period (150 BCE – 400 CE) from nine sites within the general St. Louis area ( Specifications Table1•These data are the first NAA data produced for archaeological ochres in Missouri. It is significant in that few studies have addressed provenance of ochre artifacts beyond their use as pigments, and as such may be of particular interest to those interested in the assessment of provenance for ochre artifacts in the region and elsewhere in North America.•For re-use of these data, researchers may compare newly collected compositional (NAA) data from other artifacts to the data presented here to assess potential origin locales. It is especially useful in that it contains reference sample data from multiple geological sources, as well as an array of samples from many sites, including one major site in an iron rich region of Missouri.•Data can also be compared to data from other regions to better understand the expected range of compositional variation that may exist within sources and regions.2A map of the American Bottom region shows the locations of sites included in this study , Table 133.13.1.1n = 31) collected from five more sites on either side of the Mississippi River within the St. Louis Region, and 29 samples collected from Verkamp Rockshelter in Phelps County, Missouri. The most significant of any site in this study, Verkamp Rockshelter is a multicomponent site dating from the Archaic to the Woodland Period The samples included in this study consist of 98 archaeological samples and 70 geological samples . Of thes3.1.2The samples in this study were selected to understand regional ochre procurement patterns. Previously, studies had suggested that St. Louis area archaeological ochres had originated in the Meramec River Valley. For this reason. Archaeological samples from this area (Verkamp Rockshelter) were also included in this study. St. Louis region samples were selected opportunistically based upon what was available, resulting in a range of sample sizes from two to 30 per site. Verkamp Rockshelter, on the other hand, was selected to represent the Fe-rich Meramec River Valley. Within this site, only a portion of each ochre sample was analyzed to ensure the preservation of the overall assemblage and to be able to consider specimens from multiple vertical and horizontal contexts to examine the full scope of site-wide compositional variability. Geological samples were previously selected based upon archaeological surveys of historical mines. All samples were collected from the rims and sides of mine pits to avoid potential historical disturbances within pits 3.1.313 neutrons cm−2 s−1. These samples were then allowed to decay for 25 min, and counted for 720 s to determine concentrations for Al, Ba, Ca, DY, K, Mn, Na, Ti, and V. For mid-count and long-count measurements, a second 60 mg of sample powder was placed in a high purity quartz vial and irradiated for 24 h at a thermal neutron flux of approximately 5.2 × 1013 neutrons cm−2 s−1. After 7 days of decay, the “mid” count was acquired for 2000 s to determine As, Lu, Nd, Sm, U, and Yb. Finally, after 3 weeks, the “long” count was acquired for 10,000 s to determine Ce, Co, Cr, Cs, Eu, Fe, Hf, Ni, Rb, Sb, Sc, Sr, Ta, Tb, Th, Zn, and Zr. Standards used for NAA were NIST SRM 1633a (fly ash) and SRM 688 , with quality control standards of NIST SRM 278 (Obsidian Rock) and Ohio red clay To prepare for neutron activation analysis, all samples were first ground into a powder using an agate mortar and pestle for sample homogenization. Samples were then weighed and placed in vials for irradiation and subsequent decay for variable lengths of time. For short counts, 60 mg of powdered sample was placed in a 1.2 ml high-density polyethylene tube for 5s of irradiation at a thermal flux of approximately 8.0 × 103.1.4p > 0) were used in further statistical analysis. After eliminating the negatively correlated elements, all data were then normalized to compensate for the widely ranging Fe concentrations. This was accomplished by the Fe- correction factors used in earlier studies, including the modelling the Fe-oxide concentration and accounting for the overall Fe concentration With the analysis of hematite, an additional correction is necessary to account for the trace elemental relationship with iron and modeling the contribution of Fe-oxides and -hydroxides as well as to determine the elements correlated with Fe in ochre 3.1.5Chondrite normalization was calculated based on a simple ratio of the element to the chondrite value as published by McDonough and Sun . Element3.1.6Using MURR's in-house GAUSS software, multiple multivariate statistical analyses were conducted. These include a base-10 logarithm to characterize the sample as well as determine compositional differences and similarities reflective of differential provenance, including comparing archaeological samples to reference materials collected directly from deposits previously. Once the data were corrected to account for Fe influence, principal component analyses were conducted to explore the overall composition of the assemblage collectively . ClusterNot applicableThis work abides by all ethical standards and has not included the involvement of human subjects, animal experiments, nor any data collected from social media platforms.Daniel E. Pierce: Conceptualization, Methodology, Data curation, Writing – original draft, Writing – review & editing. Rachel S. Popelka-Filcoff: Methodology, Writing – original draft, Writing – review & editing.


While experimental and informatic techniques around single cell sequencing (scRNA-seq) are advanced, research around mass cytometry (CyTOF) data analysis has severely lagged behind. CyTOF data are notably different from scRNA-seq data in many aspects. This calls for the evaluation and development of computational methods specific for CyTOF data. Dimension reduction (DR) is one of the critical steps of single cell data analysis. Here, we benchmark the performances of 21 DR methods on 110 real and 425 synthetic CyTOF samples. We find that less well-known methods like SAUCIE, SQuaD-MDS, and scvis are the overall best performers. In particular, SAUCIE and scvis are well balanced, SQuaD-MDS excels at structure preservation, whereas UMAP has great downstream analysis performance. We also find that t-SNE  possesses the best local structure preservation. Nevertheless, there is a high level of complementarity between these tools, so the choice of method should depend on the underlying data structure and the analytical needs. Dimension reduction (DR) is a key step of Cytometry by Time-of-Flight (CyTOF) data analysis. Here, the authors benchmark 21 DR methods on 110 real and 425 synthetic CyTOF samples, finding a high level of complementarity between the methods, and providing a comprehensive set of user guidelines. CyTOF captures much higher numbers of protein markers, compared with traditional flow cytometry, and has minimum spillover effect, unlike regular flow cytometry5. Compared with scRNA-seq, CyTOF profiles the proteomics makeup of the single cells and is more relevant for understanding clinical phenotypes than scRNA-seq. CyTOF can additionally capture post-translational modifications6, which is beyond the reach of scRNA-seq. CyTOF is becoming increasingly popular, at an exponential rate similar to scRNA-seq  and flow cytometry, and their variants. Mass cytometry9. Furthermore, while scRNA-seq data are integer count observations, CyTOF data are often regarded as continuous observations . On the other hand, more and more studies are now trying to generate matched CyTOF and scRNA-seq data for the single cells from the same research subjects12. These two data types are of different nature, and thus are highly complementary with each other. Therefore, analysis processes should verify that findings and conclusions generated from combining CyTOF and scRNA-seq data are consistent/concordant. Unfortunately, these issues have not been addressed thoroughly by researchers, resulting in difficulties for valid interpretation of CyTOF data.There are key differences between CyTOF and scRNA-seq data, which makes it questionable to directly translate scRNA-seq analytical methods for use on CyTOF data. For example, scRNA-seq usually profiles several thousand genes’ RNA expression for several thousand cells, while CyTOF can profile up to 120 proteins’ expression for potentially 10 to 100 times more cells. The large number of cells enable CyTOF to capture the rare populations that may be missed by scRNA-seq data. CyTOF data uniquely suffers from time-dependent signal drift during the generation of data by the CyTOF machine. But CyTOF is mostly free from the drop-out issue that is commonly seen in scRNA-seq data13, UMAP14, SAUCIE15, ZIFA16, PHATE17, scvis18, and Diffusion map19, have been developed or applied to capture the complicated non-linear relationships in the high-dimensional data. However, it’s unclear which of these DR methods is the best for CyTOF. Researchers have focused on benchmarking DR methods for scRNA-seq data in previous works21. However, the best DR methods for scRNA-seq data may not necessarily extrapolate to CyTOF.The first step of single cell data analysis is data exploration, which is usually achieved through Dimension Reduction (DR), and followed by visualization, clustering, cell type assignment, differential expression analyses, etc. Proper DR is fundamental to all downstream analyses. For example, if DR incorrectly places some cells in the wrong place in the reduced space, these cells could be mistakenly labeled as a “new” differentiation stage of another irrelevant cell type instead of their actual cell types. Traditional linear DR algorithms such as Principal Component Analyses have existed for decades. With the rise of scRNA-seq, numerous new DR algorithms, such as tSNE22, which is an expansion of mass cytometry that enables the capturing of the spatial information of the cells  global structure preservation, (2) local structure preservation, 3) downstream analysis performance, and (4) concordance of DR results with matched scRNA data. We provided a detailed description of these metrics in Supplementary Data  downstreCytomulate23 (“Method” section), which to the best of our knowledge is the first formal simulation tool that is able to well mimic the behaviors of CyTOF data for general-purpose usage. Prior works, like LAMBDA25, have created specialized procedures to generate simulated CyTOF data in the context of model-based clustering. Such ad hoc simulation procedures cannot be used to test methods developed for other purposes (e.g. DR), whereas Cytomulate aims to support all facets of CyTOF research and methodological development with flexibility.To operationalize our benchmarking framework, we gathered both publicly available and in house real datasets , and cell types in the simulated CyTOF datasets. We benchmarked the accuracy of the DR methods on these simulated data with the same criteria , which is a desirable characteristic overall. However, when examined in greater details, SQuaD-MDS Hybrid and tSNE render some of the cell types mixed and hard to distinguish from one another. These two methods have superb local structure preservation, which may help explain their placement of small clusters and the tight rendering of clusters, but their downstream performance, which relies on distinguishing between clusters and cell types, is not as impressive.Data visualization is an important downstream task contingent on DR. We embedded the Levine32 cohort with five top and popular DR methods in Fig. 29. In the SAUCIE embedding, HLA.DR-high B cells are clustered together towards the left side, whereas UMAP placed these cells at three distinct places pointing to different directions. Next, we examined CD22, which is a surface molecule expressed early during the ontogeny of B cells30. We found that B cells with high CD22 expressions are clustered towards the bottom cluster in the SAUCIE embedding. However, these cells are clustered at the outer rim of one of the clusters of cells in the UMAP embedding, which is counterintuitive  would likely be higher, influenced more by the spatial gradient in protein expression. But if the DR space is not accurate, making the clustering also inaccurate, this correlation will be confused by the abrupt changes in protein expression due to the mixture of cell types, and the correlation will be likely lower. Indeed, we found that SAUCIE is overall the best, in terms of achieving the highest positive correlation overall .Due to the scalability issue of many of these methods, subsampling the whole CyTOF datasets is sometimes necessary for some of these DR methods. For example, a number of methods, including Diffmap, Isomap, KPCA Poly/RBF, LLE, and Spectral, all tend to abort with increasing sample sizes. On the other hand, the original publications of some DR methods  have demonstrated the robustness of their results with regard to subsampling. For the purpose of observing the overall cell state space of the cells (such as major cell populations and their relationships), subsampling, up to a certain extent, will probably not negatively impact the conclusions to be drawn. But all downstream analyses will have to be limited to the subsampled cells and rare cell populations may be missed in the down-sampled subset, which is the detrimental effect of subsampling. So whether to adopt subsampling or not also depends on the purpose of the analyses.To address the scalability issue encountered while benchmarking the accuracy of DR methods, we took a two-pronged approach in our benchmark studies. For all accuracy benchmarks , which is typical for most current CyTOF datasets. In consideration of the possibility that future iterations of the CyTOF technology could incorporate more protein markers, we up-sampled the Oetjen dataset to create up to 60 protein markers while fixing the cell number at 100,000. Then we performed the same benchmark study in terms of runtime Fig.  and memoFinally, we also noticed, during our DR benchmark practices above, that some of these DR methods will abort with errors due to increasing dimensionality. To explicitly quantify the dimensionality threshold, we recorded whether the DR methods would abort while performing the scalability benchmarks of various dimensionalities. Specifically, the methods that produced errors in Fig. Synthesizing all our scalability analysis results, it is obvious that the number of cells, but not number of protein markers, in the CyTOF data is the driving-factor in determining each method’s scalability and thus becomes an important consideration for end users. But unfortunately, CyTOF data inherently profiles at least an order of magnitude more cells than scRNA-seq data. Therefore, optimizing methods for data throughput should be one of the top priorities for researchers.; Wilcoxon Rank Sum test While the accuracy of DR methods reflects their ability to reproduce each sample faithfully in the embedding space, a crucial part of assessing their performance is also to ask whether each method can produce good results consistently. To test the stability of each method, we executed each method on 100 bootstrap samples of the Oetjen cohort’s Sample A (with the same tuning parameters), and calculated the stability of the DR methods, across different bootstrap datasets. Stability is defined as the average Earth Mover’s Distance (EMD) between pairwise distance measures from the embedding from the original sample and the bootstrap samples’ embeddings. This is a measure of overall structural difference globally using the difference in distribution, and the details of implementation are included in the “Methods” section. Smaller EMD values suggest that the DR method produces similar embedding under perturbations of the input. We found that most methods, except for PHATE and LLE, are relatively stable Fig. . The top6aentclass1pt{minimaAnother important aspect of the DR methods related to the stability of computation is how much the results will vary as a function of their tuning parameters, given the same input data. Sometimes, it may be preferable that the DR results do not vary too much even given different parameters. But in other times, the data analytics practitioners may prefer that parameter tuning will lead to more diverse results given different parameters, so that they can choose a parameter setting that yields the optimal DR results. While we leave the choice of preference to the practitioners, we performed a study of parameter tuning. For several top DR performers , we tuned their parameters, as is shown in Supplementary Data While not directly related to the accuracy of DR, it is also important to closely inspect each method’s implementation and evaluate methods based on both optimal user experience and quality of software. Based on our own experience working with each DR method, we employed a clearly defined checklist to score each method’s usability, which includes documentation, automated testing and integration, regular update, cross-platform compatibility, capability of accepting general matrix input, etc  are the overall top runners, proving the need for a CyTOF-specific benchmark study of DR methods. But at the same time, none of DR methods is perfect and there is a large degree of complementarity between them. On the other hand, our comparative results echo some of the previous conclusions regarding the performance of DR methods on scRNA-seq data. For example, previous studies have shown that tSNE performs better than UMAP in terms of local structure preservation on general and scRNA-seq data40. Our analyses on real and simulated CyTOF data demonstrate the same in general.In this work, we present a comprehensive review and benchmark of popular and domain-specific DR methods across many different CyTOF datasets. The performances of the DR methods on the real and simulated CyTOF data are overall comparable, indicating the validity of our evaluation results as well as the simulation approach. Previous comparative works in the field of scRNA-seq have supported the notions that tSNE and UMAP are the top performers in general and even linear methods are well-suited for certain workflowsFrom the perspective of a practitioner working on CyTOF data, our results provide a set of good guidelines and conclusions. SAUCIE and scvis are all-rounders that perform admirably across all categories without major flaws, but SAUCIE is far more efficient than scvis. Thus, it is reasonable to use SAUCIE as part of the general workflow. The one caveat with SAUCIE is its questionable usability: while we would have easily recommended SAUCIE for rapid prototyping given its efficiency, the effort needed to set up SAUCIE renders it inferior to other efficient methods like PCA for this purpose. SQuaD-MDS Hybrid combines the local performance advantage of tSNE with the global performance of SQuaD-MDS, making it an overall excellent method for structural preservation. However, its downstream performance has some weaknesses, especially regarding cell type cluster concordance. On the other hand, UMAP excels in downstream analyses with clustering workflows, but it sacrifices Global structure performance in return. We still recommend UMAP for tasks such as cell typing, clustering, and others that rely on tight and distinct clusters. Both tSNE variants have a unique set of strengths, but given the performance advantage of SQuaD-MDS Hybrid, we would recommend the latter instead for users who seek local accuracy. One key advantage of tSNE is its usability and cross-platform compatibility, which again highlights the inherent advantage of a more established method as compared to newer methods. PHATE is potentially good for some other downstream tasks that involve cell differentiation. On the occasions when Global structure is of concern, linear methods like NMF and PCA still make a strong case, but these methods are not as useful for downstream analyses. Overall, these results suggest that no one method dominates: users should be aware of each’s strengths and limitations. A holistic approach will be to start with a top and efficient method like SAUCIE or SQuaD-MDS Hybrid and then select some other methods for specific tasks and validation (e.g. UMAP for cell typing).Among these DR methods, we observed a polarized phenomenon that those popular DR methods usually have exceptional software and superior documentation, whereas some more accurate but less famous ones are severely lacking in this regard. The latter’s poor usability further aggravates the situation by discouraging practitioners from trying them out, which in turn demotivates the improvement and optimization of such methods. The current prevalent methods, such as UMAP and tSNE, turn out a result of a mix of outstanding performance, great usability, and good scalability. Their success, however, highlights the necessity to strike a balance between strong methodological research and user-oriented development. In light of these observations, we urge researchers to take end-user experiences and the increasing sample sizes of CyTOF into consideration and to develop competitive methods that shine on both theoretical performance and practical user experience. For example, SAUCIE is a neural network, which theoretically should have the capability of mapping new data points into an existing DR space. However, no such mapping functionality is provided in its software implementation yet.5. They also ignore all the sophisticated algorithms developed in recent years for cytometry data 41 and also for scRNA-seq data, which might be applied for analyzing CyTOF data. These new algorithms could reveal complicated biological insights that cannot be afforded by standardized commercial software. For example, it might be of interest to evaluate in the future whether the pseudotime inference algorithms that work well for scRNA-seq data are also suitable for pseudotime inference of CyTOF data24. On the other hand, we envision the DR benchmark results we obtained for CyTOF data in this study could be applicable for traditional flow cytometry data as well, including spectral flow cytometry42, which measures a number of protein markers that is on par with CyTOF, has higher cell throughput, but also still possesses higher spillover.Some researchers seem to analyze CyTOF in the same way as how flow cytometry data have been traditionally analyzed, using commercial software offering simplistic functionalities. They ignore the fact that CyTOF is a much more advanced technique that offers capturing of a lot more protein channels, but has minimum spill-over effects43) can still play a role in accuracy. Their overall lackluster Global performance compared to DL-based methods indicate the need for further theoretical considerations.One curious observation we made is that, while SAUCIE and scvis are less well known than tSNE and UMAP for DR of scRNA-seq data, these two methods turn out to be among the top tier of methods for DR of CyTOF data. This may in part be due to the fact that they are based on deep learning (DL) techniques, and their great performance for CyTOF data could be owing to the tremendous success of DL for handling big data (CyTOF captures much more number of cells than that for scRNA-seq data). Further, SAUCIE’s conventional autoencoder and scvis’s variational autoencoder both assume an underlying latent variable model with smaller dimension for features. CyTOF has much lower dimensionality, in terms of numbers of genes/proteins, which might have rendered this latent space inferred by SAUCIE and scvis more appropriate for capturing the true structure of CyTOF data. Beyond SAUCIE and scvis, many other methods (e.g. UMAP and tSNE) fall under the class of methods that are based on neighbors and optimizing distance measures, rather than DL approaches. Although the range of protein/gene features from CyTOF to scRNA-seq poses no computational concerns for these methods, the curse of dimensionality . CITE-seq can capture much less number of cells than CyTOF, and CITE-seq can only profile the expression of protein surface markers, while CyTOF is capable of capturing the expression of both intracellular and cell surface proteins46. CITE-seq also suffers from drop-outs in the protein expression measurements by the UMI counting approach48, while CyTOF is relatively free from this caveat.The fields of genomics and proteomics have been developing mostly independently. The ability to combine both approaches will be extremely powerful. There has been a growing number of studies generating single cell datasets with both scRNA-seq and CyTOF dataCyTOF possesses a much higher potential of being developed into a clinically applicable diagnostic/prognostic/predictive tool, compared with scRNA-seq, due to its cheaper per-cell cost, shorter workflow, and direct measurement of proteomics changes. Our study of the performances of DR methods on CyTOF data urges the field to rethink analytical strategies for CyTOF data and to develop state-of-the-art tools to address this key gap. This effort should ultimately lead to effective translational application of the CyTOF technology for personalized medicine.All compute- and memory-intensive benchmarking exercises were performed on server nodes with dual Intel Xeon E5-2695 v4 CPUs. Each node has 36 physical cores and was allocated 200 Gigabytes (GB) of RAM, except for all scalability benchmarks, where nodes with the same CPUs but 732 GB of RAM were assigned. All jobs were run in CentOS Linux and managed with Slurm with no GUI to ensure reproducibility. Each computation job was allowed to run for up to 7 days if necessary. Since no method exceeded this time limit, all errors and exceptions reported in the paper resulted from either out-of-memory errors or other unforeseen implementation reasons that we could not control. Statistical analyses, preprocessing, and other workflows for this paper were performed in R 4.0.4 with various consumer-grade hardware running Windows and Ubuntu.https://github.com/KrishnaswamyLab/SAUCIE), GrandPrix 0.1, and SQuaD-MDS (the version on GitHub https://github.com/PierreLambert3/SQuaD-MDS-and-FItSNE-hybrid). One notable method to mention here is Diffmap since its reference implementation is available only in R (“destiny” 2.15.0 with R 3.6.3). As a convenient wrapper, our pipeline and our CyTOF DR Package offer interfaces for most methods. In our package and all analyses in this study, except for the parameter-tuning section, we used all default settings for tuning parameters to ensure fairness across all methods. But multicore optimizations (e.g. the “n_jobs” option in sklearn methods), if available, were enabled to utilize all cores on our server for efficiency.The benchmark pipeline was implemented mostly in Python (3.7 and 3.8) for its extensive support for DR methods and command-line operability. If available, Sklearn 0.24 implementations were used, as they are not only well maintained and but oftentimes also the default choice for Python users. For other DR methods, we utilized the implementations of the original authors if possible, with the exception of tSNE, which will be further discussed. Additional Python DR packages used are: openTSNE 0.6, umap-learn 0.5, zifa 0.1, PHATE 1.0, scvis 0.1, SAUCIE , which is the original authors’ implementation, technically offers faster speed and better memory efficiency, but it comes at a cost of usability with no discernable embedding improvement. Thus, openTSNE is a reasonable compromise that both improves upon efficiency as compared to BH and offers a user-friendly interface with good documentation. We also included “tSNE (sklearn)” in our main figures and benchmark study as it is one of the most popular choices with BH implementation. Other methods included in Supplementary Fig. “tSNE” in the main figures refers to the openTSNE implementation of FIt-SNE. Works such as Kobak et al.For popular as well as top methods from our benchmark, we identified their tuning parameters and performed extensive parameter tuning. We list all the parameters in Supplementary Data 50 for clustering in both the original space and the embedding space. FlowSOM has been the recommended algorithm for CyTOF because of its accuracy and efficiency52. To ensure comparisons of DR methods are fair, we used the same original space clustering labels for each sample during all benchmarks. FlowSOM requires an estimated number of clusters to be provided by the user. For cohorts with provided cell typing information by their authors, we used the number of cell types as prior knowledge. For other cohorts, we determined the estimated number of clusters empirically using cluster variance and the elbow rule. Since FlowSOM also performs a meta-clustering step to consolidate clusters if necessary, we provide a larger number of clusters than the baseline obtained from the above methods . Our approach aims to preserve the granularity of rare cell types while allowing the meta-clustering step to find the optimal clustering. All other settings were kept as default. Our study also includes a few matched scRNA-seq samples from the TuPro cohort and the Oetjen cohort. In these cases, we used the clustering algorithm provided by the Seurat56 pipeline as per the standard workflow for scRNA-seq data analyses. The number of clusters were determined automatically by the Seurat algorithm.For all CyTOF samples benchmarked in the study, we used FlowSOMAdmittedly, choosing different clustering algorithms and different tuning parameters could inevitably lead to a somewhat different benchmark result. This could happen no matter which algorithm and what set of tuning parameters we chose. Therefore, to ensure maximum fairness in good faith, we chose FlowSOM for CyTOF data as FlowSOM has been the recommended algorithm for CyTOF data analyses, and the clustering algorithm provided in Seurat, per the standard workflow for scRNA-seq data analyses. These two choices of clustering algorithms are likely the most common choices for CyTOF and scRNA-seq data users.Our main evaluation framework assesses the accuracy of each DR method’s embeddings. The framework consists of four major categories: global structure preservation , local structure preservation , downstream analysis performance (Downstream), and scRNA-seq concordance. Downstream and scRNA-seq Concordance have sub-categories to reflect different, equally weighted aspects of the major categories. For each method, we utilized the original space expression data, the reduced space embeddings, clusterings before and after DR, assigned cell types, and, if available, the corresponding scRNA-seq data to execute each individual evaluation metric under major and sub-categories. The details and implementations of these methods are described later on in each of their own sections.Category Weight column of Supplementary Data For each dataset, we first ranked all the methods based on each individual metric. Methods that aborted for whatever reason are ranked last for all metrics because they were unable to produce an embedding at all. In our weighting scheme for all metrics, we ensured that all major categories, all sub-categories, and all individual metrics within categories have the same weight . In practice, we first averaged the ranks of individual metrics under each sub-category if applicable. Then, the scores for each sub-category are averaged again as the scores for each major category. The Global and Local Category has no sub-category, and thus we directly averaged their respective metrics’ ranks. Finally, we averaged the scores of major categories to form a final accuracy score. The exact weight of each individual metric as part of its corresponding major category is detailed in the Given the rank-based nature of the accuracy scores, we compared and averaged scores across datasets from different cohorts, including both real and simulation data Fig. . For accThis category aims to assess the degree to which each embedding preserves the structure of the original, high-dimensional data on a large scale. In practical terms, we aim to evaluate the relationship between each cell with not only its neighbors but also other cells that are vastly different from themselves. Metrics under this category are critical in helping practitioners decide whether the overall structure present in an embedding, such as the spatial relationship between a few clusters, is interpretable or significant.20 has called for using pairwise distances in the original and dimension-reduced space to capture the relationship between all cells. The problem of implementing this for CyTOF datasets is that the memory complexity is Previous researchmentclass2pt{minimThis metric measures the Spearman’s correlation of the original and dimension-reduced space using PCD. The PCD matrices from the original and DR space are flattened, and the correlation  as empirical one-dimensional distributions: in practice, we flatten the 20, we perform min-max normalization of each vector to account for the difference in scale . Subsequently, EMD is calculated after normalization. Thus using this metric, we are measuring the overall distributional differences between the original space PCDs and DR embedding space PCDs. In contrast with Spearman’s correlation, which captures the granularity of each cell’s positional change in terms of ranks, EMD treats the PCD distributions as a whole, providing insights on whether the embedding resembles the original data overall. To compute this metric, we used the SciPy implementation of the Wasserstein Distance with default parameters, which computes the distances between empirical cumulative distribution functions of the two vectors.EMDEMD and COR metrics are complementary to each other in the global category. Specifically, COR utilizes each individual cell’s relationship with all the others after DR, which is one perspective of the Global structure. On the other hand, EMD offers a zoomed-out perspective by treating all the distances as a distribution rather than distinguishing among each one of them individually. In practice, we believe that both approaches are important and highly complementary with each other.58 to approximate distances along with using trees to divide data into subspaces, which are represented by a random sample of points, for fast nearest neighbor computations. As with the case of PCD, ANNOY offers a good, practical compromise for CyTOF data in terms of efficiency and accuracy. In our benchmarks, we found that the number of neighbors, We focus on neighbors for local metrics. Namely, we aim to assess whether each cell’s neighbors in the original space are still neighbors after performing DR. This category is important for not only clustering and cell type assignment but also understanding relationships between cells. The conventional approach of finding neighbors using BallTree or KDTree in sklearn proves to be quite slow for large datasets in the case of CyTOF. To realistically benchmark numerous methods on a large number of samples while also offering a quick interface for end users, we chose an approximate but fast algorithm called Approximate Nearest Neighbor Oh Yeah . It utilizes a locality-sensitive hashing algorithmk. The resulting percentage was subsequently averaged across all cells. A larger KNN score close to 1 indicates nearly perfect alignment of neighbors, whereas a relatively small score shows poor neighborhood preservation. Importantly, here we are not using the machine learning algorithm of KNN to define cell types or cell clusters, as in the Random Forest Cluster Prediction metric below. So it is not to be confused with the RF metric below, which is a downstream analysis performance metric.For each cell in the original and embedding space, we found k-nearest neighbors using ANNOY, excluding the cell itself. We define neighborhood preservation by calculating the percentage of neighbors that overlap in both sets. Specifically, for each cell, we took the intersection of the two neighborhood sets from the original and embedding space, and the percentage of preserved neighborhood was calculated by dividing the cardinality of the intersection set by 59, NPE measures how many of the neighboring cells of the same class are preserved. This metric takes into consideration the classification of cells in neighborhoods, which are useful indications of whether clusters are well-separated after embedding. We implemented NPE in house using the following algorithm:We defined For each cell, we found the number of neighboring cells that belong to the same cluster. We used the original space clustering in this step, and again assume that there are The two count vectors in Step 2 were divided by We subsetted in the normalized count vectors with the corresponding cells. This yields subsetted vectors of length Using similar notations with the original authors, we found the kernel density estimates of the vectors from Step 4a, which we call Finally, we calculated the the Total Variation Distance between the distribution for each cluster using the formula For cells belonging to each cluster The total variational distances were then averaged across all clusters to produce the NPE metric.As proposed by Konstorum et al.As a distance-based metric, smaller NPE indicates less error, meaning that each cell’s neighboring cells have similar composition in the original and embedding space. This method belongs to Local instead of Downstream, because it is focused on neighborhoods of individual cells instead of clusterings and cell types on a larger scale.KNN focuses on whether the exact cells in the neighborhood of each cell are preserved between the original data and the DR space, while NPE focuses on whether the neighborhood of each cell in the DR space consists of cells of the same type as this cell, with the “type” or “cluster” of cells defined in the original data. Roughly speaking, KNN operates at the cell level, while NPE works at the cluster/cell type level. They are very complementary with each other overall.A common first step of the downstream analysis pipeline is clustering and subsequently cell type assignment. The Downstream category specifically examines how each DR method affects the clustering performance, which will in turn affect other downstream workflows. Downstream has three sub-categories: Cluster Reconstruction measures how well original space clustering performs on the embedding space data; Cluster Concordance examines clusterings of the original space and the embedding space at the same time; and Cell Type-Clustering Concordance compares manually assigned cell types and embedding-space clustering. For the latter two sub-categories, the same metrics are used, but their purposes, inputs, and interpretations differ.Custer Reconstruction consists of four metrics to assess how the original space clustering performs after DR. While a clustering algorithm optimizes for the original data, the results should be reasonable if a DR method faithfully captures the structure of the original. This is often how such clusterings are visualized. In this sub-category, there are four equally weighted metrics: the first three measure the clustering quality itself whereas the last utilizes the clustering as a practical metric.15 is a measure of how cells are similar, in terms of Euclidean distance, to other cells in the same cluster as compared with the cells in other clusters. A large score close to 1 indicates a good clustering with tight clusters, where on the other hand a negative score means that the cells in clusters are more similar to other cells than those in the same cluster. Davies–Bouldin Index60 measures the average maximum ratio of intra-cluster distance over inter-cluster distance. A good clustering with a good embedding should result in clusters that are both tightly-formed and distinctly far apart, thus resulting in a small score. Calinski-Harabasz Index (CHI) measures variance instead of distance. Specifically, CHI is proportional to the ratio of inter-cluster variance as computed with cluster centroids and the centroid of the dataset over the intra-cluster variance computed with points in the cluster and each cluster centroid.Silhouette Score27, the prediction accuracy is a practical metric to assess whether original clusters are distinguishable. We split the embedding data into 67% training data and 33% testing data. For each DR method, we trained a random forest classifier using sklearn and recorded the prediction accuracy using the testing set. Obviously, a higher accuracy is desired.We also incorporated Random Forest Cluster Prediction (RF) to assess how well a random forest classifier can predict the cluster labels using embedding space data. As proposed by Becht et al.61 and Normalized Mutual Information (NMI)62 to measure how clusterings of the original and DR space align. Given two clustering labels, a pair confusion matrix can be formed by considering every pair of data. The original Rand Index simply finds the ratio of correctly aligned results—the sum of true positive and true negative—over the total number of observations. The ARI adjusts the Rand Index with expected values. Intuitively, a higher ARI close to 1 means that the two clustering results are well aligned, suggesting that DR has lost little information. NMI, on the other hand, treats two clustering results as distributions, and the mutual information between them is computed. The resulting mutual information is then normalized by the entropy of the marginal. For NMI, a larger value close to 1 is preferred, whereas NMI of 0 means that the two clustering results are independent.Instead of focusing on original space clustering, the Cluster Concordance sub-category uses Adjusted Rand Index (ARI)While cluster concordance measures the efficacy of clustering after DR, a valid concern is whether such clustering is concordant with true cell types instead of arbitrary cluster labels. In this sub-category, we utilized cell types provided by the original authors when available as an independent source of true labels. If not, we manually assigned cell types using original space clustering. For good embeddings, we expect that cells of the same type are also clustered together, which also suggests that if practitioners use embedding space clusterings for cell type assignment, they will likely obtain reasonable results. We applied ARI and NMI to cell types and embedding space clusterings for each method. The algorithms are the same as the ones described in Cluster Concordance but with different inputs.For datasets with matched CyTOF and scRNA-seq samples, analyzing these samples in conjunction is a great asset. One critical task in both workflows is DR, and as one would expect, DR results from matched samples need to reasonably be concordant. This will allow practitioners to not only employ a unified workflow for both but also make conclusions based on the same methods and assumptions. Further, many DR methods have indeed been used indiscriminately across different technologies. In this major category with three equally weighted sub-categories, we assessed the versatility of DR methods across technologies.For this category, we assigned cell types for CyTOF samples using embedding space data produced by each DR method. scRNA-seq cell types were assigned with original space data after preprocessing. The goal is to assess which DR method’s results are the most concordant with the reference scRNA-seq data of the same sample.a and cell type b. Then, pairwise distance between a and all other centroids was computed and ranked. The rank of the distance between a and b was then normalized by the number of cell types and recorded. A small rank means that cell type b is close to the cell type a from the vantage point of a. The process is repeated for all pairs and for both the CyTOF embedding of interest and the scRNA-seq reference. One thing to note here is that the rank is not necessarily symmetric because, although the Euclidean distance between any a and b pair is symmetric, the rank obtained in this case depends on neighboring cell types. For example, If a has numerous neighboring cell types, whereas b is a lone outlier, the ranked distance between them from the vantage of a is large. But from b’s perspective, the rank can be much smaller. With each DR method, we have two resulting distance vectors of length m, where m is the number of permutations of cell type pairs. Define X as the vector from a given CyTOF embedding and Y as the vector from the reference scRNA-seq sample. We then defined Cluster Distance as the L1 distance between X and Y.This metric measures how the relationships between common cell types change between CyTOF and scRNA-seq samples. For example, if there are two closely related cell types with similar expressions, we expect them to be close to each other in both settings. To operationalize this concept, we first computed the centroid of each cell type for both CyTOF embeddings and scRNA-seq data, and only common cell types between two datasets are retained. Using permutation, we found all pairs of cell type centroids. Let an arbitrary pair be cell type Given that Cluster Distance is a distance-based metric, the interpretation is simple. A smaller value means that the relationships between cell types are well preserved. Specifically, relationship means the rank distance between cell types. The actual value of the Euclidean distance is not really meaningful given the different nature of CyTOF and scRNA data, but the rank captures the overall structure. This metric was implemented in house and available in CyTOF DR package.a and all cells in cell type b, which are subsequently ranked and normalized by sample size. The same procedures are performed on both the CyTOF embedding of interest and the scRNA-seq reference. This more fine-grained approach is possible due to the fact that EMD compares two distributions. Using the same notation as Cluster Distance, here we compute the EMD between X and Y.Analogous to the contrast between Spearman’s correlation and EMD in Global, here the role of EMD is the same. In this metric, we use the same definitions of pairs of cell types as obtained from permutation. Instead of using only centroids, we calculate the distances between cell type X and Y. As with EMD in Global, this metric offers a good sense of whether the overall structures of cell types as clusters are well preserved using a DR method in CyTOF.We prefer a smaller EMD to suggest good cell type relationship preservation on a global scale. In this case, the distribution of ranks is being measured Gating concordance is a validation metric to ensure that the cell type assignment in CyTOF makes sense for each DR method. Specifically, we manually assigned cell types using both original and embedding space data. For the latter, cell types were assigned using clusterings based on embeddings of each DR method. As with other concordance categories, we used ARI and NMI to assess whether embedding space cell types align well with original space cell types. This sub-category is warranted in the scRNA-seq concordance category because the quality of embedding space cell type assignment is critical. If Gating Concordance is poor, then the results of EMD and Cluster Distance are not as meaningful by assumption. Thus, for a method to have good scRNA-seq Concordance, we expect it to excel in all three categories.N = 147,570). For the reference embedding from the original dataset and all the embeddings from bootstrapped datasets, we computed the PCD for each. As previously described, the PCD algorithm results in a We define stability as whether a certain method’s embeddings remain similar while executed on similar datasets. We bootstrapped Sample A of the Oetjen cohort 100 times and then executed all DR methods on the 100 bootstrapped samples  time” as the measurement of runtime and “Maximum resident set size” as memory usage. Each measurement consists of the benchmark for one method on one dataset configuration, and all runs were sequential to allow methods to utilize as much system resource as possible without aborting. All other variables, such as file input and output, were kept the same across all runs and their impact was overall negligible. We then calculated and reported runtime in minutes and memory in GB for convenience.We defined usability with 13 criteria in five categories related to code quality and user experience. For each method, we manually inspected the quality of documentation, level of software support, the interface available to users, and implementation details that would benefit practitioners. All information is gathered from their websites, GitHub repositories, in-software docstrings and examples, and our own benchmark processes. We scored all methods using a binary scoring system: if a criterion is met, the method receives a score of 1; otherwise, it receives 0. Under circumstances where ambiguity may occur, such as whether the package has been updated as needed, we generally give such edge cases the benefit of the doubt unless clear incompatibility or lack of support is present. The final ranking of usability is based on the sum of scores received by each method Fig. . The exaDocumentation: (1) Examples: The project repository or website provides examples for users to follow. (2) Website: The method has a dedicated website for detailed documentation.Implementation: (1) CLI: The method has a command-line interface that is capable of directly running the DR. (2) Cross platform: The method is easily performed in both R and Python. Installation of packages and alternative implementations are allowed. (3) Import: The package can be imported within its respective software environment. This allows users to integrate methods into their existing workflow without resorting to custom command-line scripts. (4) Matrix: The method takes a general expression matrix or array as input instead of requiring a specific data structure.Maintenance: (1) CI/CD: The package has an automated workflow to test codes with new development. (2) Update: The package has been updated within the past year or as needed.Publication: (1) Built-in: The implementation of the method is built into R or Python without needing any installation. (2) Open Source: The package is open source. (3) Published Package: The package has been published on a standard package repository, such as PyPI, CRAN, or conda.Usage: (1) Mapping: The method has a mapping function that can map new data onto existing embeddings. (2) Robust: The method does not produce unexpected exceptions or errors, except for sample size limitations.10, the Oetjen cohort33, the CyAnno cohort63, the LG cohort64, the Brain cohort65, the BC cohort66, the Levine32 cohort67, the Levine13 cohort67, the Samusik cohort68, the Lung cancer cohort69, and the Covid cohort (CyTOF_00000000000001 to CyTOF_00000000000012 on DBAI). For all datasets, we either obtained their cell typing results from the original publications, or performed cell typing ourselves based on clustering of the cells followed by manual assignment. We showed the manual cell typing results of the Lung cancer cohort, in Supplementary Fig. We accessed a total of 11 cohorts of 110 real CyTOF samples for this benchmark study  unless they have been preprocessed beforehand by the original authors. Our pipeline includes identifying lineage channels, ArcSinh transformation with a co-factor of 5, gating to remove debris using instrument channels, and bead normalization. In general, we did not perform cross-batch normalization to avoid any bias introduced by such algorithms. For the cases where the data have been previously processed, such as the CyAnno cohort, we manually inspected the data and applied specific steps of the pipeline on an as-needed basis: if cell types have already been assigned, we did not gate to remove debris; when not all bead channels were available, we assumed that bead normalization had already been performed on instrument. For all samples, we cross-checked the preprocessing results with DR embeddings of the cells labeled with their cell types.We devised a probabilistic simulation model that captured the key features of the Given the cell type To simulate cellular lineage, we assume that the underlying cell lineage is composed of several non-overlapping cell differentiation trees, where cell type With these setups, the simulated expression level for the In all accuracy benchmarks, we employed two strategies when a method fails due to large sample size. (1) If such a method has a mapping function to accommodate new data (e.g. LLE and Isomap), we initialized and trained the method with 10% subsampled data and then mapped the rest onto the embedding; (2) If a method has no mapping function and aborts during benchmarking, we treated such methods as “Error”, and they will be ranked last. This approach ensures that all evaluations are based on full samples rather than a mix of sample sizes. In a separate benchmark, we also executed all methods on subsampled data: if a method fails in this case, no further subsampling was allowed and they were treated as “Error”. For all subsampling in these benchmarks, we randomly sampled 10% of cells from each sample without replacement.scvis) and without subsampling (denoted as scvis (Full)). With the help of its mapping function, all evaluations were performed with complete samples.Although scvis does not have a memory constraint like LLE and Isomap, its authors recommend training the model using a small sample, as a unique feature of scvis. Thus, we performed scvis both with (denoted as https://dbai.biohpc.swmed.edu/)70.We developed an online web tool for displaying the results of benchmarking DR algorithms on CyTOF data. The technical stack includes HTML, CSS, JavaScript and DataTables. The visualization table to display the benchmarking results of the real and the simulation datasets in this web tool was created with the DataTables jQuery plugin. This table provides searching, sorting and pagination features and progressive bars to visualize the performances of the DR methods according to different metrics. There are two main features in this web tool. The first one is to search for DR results based on multiple input conditions (left column). And the other is to dynamically calculate the average of the performance scores for the user-chosen metrics and to rank the chosen DR methods based on the average scores (top row). This web tool is hosted on the Database for Actionable Immunology (DBAI) website (Computations were mainly performed in the R (3.6.3 and 4.1.3) and Python (3.7 and 3.8) programming languages. All statistical tests were two-sided unless otherwise described. We used the nonparametric Wilcoxon Signed Rank Test with “wilcox.test” and “pairwise.wilcox.test” procedures in R for testing ranking differences between samples in terms of accuracy. All pairwise tests’ p-values were adjusted with the Benjamini–Hochberg Procedure to control the false discovery rate (FDR). For correlation, we used Spearman’s rank-based correlation coefficient.For all rank-based metrics and results, higher ranks indicate superior performance. In case of tied ranks, we employed the maximum rank to allow tied values to all have the highest rank among them. The highest possible rank is 21 while the lowest is 1, unless otherwise noted such as in the case of parameter tuning. Decimal ranks are results of the weighting scheme of our evaluation framework.No statistical method was used to predetermine sample size. We used all CyTOF samples from the original sources, except for the Imaging CyTOF cohort. There were too many samples in this cohort. So to avoid biasing our benchmarking results, we selected the top 14 datasets with more than 5000 cells. This study was not a clinical trial, so the experiments were not randomized. The investigators were not blinded to allocation during experiments and outcome assessment.Further information on research design is available in the Supplementary InformationDescription of Additional Supplementary Files DocumentSupplementary Data 1Supplementary Data 2Supplementary Data 3Supplementary Data 4Reporting Summary