Breast cancer is the most prevalent kind of cancer among women and there is a need for a reliable algorithm to predict its prognosis. Previous studies focused on using gene expression data to build predictive models. However, recent advancements have made multi-omics cancer data sets  accessible. This has acted as the motivation for the creation of a novel model that utilizes a graph convolutional network (GCN) and Choquet fuzzy ensemble, incorporating multi-omics and clinical data retrieved from the publicly available METABRIC Database. In this study, graphs have been used to extract structural information, and a Choquet Fuzzy Ensemble with Logistic Regression, Random Forest, and Support Vector Machine as base classifiers has been employed to classify breast cancer patients as short-term or long-term survivors. The model has been run using all possible combinations of gene expression, copy number alteration, and clinical modality, and the results have been reported. Furthermore, a comparison has been made between the obtained results and different baseline models and state-of-the-art to demonstrate the efficacy of the proposed model in terms of different metrics. The results of this model based on Accuracy, Matthews correlation coefficient, Precision, Sensitivity, Specificity, Balanced Accuracy, and F1-Measure are 0.820, 0.528, 0.630, 0.666, 0.871, 0.769, and 0.647, respectively. It can be described as a censored survival analysis issue, which is used to determine whether and when an event (like a patient’s death) will occur within a given time frame2. The five-year survival rate is a frequently used statistic for the prognosis of breast cancer3. Prognostication of a patient’s life expectancy is crucial because when a sufficiently accurate prognosis estimate is supplied, it helps in clinical decision-making5. Moreover, breast cancer is a complex disease with diverse clinical outcomes, making prediction and treatment challenging. Such as the high dimensionality of multimodal data poses difficulties for medical professionals in manual interpretation. Hence, the development of computational algorithms becomes crucial in accurately predicting the prognosis of breast cancer. These algorithms can assist physicians in determining the most appropriate treatment plan for patients, emphasizing the significance of such approaches in clinical decision-making.Breast cancer is a very heterogeneous disease that affects women of different ages. The breast consists of different tissues such as fatty tissue and dense tissue, comprising lobes, lobules, and milk glands. Breast cancer occurs when breast cells multiply uncontrollably, leading to tumor formation. If breast cancer spreads to other organs, it becomes metastatic. There are two biological types: invasive breast cancer, which spreads to adjacent tissues or distant organs, and non-invasive breast cancer, which remains confined to the lobuler portion of breasts. According to the report given by WHO, there were 6,85,000 deaths worldwide and 2.3 million new cases of breast cancer in women in 2020. Therefore, early prognosis prediction is necessary for more specific therapy and treatment. In this context, calculating a patient’s likelihood of survival is a significant problem with cancer prognosis prediction6 developed a prognosis prediction system using support vector machines by including a recursive feature reduction strategy based on gene expression data. Furthermore, Nguyen et al.7 created a prediction model with random forest (RF) to select features that outperform the pre-existing Wisconsin Breast Cancer Prognostic model. Yet, multi-modal modeling techniques have demonstrated that incorporating many forms of data can improve forecast accuracy. Regarding multi-modal data, Sun et al.4 have suggested a methodology that employs a hybrid signature made up of Clinical and Gene expression data. Further, a probabilistic model which includes Bayesian Network was proposed by Gevaert et al.5. They considered genetic and clinical data for their study. Moreover, the recent advancements in deep learning methodologies showed that the models which use data with multiple modalities often perform better than uni-modal models. Sun et al.8 established this fact by constructing a deep neural network-based model by incorporating multi-omics and clinical data. Furthermore, Sun et al.9 presented the GPMKL model for determining the breast cancer survival prognosis by integrating pathological images with genomics data. Hsu et al.10 also combined gene expression data with clinical data for the breast cancer survival prediction study. Further, Arya et al.15 also introduced some new models by incorporating deep learning-based architecture for this study. However, predicting cancer survival using clinical and multi-omics data remains challenging due to limited feature size and complex structured data. Therefore, thoroughly exploring clinical and multi-omics data using various machine-learning techniques is crucial for accurate prognosis estimation in cancer research. Here the details of some of the breast cancer survival prognosis prediction studies in recent past years have been depicted in Table In past studies, the problem of forecasting a cancer patient’s prognosis was addressed by many researchers. However, due to the complexity or implausibility of combining many sources, many prognostic prediction studies typically rely on a single type of data. Such as Xu et al.12 utilized a Convolutional Neural Network (CNN)17 for feature extraction as it can generate a comprehensive stacked feature set by producing a large number of hidden features from input data. However, CNNs are limited in that, they only consider data points that are neighboring based on the kernel size and do not account for other data points that may have high correlations but are not directly connected. This is a significant limitation of their approach. Whereas, graph convolution takes into account the irregular structure of data and selects neighboring nodes for information sharing and convolution based on their correlations. Notably, the number of neighboring nodes can vary depending on a data point’s connectivity with other nodes, making information-sharing dynamic and independent of fixed kernel size. Moreover, typically, a variety of ensemble learning techniques lead to positive predictive outcomes18. Such as, after examining state-of-the-art model MDNNMD8, it has been noticed that a classifier level ensemble has been utilized to compute the result. This is a simple method where each classifier in the ensemble makes a prediction, and the end result is based on the weighted vote of the classifiers. But in the proposed model, the utilization of the Choquet fuzzy integral has been considered to use the classifier’s probabilistic results for producing the end predicted values. The Choquet fuzzy integral is a more sophisticated approach that can produce more accurate results by taking into account the degree of agreement among the classifiers and handling non-linear relationships among them. Recent study 19 has provided compelling evidence that fuzzy-based ensemble techniques outperform simple ensemble techniques.Even if the aforementioned deep learning-based methods for predicting cancer survival have produced promising results, the subsequent area still has a lot of room for growth. The question of exploiting the structural information between samples’ underlying linkages has not been addressed in prior studies. To address this issue, considering muti-omics and clinical data, a novel study using Graph Convolutional Network with a Choquet Fuzzy Ensemble (ChoqFuzGCN) has been proposed for the survival prediction of cancer patients. It is also observed that there exist certain drawbacks in some of the latest proposed models. For instance, the authors of a recent work20 to each graph. After obtaining the feature embedding of each sample, a Choquet Fuzzy Integral-based ensemble21 of three base classifiers, namely Random Forest (RF)22, Support Vector Machine (SVM) with Radial Basis Function (RBF) Kernel3, and Logistic Regression (LR)23, has been applied. This fuzzy ensemble uses the probabilistic outcomes of different classifiers to obtain the prediction value. Here a threshold value (5 years) has been chosen to make the distinction between the two prediction classes. Patients who survived more than five years are considered long-term survivors whereas patients who survived lesser than five years are considered short-term survivors.In this work, graphs have been created using different modalities of data. The graphs investigate the natural correlations between samples or patients. Also, embedding representation for all samples was computed by applying the graph convolution techniqueIn brief, the contributions made by the current study are as follows:CLN), Copy Number Alteration (CNA), and Gene expression (GE) data. To compare the performance improvement, experimental results using multi-omics and clinical data have been reported. Also, the results based on individual modality have been reported.Contribution I: From single to multi-omics approach: The aim of addressing whether transitioning from a single-modality approach to a multi-omics approach works well has been pursued. An extensive study has been performed using Clinical https://www.cbioportal.org/study/summary?id=brca_metabric) has been taken for this study. The pre-processed version of the dataset is available on Github: (https://github.com/USTC-HIlab/MDNNMD). This dataset consists of three different modalities, namely, Gene Expression data (GE), Copy Number Alteration (CNA) profile, and Clinical data (CLN), and each has around 24000, 26000, and 27 features, respectively. The dataset has 1980 samples. Among these, there are 491 short-time survivors and 1489 long-time survivors. Here, survivors with fewer than five years are referred to as short-time survivors, and those with more than five years are referred to as long-time survivors. Features from different modalities play a crucial role in capturing distinct biological and clinical information. GE features highlight coordinated gene expression changes based on different classes like cancerous or non-cancerous. CNA profiles provide insights into genomic alterations, such as amplifications or deletions, aiding in the identification of important genes. On the other hand, CLN data, encompassing patient demographics and clinical variables, offer contextual information that complements molecular data and enables personalized prognosis and treatment decisions. Integrating these diverse features enhances predictive models and improves disease understanding.The pre-processed version of the publicly available METABRIC dataset  value, and a search is conducted to select the best N features . Ultimately, 400 genes from gene expression profiles and 200 genes from CNA profiles are chosen.Each patient in the study is associated with 27 clinical features, including variables such as age at diagnosis, tumor size, and lymph nodes positive. From these clinical features, a final set of 25 features is selected for analysis. To address missing values in the GE profile and CNA profile data, a weighted nearest neighbor algorithm was employed for estimation. The GE features were normalized and discretized into three categories: under-expression  Kernel3, LR23) for breast cancer patient categorization into long-term and short-term survivors. Different graphs have been constructed for incorporating structural relationships among data instances. For the construction of the graph, data instances are considered as nodes and edges have been formed by computing correlation among the instances. A graph has been formed for gene expression data (GE). Also, another graph was formed for copy number alteration data (CNA). Here, based on experimental findings, a certain correlation threshold (0.3 for GE and 0.6 for CNA) has been chosen to create an edge between any two nodes. In this case, Pearson correlation25 has been used. Furthermore, convolution20 has been applied to each of the graphs. This convolution technique helps to learn the irregular structure of the data which may be present among connecting nodes. Then, the end node embedding has been recovered, and the feature vector has been retrieved following the completion of the training of the model using GCN. The feature vectors from the various modalities have been combined or stacked to generate the final feature vector. Here, GCN struggles to provide proper node embedding for Clinical data because it has a substantially smaller number of characteristics. Therefore, to build the stacked feature set, clinical data has been directly concatenated with the derived features from the other two modalities. The final classification has been carried out via an ensemble of base classifiers using Choquet Fuzzy Integrals21. The selection of these base classifiers is primarily motivated by their widespread usage in state-of-the-art26 methods within the field. Random Forest (RF)22 works by constructing multiple decision trees using bootstrapped samples from the dataset and aggregating their predictions through voting or averaging, resulting in a robust and accurate classification or regression model. Logistic Regression23 works by fitting a logistic function to the input data, estimating the probability of binary outcomes, and making predictions based on a specified decision boundary. On the other hand, SVM with RBF3 kernel is a valuable approach when dealing with datasets that exhibit nonlinearity, as it employs a transformation to a higher-dimensional space, enabling the identification of an optimal hyperplane that maximizes the separation between distinct classes.In this proposed work Fig. , a GraphThis study presents an original contribution by introducing a prediction model that utilizes a Graph Convolutional Network (GCN) to extract important features from multi-omics data. Moreover, a novel ensemble method based on Choquet Fuzzy Integrals (CGI) is proposed, which combines the probabilistic outputs of individual classifiers to produce the final prediction. By leveraging the inherent uncertainty in the decision scores, the Choquet Fuzzy Integral-based ensemble enhances the prediction performance. The CGI is considered a dynamic technique as it enables the dynamic integration and aggregation of contributions of base classifiers in the final ensemble. It takes into account the varying levels of uncertainty, importance, or relevance associated with each base classifier. Therefore, this model not only captures essential features from multi-omics data using GCN but also improves predictions through the integration of Choquet Fuzzy Integrals, effectively leveraging probabilistic outputs and enhancing overall accuracy.27 are able to handle irregular data patterns. It is a semi-supervised technique. These networks consider non-Euclidean structured data and leverage the examination of neighboring nodes to extract features. Consider a graph G with the sets of nodes and edges V and E, respectively. In this instance, the adjacency matrix, represented as A, stands for the edge connections between nodes. The forward pass equation  and performing the dot product operation of the inverse of D with AX , kernels are used for learning the features of the neighboring cells by moving over the entire dataset. As the shape of the kernels remains fixed, CNN is able to learn the neighboring features by considering the dataset as a regular structure. However, to explore the nonregular pattern which may exist in a dataset, CNN is not a helpful technique. In such case, graph convolutional networkstion Eq.  for the on20 Eq.  from nortion Eq.  is:4\doc23, support vector machine3, and random forest22) has been proposed, utilizing the Choquet fuzzy integral method21. The pre-eminence of individual base classifiers towards the final prediction goal is determined with the help of this approach. The input of the fuzzy fusion is constituted by the confidence ratings obtained from various classifiers. In this fusion approach, the decision scores’ uncertainty, which refers to the level of confidence or uncertainty associated with the predictions made by a classifier, is considered as an additional piece of information that is available for the fusion process.In this study, the ensemble of three machine learning base classifiers (logistic regressionS), it will have maximum pre-eminence, i.e., If all classifiers are present in the combination  of classifiers are mutually exclusive  and Copy Number Alteration (CNA) modalities, where samples are represented as nodes. The presence of an edge between two nodes is determined based on the Pearson correlation exceeding a specific threshold value (0.3 for Gene expression and 0.6 for CNA ) derived from experimental observations.In this research, separate graphs are constructed for the gene expression  due to its limited number of attributes, which is insufficient for constructing a graph to convey structural information.The final node embeddings of the CLN features are combined with the extracted features from the final layer of the trained GCN models for the GE and CNA modalities. These features are then concatenated together for further analysis.The 28 on the concatenated data.Further, up-sampling is applied using Synthetic Minority Oversampling Technique (SMOTE)21. As base classifiers, we have used Random Forest (RBF Kernel)22, Support Vector Machine3, and Logistic Regression23.Following that, an ensemble of base classifiers employing Choquet Fuzzy Integrals performs the final classificationThe following steps show the suggested method’s working mechanism. N. kth weight matrices where K total weight matrices in existence.The proposed model is implemented using a supervised setting. The over-fitting of the model is addressed in this instance using the L2 regularization procedure. The loss function applied in this scenario is the cross-entropy loss shown in Eq. . In thisO(D). For N data instances, the number of computations: Computational complexity for graph construction: This model incorporates graph construction using Pearson correlation computation between every pair of samples. Let’s consider training dataset size N of dimension F. So, Pearson Correlation computational complexity between any two instances with dimension D is lth layer, the number of computations for matrix multiplication in a dense layer will be of size 29. So, for I number of training iterations, it will be Computational complexity of graph convolution : Consider a graph 30.Computational complexity of ensemble method : Let’s, consider the number of classes is C, and the number of classifiers is M, then the complexity of the Choquet fuzzy integral is Computational complexity of the model training: The overall complexity depends on the size of the training dataset (N), dimensionality (D), number of layers (L), number of classes (C), number of classifiers (M), and number of training iterations (I).We have discussed various experimental findings in this section.31, Precision (Pre)31, Matthews correlation coefficient (Mcc)31, Sensitivity (Sn)31, Specificity (Sp)31, F1Measure32, and Balanced Accuracy 33 values are taken into account as assessment metrics. The Matthews correlation coefficient is a more reliable metric. Only if the prediction yields accurate outcomes in terms of all true positives, false negatives, true negatives, and false positives values, then Mcc produces a high score. As one class appears significantly more frequently than the other, Balanced Accuracy has been provided because this statistic is useful when there is an imbalance between the two classes. It is the Sensitivity and Specificity’s arithmetic mean. F1-Measure is also provided a better assessment of model performance if the dataset is imbalanced.Accuracy (Acc)CNA. Ten-fold cross-validation is used to train the suggested model. These threshold values are determined empirically using the validation dataset as a base. Here, we have constructed GCN with three hidden layers containing 200, 150, and 100 units, respectively. We have used the ReLU activation function in hidden layers and the Softmax activation function in the output layer. The number of training epochs is 200, and the learning rate is 0.001. Adam optimizer has been used here. The loss function consists of binary cross-entropy with L2 regularization. Here, grid search has been used to find the optimal parameters of the base classifiers. In the case of the random forest, the parameter ‘n estimators’ (number of trees in the forest) has been set within the range of 50 to 500, while the ‘maximum depth’ and ‘minimum samples split’ parameters have been kept at their default values. It has been found that 70 is the best value for ‘n estimators’. In the case of SVM with RBF kernel, and logistic regression, default parameters have been considered for the experiment.Python (version 3.9) is utilized to implement the suggested strategy. To extract features, GCN is applied, and to predict class labels, Choquet Fuzzy Integral-based ensemble technique is used. Using the Tensorflow 2-based Steller API, the GCN model is created and trained in this experiment. The correlation thresholds of 0.6 and 0.3 were selected, respectively, to build the graphs for the data on GE and CLN+GE+CNA) consistently outperformed other combinations of modalities. We also plotted the ROC curve (Receiver Operating Characteristics curve)31 prediction are shown in Table see Fig.  and obse23, Random Forest (RF)22, Support Vector Machine (SVM)3, and Majority Voting Ensemble34 on the extracted features from GCN layer. In the case of the Majority Voting Ensemble, we have used LR, SVM, and RF as base classifiers. Whereas, Radial Basis Function (RBF) Kernel of SVM has been used here. Table The proposed model has been compared with Logistic regression (LR)STACKED RF12, SiGaAtCNN+ Input STACKED RF11 and  MDNNMD8. STACKED RF has used Convolution neural network-based architecture whereas  SiGaAtCNN+ Input STACKED RF has used sigmoid gated attention convolutional neural network. Here, all the results are reported based on 10-fold cross-validation architecture. As per the results reported by the  MDNNMD’s work, which is deep neural network-based architecture, they have considered two separate stringency levels: Sp MDNNMD in terms of Sensitivity (0.666), Balanced Accuracy (0.769), and F1-Measure (0.647). When the stringency level is Sp MDNNMD with 46.6%, 17.4%, and 32.1% improvements in Sensitivity, Balanced Accuracy, and F1-Measure respectively. Similarly, when the stringency level is Sp MDNNMD with 21.6%, 6.9%, and 8.5% improvements in Sensitivity, Balanced Accuracy, and F1-Measure respectively. The comparison of the proposed model with SiGaAtCNN+ Input STACKED RF11 reveals that ChoqFuzGCN model has shown improvements of 3.2%, 14.3%, 6.5%, and 9% in Precision, Sensitivity, Balanced Accuracy, and F1-Measure values, respectively. Similarly, when compared to the STACKED RF model, ChoqFuzGCN model has exhibited performance improvements of 12.2%, 3%, and 2.5% in terms of Sensitivity, Balanced Accuracy, and F1-Measure, respectively. Furthermore, it is evident from all comparisons that F1-Measure and Balanced Accuracy are consistently superior to the existing models. Here, a high F1-Measure demonstrates that positive samples’ accurate prediction  is superior to the other techniques. Furthermore, the high Balanced Accuracy demonstrates that the suggested model is a superior method for addressing imbalanced data. Here, both the true positive and true negative predictions are acceptable.In Table The Cancer Genome Atlas Program for Breast Cancer (TCGA-BRCA) dataset, which has been downloaded from https://xenabrowser.net/datapages/. A total of 1035 breast cancer patients’ private information has been utilized for the investigation. This dataset encompasses information on copy number variants, cancer-related gene expression, and clinical details. The data is preprocessed following a similar methodology as discussed in the “Dataset” section. Similarly, as described in the “Data preprocessing” section, the data is preprocessed. The TCGA-BRCA dataset consisted of 239 samples categorized as short-time survivors and 796 samples as long-time survivors. With the preprocessed and labeled version of the TCGA-BRCA dataset, the ChoqFuzGCN model is trained and validated using tenfold cross-validation. The observed findings of the model, along with those of other approaches, have been presented in Table The viability of the proposed model has been validated using the 35 was performed on the recorded Balanced Accuracy, Sensitivity, Specificity, and F1-Measure using the scipy library. The results have shown that the obtained t-values (t-test) of 9.45, 26.4, 38.5, and 3.7 are statistically significant in terms of Balanced Accuracy, Sensitivity, Specificity, and F1-Measure, respectively, and p-values of 0.00 were obtained for all the mentioned metrics. The t-values of the ANOVA test are 438.54, 501.23, 612.23, and 396.12 for Balanced Accuracy, Sensitivity, Specificity, and F1-Measure, respectively, and the p-values are 0.00. These t-values and p-values indicate that our suggested model holds statistical significance and will be beneficial for the estimation of breast cancer survival.The validity of this prediction task can be considered established when the findings from tests such as the t-test and ANOVA are statistically significant. To validate the proposed study, the STACKED RF model and ChoqFuzGCN model were executed 10 times on the METABRIC dataset. Subsequently, t-testIn the proposed work, we have introduced a Graph Convolutional Network-based Choquet Fuzzy Ensemble model to correctly predict breast cancer prognosis. The obtained Accuracy, Matthews correlation coefficient, Precision, Sensitivity, Specificity, Balanced Accuracy, and F1-Measure values from this model are 0.820, 0.528, 0.630, 0.666, 0.871, 0.769, and 0.647, respectively. If uni-modal CLN modality is considered, it is capable enough to provide a high Sp value. In other words, we can say that the CLN modality contains certain clinical features about the patients which help the model to correctly identify the long-term survivors belonging to class 0. But, this high Sp value (0.863) is obtained at the cost of low Sensitivity (Sn) value (0.316) means miss-classification of short-term survivors. The biological interpretation of this case suggests that the oncologist can provide a less aggressive treatment plan for the long-term survivors while relying on the CLN modality only. Unfortunately, they might end up following a similar treatment plan for short-term survivors also due to the miss-classification of short-term patients as long-term patients. Now considering the bi and multi-modal combinations where CLN modality is present achieves better Sn values while maintaining comparable or much higher Sp values, respectively when compared with CLN  modality. When using multi-modal approaches, the phenomenon of increased Sensitivity without losing Specificity occurs where the supplementary modalities provide complementary information for positive instance detection. Hence, incorporating additional modalities is necessary for a better survival prognosis for both classes. It is evident that the integration of all three modalities in the proposed model has led to improved performance and achieved better results across various metrics. The expanded investigation combining different modalities has been done to enhance the overall study (see Table


With the emergence of hundreds of single-cell RNA-sequencing (scRNA-seq) datasets, the number of computational tools to analyze aspects of the generated data has grown rapidly. As a result, there is a recurring need to demonstrate whether newly developed methods are truly performant—on their own as well as in comparison to existing tools. Benchmark studies aim to consolidate the space of available methods for a given task and often use simulated data that provide a ground truth for evaluations, thus demanding a high quality standard results credible and transferable to real data.Here, we evaluated methods for synthetic scRNA-seq data generation in their ability to mimic experimental data. Besides comparing gene- and cell-level quality control summaries in both one- and two-dimensional settings, we further quantified these at the batch- and cluster-level. Secondly, we investigate the effect of simulators on clustering and batch correction method comparisons, and, thirdly, which and to what extent quality control summaries can capture reference-simulation similarity.Our results suggest that most simulators are unable to accommodate complex designs without introducing artificial effects, they yield over-optimistic performance of integration and potentially unreliable ranking of clustering methods, and it is generally unknown which summaries are important to ensure effective simulation-based method comparisons.The online version contains supplementary material available at 10.1186/s13059-023-02904-1. Single-cell RNA-sequencing (scRNA-seq) has become an established tool for studying the transcriptome at individual cell resolution. Since the first scRNA-seq study’s publication in 2009 , there hIn this context, simulations have become an indispensable tool, for example, to investigate how methods respond to varying parameter inputs and quantify their scalability in terms of computational cost, as well as ensuring that a method is performant across a range of scenarios and in comparison to other available tools. The attractiveness of simulation studies is largely due to being able to specify a ground truth, which is often challenging or infeasible to establish in experimental data . For exasynthetic data. On the one hand, conclusions drawn from simulation studies are frequently criticized, because simulations cannot completely mimic  experimental data. On the other hand, it is often too expensive, or even impossible, to generate experimental data that is suitable for formal performance comparison. Nonetheless, setting a high-quality standard for simulations is all the more important to ensure results based on them are transferable to corresponding experimental datasets.By definition, simulations generate Typically, new simulation methods come with minimal  benchmarks that focus on one-dimensional evaluations, i.e., how similarly a set of summaries is distributed between a reference and simulated dataset . In somMethods for simulating scRNA-seq data may be categorized according to various factors. Most importantly, there is a dichotomy between methods that generate synthetic data de novo and those that rely on a reference dataset. The former depend on user-defined parameter inputs to generate counts, and introduce artificial effects between, e.g., different groups of cells or samples. Conversely, reference-based methods estimate parameters to mimic the gene expression profiles observed in the reference dataset. However, many methods employ a hybrid framework where, e.g., baseline parameters are estimated from a “singular” reference , and additional layers of complexity  are added post hoc. Both strategies have their advantages and disadvantages: de novo simulators offer high flexibility in varying the strength and specificity of different effects, but might not generate realistic data; in contrast, reference-based methods are limited to the complexity of the input data and consequently less flexible, but are by default more realistic. Taken together, there is a trade-off between how applicable methods are in benchmarking single-cell analysis tools across a wide range of scenarios versus whether simulation study results are directly transferable to real data.Here, we evaluated 16 scRNA-seq simulation methods in their ability to replicate important aspects of a real reference dataset. We considered various global, gene- and cell-level summaries, and compared them between reference and simulated data, in both one- and two-dimensional settings. In addition to global distributions , we made batch- and cluster-level comparisons to capture structural differences in the summaries.Our results suggest that there is a noticeable shortage of simulators that can accommodate complex situations and that popular simulators do not adequately mimic real datasets. In particular, some current methods are able to simulate multiple groups of cells , but do so in an ad hoc manner, e.g., by introducing arbitrary differences based on parameter inputs. Few methods attempt to estimate and mimic group effects from reference datasets, but this comes at a loss of supplying a ground truth.n are “singular” references that contain cells from a single batch and cluster; type b contain cells from multiple batches; and type k contain cells from multiple clusters . Each method was only run on corresponding reference datasets. A more detailed overview of the computational pipeline for this study is given in Fig. Because different simulators can generate different levels of complexity , we tagged each method according to their capabilities , or a combination of simulated and real data, where an experimental ground truth exists. For each benchmark, we summarized the task of interest and, if any, which simulator(s) are used. Across all considered benchmarks, these amounted to only five, namely uscat (1 ), scDesiesign (1 ), powsimwsimR 2 , 23), sc, scmusca[scDD 3 , 11, 24), 24muscatter 13 , 25–34).–34.muscatter , coefficient of variation, gene detection frequency, gene-to-gene correlation, log-transformed library size , cell detection frequency, cell-to-cell correlation, local density factor, cell-to-cell distance, and k-nearest neighbor (KNN) occurrences  at the gene-level ; and the silhouette width [b/k)—that aim to capture global batch or cluster effects on gene expression variability and cell-to-cell similarity, respectively.Since some summaries  can vary between batches  and clusters , we computed them globally, i.e., across all cells, as well as for each batch and cluster. Thus, for a given dataset with te width  and cellte width  at the cTo evaluate simulator performance, we compared summaries between reference and simulated data in one- and two-dimensional settings by computing the Kolmogorov-Smirnov (KS) distance  and Wasssplatter ranked in the middle for the majority of summaries. scDD ranked poorly for most summaries, preceded by hierarchicell and ESCO. Considering all summaries, ZINB-WaVE, scDesign2, and muscat were among the best performing simulators, yielding low KS test statistics across a large number of metrics and datasets .Finally, we ranked simulators according to their overall performance. In order to weight datasets equally and independently of the number of subsets drawn from them, we first averaged statistics across subsets, then datasets. Secondly, because simulators ranked similarly in both one- and two-dimensional comparisons, and performances were often linked for certain metrics, we limited rankings to one-dimensional evaluations only, and averaged across all gene- and cell-level metrics. This resulted in three independent rankings, one for each set of methods that can accommodate a given simulation type Fig. . Notablyn, ZINB-WaVE, scDesign2, muscat, and SPsimSeq performed similarly well, with POWSC, ESCO, hierarchicell, and scDD ranking last across various summaries. ZINB-WaVE and muscat were also the most performant among type b and k simulators, joined by SPARSim and scDesign2, respectively. LDF, cell-to-cell distance and correlation , and global summaries (PVE and silhouette width for type b and k) were poorly recapitulated.For type scDesign2, ZINB-WaVE, and zingeR was relatively slow, but data simulation was not. In contrast, scDD and SymSim took longer for simulation than estimation. Overall, BASiCS was by far the slowest. SPsimSeq, SPARSim, SCRIP, and SymSim were approximately tenfold faster. The remaining methods  were the fastest. Memory usage  cell-specific mixing scores (CMS) and (ii) difference in local density factors  should be the same for experimental and simulated data. In order to investigate how method comparison results are affected by simulation, we used the 8 type egration . In ordeSPsimSeq, ZINB-WaVE, SPARsim, and SCRIP gave results most similar to real data, followed by BASiCS, and lastly muscat and SymSim  gave almost identical F1 scores and rankings for multiple references. Overall, method rankings (according to F1 scores) were lowly correlated between simulated and reference data, as well as between simulations . Consequently, methods to correct of such effects (integration) or group together similar cells (clustering) perform over-optimistically in simulated data compared to more complex and noisy experimental data and are almost indistinguishable in their performance for ‘simple’ datasets.Inevitably, summaries used to assess whether simulated data mimics real data may be redundant, and we expect that some summaries are more important than others. To quantify the relationship between summaries, we correlated all comparable summaries, i.e., gene- and cell-level summaries, respectively, excluding those that include sampling, i.e., correlations and cell-to-cell distances Fig. a. Gene dNext, we correlated the KS test statistics obtained from comparing reference-simulation pairs of summaries across all datasets Fig. b. SummarTo investigate the overall similarity of summaries, we performed multi-dimensional scaling (MDS) on KS statistics across methods and datasets  on test statistics of summaries across methods and datasets , and there may exist other informative summaries not considered here.Taken together, our analyses suggest that several summaries convey similar information, with gene-level summaries being particularly redundant, and global summaries the least redundant. Accordingly, simulator performance may be sufficiently quantifiable by a combination of one gene-level, few cell-level, and various global summaries. However, the number (and nature) of summaries to comprehensively cover the inherent data structure is dependent on its complexity  structure, and stratified simulators according to type .In this study, we compared scRNA-seq data simulators in their ability to generate synthetic data that can recapitulate key characteristics of real reference datasets. We considered a range of gene- and cell-level summaries, as well as ones specific to capturing local and global group-effects . By comparing the distribution of summaries as well as pairs thereof between reference and simulated data, we evaluated how well simulations capture the structure exhibited by a given reference dataset. We ranked simulators by averaging their performance across summaries and simulations, thus evaluating each method across a multifaceted range of criteria  and datasets (from various tissues and technologies). Our results confirm some aspects of those in the study from Cao et al.  , the vast majority of methods can simulate only one group of cells . As a result, most current simulators are rather limited in their applicability.Both neutral benchmark studies as well as non-neutral comparisons performed alongside newly presented methods rely on a ground truth for evaluation. Thus, future work should be focused on the development of flexible, faithful simulation frameworks to fill this gap, especially in scenarios where an experimental ground truth is challenging or infeasible to establish. For example, which genes and cell subpopulations are affected by batch effects cannot be controlled, independent of whether control samples might be used to quantify these effects. Similarly, intra- and inter-cluster effects are unclear, even if cluster annotations might be obtained through cell-sorting or manual annotation by an expert. And, effects on gene expression remain unknown, despite controlled perturbation or time-series studies through which discrete labels might be given. Taken together, although some level of ground truth may be experimentally attainable, simulations remain indispensable owing to (i) their feasibility and (ii) the information they provide .realistic changes, non-reference based simulations are difficult to evaluate, and conclusions drawn from de novo simulations in terms of method evaluations should be treated with caution.The most truthful model for real data is real data. Artificial data alterations  are unlikely to mimic biological differences. Even if founded on a thorough investigation of scater [countsimQC [While tools to evaluate the quality of simulated data exist, they are seldomly taken advantage of. For example, scater  offers auntsimQC  can genemuscat was among the highest ranked simulators in our study, but integration and clustering method ranking obtained from muscat simulations were rather inconsistent with those from real data. On the other hand, SPsimSeq ranked mediocre in terms of mimicking real datasets, but gave the most faithful integration method ranking. In the context of clustering, there was a consistent over-optimistic performance of methods, independent of the simulator used.In addition to evaluating the faithfulness of simulated data, we investigated whether and to what extent benchmark results are affected by the simulator used. Our results suggest that method performances for integration and clustering of scRNA-seq data deviate from those obtained from real data; in addition, simulators that better mimic reference datasets do not necessarily yield more similar method comparison results. For example, This discrepancy between the faithfulness of simulated data and benchmark results brings to question which set of summaries is sufficient to capture relevant data structure. Here, simulators were ranked by their average performance across summaries. However, many of these may be redundant (see below) or differ in their suitability to capture group-related structures . Thus, simulators that are performant “overall” are not guaranteed to be suitable for evaluating methods for a specific task , where global structure should take priority over gene-/cell-specific summaries. An open question that needs to be answered is what summaries are important for a given task.splatter offers a well-documented, easy-to-use framework that is both flexible and interpretable. While other methods might outperform splatter, they return parameters that are less applicable to benchmarking computational tools. For example, artificially introducing DE genes provides a binary ground truth , whereas estimating and mimicking cluster effects might not .Besides the capabilities each method has to offer and its performance, i.e., how realistic its simulations are, there are other criteria we did not explore thoroughly. For example, n), multi-batch (type b), and multi-cluster (type k) datasets. However, there are various methods that are aimed at simulating data where gene expression profiles evolve along a discrete or continuous trajectory or time-course . These  [SERGIO .TedSim [The combination of scRNA-seq with CRISPR/Cas9 genome editing has enabled the joint readout of gene expression and cell lineage barcodes . SalvadoTedSim , is capaMost of these methods employ fairly sophisticated and well-designed models for data generation, but require a complex set of inputs that is specific to each method and difficult to justify. Meanwhile, very few trajectory simulators support the estimation of simulation parameters from a reference dataset, making it challenging to evaluate them and opening the question of how faithful performance assessments based on them are. Overall, validating the faithfulness of synthetically generated trajectories in single-cell data remains challenging.Taken together, while a set of performant methods to generate synthetic scRNA-seq data exist, current methods are (i) limited in the level of complexity they are able to accommodate; (ii) often reliant—in full or in part—on inputs by the user to introduce  expression differences; and (iii) more or less suitable to evaluate other tools, depending on the data characteristics they can capture faithfully. Secondly, simulation-based benchmark studies are affected by the simulator used, and more performant simulators do not necessarily yield more reliable readouts of, e.g., integration and clustering methods. And thirdly, the chosen quality control summaries and their prioritization have an impact on the assessment of simulations and, consequently, the conclusions drawn from them. Thus, identifying the nature, number, and significance of summaries to faithfully capture scRNA-seq data structure warrants future work in order to improve method evaluations.ExperimentHub [SingleCellExperiment [as is otherwise. Datasets cover various organisms, tissue types, technologies, and levels of complexity . A summary of each dataset’s characteristics and source is given in Additional file Each reference dataset was retrieved from a publicly available source, including public GitHub repositories, Bioconductor’s imentHub , and datperiment , 62 and,References underwent minimal filtering in order to remove groups  with an insufficient number of cells, as well as genes and cells of low quality . Secondly, we drew various subsets from each reference to retain a reduced number of observations (genes and cells), as well as a known number of batches, clusters, or neither .scater’s calculateCPM function [b/k) structure. For PC-based summaries, we ran scran’s [modelGeneVar (on logCPM) and getTopHVGs to select the n = 500 most highly variable features, and scater’s calculatePCA to compute their first ncomponents = 50 PCs. For datasets other than type n, each summary was computed for each of three cell groupings: globally , at the batch-, and at the cluster-level. Three additional summaries—the percent variance explained (PVE) [b/k). Summaries are described in more detail in Additional file We computed a set of five summaries at the gene-level: average and variance of logCPM, coefficient of variation, detection frequency , and gene-to-gene correlation of logCPM. Here, logCPM correspond to log1p-transformed counts per million computed with function . We alsofunction  (LDF) th scran’s  modelGened (PVE)  at the ged (PVE) , and siled (PVE)  at the cks.test function of the stats R package, and the Wasserstein metric using the wasserstein_metric function of the waddR R package [MASS’ kde2d function [emdist’s emd2d function [For each reference-simulation pair of summaries, we computed the Kolmogorov-Smirnov (KS) test statistic using the  package . In addi package  (using Mfunction ) and earfunction  (using efunction ) betweenTo quantify simulator runtimes, we selected one reference per type and drew five random subsets of 400–4000 genes (fixing the number of cells) and 100–2600 cells (fixing the number of genes). For each method and subset , we separately measured the time required for parameter estimation and data simulation. For each step, we set a time limit of ComBat [Harmony [fastMNN and mnnCorrect [limma [Seurat [cms and ldfDiff function, respectively, of the CellMixS package [Integration methods were implemented as in Chazarra-Gil et al.  , Table S3 , and Table S4 (method parameters); description of evaluation statistics , computational Snakemake workflow, and supplementary R data objects.Additional file 2: Figures S1-3 (t-SNEs of reference datasets), Figs. S4-10 (comparison of 1/2D test statistics), Figs. S11-13 (runtime and memory usage), Figs. S14-16. (integration results), Figs. S17-24 (t-SNEs of integrated datasets), Figs. S25-28 .Additional file 3. Review History.