Acute myeloid leukemia (AML) is an aggressive type of leukemia, characterized by the accumulation of highly proliferative blasts with a disrupted myeloid differentiation program. Current treatments are ineffective for most patients, partly due to the genetic heterogeneity of AML. This is driven by genetically distinct leukemia stem cells, resulting in relapse even after most of the tumor cells are destroyed. Thus, personalized treatment approaches addressing cellular heterogeneity are urgently required. Reconstruction of Transcriptional regulatory Networks (RTN) is a tool for inferring transcriptional activity in patients with various diseases. In this study, we applied this method to transcriptome profiles of AML patients to test if it provided additional information for the interpretation of transcriptome data. We showed that when RTN results were added to RNA-seq results, superior clusters were formed, which were more homogenous and allowed the better separation of patients with low and high survival rates. We concluded that the external knowledge used for RTN analysis improved the ability of unsupervised machine learning to find meaningful patterns in the data. Transcription factors (TFs) bind specific DNA sequences and activate or repress the expression of corresponding genes. The genes controlled by a TF form a regulon. TFs’ activity, and the activity of their associated regulons, is hard to assess as assays that measure regulon activity need to be optimized for each TF separately. The difficulty of assessing TF activity stems from the plethora of control mechanisms that determine this activity: the regulation of transcription level , the inA computational method was described that can reconstruct regulon activity levels from global expression profiles. This method is based on a database that identifies TFs. It uses global expression profiles, either from RNA-seq or microarrays, to estimate the pattern of regulon activity that best explains the given expression pattern. This tool is readily available as an R package called “RTN: Reconstruction of Transcriptional regulatory Networks and analysis of regulons” (RTN) ,7.Leukemia is a group of blood cancers that develop in the bone marrow and result in high numbers of abnormal white blood cells (leukemic blasts) . In thisIn AML, 60–80% of younger patients achieve complete remission after standard cytotoxic chemotherapy with high-dose cytarabine  and daunorubicin. Unfortunately, most are expected to relapse and die, most likely due to the outgrowth of residual therapy-resistant AML cells in the bone marrow . In oldeDespite the remarkable genotypic and phenotypic heterogeneity of AML cells among patients, most AML studies investigate averaged responses at the population level, making the development of targeted therapy very challenging. It has been shown that different molecular subtypes of AML tumors, which can be pathologically indistinguishable, respond differently to the same drug regimens . Thus, sIn this work, we directly assessed the contribution of RTN-derived features to the clustering of AML patients, using the older FAB classification. We considered three clustering methods and compared their performances using a quality of clustering measure  and a biological measure, namely, survival. Our results support the hypothesis that RTN-derived features contain information that cannot be teased from RNA-seq data with standard clustering methods. This finding is further supported by recreating these results in an independent dataset. In future studies we plan to turn to single cell analyses, where RTN-like approaches may help predict the responses of single cells to therapy. However, in this work, we exclusively use bulk RNA.Population 1: TCGA LAML. The TCGA-LAML  data werPopulation 2: OHSU AML. The OHSU AML  data werThe RTN package (version 2.15.1) was run using R (R Core Team 2012). While a complete description of RTN can be found elsewhere ,7, a briFor population 1, feature selection was not used.For population 2 the following procedure was used for feature selection: 75% of the data were randomly chosen for feature selection. Median survival was used to label patients as survivors or non-survivors . The FeatureWiz package  of the PClustering algorithms were applied to the TCGA-AML mRNA expression dataset, including all the genes, with default parameters (k = 2 included). The resulting clusters were used to define patients’ sets, whose survival was compared using the Kaplan–Meier Survival plot and the log-rank test for the significance of difference in survival. To do this, we used the sklearn.cluster, kaplanmeier, matplotlib and fcmeans packages in the python programming language (version 3.8). Elbow analysis  suggesteTo evaluate clusters’ quality, we arbitrarily chose the Silhouette Coefficient score. Other cluster quality measures such as the Rand Index, Adjusted Rand Index, Mutual Information, Calinski–Harabasz Index or the Davies–Bouldin Index were not tested.In a previous article, we showed that the RNA-seq dataset possess crucial information above and beyond the mutations’ data. In this work, we applied a related approach to test the hypothesis that RTN provides further information that cannot be learned from RNA-seq data. If true, this hypothesis predicts that supplementing or replacing expression profiles with RTN results should outperform RNA-seq data in machine learning tasks. To test this prediction, we chose to use unsupervised machine learning, comparing the grouping of AML patients in terms of survival with RNA-seq results alone or after adding RTN-derived regulon activities as features. For a detailed description of the study population see p-value = 0.006, 0.0054 and 0.0014 in K-means, FCM and BIRCH, correspondingly). A visualization of the clusters can be seen in the First, we identified the clustering method that is most suitable for the task at hand. For this reason, we compared the performances of three clustering algorithms . Interestingly, using only RTN-inferred regulon activity to cluster patients performed the worst in defining a patients group with distinct survival . When the separation into subtypes was compared between RNA alone and RNA with RTN , the groups were purer when RTN was added . This suggests that even though RTN adds novel information, RNA-seq with RTN is still not enough to classify AML patients into homogenous subtypes. Further research is required to determine what, if any, information should be added to RNA-seq data to achieve better subtype definition. It has not escaped our attention that in our analysis, RTN alone created more dispersed clusters than those created with RNA-seq alone in TCGA-LAML. This could be, at least in part, due to the shortcomings of the RTN approach. RTN uses data on TFs published previously in the literature . HoweverIn the context of machine learning tasks, deriving regulon activity from expression profiles can be considered as knowledge-driven feature engineering. Prior knowledge of regulation networks, in the form of ChIP-seq results, is used to derive new features from RNA-seq results. The present work demonstrates that this is a viable approach to improving the performance of machine learning in biomedical tasks. While it is possible to learn from the data, the knowledge that can be derived from external sources might “consume” a large fraction of the data. Since in biomedical research data tend to be “fat” , injecting external knowledge gained through carefully designed controlled experiments may be beneficial in cases where creating large datasets may be impossible due to limited access to samples  or the costly procedures required to extend the datasets . Thus, using RTN for feature engineering may be just a particular case of a more general collection of methods for prior knowledge-based feature engineering.Can RTN aid machine learning to define purer subgroups simply by the dimensionality of this problem? Our results suggest that this is not the case. We showed that using a simple dimension reduction algorithm for feature engineering, namely, PCA, does not achieve the same improvement in the clustering of AML patients as RTN achieves.Our results do not uncover a new subtype of AML patients. The cluster with a better prognosis after adding RTN features to RNA-seq is purely made of subtype M3 of AML patients, which is sufficiently distinct from other AML subtypes that it was later re-classified as APL. In fact, all the differences between patients assigned to this cluster and other clusters can be explained by the subtype/disease differences. However, we show that this distinction is more clear when RTN features are added.This work focused on bulk RNA-seq, and we hope that when this study is performed on single-cell data, the outcomes will be more significant.In this work we showed that RTN can be used to design new features that improve the performance of unsupervised machine learning methods in identifying biologically meaningful subtypes of AML patients. We showed that generating new features with RTN based on RNA-seq features improved the quality of clusters generated with both types of data (RNA-seq and RTN) compared to each type alone. We concluded that injecting prior knowledge into unsupervised learning, at least in the case of RTN and AML patient clustering, can help in identifying biologically meaningful subgroups.


Spatially resolved transcriptomics (SRT)-specific computational methods are often developed, tested, validated, and evaluated in silico using simulated data. Unfortunately, existing simulated SRT data are often poorly documented, hard to reproduce, or unrealistic. Single-cell simulators are not directly applicable for SRT simulation as they cannot incorporate spatial information. We present SRTsim, an SRT-specific simulator for scalable, reproducible, and realistic SRT simulations. SRTsim not only maintains various expression characteristics of SRT data but also preserves spatial patterns. We illustrate the benefits of SRTsim in benchmarking methods for spatial clustering, spatial expression pattern detection, and cell-cell communication identification.The online version contains supplementary material available at 10.1186/s13059-023-02879-z. A wide variety of spatially resolved transcriptomics (SRT) techniques have been recently developed to enable gene expression profiling with spatial localization information on complex tissues. Some of these SRT techniques are based on single-molecule fluorescence in situ hybridization (smFISH) and examples include seqFISH , seqFISHThe development of SRT-specific computational methods and software tools requires the routine use of simulated/synthetic SRT data. Synthetic SRT data contains a known underlying truth, which enables objective evaluation of method performance. Synthetic SRT data also allows method evaluation to be carried out under a wide range of parameter settings, helping characterize the unique benefits and limitations of the developed method across different scenarios. In addition, synthetic SRT data allows the power and accuracy of specific analytical tasks to be evaluated under different SRT techniques, sequencing depths, and other experimental conditions, thus facilitating experimental design for maximizing the benefits of SRT experiments. Unfortunately, many existing simulations of SRT data in different methodology studies are often poorly documented, not reproducible, and/or fail to generate synthetic data that resemble real SRT datasets. In addition, simulations for assessing a particular method are often carried out under the same modeling assumptions underlying the method, which can lead to an overestimation of method performance. Therefore, it is important to develop an independent, reproducible, and realistic SRT simulation framework that can be used to facilitate the development of SRT methods for a wide variety of SRT-specific analyses. Such simulation frameworks have long been established in the parallel research field of single-cell RNA sequencing (scRNA-seq) studies, with multiple popular scRNA-seq simulators available for facilitating scRNA-seq methodology development and scRNA-seq analysis –55. UnfoSpatially Resolved Transcriptomic simulator), for generating synthetic SRT data based on a wide variety of SRT techniques. SRTsim incorporates spatial localization information to simulate SRT expression count data in a reproducible and scalable fashion, thus facilitating SRT experimental design and methodology development. A key benefit of SRTsim is its ability to not only maintain various location-wise and gene-wise SRT count properties but also preserve the spatial expression patterns of the SRT data on the tissue, thus making it feasible to evaluate SRT method performance for various SRT-specific analytic tasks using the synthetic data. We characterize the properties of the synthetic data generated by SRTsim in a comprehensive way and illustrate the benefits of SRTsim for assessing the performance of spatial clustering methods, spatial expression analysis methods, and cell-cell communication identification methods.Here, we develop the first SRT-specific simulator, which we call SRTsim  varying the total sequencing depth while keeping the number of measured locations constant, (II) varying the number of measured locations while keeping the total sequencing depth constant, and (III) keeping the average sequencing depth per-location constant, while varying both the number of locations and the total sequencing depth Fig. G. We fouFinally, we carried out simulations to examine the influence of the number and size of tissue domains on the performance of spatial clustering. In particular, we used a modified version of the domain-specific simulations to create synthetic data based on DLPFC, where the modification allows us to change the number and size of the tissue domains. To do so, we merged the original seven cortical layers Fig. A into thWe further demonstrated the utility of SRTsim in benchmarking spatial expression (SE) analysis methods that aim to identify genes with spatial expression patterns. Here, we used synthetic data generated by SRTsim for benchmarking four SE methods that include SPARK , SPARK-GP values, while SpatialDE yields overly conservative P values  to ensure fair comparison among methods. In the alternative simulations based on DLPFC, we found that SPARK and SPARK-G are more powerful than SPARK-X and SpatialDE for most spatial expression patterns, except for the layer 1 specific expression pattern for which SPARK-X is more powerful than the other three methods. In the simulations, we found that the power of SE analysis depends on two critical factors: the SE strength and the size of the tissue domain on which the SE gene displays a domain-specific expression pattern. For example, it is challenging to detect genes that display layer-specific expression patterns in the thin layers 2 and 4 compared to the other wider layers , Giotto with spatial information achieves the lowest F1 scores. For example, at a While Giotto with spatial information achieves the best performance in terms of F1 score, we note that all methods can be advantageous when measured with the other metrics. In particular, Giotto with spatial information achieves the highest precision  that contains 7 spatial domains, while STARmap data was collected in a much smaller capture area (11.78×11.78um) with only 4 spatial domains. Because HMRF relies on a Potts model to incorporate neighborhood similarity in the spatial domain assignment, it may perform well when the tissue structure is simple and small but less so when the tissue structure is large and complex.We have demonstrated SRTsim’s ability to generate synthetic datasets resembling the real data collected from eight different SRT techniques. These techniques include the three commonly applied commercial platforms such as 10x Visium, NanoString CosMx SME, and Vizgen MERFISH. We note that another SRT commercial platform is NanoString GeoMx DSP –67, whicOne important benefit of SRTsim is its ability to produce realistic synthetic datasets under different parameter settings for evaluating the performance of various SRT-specific methods in a cost-effective fashion. We have illustrated such benefit by using SRTsim to assess the performance of spatial clustering methods, spatial expression analysis methods, and cell-cell communication identification methods. Importantly, SRTsim is not limited to these three applications and can be used for method evaluation in many other SRT analytic tasks. For example, one can generate synthetic SRT data with continuous expression programs using the reference-free module in SRTsim for evaluating trajectory analysis in the spatial context , 68. In While we have primarily focused on using synthetic data from SRTsim to evaluate computational method performance, we note that SRTsim also has the potential to guide experimental design. For example, in the spatial clustering application, we have explored how the accuracy of spatial clustering for different computational methods may change under the influence of sequencing depth and the number of spatial locations measured on the tissue. In that setting, we showed that SRTsim could help us identify the likely optimal sequencing depth and the number of measured locations for detecting spatial domains on the mouse cortex, thus facilitating experimental design for maximizing the effectiveness of spatial transcriptomics.In conclusion, SRTsim is an effective simulation tool for SRT studies. SRTsim provides scalable, reproducible, and realistic simulations, while preserving expression characteristics and spatial patterns. We have demonstrated the utility of SRTsim in evaluating the performance of spatial clustering methods, spatial expression analysis methods, and cell-cell communication identification methods.spsample, again with the number of locations determined by the user. The first approach mimics the spatial location layout in certain spatial transcriptomics technologies such as 10x Visium while the second approach mimics the spatial location layout in certain spatial transcriptomics technologies such as MERFISH or seqFISH+.Step 1. Obtain or create location coordinates in the synthetic data. This is achieved by either directly obtaining the coordinates of the tissue locations measured in the reference data to serve as the locations in the synthetic data, or, if requested by the user, creating new locations in the synthetic data with the number of locations specified by the user. To create new locations, SRTsim first fits the measured locations in the reference data using a concave hull algorithm  to obtaiSRTsim is a general and flexible computational framework for simulating gene expression count data for spatial transcriptomics. SRTsim primarily performs reference-based simulations where the simulated gene expression resembles a reference spatial transcriptomics data supplied by the user. SRTsim can also perform reference-free simulations without using a reference dataset, which will be described in detail later. In the reference-based simulations, SRTsim relies on the reference spatial transcriptomics data to obtain the spatial coordinates of the measured locations, create a new set of tissue locations for a user-specified location number if needed, and simulate expression count data based on the parameters estimated in the reference data. SRTsim provides two simulation options. The first option is a tissue-based simulation, where the expression data are directly simulated on all spatial locations of the entire tissue. The second option is a domain-specific simulation, where the expression data are first simulated on locations within each tissue domain separately before being combined across domains. Like existing scRNA-seq simulators, the gene expression data simulated by SRTsim resembles the gene expression pattern in the reference data based on a wide variety of count characteristics. Unlike existing scRNA-seq simulators, the expression data simulated by SRTsim also preserves the spatial expression pattern of the reference data. The spatial expression pattern is preserved either on the entire tissue for tissue-based simulations or on each specific tissue domain for domain-specific simulations. The reference-based simulations in SRTsim consist of the following three steps:k (default = 3) nearest measured locations through a Boyer-Moore majority voting algorithm. The domain labels are then used along with the location coordinates in the next two steps for the domain-specific simulations.Step 2. Infer the gene-specific expression count model based on the reference data and generate the expression counts for each gene in the synthetic data. For tissue-based simulations, SRTsim uses all measured locations on the entire tissue to infer the expression model. For domain-specific simulations, SRTsim uses the measured locations on each tissue domain to infer domain-specific expression models. In either case, SRTsim examines one gene at a time and fits the reference expression count data using four different count models. The four count models include the Poisson model, the zero-inflated Poisson model (ZIP), the negative binomial model (NB), and the zero-inflated negative binomial model (ZINB). After modeling fitting, SRTsim selects the model with the lowest Akaike information criterion (AIC) as the gene-specific count model and uses the estimated model parameters to generate expression counts for the gene of focus in the synthetic data. The expression counts are generated for all locations on the entire tissue in the tissue-based simulations or locations on each tissue domain in turn in the domain-specific simulations.k (default = 3) nearest neighboring locations measured in the reference data, randomly samples an expression count for the gene of focus from the identified neighbors, assigns the sampled count to the new location, ranks all new locations in the synthetic data based on the assigned reference expression counts, and finally assigns the simulated expression counts to the new locations based on the resulting rank. This way, the spatial expression pattern of the gene of focus in the synthetic data will mimic the spatial expression pattern of the same gene in the reference data, thus preserving the spatial expression pattern. In the tissue-based simulations, the ranking of locations and the assignment of the simulated expression counts are performed for all locations on the entire tissue. In the domain-specific simulations, the ranking of locations and the assignment of the simulated expression counts are performed on each tissue domain in turn.Step 3. Assign the generated gene expression counts to the locations in the synthetic data in a way that preserves the spatial expression pattern of every gene. To preserve the spatial expression pattern, SRTsim first rank orders the locations in the synthetic data for every gene based on its expression level across locations in the reference data. Afterwards, SRTsim assigns the simulated gene expression to the locations in the synthetic data based on the obtained rank order. Specifically, in the setting where the locations in the synthetic data are identical to the locations in the reference data, SRTsim directly ranks the locations in the synthetic data based on the reference and assigns the simulated expression counts to locations in the synthetic data based on the rank. In the setting where new locations are created in the synthetic data, SRTsim first identifies for each new location the Once the coordinates for the locations are obtained or created in the synthetic data, the user can then proceed to perform either tissue-based or domain-specific simulations. For tissue-based simulations, only the location coordinates are needed for the next two steps described below. For domain-specific simulations, SRTsim also requires the user to input the tissue domain labels for the measured locations in the reference data. The input tissue domain labels are either directly used when the locations in the synthetic data are the same as the measured locations in the reference data or are used to extrapolate the tissue domain labels for the newly created locations in the synthetic data. In the latter case, for each new location, SRTsim assigns its domain label based on the domain labels from the k (default = 3) nearest neighboring projected locations, randomly samples an expression count for the gene of focus from the identified neighbors, assigns the sampled count to the location of focus, ranks all locations in the target region in the synthetic data based on the assigned projected reference expression counts, and finally assigns the simulated expression counts to locations in the target region based on the resulting rank. This way, the spatial expression pattern of the gene of focus in the target region will mimic the spatial expression pattern of the same gene in the reference region, thus creating potentially new and realistic spatial patterns on the tissue.In the reference-based simulations, besides preserving the spatial expression pattern of the gene of focus, SRTsim can also allow the user to redesign the spatial expression pattern of the gene in any target tissue region in the synthetic data by extrapolating the spatial expression pattern of the gene from another tissue region in the reference. To do so, SRTsim first uses a concave hull algorithm to obtain the shape profile for the target region, which consists of a set of measured locations that define the outskirt of the target tissue region in the synthetic data. In addition, SRTsim obtains the shape profile for the reference region, which consists of a set of measured locations that define the outskirt of the reference tissue region. SRTsim then maps the estimated shape profile of the reference region to the shape profile of the target region through the affine transformation , projectBesides reference-based simulations, SRTsim can also perform reference-free simulations without using a reference spatial transcriptomics dataset. To do so, SRTsim allows the user to first create an arbitrary tissue shape in two different ways: the user can either choose from a set of customizable shapes that are constructed based on various circles and squares, or provide a shape profile in the form of a set of location coordinates that define the outskirt of the desired tissue shape. With the defined tissue shape, SRTsim creates spatial locations on the tissue based on a user-specified location number. The spatial locations are then created either through a grid-based approach or a random sampling-based approach, as detailed in Step 1. For domain-specific simulations, the user is also required to assign a tissue domain label for each spatial location and specify a domain-specific fold change parameter that defines the mean expression fold-change for gene expression inside the domain versus that on the entire tissue. SRTsim then simulates the expression count data for one gene at a time on each location based on a user-specified count model. The user has the option to choose one of the four count models described in Step 2 and provide a set of model parameters that include the mean parameter , the dispersion parameter (for the two over-dispersed models), and the zero-proportion parameter (for the two zero-inflated models). Afterwards, SRTsim allocates the simulated expression counts to locations either randomly across all locations on the entire tissue for tissue-based simulations or randomly across locations within each tissue domain for domain-specific simulations. Note that the genes in the reference-free simulations are restricted to display domain-specific expression patterns but not general spatial expression patterns obtainable in reference-based simulations. Importantly, all reference-free simulations can be carried out in an R shiny framework, which allows the user to easily visualize the simulated spatial transcriptomics data.For both reference-based and reference-free simulations, the output is an S4 object that mainly contains two matrices for the synthetic data: one matrix contains the spatial coordinates for all tissue locations, along with their domain labels for domain-specific simulations; and the other matrix contains the simulated expression counts for all genes across all locations. The output S4 object also contains all the estimated model parameters and user-defined parameter inputs, which, when paired further with the input random seed, ensure the reproducibility of the synthetic data. After the simulation, user can also compare the synthetic dataset to the corresponding reference dataset using functions implemented in the package.https://jiaqiangzhu.shinyapps.io/srtsim, which is powered by RStudio. The software SRTsim, along with the scripts for reproducing all results in the present study, is freely available at www.xzlab.org/software.html.Both reference-based and reference-free simulation procedures are implemented in an R CRAN package, SRTsim. The R package contains the R shiny app, which can be easily installed and accessed on any local computer. The reference-free simulations in SRTsim can also be carried out through an online shiny app available at We obtained 49 SRT datasets across eight different spatial transcriptomics platforms , SPARSim (v.0.9.5), SymSim (v.0.0.0.9000), two variations of scDesign2 , and three variations of Splat in the Splatter package . These single-cell simulators allow users to perform reference-based simulations with provided single-cell reference data. We supplied each of the 40 SRT datasets one at a time to serve as the reference data and applied these simulators to simulate spatial transcriptomics. We evaluated the performance of different simulators by six metrics that include four gene-specific metrics and two location-specific metrics. The gene-specific metrics measure for each gene how similar the simulated gene expression counts resemble that of the reference data across locations. The four gene-specific metrics include expression mean, variance, coefficient of variation, and the proportion of zero counts. The location-specific metrics measure for each location how similar the simulated gene expression counts resemble that of the reference data across genes. The two location-specific metrics include the proportion of zero counts and the library size. We examined the empirical distribution of each gene-specific metric across genes and the distribution of each location-specific metric across locations. In addition, we examined whether the simulated gene expression counts preserve the spatial expression pattern of the reference gene through visualization. We also examined the computational efficiency of different simulators by recording the computation time and memory usage and examined their stability by recording the number of datasets each simulator failed. For computation time and memory usage, we focused on eight datasets, one from each spatial transcriptomics platform: ST (Rep11_MOB), 10x Visium (LIBD_151673), DBiT (GSM4189615_0719cL), STARmap , seqFISH+ (cortex_svz), CosMx SMI (Lung12_FOV1), MERFISH (HumanOvarianCancerPatient1_FOV1051), and SlideseqV2 (Puck_190926_03). The number of genes in these seven datasets ranges from 550 to 23,124, and the number of locations ranges from 262 to 51,649.We illustrate the benefits of SRTsim for benchmarking spatial domain detection methods using simulations. To do so, we obtained two reference SRT datasets for simulations: a non-single-cell resolution 10x Visium dataset, which has relatively low gene expression counts with the median value of the mean gene-specific expression count across genes being 0.03, and a single-cell resolution STARmap dataset, which has relatively moderate gene expression counts with the median value of the mean gene-specific expression count across genes being 0.38.n=3611) were clustered by the original study based on cytoarchitecture and gene markers into seven tissue domains that include six dorsolateral prefrontal cortex (DLPFC) layers and the white matter  for analysis.The 10x Visium dataset is a human dorsolateral pre-frontal cortex (DLPFC) data obtained from spatialLIBD and has been widely used for benchmarking studies , 39, 59.The STARmap dataset is a primary visual cortex data, with 1020 measured genes on 930 cells. Following the annotations based on the spatial expression pattern of gene markers in , 77, we For either dataset, we performed both tissue-based and domain-specific simulations. In the domain-specific simulations, in addition to directly using the tissue domains from the reference data, we also examined an alternative simulation scenario where we varied the number of tissue domains and their sizes while maintaining their domain-specific spatial gene expression pattern. To do so, we focused on the DLPFC data and grouped the original seven cortical layers into three new layers , representing 50%, 100%, 150%, 200%, and 300% of the observed sequencing depth of the reference data, respectively. In scenario II, we examined the influence of the measured location number, where we varied the number of locations on the tissue but kept the overall sequencing depth constant. In this scenario, we varied the number of spatial locations on the tissue to be 100, 300, 1000, 2000, 3611 (=observed location number in the reference), and 5000. In scenario III, we varied both the number of locations and the overall sequencing depth but kept the average number of reads in each spatial location fixed. In this scenario, we examined all pairs of combinations based on the six location number choices in scenario II and five overall sequencing depth choices in scenario I.We evaluated the performance of six different spatial clustering methods in detecting spatial domains on the tissue in the synthetic data. The six evaluated methods include the SNN (shared nearest neighbor) implemented in Seurat (v.4.0.1), BayesSpace (v.1.2.0), stLearn with Kmeans, stLearn with Louvain (v.0.3.2), SpaGCN (v.1.0.0), and HMRF . For stLearn, its two methods would ignore the spatial location information and perform regular Louvain and Kmeans clustering, as the synthetic data do not contain corresponding histological images. For Seurat and Louvain, we examined a range of resolution values and chose the one value that leads to the number of clusters closest to the truth. For SpaGCN, we also included the results with the cluster refinement (refined-SpaGCN) in the DLPFC-based simulation, which utilizes the array coordinates on top of the pixel coordinates information. For HMRF, we varied the spatial parameter beta  and only reported the results for the beta value that achieves the highest performance in the fitted data—because of this, the results from HMRF may represent model over-fitting and may over-state its performance. BayesSpace is developed specifically for ST/Visium platforms, where the neighborhood structures are well-defined in either square or hexagonal arrangements. Therefore, when applied to the STARmap data, where the spatial locations are randomly distributed, BayesSpace can only perform regular mclust-based clustering. Following , we measWe also illustrate the benefits of SRTsim for benchmarking spatial expression analysis methods using simulations. To do so, we performed similar reference-based simulations using the two reference datasets (DLPFC and STARmap). For each reference data, we considered two simulation settings: a null setting where we simulated 1000 non-SE genes with random spatial expression patterns and an alternative setting where we simulated 100 SE genes that display spatial expression patterns along with 900 non-SE genes with random spatial expression patterns. Here, we used a negative binomial model to simulate the expression counts for both SE and non-SE genes, following the recommendations of , 27, 57 We evaluated the performance of four SE methods for identifying genes with spatial expression patterns. The four SE methods include SPARK (v.1.1.1), which relies on an over-dispersed Poisson model; SPARK-G, which relies on Gaussian approximation to SPARK; SPARK-X, which uses a non-parametric test; and SpatialDE (v.1.1.3), which relies on a linear mixed model. The first three methods are implemented in the R package SPARK while the last method is implemented in the SpatialDE python package. Besides these four methods, we also explored three other SE methods that include trendsceek , BOOST-GWe illustrate the benefits of SRTsim for benchmarking cell-cell communication methods using simulations. We focus on two commonly used cell-cell communication methods for spatial transcriptomics that include Giotto (v1.0.3) and CellphoneDBv3, both examining the expression of ligand-receptor gene pairs to identify pairs that are used for communication between interacting cells from two cell types. We performed simulations to evaluate and compare the performance of these methods.Specifically, we used the single-cell resolution STARmap dataset to serve as the tissue shape to generate gene expression on the tissue. To do so, we first estimated the shape profile based on the spatial locations of the reference data and randomly generated 5000 locations within the estimated shape profile to serve as the single-cell locations. We assumed that these single cells belong to four different cell types and we created four equal-sized regions on the tissue with distinct cell type composition in each region. In particular, we set the cell types with the highest to lowest proportions to be cell type 1, 2, 3, and 4 in the first spatial region; to be 2, 3, 4, 1 in the second spatial region; to be 3, 4, 1, 2 in the third spatial region; and to be 4, 1, 2, 3 in the fourth spatial region. We then considered four cell-type composition scenarios in the simulations. In the first scenario, each layer contains one dominant cell type, with 70% of the cells belonging to the dominant cell type and 10% of the cells belonging to each of the three minor cell types. In the second scenario, each layer contains two dominant cell types with equal proportions, each consisting of 45% of the cells, along with two minor cell types each consisting of 5% of the cells. In the third scenario, each layer contains two major cell types with unequal proportion, with one consisting of 60% of the cells and the other consisting of 30% of the cells, along with two minor cell types each consisting of 5% of the cells. In the fourth scenario, each layer contains three major cell types and one minor cell type, consisting of 35%, 30%, 30%, or 5% of the cells, respectively. In each scenario, we randomly assigned each single cell in each region to one of the four cell types based on a multinomial distribution with parameters set to be the cell type composition in the region. We performed 10 simulation replicates for each simulation scenario.In each simulation replicate, we first simulated the expression level of 5000 genes for each single cell in two different ways. In the first way, we simulated homogeneous gene expression by generating expression level for each gene for a total of 8000 cells from a common negative binomial distribution, with parameters inferred based on all cells in the reference data. We then randomly assigned these cells into four cell types, with 2000 cells for each cell type. In the second way, we simulated heterogeneous gene expression by randomly selecting four cell types in the STARmap dataset and then generating the expression level of each gene in 2000 cells for each cell type using a cell type-specific negative binomial distribution, where the distribution parameters are estimated based on the particular cell type in the reference data. In either way, we randomly sampled cells from the 2000 cells in each cell type and assigned them to the spatial locations according to the cell type composition described in the above paragraph.Next, we increased the expression levels for the ligand-receptor pairs that mediate the cell-cell communications. To do so, we obtained 445 ligands and 471 receptors from CellphoneDBv3. These ligands and receptors correspond to 691 unique genes and form 930 ligand-receptor (L-R) gene pairs. For each L-R pair in turn, we randomly assign it to a pair of cell types (A-B), in order to create the scenario that cell type A communicate with cell type B through the ligand on cell type A and the receptor on cell type B. We denote such interaction as L-R-A-B pair. For four cell types, we obtained a total of 16 combinations of (ordered) cell type pairs, with the number of L-R pairs for each cell type pair ranging from 47 to 77 (median=57) in the simulations. With the allocated L-R-A-B pairs, we increased the gene expression for the L and R genes so that the L-R pair on adjacent A-B cell types are higher than that in the other cell types. In particular, for each cell in turn, we first identified its nearest four cells on the tissue and treat them as adjacent cells. If an A cell is adjacent to a B cell, we then increased the expression level of the L gene in the A cell by a fixed fold, with the fold increase set to be either 1 (null), 5, or 10, representing zero, moderate, or high level of cell-cell communications.For each simulated spatial transcriptomics dataset, we evaluated the performance of four cell-cell communication methods to detect interacting L-R-A-B pairs on the tissue. These four methods include (1) Giotto with spatial information; (2) Giotto without spatial information; (3) CellphoneDBv3 with spatial information; and (4) CellphoneDBv3 without spatial information. Each method examines one L-R-A-B pair at a time and tests whether the examined pair is significant. Specifically, for Giotto with spatial information, it calculates the mean of two values: the average ligand expression level of cells in cell type A that interact with cell type B; and the average receptor expression level of cells in cell type B that interact with cell type A. Afterwards, it calculates the proportion of the means in the permutations that are higher or lower than the actual mean. For Giotto without spatial information, it calculates the mean of two values: the average ligand expression level of all cells in cell type A and the average receptor expression level of all cells in cell type B. Afterwards, it calculates the proportion of the means in the permutations that are higher or lower than the observed mean. For CellphoneDBv3 with spatial information, it calculates the mean of the average receptor expression level of cell type A and the average ligand expression level of the cell type B. Afterwards, it calculates the proportion of the means in the permutations that are as high or higher than the actual mean, considering only combinations of cell type pairs that co-exist in a spatial domain. For CellphoneDBv3 without spatial information, it calculates the mean of the average receptor expression level of cell type A and the average ligand expression level of the cell type B. Afterwards, it calculates the proportion of the means in the permutations that are as high or higher than the actual mean, considering all combinations of cell type pairs.p-value thresholds of 0.001, 0.005, 0.01, 0.025, 0.05, or 0.1 [We evaluated the performances of different methods in detecting the L-R-A-B pairs based on the F1 score, which is the harmonic mean of precision and recall, at different adjusted , or 0.1 . While w, or 0.1 –83, we aAdditional file 1: Fig. S1. An overview of the Shiny-app component of SRTsim for reference-free simulations. Fig. S2. Benchmarking SRTsim against eight existing scRNA-seq simulators for generating SRT data based on seven SRT data by 10x Visium. Fig. S3. Benchmarking SRTsim against eight existing scRNA-seq simulators for generating SRT data based on eight DLPFC SRT data by 10x Visium. Fig. S4. Benchmarking SRTsim against eight existing scRNA-seq simulators for generating SRT data based on six SRT data by DBiT-seq and one SRT data by ST. Fig. S5. Benchmarking SRTsim against eight existing scRNA-seq simulators for generating SRT data based on SRT data by single-cell resolution SRT techniques. Fig. S6. Scatter plots comparing Moran’s I in the synthetic data generated by different simulators versus that in the reference data. Fig. S7. Benchmarking SRTsim against scDesign2 and SymSim for generating SRT data with a different number of locations. Fig. S8. Computational cost of different simulators for generating synthetic data based on references from different spatial transcriptomics techniques. Fig. S9. Distributions of six metrics and Moran’s I for the simulation based on the reference DLPFC sample 151673. Fig. S10. Spatial distributions of the domain assignments by six spatial clustering methods on the synthetic data generated by SRTsim in the tissue-based simulations. Fig. S11. Spatial distributions of domain assignments by six spatial clustering methods on the synthetic data generated by SRTsim in the domain-specific simulations. Fig. S12. Distributions of six metrics for the simulations based on STARmap data. Fig. S13. Spatial distributions of domain assignments by six spatial clustering methods based on the synthetic data generated by SRTsim via the tissue-based simulation. Fig. S14. Spatial distributions of domain assignments by six spatial clustering methods based on the synthetic data generated by SRTsim via the domain-specific simulation. Fig. S15. The performance of stLearn with Louvain measured by the adjusted Rand index is evaluated under simulations with varying-sequencing-depth (y-axis) and varying-location-number (x-axis). Fig. S16. Comparisons between the reference and synthetic data generated by SRTsim. Fig. S17. Benchmarking spatial expression analysis methods using synthetic data from SRTsim based on the DLPFC reference. Fig. S18. Benchmarking spatial expression analysis methods using synthetic data from SRTsim based on the STARmap reference. Fig. S19. Benchmarking cell-cell communication identification methods using synthetic data from SRTsim (precision). Fig. S20. Benchmarking cell-cell communication identification methods using synthetic data from SRTsim . Fig. S21. Benchmarking cell-cell communication identification methods using synthetic data from SRTsim (specificity). Table S1. Summary of 49 SRT datasets used in the present study. Table S5. Computational stability of all simulators. X represents failure, while – represents success. Table S6. Total number of alternatives in SE analysis.Additional file 2: Supplementary Table S2.P-values from the Kolmogorov–Smirnov (KS) test for the DFPLC 151673 sample. The KS test is performed for each metrics between the synthetic data and reference data.Additional file 3: Supplementary Table S3.P-values from the Kolmogorov–Smirnov (KS) test for the Lung12 FOV1 sample. The KS test is performed for each metrics between the synthetic data and reference data.Additional file 4: Supplementary Table S4. Number of significance  in the Kolmogorov–Smirnov (KS) test across ten replicates for all the data (n=29). Datasets with at least one failure in selected simulators were excluded. The KS test is performed for each metrics between the synthetic data and reference data.Additional file 5. Review history.


Background: Lung cancer is a complex disease composed of neuroendocrine (NE) and non-NE tumors. Accurate diagnosis of lung cancer is essential in guiding therapeutic management. Several transcriptional signatures have been reported to distinguish between adenocarcinoma (ADC) and squamous cell carcinoma (SCC) belonging to non-NE tumors. This study aims to identify a transcriptional panel that could distinguish the histological subtypes of NE tumors to complement the morphology-based classification of an individual.Methods: A public dataset with NE subtypes, including 21 small-cell lung cancer (SCLC), 56 large-cell NE carcinomas (LCNECs), and 24 carcinoids (CARCIs), and non-NE subtypes, including 85 ADC and 61 SCC, was used as a training set. In the training set, consensus clustering was first used to filter out the samples whose expression patterns disagreed with their histological subtypes. Then, a rank-based method was proposed to develop a panel of transcriptional signatures for determining the NE subtype for an individual, based on the within-sample relative gene expression orderings of gene pairs. Twenty-three public datasets with a total of 3,454 samples, which were derived from fresh-frozen, formalin-fixed paraffin-embedded, biopsies, and single cells, were used for validation. Clinical feasibility was tested in 10 SCLC biopsy specimens collected from cancer hospitals via bronchoscopy.Results: The NEsubtype-panel was composed of three signatures that could distinguish NE from non-NE, CARCI from non-CARCI, and SCLC from LCNEC step by step and ultimately determine the histological subtype for each NE sample. The three signatures achieved high average concordance rates with 97.31%, 98.11%, and 90.63%, respectively, in the 23 public validation datasets. It is worth noting that the 10 clinic-derived SCLC samples diagnosed via immunohistochemical staining were also accurately predicted by the NEsubtype-panel. Furthermore, the subtype-specific gene expression patterns and survival analyses provided evidence for the rationality of the reclassification by the NEsubtype-panel.Conclusion: The rank-based NEsubtype-panel could accurately distinguish lung NE from non-NE tumors and determine NE subtypes even in clinically challenging samples (such as biopsy). The panel together with our previously reported signature (KRT5-AGR2) for SCC and ADC would be an auxiliary test for the histological diagnosis of lung cancer. Lung cancer is the most common malignant tumor and one of the main causes of cancer-related deaths in humans. The most common histological classification of lung cancer is small-cell lung cancer (SCLC) and non-small cell lung cancer (NSCLC), which is based on cell morphology, according to the World Health Organization (WHO) criteria. In 2015, the WHO updated this classification by providing a new criterion that classifies lung cancer into neuroendocrine (NE) and non-NE tumors based on NE morphology , to provLung NE tumors account for approximately 25% of all lung tumors and include SCLC (∼20%), large cell neuroendocrine carcinomas , and carcinoids  . The treMicroscopic morphological features observed using hematoxylin–eosin (HE)-stained specimens are the “gold standard” for elucidating lung cancer histological classification. NE tumors have some unique morphological characteristics  and ultra-microstructures (dense core particles) , which cTherefore, immunohistochemical (IHC) detection of subtype-specific markers has been proposed for assisting histological classification. NE markers, such as chromogranin (CgA), synaptophysin (Syp), and CD56, can be used as auxiliary diagnostic tools for discriminating NE from non-NE tumors . HoweverIt is important that even with the auxiliary immunomarkers, there is still a certain percentage of misclassified cases because of the subjective diagnoses of HE staining or immunostaining results made by pathologists using varying pathological criteria or interpretations, resulting in low reproducibility of pathological diagnosis between LCNEC and SCLC in particular . Two preKRT5 and AGR2) for distinguishing SCC and ADC (non-SCC) subtypes based on the REO approach . A tentative clinical cohort of 10 SCLC samples was collected to test the clinical feasibility. Gene expression patterns of the specific immunomarker genes and survival analyses were also conducted to support the reclassification obtained by the NEsubtype-panel.http://www.ncbi.nlm.nih.gov/geo/) and The Cancer Genome Atlas . Two datasets were collected through a literature search of the NCBI PubMed database (https://pubmed.ncbi.nlm.nih.gov/) using multiple keywords related to lung NE: “lung cancer” AND “lung neuroendocrine tumors” AND  AND (“gene expression profiles” OR “RNA-seq data”). Datasets needed to fulfill the following criteria: 1) containing at least one NE subtype, or only containing non-NE subtypes but providing follow-up information; and 2) providing raw data or processed gene expression profiles with clear preprocessing and normalized methods. All datasets used in this study are displayed in The 22 public gene expression datasets of lung tissues used in this study were downloaded from the Gene Expression Omnibus , including pathologically determined samples of 21 SCLC, 56 LCNEC, 24 CARCI, 85 ADC, and 61 SCC, was used to investigate the molecular landscape across lung cancer subtypes; data from 198 patients who had undergone only curative surgical resection were used for survival analysis. The dataset was further used as a training set to develop a panel of qualitative transcriptional signatures.The qualitative signatures were tested step by step in 18 datasets that had fresh-frozen lung specimens, one dataset that had FFPE specimens, two datasets that had small biopsy specimens, two datasets that had mixed tumors with varied proportions of tumor cells, and one single-cell dataset, and these included 122 SCLC, 25 LCNEC, 137 CARCI, 6 NE, 2,155 ADC, 1,003 SCC, 4 adenosquamous carcinoma, and 12 other non-NE samples in total. LCC samples in these datasets, diagnosed according to the WHO 2004 criteria, were removed from this study since they might have included LCNEC samples. For the single-cell RNA-sequencing data (GSE131907), 32,764 tumor cells were derived from 58 samples of 44 ADC patients, including the primary tissues of 22 early-stage lung cancers (tLung) and advanced-stage lung cancers (tL/B), 17 brain metastases (mBrain) and metastatic lymph nodes (mLN) samples, and 5 pleural effusion samples.In addition, among the 18 datasets with fresh-frozen samples, nine datasets  with survival information were integrated for survival analysis, and these included 1,071 stage I–III ADC and SCC (non-NE) patients who had undergone only curative surgical resection. For the microarray datasets generated by Affymetrix platforms, a robust multiarray average algorithm  was usedFrozen biopsy specimens were obtained from 10 SCLC patients who underwent bronchoscopic intervention at the Harbin Medical University Cancer Hospital. Among them, nine patients were directly diagnosed by pathologists based on HE staining results, while one patient with a poorly differentiated tumor was further performed IHC detection for NE markers and finally diagnosed as SCLC by pathologists, which showed positivity for CD56, CgA, Syp, TTF-1, and CK7 and negativity for CK5/6 and P63. The samples were obtained under the ethical approval of the Institutional Review Boards of the Harbin Medical University Cancer Hospital, and written informed consent forms were obtained from all participants.Total RNA was extracted according to the manufacturer’s protocol. The RNA quality was checked using Nanodrop . The purity and concentration of total RNA were determined using a Nanodrop spectrophotometer  according to the OD260/280 reading and a Qubit fluorescence quantifier , respectively. Paired-end sequencing with a read length of 100 bp was conducted using the Illumina Hiseq 2500/3000 platform , and the final processed RNA-sequencing data were termed as SCLC data of Harbin Medical University (HMU-SCLC) . Data ank groups (k = 2–10) via Pearson’s correlation distance using the top 1,000 most variable genes across all samples in a cohort. The k value that corresponded to the first downward inflection in the cumulative distribution function was selected as the optimum number of clusters.Consensus clustering was performed using the “ConsensusClusterPlus” package version 1.52.0 according to the Ward method for hierarchical clustering . The samA hierarchical rank-based method was developed to construct multiple qualitative signatures of lung cancer subtypes step by step.t test was used to identify differentially expressed genes (DE genes) between the two clustering-adjusted subtype groups. The p values were adjusted using the Benjamini–Hochberg method for multiple testing to control the false discovery rate (FDR)  . Genes wa and Gb, derived from the candidate genes, Fisher’s exact test was used to assess whether the frequency of a specific REO pattern (Ea > Eb or Ea ≤ Eb) in one clustering-adjusted subtype sample was significantly different from that in another clustering-adjusted subtype sample. Here, Ea and Eb are the gene expression values of Ga and Gb, respectively. Gene pairs with FDR of < 5% were defined as significantly reversed gene pairs between the two subtypes.For a pair of genes, GS is the number of samples whose classification subtypes predicted by the gene pair (Ga and Gb) were consistent with their original pathological subtypes, and N is the total number of corresponding subtype samples used in the dataset.A gene pair signature was constructed from all significantly reversed gene pairs as follows: First, for each significantly reversed gene pair, its classification consistency with pathologically diagnostic subtypes was calculated. Here, the classification consistency was termed as the “apparent” accuracy, since the pathological assessments were not 100% reliable and there may be misclassified cases according to clinical pathological methods.a and Gb) in all samples between the two subtype groups (g1 and g2), respectively.Second, all the significantly reversed gene pairs were chosen as an initial set, and all the genes contained in the initial set were used as seed. Then, a de-redundant method was utilized to obtain an optimal gene pair set based on the filter rule as follows: For any gene in the seed, if there were multiple gene pairs containing the gene, the one with the highest apparent accuracy was retained. If multiple gene pairs achieved the same maximum apparent accuracy, the gene pair with the largest absolute rank difference  between At last, the classification score for each sample was calculated as the sum of the classification votes of all the gene pairs in the set. The majority voting rule of the reversed gene pairs within a sample was adopted for classification, where if more than half of the gene pairs within the sample voted for one subtype, the sample was classified into that subtype.In the training dataset (GSE30219), the above method was utilized to develop the NEsubtype-panel composed of three transcriptional signatures to distinguish the NE from non-NE tumors, CARCI from non-CARCI tumors, and SCLC from LCNEC tumors. To improve the robustness of the signatures to RNA degradation or low RNA input, which usually occur in clinically challenging samples, such as FFPE and biopsy samples, the gene pairs that have the gene with undetected expression value were removed and the majority voting rule of the remaining gene pairs in the signature was adopted for classification.“ClusterProfiler” R package  was perfCD56, SYP, and CHGA) (KRT5 and TP63), and one ADC marker gene (NAPSA) (TTF-1) was excluded, since it is also highly expressed in partial SCLC samples  was used to test the differences across multiple groups. RankProd (RP) algorithm of the “RankProd” R package version 3.14.0 , a nonpand CHGA) , two SCC (NAPSA) . Here, a samples . WilcoxoOverall survival (OS) is defined as the time from the date of initial surgical resection to the date of death or last contact (censored). To avoid deviations in the patient follow-up time among the different datasets, patient OS was truncated at 60 months. Survival curves were estimated using the Kaplan–Meier method and were statistically compared using the log-rank test . A multihttp://www.r-project.org/). Significance was defined as p < 0.05 or FDR < 0.05 for multiple testing.All statistical analyses were conducted using R 3.6.2 software . The CARCI subtype was characterized by low levels of immune score, cell infiltration, and immune checkpoint gene expression, while partial LCNEC and SCLC samples showed high levels of the three immune indexes , which was more than the 12,917 DE genes extracted between the original pathologically determined subtypes . Furthermore, 92.44% of the 12,623 overlapped DE genes had a higher FC value than that in the original pathological subtypes, indicating the rationality of removing the dubious samples. From the 13,216 DE genes between the two clustering-adjusted subtype samples, the top 1,000 DE genes with a large FC difference were selected to construct gene pairs. Next, 373,502 NE-specific gene pairs were extracted, whose specific REO patterns (Ea > Eb) occurred more frequently in the clustering-adjusted NE samples than those in the clustering-adjusted non-NE samples . For each NE-specific gene pair, if its REO in a sample was Ea > Eb, it voted the sample as NE, and vice versa. At last, the de-redundant method  was utilized to generate an optimal gene pair set consisting of 22 gene pairs  samples than in non-NE  samples (KRT16) is a type I cytokeratin, whose overexpression promotes tumorigenicity in ADC  with the highest degree in SCLC-signature was essential for normal mitotic spindle function-dependent cell division , indicating that ASPM might be a therapeutic target for SCLC  database, genes in the three signatures were mapped into the protein–protein interaction (PPI) network . Then, tfeatures , and YAPsubtypes . It is w samples ; howevery in ADC . The reldivision . Besides in SCLC . Our resfor SCLC .Therefore, PPI network construction and functional analyses of genes in the three transcriptional signatures provided biological evidences for their ability to determine the histological classification and clues for the treatment of lung cancer.The NEsubtype-panel was tested on multiple independent lung cancer datasets. First, the NE-signature in the panel was tested on 18 fresh-frozen tissue datasets, including 200 NE and 2,048 non-NE samples . In totaThen, the classification accuracy of the CARCI-signature in the panel was verified in the 280 signature-confirmed NE samples across nine validation datasets . The appNext, the SCLC-signature in the panel was validated in the signature-confirmed non-CARCI samples . The appAt last, we collected 10 SCLC biopsy samples from the clinic (HMU-SCLC), and the NEsubtype-panel exhibited 100% accuracy for these samples, indicating its clinical feasibility.In total, the NEsubtype-panel had a good performance in distinguishing NE tumors from non-NE tumors and determined the NE subtypes not only in fresh-frozen specimens but also in samples with RNA degradation (FFPE) and low RNA input .CD56: p = 0.0023, TP63: p = 0.0198, CD56: p = 0.0253; SYP: p = 0.0253; CHGA: p = 0.0045; NAPSA: p < 0.0001, CD56: p < 0.0001; SYP: p = 0.0009; CHGA: p = 0.0302, KRT5: p < 0.0001; TP63: p = 0.0001, As the subjective diagnoses of HE staining or immunostaining results by pathologists may lead to some misclassified cases , severalp = 0.0087, HR = 2.22, 95% CI = 1.22–4.01, p = 0.0146, HR = 2.06, 95% CI = 1.15–3.67, Next, the accuracy of reclassification by these signatures was further evaluated through survival analyses. Nine datasets were integrated and included 1,071 stage I–III ADC and SCC (non-NE) patients who were treated with only curative surgical resection and recorded survival information. From all the non-NE samples, 1,051 patients were confirmed by the NE-signature, and 20 patients were reclassified as NE, of which 3 and 17 cases were further reclassified as CARCI and LCNEC, respectively, by the CARCI- and SCLC-signatures. As expected, survival analysis showed that the three reclassified CARCI patients had significantly longer OS, while the 17 reclassified LCNEC patients showed significantly shorter OS than the other ADC and SCC patients . These results suggested the advantage of the subtype panel in clinical applications. It is worth noting that the overall apparent sensitivity of the SCLC-signature for SCLC was 88.60%, which did not seem to be perfect. As our results showed that SCLC displayed higher proliferation ability, the reclassified LCNEC samples had significantly lower proliferation abilities than the signature-confirmed SCLC samples in two of the three datasets   still deserved follow-up study. Although the accuracy of the NEsubtype-panel could reach more than 92%, there was still a certain percentage of discordant samples identified by pathological diagnosis and the NEsubtype-panel, which may lead to some misclassification because of subjective diagnosis of HE staining or immunostaining results by pathologists. The subtype-specific marker genes analysis provided transcriptional evidence to support the reclassifications obtained by our panel. Further, the reclassification of these signatures was supported using survival analyses by the knowledge that LCNEC patients have poorer prognoses and CARCI patients have better prognoses than those ADC and SCC patients. Such biological evidences support the classification accuracy of the NEsubtype-panel.However, there are still some limitations of this study. One limitation is that the NEsubtype-panel could not distinguish between typical and atypical CARCIs in the CARCI samples because the samples of these two subtypes are associated with a low incidence of lung cancer, and thus, there are very few samples present currently to develop robust signatures. Another limitation is that most samples in the public datasets are diagnosed according to the WHO 2004 criteria, which might not be detected by IHC and needs further validation based on the samples diagnosed using the WHO 2015 criteria.KRT5 and AGR2) used for distinguishing SCC from non-SCC (ADC) samples could be used as an RNA-sequencing panel to complement the morphology-based classification of lung tumors. This would also help in preserving precious tissue samples that can then be used for conducting other molecular tests.The novel transcriptional NEsubtype-panel, consisting of three gene pair signatures, was developed that could effectively distinguish lung NE tumors from non-NE tumors and determine the NE subtypes individually, even in clinically challenging samples (FFPE and biopsy samples). The combination of these signatures with our previously published signature (


Single cell RNA sequencing is a promising technique to determine the states of individual cells and classify novel cell subtypes. In current sequence data analysis, however, genes with low expressions are omitted, which leads to inaccurate gene counts and hinders downstream analysis. Recovering these omitted expression values presents a challenge because of the large size of the data. Here, we introduce a data-driven gene expression recovery framework, referred to as self-consistent expression recovery machine (SERM), to impute the missing expressions. Using a neural network, the technique first learns the underlying data distribution from a subset of the noisy data. It then recovers the overall expression data by imposing a self-consistency on the expression matrix, thus ensuring that the expression levels are similarly distributed in different parts of the matrix. We show that SERM improves the accuracy of gene imputation with orders of magnitude enhancement in computational efficiency in comparison to the state-of-the-art imputation techniques. Recovering dropout-affected gene expression values is a challenging problem in bioinformatics. Here, the authors propose a data-driven framework, that first learns the underlying data distribution and then recovers the expression values by imposing a self-consistency on the expression matrix. Despite its prevalence in computational biology, due to noise and low transcript capture efficiency, the resulting gene expression matrix from scRNA-seq is typically sparse, which often results in a loss of important biological information14. In the past decade, intense research has been devoted to address the computational challenges in recovering omitted gene expressions. The developed techniques can be broadly divided into three categories15. In the first category, the sparsity of expression data is modeled using probabilistic theories. SAVER12, SAVER-X16, bayNorm17, scImpute13, and VIPER18 are the prominent methods of this group. The second category includes the imputation methods that utilize averaging/smoothing to recover the expression values. DrImpute19, MAGIC14, and k-NN smoothing20 are the most popular techniques of this category. The last category is based on the reconstruction of data either using deep learning  or low-rank matrix assumption . Overall, the model-based techniques assume a specific model for the expression data, which may limit their applicability in some practical cases. For  example, SAVER extracts information from correlated genes and employs a penalized regression model to impute the data. It assumes that the gene expressions in a cell follow a gamma-Poisson distribution, which may not be appropriate in many cells, including those with low gene expression values12. The smoothing-based methods extract information from similar cells to impute the expression data. However, the averaging effect in these methods may potentially eliminate variability in gene expressions across cells14. The deep learning-based methods learn from the data to perform imputation without imposing any assumption. However, as the deep learning techniques are purely data-driven, the data imputation process is essentially a complex black-box operation with limited transparency. Low-rank matrix-based techniques, such as mcImpute, use a matrix completion approach to impute the data with considerations of both gene-gene and cell-cell relationships. However, in mcImpute, all zero expression values are treated as dropout events, which may lead to spurious results in some practical scenarios. Generally, the analytical techniques  are not scalable to large datasets since the entire dataset must be processed as a whole.Single-cell RNA sequencing (scRNA-seq) has emerged as an effective tool for a variety of cellular analysis tasks such as quantifying the state of individual cells31. Computationally, SERM first learns the underlying data distribution (pattern of self-consistency) using deep learning and then imputes the expression values by ensuring the self-consistency of the data. The approach leverages a deep learning optimization to extract the latent representation of the data and reconstruct the denoised expression values , to recover the missing gene expression values by enforcing data self-consistency, a unique characteristic of high-dimensional datasets. In general, the self-consistency in datasets implies that if two sets of data points arise from similar sources (such as gene counts in similar cells), they should have similar distributionsWe showcase the efficacy of SERM on various benchmark datasets and demonstrate its superior performance over the existing baseline methods. Compared to alternative techniques, SERM consistently achieves significantly better imputation accuracy and speed. The method promises to substantially improve the way that scRNA-seq data is processed and utilized in biomedical research.32, and (5) two cluster quality indices (adjusted Rand (AR)33, and Hubert34). We then show the superior performance of SERM in analyses of the cell trajectory datasets. Here the performance of different methods is assessed by using (1) trajectory quality, (2) correlation of the imputed data with original data, and (3) correlation of computed pseudotime from the imputed data with the ground truth time points. At the end, the capability of SERM in imputing large genomic datasets (the human cell landscape and the mouse cell atlas) is demonstrated to show the strong computational ability of SERM.In this section, we first present the imputation results for synthetic and benchmark gene sequence datasets and show the benefits of SERM in visualization and clustering applications. In these studies, the performance of different imputation methods  are measured in terms of (1) visualization quality, (2) correlation of the imputed data with the reference data, (3) clustering accuracy, (4) normalized mutual information (NMI)35, where each class consists of 500 cells and 1200 genes (see “Methods” for simulation parameters). The dropout-free data are referred to as the ‘reference data’. We sampled the reference data following ref. We first use the Splatter simulator to generate simulated data of 5 classes without dropout36, classification of cells from mammalian brains37, data from mouse intestinal epithelium38 and comparison of engineered 3D neural tissues39). Descriptions of the data can be found in the “Methods” section. In our analysis, the raw data are filtered by choosing cells and genes with high expression values to create reference datasets (see “Methods”). The reference datasets are then sampled at different efficiency following ref. In the next study, we use data from four different scRNA-seq experiments  can be degraded in the presence of technical noise, such as dropouts. SERM mitigates the effect of dropouts and facilitates inference methods such as PHATE to compute  cell trajectories with high fidelity.The following studies demonstrate the quality and accuracy of cell trajectory analysis enabled by SERM. For example, the recently developed dimensionality reduction method PHATE41. In our experiments, the raw data is filtered by choosing cells and genes with high expression values to create a reference dataset (see “Methods”). The reference data is then sampled at different efficiencies to create observed datasets. PHATE visualizations of the reference data, observed data (sampled at 0.1% efficiency), and the data imputed by different methods are shown in Fig. The first dataset we consider is acquired by profiling 38,731 cells from 694 embryos across 12 stages of early zebrafish development42. The PHATE embedding of the EB data, reference, observed, and imputed by different methods, are shown in Fig. The second dataset is scRNA-seq data of human Embryonic Stem (ES) cells differentiated to embryoid bodies (EBs)7. The Pearson correlation coefficients between the ground truth labels and computed pseudotimes by monocle are shown in Fig. 43 and Slingshot44, and the results are presented in supplementary Figs. To quantify the efficacy of the imputation methods in finding the cell trajectories, the observed data and imputed data from different methods are analyzed using monocleThe goal here is to demonstrate that SERM can be used to impute datasets of very large size and complexity. To this end, we analyze two large-scale datasets: the human cell landscape (HCL) and the mouse cell atlas (MCA). These two datasets have about 16,403 million data elements  and 11,665 million data elements , respectively. The HCL dataset has 63 unique cell types and 59 unique tissue types, whereas the MCA dataset has 52 unique cell types and 47 unique tissue types. We found that none of the traditional analytical methods  could impute these datasets within a week on a personal computer (PC) with an Intel Core i9 processor and 64GB RAM. The deep learning methods  can analyze the datasets within a reasonable time.To benchmark the performance of SERM for these datasets, we subsampled the raw data by 20-times randomly and analyzed the subsampled data using different methods  a specific model for gene expression values is often assumed, which may not reflect the actual data distribution in many practical cases; (2) the data is imputed based on the similarity among a few cells or genes, which is susceptible to errors as the global relationship among the cells and genes is not leveraged; (3) all zero expression values are often viewed as dropout events, which may not be accurate  in practical settings; and (4) a priori information about the data is required to tune the hyperparameters of some methods. The proposed SERM mitigates these limitations through an effective learning strategy and significantly improves the imputation accuracy. It should be noted that SERM also assumes specific distributions for the expression data . However, the distribution parameters are learned in SERM, which ensures proper model adaptation to the data.Data distortion, a condition when dropout positions in an expression matrix are filled with inaccurate values, is one of the main concerns in gene imputation. Preventing distortion is critically needed in applications like trajectory mapping, where the accurate position of each cell is required to find the trajectory continuum. In our study, we found that most methods, except SERM, distorted the data in many cases and could not  reliably uncover the underlying biological information. With the learning of the data distribution, SERM shows remarkable performance in imputing the gene expressions for trajectory inference, as revealed in our extensive experiments. The technique consistently  outperforms the existing methods for low and high dropouts.In SERM, a data-driven histogram equalization is performed on each ROI to impute the gene expression values. Although histogram equalization is a popular technique in image processing for contrast enhancement, the assumption of a gene expression matrix as an image with ordered data elements is not necessary here. An implicit assumption in our formulation is that the coverage of ROI is sufficiently large so that the captured data distribution in the ROI is representative of the entire matrix. As only the distribution of expression values in a sufficiently large ROI is considered, the order of rows and columns is generally not important. In other words, if a gene (column) or a cell (row) is interchanged with another one, the distribution of the gene expression values and  the resultant imputation should not change much. To ensure unbiased data distribution in different parts of the matrix, an optional step in which the columns/rows can be permuted randomly is included in our implementation. However, in most cases, this randomization step is unnecessary as the genomic data are generally not ordered. To illustrate this, we computed the correlation values from SERM imputed data for more than twenty expression datasets with 1000 different random alterations of cells and genes  to model the data in SERM. Other distributions can also be included in SERM (see Python/Matlab codes of SERM). One can also use empirical distribution to characterize the data. In Supplementary Fig. 26 and CarDEC47, have been reported. Although batch effects are not considered explicitly in the formulation, SERM is generally less susceptible to data obtained using different technologies or protocols . In the visualizations of projected data in a low dimension, many new cell clusters are found after SERM imputation. Thus, using SERM, it may be possible to discover novel biological information, such as cell or gene types, from large scRNA-seq data.18. To prove the versatility of SERM in single-cell imputation, we have added Supplementary Figs. 18. It is seen that, in all cases, SERM can improve the data quality (which no other method can).In the evaluation of SERM, the reference data created by choosing only the cells and genes with high expressions provides a dropout-free representation. The observed data is created from the reference data by simulating efficiency loss that introduces zeros and represents the dropout-affected data. Different dropout rates were simulated by varying the parameters of Gamma distribution. The applications of SERM on a large number of datasets with various  dropout rates show its broad applicability. Different methods are followed in  single cell research community to create the dropout-affected data50. Since SERM is based on learning from the data, it is not restricted to a specific type of data. The method can thus adapt to new data easily and provides a broadly applicable strategy for missing data recovery across different disciplines. For this reason, SERM is equally applicable for both raw and normalized  scRNA-seq datasets  are a class of DNNs for data compression in which low dimensional latent features are learned from high dimensional data. It learns only the essential latent features while ignoring non-essential sources of variation such as random noise22. Therefore, the compressed latent data of the network is a representation of the high dimensional ambient data space in lower dimensionality and captures the underlying true data manifold. Thus, when the autoencoder reconstructs the high-dimensional data, it represents the ideal version of the high-dimensional denoised data22. We use a sub-matrix (Ss of size qs × ms) of the expression data (S of size q × m) as the input to the autoencoder network to learn the distribution of the denoised expression data.Deep neural networks (DNNs) have shown promising results in classifying  biomedical dataxd is a row of Ss) and maps it to a latent space ms > P and h is defined asσ is an element-wise activation function which can be a linear or nonlinear (such as sigmoid) function. W and b are the weight matrix and bias vector, respectively. The decoder of the network maps the latent variable h to the reconstructed data So, which is of the same size as Ss.The deep learning architecture (autoencoder) used in SERM consists of two parts, an encoder and a decoder. The encoder of the network takes the input qs is the number of cells and ms is the number of genes. For each expression value, the number of elements with that value in So is counted. The collection of these counts for all the expression values is referred to as the histogram of that region. This function is an empirical estimate of the expression density function. Let So. Furthermore, we define the normalized histogram associated with the possible gene expression values in So as followsn such that Consider the gene expression matrix fg, fr and fe denote Gaussian, Rayleigh and exponential PDFs. The PDF with parameter λ that results in minimum root mean square (RMSE) is chosen for next steps of SERM. We note that although we used three PDFs in our analysis, other PDFs  can be easily incorporated into SERM implementation (see implementation codes). Below we describe the mathematical analysis for histogram equalization of exponential PDF. Analysis for other PDFs can be performed in a similar manner.After applying the histogram equalization, we fit different probability distribution functions (PDF) to S is divided into N ≥ 4 ROIs, which we denote by Si, i = 1, 2, ⋯ , N. Let Si, respectively. Furthermore, we define the normalized histogram associated with the possible gene expression values in the ROI Si as followsn such that We divide the expression matrix f(x), where f(x) can be fg(x), fr(x) or fe(x). As example, the CDF of an exponential distribution can be written asλ denotes the decay parameter. SERM uses the transformation x⌋ is the biggest integer not exceeding x. The map Si.The goal of the histogram adaptive equalization is to find a transformation for the expression values such that the transformed CDF is approximated by the CDF of X denotes continuous random variables whose CDF is strictly increasing on the possible value on pX and pY, respectively. Consider the transformationY has the uniform distribution, i.e., The main intuition behind this transformation comes from the inverse sampling method. In particular, let Z ~ F(t). Combining the transformations T0 and T1, we conclude thatT1∘T0. Similar analytical equations to Eqs.  can can14\doThe ROI is slided throughout the whole expression matrix and gene imputation is performed at each ROI to obtain the resulting imputed gene expression matrix from SERM.i ∈ {1, 2, ⋯  , q} and j ∈ {1, 2, ⋯  , m}. The interpolation formula in Eq.  in the encoder-decoder network was set to 2. L2 and sparse regularizers were used with parameter values of 0.05 and 0.9. The learning rate was fixed at 1e − 5, and the max epoch was set to 20 with a batch size of 64. The adam optimizer was used to train the model. The neuron values were initialized randomly from a uniform distribution in the range of 0–1.Python and Matlab  implementations of the SERM technique were performed. Although the whole dataset can be used for learning the distribution of denoised data, to reduce the computational burden, 2000 data points were randomly selected  embedding dimensions equal to 2, number of nearest neighbors equal to 5, an alpha value equal to 40, automatically determined the optimal number of diffusion steps (t), maximum t equal to 40, number of principal components equal to 100, ‘metric MDS’ as the method of multidimensional scaling, euclidean as the distance function, log as the method of computing the PHATE potential distance, gamma value equal to 0.5, epsilon value equal to 1e − 7, number of landmarks equal to 2000, and number of singular vectors for spectral clustering equal to 100. Monocle, Slingshot, and TSCAN methods were applied to  the datasets using default settings provided by the authors. The computational speed of SERM implemented in Matlab was used to benchmark its computational efficiency against other methods.UMAP implementation from the original authors was used with default parameters56. Indeed, for the examples presented in Supplementary section If a dataset consists of data from different batches, data from any of the batches can be used as the reference. It should be noted that setting one batch-data as reference is a common practice in batch-effect correction techniquesThe following datasets were used to benchmark SERM against different existing imputation techniques:36. Seventeen stromal subsets were identified expressing distinct hematopoietic regulatory genes spanning new fibroblastic and osteoblastic subpopulations including distinct osteoblast differentiation trajectories. Emerging acute myeloid leukemia impaired mesenchymal osteogenic differentiation and reduced regulatory molecules necessary for normal hematopoiesis. This taxonomy of the stromal compartment provides a comprehensive bone marrow cell census and experimental support for cancer cell crosstalk with specific stromal elements to impair normal tissue function and thereby enable emergent cancer.In this dataset, scRNA-seq was used to define a cellular taxonomy of the mouse bone marrow stroma, and its perturbation by malignancy37, which provides unbiased isolation of intact single cells from complex tissues such as adult mammalian brains. The authors profiled 18,194 nuclei isolated from cortical tissues of adult mice. The authors demonstrated through extensive validation that sNucDrop-seq not only accurately reveals neuronal and non-neuronal subtype composition with high accuracy but also allows in-depth analysis of transient transcriptional states driven by neuronal activity.The dataset is from a massively parallel scRNA-seq technology namely sNucDrop-seq (single-nucleus RNA-seq approach)38, which enabled the identification and characterization of previously unknown subtypes of intestinal epithelial cell and their gene signatures.Intestinal epithelial cells absorb nutrients, respond to microbes, function as a barrier, and help to coordinate immune responses. 53,193 individual epithelial cells from the small intestine and organoids of mice were profiled by Haber et al.39. ScRNA-seq data of human-induced neuronal cells were cultured in two different conditions: (1) with mouse astrocytes or (2) with differentiated human astrocytic cells. We analyze the data from the second condition, where hESC (human embryonic stem cells) induced neuronal cells and human astrocytic cells differentiated from hESCs were co-cultured at a 1:1 ratio in a 3D composite hydrogel.The dataset is from a study on human-engineered neural tissues41 was profiled using a massively parallel scRNA-seq technology named Drop-seq2. Data were acquired from the high blastula stage (3.3 hours postfertilization (hpf), moment after transcription starts from the zygotic genome) to six-somite stage . Most cells are pluripotent at the high blastula stage, whereas many cells have differentiated into specific cell types at the six-somite stage.The dataset42.EB differentiation recapitulates key aspects of early embryogenesis. It has been successfully used as the first step in differentiation protocols for certain types of neurons, astrocytes and oligodendrocytes, hematopoietic, endothelial and muscle cells, hepatocytes and pancreatic cells, and germ cells. Approximately 31,000 cells were measured, equally distributed over a 27-day differentiation time course by the authors of ref. 57. Donated tissues were perfused or washed and prepared as single-cell suspensions using specific standard protocols. The analyses included fetal and adult tissue samples and covered 60 human tissue types. Seven types of cell culture, including induced pluripotent stem (iPS) cells, embryoid body cells, hematopoietic cells derived from co-cultures of human H9 and mouse OP9 cells58, and pancreatic beta cells derived from H9 cells using a seven-stage protocol were also analyzed59. Single cells were processed using Microwell-seq60 and sequenced at around 3000 reads per cell; data were then processed using published pipelines8. In total, 702,968 single cells passed the quality control tests , uterus, bladder, ovary, intestine, kidney, lung, testis, pancreas, liver, spleen, muscle, stomach, bone marrow, thymus, prostate, cKit+ bone marrow, bone marrow mesenchymal cells and peripheral blood samples from 6- to 10-week-old C57BL/6 mice were collected. Single cells were then sequenced with Microwell- seq. The sequencing data were processed using published pipelines8. Following the work of the authors60, we analyzed 333,778 cells with 52 unique cell types and 47 unique tissue types.For creating the MCAλgc. We generated the observed datasets by drawing from a Poisson distribution with mean parameter τcλgc, where τc is the cell-specific efficiency loss. We aimed to select roughly 10–20% of genes with the highest proportion of cells with nonzero expression and 50–60% of the cells with the largest library size. All the datasets were converted to transcripts per million (TPM) before filtering so that downsampled datasets for a range of efficiency losses (0.1–10%) can be created. The specific filters used for each dataset are described in supplementary section To generate a reference dataset from real scRNA-seq data, following ref. τc as follows12:τc ~ Gamma10% efficiency: τc ~ Gamma5% efficiency: τc ~ Gamma2% efficiency: τc ~ Gamma1% efficiency: τc ~ Gamma0.5% efficiency: τc ~ Gamma0.2% efficiency: τc ~ Gamma0.1% efficiency: To mimic variation in efficiency across cells, we sampled Detailed statistics of the reference and observed datasets at different sampling efficiencies are added in section 12  by k-means clustering technique with Euclidean distance. Clusters were initialized using k-means++61. We then find the best map of cluster labels compared  to the ground truth labels. These cluster labels are then used to compute the NMI, accuracy, and cluster quality indices: adjusted Rand (AR) and Hubert. NMI is the normalized mutual information62 between the estimated labels and true labels computed following the work of Becht et al.32. Accuracy is the number of correctly found class labels divided by total number of class labels. Hubert and AR indices  are computed using the formula reported in the work of Hubert et al.34.We at first cluster the data into No statistical method was used to predetermine sample size. No data were excluded from the analyses. The experiments were not randomized. There was no blinding. The analyses performed do not involve evaluation of any subjective matters.Further information on research design is available in the Supplementary InformationPeer Review FileReporting Summary