Accurate assessment of the association between continuous variables such as gene expression and survival is a critical aspect of precision medicine. In this report, we provide a review of some of the available survival analysis and validation tools by referencing published studies that have utilized these tools. We have identified pitfalls associated with the assumptions inherent in those applications that have the potential to impact scientific research through their potential bias. In order to overcome these pitfalls, we have developed a novel method that enables the logrank test method to handle continuous variables that comprehensively evaluates survival association with derived aggregate statistics. This is accomplished by exhaustively considering all the cutpoints across the full expression gradient. Direct side-by-side comparisons, global ROC analysis, and evaluation of the ability to capture relevant biological themes based on current understanding of RAS biology all demonstrated that the new method shows better consistency between multiple datasets of the same disease, better reproducibility and robustness, and better detection power to uncover biological relevance within the selected datasets over the available survival analysis methods on univariate gene expression and penalized linear model-based methods. The realization of precision/personalized medicine requires quantitation of relevant prognostic biomarkers for each individual that guide their diagnosis and treatment. Of course, beyond the identification of disease subtypes, one component in the identification of the relevant panel of biomarkers is the association of those variables with patient survival outcomes –4. TheseIn contrast to the categorical assessment described above, many of the clinical determinants and genomic features such as age, smoking duration or gene expression represent continuous variables. The abundance of gene expression data has facilitated the validation of derived biomarkers in datasets from different sources –15 addreAs such, there is a great interest in identifying gene expression biomarkers that show prognostic value for survival outcome. This effort will not only help uncover underlying mechanisms for the diseases of interest but also provide more reliable and feasible biomarkers for clinical application (reviewed in –19).Perhaps surprisingly, gene expression has shown great promise in producing better prognostic biomarkers than clinical factors. For example, gene expression-based classification of cancer patient samples showed a better association with survival outcome than histological classification –22. HoweMany survival analysis methods which have been developed and applied to gene expression-based survival analysis have helped understand the outcome related underlying biology. A partial list of these methods includes gene expression cluster-based sample classification –29, multUsing the Kaplan-Meier method and logrank test to help visually show the estimated survival functions and statistically assess the difference in survival outcome between groups respectively using derived gene expression-based prognostic biomarkers has become a popular approach. However, to our knowledge, there is no single report using them as a discovery tool to directly derive prognostic biomarkers, particularly in the case of gene expression data. Part of the reason for this is because of difficulty in classifying samples with continuous gene expression levels categorically. As a result, it is very common to cast gene expression from a continuous to a binary variable or represent it as a multi-segmented covariate to assess the prognostic impact on survival outcome. Median or quartile-based cutpoint approaches are commonly used to classify the samples into two or more groups to assess differences in their survival outcome in many publicly available survival analysis tools (reviewed in ). The sahttp://www.kmplot.com) that mainly focuses on Kaplan-Meier plots to compare two patient groups that are defined according to a median point (or quartile/tertile points) of expression of the selected gene [Although the discovery of outcome-related biomarkers from gene expression data faces large challenges in validation, many analysis tools and databases that use the logrank test as the primary method have been made available to the medical research community. Many of these tools either claim to be or are intended to be biomarker validation tools. For example, there is a popular web-based tool , a maxin method  was appl p-value . Thus, Pn method  was donen method , it was n method .Gradient-based Cutpoint Scan for Survival Analysis approach, we call GradientScanSurv, which extends the logrank test method from handling discrete variables to handling any continuous variable. Since our method provides a summary across the entire gradient with coupled statistical significance, we feel that this ameliorates the aforementioned pitfalls in many existing tools. The novel method appears to have better overall performance within the tested example datasets in uncovering and validating the association of gene expression with survival outcome.In summary, as we will demonstrate in this report, many of the survival analysis tools and published studies that have utilized these tools may be sensitive to an analysis pitfall that can lead to an incorrect assessment of survival association of the data. With kmplot, the pitfall is that the user may be misled by fortuitously evaluating the only user-selected cutpoint that shows significance, or worse, by reporting only the point(s) that did. With PrognoScan and indeed kmplot as well, the pitfall is that with only a single readout for the association, the trend across the gradient is not made available. Our sense is that with these pitfalls, the robustness of the identified biomarkers, particularly in the context of a discovery approach has room for improvement. Therefore, we propose a novel http://www.prognoscan.org/). Briefly, Unix curl command-based customized scripts were used to download all of the LUAD expression data and metadata for survival outcome, as well as the analysis results of the PrognoScan method from PrognoScan website for all RAS pathway genes. Customized R scripts were used to parse the downloaded data to be used for comparison. The survival data and RNAseq data from TCGA (The Cancer Genome Atlas) for selected tumor types were downloaded from the TCGA data portal . Australian pancreatic adenocarcinoma (PAAD) cancer data was downloaded from the ICGC website (https://dcc.icgc.org/projects/PACA-AU) as AusPanc_Set dataset.Survival data and gene expression data from lung adenocarcinoma (LUAD) including microarray data were downloaded from the PrognoScan website  cancer data and TCGA data were analyzed in a similar way using customized R scripts that implemented the same methods.The R code for the PrognoScan analysis method was obtained through email communications with the original author of PrognoScan , who refPenalized linear model-based methods including Lasso and Elastic Net methods, which are Cox models with the added feature that allows them to be able to do variable selection, were implemented in customized R scripts using glmnet R package, and run on the TCGA data of different tumor types using multiple seeds (100) for multiple trials. The recurrent genes across multiple trials were collated into the aggregated lists based on a dynamic cutoff empirically determined for each tumor type with a minimal 85% percent times of all trials.For multiple genes’ results such as results for all 225 RAS pathway genes, multiple comparisons were corrected by B-H (Benjamini & Hochberg method ) using tFig 1, the GradientScanSurv method first ordered all the measured values of the input gene or biomarker across the gradient. Then at each cutpoint along the ordered gradient, a logrank test was used to derive a p-value to assess the difference of survival outcome between the two groups split by the corresponding cutpoint. These p-values are collected, assessed, and counted to obtain a GoodCount for the real data. GoodCount was defined as numbers of cutpoints where the corresponding separated two groups had significant p-value (p< = 0.05 as the default setting) for their logrank tests.As illustrated in yn×1 and censoring indicator δn×1. For gene expression j, we observe n subjects with gene expression value xj = T. Our approach first calculates the logrank test at each possible cutting point of the ordered gene expression values xj. To be specific, denote the ordered gene expression values as Suppose we observe n observations with the vector of survival outcome This is essentially treating the binary indicator Note that these logrank test p-values at each cutpoint do not directly provide grounds for significance for the overall association relation, but they do form the basis for the next level of aggregate statistics described below as GoodCount and GoodCount p-value that are indicative of a significance level of the overall association relationship. In our typical graph view, our method provides green lines as tentative significant cutpoints along the gradient, which provides not only the basic metrics for the next level of aggregate statistics but also the visual cue for the likelihood of the significance for the overall association .α as the significance level, the default setting of 0.05) across all possible cutpoints:A GoodCount is then defined as the number of significant tests  was defined as the proportion of the times that the GoodCounts of permutated datasets were no less than that of the original dataset. By comparing the GoodCount values from real data and those from permutated data, which work against each other and respond to the alpha level simultaneously, the derived GoodCount p-value is the final deterministic statistic for whether there is a significance of the association. Explicit derivation of the theoretical p-value under this situation could be difficult. Hence, we choose to use the robust bootstrapping method, to derive the GoodCount p-value for an unknown distribution.Practically, to create the permutated datasets, the original survival data and expression data were permutated for at least M> = 1000 times by shuffling the expression data between sample names. The GoodCountPval is the key statistics produced by the GradientScanSurv method that is used to assess the significance of the association of the individual gene expression or biomarker with the survival outcome.ROC analysis was performed on the results of each survival analysis method under evaluation (see above) using independently derived datasets of the same disease from different sources as the Training Set and Testing Set respectively. Alternatively, shared positive gene lists from at least two of the methods from the Training Set were used as Truth lists for each method in the Testing Set data. To test whether the selection of which dataset was used as the Training Set and which was used as the Testing Set had an impact on the results, the datasets used for Training Set or Testing Set were switched so that any potential bias imposed would be tested and evaluated in our comparison. Specifically, PrognoScan LUAD data and TCGA LUAD data were downloaded and have been used separately as either the Training Set or the Testing Set in the ROC analysis respectively. Alternatively, the TCGA LUAD data was used as the Training Set and the PrognoScan LUAD data was used as the Testing Set. Thus, our aggregate results are composed of multiple datasets of PrognoScan LUAD data and multiple trials of analysis results using the TCGA LUAD dataset.To make sure the results of the GradientScanSurv method were reliable and robust, multiple trials (100) of analysis results for GradientScanSurv method were obtained. Since other methods do not have a permutation test, their results for an input data set remain constant. Our comparison applied the results from multiple trials of GradientScanSurv with results from other methods analyzing the same TCGA data as described. There are also multiple PrognoScan datasets  from various microarray platforms and derived from different studies. Each of these datasets was analyzed by all methods to obtain corresponding results, which were combined as the PrognoScan results of multiple trials.Ultimately, the ROC analysis was then used to assess the performance of each method between these Training Sets and Testing Sets of multiple trials to derive the overall performance statistics for each method i.e., percentage of the total times that each of these selected methods being compared had the largest AUCs compared to other methods in each of the combined trials. In addition, the average AUCs were derived for comparison between the selected methods for each subset of total trials of data defined by each of the 9 PrognoScan datasets in combination with the 100 TCGA trials of data, simply because the PrognoScan datasets are much more diverse in platforms and data sources compared to trials of the same TCGA data.The TCGA and PrognoScan data are independent datasets from different sources but derived from the same disease . The corresponding Training and Testing Sets were derived for a total of 225 RAS pathway genes for each method for each trial of the analysis. The analysis results of each trial are essentially adjusted p-values of each gene derived from each method that were subjected to multiple test correction by the Benjamini & Hochberg method .The ROC analysis used to compare methods was based on the assumption that when we used independent datasets of the same disease for ROC analysis with one as the Training Set and the other as the Testing Set , if the method is better, we would get better AUCs in the ROC analysis, since we would expect more consistent results in the classification of Ras pathway genes either associated or not associated with survival outcome. We did the analysis this way since there is no valid truth set of survival data available for our purpose that reports which genes are truly associated with survival outcome for this particular disease.p-value. Permutations naturally caused some variation in results from trials to trials. However, the method is not impacted by those fluctuations. The default setting for the number of permutations was 1000 that was identified as a practical level from our initial tests with many datasets that were observed to have relatively stable results.By design, the GradientScanSurv method uses a permutation (bootstrap) method to derive the GoodCount Although functions of the pROC R package were used as the main procedure for ROC analysis and plotting, customized R scripts were used to collect the statistics from pROC analysis on multiple trials of both Training and Testing Sets for the purpose of performance comparison of the selected survival analysis methods.We began by characterizing the pitfalls associated with methods that offer a limited set of predefined cutpoints that split the data into categories for subsequent survival association assessment. We observed several common pitfalls in many publicly available survival analysis web tools that are used for putative biomarker validation or analysis of the association between survival and gene expression data. These tools have been previously described  and haveTable A in S1 File). In no way does our selection reflect or judge those manuscripts other than to use them as examples of the potential pitfalls associated with these tools. In the original Fig 3B in [kmplot.com, [Figure A in S1 File. Interestingly, we found the result of one option (lower quartile) gave a very similar result to that reported in the original Fig 3B in [Figure A in S1 File). Thus, it appears that the authors reported the only positive result of the 5 options and made no reference to the other possible outcomes. We are by no means condemning their work as they simply used the tool as advertised. However, this is a perfect example of how a user can become the victim of unintended consequences through inadvertent or otherwise selective parameter choices. This suggests the need for a more unbiased method for parameter selection in these survival association tools.We detail two examples chosen as representative of the issues we identified that appeared in highly ranked journals –36, althlot.com, ). Their kmplot.com) on the RELN gene for lung adenocarcinoma (LUAD) is reported. In this study [Figure B in S1 File). In contrast to the first study [In the original Fig 6E in  of the sis study , the figst study  where onFigure C in S1 File). Here, the logrank p-value is non-significant at either the median point or all of the quartile or tertile points, in contrast to the result using microarray data [Figure C in S1 File). Of course, consistency across different platforms would be desired of any useful biomarker and so this observation is worrisome if RELN indeed is associated with lung cancer survival.To test the robustness of this result, we ran our method (see below) on the RELN gene using downloaded TCGA LUAD RNAseq data  that produces the desired result. This led us to propose a novel approach, termed GradientScanSurv, to assess and visually present the gene expression and survival relationships in a more comprehensive way by considering all the cutpoints along the expression gradient.More examples of common pitfalls of survival analysis in publicly available survival analysis web tools and biological studies that have used these tools are listed in Figure C in S1 File), indicative of the potential association of this gene with survival outcome (see below). This result is consistent with the microarray data result and would have been missed if using the methods of Kaplan-Meier Plotter website (kmplot.com) to analyze the TCGA RNAseq data.As a quick proof of concept, with respect to the RELN gene example just described, we do observe a significant GoodCountPval as 0.0405 using GradientScanSurv to analyze the TCGA RNAseq data . In addition, the graphical display of the GradientScanSurv results provides additional insights regarding the extent and direction of the association that facilitates the assessment of association significance . Most, if not all of the publicly available survival analysis web tools only provide directionality for the overall relationship between survival outcome and expression change direction  at the predetermined points by Kaplan-Meier plots. Our tool also provides visual cues including vertical green lines across the expression gradient for cutpoints with tentatively significant p-values from logrank tests and brown diamonds at either the top or at the bottom for directionality of the gene expression changes versus survival outcome. This provides a visual measure of the consistency of the direction across the gradient as well.In addition, we introduce the GoodCount p-value  metric as the key statistical assessment of whether there is a significant number of cutpoints along the expression gradient that shows a significant difference in survival outcome compared with a random chance that was assessed through expression data permutation-based trials of datasets  for genes from the RAS pathway such as NRAS, KRAS, and MAP2K1genes for Lung adenocarcinoma (LUAD) data. We then downloaded the raw data from the PrognoScan website for these genes and then ran our GradientScanSurv method so that we could compare the results side by side with those produced by the PrognoScan method. Interestingly, PrognoScan has multiple datasets for the same disease for the same genes, which allowed us to assess how consistent the results were for each method.As described in the introduction, the PrognoScan method addresses the selection of cutpoints from a predefined list by evaluating all possible cutpoints and returning a corrected p-value for the optimal cutpoint. In order to directly compare our method with the PrognoScan method, we ran the PrognoScan analysis on its website . This is confirmed by the GradientScanSurv method with a very significant GoodCountPvals (= 0.001 with permutation n = 1000)  using the TCGA LUAD dataset with RNAseq data for expression values. This shows that even extending to data derived from a different platform (RNAseq), our method shows better consistency compared with the microarray platform for PrognoScan data. Given the sample size limitation of these datasets, platform differences, data source differences, the relatively high proportion of the multiple datasets showing consistent positive results we observed in our validation is very reasonable and of relatively high confidence.KRAS and NRAS mutations are known to be critical oncogenesis driver genes mutated in LUAD, although NRAS mutations are relatively rare . NRAS exFig 2A, where the GradientScanSurv method obtained a significant p-value that the PrognoScan method missed, the GradientScanSurv method accumulates the significant logrank test p-values that are scattered along the gradient making the GoodCountPvals significant , whereas the PrognoScan method did not attain significance because the minimal p-value is of insufficient magnitude  in spite of widespread cutpoints along the expression gradient showing significant p-values. This confirms the importance of aggregating statistics across all cutpoints along the expression gradient in the GradientScanSurv method rather than just using predefined cutpoints  used by other methods.For the dataset indicated by the red arrow in Figures F and G in S1 File), whereas only 1 out of 11 datasets showed a significant association with the PrognoScan method . This is also confirmed by the GradientScanSurv method with a significant GoodCountPvals (0.031 with permutation n = 1000)  with the TCGA LUAD dataset using the RNAseq platform for expression data. For the dataset indicated by arrows on Figure F, panel A and Figure G, panel A in S1 File, the GradientScanSurv method obtained a significant p-value whereas the PrognoScan method did not.We also compared another important RAS pathway gene, MAP2K1, which is downstream of RAS and may also be expected to be associated with survival outcome. We observed 3 out of 11 datasets that showed a significant association with survival outcome for MAP2K1 by the GradientScanSurv method . Conversely, only 3 out of 17 datasets showed a significant association with the PrognoScan method . This is also confirmed with the GradientScanSurv method by a significant GoodCountPvals (= 0.004 with permutation n = 1000)  using the TCGA LUAD dataset with the RNAseq platform for expression data. In addition, the cue from the GradientScanSurv plot shows that higher expression of the KRAS gene is associated with faster death , which is expected from the known RAS biology in LUAD cancer study [At the mutation level, KRAS is believed to be one of the most critical oncogenic genes in LUAD. When we examined the association between KRAS expression and survival, we found that 4 of 17 datasets showed a significant association with survival outcome using our GradientScanSurv method . This is further supported by the GradientScanSurv result with two additional KRAS datasets measured by the same probeset 204010_s_at . In contrast, another dataset using a different probeset 214352_s_at that also measures KRAS gene expression showed a consistent result with TCGA result: higher expression of KRAS gene is associated with faster death . Interestingly, a dataset that used another probeset (204009_s_at) for the KRAS gene also showed the same direction at the cutpoints with significant logrank test p-values , where higher expression of KRAS gene is associated with faster death, even if the dataset overall did not show significant GoodCountPvals.Looking at different probes, however, the GradientScanSurv result using microarray data from PrognoScan showed that lower expression measured by KRAS probeset 204010_s_at is associated with faster death (https://genome.ucsc.edu/cgi-bin/hgTracks) to visualize their genomic locations within the KRAS locus that was provided by Affymetrix . Interestingly, the two probesets that showed consistent results with TCGA result (204009_s_at and 214352_s_at) aligned with the last exon in KRAS , and they correlate very well with each other in the expression data of these datasets from PrognoScan . In contrast, the probeset 204010_s_at that showed the opposite direction with the TCGA result indeed is at the far end and off the last exon of the KRAS gene . In addition, this probeset correlated to a much lower extent with the other two probesets . It is very likely that the last probeset may not really measure the KRAS gene expression or is interfered with by a transcript from the gene on the other side of the KRAS locus. Thus, its directionality of survival change may be opposite to that of the real KRAS gene. Therefore, all of these GradientScanSurv results are consistent in their significance as well as in their change directions. This suggests that the ability to visualize the direction of the survival change by the GradientScanSurv method is valuable and useful in further validation of the analysis results. This allows for consistency of the survival status direction along the expression gradient to be visualized, a feature absent from all other methods that focus on predefined or limited cutpoints.To resolve this seemingly contradictory observation for the different probesets of the KRAS gene, we used UCSC Genome Browser (kmplot.com). Univariate Cox regression is also a commonly used method along with the Kaplan-Meier estimator and logrank test. Rank-based Cox regression is an alternative way to perform univariate Cox regression on gene expression but using the ranks of gene expression for regression instead of the absolute expression levels.The examples above demonstrate that on a limited basis, our new method appears to outperform PrognoScan with the tested datasets, but we wanted to perform a more systematic assessment of performance across multiple available methods. In order to test whether the GradientScanSurv method performs better than other survival analysis methods using univariate expression of each gene, receiver operating characteristic (ROC) analysis was performed on the same datasets for several commonly used methods in the field including PrognoScan, univariate Cox Regression on gene expression levels, univariate Cox Regression on ranks of gene expression, median cutpoint-based logrank test, and tertile cutpoint-based (top quantile vs bottom quantile) logrank tests. The last two methods are typical methods commonly used by Kaplan-Meier Plotter website . Then, ROC analysis was performed on the TCGA LUAD data as the Testing Set . Within the multiple trials of test data that were produced, the GradientScanSurv method has the largest percentage (48.41%) of total trials with the largest AUCs compared to other methods . In addition, the GradientScanSurv method has the largest average AUCs compared to other methods in 3 out of 4 subsets of trials of data defined by 100 trials of TCGA data combined with one of the PrognoScan datasets, which have valid AUCs in ROC analysis . It should be noted that the results are very different from trial to trial (Tables B and C in S1 File).We evaluated the performance of different methods using genes shared between methods as a Truth Set and also using genes identified by each individual method. In the first case, ROC analysis was first performed using the consensus gene lists identified and shared between the selected methods with datasets of Training Sets for validation by each of these individual methods in datasets of Testing Sets. Briefly, the PrognoScan LUAD data was first used as a Training Set to create a series of Truth Sets defined by being shared between the comparable lists produced by the different methods at different frequencies . In spite of smaller margins, the GradientScanSurv method still had the largest percentage (20.32%) of total trials with the largest AUCs compared to other methods . In addition, the GradientScanSurv method has the largest average AUCs compared to other methods in 4 out of 9 subsets of trials of data in combination with one of the PrognoScan datasets, which is more than any other methods . We used the Truth sets that were shared Truth lists from two or more of these methods to increase the reliability of the input Truth lists. Again, the GradientScanSurv method consistently performed the best in spite of variations between the datasets.In order to test whether the selection of which dataset was used as the Training Set and which was used as the Testing Set impacted the results, the TCGA LUAD data was first used as the Training Set to create a series of Truth sets as shared Truth lists at different frequencies of positive lists derived from each of selected methods. As before, ROC analysis was then performed on the PrognoScan LUAD data as the Testing Set , the GradientScanSurv method has the largest percentage (49.56%) of total trials with the largest AUCs compared to other methods . In addition, the GradientScanSurv method has the largest average AUCs compared to other methods in 5 out of 9 subsets of trials of data defined by 100 trials of TCGA data combined with one of the PrognoScan datasets . In this particular analysis, the Median cutpoint-based logrank test showed the worst performance with no single positive calls in the Training Set identified in contrast to all other methods (Tables D and E in S1 File).In order to directly test which of the methods made more consistent calls with different datasets of the same disease, ROC analysis was performed using the individual gene lists identified by each of the selected methods with datasets of Training Sets for validation by each of these individual methods in datasets of the Testing Sets. Either TCGA LUAD data was used as the Training Set and PrognoScan LUAD data as the Testing Set, or vice versa. For the first scenario where TCGA LUAD data was used as the Training Set and the PrognoScan LUAD data as the Testing Set , the GradientScanSurv method still achieved the largest percentage (36.71%) of total trials with the largest AUCs compared to other methods . In addition, the GradientScanSurv method has the largest average AUCs compared to other methods in 3 out of 7 subsets of trials of data defined by 100 trials of TCGA data combined with one of the PrognoScan datasets with valid AUCs in ROC analysis, which is more than that of any other methods . A relatively large proportion of the trials with NA’s for most or all methods may imply that some datasets may have intrinsic issues such as the curation of survival data or technical issues on expression data (Tables F and G in S1 File) so that none of the methods could achieve any positive calls.For the second scenario where PrognoScan LUAD data was used as the Training Set and the TCGA LUAD data as the Testing Set , although in one case (shown in a row in bold in Tables H and I in S1 File), only the GradientScanSurv method has a significant p-value indicating that GradientScanSurv has the most consistent performance using the standard adjusted p-value cutoff (< = 0.05). In addition, NRAS was consistently identified by the GradientScanSurv method as well as by PrognoScan and Cox regression method (on gene expression), indicative of presumed biologically relevant results, since NRAS expression is correlated with KRAS mutational status in LUAD [In order to further compare the actual results in terms of their underlying biological meaning, we collected the shared genes identified by all of the methods from multiple trials of LUAD datasets combined from both TCGA and PrognoScan data. Listed in  in LUAD  and thusTable 1): one from the TCGA PAAD dataset (TCGAPAAD_Set), and the other from Australian Pancreatic Cancer dataset (AusPanc_Set). Again, only the cases with significant overlapping genes between the two datasets were considered (Table 1). Interestingly, the NRAS gene was consistently found in the lists of multiple trials of data only from the GradientScanSurv method and only when there is a statistically significant overlap between the TCGAPAAD_Set and AusPanc_Set (Table K in S1 File). NRAS was only seen in the shared gene lists from the GradientScanSurv method but not from those of all other methods, indicative of its best performance in uncovering genes with likely biological relevance. In the case of PAAD, where more than 94% Pancreatic cancer patients harboring KRAS mutations [To further evaluate the underlying biological relevance from the resulting gene lists from these methods using different data sources, we compared the shared genes derived from each of these methods between two independent pancreatic cancer (PAAD) datasets . However, the GradientScanSurv method identified the other two RAS genes (KRAS and HRAS) as positive genes in PAAD, which were missed by the Lasso method. Once again, with more than 94% Pancreatic cancer patients carrying KRAS mutation [Table 2). Very similar results were observed for the Elastic Net method, which also missed the FOSL1 gene in both LUAD and PAAD results (Table M in S1 File). FOSL1 was recently identified as a novel downstream effector of KRAS and higher FOSL1 expression was associated with poor survival outcome in both lung and pancreatic cancer patients [As expected and discussed above, NRAS was once again identified by both the GradientScanSurv and Lasso method in LUAD showing strong association with patient survival outcome  or Elastic Net method (Table M in S1 File). However, both the Lasso and Elastic Net methods identified many more genes in BRCA data than the GradientScanSurv method. This may suggest that the number of genes in the positive lists of each method are dataset-dependent rather than method-dependent. Similarly, although LUAD, PAAD, COAD, and READ are associated with RAS mutational status and oncogenic activity, COAD has no positive genes for all of the test methods. READ had only one gene (MET) identified by the GradientScanSurv method. Interestingly, high MET expression has been found to be related to a worse prognosis and mortality for colorectal cancer patients [We also noticed that there were many genes identified by the GradientScanSurv method in PAAD, many more than those detected by Lasso (patients . READ anpatients .Table 2 and Table M in S1 File). All of these observations suggest that despite the caveats associated with this comparison, the GradientScanSurv method appears to have better detection power compared to Lasso and Elastic methods to uncover the biologically relevant genes that are associated with cancer patient survival outcomes.In addition, the genes DUSP6 , PIK3CA Historically, survival analysis has been focused on three types of goals: outcome-related gene finding, class discovery, and supervised prediction . In thisThe highlighted examples illustrate the common pitfalls in publicly available survival analysis and validation web tools as well as the biological studies that have used these tools. Looking through the literature, one can easily find examples of the Kaplan-Meier estimator method and logrank tests that applied a limited set of predefined cutpoints to separate samples into high and low categories of genes, biomarkers, or prognostic scores, etc. In most, if not all cases, there is no justification or rationale provided in the published studies regarding why the particular cutpoint(s) was chosen if in fact that information is provided at all. They also do not discuss whether using other cutpoints did or did not produce significance. The first example regarding the KIAA1522 gene expression illustrated an implied bias using Kaplan-Meier Plotter website resulting from the fact that the given options for the tool include only a limited set of pre-defined cutpoints. The second example regarding RELN gene expression especially with the TCGA RNASeq LUAD dataset showed how we could miss supporting evidence, leading to a conflicting conclusion. This results from the fact that by chance the relation does not have logrank test significance at the commonly applied cutpoint.We have introduced these papers simply as examples of potential pitfalls when using the available tools in the community. Their use was not meant to criticize the authors of these two papers for using a popular survival analysis tool, but rather to precaution the field as to how pitfalls can exist in commonly used tools. We used these two examples to show how a limited selection of analysis results from used tools could impact research results. In any case, our goal was to expose these aspects in order to avoid recurrence of such cases.Such common pitfalls exist widely in the research community simply because no method has been reported that correctly handles the extrapolation of continuous data to categorical analysis or that comprehensively derives a p-value to assess, uncover, and validate the association of gene expression with survival outcomes. Our GradientScanSurv method potentially fills this gap by enabling the logrank test method to handle continuous variables and systematically assess the overall association with survival outcome along the ordered gradient in a comprehensive way. In addition, the visualization cues presented in the GradientScanSurv plots as vertical green lines representing the cutpoints, at which tentative cutpoint-wise significances of survival outcome were observed, provide a novel and informative way to visually sense the extent of the association of the expression of the genes with survival outcome. Furthermore, the directionality of the changes in outcome related to the expression levels can be seen from the GradientScanSurv plots as well. These visual cues provide good overall indicators of the data along the expression gradient that can consolidate the GoodCountPval regarding how well the gene expression is associated with survival outcome.The PrognoScan method that derives a corrected p-value from the minimal p-value obtained along the expression gradient, is probably the most comprehensive logrank test method-based approach prior to the development of our GradientScanSurv method. Its main drawback is that it fails to provide a comprehensive consideration of the overall behavior along the whole expression gradient on the survival difference between the high and low expression groups. Rather, it produces a minimal p-value that is derived from one, albeit optimal, cutpoint. Although PrognoScan does evaluate the continuum of cutpoints to identify the optimal cutpoint and then derive a corrected minimal p-value from that, it fails to convey information about the rest of the continuum to uncover the association of gene expression with survival outcome. We assert that the GradientScanSurv method presumably corrects this deficiency by assessing the overall behavior of data for all the cutpoints along the expression gradient for the extent of association between gene expression and survival outcome. The side-by-side comparisons between the GradientScanSurv method and the PrognoScan method showed that GradientScanSurv better tolerates variations in datasets from the same disease but derived from different sources. Further, a more extensive global comparison with other survival analysis methods showed that our GradientScanSurv method has better overall performance in terms of consistency in multiple datasets tested by ROC analysis. It also appears to provide more biologically relevant results by evaluating the positive genes shared within the analysis results from independent datasets of the same diseases using the same analysis method.It is possible to conclude that our method categorizes a continuous variable and there are plenty of methods that treat continuous variables as continuous such as the Cox model etc. However, we have directly compared our method with Cox models that treat expression data as a continuous variable in the model or even treat the ranked orders of the expression data as a continuous variable and indeed showed that our method is better than Cox models in both cases. In our view, considering many covariates such as age, tumor size etc simultaneously would likely confound the impact of gene expression on survival outcome when using the Cox model. Our method simplifies the question by testing how strongly the expression of an individual gene is associated with survival outcome by considering one gene at a time so that only the strong potential biomarker(s) are captured to move forward with more robust validation with multiple factors.In addition, we would like to point out that by considering all cutpoints, comparing them and combining their logrank test statistics to an aggregate statistic, our proposed method achieves something that cannot be done by simply categorizing a variable. On the other hand, nonparametric methods all inevitably involve tuning parameters, for example, in spline, we have to choose a number of tuning parameters, such as the degrees, the number of knots, etc, and tuning all these parameters is a much more involved task. In contrast, our method is extremely simple and straightforward, and it is able to detect most nonlinear relationships.https://www.cancer.gov/research/key-initiatives/ras). Thus, the analysis results can be consolidated with expected biological themes with much better confidence.When we used ROC analysis for global performance comparison, we alternately used either TCGA LUAD data or PrognoScan LUAD data as the Training Set or Testing Set. Since these datasets are derived from the same disease, they can be used as Training and Testing datasets respectively for cross-validation. This is more stringent than k-fold cross-validation within each dataset as is usually done. We prefer this approach because sample size would be reduced in the Training and Testing datasets due to a further partitioning of the samples if k-fold cross-validation were used. Of course, the reduction in sample size would be expected to have a significant impact on survival analysis. Therefore, cross-validation within two independent datasets of the same disease keeps the sample size as large as in the Training Set and Testing Set respectively. It also adds more stringent criteria to the cross-validation due to the independent data sources and even different platforms and technologies that created the datasets. In addition, it leverages the much better understanding of these diseases for their impact of the genes in the RAS pathway that can be credited to the accumulated studies in the field over the years as well as the more recent RAS Initiative , which may lead to the sub-optimal power of the proposed method. However, our comparison results may suggest the Pro may outweigh the Con side, since there are so many genes encoded by the genome, considering them altogether may make the model clumsy and hard to interpret. If we scan the genes one at a time with GradientScanSurv to collect a list, and then use these integration methods to consider the positive hit genes derived from the first-round of GradientScanSurv analysis later, this may be a feasible and reasonable way to study the interactions of these genes, but this is outside of scope of this study.Our GradientScanSurv method assesses one gene at a time, which has Pros and Cons. An obvious Pro is that this method does not have to include other factors and other genes in the analysis model presumably diminishing a tendency to over-fit the model since most datasets have a limited number of patients. This simplifies the method to only consider one variable at a time, and also make its interpretation more straightforward . The Connd round. This procedure can be flexibly used for biomarker discovery at the individual gene level or at the aggregate level of multiple “good” biomarker genes. The aggregate biomarkers can be flexibly tested for permutated combinations of the identified potential individual biomarkers until better aggregate biomarkers are obtained.As mentioned earlier, gene expression cluster or gene signature-based survival analysis biomarkers have encountered issues of reproducibility and robustness . Many ofOur method is not limited to application with gene expression data. In fact, any continuous variable can be used with our method for survival analysis for its association with outcome. The univariate Cox regression model is sensitive to the actual data range and scale, since as we tested the Cox regression model with either actual expression levels or ranks based on relative expression levels and these gave quite different results. This is why we also compared Cox regression results using the rank of gene expression in addition to using the actual expression intensities. Our method can be used to derive risk scores that classify the cancer patients either through a combination of risk scores of each potential gene or through aggregate risk scores of combined potential gene lists.Taken together, our results show that the GradientScanSurv method can be used as either a validation or discovery tool for biomarkers of continuous variables. It produces better performance than all of the selected methods we compared for the selected example datasets. Our comparisons included most typically existing methods and commonly used public web tools in the field to directly assess the association of gene expression with survival outcome to our knowledge, although it is impossible to compare all of the tools in the field at one time. Our comparative analysis has also revealed the importance of carefully capturing survival data along with the associated metadata for ongoing genomic disease studies. The data we accessed have relatively high levels of sample drop out in terms of life-status reporting that significantly hampered our analysis and we look forward to applying this method to more complete datasets as they emerge.S1 FileTables A-M and Figures A-Q that were pooled into one single Supplementary Data file.Supporting information including (DOCX)Click here for additional data file.


Dimensionality reduction is an indispensable analytic component for many areas of single-cell RNA sequencing (scRNA-seq) data analysis. Proper dimensionality reduction can allow for effective noise removal and facilitate many downstream analyses that include cell clustering and lineage reconstruction. Unfortunately, despite the critical importance of dimensionality reduction in scRNA-seq analysis and the vast number of dimensionality reduction methods developed for scRNA-seq studies, few comprehensive comparison studies have been performed to evaluate the effectiveness of different dimensionality reduction methods in scRNA-seq.We aim to fill this critical knowledge gap by providing a comparative evaluation of a variety of commonly used dimensionality reduction methods for scRNA-seq studies. Specifically, we compare 18 different dimensionality reduction methods on 30 publicly available scRNA-seq datasets that cover a range of sequencing techniques and sample sizes. We evaluate the performance of different dimensionality reduction methods for neighborhood preserving in terms of their ability to recover features of the original expression matrix, and for cell clustering and lineage reconstruction in terms of their accuracy and robustness. We also evaluate the computational scalability of different dimensionality reduction methods by recording their computational cost.www.xzlab.org/reproduce.html.Based on the comprehensive evaluation results, we provide important guidelines for choosing dimensionality reduction methods for scRNA-seq data analysis. We also provide all analysis scripts used in the present study at  Single-cell RNA sequencing (scRNA-seq) is a rapidly growing and widely applying technology –3. By mek-means clustering in the PCA space to eventually produce linear trajectories [Dimensionality reduction is indispensable for many types of scRNA-seq analysis. Because of the importance of dimensionality reduction in scRNA-seq analysis, many dimensionality reduction methods have been developed and are routinely used in scRNA-seq software tools that include, but not limited to, cell clustering tools , 13 and ectories . SLICER ectories . Monocleectories , 28. Wisectories .Besides the generic dimensionality reduction methods mentioned in the above paragraph, many dimensionality reduction methods have also been developed recently that are specifically targeted for modeling scRNA-seq data. These scRNA-seq-specific dimensionality reduction methods can account for either the count nature of scRNA-seq data and/or the dropout events commonly encountered in scRNA-seq studies. For example, ZIFA relies on a zero-inflation normal model to model dropout events . pCMF moWith all these different dimensionality reduction methods for scRNA-seq data analysis, one naturally wonders which dimensionality reduction method one would prefer for different types of scRNA-seq analysis. Unfortunately, despite the popularity of scRNA-seq technique, the critical importance of dimensionality reduction in scRNA-seq analysis, and the vast number of dimensionality reduction methods developed for scRNA-seq studies, few comprehensive comparison studies have been performed to evaluate the effectiveness of different dimensionality reduction methods for practical applications. Here, we aim to fill this critical knowledge gap by providing a comprehensive comparative evaluation of a variety of commonly used dimensionality reduction methods for scRNA-seq studies. Specifically, we compared 18 different dimensionality reduction methods on 30 publicly available scRNA-seq data sets that cover a range of sequencing techniques and sample sizes , 14, 41.FIt-SNE R package [www.xzlab.org/reproduce.html or https://github.com/xzhoulab/DRComparison.We evaluated the performance of 18 dimensionality reduction methods  and evaluated the performance of these dimensionality reduction methods based on neighborhood preserving. Neighborhood preserving measures how the local neighborhood structure in the reduced dimensional space resembles that in the original space by computing a Jaccard index   is rarely, if ever, used directly in any downstream single-cell applications including clustering and lineage inference, even though it is computationally easy to do so. Therefore, we will focus our main comparison in two important downstream single-cell applications listed below.Splatter package , but generally increases with the number of included components. In addition, consistent with the k-means clustering results, we found that generic dimensionality reduction methods often yield results comparable to or better than scRNA-seq-specific dimensionality reduction methods , but generally increases with the number of included components. With a low number of low-dimensional components, ZINB-WaVE (0.72) achieves the best performance  and LTSA (0.21) do not fare well. We also note that the clustering results obtained by Louvain method are often slightly worse than that obtained by k-means clustering and slightly better than that obtained by hierarchical clustering across settings , in addition to the log2 count we used in the previous results , PCA achieves an NMI value of 0.82, 0.82, and 0.81, for log2 count transformation, log2 CPM transformation, and z-score transformation, respectively  directly model count data, many dimensionality reduction methods  require normalized data. The performance of dimensionality reduction methods that use normalized data may depend on how data are normalized. Therefore, we investigated how different normalization approaches impact on the performance of the aforementioned dimensionality reduction methods that use normalized data. We examined two alternative data transformation approaches, log2 CPM , and scRNA-seq data generated from UMI-based technologies  are often of large scale, come with almost no amplification bias, do not display apparent dropout events, and can be accounted for by over-dispersed Poisson distributions. In contrast, data generated from non-UMI-based techniques  are often of small scale, have high capture rate, and come with excessive dropout events. Subsequently, the unwanted variation from these two types of dataset can be quite different. To investigate how different dimensionality reduction methods perform in these two different types of data sets, we grouped 14 cell clustering data sets into a UMI-based group (7 data sets) and a non-UMI-based group (7 data sets). In the UMI-based data sets, we found that many dimensionality reduction methods perform reasonably well and their performance is relatively stable across a range of included low-dimensional components  are 0.74, 0.79, 0.79, and 0.79, respectively  do not influence the performance of dimensionality reduction methods. For example, with the lowest number of low-dimensional components, UMAP achieves a Kendall correlation of 0.49, 0.47, and 0.47, for log2 count transformation, log2 CPM transformation, and z-score transformation, respectively  and applied different dimensionality reduction methods to analyze the final set of 50,000 cells. Because most dimensionality reduction methods are not scalable even to these 50,000 cells, we only examined the performance of dimensionality reduction methods when paired with the sub-sampling procedure  to evaluate the performance of different dimensionality reduction methods for downstream clustering analysis. We have also primarily focused on two lineage inference methods (Slingshot and Monocle3) to evaluate the performance of different dimensionality reduction methods for downstream lineage inference. In our analysis, we found that the performance of dimensionality reduction methods measured based on different clustering methods is often consistent with each other. Similarly, the performance of dimensionality reduction methods measured based on different lineage inference methods is also consistent with each other. However, it is possible that some dimensionality reduction methods may work well with certain clustering approaches and/or with certain lineage inference approaches. Subsequently, future comparative analysis using other clustering methods and other lineage inference methods as comparison criteria may have added benefits. In addition, besides cell clustering and trajectory inference, we note that dimensionality reduction methods are also used for many other analytic tasks in scRNA-seq studies. For example, factor models for dimensionality reduction is an important modeling part for multiple scRNA-seq data set alignment [In the present study, we have primarily focused on three clustering methods  will generate scRNA-seq data sets that contain millions of cells . The larhttps://www.ncbi.nlm.nih.gov/geo/) or the 10X Genomics website (https://support.10xgenomics.com/single-cell-gene-expression/datasets). These data sets cover a wide variety of sequencing techniques that include Smart-Seq2 (8 data sets), 10X Genomics (6 data sets), Smart-Seq (5 data sets), inDrop (1 data set), RamDA-seq (1 data set), sci-RNA-seq3 (1 data set), SMARTer (5 data sets), and others (3 data sets). In addition, these data cover a range of sample sizes from a couple hundred cells to tens of thousands of cells measured in either human (19 data sets) or mouse (11 data sets). In each data set, we evaluated the effectiveness of different dimensionality reduction methods for one of the two important downstream analysis tasks: cell clustering and lineage inference. In particular, 15 data sets were used for cell clustering evaluation while another 15 data sets were used for lineage inference evaluation. For cell clustering, we followed the same criteria listed in [true cell type labels for all cells. The remaining 4 data sets contain cell labels that were determined in the original study and we simply treated them as truth though we do acknowledge that such “true” clustering information may not be accurate. For lineage inference, we followed the same criteria listed in [true lineage information for all cells. The remaining 11 data sets contain cells that were collected at multiple time points during the development process. For these data, we simply treated cells at these different time points as part of a single linear lineage, though we do acknowledge that different cells collected at the same time point may represent different developmental trajectories from an early time point if the cells at the early time are heterogeneous. In either case, the true lineages in all these 15 data sets are treated as linear, without any bifurcation or multifurcation patterns.We obtained a total of 30 scRNA-seq data sets from public domains for benchmarking dimensionality reduction methods. All data sets were retrieved from the Gene Expression Omnibus (GEO) database . We then applied log2 transformation on the normalized counts by adding one to avoid log transforming zero values. We simply term this normalization as log2 count transformation, though we do acknowledge that such transformation does take into account of cell size factor, etc. through the scater software. In addition to log2 count transformation, we also explored the utility of two additional data transformation: log2 CPM transformation and z-score transformation. In the log2 CPM transformation, we first computed counts per million reads (CPM) and then performed log2 transformation on the resulted CPM value by adding a constant of one to avoid log transformation of zero quantities. In the z-score transformation, for each gene in turn, we standardized CPM values to achieve a mean of zero and variance of one across cells using Seurat package (v2.3).A detailed list of the selected scRNA-seq datasets with corresponding data features is provided in Additional file v6.5 mouse 2i+LIF) in the Kumar data as input. We simulated scRNA-seq data with 500 cells and a known number of cell types, which were set to be either 4 or 8, using the Splatter package v1.2.0. All parameters used in the Splatter  were set to be approximately those estimated from the real data. In the case of 4 cell types, we set the group parameter in Splatter as 4. We set the percentage of cells in each group as 0.1, 0.15, 0.5, and 0.25, respectively. We set the proportion of the differentially expressed genes in each group as 0.02, 0.03, 0.05, and 0.1, respectively. In the case of 8 cell types, we set group/cell type parameter as 8. We set the percentage of cells in each group as 0.12, 0.08, 0.1, 0.05, 0.3, 0.1, 0.2, and 0.05, respectively. We set the proportion of the differentially expressed genes in each group as 0.03, 0.03, 0.03, 0.1, 0.05, 0.07, 0.08, and 0.1, respectively.Besides the above 30 real scRNA-seq data sets, we also simulated 2 additional scRNA-seq data sets for cell clustering evaluation. In the simulations, we used all 94 cells from one cell type , principal component analysis , independent component analysis , Diffusion Map , nonnegative matrix factorization , Kullback-Leibler divergence-based NMF , zero-inflated factor analysis , zero-inflated negative binomial-based wanted variation extraction , probabilistic count matrix factorization , deep count autoencoder network , a scalable deep-learning-based approach , generalized linear model principal component analysis , multidimensional scaling , locally linear embedding , local tangent space alignment , Isomap (Rdimtools R package v.0.4.2), t-distributed stochastic neighbor embedding , and uniform manifold approximation and projection . One of these methods, tSNE, can only extract a maximum of two or three low-dimensional components [FIt-SNE R package [Dimensionality reduction methods aim to transform an originally high-dimensional feature space into a low-dimensional representation with a much-reduced number of components. These components are in the form of a linear or non-linear combination of the original features   and in tmponents , 58, 59. package  in all fk-nearest neighbors for each single cell in the original space (denoted as a set A) and in the reduced space (denoted as a set B). We set k = 10, 20, or 30 in our study. We then computed the Jaccard index (JI) [We first evaluated the performance of dimensionality reduction methods by neighborhood preserving that aims to access whether the reduced dimensional space resembles the original gene expression matrix. To do so, we first identified the dex (JI)  to measuTherefore, more importantly, we also evaluated the performance of dimensionality reduction methods by evaluating how effective the low-dimensional components extracted from dimensionality reduction methods are for downstream single-cell analysis. We evaluated either of the two commonly applied downstream analysis, clustering analysis, and lineage reconstruction analysis, in the 32 data sets described above. In the analysis, we varied the number of low-dimensional components extracted from these dimensionality reduction methods. Specifically, for cell clustering data sets, in a data with less than or equal to 300 cells, we varied the number of low-dimensional components to be either 2, 6, 14, or 20. In a data with more than 300 cells, we varied the number of low-dimensional components to be either 0.5%, 1%, 2%, or 3% of the total number of cells. For lineage inference data sets, we varied the number of low-dimensional components to be either 2, 6, 14, or 20 for all data sets, since common lineage inference methods prefer a relatively small number of components.hclust; stats v3.5.3), k-means clustering , or Louvain method  to perform clustering on the reduced feature space. The k-means clustering is a key ingredient of commonly applied scRNA-seq clustering methods such as SC3 [k to be the known number of cell types in the data. We compared the cell clusters inferred using the low-dimensional components to the true cell cluster and evaluated clustering accuracy by two criteria: the adjusted rand index (ARI) [For clustering analysis, after dimensionality reduction with these dimensionality reduction methods, we used three different clustering methods, the hierarchical clustering   and the ex (ARI) . The ARIP = T denotes the inferred cell type cluster labels from clustering analysis while T = T denotes the known true cell type labels for n samples in the data; l and s enumerate the clusters, with l = 1, ⋯, r and s = 1, ⋯, k where r and k are the number of inferred cell type clusters and the number of true cell type clusters, respectively; nls = ∑ijI(pi = l)I(tj = s) is the number of times where the ith cell belongs to the cluster l in the inferred cluster labeling and jth cell belongs to the cluster s in the true cluster labeling; note that nls is an entry of contingency table which effectively measures the number of cells that are in common between P and T, with I(∙) being an indicator function; al = ∑snls is the sum of the sth column of the contingency table; and bs = ∑lnls is the sum of the lth row of the contingency table; compare function in the igraph R package (v1.0.0) to compute both ARI and NMI criteria. For rare cell type identification, we used the F-measure that is commonly used for quantifying rare cell type identification performance [F-measure is the harmonic mean of the clustering’s precision and recall, and is formulated as:where formance , 55. TheP represents the precision for identifying the rare cluster, with R represents the recall for identifying the rare cluster, with where dropClust to scale them to large data [While it is straightforward to apply different dimensionality reduction methods to most scRNA-seq data sets, we found that many dimensionality reduction methods are not computationally scalable and cannot be directly applied for clustering analysis in two large-scale scRNA-seq data sets we examined in the present study. For these non-scalable dimensionality reduction methods, we made use of a recently developed sub-sampling procedure described in rge data . In partrge data ). AfterwSlingshot [Slingshot software is the recommended lineage inference method based on a recent comparative study [k-means, hierarchical clustering algorithm, or Louvain method on the extracted low-dimensional components to obtain cluster labels. For the starting state, we supplied with the true beginning state of the lineage in the data. After obtaining the two types of input through the slingshot function, we used the getLineages function to fit a minimum spanning tree (MST) to identify lineage. The final output from Slingshot is an object of class SlingshotDataSet that contains the inferred lineage information. We follow the original Slingshot paper [m genes based on this rank order and denoted the corresponding rank in ascending order for these genes as {x1, ⋯, xm}, where xi ≤ xi + 1. Note that the true lineage is linear without any bifurcation or multifurcation patterns, while the inferred lineage may contain multiple ending points in addition to the single starting point. Therefore, for each inferred lineage, we examined one trajectory at a time, where each trajectory consists of the starting point and one of the ending points. In each trajectory, we ranked genes in order based on their position in the trajectory. We denote the corresponding rank order in the inferred trajectory for all m genes as {y1, ⋯, ym}, where we set yl as missing if lth gene is not included in the inferred trajectory. For each pair of non-missing genes, we labeled the gene pair  as a concordant pair if their relative rank in the inferred lineage are consistent with their relative rank in the true lineage; that is, either (xi ≥ xj & yi ≥ yj) or (xi < xj & yi < yj). Otherwise, we labeled the gene pair  as discordant. We denoted C as the number of concordant pairs, D as the number of discordant pairs, and U as the total number of non-missing genes. The Kendell correlation coefficient is then computed asFor trajectory inference, after dimensionality reduction with these dimensionality reduction methods, we used lingshot  (R packalingshot  (R packave study . Monocleot paper  to evaluτ over all these trajectories as the final Kendall correlation score to evaluate the similarity between the inferred lineage and the true lineage. For each data set, we repeated the above procedure five times and report the averaged results to avoid the influence of the stochasticity embedded in some dimensionality reduction methods and/or the lineage inference algorithm. For the large-scale data application to Cao et al., we also applied the sub-sampling approach dropClust to scale different dimensionality reduction methods for lineage inference.Afterwards, we obtained the maximum absolute Kumar data set for cell clustering, and the Hayashi data set for lineage inference. For each data, we randomly split the data into two subsets with an equal number of cells in each cell type in the two subsets. We repeated the split procedure 10 times to capture the potential stochasticity during the data split. In each split replicate, we applied different dimensionality reduction methods to analyze each subset separately. We used k-means clustering algorithm to infer the clustering labels in each subset. We used NMI to measure cell clustering accuracy and used Kendall correlation to measure lineage inference accuracy.We investigated the stability and robustness of different dimensionality reduction methods in both cell clustering and lineage inference applications through data splitting. Here, we focused on two representative scRNA-seq data sets, the Finally, to summarize the performance of the evaluated dimensionality reduction methods across the range of criteria in Fig. Additional file 1. Supplementary figures and tables.Additional file 2. Review history.


Saccharomyces cerevisiae, proliferating at different growth rates, and differentiating cardiopharyngeal cell lineages in the chordate Ciona robusta. We tested the method in a technical replicate dilution study, and in a k-fold validation study.A fundamental assumption, common to the vast majority of high-throughput transcriptome analyses, is that the expression of most genes is unchanged among samples and that total cellular RNA remains constant. As the number of analyzed experimental systems increases however, different independent studies demonstrate that this assumption is often violated. We present a calibration method using RNA spike-ins that allows for the measurement of absolute cellular abundance of RNA molecules. We apply the method to pooled RNA from cell populations of known sizes. For each transcript, we compute a nominal abundance that can be converted to absolute by dividing by a scale factor determined in separate experiments: the yield coefficient of the transcript relative to that of a reference spike-in measured with the same protocol. The method is derived by maximum likelihood theory in the context of a complete statistical model for sequencing counts contributed by cellular RNA and spike-ins. The counts are based on a sample from a fixed number of cells to which a fixed population of spike-in molecules has been added. We illustrate and evaluate the method with applications to two global expression data sets, one from the model eukaryote  k-fold cross-validation study. We illustrate our method with applications to two independent data sets from yeast and the sea squirt that were derived by different library preparation protocols. We show that our methods detect genome-wide amplification of expression, and we compare our method to others.We present a complete statistical model for the analysis of RNA-seq data from a population of cells using external RNA spike-ins and a maximum-likelihood method for genome-wide estimation of transcripts per cell. The model includes biological variability of cellular transcript number and sampling noise. We derive an unbiased estimator of transcripts per cell for every transcript, given by simply multiplying the count by a library-dependent, but transcript-independent, scale factor. This is a nominal estimate that can be converted to an absolute estimate by dividing by the transcript’s relative yield coefficient, measured in a separate experiment. A negative binomial probability mass function with novel normalization (size) factors allows for parametric testing of hypotheses concerning dependence of the absolute abundance of each transcript on experimental condition. Our method integrates information from every RNA-seq experiment across all replicates and experimental conditions to determine the calibration constants. We test the method with a dilution study and a  Accurate transcriptome measurements are central to understanding the fundamental mechanisms of gene expression. A main challenge presented by the RNA-seq method for digitizing information about cellular RNA content—both its composition and abundance—is correcting noise, errors, and biases introduced in the process of making the measurement. An important step in typical library preparation for sequencing is random fragmentation of the molecules to be sequenced. The actual unit that is being digitized during RNA-seq is therefore not the RNA molecules directly, but their fragments, whose number depends on transcript length and RNA abundance. However, it is the molecular abundance of a transcript that is of interest, rather than the distribution of fragments over its gene model.et al. νj calibras DESeq , can be νj calibration method clearly verifies these earlier observations and provides further insights of a transcriptome-wide up-regulation of total RNA abundance with increasing growth rate in the yeast GR study as described in detail below.Growth rate (GR) changes result in co-directional changes in total cellular RNA content in both prokaryotic and eukaryotic cells. Previous studies found that changes in the total abundance of rRNA and tRNA contribute to the GR dependent changes in RNA content in yeast cells , 38. Ourγ, was well described by an exponential functionϕ0 and ϕ1 are parameters to be fitted, for our particular set of growth rates: 0.12, 0.20, and 0.30 h-1. This function is not meant for extrapolation outside our experimental range of growth rates. For extrapolation outside this range of growth rates, a Hill-type function, which has a saturation point, might make more biological sense, but the curvature of the RNA abundance response to growth rate would require at least 3 parameters for such a function to be consistent with our data. Technical details for hypothesis testing in this GR study are given in We found that the dependence of inferred RNA abundance on growth rate, q = 0.01) In other words, a value of the exponential constant ϕ1 in We found 4,200 out of 7,468 detected transcripts to have statistically significant  dependence of RNA abundance on growth rate  cutoff, ϕ1 > 0 in ϕ1 < 0)  . We founIn Previously published results  have shosj  [sj [None of the classical global normalization methods that have implicit assumptions that only a small fraction of genes are differentially expressed and that the total cellular RNA is roughly constant across conditions (reviewed in ) provideike-ins) . Fig 6A ins) [sj  normaliz-1, respectively) using the DESeq function in the DESeq2 R package [sj size factors) yields 1118 differentially expressed transcripts at and FDR of 0.01, half (566) up-regulated and half (552) down regulated. The symmetry between up- and down-regulated genes is characteristic of this misapplied normalization assumption. These results are incorrect and miss entirely the global gene amplification as judged by results with νj normalization method. However, to an investigator not anticipating global amplification of expression, the results are consistent with the notions that total RNA does not vary with condition, a small proportion of genes are regulated by conditions, and there is no preponderance for up-regulated or down-regulated genes.Testing for differential gene expression between the lowest and highest growth rates  normalization [Following up RUVg normalization with testing for differential gene expression between the lowest and highest growth rates . Such a global effect is consistent with the literature on global transcriptome effects since FGF-MAPk acts downstream of c-myc [In our of c-myc . The metDN cells types. For the FgfrDN/lacZ comparison in p-value equals 0.0038). For the M-RasCA/FgfrDN comparison in p-value equal to 0.0017). Of the 1,934 significant (FDR = 0.01) fold changes, 1,560 were greater than 1, and 374, less than 1. The average fold change for up- and down-regulated transcripts was 1.9 and 2.8, respectively. We found essentially no differential gene expression between the LacZ and M-RasCA cell types. Only 17 and 50 transcripts were called differentially expressed at FDR values of 0.01 and 0.10, respectively.We found dramatic pairwise abundance differences between the LacZ and FgfrDN perturbation.These results corroborate our original observations, providing quantitative measures for the global down-regulation of gene expression in the Fgfrz-values are nominal, they are only one step away from absolute molecular abundance. Once the relative yield coefficient for transcript i, αi, is measured in separate experiments, the absolute molecular abundance in library j, ni,j will be known via the equation: ni,j = zi,j/αi.In this paper, we described a detailed statistical model for cellular RNA and exogenous spike-ins in a sample prepared from a fixed number of cells to which a population of spike-in molecules of known numbers has been added. In the context of this model, we derived by maximum likelihood arguments, a calibration method for RNA-seq data that estimates cellular molecular abundance of RNA. Although our molecular abundance condition, represent a technical replicate. We evaluated the spike-in model quantitatively in a number of ways .We have shown however that our method, especially when supplemented with RUVr  correctiCiona. We found global changes in gene expression in both systems: a global increase in transcript abundance with growth rate in yeast, and a global decrease in the FgfrDN embryonic cell type in Ciona. Reanalysis of the raw data with other algorithms that hold the assumption of equivalent transcriptome sizes, as expected, were not able to reveal these global transcriptome trends.We applied our method, to quantify RNA abundance and to test for differential gene expression, using data from two studies with different library preparation protocols, and in species from different kingdoms: a growth rate study in yeast, and a low cell count differentiation study in x times more molecules per cell, on average, in condition 1 compared to condition 2, even if the corresponding RNA-seq libraries were prepared in different batteries of experiments, different studies, or even prepared in different laboratories. Such a conclusion about what might be called, an absolute ratio of abundances, can be drawn without knowing the relative yield coefficient of transcript A. In the section that follows, we discuss the links between our work and methods by which these relative yield coefficients might be measured.Our focus in this paper is on deriving a nominal cellular molecular abundance that can be converted to absolute abundance by the transcript’s relative yield coefficient, which could be measured in separate experiments. In this study however, we do not attempt to measure the relative yield coefficient values, or estimate the absolute number of molecules per cell for each transcript within a condition. The current work allows us to say, that, for example, RNA transcript A has reference spike-in, arbitrarily chosen from among those that contribute a sizable fraction of total spike-in counts. Our nominal abundance of an RNA molecule is based on the temporary assumption that this molecule has the same yield coefficient as the reference spike-in. If our calibration method is supplemented with additional data on the effect that a broad range of transcript physicochemical characteristics has on library preparation and sequencing, a more realistic relative yield coefficient could be assigned to each RNA molecule of interest.In this manuscript we offer RNA abundance estimates that are proportional to absolute transcript abundance. For this we assign a (relative) yield coefficient value of 1 to a zi,j do not disentangle true absolute molecular abundance, ni,j, and the corresponding relative yield coefficient, αi; because, by definition, zi,j = αini,j. However, once one measures absolute cellular abundance of transcript i in a preparation of cells from which library j was derived , the relative yield coefficient becomes known, at, least in the idealized situation ignoring various sorts of noise, because αi = zi,j/ni,j. For example, ni,j might be measured by single-cell Fluorescence In Situ Hybridization (FISH) methods, performed on a large population of cells from which library j was derived.A technical statement of the outstanding problem is that our inferred nominal abundances αi, provided ni,j could be estimated. Likelihood methods could be used to integrate data across several libraries in the estimation of αi. In principle, once αi is estimated from one or more libraries and a population of cells from which those libraries were derived, this estimate could be used for other libraries (prepared using the same protocol), past, present, and future, to allow the determination of absolute cellular molecular abundances of transcript i.Statistical methods taking into account biological noise and technical noise could be used to compute a confidence interval for N, and variation in counts due to variation in relative yield coefficient across transcripts with nearly identical mean abundances, μN, could be disentangled.Modeling, like that presented in Our approach lays the groundwork for investigating, testing, and modeling how the physical properties—e.g., length, GC content, folding energy—determine the relative yield coefficient of spike-ins and native RNA transcripts alike. Empirical measurements of relative yield coefficients, as we have defined them, and biophysical modeling could facilitate progress in making the connection between sequencing counts and the underling molecular cellular abundances of the corresponding transcripts.νj calibration constants are closely related to the Anders and Huber-like “technical” size factors of [νj normalization constants in the cases of 2 of our data sets with large library sizes, as predicted by theory  approaches infinity, will give an abundance measure that is proportional to our z abundance values based on νj normalization.Our work has some important features in common with the HTN method of , particuW, which is used to model nuisance sources of variation that are unrelated to the experimental design. The factor matrix W is included, in addition to a design matrix, in a generalized linear model for normalized counts. One qualitative way of thinking about the W matrix is that is adds columns to the original design matrix for explanatory variables that one didn’t originally know about. Although this method is widely effective at reducing unwanted variation in RNA-seq data, it does not allow one to infer absolute cellular molecular RNA abundances, even if the factor matrix is computed based on spike-ins or an invariant gene set , was introduced by , 43, 44 gene set , as the The continuing discovery of examples in which there are gross transcriptome differences between cellular states, has established a need for spike-in controls in RNA-seq experiments . DespiteThe model presented in this work lends itself for both absolute and relative RNA quantitation, dependent on the experimental ability to accurately isolate a fixed number cells for library preparation. In both cases, we offer evidence that our approach provides reproducible results in a wide variety of conditions and has a strong predictive power. In conclusion, the presented model allows for improved unbiased RNA-seq quantitation in any experimental setup using external RNA spike-ins.S1 Appendix(PDF)Click here for additional data file.S2 Appendix(PDF)Click here for additional data file.S3 Appendix(PDF)Click here for additional data file.S4 Appendix(PDF)Click here for additional data file.S5 Appendix(PDF)Click here for additional data file.S6 Appendix(PDF)Click here for additional data file.S7 Appendix(PDF)Click here for additional data file.S8 Appendix(PDF)Click here for additional data file.S1 Figνj maximum-likelihood normalization method, in pilot studies with 6 technical replicate RNA libraries and 2 different volumes of the stock spike-in mixture added to the RNA, 3 with high (H) volume and 3 with low (L) volume. Actual RNA abundance does not vary across replicates. Variation in the medians of the distributions of relative log expression probably reflect technical volume/dilution errors in adding spike-ins to the cellular RNA and possible within-condition errors in total RNA due to cell count or RNA extraction. Before computing relative log expression, the table of counts was filtered to include only rows corresponding to those transcripts that were detected in more than 4 of 6 libraries . Next, the value 1 was added to each count . (C) RNA abundance  measured at 3 different growth rates per cell in yeast growth-rate/quiescence study before correcting for library preparation errors. Median values were computed across all libraries. Within each condition there is variation in location of the 0.5 quantile of RLE. The between condition variation reflects different overall RNA abundances at the growth rates per cell of 0.12, 0.20, and 0.30 h-1. (D) RLE plots for yeast GR data in panel C, but after adjustment of abundances by the δj scale factor for each library. Because expected abundances do not vary within a given condition, the δ-values were computed separately for each growth rate per cell. The vector of δ-values is  in order from left to right for the corresponding libraries (x-axis labels) in panel D. These δ-values are commensurate with those in the dilution studies in panels A and B. (E) RLE plots based on data from the Ciona embryonic differentiation study. RNA abundance measured for 3 different cell types, M-RasCA, FgfrDN and LacZ, before correcting for putative library preparation errors. Filtering of the table of counts was performed as in the yeast data in panels C and D. (F) RLE plots for Ciona data in panel E, but after adjustment of abundances by the δj correction factor for each library. The vector of δ-values is  in order from left to right for the corresponding libraries (x-axis labels) in panel D.(A) RNA abundance, as estimated by the (TIF)Click here for additional data file.S2 Fig2 ratio of observed to predicted spike-in counts for each detected molecule i in one or more replicates j vs. log2 of the predicted counts. Data from the Ciona embryonic differentiation study. The mean log2 ratio of observed to predicted spike-in counts is equal to 0.046, which is consistent with unbiased prediction. Vertical bars with lower and upper endpoints  demarcate a mid 0.99 quantile range of random counts generated from the multinomial model with the maximum likelihood proportions, such that Pr{Y < L} < 0.005, and Pr{Y > U} < 0.005. Because the marginal probability mass functions are binomial, the  interval for each transcript is the same as the mid 0.99 binomial quantile for that transcript. (B) Comparison of observed and predicted counts from multinomial spike-in model, as in panel A, but based on data from the yeast dilution study. (C) Comparison of observed and predicted counts from multinomial spike-in model, as in panel A, but based on data from the yeast growth rate study. (D) CV2(mean) versus mean for normalized spike-in counts, on log-log axes, for the same Ciona spike-in data (open symbols) in panel A. For each spike-in i the mean normalized count plotted on the x-axis is the mean over all libraries j of 2 values according to the multinomial model, and it is drawn from 2 that follows from a negative binomial model for random spike-in counts Yi,j, in which the mean is given by a = 1000. It is drawn from 2 values generated in 10,000 Monte Carlo simulations in which the synthetic spike-in counts were drawn form the negative binomial (a = 1000). (E) CV2(mean) versus mean for normalized spike-in counts, as in panel B, but based on spike-in counts in the yeast dilution study. Solid black line from the multinomial model (a = 1000 (2(mean) versus mean for normalized spike-in counts, as in panel B, but based on spike-in counts in the yeast growth rate study. Solid black line from the multinomial model (a = 400 ((A) Comparison of observed and predicted counts, in MA-like plot format, according to the multinomial statistical model linking spike-in counts to abundance in the corresponding sample. Each plotted point represents log(TIF)Click here for additional data file.S3 Figαi) based on spike-in counts from 9 libraries in the Ciona embryonic differentiation study. The median α-value is 0.95, and the interquartile range (IQ) of 1.3 extends from 0.51 to 1.8. (B) Observed  in β1 = 0.013; β2 = 0.0032; and β3 = −0.0014. The root mean square error, normalized by max(α) − min(α), or normalized by sd(α) are equal to 0.16 or 0.98, respectively. (C) Histogram of relative yield coefficients (αi) based on spike-in counts from 6 libraries in the yeast dilution study (different library preparation protocol than that used in the Ciona embryonic differentiation study). The median of α-value is 0.65, and the IQ range of 0.83 extends from 0.28 to 1.1. (D) Observed  and fitted values of α, for yeast dilution study, based on β-coefficients β1 = 0.0012; β2 = 0.0045; and β3 = 0.140. All 3 variables were found to be highly significant (see text). The root mean square error, normalized by max(α) − min(α), or normalized by sd(α) are equal to 0.14 or 0.93, respectively.(A) Histogram of relative yield coefficients ((TIF)Click here for additional data file.S4 Figa for each condition in the yeast GR study, is provided by looking at the overall distributions of CV(mean), and the distributions of CV(mean) for transcripts whose means fall into each of several subintervals of the range of mean values. (A) Histograms of sample estimates of CV(mean), plotted on a density scale, corresponding to the yeast quiescence study with C-limited growth at rate per cell of 0.12 h-1. Solid grey line correspond to experimental data , and the dashed black lines are for corresponding synthetic (Monte Carlo) data generated according to the negative binomial model in a = 23, determined by a maximal marginal likelihood method  are 0.19±0.22 and 0.19±0.21, respectively. Shape parameter for synthetic data, a = 34. (I–L) Similar to (A–E) but data are from libraries corresponding to growth rate per cell of 0.30 h-1. Mean ± sd of the full distributions for lab, and synthetic data in panel (E) are 0.21±0.23 and 0.20±0.21, respectively. Shape parameter for synthetic data, a = 24.A rigorous challenge of our statistical model, with a single shape parameter d method . (B-D) C(TIF)Click here for additional data file.S5 Figνj-normalized counts  from yeast GR study  PCA biplot corresponding to panel C.(A) Diagnostic relative log expression (RLE) plots of adjusted counts obtained by applying an RUVr normalization method based on residuals to  package ). Before ± log δ . We did (TIF)Click here for additional data file.S1 Code and DataCiona embryonic differentiation study, and yeast dilution study, respectively. The corresponding count data for spike-in molecules are named similarly, but with SI instead of RNA. Data files ERCCyeast.txt and ERCCciona.txt are data frames for detected ERCC spike-in molecules in the yeast growth rate study and Ciona study, giving attomoles of each molecule added to native RNA, length (nt), GC content, and folding energy . The ERCCdilutionNmatAttomoles.txt file is a data frame for the yeast dilution study that gives attomoles of detected spike-in molecules for the high-volume (H) and low-volume (L) spike-in aliquot conditions.The YmatRNAyeast.txt, YmatRNAciona.txt, and YmatRNAdilution.txt files are data frames for native RNA count data (genes-by-replicates) for the yeast growth rate study, (ZIP)Click here for additional data file.