Dysregulation of splicing is known to be associated with disease and as a hallmark of cancer. Existing tools for differential transcript usage (DTU) analysis either lack in performance, cannot account for complex experimental designs or do not scale to massive single-cell transcriptome sequencing (scRNA-seq) datasets. We introduce Here, we specifically address differential transcript usage (DTU) analysis, one type of transcript-level analysis that studies the change in relative usage of transcripts/isoforms within the same gene. DTU analysis holds great potential: previous research has shown that most multi-exon human genes are subject to alternative splicing and can thus produce a variety of functionally different isoforms from the same genomic locus.– The dysregulation of this splicing process has been reported extensively as a cause for disease,– including several neurological diseases such as frontotemporal dementia, Parkinsonism and spinal muscular atrophy, and is a well-known hallmark of cancer.Studying differential expression (DE) is one of the key tasks in the downstream analysis of RNA-seq data. Typically, DE analyses identify expression changes on the gene level. However, the widespread adoption of expression quantification through pseudo-alignment, and Smart-Seq3 hold the promise to further increase the resolution of DTU analysis from bulk RNA-seq data towards the single-cell level, where differences in transcript usage are expected to occur naturally between cell types. However, only a few bespoke DTU methods have been developed for scRNA-seq data and they lack biological interpretation. Indeed, methods specifically developed for scRNA-seq data are either restricted to exon/event level, analysis (e.g. pinpointing exons involved in splicing events), or they can only pinpoint DTU genes without unveiling the actual transcripts that are involved. Interestingly, many DTU methods for bulk RNA-seq do provide inference at the transcript level and their performance has already been extensively profiled in benchmark studies.– Based on a subset of the simulated RNA-seq dataset from Loveet al. (see Methods), we show the performance of six DTU tools; DEXSeq, DoubleExpSeq, DRIMSeq, edgeR diffSplice, limma diffSplice and NBSplice  technologies such as Smart-Seq2plice . DEXSeq – in a single experiment. Inet al. Despite of its good performance, the popular tool DEXSeq already required more than five hours to analyze two groups of 32 cells and clearly does not scale to large bulk nor scRNA-seq datasets.In order to assess DTU in single-cell applications, however, these bulk RNA-seq DTU tools should scale to the large data volumes generated by full-length scRNA-seq platforms, which can profile the transcriptome of several thousands of cellset al. Here, we observed that the DTU tool BANDITS scales particularly poorly to large numbers of transcripts. More specifically, BANDITS did not complete the DTU analysis on the dataset with 7.500 transcripts within 137 hours on our system (see Methods); therefore, larger analyses were omitted. As such, BANDITS had to be omitted from the analyses shown inExtended data figures S1 and S3.In addition, DTU methods should allow for the analysis of datasets with large numbers of (unique) transcripts. The number of transcripts that are typically assessed depends on the coverage of the RNA-seq experiment and the adopted filtering criteria in the data analysis workflow. As the coverage of RNA-seq experiments has increased rapidly over the past few years and can be expected to continue expanding, scalability towards large numbers of transcripts will be essential to enable a transcriptome-wide view on the isoform usage changes. InBesides scalability, several other issues arise when porting bulk RNA-seq DTU tools towards scRNA-seq applications. Indeed, modelling scRNA-seq data often requires multifactorial designs, for instance when comparing expression levels across multiple cell types between multiple treatment groups. Accounting for multiple covariates, however, is not implemented in BANDITS, NBSplice and DoubleExpSeq, jeopardizing their utility for (sc-)RNA-seq DTU analysis. Another issue arises with the large numbers of zero counts in scRNA-seq data, which seems to be particularly problematic for NBSplice that fails to converge if the gene-level count of any of the samples or cells is zero. As such, NBSplice could not be evaluated insatuRn, which is an acronym for Scalable Analysis of differential Transcript Usage for RNa-seq data, a novel method for DTU analysis that (i) is highly performant, (ii) provides a good control of the false discovery rate (FDR) (iii) scales seamlessly to the large data volumes of contemporary (sc-)RNA-seq datasets, (iv) allows for modelling complex experimental designs, (v) can deal with realistic proportions of zero counts and (vi) provides direct inference on the biologically relevant transcript level. In brief, satuRn adopts a quasi-binomial (QB) generalized linear model (GLM) framework. satuRn provides direct inference on DTU by modelling the relative usage of a transcript, in comparison to other transcripts from the same gene, between groups of interest. To stabilize the estimation of the overdispersion parameter of the QB model, we borrow strength across transcripts by building upon the empirical Bayes methodology as introduced by Smythet al. In order to control the number of false positive findings, an empirical null distribution is used to obtain the p-values, which are corrected for multiple testing with the FDR method of Benjamini and Hochberg. Our method is implemented in an R package available athttps://github.com/statOmics/satuRn and is distributed through the Bioconductor project.Altogether, many of the existing DTU analysis tools are not well suited to analyze large bulk RNA-seq and full-length scRNA-seq datasets, leaving the great potential of differential splicing analysis for these data largely unexploited. In light of these shortcomings we developed or Salmon, or by classical alignment-based tools followed by transcript-level quantification . LetgtiY denote the observed expression value for a given transcriptgt = 1, … , T of geneg = 1, ..., G in cell or samplei = 1, … , n. The total expression of geneg in samplei can then be expressed asAs input, satuRn requires a matrix of transcript-level expression counts , which may be obtained either through pseudo-alignment using kallistogT transcripts belonging to geneg in samplei. The usage of transcriptt in sample or celli can then be estimated asi.e., by taking the sum of expression values for allNext, we adopt a quasi-binomial (QB) generalized linear modelling (GLM) strategy to model transcript usage. As opposed to canonical maximum likelihood models, this quasi-likelihood modelling strategy only requires the specification of the first two moments of the response distribution, i.e., the mean and the variance. We define the mean of the QB model asgti is the expected probability of observing transcriptt within the pool of transcripts  belonging to geneg in samplei and, as such, corresponds to its expected usage for that sample. We model πgti using a logit linkfunction, whereβt is a p x 1 column vector of regression parameters modelling the association between the average usage and the covariates for transcriptt. Finally,X that corresponds with the covariate pattern of samplei, withp the number of parameters of the mean model, i.e., the length of vectorβt.In this notation, πThe variance of the QB model can be described asgt a transcript-specific overdispersion parameter to describe additional variance in the data with respect to the binomial variance. We adopt the empirical Bayes procedure from Smythet al., as implemented in thesqueezeVar function of thelimma Bioconductor R package, to stabilize the estimates of ϕgt by borrowing information across transcripts, which is adopted in the default edgeR quasi-likelihood workflow for bulk RNA-seq data., Note that stabilizing the dispersion estimation is particularly useful in datasets with a small sample size.witht from the pool of transcripts from the corresponding geneg across samples. The intercept also has an interpretation of a log-odds and the remaining mean model parameters are log-odds ratios, which may thus be interpreted in terms of differential transcript usage. We adopt t-tests that are computed based on the log-odds ratio estimates of the QB model and the posterior variance, as obtained from the empirical Bayes procedure. P-values are computed assuming a t-distribution under the null hypothesis with posterior degrees of freedom calculated as the sum of the residual degrees of freedom and the prior degrees of freedom from the empirical Bayes procedure.Taken together, the quasi-binomial thus allows us to model the log-odds of drawing a particular transcriptExtended data figure S14), which could suggest that the theoretical null, the t-distribution, is no longer valid. Indeed, in large-scale inference settings, failure of the theoretical null distribution is often observed. Efron (Chapter 6) describes four reasons why the theoretical null distribution may fail; failed mathematical assumptions, correlation across features (transcript expression), correlation across subjects (samples or cells), and unobserved confounders in observational studies. To avoid these issues, Efron proposes to exploit the massive parallel data structure of omics datasets to empirically estimate the null distribution of the test statistics. To this end, Efron converts the test statistic to z-scores, which should follow a standard normal distribution under the theoretical null, and then proposes to approximate the empirical null distribution with a normal distribution with unknown mean  the sign of the t-test statisticgtS andgtZ the resulting z-score. Next, we adopt the maximum likelihood procedure, implemented in thelocfdr function of the locfdr R package from CRAN, to estimate the meanwith Φ the cumulative distribution function for the standard normal distribution, As opposed to the original p-values that were calculated based on the theoretical null distribution for the t-statistics, we found that this procedure allows for a better FDR control in single-cell applications.Finally, the resulting  p-values are corrected for multiple testing with the FDR method of Benjamini and Hochberg.Extended data and the respective original publications. Note that all methods were run in R3.6.1 using default settings.We adopted two different strategies for filtering transcripts in each of the RNA-seq datasets in the performance benchmarks.filterByExpr function implemented in edgeR. This filtering strategy only retains transcripts that have at least an expression level ofmin.count counts-per-million  in at leastn samples or cells. In addition, the sum of the CPM of the transcript across all cells or samples must be at leastmin.total.count. For the bulk RNA-seq datasets, we use the default settings  andmin.total.count = 10), with ns the number of samples or cells in the smallest group. For the scRNA-seq datasets, the settings are adjusted to;min.count = 1 , n = 0.5*ns andmin.total.count = 0. In addition, if only one transcript of a gene passes this filtering criterion, it is omitted from the analysis, as DTU analysis is meaningless when only one transcript is retained. As such, we specifically set the parameters to generate a very lenient filtering criterium.The first filtering strategy uses thedmFilter function implemented in DRIMSeq. This filter is more stringent and specifically designed for DTU analysis. The filtering process can be thought of as proceeding in three steps. The first step requires the transcripts to have a count of at least 10 in at leastsn samples. The second filtering step requires the transcript to make up at least 10% of the total count of its corresponding gene in at leastsn samples or cells. The third filtering step removes all transcripts for which the corresponding gene has a count below 10 in any of the samples or cells in the dataset. Again, if only one transcript of a gene passes this filtering criterion, it is omitted from the analysis.The second filtering strategy uses thescaledTPM abundances. Counts are the transcript-level abundances estimated by kallisto or Salmon, i.e., the number of reads mapping to each transcript.scaledTPM normalized abundances are obtained by first computing transcript-per-million (TPM) abundances by (1) dividing the transcript counts by the length of the corresponding transcript in kilobases, (2) dividing by a cell/sample specific scaling factor, which is computed by taking the sum across transcripts of all length-corrected abundances from (1) and (3) multiplying by one million. Finally, the TPM abundances are scaled per cell/sample, so that the sum of the TPM abundances in the cell/sample equals the original number of counts for that cell/sample.We additionally compared the results between two quantification strategies: counts andet al. (dataset 1) and both theDrosophila melanogaster (dataset 2) andHomo sapiens (dataset 3) simulation studies from Van den Bergeet al.. All three datasets were generated based on parameter values obtained from real RNA-seq samples, to mimic real RNA-seq data as close as possible.To evaluate the performance of the different DTU analysis methods, we first adopt three simulated bulk RNA-seq datasets from previous publications: the simulated dataset from Loveet al., the origin of DTU is twofold: On the one hand, DTU was specifically introduced by swapping the transcript-per-million (TPM) abundances between two expressed isoforms. On the other hand, DTU was also obtained as a consequence of introducing DTE, where a single expressed isoform was induced to be differentially expressed at a certain log fold change, which leads to DTU if this transcript belongs to a gene expressing multiple isoforms. For datasets 2 and 3 from Van den Bergeet al., there is only one source of DTU. The number of differentially used transcripts within a gene was sampled ranging from a minimum of two up to a random number drawn from a binomial distribution with size equal to the number of transcripts and success probability 1/3, thus allowing for differential usage of more than two transcripts within the same gene. For the selected transcripts, the percentage of usage withing the corresponding gene is first computed on the TPM scale. Next, DTU was introduced by swapping these percentages between transcripts. Subsequently, the swapped percentages are multiplied by the original gene-level read count per kilobase (RPK) to obtain transcript-level RPK. Finally, TPM and counts are computed based on these transcript-level RPK. Additionally, dataset 1 uses Salmon (version 1.1.0) for estimating transcript-level abundances, whereas datasets 2 and 3 were quantified with kallisto (version 0.46.0).Notably, there is a subtle difference in how DTU is introduced between the two simulation frameworks. For dataset 1 from Love release version 8. Nine datasets were generated non-parametrically. More specifically, we first selected samples from adrenal gland tissue that were extracted with the RNA extraction method “RNA Extraction from Paxgene-derived Lysate Plate Based”. From the remaining samples we subsampled nine datasets, comprising three repeats for each of three sample sizes;5 versus 5,20 versus 20 and50 versus 50 samples. Next, DTU is artificially in 15% of the genes. The number of differentially used transcripts within a gene was determined with the strategy of Van den Bergeet al. For the selected transcripts, the counts of transcripts within the same genes are swapped in one group of samples or cells, inducing DTU signal between the groups. The GTEx data was quantified with RSEM version 1.3.0.We evaluate the performance of the different DTU methods on real bulk RNA-seq data, by subsampling a homogeneous set of samples from the large bulk RNA-seq dataset available from the Genotype-Tissue Expression (GTEx) consortium,, after which DTU is artificially introduced as discussed above . The datasets from Chenet al. and Tasicet al. were quantified using Salmon v1.1.0, whereas the dataset from Darmaniset al. was quantified with Salmon v0.8.2 as obtained from Sonesonet al.We evaluate the performance of the different DTU methods on real scRNA-seq datasets. These scRNA-seq datasets were generated non-parametrically by subsampling a homogeneous set of cells from three real scRNA-seq datasets,et al., which was used to constructExtended data Figure S7, we selected a homogeneous population of cells by considering only the EpiStem cells of female mice, resulting in a dataset of 120 cells. From this homogeneous population of cells, we then subsampled six datasets, comprising three repeats for each of two sample sizes:20 versus 20 and50 versus 50 cells. Next, DTU was artificially introduced by swapping transcript counts between groups of cells. Finally, we adopted either edgeR or DRIMSeq for filtering.For the dataset of Chenet al., which was used to construct Figure S8 (Extended data), we selected a homogeneous population of cells by considering only the Lamp5 cells in the anterior lateral motor cortex of mice without any eye conditions, resulting in a dataset of 897 cells. After introducing DTU by swapping transcript counts, we randomly subsampled 20, 75 or 200 cells from each group. For the dataset of Darmaniset al., which was used to constructExtended data Figure S9, we selected the immune cells that clustered together in tSNE cluster 8 of the original publication, resulting in a dataset of 248 cells. After introducing DTU, we randomly subsampled 20, 50 or 100 cells from each group.The other two scRNA-seq datasets were generated analogously. For the dataset of Tasic i.e. between different the cell types originating from the ALM and VISp regions of the glutamatergic L5 IT subclass. We use the quasi-likelihood method of edgeR to model the gene expression profiles and additionally adopt the edgeRglmTreat function to test differential expression against a log2-fold change threshold (log2-fold change = 1). Statistical significance was evaluated at the 5% FDR level.We perform a differential gene expression (DGE) analysis on a subset of the Tasic single-cell dataset,We assess the performance of different DTU methods on a bulk simulation dataset with scatterplots of the true positive rate (TPR) versus the false discovery rate (FDR), according to the following definitions:where FN, FP and TP denote the numbers of false negatives, false positives and true positives, respectively. The FDR is the expected value of the false discovery proportion (FDP), which is the ratio of the number of false positives to the total number of positives. The FDR-TPR curves are constructed using the Bioconductor R package ICOBRA version 1.14.0. which contains 617 cells in total. For the scalability benchmark with respect to the number of cells in the dataset, we randomly subsample a certain number of cells  from the dataset (without introducing DTU or selecting specific homogeneous cell populations). Next, we filter this subsample using the edgeR-based filtering criterion. This was done to remove very lowly abundant transcripts, which may otherwise cause problems in the parameter estimation procedure. From the remaining transcripts, we randomly subsampled to a total of 30,000 transcripts before running the DTU analysis. To allow for a scalability benchmark of BANDITS, which scales poorly to the number of transcripts (Extended data figure S1). For the scalability benchmark with respect to the number of transcripts, we randomly sampled two groups of 16 cells from the dataset. After applying the edgeR-based filter, we sampled eight distinct numbers of transcripts: 1,000, 2,000, 5,000, 10,000, 15,000, 20,000, 25,000, 30,000 and 35,000 prior to the DTU analysis.The scalability benchmark was run on subsets of the Chen scRNA-seq dataset,scripts , we also bulk RNA-seq dataset. Using the same strategy as described above for the scRNA-seq scalability benchmark, we here assess scalability with respect to the number of samples  while keeping the number of transcripts fixed at 30,000 and scalability with respect to the number of transcripts (same range as above) while keeping the number of samples fixed at sixteen. Results are displayed in theExtended data figures S20 and S21.We additionally perform a scalability benchmark on the GTExAll scalability benchmarks were run on a single core of a virtual machine with an Intel(R) Xeon(R) CPU E5-2420 v2  processor and 30GB RAM.We first evaluate the performance of our novel DTU method, satuRn, on publicly available simulated and real bulk RNA-seq data, as well as on real scRNA-seq data. In general, we found that the performance of satuRn was at least on par with the performances of the best tools from the literature. In addition, our method controls the FDR closer to the nominal level, on average. Second, we show that satuRn scales towards the large data volumes generated by contemporary bulk and single-cell RNA-seq experiments, allowing for a transcriptome-wide analysis of datasets consisting of several thousands of cells, in only a few minutes. Finally, we analyze a large full-length scRNA-seq case study dataset, where we obtain highly relevant biological results on isoform-level changes between cell types that would have remained obscured in a canonical differential gene expression (DGE) analysis.et al. and contains two groups of 12 samples each, which we subsample without replacement to evaluate 3vs3, 6vs6 and 10vs10 two-group comparisons. Datasets 2 and 3 are theDrosophila melanogaster andHomo sapiens simulation studies from Van den Bergeet al. and Sonesonet al., which both contain two groups of five samples each. In brief, all datasets were constructed by generating sequencing reads based on parameters that are estimated from real bulk RNA-seq data. DTU between groups of samples was artificially introduced in the data, prior to the quantification of expression using either Salmon (dataset 1) or kallisto (datasets 2 and 3). Notably, there are some methodological differences between the simulation framework of dataset 1 and that of datasets 2 and 3 with respect to the read generation and the simulation of DTU signal (see Methods). In terms of transcript filtering, we adopt two different strategies as implemented by edgeR and DRIMSeq, which correspond to a lenient and more stringent filtering, respectively (see Methods).To evaluate the performance of satuRn, we adopt three simulated bulk RNA-seq datasets from previous publications. Dataset 1 was obtained from Love6 versus 6 subsamples for dataset 1, after filtering with edgeR.The result of the performance evaluation of satuRn with respect to other DTU methods on the three simulated bulk datasets is displayed inExtended data figures S2, S13). Neither sample size nor filtering criterion had a profound impact on the ranking of the performances of the different DTU methods; satuRn, DEXSeq and DoubleExpSeq remain the best performing methods overall. In addition, we studied the impact of using either transcript count estimates or normalized abundance estimates  as input data for the DTU algorithms. We observed a slightly higher performance in all datasets when providing count estimates, except for Dataset 1 from Loveet al. Loveet al. introduced DTU either by swapping TPM abundances or by introducing a fold change in transcript expression in one group of samples, again on TPM abundances. However, when back-transforming the swapped TPM abundances to count abundances, the transcript lengths were not swapped. Hence, the simulation strategy can induce differences in the total count for a gene, which in turn may alter the usage of all (target and off-target) transcripts in that gene when counts are used as an input, which gives rise to many false positives in off-target transcripts of DTU genes. When introducing DTU by swapping counts , we could expect spurious signal in analyses on the TPM scale for the same reason. However, the difference between the transcript count and TPM analyses was much less pronounced, which was also the case for the Van den Bergeet al. simulation approach. Therefore, all performance evaluations in the body of this publication were generated with count estimates as input data, except forExtended data.We also evaluated the effects of sample size and different filtering criteria on the performance of the different DTU methods  consortium, generating nine datasets in total, i.e. three repeats for each of three sample sizes;5 versus 5,20 versus 20 and50 versus 50 samples. We then artificially introduced DTU in these data by swapping transcript counts between groups of samples (see Methods for details). Again, we adopt two different filtering strategies as implemented by edgeR and DRIMSeq (see Methods).While simulation studies are common for evaluating the performance of DE analysis methods, there is currently no consensus on the simulation strategy that best mimics real (sc)RNA-seq data. In addition, simulation frameworks typically generate data according to parametric assumptions on the data-generating mechanism, thus potentially favoring DE methods that adopt similar distributional assumptions in their statistical model.50 versus 50 samples), as these methods do not scale to such large datasets .The results of the performance evaluation of satuRn on the real bulk datasets upon edgeR filtering is displayed inatasets . Adoptin,, (see Methods). Again, we subsampled three repeats of different sample sizes, artificially introduced DTU with the swapping strategy and applied either the edgeR- or DRIMSeq-based filtering criterium (see Methods for details).Finally, we evaluate the performance of satuRn on single-cell RNA-seq data. As with the real bulk analysis, the single-cell datasets were generated by subsetting from three different real scRNA-seq datasetset al. dataset, we generated three repeats of two sample sizes, i.e.20 versus 20 and50 versus 50 cells. The results of the performance evaluation of satuRn on this dataset upon edgeR filtering is displayed inExtended data figure S7). The results of the performance evaluations on the other two scRNA-seq datasets, are in strong agreement with the results displayed here, with satuRn performing at least on par with DoubleExpSeq and satuRn additionally controlling the FDR around the nominal level (Extended data figures S8 and S9).By subsampling the ChenExtended data figure S14). To obtain proper p-values with satuRn in single-cell applications, we therefore estimate the null distribution of the test statistic empirically . Note that the performances in the bulk RNA-seq benchmarks are not affected by adopting the empirical null distribution, because no deviations of the theoretical null distribution occur in bulk data. However, the empirical null resulted in much improved FDR control in scRNA-seq datasets (Extended data figure S14). We therefore adopt the empirical null estimation as the default setting in satuRn. As such, all satuRn results in this publication are relying on the empirical null strategy. As a final remark, we likewise attempted to improve the FDR control of DoubleExpSeq. However, in all analyses with DoubleExpSeq we observed a large spike of p-values equal to 1, which poses a problem when estimating the empirical null distribution (Extended data figure S15). Therefore, this strategy could not be used to improve the FDR control of DoubleExpSeq.Notably, we found that the theoretical null distribution of the test statistics from satuRn failed to provide good FDR control in single-cell analyses , thus introducing DTU signal of different magnitude in the different transcripts. InWe performed a computational benchmark of satuRn to investigate its scalability with respect to the number of samples/cells and the number of transcripts in an RNA-seq dataset. All scalability benchmarks were run on a single core of a Linux machine with an Intel(R) Xeon(R) CPU E5-2420 v2  processor and 30GB RAM. The results are displayed inet al.), while keeping the number of transcripts in the dataset fixed at 30,000. From the left panel, it is clear that DRIMSeq and especially DEXSeq scale very poorly with the number of cells in the dataset, which was already shown in was not included in this benchmark, as it does not scale to datasets with this many transcripts. For a performance and scalability evaluation of BANDITS on datasets with a lower number of transcripts, we refer toExtended data figure S1. NBSplice was also omitted as it fails to converge on datasets with a large proportion of zero counts.Extended data Figure S20, we additionally evaluate the scalability of the DTU tools on different subsets of the bulk GTEx41 dataset. These results are in line with those of the single-cell scalability benchmark, with DEXSeq and DRIMSeq being markedly slower than the other methods. When directly comparing the scalability between the bulk and single-cell data on equally sized subsets , we found limited differences in scalability between the data types, except for DEXSeq. DEXSeq scales considerably better on bulk data than on single-cell data, suggesting that the estimation of the GLM parameters is slower when the data are sparse. However, as the scalability profile of DEXSeq is quadratic with respect to the number of cells/samples in the data, it is still infeasible to adopt DEXSeq in datasets with many cells/samples.In In addition, it fails to control the FDR at the desired level, especially with increasing sample sizes. With satuRn, we overcome these issues and effectively unlock DTU analysis for single cell applications.Altogether, we find that while several methods for DTU analysis exist, none are optimally suited for analyzing single-cell datasets. DRIMSeq, NBSplice, edgeR diffsplice and limma diffsplice have a much lower overall performance in our benchmarks. DEXSeq does not scale to large datasets. Finally, DoubleExpSeq does not support experimental designs that require an analysis with multiple additive effects, e.g. randomized complete block designs and designs where batch-effect correction is required, which are essential for many practical scRNA-Seq analysis settings.) RNA-seq dataset from Tasicet al. In addition, we analyze the same dataset with DoubleExpSeq and limma diffsplice, which are the only other DTU methods that scale to large scRNA-seq datasets and have a reasonable performance in our benchmarks. In the original publication, the authors studied differential gene expression between cell types originating from two areas at distant poles of the mouse neocortex; the primary visual cortical area (VISp), which processes sensory information with millisecond timescale dynamics– and the anterior lateral motor cortex (ALM), which displays slower dynamics related to short-term memory, deliberation, decision-making and planning., Based on marker genes, Tasicet al. assigned all of the 23,822 cells from the scRNA-seq dataset to one of three cell classes; glutamatergic (excitatory) neurons, GABAergic (inhibitory) neurons or non-neuronal cells. The authors then further classified the neuronal cells into several subclasses based on their dominant layer of dissection and projection patterns (through a retrograde labelling experiment). Finally, these subclasses are further classified into cell types based on the expression of specific marker genes.We use satuRn to perform a DTU analysis on a subset of the single-cell , where L5 refers to layer-of-dissection 5 and IT refers to the intratelencephalic projection type. Here, we first perform a DGE analysis with an edgeR-based workflow (see Methods) on the same comparisons between L5 IT cell types that were assessed by Tasicet al.In their original DGE analysis, TasicNext, we perform a DTU analysis for the same cell types using satuRn. In column 5 of) on the three comparisons with most DE genes and most genes with evidence for DTU . Similar gene ontology categories are returned for the set of DGE genes and the set of DTU genes, with many of the enriched processes being biologically relevant in the context of this case study. Enriched gene sets include the gene ontology classes, synapse, neuron projection, synaptic signaling and cell projection organization. This shows that the complementary information brought by the DTU analysis is indeed biologically relevant. For an extensive overview of the GSEA of the set of DGE genes and genes with evidence of DTU in comparisons 5, 6 and 7, we refer to theExtended data.We perform a gene set enrichment analysis (GSEAENSMUSG00000029470), alsb gene which is part of a family of purinergic receptors that have been implicated in functions such as learning, memory and sleep. In the DGE analysis, no evidence for differential expression of P2rx4 was found at the gene level . By contrast, in our DTU analysis the transcripts of P2xr4 displayed the highest statistical evidence for differential usage within the set of transcripts that could be assigned to the ontology class ‘neuron projection’. The mean usage of transcriptENSMUST00000081554 is estimated to be 28.9% in Tnc cells and 75.9% in Hsd11b1 Endou cells . For transcriptENSMUST00000195963, the transcript usage changes from 58.2% in Tnc cells and 16.6% in Hsd11b1 Endou cells . For the third transcript of P2rx4 that was assessed in our DTU analysis,ENSMUST00000132062, we found no statistical evidence for DTU . InENSMUST00000195963, is not protein coding. By contrast, the majority transcript in the Hsd11b1 Endou cell type,ENSMUST00000081554, is coding for the P2X purinoceptor protein (UniProt IDQ9Z256). As such, the changes in transcript usage between both cell types represent actual biological differences in the functionality of the gene products, which may be relevant to the process of neuron projection. This functional difference would have remained obscured when only performing a canonical DGE analysis.To display the utility of satuRn for identifying and visualizing DTU transcripts in scRNA-seq datasets, we focus on comparison #6 of the DTU analysis and discuss the gene P2X Purinoceptor 4 or P2rx4  and we observe that the p-values were shifted towards smaller values (Extended data figures S23 and S24). Therefore, we adopted the same empirical null strategy as implemented in satuRn to post-process the results. While this dramatically decreased the number of significant DTU transcripts, limma diffsplice still identified more transcripts  than our method. However, when we inspected the transcripts that were highly ranked in the top DTU list of limma diffsplice but lowly ranked in our top list, we found that most of these transcripts either originate from genes that are lowly expressed, or they are transcripts with a large fraction of zero counts . Limma diffsplice thus claims differential usage more often for transcripts that only contain little information for assessing DTU. This is depicted inWe also analyzed the case study dataset with limma diffsplice.t with the average log-fold change in the expression of all transcripts belonging to the same gene as transcriptt. As such, limma diffsplice does not incorporate any information on the absolute gene expression levels. In contrast, our quasi-binomial GLM framework models the log-odds of drawing a particular transcriptt from the pool of transcripts in the corresponding gene. As a consequence, transcripts belonging to lowly expressed genes are correctly considered less informative in satuRn and are thus less likely to be picked up. For example, inThis behavior can be expected. Limma diffsplice tests for DTU by comparing the log-fold change in expression of transcriptConversely, by looking at the transcripts that were highly ranked in our DTU list but lowly ranked in the top list of limma, we observe that our model is more likely to capture small changes in transcript usage that are stable across all cells and belong to genes that are highly expressed. An example of such a transcript is shown inWe therefore argue that, given the above observations, the transcripts identified by satuRn should be considered more reliable, as they generally originate from genes containing more information for assessing DTU.et al. with DoubleExpSeq. DoubleExpSeq identified a large number of DTU transcripts in all eight comparisons between cell types, ranging from 335 to 4580 DTU transcripts (Extended data figure S22). This is consistent with our performance benchmarks, which already suggested that DoubleExpSeq becomes overly liberal in single-cell datasets with a large number of cells . We therefore expect many of these transcripts to correspond to false positives. Furthermore, this is reflected in the pathological distribution of p-values obtained by DoubleExpSeq, where p-values tend to be small and therefore the analysis too liberal (Extended data figure S25). Furthermore, as discussed in the benchmark studies, we could not adopt the empirical null strategy to improve the FDR control of DoubleExpSeq. Again, a large number of p-values equal 1 poses a problem for estimating the empirical null distribution (Extended data figure S26).We additionally analyzed the dataset by TasicWhile the results of DoubleExpSeq are likely to be overly liberal, the ranking of the transcripts  might still be reasonable. InFinally, we note that while DoubleExpSeq could still be used in this case study given the simple factorial design (using a single factor to assign each cell to a cell type), DoubleExpSeq cannot be used in multifactorial designs, for instance to compare expression levels across multiple cell types between multiple samples or treatment groups.The analysis of scRNA-seq experiments often requires incorporating covariates in the statistical model, for instance to correct for batch effects or to account for subject-level baseline characteristics like age or body mass index. satuRn and most other DTU tools allow for the analysis of complex experimental designs with multiple categorical and continuous covariates (see Table 1). Exceptions are DoubleExpSeq and BANDITS that do not support modeling multiple covariates, which limits their utility for scRNA-seq data analysis and bulk RNA-seq experiments with complex designs. This may result in suboptimal data analysis. First, covariates may be able to explain a larger part of the variability in the data, which will increase the statistical power to pick up the effect of interest, as the explained variability would otherwise be incorporated in the error term. Second, the omission of covariates may result in biased parameter estimates. Third, the included covariates themselves could be key to further unravel the studied biological process.et al. but now we include a main effect for cell type, gender, and the continuous covariate age, as well as an interaction effect between cell type and gender.To demonstrate satuRn’s ability to analyze datasets with complex experimental designs, we reanalyzed the single-cell study from TasicENSMUST00000198199 dropped from the 8th to the 68th position after correcting for gender and age . Second, we assessed the interaction effect to prioritize transcripts for which the DTU between cell types differs according to gender. The differential transcript usage ofENSMUST00000145174 between Hsd11b1 Endou and Tnc cell types, for instance, is significantly different between female and male mice . Indeed, for female mice there is evidence for DTU , while no difference in usage is observed for male mice  algorithm or similar for assigning reads to transcripts. This is a challenging task, given the largely overlapping RNA sequence of different gene isoforms. Therefore, it has been suggested to perform differential usage analyses on transcript compatibility counts (TCCs),,, sometimes referred to as equivalence class (EC) counts, instead. An equivalence class is the set of transcripts that a sequencing read is compatible with. As such, reads that are pseudo-aligned to the same set of transcripts are part of the same equivalence class. A TCC is the number of reads assigned to each of the different equivalence classes. Note, that TCCs can be obtained much faster than transcript-level counts, as the former does not require running the EM algorithm. Another important advantage of TCCs is that they can also be obtained from UMI-based scRNA-seq experiments, like the 10× Chromium protocol.To obtain transcript-level abundance estimates, pseudoalignment tools like kallistoet al. The DECU analysis is performed without correction for gender and age to allow for a comparison with our elaborate transcript-level analysis from the first case study analysis. We adopt the same filtering criteria as for the transcript-level analysis. However, ECs that correspond to more than one gene are additionally removed, as the relationship between ECs and genes needs to be determined unambiguously to infer within-gene EC usages., In our analysis, approximately 15% of the reads were discarded for this reason, which is in line with previous reports. After these steps, 26.141 ECs are retained, as compared to 14.420 transcripts in the transcript-level analysis. Next, the same comparisons of interest are tested. The number of differentially used ECs for each comparison are displayed inHere we perform a differential equivalence class usage (DECU) analysis on the single-cell dataset from TasicFor several comparisons, there is a clear discrepancy between the number of differentially used transcripts and equivalence classes (ENSMUSG00000021699). As depicted inENSMUST00000134973 andENSMUST00000138938. In the transcript-level analysis, transcriptENSMUST00000138938 was not retained after filtering, as most of these reads were assigned to transcriptENSMUST00000134973 by Salmon. Note that while the interpretation at the transcript level is more sensible from a biological perspective, it also assumes an appropriate attribution of reads to transcripts.For comparison 6 (53% overlap), we investigate the genes identified by both analyses. For some genes that are identified in both analyses, there is a strong agreement between the transcript-level and EC-level results. One example is gene Phosphodiesterase 4D or Pde4d (Ensembl IDENSMUSG00000029470), which was discussed in Figure 6 of the transcript-level case study, was also identified in the EC-level analysis. Four P2rx4 equivalence classes passed feature-level filtering, which we will denote with EC1-EC4. EC1 uniquely corresponds to transcriptENSMUST00000195963, and both the transcript and EC are found to be differentially used in their respective analyses . The other ECs correspond to two, three and four different transcripts, respectively, but not to transcriptENSMUST00000195963 . For EC2 and EC3, strong evidence for differential usage was found. For EC4, there was no evidence for differential usage. However, based on these ECs alone, it would not be possible to unambiguously attribute these changes to single transcripts, complicating the biological interpretation . In contrast, while the transcript-level analyses do allow for clear functional interpretation, whether these results reflect true underlying biology depends on the ability of the quantification tools to assign reads to transcripts (seeDiscussion). Finally, none of the four ECs are compatible with transcriptENSMUST00000132062, the third transcript that passed feature-level filtering in the transcript-level analysis. In the EC-level analysis, there were many equivalence classes corresponding to this transcript but with a relatively low count, such that these individual equivalence classes did not pass filtering. The fact that the transcript was retained in the transcript-level analysis can be explained by Salmon assigning many of the reads of these low count equivalence classes that correspond to multiple transcripts to transcriptENSMUST00000132062.For most other genes in which differential usage was identified in both analyses, the comparison between the transcript analysis and the equivalence class analysis is more ambiguous, as equivalence classes often do not allow to pinpoint the responsible transcript(s) driving the observed differential signal. For example, the gene P2X Purinoceptor 4 or P2rx4 . More importantly, however, the ranking of the exons based on their respective p-values is almost identical between the two methods . Again, this is supportive of a similar performance of DEXSeq and satuRn on small bulk RNA-Seq datasets, albeit with a different type 1 error rate. InExtended data Figure S29, we additionally demonstrate how satuRn can be used to visualize changes in the usage of exons.Compared to the original DEXSeq analysis, satuRn identified fewer differentially used exons between the two experimental groups, with three statistically significant findings at 5% FDR for satuRn compared to 20 significant findings with DEXSeq. While we did not benchmark satuRn and DEXSeq on exon-level data, this could correspond to DEXSeq being overly liberal in transcript-level bulk analyses, especially with small sample sizes RNA-seq experiments. Furthermore, given the underlying GLM framework, our method can handle complex experimental designs that are commonplace in scRNA-seq experiments. Finally, satuRn can extract biologically relevant information from a large scRNA-seq dataset that would have remained obscured in a canonical DGE analysis. Currently, satuRn and all other DTU methods discussed in this manuscript, except for BANDITS, neglect the uncertainty on this abundance estimate. BANDITS models the abundance uncertainty; however, it had a markedly lower performance than our method in our benchmark evaluation (Extended data figure S1).Since most sequencing reads map to multiple transcripts, quantification tools such as Salmon or kallisto only provide an estimate of the expected number of fragments originating from each transcript. Incorporating quantification uncertainty has recently been shown to improve results in differential expression analysis of single-cell RNA-seq datasets. dataset (Extended data figure S9) are markedly lower than the performances on the other two datasets (Extended data figure S8). However, when we stratify the benchmark results according to the percentage of zero counts at the gene level, we observe that performances are negatively associated with sparsity, with all methods having worse performances for transcripts with many zero counts, across all datasets . The overall difference in performance between the different datasets is thus mainly driven by the number of transcripts with low, middle, and high fraction of zeros. A closer inspection of the Darmanis dataset showed that, after filtering, the transcript-level counts matrix contains a much larger percentage of zero counts than the other datasets . We also more frequently observed the scenario where the expression level of a gene could be attributed to a single isoform . This effectively causes a binary transcript usage pattern at the level of individual cells, with transcript usages of 0% or 100% Although that this may reflect the true underlying biology, We argue that it is more likely to be a technical artefact as a consequence of more shallow sequencing, given the lower percentage of binary usage profiles in the Chen and Tasic datasets. The supposedly binary expression of transcripts due to coverage-dependent bias and the use of more stringent filtering criteria to reduce this bias has already been comprehensively reported by Najaret al.One challenge common to all DTU methods is that the power to detect differentially used transcripts depends strongly on sequencing depth sequencing depth of the scRNA-seq dataset. This becomes clear when comparing the performances for the three different scRNA-seq benchmarks in this manuscript. The performances on the Darmaniset al., we performed a case study on both transcript-level counts and transcript compatibility counts (TCCs). TCCs have several interesting properties: they are fast to compute and avoid the ambiguity in assigning reads to transcripts with often highly similar RNA sequences. The latter can be particularly difficult with UMI-based RNA-seq  data, given that only one end of the paired-end sequencing read can be used for transcriptome alignment. Nevertheless, biological interpretation of equivalence classes that correspond to multiple transcripts of a gene is challenging. Functional interpretation will therefore be limited to pinpointing genes with differential usage of equivalence classes. Transcript-level analyses, however, do allow for clear functional interpretation but assume the correct attribution of reads to transcripts. An additional disadvantage of equivalence class-level analyses is that reads mapping to multiple genes must be discarded, which for our analysis removed approximately 15% of all counts.For a subset of the single-cell dataset of TasicWe conclude with the following recommendations for DTU analysis from an applied perspective. In case of small bulk RNA-seq datasets, satuRn, DEXSeq and DoubleExpSeq can be used interchangeably. In case of datasets with more complex designs that require the DTU model to incorporate additional covariates, e.g., batch effects, DoubleExpSeq cannot be used. For single-cell datasets, using DEXSeq will become infeasible in terms of scalability and DoubleExpSeq may give overly liberal results. As such, we recommend satuRn for performing DTU analyses in large bulk and single-cell RNA-seq datasets.https://doi.org/10.5281/zenodo.6826603.Zenodo: Datasets associated with this publication-Case_study.zip -Performance_Chen.zip -Performance_Darmanis.zip -Performance_Dmelanogaster.zip -Performance_GTEx.zip -Performance_Hsapiens.zip -Performance_Love.zip -Scalability_analysis.zip This project contains the following underlying data:Scalability_analysis.zip contain intermediate DTU analysis results that are available as.Rdata files or, in the case ofCase_study.zip,.Rds files. Data are available under the terms of theCreative Commons Attribution 4.0 International license (CC-BY 4.0).In addition, all folders excepthttps://doi.org/10.5281/zenodo.6810615.Zenodo: Extended data for satuRn publication-Supplementary_Figures.pdf: The supplementary figures to the satuRn publication, including figure captions.-DTU_Methods_Detail.pdf: A.pdf text file that describes the different DTU tools that were included in our benchmarks in greater detail as compared to the description in our main publication. For even higher detail, we refer to the respective original publications.-GSEA_MSigDB.xlsx: The output of the Gene Set Enrichment Analyses (GSEA) for the case study of our publication as generated by the online MSigDB platform.This project contains the following extended data:CC-BY 4.0 license.License: Data are available under the terms of thehttps://github.com/statOmics/satuRnsatuRn R package:https://doi.org/10.5281/zenodo.4656084.Archived source code at time of publication:https://github.com/statOmics/satuRnPaperCode to reproduce analyses and figures:https://doi.org/10.5281/zenodo.6826612.Archived analysis code at time of publication:CC-BY 4.0 license.License: The axis labels for Figure S20A should read "number of samples per group" rather than cells.Table S1 should use decimal points instead of commas for percentage values.There are some mixed-up figure references in the last paragraph in the "Performance on simulated bulk RNA-seq datasets" section (page 11). I believe the first sentence should refer to S9 rather than S13. S10-12 are also not referenced in the text.My original point about Figure 8 referred to the A-C labels that were missing. These labels still haven't been added .In version 2 of the manuscript, the authors have comprehensively addressed the suggested revisions. There are some remaining minor issues; addressing these would help with clarity:Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?YesAre the conclusions about the method and its performance adequately supported by the findings presented in the article?YesIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?PartlyReviewer Expertise:Bioinformatics, method development, transcriptomicsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thanks to the authors for addressing fully the concerns raised in my first revision. They now discuss the general limitations of scRNA-seq affect differential transcript usage analyses, and how each tool is able to capture events of different magnitudes.  Finally, I want to congratulate the authors for putting the manuscript together and a well-documented and functional software package.Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?YesAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?YesReviewer Expertise:Bioinformatics, Computational Biology, Drug DiscoveryI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The transcript swapping strategy seems quite powerful to benchmark DTU methods. This strategy is probably introducing DTU of different magnitudes. It would be highly informative if the authors can assess the performance of the different methods as a function of the strength of the induced DTU. How is the performance of satuRn and the other tools across different ranges of DTU induction?The authors describe a strategy to calculate p-values based on t-test on the estimated log-odds transcript ratios. They also describe that this strategy is too liberal for scRNA-seq analyses and provide an alternative approach that converts these p-values to z-scores and use locfdr to estimate an empirical null distribution. Based on the paper and the vignette, it was unclear to me which strategy is followed for bulk RNA-seq analyses? Does the user have the possibility of choosing which strategy to use?The filtering strategy for scRNA-seq datasets is very stringent (at least 1 count in 50% of the cells) and might result in mostly highly expressed genes/transcripts. How many genes can be tested for DTU after such thresholds are applied in real scRNA-seq data? It would also be very informative to show benchmarks with less stringent criteria to assess how the presence of many zeros affect each method. et al. (2019)). Examples would be when the transcripts are not distinguishable from the 3’ends alone. The authors should consider testing their approach and benchmarks at the level of transcript compatibility counts. Given that most scRNA-seq protocols only sequence the 3’ end of transcripts, it might be unfeasible to reliably estimate transcript expression (this is discussed in NtranosI found unclear whether stageR applied by default (as I understand from the vignette)? If so, I think it is important to apply stageR to all the methods tested in the benchmarks to distinguish the performance of saturn vs satuRn + stageR. The authors describe the applicability of their method for differential transcript usage. However, as I understand the model, it could also be used for differential exon usage, or groups of reads supporting an alternative splicing event . I think the authors could showcase these examples to expand the applications of their model.In most ROC curves it’s a bit hard to distinguish each different line due to over plotting.The authors propose a new method to test for differences in transcript usage for both bulk RNA-seq and single-cell RNA-seq data. The manuscript is clear and the method is sound and well explained. The authors provide an excellent implementation of their method as a Bioconductor package whose vignette is easy to follow. The main advantage and novelty of satuRn is that it scales well with very large samples, which is a limitation of some of the best performing methods in literature. Below are a couple of suggestions to provide more context to the DTU events on the benchmarks, assess to what extent DTU for single cells might be a limitation and a couple of requests for clarifications in the manuscript.Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?YesAre the conclusions about the method and its performance adequately supported by the findings presented in the article?PartlyIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?YesReviewer Expertise:Bioinformatics, Computational Biology, Drug DiscoveryI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We are grateful for the comments and suggestions of both reviewers, which have markedly improved the readability of the manuscript and have led to several additional analyses and findings. We here first summarise the major and minor changes to the manuscript and then address the reviewer-specific comments point-by-point:We include a case study analysis on single-cell RNA-seq data with a more complex multifactor experimental design. We first show that including additional covariates to the differential transcript usage (DTU) analysis affects the ranking of transcripts in the resulting top list as compared to an analysis without these covariates. Second, we use the more complex model to identify covariate-specific changes in DTU.We include a case study that performs a differential usage analysis at the level of equivalence classes. We qualitatively compare results between a transcript-level and equivalence class-level analysis on the same data, highlighting the advantages and disadvantages of both analyses.We included a short case study on differential exon usage on a small bulk RNA-seq dataset, comparing satuRn with DEXSeq.We elaborate on the effect of data sparsity and filtering on the benchmark performances of the different DTU tools. We additionally summarize the key characteristics of the different datasets.We elaborate on the comparison between using raw counts or scaledTPM abundances in the context of DTU analyses.We assess the performance of DTU methods as a function of the strength of the induced DTU signal as introduced by the swapping strategy in our performance benchmarks.We additionally include a scalability analysis of the different DTU tools on bulk RNA-seq data. Minor changes:et al. and the other datasets.We discuss the discrepancy of the performance benchmark results between the dataset from LoveWe add information on the quantification strategies for the single-cell RNA-seq datasets.We make more explicit which strategies were used for introducing DTU signal in the different datasets that were used for benchmarking the DTU methods.We clarify the acronym FDP.We labeled the panels of Figure 8. In addition, we provided labels to Figure 6 and adapted the captions accordingly.Extended data figures.We provide a more detailed description of the null distribution histograms in theWe clarify that, by default, satuRn adopts the testing strategy that is based on the empirical null distribution, but that users could opt for working with the unadjusted p-values as well.We clarify that we did not perform stage-wise testing adjustment in any of the analyses presented in the manuscript.We aimed to improve the layout of the performance curves. Response to comments Reviewer 2:The swapping strategy will indeed induce DTU signal of different magnitudes. We followed the reviewer’s suggestion of assessing the performance of the different methods as a function of the strength of the induced signal.Extended data , corresponding to the four benchmarks that used swapping, where we stratify FDR-TPR curves based on the strength of the induced DTU signal. As an estimate for the strength of the signal, we take the difference in the empirical average transcript usage between the two groups of cells/samples. The empirical usage was computed by dividing the transcript-level count with the corresponding gene-level count in each cell/sample. Based on this metric, the FDR-TPR curves are stratified in 6 categories: a difference in transcript usage between groups of 0-10%, 10-20%, 20-30%, 30-40%, 40-50% and 50-100%. A brief discussion of the results is included in the “Performance on real single-cell data” paragraph of the Results section of the manuscript. Specifically, we have included four additional figures in the First, as expected, the ability of all methods to detect DTU decreases when the strength of the DTU signal decreases. Second, the difference of transcript usage that can be correctly detected is dependent on the data: small differences in transcript usage can be more easily detected in bulk data as compared to single-cell data, likely owing to the larger sequencing depth and lower variability of bulk datasets. Third, inference improves for all strata when the number of cells or samples increases. Fourth, we observed in all benchmark datasets that satuRn and DoubleExpSeq are more successful in detecting small differences as compared to the other methods. As such, the performance gain of satuRn and DoubleExpSeq in the overall results (without stratification) can be attributed to picking up DTU with smaller effect sizes. As a final note, having different transcript-level filtering procedures (edgeR filter versus DRIMSeq filter) did not affect these findings. Adopting scaledTPM abundances instead of raw counts did not affect the results, either (results not shown).In the second comment, the reviewer asked if satuRn by default uses the empirical null strategy to improve FDR control, if this is also the case in bulk analyses and if this is tunable by the user.Extended data Figure S14. However, it is possible for the user to use the “raw” p-values and FDR instead, as they are also provided in the output of thetestDTU function. By default, we indeed always use the strategy that relies on the empirical null distribution, both for single-cell and bulk datasets. The empirical null strategy improved the FDR control in single-cell data and performed on par in bulk data, as shown intestDTU function, i.e., the fact that both the raw and empirical p-values are returned and can both be used for downstream analysis (see the updated vignette athttps://statomics.github.io/satuRn/articles/Vignette.html). The empirical null strategy being the default is discussed at the end of the “Performance on real single-cell data” paragraph of the Results section of the manuscript. However, in the Bioconductor vignette, we indeed failed to explicitly mention which strategy we are using. We have now added this information to the vignette, where we now also more explicitly describe the output of theIn the third comment, the reviewer requests to assess the effect of filtering and data sparsity on the performance of the different DTU tools. This relates to a comment also raised by reviewer #1, who requested to assess how the performance of the different DTU methods is affected by the percentage of zero counts in the data and if we could display some of the sparsity related summary statistics and distributions of the different benchmark datasets. We therefore address these comments jointly.Extended data Table S1) containing relevant summary statistics for the different datasets, including the number of features retained after filtering in the different datasets, as well as a figure  that visualizes some of the sparsity related properties of the datasets . These clearly show that the Darmaniset al. dataset, for which the performance was lowest for each DTU method, is by far the sparsest dataset in our benchmark analyses. We first provide a table . Third, some methods seem more sensitive than others against sparse data. In particular, the performance of DEXSeq on the Tasicet al. 20v20 dataset and the Chenet al. 20v20 dataset  is on par with the performance of satuRn and DoubleExpSeq for the strata with a low or middle percentage of zeros, but markedly lower for the stratum with the highest percentage of zeros. As such, the performance gain of satuRn and DoubleExpSeq over DEXSeq in these datasets can be largely attributed to a better performance on the sparsest part of the data. Note that this is in line with the performance of DEXSeq being on par with satuRn and DoubleExpSeq on bulk data. Second, we explored how differences in sparsity between the different datasets are reflected in the performance benchmarks. To this end, we construct FDR-TPR curves where the genes  have been stratified on the percentage of zero counts. For each dataset, we thus assess the performance on three strata of genes, i.e., for genes that have a low (< 25%), middle (25-50%) or high (> 50%) percentage of zero counts in their corresponding transcript-level count matrices. All the transcripts of the gene get stratified in this gene-level stratum. The results of this performance benchmark have been included in theExtended data Figures S30-S32 and Table S1, we included a discussion of these results in the Discussion section of the manuscript (third paragraph). In addition toWe have followed the reviewer’s suggestion of including an additional case study in the manuscript that performs a differential usage analysis on transcript compatibility counts (TCCs), sometimes referred to as equivalence class (EC) counts.et al., which was also used for the transcript-level case study, allowing us to qualitatively compare the results of both analyses. This additional analysis led to several novel insights which we describe detail in paragraph “Case study on transcript compatibility counts” of the Results section and discuss in the fourth paragraph of the Discussion section of the manuscript. In addition, Figure S27 was added to theExtended data. We here attempt to briefly summarise the key outcomes of the analysis. More specifically, we have performed an EC-level analysis on the dataset of Tasic First, we display the number of differentially used transcripts and equivalence classes in eight comparisons between cell types in Table 3. The extent to which transcript-level and EC-level results correspond differs between the different comparisons. For several comparisons, there is a clear discrepancy between the number of differentially used transcripts and equivalence classes  and in the number of unique genes in which evidence for differential usage was observed . The overlap between the genes found in the transcript-level and the TCC-level analysis ranges from 53% (54/102) in comparisons 6 and 8 to 100% in comparison 3.ENSMUST00000195963, and both the transcript and TCC are found to be differentially used in their respective analyses . The other ECs correspond to two, three and four different transcripts, respectively . For EC2 and EC3, strong evidence for differential usage was found. For EC4, there was no evidence for differential usage. However, based on these ECs alone, it would not be possible to unambiguously attribute these changes to single transcripts, complicating the biological interpretation. For comparison 6 (53% overlap), we investigate the genes identified by both analyses. For some genes that are identified in both analyses, there is a strong agreement between the transcript-level and EC-level results. One example is the gene Pde4d. As depicted in Figure 11, there is a one-to-one concordance in the usage profiles of the equivalence classes and the transcripts. In other words, for this gene, the EC-level analysis is sufficient to obtain the same qualitative results as the transcript-level analysis. This is highly interesting, as the TCCs require only a fraction of the time to compute compared to the transcript-level counts, and they have no quantification uncertainty. However, for most other genes in which DU was identified in both analyses, the comparison between transcript and EC results is more ambiguous, as ECs often do not allow to pinpoint the responsible transcript(s) driving their observed DU. We demonstrate this by investigating the gene P2rx4. Four P2rx4 equivalence classes passed feature-level filtering, which we will denote with EC1-EC4. EC1 uniquely corresponds to transcript In the Discussion, we therefore conclude that biological interpretation of equivalence classes that correspond to multiple transcripts of a gene remains challenging. Currently, functional interpretation will therefore be limited to pinpointing genes with differential usage of a certain equivalence classes, without being able to link this to a functional effect at the transcript- or protein level. Finally, we note that while an analysis on scRNA-seq data from droplet protocols (like Chromium 10X) where the transcripts are not distinguishable from the 3’ ends alone would also be very interesting, we did not include such analysis in this manuscript. In our opinion, an extensive benchmarking of the different DTU tools on such data would be required and is out of the scope of this manuscript.The reviewer asked if we by default use the stage-wise testing procedure stageR in a satuRn analysis workflow.et al. However, conceptually, a stage-wise testing procedure makes sense in the context of DTU and should result in increased statistical power for all DTU methods. In this context, stageR has also been included in the downstream analysis of DTU methods in the F1000research workflow paper by Loveet al. (https://doi.org/10.12688/f1000research.15398.3). We do not apply stageR by default. Hence, all results from the paper are generated without stage-wise testing to allow for a fair comparison between the methods. We did not systematically compare results for all methods with and without stageR, as this would further increase (double) the number of benchmarks in this paper. In addition, it is not an integral part of any DTU method and the utility of stageR in a DTU setting has been demonstrated previously in the original stageR publication of Van den Berge We only included stageR as a downstream analysis method in our package vignette, where we wanted to showcase how to use stageR in the downstream analysis of satuRn output. We have now specifically stated in the vignette that this is an optional step.Extended data.We have followed the reviewer’s suggestion of including an additional case study in the manuscript that performs a differential usage analysis at the level of exons. To briefly showcase this, we adopted the differential exon usage (DEU) analysis using the data from the DEXSeq vignette, i.e., we perform the same DEU analysis as the DEXSeq vignette using both DEXSeq and satuRn. This analysis has been included in the “Case study on exon-level counts” paragraph in the Results section of the manuscript. In addition, we added Figures S28 and S29 to theExtended data Figure S2). More importantly, however, the ranking of the exons based on their respective p-values is almost identical between the two methods . Again, this is supportive of a similar performance of DEXSeq and satuRn on small bulk RNA-Seq datasets, albeit with a different type 1 error rate. InExtended data Figure S29, we additionally demonstrate how satuRn can be used to visualise changes in the usage of exons. Compared to the original DEXSeq analysis, satuRn identified fewer differentially used exons between the two experimental groups, with three statistically significant findings at 5% FDR for satuRn compared to 19 significant findings with DEXSeq. While we did not benchmark satuRn and DEXSeq on exon-level data, this could correspond to DEXSeq being overly liberal in transcript-level bulk analyses, especially with small sample sizes . While this is computationally less challenging, it would be helpful for the authors to also show satuRn’s scalability on bulk RNA-seq, for instance, using the LoveThe Discussion notes that performance on different scRNA datasets is affected by the percentage of zero counts: It would be helpful if the authors could show the zero count distributions in the three different scRNA data sets to illustrate this.It can be quite difficult to compare across the FDR-TPR curves when assessing performance using the different filters and abundance estimation methods. Perhaps for a subset of the comparisons and only showing the best performing tools, the authors could consider adding a set of simplified figures that compares the different filtering/counting approaches on a single plot per tool. et al. data., which is presumably due to the simulation methodology. Do the authors have any insights into why this dataset performed better using pseudo-alignments?The authors state that performance of raw abundance estimates was better in all data sets, except for the LoveThe quantification methodology of the scRNA data is not described in the methods and should be included.In the two sections describing the real bulk and real single-cell study data, the authors refer to the ‘swapping strategy’ described in the Methods. However, two different swapping strategies are detailed. Please clarify.In the ‘Performance assessment’ section, the acronym FDP is not spelled out.Panels A-C in Figure 8 are missing labels.It would be helpful if the authors could explain the null distribution histograms a bit more, i.e. what the green and blue lines, red triangles and highlighted distributions show. The following points would help improve the manuscript:Is the rationale for developing the new method (or application) clearly explained?YesIs the description of the method technically sound?YesAre the conclusions about the method and its performance adequately supported by the findings presented in the article?YesIf any results are presented, are all the source data underlying the results available to ensure full reproducibility?YesAre sufficient details provided to allow replication of the method development and its use by others?PartlyReviewer Expertise:Bioinformatics, method development, transcriptomicsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We are grateful for the comments and suggestions of both reviewers, which have markedly improved the readability of the manuscript and have led to several additional analyses and findings. We here first summarise the major and minor changes to the manuscript and then address the reviewer-specific comments point-by-point:We include a case study analysis on single-cell RNA-seq data with a more complex multifactor experimental design. We first show that including additional covariates to the differential transcript usage (DTU) analysis affects the ranking of transcripts in the resulting top list as compared to an analysis without these covariates. Second, we use the more complex model to identify covariate-specific changes in DTU.We include a case study that performs a differential usage analysis at the level of equivalence classes. We qualitatively compare results between a transcript-level and equivalence class-level analysis on the same data, highlighting the advantages and disadvantages of both analyses.We included a short case study on differential exon usage on a small bulk RNA-seq dataset, comparing satuRn with DEXSeq.We elaborate on the effect of data sparsity and filtering on the benchmark performances of the different DTU tools. We additionally summarize the key characteristics of the different datasets.We elaborate on the comparison between using raw counts or scaledTPM abundances in the context of DTU analyses.We assess the performance of DTU methods as a function of the strength of the induced DTU signal as introduced by the swapping strategy in our performance benchmarks.We additionally include a scalability analysis of the different DTU tools on bulk RNA-seq data. Minor changes:et al. and the other datasets.We discuss the discrepancy of the performance benchmark results between the dataset from LoveWe add information on the quantification strategies for the single-cell RNA-seq datasets.We make more explicit which strategies were used for introducing DTU signal in the different datasets that were used for benchmarking the DTU methods.We clarify the acronym FDP.We labeled the panels of Figure 8. In addition, we provided labels to Figure 6 and adapted the captions accordingly.Extended data figures.We provide a more detailed description of the null distribution histograms in theWe clarify that, by default, satuRn adopts the testing strategy that is based on the empirical null distribution, but that users could opt for working with the unadjusted p-values as well.We clarify that we did not perform stage-wise testing adjustment in any of the analyses presented in the manuscript.We aimed to improve the layout of the performance curves. Major changes:et al. but we now include a main effect for cell type, gender, and the continuous covariate age, as well as an interaction effect between cell type and gender. In this analysis, we first show that including these additional covariates affects the ranking of transcripts in the resulting top list as compared to our original unadjusted analysis. Second, we assessed the interaction effect to prioritise transcripts for which the DTU between cell types differs according to gender. Here, we highlight a transcript that displays strong evidence for DTU between cell types in female mice, but not in male mice. The analysis has been included in the manuscript as a new paragraph, “Analyses with complex experimental designs”, under the Results section and includes the new Figure 10.We have followed the reviewer’s suggestion of including a case study analysis with a multifactor experimental design. We reanalysed the single-cell study from TasicExtended data, Figures S20 and S21.We have followed the reviewer’s suggestion of including an additional scalability benchmark on bulk data. We performed the benchmark on the GTEx dataset and not the Love dataset (as was suggested by the Reviewer), because the latter only contains 24 samples. The results of the benchmark are in line with those of the single-cell scalability benchmark, with DEXSeq and DRIMSeq being markedly slower than the other methods. When directly comparing the scalability between the bulk and single-cell data on equally sized subsets, we found limited differences in scalability between the data types, except for DEXSeq. DEXSeq scales considerably better on bulk data than on single-cell data, suggesting that the estimation of the GLM parameters is slower when the data are sparse. However, since the scalability profile of DEXSeq is quadratic with respect to the number of cells/samples in the data, it is still infeasible to adopt DEXSeq in bulk datasets with many samples. The discussion of the results has been added to the “Scalability benchmark” paragraph of the Results section. Figures of the scalability profiles on bulk data have been included in theIn the third comment, the question was raised as to how the performance of the different DTU methods is affected by the percentage of zero counts in the data. This relates to a comment raised by reviewer #2, who requested “to show [DTU] benchmarks with less stringent criteria to assess how the presence of many zeros affect [the performance of] each method”. We therefore address these comments jointly.Extended data Table S1 containing relevant summary statistics for the different datasets, including the number of features retained after filtering, as well as Figure S32 that visualizes some of the sparsity related properties of the datasets . These clearly show that the Darmaniset al. dataset, for which the performance was lowest for each DTU method, is by far the sparsest dataset in our benchmark analyses. We provideExtended data, Figures S30 and S31. As expected, we observe that performances are higher when there is a low percentage of zero counts, uniformly across all datasets and for all methods. Second, performances on genes with high percentage zero counts increase strongly when the number of cells increases . Third, some methods seem more sensitive than others against sparse data. In particular, the performance of DEXSeq on the Tasicet al. 20v20 dataset and the Chenet al. 20v20 dataset  is on par with the performance of satuRn and DoubleExpSeq for the strata with a low or middle percentage of zeros, but markedly lower for the stratum with the highest percentage of zeros. As such, the performance gain of satuRn and DoubleExpSeq over DEXSeq in these datasets can be largely attributed to a better performance on transcripts of lowly abundant genes. Note that this is in line with the performance of DEXSeq being on par with satuRn and DoubleExpSeq on bulk data. Second, we explored how differences in sparsity between the different datasets are reflected in the performance benchmarks. To this end, we construct FDR-TPR curves where the genes  have been stratified based on the percentage of zero counts. For each dataset, we thus assess the performance on three strata of genes, i.e., for genes that have a low (< 25%), middle (25-50%) or high (> 50%) percentage of zero counts in their corresponding transcript-level count matrices. All the transcripts of the gene get stratified in this gene-level stratum. The results of this performance benchmark have been included in theExtended data Figures S30-S32 and Table S1, we included a discussion of these results in the Discussion section of the manuscript (third paragraph). In addition toExtended data Figures S10-S13 and are discussed in the “Performance on simulated bulk RNA-seq datasets” paragraph in the Results section of the manuscript.We have followed the reviewer’s suggestion of including simplified figures that compare the different filtering and counting approaches on a single plot per tool. These figures have been included as These figures show more clearly that neither sample size nor the filtering strategy had a profound impact on the ranking of the performances of the different DTU methods; satuRn, DEXSeq and DoubleExpSeq remain the best performing methods overall. However, we did find that the effects of filtering and counting differ between the different datasets.et al. simulated bulk dataset, filtering more stringently strongly improves performance. In addition, both performance and FDR control are much better when using scaledTPM abundances, as compared to using counts. The strong positive effect of using scaledTPM abundances are only observed for this dataset, which we will discuss in more detail in our response to the next comment of the reviewer. For the Loveet al., we also observe a positive effect of stringent filtering. The difference between scaledTPM and raw count abundances is negligible. For the simulated bulk datasets by Van den Bergeet al. and Chenet al., the effects of filtering are limited and using counts performs slightly better than using scaledTPMabundances. For the real datasets, a quite different pattern is observed. For the GTEx bulk dataset and the single-cell datasets by Tasicet al. dataset. Note that this dataset is by far the sparsest dataset in our benchmarks . The differences in performance between scaledTPM and raw count abundances are small. Finally, stringent filtering had a strong positive impact on performance on the Darmaniset al., for reasons we explain in below. Therefore, we recommend using raw counts abundances. With respect to filtering, the more stringent DRIMSeq filtering strategy only seems beneficial in simulated data and in very sparse single-cell data. Thus, for bulk data and regular single-cell data the lenient edgeR filtering strategy seems sufficient to increase the signal-to-noise ratio. Taken together, using scaledTPM abundances is only beneficial in the simulations by Loveet al. As suggested by the reviewer, this is indeed due to the simulation methodology.In our hands, the use of scaledTPM abundances instead of raw counts was indeed only beneficial in the simulated dataset by Loveet al. introduced DTU either by swapping TPM abundances or by introducing a fold change in transcript expression in one group of samples, again on TPM abundances. However, when back-transforming the swapped TPM abundances to count abundances, the transcript lengths were not swapped. Hence, the simulation strategy can induce differences in the total count for a gene, which in turn may alter the usage of all (target and off-target) transcripts in that gene when raw counts are used as an input, which gives rise to many false positives in off-target transcripts of DTU genes. Loveet al. simulation approach. When introducing DTU by swapping raw counts , we could expect spurious signal in analyses on the TPM scale for the same reason. However, the difference between the raw count and TPM analyses was much less pronounced, which was also the case for the Van den Berge We have included this as a remark in the “Performance on simulated bulk RNA-seq datasets” paragraph in the Results section of the manuscript.et al.29 and Tasicet al.42 were quantified using Salmon v1.1.0, whereas the dataset from Darmaniset al.26 was quantified with Salmon v0.8.2 as obtained from Soneson et al.43.”The quantification methodology for the single-cell RNA-seq data was indeed missing and has now been included in the “Real single-cell study” paragraph of the Methods section as follows: “The datasets from ChenWe agree with the reviewer that it previously unclear which swapping strategy was used in the “real bulk study” and “real single-cell study” paragraphs of the Methods section. In both cases, DTU is artificially introduced by swapping raw transcript counts between groups of samples, which we now explicitly mention in both paragraphs.We have now spelled out the acronym FDP in the “Performance assessment” paragraph of the Methods section, and defined it as follows: “The FDR is the expected value of the false discovery proportion (FDP), which is the ratio of the number of false positives over the total number of positives.”As requested, we labeled the panels of Figure 8. In addition, we provided labels to Figure 6 and adapted the captions accordingly.We agree with the reviewer that the null distribution histograms in the Extended data Figures S14, S15, S24 and S26 required further explanation. We have included this information in the caption of Figure S14 and referred to that information in the captions of Figures S15, S24 and S26. The update caption of Figure S14 reads as follows: “The z-scores corresponding to the null transcripts are expected to follow a standard normal distribution . This corresponds well with the maximum likelihood estimates (MLE) for the mean and variance of the empirical null distribution  as obtained with the locfdr package. In brief, these estimates are obtained by assuming that the z-scores of all transcripts follow a mixture distribution, where the z-scores of the null transcripts are expected to follow a normal distribution and the z-scores of the DTU transcripts follow some other distribution. Two models are fitted to the z-scores. The blue dashed curve is a normal distribution that is fitted to the mid 50% of the z-scores, which are assumed to originate from null genes, thus representing the estimated empirical null component densities. The MLE and central matching estimates (CME) for the mean and standard deviation of the estimated empirical null distribution are provided in the caption at the bottom of the plot. Finally, the green solid curve represents the estimated marginal density across all z-scores and is obtained by fitting a spline model to the histogram counts.”locfdr figures. In brief, thelocfdr package returns local fdr values for each transcript. The FDR is a set property, i.e., the expected fraction of false positives in the set that is returned. Efron (2010) showed that the FDR can also be interpreted as a posterior probability, i.e., the posterior probability that a random feature in the set that is returned is a false positive. Unlike the FDR, the local fdr of a feature is the posterior probability that this individual feature is a false positive, so it relates to a single specific null hypothesis of this specific feature and not to a set of null hypotheses. Efron (Ann. Stat. 2007) defined 0.2 as a useful cutoff for the local fdr and argues that a 0.2 cutoff for the local fdr often corresponds with a conventional FDR of 0.05. Efron showed that the FDR can also be interpreted as the average local fdr over the set that is returned as significant, i.e., the features with a local fdr < 0.2. The conventional FDR over this set can be estimated by taking the sample average of the local fdr of the features that are returned, and is bound to be much smaller than 0.2.  The triangles on thelocfdr plot indicate cut points for thez-scores;z-scores that are more extreme than the triangles will result in a local FDR estimate that is smaller than 0.2. To avoid confusing the reader with concepts that are not required for the paper, we have omitted the explanation of the yellow triangles on the Response to comments Reviewer 1:


Personalized therapy has been at the forefront of cancer care, making cancer treatment more effective. Since cancer patients respond individually to drug therapy, predicting the sensitivity of each patient to specific drugs is very helpful to apply therapeutic agents. Traditional methods focus on node (molecular) information but ignore relevant interactions among different nodes, which has very limited application in complex situations, such as cancer drug responses in real clinical practice.Treatment evaluation with Quantified Network (TreeQNet) is a webserver which could predict sensitivity to drugs for patients through the innovative use of proteomic and phosphoproteomic network from tumor tissues.http://bioinfo.ustc.edu.cn/. TreeQNet source code: https://github.com/Really00/treeqnet-web-front/.TreeQNet service:  Drug treatment is most closely related to the specific origin of cancer and the location of gene mutation. Driven by the development of integrated  patient . The mai patient . Existin patient , single  patient . Signali patient . In summIn a biological system, it is the interactions (regulations) or edges among molecules rather than single molecules that facilitate a biological function or signal transduction involved in diseases. Complex diseases are generally resulted from the failure of relevant systems instead of single molecules, which should be investigated in a dynamic and network manner. Thus, it is important to identify network signatures that are associated with complex diseases for early diagnosis and clinical prognosis. Edge biomarker represents one of the network signatures that are sensitive to network perturbation in diseases. Specifically, the “node” represents each single molecule, and the “edge” indicates the interaction or correlation between each pair of genes or molecules in the biological network. The “edge biomarker” is a more sensitive type of biomarker or signature that could reflect the perturbed networks in disease treatment. Cancer, as a complex disease caused by related systems, is more appropriate to use dynamic and intermolecular networks .So if effective edge markers can be identified from the omics data may add new insights to cancer drug prediction based on the EdgeBiomarker algorithm proposed by Zhang et al. . CombiniThe TreeQNet web server is implemented using Model-View-View Model (MVVM) architecture model, which separates the user interface from the operation logic to provide users with a better experience. Javascript and Hypertext Markup Language (HTML) are the main languages for web services. The front-end web is based on HTML and Cascading Style Sheets (CSS). View layer is constructed based on Vue framework 2.5 which makes the page render and respond very quickly. Vue achieves efficient bidirectional data binding and flexible component system through MVVM idea, and provides a flexible file upload configuration. We develop Kexpress framework 1.0 based on express at the back-end.Functionally, the server includes edge feature selection and drug sensitivity prediction. Model algorithm implemented with R seamlessly integrates with Kexpress framework shown in Fig. TreeQNet provides guidance on drug selection for individual patient, creating opportunities for personalized treatments of patients. TreeQNet can predict drug sensitivity based on kinase-substrate network and the whole algorithm flow is shown in Fig. These edge features are stored in the system as built-in data. Subsequently, we predict drug sensitivity based on elastic net regression model. Elastic net use 1696 edge strength features. Users can upload protein expression values and phosphorylation site expression values of patient tumor tissue and predict tumor cell growth inhibitions (TCGIs) for afatinib, gefitinib, and regorafenib.p value 0.002, Gefitinib R-squared 0.61, p value 0.026 and Regorafenib R-squared 0.75, p value 0.003. The observed TCGI ratio varied a lot due to the limited number of samples. The prediction model to some extent could alleviate the influence of outliers and thus the predicted TCGIs showed narrower scales than the observed ones. While, the high correlation between predicted and observed ones suggested the good performance of our prediction model. In addition, two external colorectal cancer samples were used for testing . As shown in Fig. TreeQNet is a user-friendly web server for treatment evaluation of cancer drugs that facilitates the precise treatment and evaluation for cancer patients and effectively selects the most suitable targeted therapy for patients without druggable mutations. For researchers, TreeQNet assists them to predict drug response through network analysis combined with proteomic and phosphoproteomic data. In addition, TreeQNet can help researchers establish an accurate index to determine the suitable drugs for a given tumor type. TreeQNet server accessible to users via web browsers is convenient for researchers to process data quickly, and provides a visual preview of the results.Additional file 1. Introduction to TreeQNet: The usage of the TreeQNet software is described. The format of the input data (including protein expression data and phosphorylation site expression data) is described. Relevant built-in data from the software is listed. We also outline the main calculation method and the results of typical run.