Detection of differential transcript usage (DTU) from RNA-seq data is an important bioinformatic analysis that complements differential gene expression analysis. Here we present a simple workflow using a set of existing R/Bioconductor packages for analysis of DTU. We show how these packages can be used downstream of RNA-seq quantification using the Salmon software package. The entire pipeline is fast, benefiting from inference steps by Salmon to quantify expression at the transcript level. The workflow includes live, runnable code chunks for analysis using DRIMSeq and DEXSeq, as well as for performing two-stage testing of DTU using the stageR package, a statistical framework to screen at the gene level and then confirm which transcripts within the significant genes show evidence of DTU. We evaluate these packages and other related packages on a simulated dataset with parameters estimated from real data. DTU may produce functionally different gene products through alternative splicing and changes to the coding sequence of the transcript, and may also result in transcripts with different untranslated regions on the 5’ or 3’ end of the transcript, which can affect binding sites of RNA binding proteins. Reyes and Huber2 found that alternative usage of transcription start and termination sites was a more common event than alternative splicing when examining DTU events across tissues in GTEx. Specific patterns of DTU have been identified in a number of diseases, including cancer, retinal diseases, and neurological disorders, among others3. Large-scale analyses of cancer transcriptomic data, comparing tumor to normal samples, have identified that protein domain losses are a common feature of DTU in cancer, including domains involved in protein-protein interactions5.RNA-seq experiments can be analyzed to detect differences across groups of samples in total gene expression – the total expression produced by all isoforms of a gene – and additionally differences in transcript isoform usage within a gene. If the amount of expression switches among two or more isoforms of a gene, the total gene expression may not change by a detectable amount, but the differential transcript usage (DTU) is nevertheless biologically relevant. DTU is common when comparing expression across cell types: recent analysis of the Genotype-Tissue Expression Project (GTEx)DEXSeq 6,diffSpliceDGE from theedgeR package8,diffSplice from thelimma package10, andDRIMSeq11. Other Bioconductor packages which are designed around a DTU analysis includestageR12,SGSeq13, andIsoformSwitchAnalyzeR14.stageR can be used for post-processing of transcript- and gene-level p-values from DTU detection methods, and will be discussed in this workflow.SGSeq can be used to visualize splice events, and leveragesDEXSeq orlimma for differential testing of splice variant usage. The Bioconductor packageIsoformSwitchAnalyzeR is well documented and the vignette available from theIsoformSwitchAnalyzeR landing page can be seen as an alternative to this workflow.IsoformSwitchAnalyzeR is designed for the downstream analysis of functional consequences of identified isoform switches. It allows for import of data from various quantification methods, includingSalmon, and allows for statistical inference usingDRIMSeq. In addition,IsoformSwitchAnalyzeR includes functions for obtaining the nucleotide and amino acid sequence consequences of isoform switching, which is not covered in this workflow. Other packages related to splicing can be found at theBiocViews links forDifferentialSplicing andAlternativeSplicing. For more information about the Bioconductor project and its core infrastructure, please refer to the overview by Huberet al.15.While many tutorials and workflows in the Bioconductor project address differential gene expression, there are fewer workflows for performing a differential transcript usage analysis, which provides critical and complementary information to a gene-level analysis. Some of the existing Bioconductor packages and functions that can be used for statistical analysis of DTU includeDRIMSeq publication is a good reference for these, having descriptions and comparisons with many current methods11. This workflow will build on the methods and vignettes from three Bioconductor packages:DRIMSeq,DEXSeq, andstageR. This Bioconductor workflow article does not contain any new statistical methods for detection of DTU or DGE, but instead leverages existing statistical methods and software packages.We note that there are numerous other methods for detecting differential transcript usage outside of the Bioconductor project. TheedgeR andDESeq2 have collaborated to develop thetximport package16 for summarizing the output of fast transcript-level quantifiers, such asSalmon17,Sailfish18, andkallisto19. Thetximport package focuses on preparing estimated transcript-level counts, abundances and effective transcript lengths, for gene-level statistical analysis usingedgeR7,DESeq220 orlimma-voom10.tximport produces an offset matrix to accompany gene-level counts, that accounts for a number of RNA-seq biases as well as differences in transcript usage among transcripts of different length that would bias an estimator of gene fold change based on the gene-level counts21.tximport can alternatively produce a matrix of data that is roughly on the scale of counts, by scaling transcript-per-million (TPM) abundances to add up to the total number of mapped reads. This counts-from-abundance approach directly corrects for technical biases and differential transcript usage across samples, obviating the need for the accompanying offset matrix.Previously, some of the developers of the Bioconductor packagestximport to import transcript-level estimated counts, and then pass these counts to packages such asDRIMSeq orDEXSeq for statistical analysis of differential transcript usage. Following a transcript-level analysis, one can aggregate evidence of differential transcript usage to the gene level. ThestageR package in Bioconductor provides a statistical framework toscreen at the gene level for differential transcript usage with gene-level adjusted p-values, followed byconfirmation of which transcripts within the significant genes show differential usage with transcript-level adjusted p-values12. The method controls theoverall false discovery rate (OFDR)22 for such a two-stage procedure, which will be discussed in more detail later in the workflow. We believe thatstageR represents a principled approach to analyzing transcript usage changes, as the methods can be evaluated against a target error rate in a manner that mimics how the methods will be used in practice. That is, following rejection of the null hypothesis at the gene level, investigators would likely desire to know which transcripts within a gene participated in the differential usage.Complementary to an analysis of differential gene expression, one can useSalmon method  of the GEUVADIS project23, and this was used as a baseline for transcript abundances in the simulation. Transcripts that were associated with estimated counts less than 10 had abundance thresholded to 0, all other transcripts were considered “expressed” .alpine30 was used to estimate realistic fragment GC bias from 12 samples from the GEUVADIS project, all from the same sequencing center .DESeq220 was used to estimate mean and dispersion parameters for a Negative Binomial distribution for gene-level counts for 458 non-duplicated GEUVADIS samples provided by therecount2 project24, accounting for variation associated with sequencing center and human population. Note that, while gene-level dispersion estimates were used to generate underlying transcript-level counts, additional uncertainty on the transcript-level data is a natural consequence of the simulation, as the transcript-level counts must be estimated (the underlying transcript counts are not provided to the methods).First we describe details of the simulated data, which will be used in the following workflow and in the evaluation of methods. Understanding the details of the simulation will be useful for assessing the methods in the later sections. All of the code used to simulate RNA-seq experiments and write paired-end reads to FASTQ files can be found at an associated GitHub repository for the simulation codepolyester31 was used to simulate paired-end RNA-seq reads for two groups of 12 samples each, with realistic fragment GC bias, and with dispersion on transcript-level counts drawn from the joint distribution of mean and dispersion values estimated from the GEUVADIS samples. We will call this the “main simulation”. To compareDRIMSeq andDEXSeq in further detail, we generated an additional simulation in which dispersion parameters were assigned to genes via matching on the gene-level count, and then all transcripts of a gene had counts generated using the same per-gene dispersion. We will call this the “fixed per-gene dispersion” simulation. The first sample for group 1 and the first sample for group 2 followed the realistic GC bias profile of the same GEUVADIS sample, and so on for all 12 samples. This pairing of the samples was used to generate balanced data, but not used in the statistical analysis.countsimQC32 was used to examine the properties of the simulation relative to the dataset used for parameter estimation (countsimQC (per-gene mean and dispersion) were estimated over 458 samples. The fullcountsimQC report can be accessed at the associated GitHub repository for simulation code25.imation . The simDifferential expression across two groups was generated as follows: The 46,933 expressed transcripts as defined above belonged to 15,017 genes. 70%  of these genes with expressed transcripts were set as null genes, where abundance was not changed across the two groups. For 10%  of genes, all isoforms were differentially expressed at a log fold change between 1 and 2.58 (fold change between 2 and 6). The set of transcripts in these genes was classified as DGE  by construction, and the expressed transcripts were also DTE , but they did not count as DTU , as the proportions within the gene remained constant. To simulate balanced differential expression, one of the two groups was randomly chosen to be the baseline, and the other group would have its counts multiplied by the fold change. For 10%  of genes, a single expressed isoform was differentially expressed at a log fold change between 1 and 2.58. This set of transcripts was DTE by construction. If the chosen transcript was the only expressed isoform of a gene, this counted also as DGE and not as DTU, but if there were other isoforms that were expressed, this counted for both DGE and DTU, as the proportion of expression among the isoforms was affected. For 10%  of genes, differential transcript usage was constructed by exchanging the TPM abundance of two expressed isoforms, or, if only one isoform was expressed, exchanging the abundance of the expressed isoform with a non-expressed one. This counted for DTU and DTE, but not for DGE. An MA plot of the simulated transcript abundances for the two groups is shown inSalmon17 and imported into R/Bioconductor withtximport16. Details about how to runSalmon, and which type of transcript-level estimated counts should be imported, is covered in the Workflow, with the exact code used to run the DTU analysis.Salmon estimates the relative abundance of each annotated transcript for each sample in units of transcripts-per-million (TPM); the estimated TPM values should be proportional to the abundance of the transcripts in the population of cells that were assayed. One critical point is thatSalmon only considers the transcripts that are provided in the annotation; it is not able to detect expression of any novel transcripts. If many un-annotated transcripts are expressed in the particular set of samples, successful application of this workflow may require first building out a more representative set of annotated transcripts via transcriptome assembly or full transcript sequencing.As described in the Introduction, this workflow uses transcript-level quantification estimates produced bySalmon estimates theeffective length of each transcript, which can take into account a number of sample-specific technical biases including fragment size distribution (default), fragment GC content, random hexamer priming bias, and positional bias along the transcript. If a transcript had certain properties, related to its length and its sequence content, that made it difficult to produce cDNA fragments and sequence reads from these fragments, then its effective length should be lower for that sample, than if these technical biases were absent. The estimates of TPM and the effective lengths can be used to estimate the number of fragments that each transcript produced, which will be calledestimated counts in this workflow.In addition to relative abundance,Different types of estimated counts may be correlated with effective transcript length, and so we will discuss in the Workflow our recommended type to use for DTU and DGE analysis  distribution for the feature counts, and considers the counts for each feature  relative to counts for all other features of the group (the gene), using an interaction term in a generalized linear model (GLM). The GLM framework is an extension of the linear model (LM), but shares with LM the usage of a design matrix, typically represented byX, which is made up of columns of covariates that are multiplied by scalar coefficients, typically represented byβ. The design matrix with its multiple coefficients is useful for extending statistical models beyond simple group comparisons, allowing for more complex situations, such as within-patient comparisons, batch correction, or testing of ratios.DEXSeq models each feature independently in the GLM fitting stage, and so does not take into account any correlation among the counts across features in a group (e.g. transcript counts within a gene), except insofar as such correlation is accounted for by columns in the design matrix. This last point is important, as correlation induced by DTU across condition groups, or even DTU that can be associated with cell-type heterogeneity, can be modeled in theDEXSeq framework by interaction terms with relevant covariates introduced into the design matrix. In the current workflow, we provide an example of capturing DTU across condition usingDEXSeq, but future iterations of this workflow may also include controlling for additional covariates, such as estimated cell type proportions.DEXSeq was evaluated on transcript counts by Sonesonet al.33 and then later by Nowicka and Robinson11, where it was shown in both cases thatDEXSeq could be used to detect DTU in this configuration, though isoform pre-filtering greatly improved its performance in reducing the observed false discovery rate (FDR) closer to its nominal level.DRIMSeq assumes an Dirichlet Multinomial model (DM) for each gene, where the total count for the gene is considered fixed, and the quantity of interest is the proportion for the transcript within a gene for each sample. If the proportion for one transcript increases, it must result in a decrease for the proportions of the other transcripts of the gene. Genes that are detected as having statistically significant DTU are those in which the proportion changes observed across condition were large, considering the variation in proportions seen within condition. The variation in proportions across biological replicates is modeled using a singleprecision parameter per gene, which will be discussed in the workflow below.DRIMSeq also uses a design matrix, and so can be used to analyze DTU for complex experimental designs, including within-patient comparisons, batch correction, or testing of ratios.In contrast to the NB model,DRIMSeq directly models the correlations among transcript counts within a gene via the DM distribution, and so can capture these correlations across exchangeable sampleswithin a condition.DEXSeq with the NB distribution only can capture correlations among transcript counts within a gene when the DTU isacross sample groups defined by covariates in the design matrix. On the other hand,DRIMSeq is limited in modeling a single precision parameter per gene. If there are two transcripts within a gene with very different biological variability – perhaps they have separate promoters under different regulatory control – thenDEXSeq may do a better job modeling such heterogeneity in the biological variability of transcript expression, as it estimates a separate dispersion parameter for each transcript.A critical difference between the two models is thatDRIMSeq,DEXSeq,stageR, andtximport packages from Bioconductor version 3.7. As the code for this article is linked to Bioconductor version 3.7 (released April 2018), please consult thelive Bioconductor workflow as the correct code for running the packages may change over time. Bioconductor packages should always be installed following theofficial instructions. The workflow uses a subset of all genes to speed up the analysis, but the Bioconductor packages can easily be run for this dataset on all human genes on a laptop in less than an hour. Compute time for the various packages is included in the respective evaluation sections.This workflow was designed to work with R 3.5, and theSalmon version 0.10.0 to quantify abundance and effective transcript lengths for all of the 24 simulated samples. For this workflow, we will use the first six samples from each group. We quantified against theGENCODE human annotation version 28, which was the same reference used to generate the simulated reads. We used the transcript sequences FASTA file that contains “Nucleotide sequences of all transcripts on the reference chromosomes”. When downloading the FASTA file, it is useful to download the corresponding GTF file, as this will be used in later sections.We usedSalmon index, we used the following command. Recent versions ofSalmon will discard identical sequence duplicate transcripts, and keep a log of these within the index directory. The–gencode command trims the GENCODE FASTA headers to only include the transcript identifier.To build thesalmon index --gencode -t gencode.v28.transcripts.fa \      -i gencode.v28_salmon-0.10.0sample and using as input these two reads files. The library type is specified by-l IU (inward and unstranded) and the options are discussed in theSalmon documentation. Recent versions of Salmon can automatically detect the library type by setting-l A. Such a command can be automated in a bash loop using bash variables, or one can use more advanced workflow management systems such as Snakemake34 or Nextflow35.To quantify each sample, we used the following command, which says to quantify with six threads using the GENCODE index, with inward and unstranded paired end reads, using fragment GC bias correction, writing out to the directorysalmon quant -p 6 -i gencode.v28_salmon-0.10.0 -l IU \      --gcBias -o sample -1 sample_1.fa.gz -2 sample_2.fa.gztximport to import the estimated counts, abundances and effective transcript lengths into R. Thetximport package offers three methods for producing count matrices from transcript-level quantification files, as described in detail in Sonesonet al.16, and already mentioned in the introduction. To recap these are: (1) original estimated counts with effective transcript length information used as a statistical offset, (2) generating “counts from abundance” by scaling TPM abundance estimates per sample such that they sum to the total number of mapped reads , or generating “counts from abundance” by scaling TPM abundance estimates first by the average effective transcript length over samples, and then per sample such that they sum to the total number of mapped reads . We will usescaledTPM for DTU detection, with the statistical motivation described below, and the original estimated counts with length offset for DGE detection.We can usernaseqDTU. If a user is applying the code in this workflow with her own RNA-seq data, she does not need to load thernaseqDTU package. If a user is running through the code in this workflow with the workflow simulated data, she does need to load thernaseqDTU package.We recommend constructing a CSV file that keeps track of the sample identifiers and any relevant variables, e.g. condition, time point, batch, and so on. Here we have made a sample CSV file and provided it along with this workflow’s R package,extdata directory where the CSV file is located. These two lines of code load the workflow package and find this directory on the machine. Again, these two lines of code would therefore not be part of atypical analysis using one’s own RNA-seq data.In order to find this CSV file, we first need to know where on the machine this workflow package lives, so we can point to thelibrary(rnaseqDTU)csv.dir <- system.filesample_id will be used later byDRIMSeq, and so using this column name allows us to pass thisdata.frame directly toDRIMSeq at a later step.The CSV file records which samples are condition 1 and which are condition 2. The columns of this CSV file can have any names, althoughsamps <- read.csv)head(samps)##   sample_id condition## 1      s1_1         1## 2      s2_1         1## 3      s3_1         1## 4      s4_1         1## 5      s5_1         1## 6      s6_1         1samps$condition <- factor(samps$condition)table(samps$condition)#### 1 2## 6 6files <- file.pathnames(files) <- samps$sample_idhead(files)##                         s1_1                         s2_1## "/path/to/dir/s1_1/quant.sf" "/path/to/dir/s2_1/quant.sf"##                         s3_1                         s4_1## "/path/to/dir/s3_1/quant.sf" "/path/to/dir/s4_1/quant.sf"##                         s5_1                         s6_1## "/path/to/dir/s5_1/quant.sf" "/path/to/dir/s6_1/quant.sf"tximport. For DTU analysis, we suggest generating counts from abundance, using thescaledTPM method described by Sonesonet al.16. ThecountsFromAbundance option oftximport uses estimated abundances to generate roughly count-scaled data, such that each column will sum to the number of reads mapped for that library. By usingscaledTPM counts, the estimated proportions fit byDRIMSeq, which are generated from counts, will be equivalent to proportions of the abundance of the isoforms.We can then import transcript-level counts usingscaledTPM, we used the original estimated transcript counts (countsFromAbundance="no"), or if we usedlengthScaledTPM transcript counts, then a change in transcript usage among transcripts of different length could result in a changed total count for the gene, even if there is no change in total gene expression. For more detail and a diagram of this effect, we refer the reader to Figure 1 of Trapnellet al.21. Briefly, this is because the original transcript counts andlengthScaledTPM transcript counts scale with transcript length, whilescaledTPM transcript counts do not. A change in the total count for the gene, not corrected by an offset, could then bias the calculation of proportions and therefore confound DTU analysis. For testing DTU usingDRIMSeq andDEXSeq, it is convenient if the count-scale data donot scale with transcript length within a gene. This could be corrected by an offset, but this is not easily implemented in the current DTU analysis packages. Forgene-level analysis (DGE), we can use gene-level original counts with an average transcript length offset, gene-levelscaledTPM counts, or gene-levellengthScaledTPM counts, as all of these three methods control for the length bias described by Trapnellet al.21 and Sonesonet al.16. Our DTU and DGEcountsFromAbundance recommendations are summarized inIf instead ofscaledTPM counts hinges on the fact that estimated fragment counts scale with transcript length in fragmented RNA-seq data. If a different experiment is performed and a different quantification method used to produce counts per transcript whichdo not scale with transcript length, then the recommendation would be to use these counts per transcript directly. Examples of experiments producing counts per transcript that would potentially not scale with transcript length include counts of full-transcript-length or nearly-full-transcript-length reads, or counts of 3’ tagged RNA-seq reads aggregated to transcript groups. In either case, the statistical methods for DTU could be provided directly with the transcript counts.A final note is that, the motivation for usingrnaseqDTU package due to size restrictions. Instead we will load a pre-constructed matrix of counts below. In a typical workflow, the code below would be used to generate the matrix of counts from the quantification files. All of the quantification files and simulated reads for this dataset have been made publicly available onZenodo; see theData availability section at the end of this workflow.The following code chunk is what one would use in a typical analysis, but is not evaluated in this workflow because the quantification files are not provided in thelibrary(tximport)txi <- tximportcts <- txi$countscts <- ctsTxDb object, a transcript database that can be used to link transcripts to genes (among other uses). The following code chunks were used to generate aTxDb, and then use theselect function with theTxDb to produce a correspondingdata.frame calledtxdf which links transcript IDs to gene IDs. In thisTxDb, the transcript IDs are calledTXNAME and the gene IDs are calledGENEID. The version 28 human GTF file was downloaded from the GENCODE website when downloading the transcripts FASTA file. Due to size restrictions, neither thegencode.v28.annotation.gtf.gz file nor the generated.sqlite file are included in thernaseqDTU package.Bioconductor offers numerous approaches for building alibrary(GenomicFeatures)gtf <- "gencode.v28.annotation.gtf.gz"txdb.filename <- "gencode.v28.annotation.sqlite"txdb <- makeTxDbFromGFF(gtf)saveDbTxDb database has been generated and saved, it can be quickly reloaded:Once thetxdb <- loadDb(txdb.filename)txdf <- select, "TXNAME", "GENEID")tab <- table(txdf$GENEID)txdf$ntx <- tab[match)]cts object as created in thetximport code chunks. This contains count-scale data, generated from abundance using thescaledTPM method. The column sums are equal to the number of mapped paired-end reads per experiment. The experiments have between 31 and 38 million paired-end reads that were mapped to the transcriptome usingSalmon.We load thedatacts##                         s1_1       s2_1       s3_1## ENST00000488147.1 179.798908 184.437348 229.046306## ENST00000469289.1   0.000000   0.000000   0.000000## ENST00000466430.5   5.004159   3.627831   9.463167range(colSums(cts)/1e6)## [1] 31.37738 38.47173txdf object giving the transcript-to-gene mappings . This is contained in a file calledsimulate.rda that contains a number of R objects with information about the simulation, that we will use later to assess the methods’ performance.We also have thedata(simulate)head(txdf)##               GENEID            TXNAME ntx## 1 ENSG00000000003.14 ENST00000612152.4   5## 2 ENSG00000000003.14 ENST00000373020.8   5## 3 ENSG00000000003.14 ENST00000614008.4   5## 4 ENSG00000000003.14 ENST00000496771.5   5## 5 ENSG00000000003.14 ENST00000494424.1   5## 6  ENSG00000000005.5 ENST00000373031.4   2all(rownames(cts) %in% txdf$TXNAME)## [1] TRUEtxdf <- txdfall(rownames(cts) == txdf$TXNAME)## [1] TRUEDRIMSeq, we build adata.frame with the gene ID, the feature (transcript) ID, and then columns for each of the samples:In order to runcounts <- data.frameDRIMSeq package and create admDSdata object, with ourcounts andsampsdata.frames. Typing in the object name and pressing return will give information about the number of genes:We can now load thelibrary(DRIMSeq)d <- dmDSdatad## An object of class dmDSdata## with 16612 genes and 12 samples## * data accessors: counts, samplesdmDSdata object has a number of specific methods. Note that the rows of the object are gene-oriented, so pulling out the firstrow corresponds to all of the transcripts of the first gene:Themethods(class=class(d))## [1] [           coerce      counts      dmFilter      dmPrecision length## [7] names       plotData    show## see ’?methods’ for accessing help and source codecounts##              gene_id        feature_id       s1_1       s2_1## 1 ENSG00000000419.12 ENST00000371588.9 1394.71411 1210.12539## 2 ENSG00000000419.12 ENST00000466152.5  135.15850   18.20031## 3 ENSG00000000419.12 ENST00000371582.8  154.77943   35.39425## 4 ENSG00000000419.12 ENST00000371584.8   42.85733   86.04958## 5 ENSG00000000419.12 ENST00000413082.1    0.00000    0.00000n to be the total number of samples, andn.small to be the sample size of the smallest group. We use all three of the possible filters: for a transcript to be retained in the dataset, we require that (1) it has a count of at least 10 in at leastn.small samples, (2) it has a relative abundance proportion of at least 0.1 in at leastn.small samples, and (3) the total count of the corresponding gene is at least 10 in alln samples. We used all three possible filters, whereas only the two count filters are used in theDRIMSeq vignette example code.It will be useful to first filter the object, before running procedures to estimate model parameters. This greatly speeds up the fitting and removes transcripts that may be troublesome for parameter estimation, e.g. estimating the proportion of expression among the transcripts of a gene when the total count is very low. We first definen was large, it would make sense to allow perhaps a few samples to have very low counts, so loweringmin_samps_gene_expr to some factor multiple (< 1) ofn, and likewise for the first two filters forn.small. The second filter means that if a transcript does not make up more than 10% of the gene’s expression for at leastn.small samples, it will be removed. If this proportion seems too high, for example, if very lowly expressed isoforms are of particular interest, then the filter can be omitted or themin_feature_prop lowered. For a concrete example, if a transcript goes from a proportion of 0% in the control group to a proportion of 9% in the treatment group, this would be removed by the above 10% filter. After filtering, this dataset has 7,764 genes.It is important to consider what types of transcripts may be removed by the filters, and potentially adjust depending on the dataset. Ifn <- 12n.small <- 6d <- dmFilterd## An object of class dmDSdata## with 7764 genes and 12 samples## * data accessors: counts, samplesdmDSdata object only contains genes that have more that one isoform, which makes sense as we are testing for differential transcript usage. We can find out how many of the remaining genes haveN isoforms by tabulating the number of times we see a gene ID, then tabulating the output again:Thetable(table(counts(d)$gene_id))####    2    3    4    5    6    7## 4062 2514  931  222   34    1samples. Here we use a simple design with just two groups, but more complex designs are possible. For some discussion of complex designs, one can refer to the vignettes of thelimma,edgeR, orDESeq2 packages.We create a design matrix, using a design formula and the sample information contained in the object, accessed viadesign_full <- model.matrix)colnames(design_full)## [1] "(Intercept)" "condition2"Only for speeding up running the live code chunks in this workflow, we subset to the first 250 genes, representing about one thirtieth of the dataset. This step would not be run in a typical workflow.d <- d7764 / 250## [1] 31.056precision, which is related to the dispersion in the Dirichlet Multinomial model via the formula below. Because precision is in the denominator of the right hand side of the equation, they are inversely related. Higherdispersion – counts more variable around their expected value – is associated with lowerprecision. For full details about theDRIMSeq model, one should read both the detailed software vignette and the publication11. After estimating the precision, we fit regression coefficients and perform null hypothesis testing on the coefficient of interest. Because we have a simple two-group model, we test the coefficient associated with the difference between condition 2 and condition 1, calledcondition2. The following code takes about half a minute, and so a full analysis on this dataset takes about 15 minutes on a laptop.We then use the following three functions to estimate the model parameters and test for DTU. We first estimate theset.seed(1)system.time  d <- dmFit  d <- dmTest})## ! Using a subset of 0.1 genes to estimate common precision !## ! Using common_precision = 21.2862 as prec_init !## ! Using 0 as a shrinkage factor !##    user  system elapsed##  34.213   0.450  35.846results function. We can generate a single p-value per gene, which tests whether there is any differential transcript usage within the gene, or a single p-value per transcript, which tests whether the proportions for this transcript changed within the gene:To build a results table, we run theres <- DRIMSeq::results(d)head(res)##              gene_id        lr df       pvalue   adj_pvalue## 1 ENSG00000000457.13  1.493561  4 8.277814e-01 9.120246e-01## 2 ENSG00000000460.16  1.068294  3 7.847330e-01 9.101892e-01## 3 ENSG00000000938.12  4.366806  2 1.126575e-01 2.750169e-01## 4 ENSG00000001084.11  1.630085  3 6.525877e-01 8.643316e-01## 5 ENSG00000001167.14 28.402587  1 9.853354e-08 5.007113e-07## 6 ENSG00000001461.16  9.815460  1 1.730510e-03 6.732766e-03res.txp <- DRIMSeq::resultshead(res.txp)##              gene_id         feature_id         lr df    pvalue adj_pvalue## 1 ENSG00000000457.13 ENST00000367771.10 0.16587607  1 0.6838032  0.9171007## 2 ENSG00000000457.13  ENST00000367770.5 0.01666448  1 0.8972856  0.9788571## 3 ENSG00000000457.13  ENST00000367772.8 1.02668495  1 0.3109386  0.6667146## 4 ENSG00000000457.13  ENST00000423670.1 0.06046507  1 0.8057624  0.9323782## 5 ENSG00000000457.13  ENST00000470238.1 0.28905766  1 0.5908250  0.8713427## 6 ENSG00000000460.16  ENST00000496973.5 0.83415788  1 0.3610730  0.7232298pvalue column may containNA values, we use the following function to turn these into 1’s. TheNA values would otherwise cause problems for the stage-wise analysis. From investigating theseNA p-value cases forDRIMSeq, they all occur when one condition group has all zero counts for a transcript, but sufficient counts from the other condition group, and sufficient counts for the gene.DRIMSeq will not estimate a precision for such a gene. These all happen to be true positive genes for DTU in the simulation, where the isoform switch is total or nearly total.DEXSeq, shown in a later section, does not produceNA p-values for these genes. A potential fix would be to use a plug-in common or trended precision for such genes, but this is not implemented in the current version ofDRIMSeq.Because theno.na <- function(x) ifelse(is.na(x), 1, x)res$pvalue <- no.nares.txp$pvalue <- no.naWe can plot the estimated proportions for one of the significant genes, where we can see evidence of switching .idx <- which[1]res##              gene_id       lr df       pvalue    adj_pvalue## 5 ENSG00000001167.14 28.40259  1 9.853354e-08  5.007113e-07plotProportionsDRIMSeq functions on the entire dataset.Because we have been working with only a subset of the data, we now load the results tables that would have been generated by runningdata(drim_tables)nrow(res)## [1] 7764nrow(res.txp)## [1] 20711stageR package is designed to allow for such two-stage testing procedures, where the first stage is called ascreening stage and the second stage aconfirmation stage12. The methods are general, and can also be applied to testing, for example, changes across a time series followed by investigation of individual time points, as shown in thestageR package vignette. We show below howstageR is used to detect DTU and how to interpret its output.A typical analysis of differential transcript usage would involve asking first: “which genes contain any evidence of DTU?”, and secondly, “which transcripts in the genes that contain some evidence may be participating in the DTU?” Note that a gene may pass the first stage without exhibiting enough evidence to identify one or more transcripts that are participating in the DTU. ThestageR package will combine transcript and gene names, we need to strip the gene and transcript version numbers from their Ensembl IDs (this is done by keeping only the first 15 characters of the gene and transcript IDs).We first construct a vector of p-values for the screening stage. Because of how thepScreen <- res$pvaluestrp <- function(x) substrnames(pScreen) <- strp(res$gene_id)We construct a one column matrix of the confirmation p-values:pConfirmation <- matrixrownames(pConfirmation) <- strp(res.txp$feature_id)data.frame with the transcript and gene identifiers.We arrange a two columntx2gene <- res.txpfor (i in 1:2) tx2gene <- strpstageR analysis. We must specify analpha, which will be theoverall false discovery rate target for the analysis, defined below. Unlike typical adjusted p-values or q-values, we cannot choose an arbitrary threshold later: after specifyingalpha=0.05, we need to use 5% as the target in downstream steps. There are also convenience functionsgetSignificantGenes andgetSignificantTx, which are demonstrated in thestageR vignette.The following functions then perform thelibrary(stageR)stageRObj <- stageRTxstageRObj <- stageWiseAdjustment suppressWarnings})head(drim.padj)##            geneID            txID         gene transcript## 1 ENSG00000001167 ENST00000341376 1.446731e-05   0.000000## 2 ENSG00000001167 ENST00000353205 1.446731e-05   0.000000## 3 ENSG00000001461 ENST00000003912 8.263160e-03   0.000000## 4 ENSG00000001461 ENST00000339255 8.263160e-03   0.000000## 5 ENSG00000001631 ENST00000394507 1.287012e-04   0.060474## 6 ENSG00000001631 ENST00000475770 1.287012e-04   1.000000screened genes. The transcripts with values in the column,transcript, less than 0.05 pass theconfirmation stage on a target 5%overall false discovery rate, or OFDR. This means that, in expectation, no more than 5% of the genes that pass screening will either (1) not contain any DTU, so be falsely screened genes, or (2) contain a falsely confirmed transcript. A falsely confirmed transcript is a transcript with an adjusted p-value less than 0.05 which does not exhibit differential usage across conditions. ThestageR procedure allows us to look at both the genes that passed the screening stage and the transcripts with adjusted p-values less than our targetalpha, and understand what kind ofoverall error rate this procedure entails. This cannot be said for an arbitrary procedure of looking at standard gene adjusted p-values and transcript adjusted p-values, where the adjustment was performed independently.The final table with adjusted p-values summarizes the information from the two-stage analysis. Only genes that passed the filter are included in the table, so the table already representsDRIMSeq was sensitive to detect DTU, but could exceed its false discovery rate (FDR) bounds, particularly on the transcript-level tests, and that a post-hoc, non-specific filtering of theDRIMSeq transcript p-values and adjusted p-values improved the FDR and OFDR control. We considered the standard deviation (SD) of the per-sample proportions as a filtering statistic. This statistic does not use the information about which samples belong to which condition group. We set the p-values and adjusted p-values for transcripts with small per-sample proportion SD to 1. We do not recompute adjusted p-values, although we will provide the filtered p-values to thestageR procedure.We found thatmore conservative: excluding transcripts with small SD of the per-sample proportions brought both the observed FDR and the observed OFDR closer to their nominal targets, as will be shown in the evaluations below.We note that the p-values are no longer necessarily uniform after filtering out small effect size transcripts and genes, although we find that in this simulation at least, the filtering made the procedureres.txp.filt <- DRIMSeq::resultssmallProportionSD <- function {  cts <- as.matrix(subset(counts(d), select=-c))  gene.cts <- rowsum$gene_id)  total.cts <- gene.cts  props <- cts/total.cts  propSD <- sqrt(rowVars(props))  propSD < filter}filt <- smallProportionSD(d)res.txp.filt$pvalue[filt] <- 1res.txp.filt$adj_pvalue[filt] <- 1DRIMSeq modeling steps, and to avoid interfering with the modeling, we run it afterDRIMSeq. The other three filters used before have been tested by theDRIMSeq package authors, and are therefore a recommended part of an analysis before the modeling begins. We do not apply this post-hoc filter toDEXSeq in this workflow, asDEXSeq seemed to be closer to controlling its FDR and OFDR in the evaluations, after using theDRIMSeq filters recommended in this workflow.The above post-hoc filter is not part of theDEXSeq package was originally designed for detecting differential exon usage6, but can also be adapted to run on estimated transcript counts, in order to detect DTU. UsingDEXSeq on transcript counts was evaluated by Sonesonet al.33, showing the benefits in FDR control from filtering lowly expressed transcripts for a transcript-level analysis. We benchmarkedDEXSeq here, beginning with theDRIMSeq filtered object, as these filters are intuitive, they greatly speed up the analysis, and such filtering was shown to be beneficial in FDR control.TheDRIMSeq filtering procedure dramatically increase the speed ofDEXSeq, compared to running an exon-level analysis. Another advantage is that we benefit from the sophisticated bias models ofSalmon, which account for drops in coverage on alternative exons that can otherwise throw off estimates of transcript abundance30. A disadvantage over the exon-level analysis is that we must know in advance all of the possible isoforms that can be generated from a gene locus, all of which are assumed to be contained in the annotation files (FASTA and GTF).The two factors of (1) working on isoform counts rather than individual exons and (2) using theDEXSeq package and then build aDEXSeqDataSet from the data contained in thedmDStest object (the class of theDRIMSeq object changes as the results are added). The design formula of theDEXSeq-DataSet here uses the language “exon” but this should be read as “transcript” for our analysis.DEXSeq will test – after accounting for total gene expression for this sample and for the proportion of this transcript relative to the others – whether there is a condition-specific difference in the transcript proportion relative to the others.We first load thelibrary(DEXSeq)sample.data <- DRIMSeq::samples(d)count.data <- round(as.matrix(counts(d)))dxd <- DEXSeqDataSet$feature_id,                       groupID=counts(d)$gene_id)DEXSeq analysis. While we are only working on a subset of the data, the full analysis for this dataset took less than 3 minutes on a laptop.The following functions run thesystem.time({  dxd <- estimateSizeFactors(dxd)  dxd <- estimateDispersions  dxd <- testForDEU})##    user system elapsed##   7.451  0.032   7.488DRIMSeq functions). We compute a per-gene adjusted p-value, using theperGeneQValue function, which aggregates evidence from multiple tests within a gene to a single p-value for the gene and then corrects for multiple testing across genes6. Other methods for aggregative evidence from the multiple tests within genes have been discussed in a recent publication and may be substituted at this step36. Finally, we build a simple results table with the per-gene adjusted p-values.We then extract the results table, not filtering on mean counts qval <- perGeneQValue(dxr)dxr.g <- data.frame,qval)data.frame:For size consideration of the workflow R package, we reduce also the transcript-level results table to a simplecolumns <- cdxr <- as.data.framehead(dxr)##	                                          featureID	          groupID## ENSG00000000457.13:ENST00000367771.10 ENST00000367771.10    ENSG00000000457.13## ENSG00000000457.13:ENST00000367770.5	  ENST00000367770.5    ENSG00000000457.13## ENSG00000000457.13:ENST00000367772.8	  ENST00000367772.8    ENSG00000000457.13## ENSG00000000457.13:ENST00000423670.1	  ENST00000423670.1    ENSG00000000457.13## ENSG00000000457.13:ENST00000470238.1	  ENST00000470238.1    ENSG00000000457.13## ENSG00000000460.16:ENST00000496973.5	  ENST00000496973.5    ENSG00000000460.16##	                                    pvalue## ENSG00000000457.13:ENST00000367771.10 0.5620081## ENSG00000000457.13:ENST00000367770.5	 0.8399434## ENSG00000000457.13:ENST00000367772.8	 0.5675043## ENSG00000000457.13:ENST00000423670.1	 0.7032904## ENSG00000000457.13:ENST00000470238.1	 0.8476920## ENSG00000000460.16:ENST00000496973.5	 0.2108527DEXSeq functions on the entire dataset.Again, as we have been working with only a subset of the data, we now load the results tables that would have been generated by runningdata(dex_tables)stageR package has not already been loaded, we make sure to load it, and run code very similar to that used above forDRIMSeq two-stage testing, with a targetalpha=0.05.If thelibrary(stageR)strp <- function(x) substrpConfirmation <- matrixdimnames(pConfirmation) <- list(strp(dxr$featureID),"transcript")pScreen <- qvalnames(pScreen) <- strp(names(pScreen))tx2gene <- as.data.frame])for (i in 1:2) tx2gene <- strpThe following three functions provide a table with the OFDR control described above. To repeat, the set of genes passing screening should not have more than 5% of either genes which have in fact no DTU or genes which contain a transcript with an adjusted p-value less than 5% which do not participate in DTU.stageRObj <- stageRTxstageRObj <- stageWiseAdjustmentsuppressWarnings})head(dex.padj)##            geneID            txID         gene transcript## 1 ENSG00000001167 ENST00000341376 1.379695e-05         0## 2 ENSG00000001167 ENST00000353205 1.379695e-05          0## 3 ENSG00000001461 ENST00000003912 1.011322e-03          0## 4 ENSG00000001461 ENST00000339255 1.011322e-03          0## 5 ENSG00000001630 ENST00000003100 4.979296e-03         0## 6 ENSG00000001630 ENST00000450723 4.979296e-03          0DRIMSeq11,DEXSeq6,stageR12,tximport16, orSalmon17 in published research, please cite the relevant methods publications, which can be found in the References section of this workflow.This concludes the DTU section of the workflow. If you usetximport and summarize counts to the gene level and compute an average transcript length offset for count-based methods16. We will then show code for usingDESeq2 andedgeR to assess differential gene expression. Because we have simulated the genes according to three different categories, we can color the final plot by the true simulated state of the genes. We note that we will pairDEXSeq withDESeq2 results in the following plot, andDRIMSeq withedgeR results. However, this pairing is arbitrary, and any DTU method can reasonably be paired with any DGE method.In the final section of the workflow containing live code examples, we demonstrate how differential transcript usage, summarized to the gene level, can be visualized with respect to differential gene expression analysis results. We usetxi.g which contains the gene-level counts, abundances and average transcript lengths.The following line of code is unevaluated, but was used to generate an objecttxi.g <- tximporttxi.g object which is saved in a filesalmon_gene_txi.rda. We then load theDESeq2 package and build aDESeqDataSet fromtxi.g, providing also the sample information and a design formula.For the workflow, we load thedatalibrary(DESeq2)dds <- DESeqDataSetFromTximport## using counts and average transcript lengths from tximportDESeq2 analysis20.The following two lines of code run thedds <- DESeq(dds)dres <- DESeq2::results(dds)Because we happen to know the true status of each of the genes, we can make a scatterplot of the results, coloring the genes by their status .all(dxr.g$gene %in% rownames(dres))## [1] TRUEdres <- dres# we can only color because we simulated...col <- rep)col[rownames(dres) %in% dge.genes] <- 1col[rownames(dres) %in% dte.genes] <- 2col[rownames(dres) %in% dtu.genes] <- 3DEXSeq outputs an adjusted p-value of 0 for some of the genes, we set these instead to a jittered value around 10−20, so that their number and location on the x-axis could be visualized. These jittered values should only be used for visualization.bigpar# here cap the smallest DESeq2 adj p-valuecap.padj <- pmin(-log10(dres$padj), 100)# this vector only used for plottingjitter.padj <- -log10jp.idx <- jitter.padj == 20jitter.padj[jp.idx] <- rnorm(sum(jp.idx),20,.25)plotlegend,        col=c, pch=20, bty="n")        edgeR as the inference engine7. The following code incorporates the average transcript length matrix as an offset for anedgeR analysis.We can also perform differential gene expression analysis usinglibrary(edgeR)cts.g <- txi.g$countsnormMat <- txi.g$lengthnormMat <- normMat / exp(rowMeans(log(normMat)))o <- log) + log(colSums(cts.g/normMat))y <- DGEList(cts.g)y <- scaleOffset) + o))keep <- filterByExpr(y)y <- yedgeR model fitting and results extraction can be accomplished with the following lines:The basicy <- estimateDispfit <- glmFitlrt <- glmLRT(fit)tt <- topTags, sort="none")[[1]]Again, we can color the genes by their true status in the simulation:common <- intersect)tt <- ttres.sub <- res# we can only color because we simulated...col <- rep)col[rownames(tt) %in% dge.genes] <- 1col[rownames(tt) %in% dte.genes] <- 2col[rownames(tt) %in% dtu.genes] <- 3DRIMSeq andedgeR. One obvious contrast withDRIMSeq outputs lower non-zero adjusted p-values thanDEXSeq does, whereDEXSeq instead outputs 0 for many genes. The plots look more similar when zooming in on theDRIMSeq y-axis, as can be seen in the right panel ofbigparplot(-log10(tt$FDR), -log10, col=col,     xlab="Gene expression",     ylab="Transcript usage")legend,        col=c, pch=20, bty="n")bigparplot(-log10(tt$FDR), -log10, col=col,     xlab="Gene expression",     ylab="Transcript usage", ylim=c)legend,        col=c, pch=20, bty="n")workflow section of the article. The following sections provide anevaluation of the methods presented in this workflow for DTU and DGE, alongside evaluation of other popular methods for DTU and for DGE. We additionally provide an evaluation of popular methods for DTE. While the workflow does not contain any code for performing DTE, we felt it was valuable to include an evaluation at this level of analysis as well. In practice, for count-based methods such asDESeq2 andedgeR, performing DTE uses the same code as for DGE, but the counts are provided at the transcript level rather than summarized to the gene level. All of the analysis code used in the evaluations is provided in the associated GitHub repository25.This marks the end of theDRIMSeq andDEXSeq for DTU,DESeq2 andedgeR for DGE, relative to other popular methods for DTU and DGE. It is useful to assess the performance of methods for DGE in a simulation which also includes DTU – to see whether there is potentially an enrichment of false positives for certain types of genes according to the simulation. We also considered the question of DTE, and evaluated a number of methods designed for DGE – as well as methods designed for either DGE or DTE – by testing at the transcript level. DTE is not one of the analyses included in the workflow, but it is straightforward to perform with many of the DGE methods as well as with the methods explicitly designed to perform DGE or DTE.We investigated the performance of the Bioconductor packages used in the workflow above,As in the last plots presented in the workflow, in the evaluation we categorized genes by their simulation type, using the terms “DGE”, “DTE”, and “DTU”. When referring to the gene types in the simulation: these refer to the 10% of genes wherein all expressed transcripts had a constant fold change across condition (DGE), the 10% of genes where a single expressed transcript had a fold change across condition (DTE), and the 10% of genes where two transcripts had their expression switched across condition (DTU). Thus, the DTU genes counted as false positives for the DGE analysis, and vice versa. The DTE genes counted as true positives for the DGE analysis , and counted as true positives for DTU analysis if there were other expressed transcripts in the gene, or a false positive for DTU analysis if there were no other expressed transcripts (and so the proportions did not change).iCOBRA package37 was used to construct plots to assess the true positive rate (TPR) over the false discovery rate (FDR) at three nominal FDR thresholds: 1%, 5%, and 10%. We additionally used bar plots to show the number of false positives for each method across simulated gene-type categories (these plots referred to here asbreakdown plots). We can do this at both the gene and transcript level: a false positive transcript can be categorized according to the type of gene to which it belongs. Finally, we created anOFDR plot for assessing the use ofstageR for constructing gene-transcript OFDR sets, after applyingstageR to the output of the DTU detection methods. The OFDR plot displays the observed OFDR on the x-axis and the sensitivity in recovering DTU transcripts on the y-axis. We used a fixed target OFDR for these plots of 5%. The code for evaluating all methods and constructing theiCOBRA plots is included in the simulation repository25.We used three types of plots to explore the results. For assessing overall method performance for DTU, DGE, and DTE analysis, theSUPPA238 andRATs39, both of which can takeSalmon quantifications as input. For statistical testing of DTU,SUPPA2 computes, for a given transcript, the difference in proportion across condition and the differences in proportion seen between biological replicates.SUPPA2 then compares the difference in proportion across condition to the distribution of between-replicate differences for transcripts with similar average abundance by TPM. The transcript p-value is the tail probability from this empirical distribution, divided by two.SUPPA2 is implemented as a command-line software package written in python, with a number of distinct features, including the ability to translate fromSalmon transcript-level quantifications to individual splicing events, which are cataloged using a specific vocabulary described in theSUPPA2 software usage guide.SUPPA2 additionally offers differential analysis on the splicing events, which may be more valuable to investigators than per-transcript results, depending on the research goals (similar to the exon-level primary use case ofDEXSeq).We assessed two other methods for DTU,RATs uses a G-test of independence40 at both the gene level and transcript level: at the gene level it compares the sets of abundances for each transcript across condition, and at the transcript level it compares the abundance of each transcript against the pooled abundance of the other transcripts in the gene, similar to the approach ofDEXSeq in detecting differential exon usage, although with a different statistical test.RATs uses gene- and transcript-level expression filters before statistical testing. Unlike the other DTU methods discussed,RATs uses the inferential replicates (bootstrap or Gibbs samples) to repeat the testing multiple times, and then calculates the fraction of inferential replicates which achieve statistical significance.RATs also repeats the statistical testing multiple times using subsets of samples as a secondary assessment of reproducibility. TheRATs software version we used additionally performs a filter on effect size, such that only genes or transcripts which were both reproducible according to inferential replicates and sub-sampling, and having a sufficiently large effect size are reported as DTU.RATs is implemented as an R package designed to detect DTU from transcript quantification as produced bySalmon orkallisto19. As mentioned above, it can operate either on estimated counts alone, or on inferential replicates of the counts (bootstrap or Gibbs samples) as generated by either of these quantification tools. It is recommended in theRATs software guide to use a counts-from-abundance approach to generate the transcript counts.SUPPA2 in its differential transcript usage mode. We enabled a filter to remove transcripts with less than 1 TPM. TPM filtering is a command-line option available during thediffSplice step ofSUPPA2 and this greatly improved the running time without loss of sensitivity . We did not use theSUPPA2 optional gene-correction, which does not correct for false discovery rate across genes, as we wanted to apply the aggregation and correction methodperGeneQValue fromDEXSeq to obtain an FDR-bounded set of genes and transcripts as output. We ranRATs with 30 bootstrap replicates fromSalmon, generating counts from abundance by scaling up TPMs. The bootstrap replicates approach performed similarly to the approach without bootstrap replicates, with a minor improvement in the FDR and OFDR with including the bootstrap replicates. For easier visualization and to avoid overlapping data points, we only include theRATs bootstrap results in the evaluation plots.We ranDRIMSeq filters for minimum gene and transcript counts and minimum proportion. This eliminated genes which had expression too low to have very much statistical power for detecting DTU, and transcripts which were very lowly expressed in both conditions, and so not contributing useful information for DTU. We assessed that excluding these lowly expressed genes and transcripts did not change the relative differences in sensitivity of the methods, as they were not detectable by any of the methods with regards to DTU. ForSUPPA2, we performedperGeneQValue only on those genes and transcripts passing theDRIMSeq filters. ForRATs, we provided the bootstrap replicate counts-from-abundance for the genes and transcripts that passed theDRIMSeq filters. We performed identical stage-wise analysis withstageR onSUPPA2 andRATs output, to allow direct comparison withDRIMSeq andDEXSeq stage-wise results and observed OFDR. Exact code for runningSUPPA2 andRATs is provided in the respective directories in the associated GitHub repository25.To facilitate comparisons across methods, we only considered the genes and transcripts passing theDRIMSeq,DEXSeq,RATs, andSUPPA2 at per-group sample sizes of 3, 6, 9, and 12. The following evaluation corresponds to the “main” simulation as described in the Methods.In the workflow, we showed a typical analysis for a comparison of 6 vs 6 samples. As we were interested in the performance at various sample sizes, we performed the entire analysis forSUPPA2 always controlled its FDR,RATs controlled its FDR except for the 1% threshold for sample size 3,DEXSeq controlled its FDR except for the 1% threshold at all sample sizes and 5% threshold for sample size 3, andDRIMSeq exceeded its FDR but approached the target for larger sample sizes . The true positives for most methods matched the gene-type proportion of true genes with transcript usage changes . About tchanges . The metstageR using gene- and transcript-level p-values fromDRIMSeq,DEXSeq,RATs, andSUPPA2, for a 5% target OFDR.SUPPA2 andRATs controlled the target OFDR at all sample sizes, withRATs having nearly exactly 5% OFDR at the smallest sample size.DEXSeq input tostageR was close to the 5% OFDR target except for a sample size of 3, which had an OFDR around 10%. ForDRIMSeq, we assessed whether raising the p-values to 1 for transcripts with small proportion SD helped to recover OFDR control. The observed OFDR forDRIMSeq with proportion SD filtering was at lowest around 12% at per-group sample size of 6 and higher  procedure implemented with higher . WithoutstageR procedure, we also evaluated the transcript-level adjusted p-values alone forDRIMSeq,DEXSeq,RATS, andSUPPA2. This evaluation corresponds to an analysis which does not use any gene-level aggregation, and does not usestageR, but considers only the adjusted p-values per transcript from each method. Here we computed the standard FDR, where the unit of false discovery is thetranscript, in contrast to the OFDR where the unit of false discovery is thegene.SUPPA2 andRATs tended to control their FDR as in the gene-level analysis . ForDRIMSeq andDEXSeq, we include the compute time of the estimation steps (importing counts withtximport and filtering takes only a few seconds).InDRIMSeq andDEXSeq, we generated an additional simulation we called “fixed per-gene dispersion” in which genes were assigned Negative Binomial dispersion parameters by matching the gene-level count to the joint distribution of mean and dispersions on the GEUVADIS dataset. Then transcript-level counts were generated with all transcripts of a gene being assigned the same Negative Binomial dispersion parameter. This contrasts with the “main” simulation, in which each transcript was assigned its own dispersion parameter, resulting in heterogeneity of dispersion within a gene. As we do not know the degree to which transcripts of a gene would have correlated biological variability in an experimental dataset, we also include the results for the count-based methods that estimate precision/dispersion,DRIMSeq andDEXSeq, for this additional simulation.In order to further investigate performance differences between the two methods highlighted in the workflow section,DRIMSeq, which estimates a single precision parameter per gene, performed better in terms of FDR control on this simulation at the gene level 42,EBSeq43, andDESeq220, use a Negative Binomial distribution to model the counts, and empirical Bayes techniques to estimate per-gene parameters despite limited sample size. The Negative Binomial is a useful distribution for counts, in that it has a parameter for the location of the mean count, as well as adispersion parameter for the expected spread of counts. For high counts, the dispersion parameter is approximately equal to the square of the coefficient of variation (the standard deviation over the mean), and so can be interpreted for high counts as how much the data can be expected to vary around the mean count, relative to the size of the mean.A number of the methods,EBSeq, uniquely among these Negative-Binomial-based models, was also specifically designed to accommodate extra uncertainty in transcript counts when assessing DTE.EBSeq has a DTE mode in which the number of transcript isoforms per gene is supplied as a piece of information before running the main analysis function.edgeR-QL differs fromedgeR andDESeq2 in that it accounts for uncertainty in the dispersion estimate via a quasi-likelihood framework.limma withvoom transformation10 andsleuth44 model the log of scaled counts, withsleuth additionally taking into account inferential variance on the transcript- and gene-level counts, unlike any of the other DGE or DTE methods we assessed. Finally,SAMseq45 scales counts via a resampling approach and applies rank-based statistical tests to detect differences in samples across condition; by operating on ranks, it is much less sensitive to count outliers or in general to mis-specified parametric modeling.filterByExpr foredgeR,edgeR-QL, andlimma withvoom,sleuth_prep forsleuth, and a custom filter requiring a count of 10 or more for half the samples forDESeq2,EBSeq, andSAMSeq, which do not come with their own filtering functions. For evaluation, all genes  were included, except those for which no software provided an adjusted p-value.For DGE and DTE, the following filtering functions or rules for each package were used:We assessed the aforementioned R packages for differential gene expression, to determine true positive rate and control of false discovery rate on the “main” simulated dataset. In this analysis, the simulated “DGE” genes , and the “DTE” genes  should count as true positives for differential gene expression, while the simulated “DTU” genes should count as false positives for differential gene expression, as the total expression of the gene remains constant.DESeq2,EBSeq,edgeR,limma withvoom transformation,SAMseq, andsleuth. We usedtximport to summarizeSalmon abundances to the gene level, and provided all methods other thanDESeq2 andsleuth with thelengthScaledTPM count matrix.sleuth takes as input the quantification fromkallisto19, which was run with 30 bootstrap samples and bias correction. For gene-level analysis insleuth, the argumentaggregation_column="gene_id" was used. AsDESeq2 has specially designed import functions for taking in estimated gene counts and an offset fromtximport, we used this approach to provideSalmon summarized gene-level counts and an offset.edgeR andedgeR-QL had the same performance using the counts and offset approach or thelengthScaledTPM approach, so we used the latter for code simplicity. The exact code used to run the different methods can be found at the simulation code repository25. Compute time for the different gene-level methods are presented inWe comparediCOBRA plots with true positive rate over false discovery rate for gene-level analysis across four different pergroup sample sizes are presented inDESeq2 andEBSeq tended to control the FDR, while those two methods had, for example, 15% and 18% FDR respectively at the nominal 10% rate.SAMseq, with so few samples, did not have adequate sensitivity to detect DGE. At the per-group sample size of 6, all methods exceptDESeq2 andSAMseq tended to control the FDR. At this sample size,EBSeq controlled its FDR. For the largest per-group sample sizes, 9 and 12, the performance of many methods remained similar as previously, exceptsleuth did not control its nominal FDR. For ease of comparison, we also providesleuth at higher sample sizes was related to the realistic GC bias parameters used in the simulation, but simulating fragments uniformly from the transcripts revealed the same performance at per-group sample sizes 9 and 12 , from this simulation study we would recommend per-group sample sizes larger than 3 to achieve greater than 50% sensitivity for detecting DTU. The maximal gene-level DTU sensitivity achieved in the “main” simulation was around 80% at a per-group sample size of 12. This reflects the fact that for some of the DTU genes, the change in proportions across conditions was small, and was not detectable relative to the within-condition biological variability. As the “main” simulation usedstageR in DTU analysis for its use of a formal statistical procedure involving a screening and confirmation stage, as this fits closely to what we expect a typical analysis to entail. It is likely that an investigator would want both a list of statistically significant genesand transcripts participating in DTU, andstageR provides error control on this pair of lists, assuming that the underlying tests are well calibrated.We recommend the use ofedgeR had better control of FDR thanDESeq2.DESeq2 approached its target FDR as sample size grew. Popular methods that had relatively high sensitivity and control of FDR across all sample sizes includelimma withvoom transformation andedgeR-QL, both of which had better control thanedgeR at per-group sample size of 3.From the DGE and DTE analyses of this particular simulation data, we found thatDEXSeq analysis,SUPPA2, orLeafCutter46, here we considered and detected expression switching between annotated transcripts. Other methods such asDEXSeq (exon-based),SUPPA2, orLeafCutter may benefit in terms of power and interpretability from performing statistical analysis directly on exon usage or splice events. Methods such asDEXSeq (exon-based) andLeafCutter benefit in the ability to detect un-annotated events. The workflow presented here would require further processing to attribute transcript usage changes to specific splice events, and is limited to considering the estimated abundance of annotated transcripts.One potential limitation of this workflow is that, in contrast to other methods such as the standardThe following provides the session information used when compiling this document.devtools::session_info## Session info -------------------------------------------------------------##  setting  value##  version  R version 3.5.0 (2018-04-23)##  system   x86_64, darwin15.6.0##  ui       X11##  language (EN)##  collate  en_US.UTF-8##  tz       America/New_York##  date     2018-06-17## Packages -----------------------------------------------------------------##  package              * version   date       source##  acepack                1.4.1     2016-10-29 CRAN (R 3.5.0)##  annotate               1.58.0    2018-05-01 Bioconductor##  AnnotationDbi        * 1.42.1    2018-05-08 Bioconductor##  assertthat             0.2.0     2017-04-11 CRAN (R 3.5.0)##  backports              1.1.2     2017-12-13 cran (@1.1.2)##  base                 * 3.5.0     2018-04-24 local##  base64enc              0.1-3     2015-07-28 CRAN (R 3.5.0)##  Biobase              * 2.40.0    2018-05-01 Bioconductor##  BiocGenerics         * 0.26.0    2018-05-01 Bioconductor##  BiocInstaller        * 1.30.0    2018-05-04 Bioconductor##  BiocParallel         * 1.14.1    2018-05-06 Bioconductor##  BiocStyle              2.8.0     2018-05-01 Bioconductor##  BiocWorkflowTools      1.6.1     2018-05-24 Bioconductor##  biomaRt                2.36.0    2018-05-01 Bioconductor##  Biostrings             2.48.0    2018-05-01 Bioconductor##  bit                    1.1-12    2014-04-09 CRAN (R 3.5.0)##  bit64                  0.9-7     2017-05-08 CRAN (R 3.5.0)##  bitops                 1.0-6     2013-08-17 CRAN (R 3.5.0)##  blob                   1.1.1     2018-03-25 CRAN (R 3.5.0)##  bookdown               0.7       2018-02-18 CRAN (R 3.5.0)##  checkmate              1.8.5     2017-10-24 CRAN (R 3.5.0)##  cluster                2.0.7-1   2018-04-13 CRAN (R 3.5.0)##  codetools              0.2-15    2016-10-05 CRAN (R 3.5.0)##  colorspace             1.3-2     2016-12-14 CRAN (R 3.5.0)##  compiler               3.5.0     2018-04-24 local##  data.table             1.11.2    2018-05-08 CRAN (R 3.5.0)##  datasets             * 3.5.0     2018-04-24 local##  DBI                    1.0.0     2018-05-02 CRAN (R 3.5.0)##  DelayedArray         * 0.6.0     2018-05-01 Bioconductor##  DESeq2               * 1.20.0    2018-05-01 Bioconductor##  devtools             * 1.13.5    2018-02-18 CRAN (R 3.5.0)##  DEXSeq               * 1.26.0    2018-05-01 Bioconductor##  digest                 0.6.15    2018-01-28 cran (@0.6.15)##  DRIMSeq              * 1.8.0     2018-05-01 Bioconductor##  edgeR                * 3.22.2    2018-05-24 cran (@3.22.2)##  evaluate               0.10.1    2017-06-24 CRAN (R 3.5.0)##  foreign                0.8-70    2017-11-28 CRAN (R 3.5.0)##  Formula                1.2-3     2018-05-03 CRAN (R 3.5.0)##  genefilter             1.62.0    2018-05-01 Bioconductor##  geneplotter            1.58.0    2018-05-01 Bioconductor##  GenomeInfoDb         * 1.16.0    2018-05-01 Bioconductor##  GenomeInfoDbData       1.1.0     2018-01-10 Bioconductor##  GenomicRanges        * 1.32.2    2018-05-06 Bioconductor##  ggplot2                2.2.1     2016-12-30 CRAN (R 3.5.0)##  git2r                  0.21.0    2018-01-04 CRAN (R 3.5.0)##  graphics             * 3.5.0     2018-04-24 local##  grDevices            * 3.5.0     2018-04-24 local##  grid                   3.5.0     2018-04-24 local##  gridExtra              2.3       2017-09-09 CRAN (R 3.5.0)##  gtable                 0.2.0     2016-02-26 CRAN (R 3.5.0)##  Hmisc                  4.1-1     2018-01-03 CRAN (R 3.5.0)##  htmlTable              1.11.2    2018-01-20 CRAN (R 3.5.0)##  htmltools              0.3.6     2017-04-28 CRAN (R 3.5.0)##  htmlwidgets            1.2       2018-04-19 CRAN (R 3.5.0)##  httr                   1.3.1     2017-08-20 CRAN (R 3.5.0)##  hwriter                1.3.2     2014-09-10 CRAN (R 3.5.0)##  IRanges              * 2.14.9    2018-05-15 Bioconductor##  knitr                * 1.20      2018-02-20 CRAN (R 3.5.0)##  labeling               0.3       2014-08-23 CRAN (R 3.5.0)##  lattice                0.20-35   2017-03-25 CRAN (R 3.5.0)##  latticeExtra           0.6-28    2016-02-09 CRAN (R 3.5.0)##  lazyeval               0.2.1     2017-10-29 CRAN (R 3.5.0)##  limma                * 3.36.1    2018-05-05 Bioconductor##  locfit                 1.5-9.1   2013-04-20 CRAN (R 3.5.0)##  magrittr               1.5       2014-11-22 CRAN (R 3.5.0)##  Matrix                 1.2-14    2018-04-13 CRAN (R 3.5.0)##  matrixStats          * 0.53.1    2018-02-11 CRAN (R 3.5.0)##  memoise                1.1.0     2017-04-21 CRAN (R 3.5.0)##  methods              * 3.5.0     2018-04-24 local##  munsell                0.4.3     2016-02-13 CRAN (R 3.5.0)##  nnet                   7.3-12    2016-02-02 CRAN (R 3.5.0)##  parallel             * 3.5.0     2018-04-24 local##  pillar                 1.2.2     2018-04-26 CRAN (R 3.5.0)##  plyr                   1.8.4     2016-06-08 CRAN (R 3.5.0)##  prettyunits            1.0.2     2015-07-13 CRAN (R 3.5.0)##  progress               1.1.2     2016-12-14 CRAN (R 3.5.0)##  R6                     2.2.2     2017-06-17 CRAN (R 3.5.0)##  rafalib              * 1.0.0     2015-08-09 CRAN (R 3.5.0)##  RColorBrewer         * 1.1-2     2014-12-07 CRAN (R 3.5.0)##  Rcpp                   0.12.17   2018-05-18 cran (@0.12.17)##  RCurl                  1.95-4.10 2018-01-04 CRAN (R 3.5.0)##  reshape2               1.4.3     2017-12-11 CRAN (R 3.5.0)##  rlang                  0.2.1     2018-05-30 cran (@0.2.1)##  rmarkdown            * 1.9       2018-03-01 CRAN (R 3.5.0)##  rnaseqDTU            * 0.1.0     2018-06-18 local (mikelove/rnaseqDTU@NA)##  rpart                  4.1-13    2018-02-23 CRAN (R 3.5.0)##  rprojroot              1.3-2     2018-01-03 cran (@1.3-2)##  Rsamtools              1.32.0    2018-05-01 Bioconductor##  RSQLite                2.1.1     2018-05-06 CRAN (R 3.5.0)##  rstudioapi             0.7       2017-09-07 CRAN (R 3.5.0)##  S4Vectors            * 0.18.1    2018-05-02 Bioconductor##  scales                 0.5.0     2017-08-24 CRAN (R 3.5.0)##  splines                3.5.0     2018-04-24 local##  stageR               * 1.2.22    2018-06-14 cran (@1.2.22)##  statmod                1.4.30    2017-06-18 CRAN (R 3.5.0)##  stats                * 3.5.0     2018-04-24 local##  stats4               * 3.5.0     2018-04-24 local##  stringi                1.2.2     2018-05-02 CRAN (R 3.5.0)##  stringr                1.3.1     2018-05-10 CRAN (R 3.5.0)##  SummarizedExperiment * 1.10.1    2018-05-11 Bioconductor##  survival               2.42-3    2018-04-16 CRAN (R 3.5.0)##  tibble                 1.4.2     2018-01-22 CRAN (R 3.5.0)##  tinytex                0.5       2018-04-16 CRAN (R 3.5.0)##  tools                  3.5.0     2018-04-24 local##  utils                * 3.5.0     2018-04-24 local##  withr                  2.1.2     2018-03-15 CRAN (R 3.5.0)##  xfun                   0.1       2018-01-22 CRAN (R 3.5.0)##  XML                    3.98-1.11 2018-04-16 CRAN (R 3.5.0)##  xtable                 1.8-2     2016-02-05 CRAN (R 3.5.0)##  XVector                0.20.0    2018-05-01 Bioconductor##  yaml                   2.1.19    2018-05-01 CRAN (R 3.5.0)##  zlibbioc               1.26.0    2018-05-01 BioconductorDRIMSeq - 1.8.0,DEXSeq - 1.26.0,stageR - 1.2.21,tximport - 1.8.0,DESeq2 - 1.20.0,EBSeq - 1.20.0,edgeR - 3.22.2,limma - 3.36.1,RATs - 0.6.4,samr - 2.0,sleuth - 0.29.0,SUPPA2 - 2.3. The samples were quantified withSalmon version 0.10.0 andkallisto version 0.44.0.polyester version 1.16.0 andalpine version 1.6.0 were used in generating the simulated dataset.The statistical methods were evaluated using the following software versions:The simulated paired-end read FASTQ files have been uploaded in three batches of eight samples each to Zenodo-https://doi.org/10.5281/zenodo.129137526https://doi.org/10.5281/zenodo.129140427https://doi.org/10.5281/zenodo.12914432810.5281/zenodo.129152229.The quantification files are also available as a separate Zenodo dataset:https://github.com/mikelove/swimdown) and archived here-https://doi.org/10.5281/zenodo.141044325.The scripts used to generate the simulated dataset are available at the simulation GitHub repository .The counts associated with SonesonAll data is available under a CC BY 4.0 license.1. All software used in1 this workflow is available as part of Bioconductor version 3.7.2. https://github.com/mikelove/rnaseqDTUSource code for the workflow:3. https://doi.org/10.5281/zenodo.141044248Link to archived source code as at time of publication:4. License: Artistic-2.0 Although we still believe some of the suggestions from our last report would make the article appeal to a larger audience we agree with the authors that they are not necessary as the changes in the last update does take care of our major concern.We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The authors have made a great improvements to the article along the lines suggested by the referees.  I still think there is a considerable amount of work that could be added to expand the study, as I suggested in my original report. I understand, however, that the authors feel that a more expansive exploration of the tool parameter space and the addition of  assessment on real data is beyond the scope of the article. I am of the opinion that, with these suggestions in hand, they are best positioned to define the scope of their own work. In my opinion the improvements they have made round out the article nicely and certainly make it acceptable for publication and indexing. Minor comment: 1) Introduction, line 1: This sentence is awkward. I think total gene expression is self explanatory, but if the authors feel it requires more explanation perhaps they could make the sentense clearer - something like: "RNA-seq experiments across groups of samples can be analyzed to detect differences in the total expression of all isoforms of a gene and, additionally, differences in transcript isoform usage within a gene." P.S. I particularly like the use of the hex 'stickers' for the tools in the workflow diagram - nice.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. We feel the authors have done an excellent job in improving the structure and clarity of the paper. We believe the improved introduction and workflow diagram will be particularly helpful to readers and we accept the paper for indexing. A few very minor points remain:p18, could the authors please rewrite this sentence for clarity: "To repeat, the set of genes passing screening should not have more than 5% of either genes which have in fact no DTU or genes which contain a transcript with an adjusted p-value less than 5% which do not participate in DTU."p32, second paragraph: could the authors hypothesise why there was a slight increase of false positives for DTU genes in the DGE analysis?p34, fourth paragraph: mentions of SUPPA2, DEXseq and leafcutter are repeated across two sentences in the same context.  We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Summaryet al presents a combined workflow and benchmark for differential transcript usage. This is a vital paper as there is no consensus on which differential transcript usage tools works better (here addressed by the benchmark part) and very few people are aware of the feasibility of analysis of differential transcript usage – something the workflow can hopefully help with. Of special note is the extent to which open source have been embraced by Loveet al – an approach that is commendable (and worthy repeating). In “Swimming downstream: statistical analysis of differential transcript usage following Salmon quantification” Love In the revised version (version 2) the workflow and benchmark have been separated and the workflow has been simplified which, together with the overview figure and the general improvements makes the article much more readable. With regards to analysis especially the benchmark session have been extended. In summary, the revised manuscript version address the majority of our concerns but we still believe the introduction and benchmark could be improved, as detailed below.Commentshttps://github.com/mikelove/rnaseqDTU) but seem to accidentally have forgotten to correct this in the workflow part of the article (page 17)The authors have updated the benchmark to use testForDEU instead of nbinomLRT for the DEXSeq analysis . A more detailed introduction here would also make the concepts behind StageR clearer to readers not familiar with the subject.The introduction lacks:et al which instead illustrates the DTU problem.It could be even further highlighted that the difference between “scaledTPM” and the other scaling methods is whether the issue with transcript lengths are normalized away. This could also be highlighted by making a figure like Fig1 in TrapnellWe further recommend that the documentation for tximport with regards to scaling is updated with the descriptions made here.The scaling section (page 9)The section regarding the theory behind DRIMSeq (bottom of page 12) (could be with the rest of the theory regarding DRIMSeq (page 7)The section regarding StageR (page 14)The article would benefit from a one or two sentence layman introduction to all the tools (for people not in the field).The workflow/benchmark still contains large chunks of text that belongs in the introduction/methods. Specifically we are thinking of:The DTU benchmark should also include an analysis on unmodified simulated data to test how many false positives are found if there truly are no DTU (which might be the case for some datasets). We suggest comparing samples internally in either set 1 or set 2.To reflect a very common use case scenario the DGE / DTE benchmark should also be performed with 2 replicates.All iCOBRA plots would benefit from zooming in on the y-axis to the min and max of any tool across all samples . This is especially problematic for the DGE/DTE benchmark.The section on page 30 describing the results in figure 13 is very hard to understand.Given the success of repurposing DEXSeq to DTU, the good performance of limma/edgeR for DTE/DGE and the recent incorporation of RATS, the current benchmark could also test a repurposing of limma’s (and edgeR’s) differential exon usage test (diffsplice/diffSpliceDGE). This is optional – but it would be a huge step forward for testing differential isoform usage as it would bring a lot of clarity to the field.We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. Thank you for reviewing our revised version. Yes, unfortunately, while we updated the results in version 2 to use testForDEU and the code in the Bioconductor git repository, we neglected to update the displayed code chunk with nbinomLRT to testForDEU. A version 3 is already in production which will show the correct code. Regarding the placement of the text in various sections, an F1000Research workflow article does not have exactly the same structure as a typical research article, and we feel our current structure is appropriate and fits with the pattern of other Bioconductor workflow article published in F1000Research. In particular, we feel it is appropriate to introduce concepts as they arise in the natural flow of the workflow, rather than all in the Introduction or in the terse Methods section. In version 2, we feel we have included sufficient background and citations to research and review articles about the importance of detecting DTU in the Introduction (see the first paragraph of the Introduction). For full details on the theory of DRIMSeq, we have instead recommended that readers go to the primary source. We will not be adding additional simulations or evaluations to the workflow in the near future, as it already contains much more evaluations and simulation than the comparable workflows published in the Bioconductor channel of F1000Research. In 'Swimming downstream: statistical analysis of differential transcript usage following Salmon quantification' Love, Sonesson & Patro present both 1) a workflow for identifying the signatures of differential transcript usage between RNA-seq samples in two conditions, based on a suite of tools, and 2) a benchmarking analysis of the performance of these tools based on simulated data. The aims of this work are laudable and I have no doubt it will be a valuable addition to the literature, the resulting paper suffers from several flaws and needs considerable additional work, in my opinion.Major comments: 1) The intermingling of the benchmarking and workflow sections of this manuscript make the text confused and difficult to read. I'd suggest that the authors either restructure the manuscript beginning with the workflow section and then following with the benchmarking section, or split the work in to two and concentrate separately on the two areas. 2) This work is listed as a Method article. I am not convinced that an example of stringing existing tools together fits the description required for this section ."). The benchmarking part of the work is better suited to a Research Article, whilst the workflow part is more like a computational protocol and might be better suited for publication as a Study Protocol. 3) Quantifying transcript expression from RNA-seq data is challenging but has become common-place and relatively straight-forward thanks to the development of high-performance tools such Salmon and Kallisto. These tools typically provide a transcripts-per-million estimation of a transcripts expression. With these quantifications in place the inevitable, and even more challenging, next step is to identify those transcripts where their expression is changing between samples. To date there has not been a clear data-driven exploration of the underlying statistical properties of TPM quantifications (or estimated transcript counts from TPMs) as a function of biological and technical replication - instead, much as was the case for differential gene expression from RNA-seq data until relatively recently - the tools for identifying DTE are built on the strong assumption of a distribution for the quantifications and, typically, assume a negative binomial distribution. Although this looks to be a good assumption in the case of gene expression, it is far from clear to me that the assumption of a negative binomial distribution for the distribution of a transcripts TPM or estimated counts across biological replicates is a good assumption for TPMs or estimated counts from TPMs, particularly given that - in the context of biological DTU - the expression of a transcript can be strongly correlated with the other child transcripts of the gene. The fixed per-gene dispersion section seems like the beginnings of an exploration in this area but this assumption too is without any justification. Perhaps the authors could use some highly replicated data from a complex eukaryote to actually measure these distributions and give clarity on whether these assumptions are valid? Or, failing that, explore the impact of different potential distributions of the tool performance?The low overall TPRs exhibited by all the tools; 25-80% for DTU, 50-80% for DGE & only 20-50% for DTE. What this means for these field and how might these be improved?The TPR/FPR performance of the tools not only as function of the sample size, but also as a function of the annotation used in the original transcript quantitations, as a function of the effect-size threshold used and as a function of the low-count-rate filtering used for each tool. These are all critical parameters in the tools performance.An expanded discussion of the extremely poor FPR performance of DRIMseq, that is largely glossed-over in the current text. Why is DRIM-seq performing so poorly? It is more or less dependant on the specific parameters used, or the details of the simulated data, than the other tools - or is it just generically over-sensitive across all the parameter space.The overlap between the sets of DTU, DGE & DTE identified by each tool, instead the authors just give us some numbers and the TPR/FPR performance metrics. Are these tools reliably identifying the same thing or are they finding wildly different sets of results? .1.The use of p-values, adjusted or not, as a threshold for subsetting these results for scientific relevance - particularly given Blume et. al 2018Some discussion of why the authors limit themselves to discussing DRIMseq, DEXSeq and SUPPA2 despite listing five additional alternative methods in the introduction. Alternatively, the authors could include these tools in their benchmarking, particularly if they decided to split the work into two papers with one of these focussing on the benchmarking.Some discussion of the impact that the development of long-read sequencing of native RNAs will have on this field, these tools, and their results in the next few years - perhaps the authors could even use some of the publically available data from the Oxford Nanopore RNA consortium (https://github.com/nanopore-wgs-consortium/NA12878/blob/master/RNA.md) to contrast the performance of this new technology with the tools they examine here for detecting DTE and DTU.2? What approaches would the authors then recommend?How do these tools cope with RNA-seq experiments with more complex designs? For example, what about if there are 7 conditions, or a time-series  The entire discussion section of the benchmarking results is essentially missing and the current discussion section of more like a brief conclusion. Points that I would like to see the authors discuss in detail include: 5) No effort has been made to test these workflows with real data with validated instances of DTU. These exist in the published literature. For a workflow description this is fine, but for the benchmarking aspect of the work I would like to see the authors use this pipeline in anger, with real data, and see what the results are and how they match up with the validated results. 6) The introduction does not motivate the importance of identifying DTU in biology. I'd like to see the introduction present the biological relevance of DTU, the relative sparsity of existing validated DTU instances, and the scope DTU has for being an explored layer of regulation for basic biological processes. 7) The only conclusion from the paper seems to be that the authors recommend the use of stageR - based largely on the fact that its two-stage model matches what the authors think a typical analysis workflow is. This conclusion may be sound advice but a) this paper does not present any compelling *evidence* that this is a typical workflow, and b) stageR is not really what this paper is about" Indeed, here stageR is used as a framework to assist with assessing the performance of the other tools. I'd like to see the authors instead draw some clear conclusions about which tools are the best to use for identifying DTU.Minor Comments: 1) The workflow section really needs some workflow diagrams to highlight the chain for each tool and where they are similar and different. 2) The plots in the paper are not as high quality as I'd expect:  - Figures need to be higher resolution   - Figures 3,5,6,8,12 & 13 are multi-panel figures with the same axes on each figure. They would benefit from being plotted with shared axes allowing the performance between different samples sizes to be more clearly visible to the reader.  - Figures 9-11: perhaps consider using a multi-panel 2d histogram to show the density profiles for each group, or at least using a better point symbol.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We have addressed all minor text or grammatical suggestions by the reviewers.We have re-organized the article into distinct and more separated Workflow and Evaluation sections, which was suggested by all reviewers. We begin the article with a clear outline, titled: "Structure of this article", which outlines the Workflow part and the Evaluation part. This outline has direct links to relevant sections and subsections which follow. We have also included an overview diagram of the methods and packages included in the Workflow section, and how they are interconnected.We have added to the Introduction more motivational text on why a DTU analysis is relevant for biology and biomedical research.We have added a large section describing the methods DEXSeq and DRIMSeq, before the Workflow section.We have expanded the original sections discussing counts-from-abundance and their use in the workflow, to make our use of the tximport method more clear.For the DEXSeq section, we have corrected an earlier incorrect use of nbinomLRT, which is now replaced with the correct testForDEU. The practical result is that DEXSeq performs somewhat less conservatively, but the original code was incorrect, and the fix is necessary. The incorrect use of nbinomLRT in this context will now produce an error in future releases of Bioconductor, to avoid possible incorrect usage.We have added RATs to the DTU Evaluation.We now apply stageR to all DTU methods that are evaluated: DRIMSeq, DEXSeq, RATs, and SUPPA2. The RATs and SUPPA2 methods are described, but the code is not provided, as these packages are not part of the Workflow.We use consistent x-axes and y-axes whenever possible, and use PDF instead of JPG to reduce compression artifacts. When a consistent x-axis is not used in the main text, we include Supplementary Figures with the same plots with outlying methods dropped to keep the x-axis consistent.We use a palette in which colors are more discernable for color-blind readersIn the Evaluation sections, we include additional plots which examine the simulated gene type source of false positives for the DTU, DGE, and DTE analyses.We added a new evaluation to examine performance differences between DRIMSeq and DEXSeq, using the identical simulated data that was used in Soneson et al (2016) and Nowicka and Robinson (2016).We have added a 2 vs 2 simulation for the DTU Evaluation.We added a brief overview description of all methods assessed in the DGE and DTE Evaluations.We have added more recommendations in the Discussion. Reviewer-specific comments:We thank all reviewers for their insightful comments and suggestions that we feel have greatly improved the readability and usefulness of the workflow. We summarize the main changes and then address reviewer-specific comments point-by-point: 1) We have followed the reviewer's suggestion, and have separated the Workflow and Evaluation sections, with an outline at the beginning clearly delineating the two sections, and an overview diagram. 2) We originally submitted our Bioconductor workflow as a "Research" article, but the Editorial Office recommended to change the categorization to "Method", which is the categorization of many of the other Bioconductor workflows. Bioconductor workflows are not intended to introduce new computational methods or new software packages, but to demonstrate, with live code that resides in an Rmarkdown vignette within an R package structure, how to use a number of different existing Bioconductor packages to analyze a dataset.  We asked for comment from the Editorial Office on the recommended categorization of Bioconductor workflows under the F1000Research article types, and they provided us with the following statement:In general, Bioconductor workflows are classified as Method articles in F1000Research, since Research articles must present novel research findings, and Software Tool articles must present novel software tools. Since this article by Love et al neither presented novel research findings nor a new software tool, the F1000Research editorial office felt that classifying this article as a Method article was most appropriate. The majority of workflows submitted to the Bioconductor gateway will fall into this article type." -F1000Research Editorial Office " 3) We have followed the reviewer's suggestion and included, in addition to the fixed per-gene dispersion simulation, an additional simulation from Soneson et al. (2016), to assess differences between DRIMSeq and DEXSeq, the two methods that are the focus of the workflow. This simulation involved generation of Negative Binomial gene counts, and then the expression was distributed from genes to transcripts by per-sample draws from a Dirichlet distribution, with a minority of genes undergoing DTU across condition. Analysis of additional datasets, and a final determination of which type of data-generating process is closer to various real RNA-seq datasets, is beyond the scope of this workflow, but we feel that the existing simulations cover a range of possibilities and are useful to the readers of the workflow. We comment in a number of places on the limitations of the simulation, including in the overview:While the evaluations rely on simulated data, and are therefore relevant only to the extent that the simulation model and parameters reflect real data, we feel the evaluations are useful for a rough comparison of method performance, and for observing relative changes in performance for a given method as sample size increases." " Also at the end of the DTU Evaluation:Again, a caveat of all of our comparative evaluations of DRIMSeq and DEXSeq is that we do not know whether various real RNA-seq experiments will more closely reflect heterogeneous dispersion or fixed dispersion within genes, or if the counts within gene are better modeled by distributing gene-level abundance to transcripts via a Dirichlet distribution as in Soneson et al (2016). However, we have examined simulations reflecting each of these cases, and confirmed that minimum count and minimum proportion filtering benefit both DRIMSeq and DEXSeq." " 4) We now include more discussion on the results of the evaluations in the Discussion, including a comment on statistical power. We include a breakdown of false positives by the simulated gene type. Further cross-section of all methods' performance by incomplete annotation, effect size filters, and various count or proportion filters is beyond the scope of the article. Complete analysis of overlap of calls across the various simulations and analyses is also beyond the scope of the article. We now explore DRIMSeq's performance in the "main" and "fixed per-gene dispersion" simulations, wherein we see that many of the excess false positives at the transcript-level arise from simulated DTU genes, so other transcripts not participating in DTU were being reported as significant. In the “main” simulation, where DRIMSeq has the most problem with FDR control, it only slightly exceeds a target 10% FDR at the gene level at per-group sample sizes 6 and higher. With proportion SD filtering, DRIMSeq at the transcript level also has small inflation of target 10% FDR for per-group sample sizes 6 and higher.  We now include RATs as an additional method evaluated on the "main" simulation for DTU analysis. RATs performs similar to SUPPA2, in that it nearly always controls the FDR, although in some cases, it displays higher gene-level sensitivity than SUPPA2. We do not intend the article to be a complete evaluation of all existing methods for DTU, but to compare the two Bioconductor methods that are the focus of the workflow with a few key DTU methods. Extended discussion of long-read sequencing is beyond the scope of the article, although we added the following comment to the workflow section on importing counts:If a different experiment is performed and a different quantification method used to produce counts per transcript which do not scale with transcript length, then the recommendation would be to use these counts per transcript directly. Examples of experiments producing counts per transcript that would potentially not scale with transcript length include counts of full-transcript-length or nearly-full-transcript-length reads, or counts of 3' tagged RNA-seq reads aggregated to transcript groups. In either case, the statistical methods for DTU could be provided directly with the transcript counts." " A relevant quote from Nowicka and Robinson (2016) is:With emerging technologies that sequence longer DNA fragments , we may see in the near future more direct counting of full-length transcripts, making transcript-level quantification more robust and accurate.” " In the "DTU testing" section, we now discuss how DEXSeq and DRIMSeq can be used to evaluate experiments with complex designs, with little limitation as long as the coefficients for each sample can be encoded as a design matrix multiplied by a vector of coefficients. 5) Comprehensive evaluation of the methods on additional datasets is beyond the scope of the article. 6) Following this and other reviewers' suggestion, we have now added motivation to the first part of the Introduction as to why DTU is relevant for biological or biomedical research. 7) We have revised some of our description of the stageR framework to be more clear about why we recommend its use in a DTU workflow:It is likely that an investigator would want both a list of statistically significant genes and transcripts participating in DTU, and stageR provides error control on this pair of lists, assuming that the underlying tests are well calibrated." " We also provide some more details in the Discussion regarding the various methods and their performance. Minor Comments: 1) We have added an overview diagram as Figure 1. 2) We have updated figures to be PDF instead of JPG, and made the axes more consistent when possible. A workflow to enable more people to perform differential transcript usage on their RNA-seq data set is a useful addition to the literature. Benchmarking methods and combinations of workflows are also an important part of the literature. In this manuscript, both things have been attempted, which unfortunately makes the manuscript a little blurred in its focus. We view a workflow as an instructional manuscript in which a step-by-step analysis can be reproduced with a new data set that a user wants to bring to the analysis. This is presented in the sections Quantification and data import and Statistical analysis of differential transcript usage and, in our view, should be the focus of the manuscript. These are complex analyses combining several packages with several alternative paths. It would really help the user if a flowchart for this analysis could be made that shows the common parts of the workflow , how the alternatives split and which packages are used for alternative parts of the workflow. For example, DRIMseq is an alternative to DEXseq, which can then be followed by stageR, and Suppa is a complete  workflow. The evaluation sections are somewhat useful and interesting in their own right, but rely on simulated data and are therefore not directly applicable to readers who are looking for workflows to guide them in their own data analysis. However, they do help users decide which workflows to choose in their own analysis. Overall we wonder if this manuscript could be two separate manuscripts: a workflow for DTU and an evaluation of methods based on simulated data? Another (preferable) alternative would be to only focus on DTU in the evaluation and keep the section Evaluation of methods for DTU as a guide to help the user to choose the workflow (with this clearly stated). We felt there were too many additional analysis introduced after this point which relied on more in-depth understanding of the DGE literature, which was not really the focus of the workflow.Minor comments:page 6: "We recommend scaledTPM for differential transcript usage so that the estimated proportions fit by DRIMSeq in the following sections correspond to the proportions of underlying abundance." Could the authors please rewrite/break up this sentence to improve readability?page 6, section 'Import counts into R/Bioconductor': the authors should clarify whether the referenced R package is for demonstration purposes only .page 6: could the concept of using counts from abundance be introduced/explained before referring to specific package parameters and settings?page 6: "The following code chunk is not evaluated, but instead we will load a pre-constructed matrix of counts". Could the authors please clarify this sentence? We assume this means that instead of constructing a matrix of counts , pre-constructed data is loaded.page 7 "We ran the following unevaluated code chunks": does 'unevaluated' refer to not run in a typical workflow?page 7, 'Statistic analysis of differential transcript usage', second paragraph: could the description of txdf be moved to the previous section where it is constructed? This would help improve the flow.page 12: "(2) contain a transcript with a transcript adjusted p-value less than 0.05 which does not participate in DTU, so contain a falsely confirmed transcript": could the authors please rewrite this sentence for clarity.page 13: sentence "The testing of “this” vs “others”..." could be improved for clarity, e.g.: "DEXseq in its original version requires fitting of coefficients for each exon within a gene. Running DEXseq at a transcript-level considerably improves performance as fewer features per gene require fitting of coefficients."page 14, after the line "dxr <- as.data.frame could help in clarifying the output.page 15, in the code "paste0("suppa/group1.tpm")": the paste function is not necessary here.Section 'Evaluation of methods for DTU': could the authors offer an explanation why SUPPA2 only reported one DGE gene as DTU?Could the y and x axes on the plots on pages 17-20 and 25 be made consistent with each other? Also, very minor point, but these plots have some jpeg artefact. Could pdf or png plots be used instead?page 19 "DRIMSeq [...] performed slightly better": could a metric be referenced in how the package performed better?page 22: "We can repeat the same analysis...": 'same analysis' is misleading as this section tests only DGE.page 24: could the authors formally introduce or describe EBSeq and SAMseq packages, preferably earlier in the manuscript?page 26: could the authors use 'compute time' instead of 'timing'? Several sections should be edited for clarity and flow of ideas. Specifically,page 5: "We recommend [constructing] a CSV file..."page 6: "We suggest for DTU analysis to generate counts from abundance..." reword to "For DTU analysis, we suggest generating counts from abundance..."page 16: "DEXSeq controlled [the FDR] except for..."page 16: "DRIMSeq had [an] observed FDR.."page 16: "...reported 2 extra genes more than..." change to "reported two more genes than"page 16: "...DEXseq were the most sensitive methods [for] recovering"page 19 "...DRIMSeq and DEXSeq [in] this additional simulation"page 19: "Again, we caveat our comparative evaluation of DRIMSeq and DEXSeq by noting that we do not know..." change to "Again, a caveat of our comparative evaluation of DRIMSeq and DEXSeq is that we do not know..."page 24: "did not have [adequate] sensitivity to detect DGE"page 24: "while those two method[s] had" We identified the following typographical errors and grammatical issues:We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. We have addressed all minor text or grammatical suggestions by the reviewers.We have re-organized the article into distinct and more separated Workflow and Evaluation sections, which was suggested by all reviewers. We begin the article with a clear outline, titled: "Structure of this article", which outlines the Workflow part and the Evaluation part. This outline has direct links to relevant sections and subsections which follow. We have also included an overview diagram of the methods and packages included in the Workflow section, and how they are interconnected.We have added to the Introduction more motivational text on why a DTU analysis is relevant for biology and biomedical research.We have added a large section describing the methods DEXSeq and DRIMSeq, before the Workflow section.We have expanded the original sections discussing counts-from-abundance and their use in the workflow, to make our use of the tximport method more clear.For the DEXSeq section, we have corrected an earlier incorrect use of nbinomLRT, which is now replaced with the correct testForDEU. The practical result is that DEXSeq performs somewhat less conservatively, but the original code was incorrect, and the fix is necessary. The incorrect use of nbinomLRT in this context will now produce an error in future releases of Bioconductor, to avoid possible incorrect usage.We have added RATs to the DTU Evaluation.We now apply stageR to all DTU methods that are evaluated: DRIMSeq, DEXSeq, RATs, and SUPPA2. The RATs and SUPPA2 methods are described, but the code is not provided, as these packages are not part of the Workflow.We use consistent x-axes and y-axes whenever possible, and use PDF instead of JPG to reduce compression artifacts. When a consistent x-axis is not used in the main text, we include Supplementary Figures with the same plots with outlying methods dropped to keep the x-axis consistent.We use a palette in which colors are more discernable for color-blind readersIn the Evaluation sections, we include additional plots which examine the simulated gene type source of false positives for the DTU, DGE, and DTE analyses.We added a new evaluation to examine performance differences between DRIMSeq and DEXSeq, using the identical simulated data that was used in Soneson et al (2016) and Nowicka and Robinson (2016).We have added a 2 vs 2 simulation for the DTU Evaluation.We added a brief overview description of all methods assessed in the DGE and DTE Evaluations.We have added more recommendations in the Discussion. Reviewer-specific comments:We have tried to separate and clarify the Workflow section and the Evaluation section. We now include an overview diagram, as helpfully suggested here.We have expanded the section on counts-from-abundance, added a section before the counts are imported, and clarified the sentences highlighted by the reviewers.We have clarified a number of the "not evaluated" sentences in the original workflow.The description of txdf is given in the section where it is constructed, under the heading "Transcript-to-gene mapping".We have clarified the OFDR description in the sentence highlighted by the reviewers, and have removed the "this" vs "other" sentence, as the history of DEXSeq method development is not necessary or useful for the readers of this workflow.We have added `head(dxr)` to demonstrate the output.We have removed the SUPPA2 code, as now the workflow focuses on the Bioconductor package DRIMSeq and DEXSeq, which have live code examples (SUPPA2 is a python package and so cannot have live code examples in a Bioconductor workflow).We have made the x- and y-axes consistent whenever possible.We have revised the Workflow and Evaluation sections following all of the reviewers' helpful comments, error spotting, and suggestions on improved wording.We thank all reviewers for their insightful comments and suggestions that we feel have greatly improved the readability and usefulness of the workflow. We summarize the main changes and then address reviewer-specific comments point-by-point: Summaryet al presents a combined workflow and benchmark for differential transcript usage. This is a vital paper as there is no consensus on which differential transcript usage tools works better (here addressed by the benchmark part) and very few people analyze differential transcript usage – something the workflow can hopefully help with. Of special note is the extent to which open source have been embraced by Loveet al – an approach that is commendable (and copy worthy). Although the manuscript has a lot of potential it can, in its current form, be challenging to read and the benchmark of differential transcript usage part needs to be extended. Revisions are therefore required. In “Swimming downstream: statistical analysis of differential transcript usage following Salmon quantification” LovePrefaceMalte Thodeberg helped me review this paper – thanks Malte!Since neither of us are native English speakers/writers we have not attempted to corrected for potential gramma and/or spelling mistakesI'm the developer of IsoformSwitchAnalyzeR.General commentsOne solution would be to have the benchmark first and the workflow afterwards. It would then be natural that workflow used the tool(s) deemed better by the benchmark.The article switches between describing a workflow, which users can follow to perform differential transcript usage on their own data, and a benchmark of differential expression/usage tools. The two sections should be much more clearly separated and each should be more concisely written.The main problem with the workflow part of the manuscript is the intermixing of the workflow and benchmarking (and the intro/methods) sections which makes it necessary to include a lot of callouts, omissions and special cases. This has the unintended effect of cluttered the workflow making it hard to read and/or follow. This would however be solved by the above suggested re-structuring. If such restructure were implemented it would also seem more natural that the workflow consistently only use a small dataset (either a subset of the simulated data or another dataset entirely) whereby the workflow could be simplified a lot.Although the benchmark is of high quality it still needs to be a bit more exhaustive.(Even with the suggested re-structure) The whole article would highly benefit from an overview paragraph and/or figure to give the reader the high-level overview of the outline before jumping into it (something like a table/figure/description of content). This could also be a table of content (with links included to enable easy jumping in the article).TitleThe title should reflect it is a workflow and/or benchmark. The current title suggests the authors developed a new tool for differential transcript usage which were specifically designed to integrate with Salmon. Furthermore, it could be considered to change the title so it also indicates the differential gene/transcript expression performed in the manuscript.IntroductionThe introduction lacks a section describing why differential transcript usage are of interest in the first place.Optional suggestion: include a lay-man introduction to how the tools work .Large parts of what would normally be in the introduction and methods have been moved into the results. Introduction to tools and methods including descriptions of how they work belongs in the introduction. Description of parameter choice for e.g. scaling during tximport also belongs in intro/methods.In the section where tools for DTU are mention please remove (or argue for inclusion of) BITSeq and stageR. StageR is for post analysis of p-values (no test). Although BITSeq is mentioned in some of the BiocViews of alternative splicing neither the article nor the vignette shows anything but DTE (aka no DTU). Mention that SGSseq wraps DEXSeq.The test build into IsoformSwitchAnalyzeR in not rank-based – but it is obsolete and will be removed from the next update – so it could be skipped entirely .https://www.bioconductor.org/packages/devel/BiocViews.html#___AlternativeSplicing ).Please reference IsoformSwitchAnalyzeR for its main purpose: the downstream analysis of functional consequences of identified isoform switches. Consider also mentioning other tools for downstream analysis The simulations performed should either be named or numbered to allow for clear reference to which of the simulated datasets are used.In the countSimRepport please compare the simulated data to the 12 samples which were used for the basis of the simulation (comparing 12 to hundreds of samples is not easy to interpret).Please elaborate on discussion of the different options for scaling-from-TPM-to-counts. It is unclear what the difference is and when it matters. Furthermore you write “if we used lengthScaledTPM transcript counts, then a change in transcript usage among transcripts of different length could result in a changed total count for the gene, even if there is no change in total gene expression” is there a mixup here? If not, why do you then use lengthScaledTPM in the DGE/DTU section? Please include a recommendation of when to use which option for analysis of DGE/DTE, DTU and if both are present in the data.Include a paragraph on quantification before introducing the modifications. If any expression filtering was done (as fig 1 indicate and mention above) it should be clearly stated.Currently it is unclear how many genes were modified in which way. To remedy that please provide a table indicating the number genes modified for DTU or DGE by each of the changes you introduce .Modifications  In the workflowPlease add a comment of why DRIMSeq have NA as p-values (that will confuse many people)  Post-hoc filtering on DRIMSeqWhat is the reasoning beheading this filtering step? And is it statistically valid to do this filtering – the proportions and p-values are not independent. Is the modified p-value distribution still uniform in the interval , but we do not attempt to provide empirical evidence to support this. Importantly, we apply the post-hoc filtering because we have empirical evidence that DRIMSeq was not providing uniform p-values for null transcripts on the simulated data explored in this article. Therefore, we begin with a non-uniform distribution of p-values for the null transcripts. The filtering is shown empirically to improve the FDR control. We do not perform the simulation multiple times, and we have not extended iCOBRA to support multiple iterations on a single plot, which is beyond the scope of this article. We are most interested in the relative performance of the various methods, and their general location on the TPR-FDR plots, which is achieved with the current evaluation. We did explore running DEXSeq 25 times on the 3 vs 3 "main" simulation, and the inter-simulation variation in the TPR-FDR plot was minimal. We have uploaded all 24 of the simulated paired-end reads to Zenodo, and the dataset is already quite large. We do not run the methods on entirely null datasets, which is beyond the scope of this article. We have now used stageR on all methods. stageR accepts gene-level p-values  and transcript-level p-values. If gene-level p-values are not provided by a method then DEXSeq's perGeneQValue was used to generate gene-level adjusted p-values, for use with stageR. We do not evaluate other methods for exon usage, as we focus in the workflow on Bioconductor methods that have been already proposed and evaluated for DTU analysis in publications. We now use consistent axes, and include the group size in the strip titles. We now evaluate DRIMSeq and DEXSeq on the identical simulation dataset used in both Soneson et al (2016) and Nowicka and Robinson (2016). We find similar performance of DEXSeq as reported in those papers using a less stringent transcript filter, but when we use DRIMSeq count and proportion filters as recommended in this workflow, the performance of DEXSeq is greatly improved, to levels consistent with what we see in the "main" simulation.Evaluation of DGE/DTE We clarify why a DGE and DTE evaluation is included. We do not perform a 2 replicate DGE or DTE evaluation, as this is beyond the scope of the article. We now breakdown the DGE and DTE results by simulated gene type. We do not see any strong enrichment of one simulated gene type in the false positive breakdown plots. We believe our evaluation may differ from others in exploring the consistency of results as sample size increases.Discussion We now include in the Discussion some recommendations on tool usage and performance.