The abundance of new computational methods for processing and interpreting transcriptomes at a single cell level raises the need for in silico platforms for evaluation and validation. Here, we present SymSim, a simulator that explicitly models the processes that give rise to data observed in single cell RNA-Seq experiments. The components of the SymSim pipeline pertain to the three primary sources of variation in single cell RNA-Seq data: noise intrinsic to the process of transcription, extrinsic variation indicative of different cell states (both discrete and continuous), and technical variation due to low sensitivity and measurement noise and bias. We demonstrate how SymSim can be used for benchmarking methods for clustering, differential expression and trajectory inference, and for examining the effects of various parameters on their performance. We also show how SymSim can be used to evaluate the number of cells required to detect a rare population under various scenarios. Simulated single cell RNA sequencing data is useful for method development and comparison. Here, the authors developed SymSim, a simulator that explicitly models the main factors of variation in single cell data. While these computational methods are often based on reasonable assumptions it is difficult to compare them to each other and assess their performance without gold standards. One approach to address this is through simulations12.The advent of single cell RNA sequencing has led to a surge of computational and statistical methods for a range of analysis tasks. Some of the methods or the tasks that they perform have originated from bulk sequencing analysis, while others address opportunities  rely primarily on fitting distributional models to observed data and then drawing from these distributions. While the resulting models provide a good fit to observed data, their parameters are often abstract and do not directly correspond to the actual processes that gave rise to the observations. This leaves an important unaddressed problem in designing and using a simulator: the need to modulate and then study the effects of specific aspects of the underlying physical processes, such as the efficiency of mRNA capture, the extent of amplification bias , and the extent of transcriptional bursting. To address this, we present SymSim (Synthetic model of multiple variability factors for Simulation), a software for simulation of single cell RNA-Seq data. SymSim explicitly models three of the main sources of variation that govern single cell expression patterns2: allele intrinsic variation, extrinsic variation, and technical factors .The first knob for controlling the simulation allows us to adjust the extent to which the infrequency of bursts of transcription adds variability to an otherwise homogenous population of cells. We use the widely accepted two-state kinetic model, in which the promoter switches between an on and an off states with certain probabilitieskon, koff, and s) for each gene in each cell are first calculated using a product of cell-specific and gene-specific factors, then adjusted by the parameter distributions estimated from experimental data .The coordinates of a cell’s vectors represent factors of cell to cell variability that are extrinsic to the noise generated intrinsically by the process of transcription (which we model by drawing from the stationary distribution above). These values, which we term extrinsic variability factors (EVF) represent a low dimension manifold on which the cells lie and can be interpreted as concentrations of key proteins, morphological properties, microenvironment and more. When simulating a homogeneous population, the EVFs of the cells are drawn from a normal distribution with a fixed mean of 1 and a standard deviation on rate of a certain promoter (if the EVF and gene effect vectors are both for parameter kon). The gene effect values are first drawn independently from a standard normal distribution. We then replace each gene effect with a value of zero with probability η, thus ensuring that every gene is only affected by a small subset of EVFs. The sparseness parameter η can be set by the user; in this paper we set it to a fixed value of 0.7.The coordinates of the gene effect vectors can be interpreted as the dependence of its kinetics on the levels of EVFs. For instance, a positive value means that higher concentration of the corresponding EVF can give rise to a higher 4 and MAGIC18) to reduce technical variation in real data. Then we obtain aggregated distributions from the results of all the settings we considered . To gain a robust estimation of the distribution of kinetic parameters to be used by SymSim, we performed the estimation multiple times with 1) different subpopulations of a dataset; (2) different imputation methods  than others , possibly due to selection bias which can be exacerbated by low capture rate. In SymSim, we model the high-expression outlier genes by designating a small subset of genes (whose proportion is determined by the parameter prop_hge) as constitutively transcribed, and adjusting their transcription rate s by a factor determined by the parameter mean_hge .Finally, we account for the possibility of outlier genes with unusually high-expression level, commonly observed in real data. These outlier genes are hard to model with distributional methods, and require additional parametrization17 and extent of the intrinsic noise15 in the expression of a gene in a homogenous population of cells  can vary for the different ranges of kon, koff, and s. Specifically, one can distinguish the following three types of gene-expression distributions by the number of inflection points in the smoothed density function: unimodal with highest frequency at 0 (no inflection point), unimodal with highest frequency at non-zero value (one inflection point), and bimodal (two inflection points). Figure kon and koff with given s = 10. This gives a clear correspondence between kinetic parameter configurations and types of gene-expression distributions. For example, when s is relatively large, we obtain bimodal distributions when kon and koff are smaller than 1.An intriguing question in the analysis of single cell RNA-seq data is the extent to which the conclusion drawn from the data  may be confounded by transcriptional bursting and transcriptional noise. SymSim provides a way to explore this. We first note that modalitykon and koff values by bimod can take value from 0 to 1. This way, other properties such as burst frequency (kon/(kon + koff)) and synthesis rate (s) remain the same. In Fig. bimod parameter on gene-expression distribution in a simulated homogenous population. Expectedly, as bimod increases, so does the number of bimodal genes, as well as the average Fano factor  all While the first knob focuses on variation within a homogeneous set of cells, the second knob allows the user to simulate multiple, different cell states. This added complexity is achieved by setting different EVF values for different cells, in a way that allows users to control cellular heterogeneity and generate discrete subpopulations or continuous trajectories. To this end, SymSim represents the desired structure of cell states using a tree (which can be specified by the user), where every subpopulation (in the discrete mode) or every cell (in the continuous mode) is assigned with a position along the tree. Different positions in the tree correspond to different expected EVF values, and the expected absolute difference between the value of an EVF of any two cells is linearly proportional to the square root of their distance in the tree . The EVF values are then drawn from a normal distribution where the mean is determined by the position in the tree, and the standard deviation is defined by σ  full-length mRNAs profiling without the use of UMIs ; and (2) profiling only the end of the mRNA molecule with addition of UMIs . The former protocol is usually applied for a small number of cells and with a large number of reads per cell, providing full information on transcript structure36. The latter is normally applied for many cells with shallower sequencing, and it is affected less by amplification and gene length biases33.The third knob of SymSim allows users to control technical variation, which accounts for a large part of the variation observed in scRNA-seq datasetsα and standard deviation β, which can be set by the user. The second step is amplification, where in every cycle SymSim selects each available molecule with a certain probability and duplicates it. The expected amplification efficiency and the number of PCR cycles can be set by the user. As an optional step, SymSim provides the option of linear amplification . We do not apply this option in this manuscript. In the third step each amplified molecule is broken down into fragments, in preparation for further amplification, size selection, and sequencing (Methods).The workflow of these steps is shown in Fig. Depth, which, along with the respective standard deviation (Depth_sd) can be provided by the user. To derive the final observed expression values we do not account for sequencing errors, and assume that every sequenced fragment is assigned to the correct gene it originates from. For the non-UMI option, we define the raw measurement of expression as the number of reads per gene. If UMIs are used, SymSim counts every original mRNA molecule only once by collapsing all reads that originated from the same molecule. Notably, for certain depth values, the resulting distribution of number of reads per UMI is similar to the one observed in a dataset of murine cortex cells38  is drawn from a normal distribution whose mean is determined by the parameter 39, whereas the use of UMIs can correct these biases40. In particular, we have observed a negative correlation between gene length and length-normalized gene-expression in our reference non-UMI dataset . As a result, our simulated data with a non-UMI protocol show a similar dependence of gene-expression on gene length as in experimental data . In cases where UMIs are used, gene length effects are also modeled during amplification, but these effects are mitigated since each molecule is counted at most once. We therefore do not observe gene length bias in the UMI-based simulated data, similarly to the experimental data . Finally, we model batch effects with multiplicative factors that are gene- and batch-specific. In Supplementary Fig. It has been previously shown that estimation of gene-expression levels from full-length mRNA sequencing protocols has amplification biases related to sequence-specific properties like gene length and GC-contentα), extent of amplification bias (MaxAmpBias), and sequencing depth (Depth). Using “bad” technical parameters introduce more noise to true counts, and compared to the non-UMI simulation the UMIs reduce technical noise. The histograms of true counts and four versions of simulated counts are shown in Supplementary Fig. In Fig. The total computation time to simulate a dataset consists of time to generate true counts and time to generate observed counts from true counts. We show the runtime and memory usage for different parameter configurations in Methods.13) and sets the values of ten parameters from both the first and third knobs (Methods). We test this function with the non-UMI Th17 dataset41 , the UMI cortex dataset38  and two UMI datasets from 10x Genomics, denoted by “UMI 10x t4k” and “UMI 10x pbmc8k” . See Supplementary Note For a given real dataset, SymSim can produce observed (read or UMI) counts that have similar statistical properties to the real data Fig. , by sear13 and powSimR12 with the same experimental datasets as input, and found that SymSim matches this data significantly better  and observed counts , indicates that SymSim can transform the simulated ground truth Fig. , left inN), within population variability (σ), mRNA capture rate (α), and sequencing depth (Depth). We also test the effect of the proportion of cells associated with the smallest subpopulation of cells (Prop), using population 2 in the tree as our designated rare subpopulation.SymSim can be used to benchmark methods for single cell RNA-Seq data analysis as it provides both observed counts and a reference ground truth. In the following sections we demonstrate the utility of SymSim as tool for benchmarking methods for clustering, differential expression, and trajectory inference in a sample consisting of multiple subpopulations, using the structure depicted in Fig. 38 and specified in Supplementary Table 4 and SIMLR42. In all cases we set the expected number of clusters to the ground truth value (k = 5). The accuracy of the methods is evaluated using the adjusted Rand index . To inspect the effects of the various parameters on clustering performance, we performed multiple linear regression between the parameters and the ARI. The regression coefficients are shown in Fig. σ appears to be the most dominant factor, and the proportion of the rare population (Prop) is clearly positively associated with better performance. Among the technical parameters, while α plays a role on the performance especially for the rare population, the impact of Depth is minor.We begin by inspecting the impact of each parameter on the performance of clustering methods. To this end, we simulated observed counts using the UMI option, and traversed a grid of values for the five parameters with 18 simulation runs per configuration. The values of the remaining parameters are largely determined according to the cortex datasetN, which we discuss in the next section), namely, the within-population variation parameter σ and mRNA capture efficiency α, for the benchmarking analysis henceforth. In particular, the range of values for σ is set such that we cover various data characteristics, from well separated populations, to almost entirely mixed ones  or k-means. We observe better accuracy as the quality of the data increases or the within-population variation decreases , especially with scVI. Comparing the methods to each other, we see that scVI/k-means has the highest ARI in most cases; when the rare population accounts for 5% of all cells, Seurat is the second best and PCA/k-means and SIMLR are comparable; when the rare population accounts for 10% of all cells, SIMLR performs slightly worse than Seurat and PCA/k-means.We then focus on varying the dominant factors  genes between populations  and mRNA capture rate (α) with 10 simulation runs per parameter configuration. We use two accuracy measures: (a) AUROC (area under receiver operating characteristic curve), obtained by treating the p-values output from each method as a predictor , and therefore we used its unadjusted p-values in Fig. 47). To make use of this filtering, we conducted an additional analysis where we used the adjusted p-values for DESeq2 and compare it to all other methods using only the non-filtered (non NA) genes  on the performance of clustering and differential expression algorithms. This analysis is presented in Supplementary Note We also investigate the effects of bimodality  algorithm implemented in the package dynverse52 (Methods). We generate datasets with different values of σ and α with the input tree shown in Fig. k-nearest neighbor purity (knn purity) of cells, that is, for each cell, we calculate the Jaccard Index between its k-nearest neighbors in the true trajectory and that in the inferred trajectory. Results are shown in Fig. k is set to 100. In Fig. σ and fix α as 0.1. Both the correlation and knn purity decrease when σ increases. In Fig. α and fix σ as 0.6. All methods show an overall increasing trend along with α with both measures. Consistent with a recent benchmark study52, we observe that overall Slingshot clearly outperforms the other two methods.The ability of SymSim to generate a continuum of cell states makes it a convenient choice to benchmark trajectory inference methods. We compare three methods including Monocle53 assumes that the goal of the experiment is to identify subpopulations of cells and provides a theoretical lower bound for the problem. This bound considers the aspect of counting cells , but it does not account for the identifiability of each subpopulation, which may be hampered by both technical and biological factors as well as the performance of clustering algorithms.Deciding how many cells to sequence is a decision many researchers face when designing an experiment, and the optimal number of cells to sequence highly depends on the nature of the biological system under investigation and the respective technical hurdles. A previous approach to this problemN) ranging from 600 to 7000. These simulations were based on the parameters fit to the cortex dataset38 with varying levels of σ and α (250 simulations per parameter configuration).In the following we demonstrate how SymSim can be used to shed more light on this important problem. Importantly, in its current form SymSim does not use real data to model between-population variability. We therefore interpret the results in a relative manner—how do different variability factors shift the required number of cells, compared to each other and to the theoretical lower bound. Our example focuses on a case of one rare subset, represented by cells from population 2  and SIMLR). We say that a given algorithm was successful in detecting the rare population if at least 50 cells from this set are assigned to the same cluster, and form at least 70% of the cells in that cluster. We use these labels to compute an empirical success probability P for each algorithm and each parameter configuration. Out of the 250 simulations for each parameter configuration, we randomly sample 100 randomizations 20 times, and for each 100 randomizations we can calculate a value of P. We then plot the mean and standard deviation as error bars of the 20 values of P for each N under each configuration  reaches almost 1 at N = 1400. Conversely, the empirical curves vary dramatically, based on parameter values. For an easy case of low within-population variability (σ = 0.4) and high capture efficiency (α = 0.1) the empirical upper bound curve is close to the theoretical one , we need at least 3838 cells (corresponding to α = 0.01), while with the theoretical curve, we need only less than 1200 cells. In general, the number of cells needed increase when the desired success rate and σ increase. Very low capture efficiency  tend to require high number of cells. Considering only the binomial sampling of cells may therefore underestimate the number of cells needed for a realistic scenario, and considerations of biological and technical variations with simulators like SymSim is merited.To understand the implications on the number of cells required in a given setting, we calculated how many cells are required, in each configuration, to achieve a success rate of respectively 0.6, 0.7, 0.8, and 0.9 Fig. . As expe13. (iv) When generating observed counts, we simulate key steps in real experimental protocols, which automatically gives us dropout events, length bias, and distribution of library sizes. We also provide choices to use UMI-based protocols or non-UMI full-length mRNA protocols, as the properties of data output from these two categories can be very different.SymSim has the following features, which are advantageous over existing simulators: (i) We simulate true transcript counts from a kinetic model that can be interpreted in terms of transcript synthesis rate, promoter activation, and deactivation. (ii) When generating multiple discrete or continuous populations, instead of generating biological differences through directly altering the true transcript count distribution, we set Diff-EVFs, which can be interpreted as biological conditions that cause the differences between subpopulations of cells. This is a more natural and realistic way to simulate biological transcriptional differences. (iii) The EVF formulation provides an intuitive way to specify and simulate complex structures of cell–cell similarity, without the need for manual specifications of the numbers of DE genes4. This type of extension will facilitate simulation of between-subpopulation diversity that better mimics experimental observations, albeit at the cost of using parameters that are less interpretable biologically. Another extended application of interest is to use different tree structures for different Diff-EVFs when generating multiple populations of cells, such that every tree represents a different aspect of variability between cells. For instance, using this approach, one tree can represent a differentiation process and the other can represent variability due to the physical location of the cell.The main input parameters to SymSim, mostly the parameters in the third knob, are self-explanatory with their own technical meanings, which users can adjust to match an experimental dataset of interest. SymSim allows users to simulate datasets with desired properties or matched with experimental data. While the procedure of parameter fitting was developed in order to generate simulated datasets with similar properties, it may also provide additional insight, as the parameters are biologically or technically interpretable. For instance, comparing the parameters fit to the the UMI and non-UMI datasets in this study we note that the capture efficiency inferred to the latter is much higher . The need for robust analytical methods therefore increases, and so does the means for proper evaluation of these methods. SymSim provides a starting point to address this challenge of flexible and feature-rich simulation for method evaluation, as it aims to directly mimic the key mechanistic properties of single cell RNA sequencing.As the number and extent of biological applications of single cell genomics continues to grow, so does the extent of analytical questions one can tackle, which go beyond standard bulk era analysis steps  for all genes in all cells, sampling the same number of values from the experimental kon distribution , and updating the kon values to the ones sampled from the experimental distribution with the same rank. The values of koff and s are calculated in the same way.For a gene in a cell, the parameters for the kinetic model ene Fig. . To alloprop_hge and mean_hge) to SymSim for these outlier genes. First, we use parameter prop_hge to represent the proportion of these genes. For randomly selected genes according to this proportion, we consider that they are always on and their gene-expression levels are sampled from a Poisson distribution with parameter s (the transcription rate). To reflect their high gene-expression level, we increase their transcription rate s calculated from EVFs and gene effects with an inflation factor. The inflation factor for each gene is the exponential (base 2) of (mean_hge-1 + rank/n), where rank is the rank of mean s of a gene among all outlier genes , n is the total number of outlier genes and mean_hge specifies the extent of inflation to original s. The values of prop_hge and mean_hge can be fitted from real data.Finally, to better model the high-expression outlier genes which usually account for a very small proportion of all genes, we introduce two parameters  by Gaublomme et al.41 . To mitigate the effects of low sensitivity, the UMI-based data were imputed using scVI4 and MAGIC18, and the non-UMI data were imputed using MAGIC18 (as scVI is only applicable to large datasets). To reduce the effects of extrinsic variation, we performed the parameter estimation separately in each of the three largest clusters in the cortex dataset (each cluster is assumed to represent a relatively homogenous subpopulation), and on the entire T cell data  and obtained similar distribution ranges , and at each time point n σ Fig. .σ.For the continuous mode we also provide an alternative option to the Brownian motion model: using impulse functions for modeling the path-specific variation. When impulse function is used, for cells sampled from branches that are not on the root-tip path for the specific EVF, they are sampled from a univariate normal with mean equal to the EVF value at their most recent common ancestor with the varying path, and standard deviation 28 and sequences full-length mRNAs (using procedures in Smart-seq229 as template), and the other uses UMIs and sequences only the 3′ end of the mRNA (using the Chromium chemistry by 10x Genomics as template). In the pre-amplification step, we provide option of using linear amplification to mimic the CEL-seq protocol. As shown in Fig. We simulate two categories of library preparation protocols, one does not use UMIs (unique molecular identifiers)α.(1) Capturing step: molecules are captured from the cell with probability lenslope can be used to control the amount of length bias, and MaxAmpBias is used to tune the total amount of amplification bias. If using CEL-Seq protocol this step is the in vitro transcription (IVT) linear amplification.(2) Pre-amplification step: if using non-CEL-Seq protocols, this step involves N rounds of PCR amplifications. We introduce sequence-specific biases during amplification, which includes transcript length bias and other bias assigned randomly. Parameter 3) Fragmentation step: the mRNAs are chopped into fragments for sequencing. If sequencing full-length mRNA, all fragments with acceptable length are kept for sequencing. If sequencing only the 3′ end for UMI protocols, only fragments on the 3′ end are kept for sequencing. The lengths of the simulated transcripts are obtained from the human reference genome, and the fragmentation is calibrated so that the average fragment length is 400 bp, which is typical for RNA sequencing (Supplementary Note  Fragment(4) Amplification step: fragments go through another k rounds of PCR amplifications for all protocols, including CEL-Seq and non-CEL-Seq protocols.(5) Sequencing step: amplified fragments from the previous step are randomly selected according to a given value of sequencing depth, which is the total number of reads (fragments) to sequence.(6) After the sequencing step , we can get the UMI counts for UMI protocols and read counts for non-UMI protocols.Note that for simplicity, this pipeline omits several steps, including reverse transcription, and library cleaning up.The runtime to simulate a dataset is dominated by the simulation of these technical steps. The total time cost varies with mRNA capture efficiency and protocol. With one CPU core of an Intel(R) Xeon(R) CPU E5–2690 v4 @ 2.60 GHz, the runtime and memory consumed to simulate a dataset with 500 cells and 10000 genes for different parameters are shown in Table rate_2PCR, which equals to the average amplification rate across all genes) plus a bias term B. The bias term B ranges from -MaxAmpBias to MaxAmpBias, where MaxAmpBias is a user-specified parameter to represent the total amount of amplification bias in our system.During PCR amplification of the full-length cDNAs, the PCR amplification rate  can vary for different transcripts. As a result, some transcripts are over- and some are under-amplified. This causes the unwanted amplification bias. To simulate this, for each gene, its PCR amplification rate is set to a sum of a basal amplification rate . We use multiple thresholds ranging from 0.6 to 1 on the |LFC| to define a gene is DE, in order to avoid being biased with one single artificial threshold.Diff-EVFs give rise to differences between populations as well as DE genes between populations. DE genes by design are the ones with non-zero gene effect values corresponding to the Diff-EVFs Fig. , as the t-test. For edgeR, we used the quasi-likelihood approach (QLF) with cellular detection rate (the fraction of genes that are detected with non-zero counts in each cell) as covariate. For DESeq2, we use local for the fittype parameter, and we evaluate its performance, respectively, based on the output p-values and adjusted p-values, which serve as filtering of genes.DE genes in observed counts are detected, respectively, with edgeR, DESeq2, Wilcoxon test, and Student specificity and sensitivity, thus calculate the area under the ROC curve. (b) Negative of Spearman correlation between the p-values of each detection method and the log fold difference of the true expression levels. Genes with high log fold change in true transcript counts should correspond to low p-value if the DE method works well. As the inferred p-values and log fold change in true counts are expected to be anti-correlated, we take the negative of this correlation, such that higher value corresponds to better performance.The output from each DE method is a p-value for each gene, with smaller values meaning the gene is more likely to be a DE gene. We use two metrics to evaluate the performance of a DE method: (a) AUROC (area under receiver operating characteristic curve), where we apply different thresholds on the p-values to obtain different sets of predicted DE genes, and we can then plot ROC curves with different combinations of 1-https://github.com/dynverse/dynwrap, version 0.1.0) and dynmethods  to run the three trajectory inference methods compared in this manuscript: Monocle (version 2.6.4), Slingshot (version 0.99.12), and MST (a basic method implemented in dynmethods). All methods were run with default parameters. Both dynwrap and dynmethods are under the collection of R packages dynverse used in the manuscript by Saelens et al.52.We use the R packages dynwrap , we randomly sample N cells from the pool of 10,000 cells according to the probability vector.During our simulation to estimate the number of cells needed to detect a rare population, we simulate the random sampling process as follows: we start with a total of 10,000 cells for all five populations with 2000 cells for each population. We set probability vector of a cell belonging to each population as , where Population 2 is the rare population with smallest probability. For each randomization and given total number of cells Further information on research design is available in the Supplementary InformationPeer Review FileReporting SummarySource Data


By offering high sequencing speed and ultra-high-throughput at a low price, Illumina next-generation sequencing platforms have been widely adopted in recent years. However, an experiment with multiplexed library could be at risk of molecular recombination, known as “index switching”, which causes a proportion of the reads to be assigned to an incorrect sample. It is reported that a new advance, exclusion amplification (ExAmp) in conjunction with the patterned flow cell technology introduced on HiSeq 3000/HiSeq 4000/HiSeq X sequencing systems, potentially suffers from a higher rate of index switching than conventional bridge amplification. We took advantage of the diverse but highly cell-specific expression of antigen receptors on immune cells to quantify index switching on single cell RNA-seq data that were sequenced on HiSeq 3000 and HiSeq 4000. By utilizing the unique antigen receptor expression, we could quantify the spread-of-signal from many different wells  due to index switching. Based on full-length T cell receptor (TCR) sequences from all samples reconstructed by TraCeR and TCR gene expression quantified by Kallisto, we found index switching in all three batches of experiments investigated. The median percentage of incorrectly detected markers was estimated to be 3.9% (interquartile range (IQR): 1.7%-7.3%). We did not detect any consistent patterns of certain indices to be more prone to switching than others, suggesting that index switching is a stochastic process. Our results confirm that index switching is a problem that affects samples run in multiplexed libraries on Illumina HiSeq 3000 and HiSeq 4000 platforms. Next-generation sequencing technology has been improved greatly in terms of its data output, sequencing speed, read quality and cost. As the throughput of sequencing platforms is increasing, sample multiplexing becomes practical and widely adopted by many studies. A common way to perform multiplexing is to introduce unique indices into each adaptor during library preparation, so that multiple samples can be pooled and sequenced in parallel. After sequencing, reads can be assigned to their original samples based on the indices. However, the process comes with a risk of molecular recombination, known as “index switching”, which causes some of the reads to be assigned to an incorrect sample. In 2015, Illumina adopted a new cluster generating mechanism, known as exclusion amplification (ExAmp), as well as patterned flow cells in their HiSeq 3000/4000/X instruments. By increasing cluster density on the flow cell, these new advances facilitate higher data output, more sequencing reads and shorter running time, thereby reducing sequencing cost substantially. However, because of the increased concentration of free primers produced during library preparation, it has been suggested that this promising chemistry results in a higher level of index switching than conventional bridge amplification [Index switching may compromise applications that rely on accurate profiling of gene expression, especially those with the need to profile lowly expressed genes. For example, in differential expression analysis of RNA-seq data, it is usually considered to be a good practice to place samples randomly over a plate in order to minimize any potential confounding, such as the edge effect, a common phenomenon in immunoassays. To balance samples from the test group and the control group, sequencing pooled samples from each group on one lane is also preferred for the same reason. However, in an experiment impacted by index switching, those practices make it possible that reads originally from a sample in a test group might be incorrectly assigned to another sample in a control group, which weakens and distorts any real signal. In a single-cell RNA-seq experiment that mainly focuses on detecting heterogeneity among single cells, it is commonly required that hundreds or even thousands of cells are analyzed, and every one of them needs a unique index to be sequenced in a single run. Double indexing, introduced by Kircher et al. , increas13. Likewise, T cells deploy a similar strategy for creating receptor diversity and an estimated 1018 different T-cell receptors can exist [To allow immune cells to be able to recognize a wide range of antigens, each naïve B and T cell expresses a unique immunoglobulin receptor (consisting of heavy and light chain) or T cell receptor (consisting of α and β chain), respectively, through somatic recombination. For human immunoglobulin heavy chain region, there are 123–129 Variable (V) gene segments , 27 Divean exist .By using unique pairs of i5 and i7 index adaptors, the rate of index switching was estimated to be up to 2% for sequencing libraries that run on patterned flow cells using ExAmp cluster generation (2). Van der Valk et al.  discoverStudy participants provided written informed consent. The study was approved by Regional Committee for Medical and Health Research Ethics South-East Norway (2010/2720).The data used for this study was originally generated to profile the transcriptomics of disease-specific immune cells from intestinal biopsies or peripheral blood from celiac disease patients. We used the Smart-seq2 protocol (8) to perform single-cell RNA-sequencing on 465 immune cells. The immune cells analysed included 215 HLA-DQ2: gluten- tetramer-sorted T cells, 247 transglutaminase 2 (TG2)-specific plasma cells, and three unassigned cells in three batches. Correspondingly, Batch 1 contained one plate of 96 wells; Batch 2 contained two (Plate 2 and Plate 3); while Batch 3 contained two plates (Plate 4 and Plate 5). Except Plate 2 which only contained T cells from peripheral blood, T cells and plasma cells were randomly placed into 96-well plates as single cells by index sorting on a FACS Aria II cell sorter. The cells were retrospectively identified based on each cell’s high-dimensional immune phenotype. Double indexing was applied in library preparation, thereby every read had an index composed of two segments, indicating which row and column its source well was located. The indices were incorporated in PCR primers that were used at a final concentration of 125 nM. Free adaptors were removed by two rounds of purification with AMPure XP beads. The final sequencing library shows undetectable or negligible number of free adaptors assessed by High sensitivity DNA assay run on the BioAnalyzer . SamplesTo acquire more complete reference transcriptome for immune receptors, we downloaded human immunoglobulin (IG) and T cell receptor (TR) gene sequences in fasta format from IMGT , and useWe applied TraCeR  to reconBy using a double indexing strategy, all reads contain two indices that denote respectively a specific column and row. During demultiplexing, a read with a switched index would be incorrectly assigned to a well corresponding to the switched index. Since all index combinations were used exhaustively in all three batches, an index-switched read could not be easily identified. However, the marker gene affected by switching of one of the indices would be detected in wells in the same row or column of the origin well, but at a low expression level. Since TCR sequences are only shared between T cells belonging to the same clonotype, we expect the large majority of cells in a batch to have a unique TCR. This can be exploited to detect spread of signal originating from a well with a given TCR sequence to other wells in the same batch. For each TCR sequences detected in more than two wells in a batch, we defined the well with the highest expression in each batch as origin well. The relative expression of all wells that expressed identical TCR sequences was defined as the ratio of expression in each well to the origin well. Wells that used identical column or row indices as the origin wells were defined as within the cross, and all remaining wells were defined as outside of the cross. To show that index switching was a major cause of signal spread, we counted the number of wells within the cross that seemingly expressed the same TCR sequence as the origin well, and compared it with the number of wells expressing the same TCR sequence outside the cross.For each TCR marker in Batch 2 and Batch 3, we divided the above calculated relative expression in wells along the column of the origin well into those in the source plate and those in the other plate, then average the relative expressions in the two groups respectively. A paired sample t-test was conducted for testing if the estimated level of signal spreading in the source plate was higher than that in the other plate in the same batch, which was sequenced on the same lane.For each marker used for quantifying index switching, either V gene or full-length TCR sequence, we identified the source well where the cell expressing the marker was located. Based on the expression data, we identified a well as a source of a certain marker for index switching if it met all the following criteria. 1) A well was in a position where the marker was also detected in wells in the same row or the same column (center of the cross). 2) Expression level of the marker was five times higher in this well than in any other well in the same column or the same row. 3) The cell type of the cell that was placed in the well during library preparation must match the cell type of the corresponding marker. For example, a source must be T cell while using TR_V gene segment or TCR as a marker. While using a BCR as a marker, the source cell must be a plasma cell. 4) The expression level of the marker was the highest among all the corresponding TR_V alpha or beta markers expressed in this well.By assuming that the expression levels for each marker are independent, a marker was eliminated in the process of quantifying the rate of index switching if it complies to any of the following three requirements. 1) A marker was detected in less than three wells. 2) A marker was detected in more than three wells outside of the cross pattern. 3) More than two sources were identified for the same marker in a batch. In cases where both TCR sequence and TR_V gene used by it were qualified, we primarily used TR_V gene segments as markers since they are more readily detected than full-length TCR sequences and most TR_V gene segments were each only expressed by one T cell in the batch. However, in cases where the same TR_V gene segment was used by different TCRs or that cells belonging to the same clone and thus expressing the same TCR were present in the same batch, the V gene segment would be observed in wells outside of the cross pattern. Therefore, we used full-length TCR sequence reconstructed by TraCeR instead of V gene segment as a marker if a V gene segment was detected in more than one well outside of the cross pattern or more than two source wells were identified for a V gene segment.If a marker was detected in more than three wells outside of the cross pattern, or more than two sources were identified for the same marker in a batch, it was eliminated from further study for quantifying the rate of index switching. The reasoning was that their spreading signals are more likely to be confounded with each other’s, thus compromising the accuracy of quantification of index switching arising from a given well. When two sources of the same marker were identified in one batch, expressions of the marker on presumably overlapped positions were allocated proportionally to the sources’ expression, and then the level of signal spread was estimated for the two sources separately. The relative expression values in recipient wells (ratio of expression in the well to the expression in its source well) in the crosses for every identified marker were summed up by columns and rows, and then adjusted by dividing the number of times the well was exposed as a recipient of the identified marker.We used TraCeR (9) to reconstruct full length TCR sequences from all samples, including samples of plasma cells, because as a result of index switching we found reads mapped to TCR also in wells originally containing plasma cells. All the reconstructed receptors, including both productive and unproductive ones, were included. TCR sequences should be unique for most cells, although occasionally two cells share identical TCR sequences due to in vivo clonal expansion, these cases are relatively rare. However, we observed in our data that 182 unique full-length TCR sequences were reconstructed from 288 wells and that 76 of them were detected in more than one well. Closer inspection revealed that the shared TCR expression were most noticeable in plates belonging to the same batch and sequenced on the same lane . This leThere could be two major causes of signal spreading, either by index switching or by contamination. Index switching is a known problem associated with libraries sequenced on the Illumina HiSeq 3000 and 4000 platforms.Since switching of both indices is extremely rare, index switching will in most cases cause the signal to spread to wells in the same row (when column index is switched) or the same column (caused by switching of the row index). In other words, index switching will typical spread the signal from the source well to the recipient wells in a cross pattern. To verify that index switching is the main cause of the observed signal spreading, we quantified the spread of signal within the cross pattern and compared it with the signal spreading to wells outside the cross pattern. After excluding TCRs detected in less than three wells, 45 wells with the highest expression of each TCR markers were identified as origin wells. Note that these excluded TCR markers had generally very low expression—the median expression of these markers was 239 TPM (IQR 3–1531), while the median expression of the included markers was 7658 TPM (IQR 3658–14870). The same TCR sequence as in the origin wells was found in 320 out of 1104 wells (29%) in the same row or column of the origin wells, i.e. within the virtual cross patterns. In comparison, among the 6820 wells outside of the cross pattern, only 13 of them (0.19%) were found to express any of the 45 TCR markers . After nFurthermore, we excluded the origin wells and wells with 0 relative expression and then looked closely at the relative expression levels of the remaining wells. They were in most cases very low . 97.7% oIn two of the batches, there were two 96-well plates that were each individually sorted and processed independently, including cDNA amplification, fragmentation and index barcoding by PCR. Purified libraries from each of the plates were kept frozen without further PCR rounds. The final pooled indexed libraries from each of the plates were mixed immediately before sequencing. Thus, contamination should only cause spread of signal of wells within the same plate, whereas index switching would occur both within and across the plate boundary. Hence, we assume that all spread of signal across plate boundaries is caused by index switching whereas the spread of signal within the plate is caused by index switching and possibly in addition by contamination. To test if contamination aggravated the signal spreading, we calculated the rate of TCR signal spread from the well with the highest expression in a batch to wells that shared the same column index. The spread of signal within the source plate was not significantly more than across the plates  by a paiFor every reconstructed TCR sequence, we profiled its expressions in all wells. Indeed, when we closely examined the expression pattern of several TCR sequences that were frequently observed in our data, we found clear cross patterns in most cases . Apart fSince we have many different TCR sequences that were uniquely expressed by one or very few cells in each batch, we decided to use the immune receptor information to quantify index switching. In addition to TCR sequences, we also explore the probability for using TR_V gene segments and BCR sequences.Expression of 84 unique TR_V gene segments for TCR alpha and beta chains were detected and quantified from the 215 T cells by using Kallisto. For every single TR_V gene segment, we profiled its expression in all wells except for well A1 of every plate, where multiple cells were added as control. The results for 23 out of 84 V gene segments showed clear cross patterns in wells within the same batch . The sigTRBV20-1 was too uncertain to be used for quantifying index switching. By using TraCeR, we reconstructed four unique TCR sequences that used this TR_V gene segment, and then profiled the expression of each of the four TCR sequences. Comparing with the overlapped expression while using only TR_V gene segment TRBV20-1 J which would lead to a failure to assemble a full-length TCR chain.Full-length TCR receptor sequences reconstructed by TraCeR were used to distinguish those commonly used V gene segments based on additional information such as J-chain usage and the highly diverse junctional sequence. TRBV20-1 , the fouTRBV20-1  were cleTRBV20-1  was deteBCR sequences expressed by plasma cells were reconstructed by BraCeR(10) and used as a marker for further verification. Since there were no plasma cells in Plate 2 of Batch 2, it was applied only to Batch 3 of our experiment. To our surprise, 521 unique BCR sequences were observed in Batch 3, in which there were only 108 plasma cells. Upon closer inspection, 322 of the BCR sequences (64%) were only detected in one cell with such a low expression that the median expression for these BCR sequences was 13.2 TPM (IQR 4.4–114.5). We speculated that aside from real cases of lowly expressed BCRs, there were considerable misassemblies by BraCeR. The high expression level of BCR genes in plasma cells combined with index switching has led to the high number of lowly expressed ‘artificial’ BCR sequences misassembled by index-switched reads from several different source wells. We therefore decided to not include any BCR markers in the quantification of index switching. However, after elimination of the BCR sequences that were detected in one well only, the results for 41 out of 199 BCR sequences showed clear cross patterns in 2 merged plates , confirmAfter quality filtering, we had found in total 48 markers (15 TR_V gene and 33 TCR sequences) in 55 wells that showed unambiguous signs of spread of signal caused by index switching. This relatively large number of different markers allowed us to quantify and compare index switching between batches and between different indices. For every identified marker, we quantified the level of signal spreading by summarizing its relative expression in its recipient wells. The median percentage of incorrectly detected markers was estimated to be 3.9% (IQR: 1.7%-7.3%). For each sequencing platform, the mean level of signal spreading and systematic variation across all identified markers were assessed. We found no significant difference in signal spreading levels between Illumina HiSseq 3000 and HiSseq 4000 platforms ,  A marker was detected in less than three recipient wells. 2) A marker was detected in more than three wells outside of the cross pattern. 3) More than two sources were identified for the same marker in a batch.Some markers were found infrequently and in general the expression levels of these markers were very low, even in the presumptive source well with the highest expression level. This is probably due to low number of transcripts amplified in the source well during library preparation or possible misassembly during reconstruction of the TCR marker. In both cases, the information of index switching we could get by using these markers would be unreliable, as the relative expressions of the marker would be confounded either by the detection efficiency or the rate of misassembly. Therefore, we imposed the first filter to exclude those markers. However, we might introduce a slight selection bias in our estimation of index switching rates by omitting these infrequently observed markers.There could be several causes for a marker to be detected in other wells than the source well and its cross. The most common scenario is that the marker is not specific enough, i.e. a V gene segment was used by multiple TCRs, or biological multiplication due to clonal expansion. Most of the cells that substantially expressed the same marker would be identified as sources, but we may fail to identify low-expression markers as the spread of signal could be too low to be detected. Failing to identify such a source would inflate the estimation of index switching rates since some of the signals we detected within the crosses could represent either additional unidentified sources or spread of signal from those. We therefore employed the second filter (excluding markers detected in more than three wells outside of the cross pattern) to eliminate these ‘unspecific’ markers from the quantification process. In addition, other causes such as sample contamination, dual index switching and misassembly, could also lead to signals outside the cross pattern. Though it is impossible to measure the effect of those potential causes in our experiment setting, it would be alleviated by the second filter. However, despite the second filter for marker selection, and that the result of the paired samples t-test did not indicate contamination to such an extent that it would confound the index switching effect, we cannot completely rule out the possibility of contamination, which might slightly inflate our estimated level of index switching. Similarly, unidentified additional source wells within the cross could also inflate our estimate.Although unlikely, it is possible that index switching occurs at both ends of a read. We were unable to measure it in this study, since with such a low frequency it might not be distinguishable from those caused by sample contamination. But it would slightly deflate our estimation if there were any.The tools that we used to reconstruct full-length antigen receptor sequence, i.e. TraCeR and BraCeR, are designed for single cell RNA-seq experiments. They are internally using Trinity  to assemThere is a theoretical possibility that one index could convert to another as a result of PCR or sequencing errors and thus resulting in a cross pattern that is indistinguishable from that caused by index switching. However, due to the highly distinct indices that were used in our experiments, we believe it is negligible. It is especially true for this particular study, where the measurements of expressions were based on either full-length antigen receptor chains or V gene segments which are approximately twice as long as a read (150 bp). Thus, in this setting, it seems improbable that a marker would be mis-assigned due to sequencing or PCR error since it is unlikely that the exact same error would occur in a sufficient number of reads covering the marker.It has been shown that the rate of index switching is dependent on the presence of free adaptors, usually primers with molecular size <100 bases . In our In this study, we estimated the level of read misassignment due to index switching in three single-cell transcriptomics libraries sequenced on the Illumina HiSeq 3000 or HiSeq 4000 sequencing platforms. We took advantage of the unique antigen receptor genes expressed by T cells and B cells and could thus quantify the level of signal spreading from 48 different gene markers in 55 different wells. The median percentage of incorrectly assigned markers was estimated to be 3.9% (IQR 1.7%-7.3%). There was no significant difference in index switching level between Illumina HiSeq 3000 and HiSeq 4000 sequencing platforms. Furthermore, we did not detect any consistent pattern of certain indices to be more prone for switching than others, suggesting that index switching is a stochastic process. Our study confirms that index switching is a problem that affects all samples run in multiplexed libraries on Illumina HiSeq 3000 and HiSeq 4000 platforms, and suggests that immune cell receptor information can be utilized to quantify the level of index switching for a given setup of single cell experiments.S1 Table(DOCX)Click here for additional data file.S2 TableNumber on diagonal position is number of unique TCRs detected in each plate. Others are numbers of common TCRs in two corresponding plates. Samples in Plate 2 and Plate 3 were indexed with the same set of column indices and sequenced on the same lane. Similarly, samples in plate 4 and plate 5 were indexed with the same set of column indices and sequenced on the same lane.(DOCX)Click here for additional data file.S1 FigCell type is labeled in the corresponding well, T for T cell, P for plasma cell, 0 for empty, 50 for mixture of multiple cells and X for unknown type. Ts for T cell identified as a source of signal spreading.(PDF)Click here for additional data file.S2 FigAfter Tagmentation, adaptors including indices were incorporated by PCR. The 96 samples from each plate were then pooled and free primers were removed by two rounds of purification with AMPure XP beads. The pooled libraries were quality-checked on the BioAnalyzer by the High-sensitivity DNA kit before kept frozen at -20C until sequencing.(PDF)Click here for additional data file.


Normalization of RNA-Seq data has proven essential to ensure accurate inferences and replication of findings. Hence, various normalization methods have been proposed for various technical artifacts that can be present in high-throughput sequencing transcriptomic studies. In this study, we set out to compare the widely used library size normalization methods  and across sample normalization methods  for RNA-Seq data using publicly available data from The Cancer Genome Atlas (TCGA) cervical cancer study. Additionally, an extensive simulation study was completed to compare the performance of the across sample normalization methods in estimating technical artifacts. Lastly, we investigated the effect of reduction in degrees of freedom in the normalized data and their impact on downstream differential expression analysis results. Based on this study, the TMM and RLE library size normalization methods give similar results for CESC dataset. In addition, the simulated datasets results show that the SVA (“BE”) method outperforms the other methods  by correctly estimating the number of latent artifacts. Moreover, ignoring the loss of degrees of freedom due to normalization results in an inflated type I error rates. We recommend adjusting not only for library size differences but also the assessment of known and unknown technical artifacts in the data, and if needed, complete across sample normalization. In addition, we suggest that one includes the known and estimated latent artifacts in the design matrix to correctly account for the loss in degrees of freedom, as opposed to completing the analysis on the post-processed normalized data. Demand for revolutionary technologies to deliver fast, inexpensive and accurate information has accelerated the development of high throughput sequencing (HTS) technologies. In the last five years, massively parallel RNA sequencing (RNA-Seq) has allowed for the large scale characterization of the transcriptomic landscape of cancer. Subsequently, many methods have been developed that not only provide accurate measurements of transcript abundance ,2, but aIn completing RNA-Seq studies, variability in measurement can be attributed to both the biological and technical factors. Sources of technical variation include: differences in library preparation across samples , sequencTable 1 presents the most recent proposed normalization methods for these three types of normalization approaches. In the last few years, a number of comprehensive studies have done to compare different normalization approaches to treat RNA-Seq data and their impact on the downstream analysis, where several publications have focused on the normalization methods that correct for differences in library size and/or gene length [In recent years, various normalization methods were proposed to correct for technical biases. Type of technical biases can be divided into three general varieties: (1) normalization for gene length, (2) normalization for library size , and (3) normalization for known or unknown technical artifacts across samples. Such normalization methods differ in the type of biases adjustment and statistical assumptions. e length ,22–24. Ie length ,18,25–27Therefore, the goals of this study are the following: 1) provide a review of existing methods for normalization of RNA-Seq data; 2) to assess the various methods for library size normalization using a large expression study in data; 3) to assess the various methods for determining latent or unknown artifacts in the data using both real and simulated data; and 4) to assess the impact of not accounting for the loss of degrees of freedom due to normalization and the impact on the type I error rate for differential expression analysis. The methods are compared using data from The Cancer Genome Atlas (TCGA) cervical study (CESC) . MoreoveFig 1). Here workflows 1 and 2 compare the effect of library size normalization methods on the differential expression analysis, while workflow 3 compares different methods to estimate the latent artifacts followed by across sample normalization for these unknown factors, and also considers the impact of not accounting for loss of degrees of freedom due to normalization for differential expression analysis. Note workflows 1–3 are used by TCGA cervical study, and for the simulation study, only workflow 3 is considered.In this section, we describe the normalization methods used in our study as well as the specific criteria used in our comparison. We also discuss the TCGA study and the simulation study used to assess the methods. We examined the effects of different normalization methods on the differential expression analysis using three analysis workflows . The goal of library size normalization is to make the library sizes comparable by scaling raw read counts in each sample by a single sample-specific factor reflecting its library size. There are three commonly used methods: Upper Quartile (UQ), Trimmed Mean of M-values (TMM), and Relative Log Expression (RLE).One source of variation between samples is the difference in library size, where library size is the total number of reads generated for a given sample. Difference in library size can be due to many factors, including differences in multiplexing of samples  or global differences in gene expression levels (Upper Quartile (UQ): Under this normalization method, after removing genes having zero read counts for all samples, the remaining gene counts are divided by the upper quartile of counts different from zero in the computation of the normalization factors associated with their sample and multiplied by the mean upper quartile across all samples of the dataset [EDASeq and edgeR Bioconductor packages [ dataset . This nopackages ,33.Trimmed Mean of M-values (TMM): This normalization method is based on the hypothesis that most genes are not differentially expressed (DE). For each sample, the TMM factor is computed while one sample is considered as a reference sample and the others as test samples. For each test sample, after exclusion of the most expressed genes and the genes with the largest log ratios, TMM is computed as the weighted mean of log ratios between this test and the reference. Because of the low DE hypothesis, the TMM should be close to 1. If it is not, its value provides an estimate of the correction factor that must be applied to the library sizes [edgeR Bioconductor package as the default normalization method [ry sizes . This non method .Relative Log Expression (RLE): Similar to TMM, this normalization method is based on the hypothesis that the most genes are not DE. For a given sample, the RLE scaling factor is calculated as the median of the ratio, for each gene, of its read counts over its geometric mean across all samples. By assuming most genes are not DE, the median of the ratio for a given sample is used as a correction factor to all read counts to fulfill this hypothesis [DESeq and DESeq2 Bioconductor packages [pothesis . This nopackages ,35.As the library size normalization methods mostly correct for sequencing depth and fail to adjust for other technical variations, across sample normalization methods have been proposed to correct for other technical artifacts to improve data quality and ability to detect biologically relevant genes. However, such technical sources of variation become problematic to deal with when they are correlated or confounded with a primary biological factor of interest; therefore good experimental design is essential when completing HTS studies. Thus, in completing normalization for technical artifacts we often assume that the primary factor of interest is independent of all artifacts ,17,25. AComBat, was proposed to provide more robust batch effect adjustments with small sample sizes [ComBat approach places less emphasis on the biological model and mostly reducing global variation even without specifying a biological model. In addition, the ComBat approach uses an empirical Bayes method to avoid over-correcting the known technical artifacts, which is critical for use with small batches (or sample size).In contrast to the more complex modeling methods is the approach involving the direct adjustment for known technical artifacts within the appropriate statistical model . As an example, a model with the primary biological factors of interest can be included in the design matrix along with the known technical artifacts, which may improve the precision of the comparisons of interest ,36. In ale sizes ,27. HoweRecently, normalization methods have been developed to assess and remove unknown technical variations by estimating the latent factors to capture these sources of variation. Several of these methods rely on singular value decomposition (SVD) or some other factor analysis techniques to determine the unwanted variation directly from the data. One issue with applying these approaches is the difficulty in distinguishing the unwanted technical variation from the biological factors of interest. Hence, remove unwanted variation (RUV) method was proposed to adjust for unknown technical variations by performing factor analysis on negative control genes  ,37. TherComBat) is used to obtain the normalized data.After estimating the latent factors using RUV, SVA and PCA approaches, an appropriate statistical approach : Under this approach, the factors of unknown technical variations are estimated and removed by performing the factor analysis on suitable sets of negative control genes or samples by keeping the primary factor of interest. Therefore, RUV [ore, RUV  method iore, RUV ,25.Surrogate Variable Analysis (SVA): In this approach, the unknown technical variations or “surrogate variables” (SV’s) are estimated by applying SVD on the computed residual matrix and selecting significant eigenvectors [nvectors ,25,42. Tnvectors ,26,43. Tnvectors  to estimPrincipal Component Analysis (PCA): This approach is completed by applying SVD to the scaled residual matrix to estimate the factors of unknown technical variations [riations . One canriations –47; or dFor practical purposes it is more convenient to perform downstream analyses on the batch adjusted or normalized data without further consideration of technical artifacts effects. However, adjustment for technical artifacts reduces the effective degrees of freedom in the dataset and thus changes the null distribution of the test statistics. Not accounting for this change in the degrees of freedom due to normalization for batch effects may lead to increase the false positive rates, especially when the primary factors of interest are not equally represented in all batches or batch effects act as a confounder . It shouhttps://gdc.cancer.gov/) (July 2017). The large-scale study the size of the TCGA unavoidably generated technical artifacts. These factors  were tracked for each sample with this information contained within the original TCGA ID. These known factors can also be downloaded from MBatch, a web-based analysis tool for normalization of TCGA data developed by MD Anderson. (https://bioinformatics.mdanderson.org/tcgabatcheffects).To compare the normalization methods and their impact on the differential expression analysis, publically available data from the TCGA cervical study (CESC) was used . Level 3For the CESC study, gene expression data was measured on 60,433 genes and 178 cervical tissue samples . The integrative clustering analysis reported in the main CESC TCGA paper used mRNA, DNA methylation, miRNA and copy number variation data identified two squamous-carcinoma-enriched groups and one adenocarcinomas-enriched group . The twoAn extensive simulation study was completed to compare the performance of SVA (“BE” and “Leek”) and PCA (based on different percent of variation) methods to identify the number of latent factors , and determine SVs, where the residual RUV method was also included. Then, the performance of different across latent factor identification methods were followed by normalization compared using Euclidean distance. The simulation study also investigated the impact of not accounting for the loss of degrees of freedom due to normalization on testing . The empirical type I error rate was computed for each “null” gene in which the proportion of the simulated datasets  with differential gene expression p-value less than 0.05. Then, for the set of simulated null genes, the average type I error rate was computed by averaging the individual “null” gene type I error rates. In the simulation of the data, we considered two main scenarios: (I) only the batch effect(s) simulated ; and (II) batch effects plus the effect of a biological variable of interest, where batch and biological effect were uncorrelated. The technical artifact was simulated to represent different mechanisms: discrete number of batches or runs of the samples  or a trend effect due to time of run with a continuous effect. The primary biological effect simulated was a binary factor, such as a treatment group and a control group. Note that in the simulation study, the genes with p-values less than 0.05 are considered to be DE.g = 1,…,G genes, n = 1,…,N samples, k = 1,2 biological groups, and l = 1,…,L batches. Let xgnkl be the count for gene g in biological group k, sample n, and batch l, with a Negative Binomial distribution: xgnkl~NegBin. The parameters μgkl and φg are the mean and dispersion, respectively. Under each scenario (I or II), we changed the sample size , the percentage of genes were affected by the batch effect(s) , and the percentage of DE genes . For each scenario, 1,000 datasets were generated, where for each dataset we simulated expression levels for G = 1,000 genes. The baseline parameters We have ωj =  and γj =  represent the jth effect of batch or primary biological variable, and J represents the total number of variables as batch and primary biological in the study (see S1 Table).Under the scenario I, we considered the binary batch , continuous batch , and both binary and continuous batches. While under the scenario II, we took into account the effect of primary biological factor along with the batch effects (both binary and continuous). Then, the non-null genes  were affected by the batch effect(s) or/and primary biological factor were generated using a mean shift l = 1,2),γ1 was generated from the Bernouli distribution and ω1 was generated from the Normal distribution . While for the continuous batch variable ,γ1 was generated from the standard Uniform distribution, U, and ω1 were generated from the Normal distribution . If we consider both binary and continuous variables (J = 2), then γj and ωj for j = 1,2 were generated as explained before. Lastly, for the scenario II, to take into account the effect of biological factor along with the batch effects (both binary and continuous), then γ3 and ω3 were generated from the Bernoulli distribution and the Normal distribution , respectively. The reason of changing the values of σ is that to provide the moderate effects for batch and primary biological variable under each scenario. The code to generate the simulated data is available at For the scenario I, to generate only the binary batch variable . To compare the library size normalization methods in terms of their impact on the differential expression analysis results, we use the edgeR Bioconductor package, that implements a Negative Binomial distribution where the empirical Bayes method is used to estimate dispersion parameters [limma by fitting the model on: (1) the normalized data  where the design matrix contains only the primary factor of interest, and (2) the log-transformed UQ normalized data where the known batch ID and estimated batch effects are included in the design matrix along with primary factor of interest. Note that the SVA method uses the log-transformed data to estimate the SVs, then, the log-transformed data is used instead of counts data to compare the performance of different methods to estimate the latent artifacts. To compare the normalization methods in terms of their impact on the differential expression analysis results, we considered the number of DE genes and number of common DE genes found among the methods. The raw p-values were adjusted for multiple comparisons by the Benjamini-Hochberg (BH) procedure [The filtered raw counts data from the CESC TCGA study were used to compare the effect of different normalization methods for library size and technical artifacts using three analysis workflows , with 4,612 and 5,454 of the DE genes (adjusted p < 0.05) overlapping between methods. Most of the differences between these two workflows were due to the inclusion of batch ID in the model. Library size normalization using the UQ found the DE genes 5,140  and 6,302  from the analyses with and without adjusting for batch ID, respectively; while, the TMM  and RLE  had the fewest DE genes detected, regardless of adjustment for batch ID. However, the number of DE genes did not vary much between any of the methods. Lastly, we observed that regardless of library size normalization method, the adjustment for known technical artifact resulted in a decrease of approximately 20% detected DE genes .We compared the performance of three library size normalization methods, UQ, TMM, and RLE. Analyses with and without inclusion of the known technical artifact (batch ID) (workflow 2 and 1) showed quite similar results for the different library size normalization methods  after using UQ library size normalization method. This approach determined three significant latent factors, while under the “BE” approach estimated 24 SVs. Thus, for the remaining methods  which do not determine the number of significant SVs, we estimated the top three latent factors to correspond with the number of SVs determined from the SVA (“Leek”) method. For the PCA method, the top three PCs each explained 10%, 6.2% and 5% of the proportion of total variation. A comparison between the top estimated latent factors between the methods showed that the SVA (“Leek”) and the residual RUV estimates were highly correlated (r = -0.88 and p < 2.2e-16). In addition, the top estimated latent factor based on the PCA were significantly correlated with the top estimated latent factor based on the residual RUV (r = 0.42 and p = 4e-07), and the top SVA estimated latent factor .Fig 3 represents the 2D plot of first two PC’s for raw counts compared with the UQ library size normalization followed by use of PCA to demonstrate the effect of across sample normalization, where the data was normalized using a model that adjusted for known batch ID and estimated technical artifacts. Fig 3C shows the large effect of batch ID with high proportion of variation for top PC that leads to consider the workflow 2. After adjusting the library size and removing the effect of batch ID and estimated SVs (using PCA method) through normalization, we observe a clearer separation of samples with regards to the biological factor  with less clustering of samples based on batch ID . We found similar results for the residual RUV and SVA (“Leek”) methods .Fig 4A presents the overlap of DE genes (adjusted p < 0.05) between the various methods to determine unknown technical artifacts, where design matrix contains only the primary factor of interest, while in Fig 4B presents the results following the log-transformed UQ normalized data, where the design matrix contains the estimated latent technical artifacts and the batch ID along with the primary factor of interest. Fig 4A and 4B show a large overlap in the detected genes when normalization is completed for library size, known batch effects and unknown latent factors , compared to 3,593 genes when using the log-transformed UQ normalized data. As the Venn diagrams illustrate , there was a large overlap in the DE genes between using the normalized data and log-transformed UQ normalized data, where more DE genes were detected using the normalized data based on the SVA (“Leek”) (37%), while the PCA and the residual RUV methods show less percentage around 20%. The residual RUV and the PCA methods found the most DE genes 7,332  and 5,796 , respectively, while SVA (“Leek”) detected 2,111  and 1,037  genes not detected by any of the other three analyses. Lastly, the residual RUV and PCA analyses produced more similar DE results, compare to SVA (“Leek”).Next, the effect of these methods on the differential expression analysis was assessed under the workflow 3, where the goal is to assess the impact of correctly accounting for loss of degrees of freedom due to normalization. Fig 5). The SVA (“BE”) procedure outperformed the other methods by correctly estimating the number of SVs in 70–100% of simulations. The PCA approach with SVs determined if explained > 7% of variation also behaves well in correctly estimating SVs when the percentage of genes were impacted by the batch effects was increased. The SVA method (“BE” or “Leek”) and the PCA based approach with SVs determined based on explaining >8% or >7% of variation perform similarly when the sample size and the percentage of genes affected by batch effects was large . Moreover, the percentage of correctly estimated SVs by SVA (“Leek”) is greater than 80% for most of the scenarios, while this percentage decreases significantly in the case when using both binary and continuous batch effects.To compare the performance of different methods in estimating technical artifacts, firstly we calculated the percentage of correctly estimated number of SVs. The SVA (“BE” and “Leek”) and PCA (based on different percent of variation) were considered, while in the residual RUV approach the number of factors of unwanted variation needs to be specified in advance . We founFig 6, where both binary and continuous batch effects were used, there was more variation in the SVA (“Leek”) method compared with other methods for estimating the SVs, while the residual RUV and SVA (“BE”) methods perform similarly. The PCA (7% of variation) performed similar to the other methods, especially when the percentage of genes was affected by batch effects was large. Lastly, considering other types of batch effects  may support similar results in assessing Euclidean distance between normalized and baseline data . Note that the outliers in the boxplots of Euclidean distance demonstrate the less similarity between the batch effect normalized data and the baseline simulate data that depends on the truly estimate of number of SVs along with how the estimated SVs can capture the technical variation.After estimating the SVs, we normalized the data by fitting a linear model for log-transformed data, where the design matrix included the primary factor of interest and the estimated latent factors. The performance of different across latent factor identification methods followed by sample normalization  were compared using the Euclidean distance as the similarity metric to measure the similarity between the normalized and baseline simulated data . To estimate the SVs based on the residual RUV method, the number of estimated SVs through the SVA (“BE”) method was used as the number of factors of technical artifacts. As presented in S6 Fig). Fig 7 shows that using the batch effect normalized data increases the average type I error rate significantly (rates as high as 30%). In addition, The SVA “Leek” is the worst performing method in accounting for loss of degrees of freedom due to normalization, since for some simulated datasets, this method considerably overestimates the number of SVs compared with other methods. Moreover, changes in the proportion of DE genes do not show any noticeable differences in the type I error rate. However, as expected the impact of decreases in the average type I error rate in the case of large sample sizes (N ≥ 200).The last aspect of the simulation study was designed to assess the impact on association analysis the lack of correctly accounting for the loss of degrees of freedom due to normalization. For each simulated data, we considered two approaches to determine DE genes : (1) analysis based on post-normalization and (2) normalization for batch effects is completed through the design matrix that also includes the primary variable of interest. Note that under the workflow 3, the UQ normalization method was not considered for the simulated data. Comparison of the approaches in terms of the average of empirical type I error rates among the 1,000 simulated datasets . As we mentioned before, it is important to keep in mind that TMM and RLE methods rely on an assumption that most genes are equivalently expressed in the samples [In applying the library size normalization methods  to the CESC data, we showed that these methods performed similarly , such as the percentage of genes affected by batch effect, the type of batch effects, and the sample size. The SVA (“BE”) procedure outperformed the other methods by correctly estimating the number of SVs, while increases in the sample size and the percentage of genes affected by batch effects can improve the performance of PCA based approach with SVs determined based on explaining >8% or >7%. In addition, comparing the performance of SVA (“BE” and “Leek”), PCA-7% variation, and residual RUV methods followed by sample normalization using Euclidean distance showed that SVA (“Leek’) method had more variation compared with other methods. Moreover, using the normalized data (SVA “Leek”) increased considerably the average type I error rate due to overestimate the number of SVs for some simulated datasets.Lastly, in completing the DE analysis step, if possible we suggest to incorporate the estimated unknown technical variations along with the known technical variation (batch ID) and the primary factor of interest in the design matrix to correctly reflect the loss of degrees of freedom due to the normalization. The accounting for the loss of degrees of freedom is often overlooked, resulting in researchers overestimating their statistical power. Thus, it is imperative for RNA-Seq studies to incorporate good experimental design principals to reduce the need for across sample data normalization.We recommend the assessment and, if required, the use of across sample normalization methods, in addition to the library size normalization, in the analysis of RNA-Seq data. As often one does not know all the technical factors that might impact gene expression experiments, assessment of both known technical artifacts  and unknown latent factors is recommended. In addition, we recommend that instead of using the normalized data in differential expression analysis, that researchers include the known technical artifacts as the covariates along with the primary factors of interest in the design matrix, which correctly accounts for the loss of degrees of freedom. Lastly, the ability to complete normalization does not replace the need for the incorporation of good experimental design principals in completing RNA-Seq experiments.S1 FigA) FPKM/RPKM normalization for gene length. (B) Illumnia sample multiplexing overview; two representative DNA fragments from two unique samples, each attached to a specific barcode sequence that identifies the sample from which it originated (i). Libraries for each sample are pooled and sequenced in parallel. Each new read contains both the fragment sequence and its sample identifying barcode (ii). Barcode sequences are used to de-multiplex, or differentiate reads from each sample (iii). Each set of reads is aligned to the reference sequence (iv).((TIF)Click here for additional data file.S2 FigA, C) residual RUV and  SVA (“Leek”) through normalization. Each point is colored based on the two keratin groups  and 14 levels of known batch ID .Plots of the top two PCs with the proportion of total variation on CESC data to cluster primary factor of interest (keratin-low and keratin-high groups) after adjusting the library size (UQ) and removing the effect of batch ID and estimated SVs by use of ((TIF)Click here for additional data file.S3 FigA) PCA, (B) residual RUV, and (C) SVA (“leek”) for CESC data: (blue) the normalized data , where design matrix contains only the primary factor of interest , (green) log-transformed UQ normalized data, where the design matrix contains the estimated technical artifacts and the known technical artifact (batch ID) along with the primary factor of interest.Overlap DE genes (adjusted p < 0.05) found under the UQ library size normalization method and the methods for determining the latent artifacts: ((TIF)Click here for additional data file.S4 FigA) N = 20, (B) N = 50, (C) N = 100, and (D) N = 200.Euclidean distance between normalized and baseline data based on simulated data with binary batch: ((TIF)Click here for additional data file.S5 FigA) N = 20, (B) N = 50, (C) N = 100, and (D) N = 200.The performance of different across latent factor identification methods followed by sample normalization  were compared. Euclidean distance between normalized and baseline data based on simulated data with continuous batch: ((TIF)Click here for additional data file.S6 FigA) N = 50 and (B) N = 200. The horizontal dashed line shows the threshold 0.05.The empirical type I error rates among 1,000 simulated data sets: ((TIF)Click here for additional data file.S1 Table(XLSX)Click here for additional data file.S1 File(DOCX)Click here for additional data file.