In shotgun metagenomics, microbial communities are studied through direct sequencing of DNA without any prior cultivation. By comparing gene abundances estimated from the generated sequencing reads, functional differences between the communities can be identified. However, gene abundance data is affected by high levels of systematic variability, which can greatly reduce the statistical power and introduce false positives. Normalization, which is the process where systematic variability is identified and removed, is therefore a vital part of the data analysis. A wide range of normalization methods for high-dimensional count data has been proposed but their performance on the analysis of shotgun metagenomic data has not been evaluated.p-values and control the false discovery rate (FDR). Our results showed that the choice of normalization method has a large impact on the end results. When the DAGs were asymmetrically present between the experimental conditions, many normalization methods had a reduced true positive rate (TPR) and a high false positive rate (FPR). The methods trimmed mean of M-values (TMM) and relative log expression (RLE) had the overall highest performance and are therefore recommended for the analysis of gene abundance data. For larger sample sizes, CSS also showed satisfactory performance.Here, we present a systematic evaluation of nine normalization methods for gene abundance data. The methods were evaluated through resampling of three comprehensive datasets, creating a realistic setting that preserved the unique characteristics of metagenomic data. Performance was measured in terms of the methods ability to identify differentially abundant genes (DAGs), correctly calculate unbiased This study emphasizes the importance of selecting a suitable normalization methods in the analysis of data from shotgun metagenomics. Our results also demonstrate that improper methods may result in unacceptably high levels of false positives, which in turn may lead to incorrect or obfuscated biological interpretation.The online version of this article (10.1186/s12864-018-4637-6) contains supplementary material, which is available to authorized users. The gene abundances are then calculated by counting the number of reads matching each gene in the reference database. Finally, statistical analysis is used to identify the genes that have a significant differential abundance between the studied conditions.In shotgun metagenomics, microorganisms are studied by sequencing DNA fragments directly from samples without the need for cultivation of individual isolates . Since sGene abundance data generated by shotgun metagenomics is however affected with multiple sources of variability which makes it notoriously hard to interpret –12. A suA wide range of different methods has been applied to normalize shotgun metagenomic data. The majority of these normalization methods are based on scaling, where a sample-specific factor is estimated and then used to correct the gene abundances. One approach is to derive the scaling factor from the total gene counts present in the sample , 18. ThiComparisons of normalization methods have previously been done for RNA-seq data , 26 as wp-values and control the FDR. Our results showed that the normalization methods had a substantially different performance in identifying DAGs. Several of the methods demonstrated a high FPR, especially when the DAGs were distributed asymmetrically between the experimental conditions. In some cases, the high FPR also resulted in an unacceptably high FDR. TMM and RLE had the overall highest performance, with a high TPR, low FPR and a low FDR in most of the evaluated scenarios and are therefore recommended methods for normalization of gene abundance data. We conclude that the choice of normalization method is critical in shotgun metagenomics and may, if not done correctly, result in incorrect biological interpretations.To address this knowledge gap and to provide guidance in choosing a suitable data-driven normalization method, we have performed a systematic evaluation of nine methods on gene abundance data from shotgun metagenomics. The evaluation was performed on datasets formed by individual resampling of three comprehensive metagenomic datasets, thereby creating a realistic setting where the unique characteristics and variance structure of the data are preserved. The methods were evaluated based on their impact on the identification of DAGs by comparing their TPR and FPR as well as their ability to correctly estimate unbiased Yij be the counts for gene i=1,…,m in sample j=1,…,n. The scaling normalization methods will derive a sample-specific normalization factor, denoted Nj, while non-scaling methods will replace Yij with the normalized In this study we evaluate the performance of nine normalization methods for count data, representing gene abundances from shotgun metagenomics Table . Seven mTotal count derives the normalization factor Nj as the sum of all gene counts in a sample j [sample j , 18, 27,n 3.2.1)  using thMedian calculates the the normalization factor Nj as the median of genes counts that are non-zero in at least one sample, i.e.  3.10.5) .Upper quartile estimates the normalization factor Nj as the sample upper quartile (75th percentile) of genes counts that are non-zero in at least one sample [e sample , i.e. \ 3.10.5) .TMM calculates the normalization factor Nj using a robust statistics based on the assumption that most genes are not differentially abundant and should, in average, be equal between the samples [r is chosen as reference. For each sample j, genes are filtered based on their mean abundance and fold-change between the sample and the reference. An an adjustment  samples . First, bundance .RLE assumes most genes are non-DAGs and uses the relative gene abundances to calculate the normalization factor [i by taking the geometric mean its abundances across all samples. The normalization factor Nj is then calculated as the median of all ratios between gene counts in sample j and the reference, i.e. n factor . First,  1.14.1) .CSS is based on the assumption that the count distributions in each sample are equivalent for low abundant genes up to a certain threshold lth quantile across all samples is calculated. The threshold j is then computed as the sum over the genes counts up to the threshold the data . First,  1.10.0) .Reversed cumulative sum scaling (RCSS) is a variant of CSS that utilize the observation that high-abundant genes in shotgun metagenomic data have, in general, a lower coefficient of variation [Nj as the sum of all genes with an abundance larger than the median. The normalization factor is thus given by, ariation . RCSS thn 3.2.1)  using thQuantile-quantile normalizes the data by transforming each sample to follow a reference distribution [lth quantile in the jth sample. The counts Yij are then replaced by Yaj=Ybj for any a,b, such that a≠b, the choice of which gene receives which quantile is made randomly. We implemented quantile-quantile in R (version 3.2.1) [ribution . The refn 3.2.1)  adapted n 3.2.1) . In ordeRarefying is a normalization method that discards fragments from each sample until a predefined number of fragments is the same for all samples [ samples , 23. ForYij in gene i and sample j was modeled using a over-dispersed Poisson generalized linear model (OGLM) [αi is the log of the baseline counts expected for a gene i, βi is the effect parameter that describes the relative abundance of gene i between the two conditions, and xj is an indicator function, such that xj=1 if sample j belongs to condition 1 and 0, otherwise. The counts Yij is assumed to follow a Poisson distribution with a gene-specific scaling of the variance . Furthermore, Nj was set to the factor corresponding to the method used to normalize the data. For method that does not use a normalization factor , Nj was set to 1. The model parameters αi and βi were estimated using maximum likelihood. Then, a gene is classified as a DAG using an F-test, which decides whether the model with an effect parameter is a better fit than the model without. FDR was estimated using the Benjamini-Hochberg algorithm [The number of counts l (OGLM) , 32, i.elgorithm . The OGL7 high quality reads per sample. Reads were mapped to a common gene catalog and quantified. The gene catalog was in turn mapped to eggNOG database v3.0 [5 reads per sample. The reads were downloaded from MG-RAST database [−5 were kept. The Marine dataset contains a set of samples from TARA ocean project, a large oceanic metagenome study with a total of 243 samples collected in 68 different locations across the globe [8 reads per sample. Reads were mapped to an oceanic gene catalog using MOCAT v1.2 [μm. For all datasets, genes with more than 75% zeros or mean abundance less than three were excluded from the analysis, resulting in 3573, 2345 and 4372 genes for Human gut I, Human gut II and Marine, respectively. The count data used in this study is available at [The normalization methods had their performance evaluated in three different publicly available metagenomic datasets, here denoted Human gut I, Human gut II and Marine. Human gut I contained 74 samples of sequenced DNA from gut microbiome of control patients in a type-2 diabetes study . The choase v3.0 . Human gase v3.0 . DNA wasdatabase , and tradatabase  using HMdatabase . Mapped CAT v1.2  using thlable at .m samples. Differentially abundant genes were introduced by random selection of genes that had their number of observed DNA fragments in one of the groups downsampled. Thus, for gene i and sample j, the counts Yij were replaced with a number generated by sampling from a binomial distribution, such that q is the effect size describing the average fold-change in abundance. In the evaluation, the number of samples in the groups as well as the total number of DAGs, the distribution of DAGs between the groups and the effect size q were varied.The normalization methods were evaluated on artificial data created by randomly sampling metagenomes without replacement from each of the comprehensive datasets. The artificial data was divided in two groups, representing two experimental conditions, each consisting of p-values calculated by the statistical analysis described above, the TPR and FPR at position k were calculated as TP(k) is the number of true positive above position k, FP(k) is the number of false positives above position k and #{DAGs} and #{non-DAGs} were the total number of DAGs and non-DAGs in the dataset, respectively. The true FDR (tFDR) at position k was calculated as k was chosen as follows: for the TPR analysis k corresponds to the position where FPR is 0.01, for the FPR analysis k corresponds to the position where TPR is 0.50, and for tFDR analysis k is the position where eFDR is 0.05.The performance of the normalization methods was evaluated based on the TPR, which represents the ability to correctly identify the DAGs, and on the FPR, which represents the amount of non-DAGs that were incorrectly identified as DAGs. Given a ranking list of genes sorted based on their lgorithm . All perIn this study, we compared the performance of nine normalization methods for shotgun metagenomic gene abundance data. The comparison was made on artificial data consisting of two groups, created by individual resampling without replacement of three comprehensive metagenomic datasets. In the resampling, DAGs were introduced by randomly selecting genes to have their number of counts in one of the two groups downsampled. Each artificial dataset was normalized using the nine different methods and the ability to correctly identify the DAGs was assessed. This set-up was used to investigate how the performance of the normalization methods changed under different characteristics of the data such as group size, proportion of DAGs and their distribution between the two groups.First, all methods were evaluated with the DAGs symmetrically distributed between the two groups. Here, 10% of the genes were selected to be DAGs with an average fold-change of 3 and the group size was set to 10+10. The Human gut I and Marine datasets showed the overall highest performance for detecting DAGs , while Human gut II, which had a substantially lower sequencing depth, had an average TPR of 0.61 , lightly-unbalanced effects , unbalanced effects  and heavily-unbalanced effects . First, the impact of unbalanced DAGs on the methods performance was measured in terms of TPR at a fixed FPR of 0.01  estimates the scaling factors directly from the absolute gene abundances. This makes it harder to exclude differentially abundant genes and as a consequence, the scaling factors may become biased, especially when the effects are asymmetric. The high performance of TMM and RLE observed in this study is in line with previous evaluations on other forms of count data. For example, McMurdie et al.  showed tOn the other end of the scale, normalization using quantile-quantile, median and upper quartile, as well as rarefying the data, had the overall lowest performance. The difference in TPR, compared to highest performing methods, was especially large in the heavily-unbalanced cases where DAGs were exclusively present in one of the samples. Taking the Marine dataset as an example, the TPR for these methods were 20 p.p. lower than TMM and RLE, which had the highest overall performance. All these methods also resulted in high FPRs that reached, in many cases, unacceptable levels. Among these low-performing methods, upper quartile and rarefying also resulted in inflated gene-gene correlation. Thus, quantile-quantile, median, upper quartile and rarefying are not recommended for normalization of metagenomic gene abundance data. Interestingly, the straight-forward total count method, which uses the total abundance of all genes in a sample as the basis for the normalization, had, overall, similar or higher performance than median and upper quartile. One argument for not using total count is that the sum of all gene abundances can be heavily dominated by the genes that are most commonly present in the microbial community. Instead, median and upper quartile should represent robust alternatives that avoid the most commonly present genes by replacing the sum with the 50th or 75th percentile of the gene count distribution as scaling factor. We did not, however, observe any tendencies that median or upper quartile had an overall higher performance than total count. On the contrary, the scaling factors estimated from total count and upper quartile had a very high correlation, which has also been shown in previous studies , suggestp-values for non-DAGs should, in theory, follow a uniform distribution. Our results showed that this was not the case. In the situations where the DAGs where distributed between the groups in a balanced way, all methods generated p-value distributions that were approximately uniform. However, when the DAGs were only present in one of the groups, the p-value distribution of the non-DAGs became skewed against low values. Metagenomic gene abundances are measured relatively to the sequencing depth and genes that are differentially abundant will therefore, indirectly, also affect non-DAGs. If a normalization method fails to compensate for this ’artificial’ effect, it may result in too low p-values for non-DAGs and, in turn, in an excessive number of false positives. Our results showed that quantile-quantile and upper quartile normalization methods had the most biased p-values, suggesting that this, at least partially, is likely the cause for their high FPR. Furthermore, previous studies have shown that statistical models for identification of DAGs in shotgun metagenomics can result in highly biased p-values if their underlying assumptions are invalid [p-values caused by invalid model assumptions, the skewed p-value distributions generated by improper normalization observed in this study, can not be addressed by replacing the parametric model (the overdispersed Poisson model), with e.g. a non-parametric method or a permutation-based approach.The  invalid . In part invalid , 43. It p-value distribution for non-DAGs. Biased FDR estimation may result in a large number of false positives genes, i.e. non-DAGs incorrectly reported as significant. Our results showed that all normalization methods achieved a correctly estimated FDR when the effects were balanced. However, similarly to the p-values, the FDR became biased when the DAGs were introduced in an unbalanced way. Already at the lightly-unbalanced case, where effects were added to 10% of the genes distributed 75%-25% between the groups, two methods  were unable to control the FDR for at least one dataset. At the unbalanced case , six of the nine methods resulted in considerably biased FDR estimates. Only TMM, RLE and CSS were able to correctly control the FDR and only showed a moderate bias at the heavily-unbalanced case . Several of the other methods however showed an unacceptable FDR bias. In particular, RCSS, quantile-quantile, upper quartile and total count had a true FDR close to 50% when the corresponding estimated FDR was fixed to 5%. Our results thus show that many normalization methods produce highly skewed p-value distribution, which results in biased FDRs, as soon as the DAGs becomes unbalanced between the groups. It is worth to note that changing the approach for controlling the FDR to the more conservative Benjamini-Yekutieli method or the Storey q-values method did not remove the bias or resulted in a considerably reduced statistical power , 2623, 2In conclusion, our evaluation showed that the choice of normalization method can greatly affect the quality of the results in the analysis of gene abundances in shotgun metagenomic data. When DAGs were asymmetrically distributed between experimental conditions, several well-established normalization methods showed a decreased TPR and an increased FPR. The high FPR resulted, for many methods, in an unacceptably biased FDR which can lead to a large number of false positives. The highest performing normalization methods in our study were TMM and RLE, and for larger group sizes CSS, which showed a high TPR and low FPR. These methods were also the best in controlling the FDR. Normalization is an essential step in the analysis of gene abundances in shotgun metagenomics. Our results emphasize the importance of selecting a sound and appropriate method for this task. They also demonstrates that the use of inappropriate normalization methods may obscure the biological interpretation of data. Further research for improved data-driven normalization of shotgun metagenomic data is therefore warranted.Additional file 1Figure S1. Histograms of Spearman correlations between normalization factors and raw counts of non-differentially abundant genes (non-DAGs). Spearman correlations were compute per gene in the Human gut I, for group size 10+10, with 10% of effects divided equally between the two group, and fold-change 3. Affected genes were randomly selected in 100 iterations. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), upper quartile (UQ), median (Med) and total count (TC). (PDF 76 kb)Additional file 2Figure S2. Scatterplot of normalization factors for each pair of scaling methods. Normalization factors estimated per sample in the Human gut I, for group size 10+10, with 10% of effects divided equally between the two group, and fold-change 3. Affected genes were randomly selected in 100 iterations. The number on the top-left of each plot indicates the Spearman correlation for the normalization factors presented in the plot. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), upper quartile (UQ), median (Med) and total count (TC). (PDF 316 kb)Additional file 3Figure S3. Mean Spearman correlation between raw and normalized counts. Spearman correlations were compute per gene before and after normalization in the Human gut I, for group size 10+10, with 10% of effects divided equally between the two group, and fold-change 3. Affected genes were randomly selected in 100 iterations. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), quantile-quantile (Quant), upper quartile (UQ), median (Med), total count (TC) and rarefying (Rare). (PDF 8 kb)Additional file 4Table S1. True positive rate at a fixed false positive rate of 0.01 for a group size of 10+10. (PDF 16 kb)Additional file 5Table S2. False positive rate at a fix true positive rate of 0.50 for a group size of 10+10. (PDF 16 kb)Additional file 6Figure S4. Effect size analysis of DAGs. Estimated effect size of differentially abundant genes (DAGs) (y-axis) for different distribution of effects between groups (x-axis): balanced (‘B’) with 10% of effects divided equally between the two groups, lightly-unbalanced (’LU’) with effects added 75%-25% in each group, unbalanced (‘U’) with all effects added to only one group, and heavily-unbalanced (’HU’) with 20% of effects added to only one group (x-axis). The results were based on resampled data consisting of two groups with 10 samples in each, and an average fold-change of 3. Three metagenomic datasets were used Human gut I, Human gut II and Marine. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), quantile-quantile (Quant), upper quartile (UQ), median (Med), total count (TC) and rarefying (Rare). (PDF 436 kb)Additional file 7Table S3. True false discovery rate at an estimated false discovery rate of 0.05 for a group size of 10+10. (PDF 16 kb)Additional file 8Figure S5. True false discovery rate for p-values adjusted using Benjamini-Yekutieli method at an estimated false discovery rate of 0.05 (y-axis) for different distribution of effects between groups (x-axis): balanced (‘B’) with 10% of effects divided equally between the two groups, lightly-unbalanced (’LU’) with effects added 75%-25% in each group, unbalanced (‘U’) with all effects added to only one group, and heavily-unbalanced (’HU’) with 20% of effects added to only one group. The results were based on resampled data consisting of two groups with 10 samples in each, and an average fold-change of 3. Three metagenomic datasets were used Human gut I, Human gut II and Marine. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), quantile-quantile (Quant), upper quartile (UQ), median (Med), total count (TC) and rarefying (Rare). (PDF 40 kb)Additional file 9Figure S6. True false discovery rate for p-values adjusted using Storey q-values method at an estimated false discovery rate of 0.05 (y-axis) for different distribution of effects between groups (x-axis): balanced (‘B’) with 10% of effects divided equally between the two groups, lightly-unbalanced (‘LU’) with effects added 75–25% in each group, unbalanced (‘U’) with all effects added to only one group, and heavily-unbalanced (‘HU’) with 20% of effects added to only one group. The results were based on resampled data consisting of two groups with 10 samples in each, and an average fold-change of 3. Three metagenomic datasets were used Human gut I, Human gut II and Marine. The following methods are included in the figure trimmed mean of M-values (TMM), relative log expression (RLE), cumulative sum scaling (CSS), reversed cumulative sum scaling (RCSS), quantile-quantile (Quant), upper quartile (UQ), median (Med), total count (TC) and rarefying (Rare). (PDF 132 kb)


Nuclear transfer techniques  are currently under development to provide a route to eliminating particular instances of mitochondrial disease from the germline. Before these kinds of techniques are implemented clinically it is of primary concern that their safety and efficacy is established. In a recent paper, Hyslop et al.  contains supplementary material, which is available to authorized users. Two main methods of mitochondrial replacement—pronuclear transfer (PNT) and maternal spindle transfer (MST)—are currently under development as potential germline therapies for eliminating some forms of mitochondrial disease. Hyslop et al.  examinedHowever, there a number of shortcomings to the analytical approaches undertaken. First, the power to detect differences between treatment groups is low due to the small number of biologically independent samples, which is at the level of blastocyst and not single cell sample. For instance, a test of the mitonuclear mismatching hypothesis would compare nine heterologous versus a maximum of two autologous/homologous blastocysts. Second, no statistical modeling of treatment effects was conducted, which obviously precludes the possibility of making any conclusions about whether or not there are statistical differences overall, or between specific treatment groups. The analyses only extend as far as plotting the results of a principal components analysis (PCA), which is a variance-orientated dimension reduction technique that can be useful for preliminary visualization of data.We investigated these issues using simulated datasets and subsequent power analysis and principal components analysis, and conclude that based on the number of samples included and the magnitude of effect sizes that might reasonably be expected to be present, the study is unable to provide clear evidence that the manipulated samples are indistinguishable from controls.The power to detect differential gene expression between treatments was examined via simulation, where simulated datasets based on the experimental design used here were analysed for differences between treatments using a mixed effects linear model. In order to resemble a transcriptomic analysis of differential gene expression, simulations were ran in batches of 100  and the power was calculated from each batch as the percentage of significant tests. These batches were repeated to produce 100 power estimates from simulated data. Two sets of simulations were ran: Set 1 tested a range of effect sizes, and Set 2 tested a range of sample sizes. All analyses used R v3.2.1 and the ‘lmer’ mixed modelling function in the ‘lme4’ package [Each simulated dataset was set up by initially specifying a small effect size for differences in gene expression between cell types, variance estimates  for botNext, the experimental design for each simulated dataset was set up as a balanced design, based on the numbers of samples in Hyslop et al.  , as well as blastocyst and error variance estimated from the data in Hyslop et al. . Note thIn the original manuscript, linear models such as those simulated here were not used to determine significance of differential expression between treatments. Instead, a principal components analysis (PCA) of the gene expression data was carried out, and the resulting scores of each sample along principal component vectors were plotted in order to visualize and distinguish between samples based on treatment group. We therefore followed this approach with a final set of simulations, where we generated gene expression values for a multivariate dataset of 12,000 genes  effect sizes were unusually strong  have a low effect size (Cohen’s d = 0.1) for differential expression between treatments Fig. , and wheOn the basis of the low power and the descriptive nature of the methods employed by Hyslop et al. , the con


Availability of large cohorts of samples with related metadata provides scientists with extensive material for studies. At the same time, recent development of modern high-throughput ‘omics’ technologies, including metabolomics, has resulted in the potential for analysis of large sample sizes. Representative subset selection becomes critical for selection of samples from bigger cohorts and their division into analytical batches. This especially holds true when relative quantification of compound levels is used.We present a multivariate strategy for representative sample selection and integration of results from multi-batch experiments in metabolomics.® was used and its p(corr) vectors were averaged to obtain combined metabolic profile. Jackknifed standard errors were used to calculate confidence intervals for each metabolite in the average p(corr) profile.Multivariate characterization was applied for design of experiment based sample selection and subsequent subdivision into four analytical batches which were analyzed on different days by metabolomics profiling using gas-chromatography time-of-flight mass spectrometry (GC–TOF–MS). For each batch OPLS-DAA combined, representative metabolic profile describing differences between systemic lupus erythematosus (SLE) patients and controls was obtained and used for elucidation of metabolic pathways that could be disturbed in SLE.Design of experiment based representative sample selection ensured diversity and minimized bias that could be introduced at this step. Combined metabolic profile enabled unified analysis and interpretation.The online version of this article (doi:10.1007/s11306-017-1248-1) contains supplementary material, which is available to authorized users. Increased availability of modern high-throughput ‘omics’ technologies resulted in the potential for generating massive amounts of chemical data. At the same time, availability of large cohorts of samples with related metadata has increased in recent years, providing scientists with extensive and well-described material for studies. These developments place additional requirements on study planning and execution, for which selection of relevant samples, extraction and integration of useful information from obtained data are important  to consider Hulley , but areWhen data complexity is reduced with multivariate characterization, several approaches for representative sampling from a multivariate space can be used. One possible method, the space-filling design, targets even distribution of the design points throughout the space of interest   is a chronic autoimmune disease predominantly diagnosed in women with very diverse manifestations as well as disease onset. It is a connective-tissue disorder characterized by immunological abnormalities and the involvement of a variety of organ systems, like skin, joints, kidneys, heart and the central nervous system  plasma samples were collected from all study participants according to standardized protocols and stored at −80 °C. All participants gave written informed consent to participate in the study which was approved by the ethical board at Karolinska University Hospital, Sweden.In this paper a multivariate approach was used to select a subset of samples from the entire Karolinska SLE cohort. Two hundred and five sample descriptors formed the basis for sample selection with the multivariate characterization approach. Since SLE predominantly affects women, only female samples were used in this study. The cohort was divided into five subgroups based on the patients’ antibody profiles. This division was inspired by previous observations . This procedure was repeated for each SLE subgroup and the control group, resulting in the selection of 23 samples from each subgroup. Only 22 samples were available from the aPL positives subgroup, and all were included for analysis. Sample selection produced 114 SLE samples and 23 controls for the study. This PCA-based sample selection procedure was repeated for sub-division of samples into four analytical batches analyzed at different days, with five to six samples from each SLE-subgroup and controls included in each batch.sk) is calculated and then each value for this variable is multiplied by 1/sk, average is not subtracted], and using peak areas of internal standards, was calculated. The t1-score vector from this model was used for normalization of the data which was done by dividing the all peak areas in each sample by its corresponding t1-score value . PCA was used for the sample selection procedure, and OPLS-DAFor each batch run order effect was checked by calculating a 1 + 1 OPLS model with all metabolite signals as X variables and sample run order as Y variable. For batch 4 high run order effect was observed (20%) as compared to other batches , which could be explained by the visible, uncontrollable drop in instrument sensitivity during the analysis.avg is the average standard error obtained from jackknife standard errors from each batch (SE1 − SE3). Results were presented as average p(corr) vector, with average confidence interval defining the significance of the metabolite in this vector. The average confidence interval was calculated by multiplying average standard error via the t-value  corresponding to the N − 3 degrees of freedom, where N is the total number of the cross-validation groups from all batches included in the study.Metabolites significant for the OPLS-DA models between SLE and controls were identified using confidence intervals calculated by multiplying the jackknife standard errors by the t-value  corresponding to the N − 1 degrees of freedom, where N is the number of the cross-validation groups. Jackknifing is a method for finding the precision of an estimate, by iteratively keeping out parts of the underlying data, making estimates from the subsets and comparing these estimates  clinical descriptors  that should fit into the score plot is based on whether a smaller space with less variation  or one covering wider variation within the samples available (wider square) is desired. The smaller square removes all outliers that could introduce unwanted variation in the subsequent data analysis, and is therefore the more optimal approach for pilot and exploratory studies. The wider square represents a wider spread of variation in the data and is better suited for more comprehensive studies with larger amount of analyzed samples .The samples were further separated into four batches using the same approach described above because of the limited, maximum, daily sample throughput of the GC–TOF–MS instrument. Five to six samples from each subgroup and the control group were analyzed in each batch. An uncontrollable drop in instrument’s sensitivity during the analysis was observed for batch 4, which was therefore excluded from further analysis.The selection method we used ensured that samples represented the multivariate space defined as samples and their related descriptors and hence were representative for the studied cohort. Dividing the samples into four batches representatively allowed for treatment of each batch as an individual study and gave the basis for independent data analysis. This approach provided full control over the analytical pipeline and ensured that, despite the exclusion of one batch from study analysis due to problems with instrument stability, remaining batches still carried information required by the study. If the samples had not been divided in a controlled manner, loss of samples due to uncontrollable factors could result in the need for a whole new analysis.2X = 0.4) revealed sample separation based on day of analysis as the main source of variation in the data . This finding suggested the need for individual analysis of each batch. Here we present multivariate approach for performing results compilation from different batches. We have focused on the differences between SLE patients and controls, since this was the first aim of the study, with the main assumption being that analysis of the metabolic profile differentiating SLE patients from the control would improve both disease diagnosis, as well as understanding of its pathology. With the balanced and representative number of samples from each disease sub-class, finding metabolic profile characterizing SLE is expected to be a reliable approach.To get a general overview and understanding of the spread of variability in the data, we performed combined analysis of all batches with PCA. In our study, the score plot from the PCA model of normalized data from three batches . A low correlation between p(corr) vectors showed that there were no strong metabolic differences between SLE and controls, what was especially seen in the weak model for the Batch 2. Still, obtaining a common metabolic profile could give relevant information about perturbations in metabolite levels between SLE and controls.To compare similarity of metabolic profiles between batches, we further investigated shared and unique structure plots (SUS-plots)  vector] in all the OPLS-DA models were selected. These metabolites were considered reliable, and formed a combined metabolic profile describing differences between SLE patients and controls. Their p(corr) values and jackknife standard errors were used to calculate the average p(corr) values and corresponding confidence intervals respectively, as summarized in Table In multivariate approach interpretation of the metabolic profile is based on an assumption that if metabolites involved in a certain metabolic pathway demonstrate changes that would not be anticipated by random chance, then this pathway is probably biologically or metabolically important. This assumption is valid even if a few single metabolites do not show significant changes.Our study showed decrease in levels of most free, long-chain fatty acids in SLE patients compared to controls. These results confirm the ones presented by Wu et al.  vectors were subsequently averaged to provide combined metabolic profile differentiating SLE patients from controls, which was later evaluated in relation to metabolic pathways that could be disturbed in SLE.Because of the applied methodology, which was based on strict control of each experimental step, we were able to obtain a reliable metabolic profile that characterized the comparison between SLE patients and controls. The work presented in this paper emphasizes the importance of applying multivariate approaches for representative sample selection and subsequent integration of ‘omics’ results obtained from different analytical batches. The applied data analysis approach may be used for compilation of results from different analytical batches and for combination of results from different studies. We believe that this methodology will lead to more reliable results and will enable not only comparison of data analyzed at different times, but also ones obtained from different research groups.Below is the link to the electronic supplementary material.Supplementary material 1 (DOCX 217 KB)


Carefully designed control experiments provide a gold standard for benchmarking different genomics research tools. A shortcoming of many gene expression control studies is that replication involves profiling the same reference RNA sample multiple times. This leads to low, pure technical noise that is atypical of regular studies. To achieve a more realistic noise structure, we generated a RNA-sequencing mixture experiment using two cell lines of the same cancer type. Variability was added by extracting RNA from independent cell cultures and degrading particular samples. The systematic gene expression changes induced by this design allowed benchmarking of different library preparation kits  and analysis pipelines. Data generated using the total RNA kit had more signal for introns and various RNA classes  and less variability after degradation. For differential expression analysis, voom with quality weights marginally outperformed other popular methods, while for differential splicing, DEXSeq was simultaneously the most sensitive and the most inconsistent method. For sample deconvolution analysis, DeMix outperformed IsoPure convincingly. Our RNA-sequencing data set provides a valuable resource for benchmarking different protocols and data pre-processing workflows. The extra noise mimics routine lab experiments more closely, ensuring any conclusions are widely applicable. Transcriptome profiling experiments are widely used in functional genomics research and have helped advance our understanding of gene regulation in health and disease. Throughout the evolution of this technology, from probe-based quantification on microarrays through to sequence-based transcript counting using second and third generation sequencing, researchers have conducted specially designed control experiments to benchmark different platforms and analysis methods. An early high profile example focused on the Affymetrix gene expression platform  using a mixture design, where two distinct samples are mixed in known proportions, inducing predictable gene expression changes across the entire series   , who des , rank correlations between microarray and qPCR platforms and consistency of differential expression results (amongst others). The study concluded that all platforms compared are capable of producing reliable gene expression measurements.The Microarray Quality Control Consortium (MAQC) used this design in a large scale inter-lab (11 sites), inter-platform comparison of 7 microarray technologies using commercially available bulk RNA sources  profiled as pure and mixed samples in 2 different proportions  . This prWith the advent of RNA-sequencing (RNA-seq), the MAQC project was extended by the sequencing quality control (SEQC) consortium  that usePrevious mixture experiments performed using either microarray or RNA-seq have a number of well-known limitations. The first is that the samples used are all identical, coming from the same source of bulk RNA, meaning that any variation observed is purely technical in nature. In practice, biological noise is a key source of variability in both microarray ,12 and RSaccharomyces cerevisiae , with the relatively small number of sex-specific genes providing inbuilt truth for methods comparisons ,15. Anotrevisiae . By analTo address some of the shortcomings outlined above, we designed and conducted a mixture experiment that simulated variability beyond the purely technical. This experiment compared two popular library preparation methods  using short reads (100 bp) obtained from the Illumina HiSeq platform. In the sections that follow we present details on the experimental design and quality control of this data set and results from the various methods compared using the inbuilt truth available from the mixture design. Popular differential expression analysis methods, differential splicing algorithms and deconvolution algorithms were compared using these data.et al. (2006)  . This de. (2006)  and SEQC. (2006) . A pilotTo obtain samples for the mixture study, cell lines from a range of passages (2–4) were grown on 3 separate occasions in RPMI media (Gibco) supplemented with Glutamax and 10% fetal calf serum to a 70% confluence. Cell lines were treated with 0.01% Dimethyl sulfoxide (Sigma), and after 6 h, cells were collected, snap-frozen on dry ice and stored at −80°C until required. Total RNA was extracted from between half a million and a million cells using a Total RNA Purification kit (Norgen Biotek) with on-column DNAse treatment according to the kit instructions. RNA concentration for each pair of replicates to be mixed was equalized using Qubit RNA BR Assay kit (Life Technologies) so that both samples in a pair had the same concentration . Replicates of pure NCI-H1975 (100:0) and pure HCC827 (0:100) and intermediate mixtures ranging from 75:25 to 50:50 to 25:75 , while the second aliquot was degraded by incubation at 37°C for 9 days in a thermal cycler with a heated lid (we refer to this as the ‘degraded’ replicate), with RNA integrity number (RIN) determined using TapeStation RNA ScreenTape (Agilent). A total of 10 μl (∼1 μg) from each replicated mixture (both good and degraded) were used for Next Generation Sequencing library preparation using two different protocols: Illumina's TruSeq Total Stranded RNA kit with Ribozero depletion and Illumina's TruSeq RNA v2 kit. Libraries were quantified and normalized by qPCR, as recommended by Illumina, and libraries prepared using the same protocol were pooled together. Library clustering was performed on a cBot with Illumina HiSeq SR Cluster kit v4 cBot. Each of the two pools of libraries was sequenced as single-end 100 base pair reads over 4 lanes on an Illumina HiSeq 2500 with an Illumina HiSeq SBS kit v4. Base calling and quality scoring were performed using Real-Time Analysis (version 1.18.61) and FASTQ file generation and de-multiplexing using CASAVA (version 1.8.2). Library quantification, clustering, sequencing, base calling and de-multiplexing were carried out at the Australian Genome Research Facility . This data set is available under GEO series accession number GSE64098.hg19 build of the human reference genome using the Subread and Subjunc software (version 1.4.6) with default settings (hg19 genome assembly) using the featureCounts procedure ; gene body annotation (reads overlapping any part of the gene body between the transcription start and end sites are counted); conservative gene-level intron annotation . Finally, we also produced a conservative intergenic count. First we produced a combined gene and intergenic annotation, which encompassed the gene body and the intergenic region preceding this gene along the genomic coordinates, regardless of the gene direction. We then counted all reads overlapping each of these amalgamated regions and subtracted the number of reads that overlapped the corresponding gene body, thus obtaining a conservative estimate of the number of reads overlapping the respective intergenic region (i.e. the reads that overlapped both the gene and the preceding intergenic region were only assigned to the respective gene). These counts were used to fit the non-linear models of gene expression (see below).FASTQ files from the same libraries were merged and aligned to the settings . Next rerocedure  in an uns Figure . To comps Figure , we summs Figure , gene-les Figure  were genet al. (2006) to get high precision estimates of the abundance of each gene in the two reference cell lines. For each gene g, let Xg be its expression level in NCI-H1975 and let Yg be its expression level in HCC827. The expression ratio (fold-change) between NCI-H1975 and HCC827 for that gene is therefore Rg = Xg/Yg.We use the non-linear model described in Holloway pi for the proportion of RNA from cell line NCI-H1975 in a particular RNA sample i, with 1 − pi being the proportion of RNA from cell line HCC827, for i = 1, …, 15. The expression level of gene g in the RNA mix must be piXg + (1 − pi)Yg. The expression FC between the RNA mix and the HCC827 reference must be {piXg + (1 − pi)Yg}/Yg = piRg + 1 − pi, which is an increasing function of Rg. This shows that, regardless of the true values of Xg or Yg, the expression values for each gene must change in a smooth predictable way across the mixture series.For each mixture series, there are 15 RNA samples in total. Write gi for the trimmed mean of M-values (TMM) normalized log2-count-per-million value obtained from voom for gene g and sample i. We fit the following nonlinear regression model to the log2-count-per-million values for each gene:gj represent measurement error and are assumed to be independent with mean zero and gene-specific standard deviation ϕg. The non-linear regression returns estimates Write logCPM2-count-per-million values using the fitmixture function in the limma package. The fitmixture function fits the non-linear regressions very efficiently to all genes simultaneously using a vectorized nested Gauss–Newton type algorithm  value of greater than 1. The number of genes is reduced to 14 981 after filtering on expression. The analysis was also repeated using a lower CPM cutoff value of 0.5, leaving 16 131 genes.baySeq methods; normalization using the median ratio method  estimated to get log-FCs and average log-CPM values.To obtain mean-difference plots in Figure et al.   and the pilot SEQC data set from Law . (2014)  was usedWe investigate simple two-group comparisons by subsetting the data for the two groups of interest. For all methods, correction for multiple hypothesis testing was carried out using Benjamini and Hochberg's FDR approach .g is the estimated log-FC for gene g, G is the total number of genes and p and q are the mixing proportions for the two groups being compared.For each two-group comparison, the estimated log-FCs were compared with the log-FCs predicted by the nonlinear regression  which left 138 042 exons from 16,500 unique genes for downstream analysis.Exon counts were obtained from the poly-A mRNA samples as described in the edgeR and limma-voom analyses are normalized by the TMM method and counts for DEXSeq are normalized using the median ratio method. As per the differential expression analyses, simple two-group comparisons are carried out by subsetting the data for the groups of interest and p-values are adjusted using Benjamini and Hochberg's FDR approach  in DeMix (ISOpure (Y = (1 − p)N + pT, where Y is the expression level from a mixed sample, N is the expression level from normal tissue, T is the expression level from tumour tissue, and p is the tumour proportion in the observed mixed sample. DeMix is a Bayesian approach available in R  do not satisfy the linear convolution structure, or (ii) are uninformative as they have the same expression levels in both sample types such that including them in the model estimation step weakens its ability to differentiate each component.The two approaches compared, DeMix  and ISOp (http://bioinf.wehi.edu.au/voom/). This data set was pre-processed by filtering genes with low counts (CPM > 1 in at least 4 samples was required), followed by TMM normalization  from Law . (2014)  obtainedlization . The est freedom  for thisEGFR: L858R and T790M mutations are found in NCI-H1975 and deletion of exon 19 is observed in HCC827), were selected for the main experiment. The intent here was to have changes that were more subtle than those typically observed in earlier mixture experiments where completely different tissue types, were compared. For example, use of the MCF-7 breast cancer cell line and Jurkat human T lymphocyte cells in Holloway et al. (2006)  and Illumina's TruSeq RNA v2 kit (which we refer to as the poly-A mRNA kit). Libraries were sequenced as single-end 100 bp reads on an Illumina HiSeq2500 instrument producing on average 50 million reads per library (range 32 to 94 million). Reads from all samples were mapped to the hg19 reference genome using the Subread alignment software  via RNA-seq. The two cell lines HCC827 and NCI-H1975, which were observed to be most similar according to the multidimensional scaling (MDS) plot  and had software . Mapped reCounts  accordinSubjunc to their 5΄ position and assigned them to non-overlapping custom annotations restricted to either exon, intron or intergenic regions (UTR regions were considered as exons). Genomic feature mapping statistics for all replicate 2 samples are shown in Figure et al. (2014) . To facilitate comparison of the protocols in terms of read distribution across different genomic features, we reduced the reads mapped using the splice aware aligner . (2014) , althougfeatureCounts behaviour) obtained from Subjunc aligned data and annotated them according to the gene type information available with the annotation. The percentage of reads assigned to each type for all replicate 2 samples is shown in Figure In order to compare the protocols with regard to read distribution across different RNA classes, we used the gene-level exon counts (default 2 counts per million (log-CPM)  Figure  and D. T) Figure , the rep) Figure  on the o2-fold-changes (log-FCs) (y-axis) versus average expression (x-axis) for the pure samples . The FCs cover a wide dynamic range and are symmetric about the log-FC = 0 (no change) line. When samples more similar in RNA composition are compared, the log-FCs are compressed and asymmetric  to visualise the attenuation in signal that occurs as the RNA samples compared become more similar. Figure c Figure  and C. Lc Figure –F, we seRNA-seq library preparation methods that rely on capturing poly-A RNA species are susceptible to RNA degradation. As RNA gets degraded, the non-poly-adenylated 5΄ end of the molecule becomes under-represented. In order to assess the 5΄-3΄ bias in our libraries, we used the RSeQC  softwareMg) and log-variance estimates  for the poly-A data and 0.52 units  for the total RNA data. The average increase in variability for the poly-A mRNA data is thus 1.85-fold greater than that observed for the total RNA data. For this reason, we chose to focus on the mRNA data in the method comparisons in the remainder of this paper as it shows the most variation due to sample quality and will allow us to compare the performance of methods in the presence of high or low baseline variability by either including or excluding the degraded samples.To assess the bias and precision genome-wide, we used the gene-wise log-ratios  and log-variances  (edgeR-glm) (baySeq) or by normal distributions (baySeq-norm); limma-voom (voom) or with sample quality weights (voom-qw)   or with geR-glm) ; baySeq geR-glm) ,37 with mma-voom ,38 eithevoom-qw) ; and DES . Thus, voom-qw retains much of its power even when transcriptional differences are reduced. Most other methods have recovery rates only moderately lower than that of voom-qw, except for baySeq and baySeq-norm that have much lower recovery rates than the other methods: 62% in both methods for 075 versus 050, and 10–12% in baySeq and 0% for baySeq-norm for 050 versus 025 and 075 versus 050. Both baySeq methods lose power more quickly than the other methods when the transcriptional differences are small.Using a CPM threshold of 1 for expression  and a 5% FDR cutoff for differential expression, s Figure . For modThe experimental design allows us to perform an analysis on the regular samples and compare our results to the analysis that includes the degraded samples. Replacing the non-degraded replicate 2 with the degraded sample results in extra within-group variability that can be observed in the vertical spread of points in the MDS plot in Figure baySeq-norm. With the increase in within-group variability, baySeq-norm achieves relatively high recovery rates that are matched by exceedingly high inconsistency rates . Amongst methods with low inconsistency, voom-qw has the highest recovery rate for 100 versus 000d and 075 versus 025d while DESeq2 has highest recovery by a narrow margin for the most subtle comparisons 050 versus 025d and 075 versus 050d.As expected, increasing within-group variability results in lower recovery rates for most methods. Here, the relative performance was observed to be similar to that observed for the good samples only for all methods except cy rates . For 100baySeq-norm, all methods detect differential expression consistently, with low inconsistency rates observed across all comparisons with or without degraded samples. For large to moderate differences in mixture proportion , inconsistency rates sit at between 0–1.7% (not including baySeq-norm). For subtle differences in mixture proportion , inconsistency is between 0–0.3% (not including baySeq-norm). Although recovery and inconsistency is assessed for each method relative to itself, it is worth noting that most genes detected as DE in the 100 versus 000 comparison are common to all methods, with an overlap of 10 015 genes between baySeq, edgeR-glm, voom-qw and DESeq2  and DESeq2 detects the least unique DE genes  over the entire data set and adjusted according to the relevant sample concentrations for the two-group comparisons . The pilot SEQC RNA-seq data set from Law et al. (2014) (edgeR-glm. The SEQC data set was pre-processed by filtering genes with low counts (CPM > 1 in at least 4 samples was required), then TMM normalized. The prior degrees of freedom was estimated to be 5.30 using the good samples, 4.53 for the good and degraded samples and 67.7 for the SEQC data set. This value measures the consistency of the gene-wise dispersions, with a smaller prior degrees of freedom indicating the values are more gene-specific  and larger values indicating that they are more consistent  and will be more heavily shrunk to a common value. Repeating this analysis with limma voom gave similar results with prior degrees of freedom values estimated as 5.29, 4.54 and 45.3, respectively for the 3 data sets. Plotting the BCV from the edgeR-glm analysis offers another way to visualise this  genes was carried out using three available Bioconductor methods for the analysis of exon counts: DEXSeq , edgeR-dedgeR-ds  and voom. The number of DS genes detected is much lower in these comparisons, with reduced recovery and inconsistency rates relative to the corresponding comparisons using good samples only refer to .Results from comparisons between the mixture samples  indicate that the differential splicing methods deteriorate more rapidly than the differential expression analysis methods when the expression changes become more subtle, with lower recovery rates and higher inconsistency rates. In general, differential splicing analysis is more complex and challenging than differential expression analysis. Detection of differential exon usage is equivalent to a statistical test of interaction rather than of a simple change in level between two treatment conditions, so there is less statistical power to detect differential splicing. Furthermore, the total gene-level read counts used for differential expression analysis are spread over several exons per gene, leaving much lower counts per feature for exon-level analysis.DeMix (ISOpure (DeMix yielded proportion estimates that were closer to the ground truth (differences of less than 10%) while ISOpure largely overestimated mixture proportions across all 12 samples. We also compared the deconvolved expression values from the two methods with the four pure samples using Pearson correlation coefficients. The correlations from DeMix are all above 0.95 and were generally higher than those obtained from ISOpure , voom and finally baySeq  is the least sensitive. Recent work has shown that combining differential expression methods in a weighted manner  and assessment criteria ,40,43–47 be best ,43, or et   , who degFurther extensions to this experiment could include other library preparation methods, such as exome capture, which has been shown to improve the results from degraded samples  or GlobiOther genomics applications, such as ChIP-seq or RNA-seq using new technologies, such as long read sequencing platforms from companies such as Pacific Biosciences (Sequel) or Oxford Nanopore Technology (MinION and PromethION) could also benefit from having available a reference data set such as this to allow methodology testing and inter-platform comparisons.RNASEQcomp , the voom-qw method that explicitly deals with sample-level variation slightly outperforms other methods, with DESeq2 second best, followed by the two edgeR methods which perform similarly, then standard voom, and finally the two baySeq methods that have the least power and consistency. In terms of bias, DESeq2 gives FCs that slightly underestimate the consensus values obtained from the nonlinear model, indicating that its shrinkage procedure gives conservative results. For differential splicing methods, edgeR-ds and voom-ds were found to both be conservative relative to the most popular method, DEXSeq, however, DEXSeq tended to have a high inconsistency rate compared to these other methods. This is the first time that a mixture experiment has been used to benchmark differential splicing methods. The results from the deconvolution analysis showed that DeMix consistently outperformed ISOpure. Our work demonstrates the utility of carefully designed control experiments for benchmarking library preparation kits and analysis methods.This mixture design also allows for internal comparisons to be made http://bioinf.wehi.edu.au/folders/rnaseqmixture/.Raw sequence data are available under GEO series accession numbers GSE86337 (pilot experiment) and GSE64098 (mixture experiment). Analysis scripts and processed data are available from the Supplementary Information website at Supplementary DataClick here for additional data file.


Models for predicting phenotypic outcomes from genotypes have important applications to understanding genomic function and improving human health. Here, we develop a machine-learning system to predict cell-type–specific epigenetic and transcriptional profiles in large mammalian genomes from DNA sequence alone. By use of convolutional neural networks, this system identifies promoters and distal regulatory elements and synthesizes their content to make effective gene expression predictions. We show that model predictions for the influence of genomic variants on gene expression align well to causal variants underlying eQTLs in human populations and can be useful for generating mechanistic hypotheses to enable fine mapping of disease loci. Although many studies show strong relationships between variation in genotype and phenotype across a range of human diseases and traits, the mechanisms through which this relationship operates remain incompletely understood . NoncodiLarge-scale consortia and many individual laboratories have mapped the epigenetic and expression profiles of a wide variety of cells . FurtherDespite this progress, models to predict cell-type–specific gene expression from DNA sequence have remained elusive in complex organisms. Existing models use experimental annotations as input , allowing them to shed light on the relationships between these annotations  but not Here, we use novel machine-learning algorithms to learn to predict thousands of epigenetic and transcriptional profiles from hundreds of human cell types using only DNA sequence as input. By using the model, we predict the difference between the two alleles of genomic variants for these thousands of data sets, focusing particularly on predicted changes to gene expression. We demonstrate the considerable potential value of this observation to identify likely causal variants and mechanisms within GWAS loci.In previous work, we introduced a deep convolutional neural network approach named Basset for modeling “peak”-based chromatin profiles, focusing particularly on DNase I hypersensitivity . Given a17=) 131-kb regions as input and, similar to Basset, performs multiple layers of convolution and pooling to transform the DNA to a sequence of vectors representing 128-bp regions (Methods). To share information across long distances, we then apply several layers of densely connected dilated convolutions (Methods). After these layers, each 128-bp region aligns to a vector that considers the relevant regulatory elements across a large span of sequence. Finally, we apply a final width-one convolutional layer to parameterize a multitask Poisson regression on normalized counts of aligned reads to that region for every data set provided .To assess how effectively the model predicts the signal in these data sets, we compared predictions to experimental coverage in the 128-bp bins for a set of held-out test sequences A. As preSupplemental Fig. S1). Note that these accuracy statistics cannot be directly compared with those reported in the Basset paper due to different sequence distributions; in the prior work, all sequences were DNAse-seq peaks in some cell type, whereas here they are unbiased genome samples.To benchmark this framework against our previous Basset , we focut-test P-value <2 × 10−78) . The mean and median cross-replicate predictions  also exceeded the correlation between replicates  . Prediction correlation can exceed replicate–replicate correlation because the replicate measurements are noisy and the predictions implicitly denoise.One thousand two hundred eighty-four replicated experiments allowed us to benchmark prediction–experiment correlation versus experiment–experiment correlation. Determining the exact nature of these replicates was not always possible, but 80%–90% are biological  replicates. Across these replicated experiments, the mean and median prediction correlation exceeded the replicate correlation . Given the limited focus on transcriptional regulation, Basenji prediction correlations approach but fall short of the replicate–replicate correlations at the high end for the most consistent data sets, particularly CAGE .As replicate–replicate correlation increased, Basenji prediction accuracy also increased, suggesting that higher quality data enable more effective modeling of the sequence dependence of the regulatory signal D. The siSupplemental Fig. S4). An eighth layer would reach outside the bounds of the sequence too frequently for this input length but may add value for longer sequences.We hypothesized that the seven dilated convolution layers enabled the model to capture the distal influences that are an established feature of human gene regulation. To isolate the influence of receptive field width, we trained similar models with one to seven dilated layers. Test accuracy increased with increasing receptive field for all data types, confirming the value added by the additional dilated convolution layers of the final network   A,B, whicctively) . Correlactively) A, suggesSupplemental Fig. S5). For these analyses, we performed quantile normalization on the predictions and experimental measurements across data sets. To quantify concordance between cell/tissue cluster structure, we performed Gaussian mixture model clustering of bootstrap gene samples for the experimental and predicted expression profiles. The adjusted Rand index distribution  indicated significant agreement between clusters   C.Supplemental Fig. S6). The stable expression across cell types of housekeeping genes relies on a particular promoter architecture . For each gene's two most variable TSSs, we computed the log2 fold change of one TSS versus the other in each CAGE sample for both the experiment and Basenji prediction. The mean Pearson correlation between experiment and prediction fold changes was 0.295 (median 0.288) over the set of genes , which is greater than zero with P-value <1 × 10−90 by t-test. This analysis suggests that Basenji predictions explain considerable variance in alternative TSS usage across samples.CAGE quantifies gene expression at TSS resolution, and genes may use distinct TSSs in different cell types . To measDistant enhancer sequences play a significant role in activating gene expression .We devised a method to quantify how distal sequence influences a Basenji model's predictions and applied it to produce saliency maps for gene regions. The saliency map scores annotate 128-bp segments with a function of the model predictions’ gradient with respect to that segment's vector representation after the convolutional layers and before the dilated convolutions share information across wider distances (Methods). Peaks in this saliency score detect distal regulatory elements, and its sign indicates enhancing (+) versus silencing (−) influence.PIM1 in the GM12878 cell line exemplifies this approach   , and CTCF binding sites (not overlapping promoters or enhancers) . We compMethods) B. PromotFunctional profiling of genotyped individuals is widely used to detect influential genomic variation in populations. The Gene-Tissue Expression (GTEx) project offers one such data collection . GTEx meGiven a SNP–gene pair, we define its SNP expression difference (SED) score as the difference between the predicted CAGE coverage at that gene's TSS  for the two alleles A. Linkag2 statistics after controlling for the total amount of genetic variation tagged by each SNP as measured by LD score . To assess the quantitative extent of this enrichment, we ranked variants by the difference between their predictions from regression models including and excluding |SED-LD| for each tissue and binned them into five quantiles. The proportion of variants called significant eQTLs was 3.2–5.8× greater in the top quantile relative to the average of the bottom three in all tissues . Thus, our analyses support a robust predictive relationship between Basenji scores and population measurements of RNA abundance, despite the additional layers of post-transcriptional regulation captured by the eQTL analysis and presently invisible to Basenji.We checked whether the absolute value of SED-LD correlated with eQTL χLD score  on a set tissues B,C. ThisBasenji's utility for analyzing human genomic variation goes beyond intermediate molecular phenotypes like eQTLs to downstream ones like physical traits and disease. Basenji also offers a substantial upside: eQTL analysis is highly informative for disease variant interpretation, but few cell types can plausibly be sampled to conduct such an investigation. With Basenji, a single experiment is sufficient to predict a genomic variant's influence on gene expression in that cell type. We hypothesized that a predictive view of the 973 human samples profiled by CAGE would offer a novel perspective on disease variants.2 ratio between allele values for each penultimate layer unit. One hundred twenty-eight principal components were sufficient to represent the full profile well . We maintained the same 10 cross-validation folds reported by DeepSEA to compare methods. A logistic regression model to predict GWAS catalog presence using these Basenji principal component features achieved 0.666 AUROC, slightly greater than the 0.657 achieved by DeepSEA using a more complicated regression model that also included conservation statistics  . A joint model adding DeepSEA's predictions as a feature to our logistic regression model increased AUROC to 0.706 , supporting the orthogonal value added by Basenji SNP scores beyond the state of the art.To test the utility of Basenji SNP scores for this application, we acquired a curated set of disease variants studied by the successful DeepSEA method to predict variant influence on TF binding and chromatin . DeepSEAGPR65 in many cells, most severely acute lymphoblastic leukemia cell lines, thyroid cells, insular cortex cells, and a variety of immune cells. GPR65 is a receptor for the glycosphingolipid psychosine and may have a role in activation-induced cell death and differentiation of T cells  via variants independent of this set ; the competing correlated variant rs74796499 has a smaller measured effect .PICS fine mapping assigns rs78461372 a 5% probability of causal association with multiple sclerosis and assigns the lead variant rs74796499 a 24% probability. Basenji predicts no effect for rs74796499 or any other variants in the PICS credible set. To validate the predicted stronger effect of rs78461372 on nearby transcription, we consulted the GTEx eQTL analysis . GTEx suGPR65 and GALC are more active, disruption of the motifs decreases the prediction. Alternatively, disruption of the same motifs increases the model's prediction for expression in several tissues, including insular cortex. Altogether, Basenji predictions shed substantial light on this complex and influential locus, offering several promising directions for future work to unravel the causal mechanism.To better understand the model prediction, we performed an in silico saturation mutagenesis  in severTranscriptional regulation is the primary driver of gene expression specificity across cell types and states . The genAlthough we demonstrated the present utility of this approach, there are several indications that we may be scratching the surface of what will be possible in this space. The data sets analyzed vary in quality, by both signal-to-noise ratio and technical variance from undersampling with limited sequencing. We observed increasing predictive accuracy for experiments with greater sequencing depth and greater consistency between replicates. Thus, sequence-based modeling will benefit from improved experimental protocols, which are an area of active research  . FurtherDilated convolutions extended the reach of our model to view distal regulatory elements at distances beyond previous models, achieving a 32-kb receptive field width. This distance contains many, but certainly not all, regulatory elements . Thus, eDespite focusing only on transcription without considering post-transcriptional regulation and interaction across regulatory layers , we founFiner resolution analysis of large genomic regions exposes expressive machine-learning models more to biases in functional genomics sequencing experiments   and repeSupplemental Table S1; We downloaded FASTQ files for 973 CAGE experiments performed by FANTOM5 , 593 DNa2 coverage estimates. Finally, we reconfigured the coverage estimates to highlight the residual coverage unexplained by the GC% model. The parameters of the bias models exhibit a wide range, both within and across assays, suggesting the absence of a common sequencing bias. A Python script implementing these procedures to transform a BAM file of alignments to a bigWig file of inferred coverage values is available in the Basenji tool suite.We normalized for GC% bias using a procedure that incorporates several established ideas, aiming to model the trend across the GC% spectrum without precluding a GC% enrichment for active regions . We assi17=) 131-kb nonoverlapping sequences across the chromosomes. We discarded sequences with >35% unmappable sequence, leaving 14,533 sequences. We randomly separated 5% for a validation set, 5% for a test set, and the remaining 90% for training. Within each sequence, we summed coverage estimates in 128 bins to serve as the signal for the model to predict.By avoiding assembly gaps and unmappable regions >1 kb, we extracted  package to search for effective hyperparameters throughout the model, including the convolution widths, convolution filter numbers, dropout rates, learning rate, and momentum parameters , enhancer (ENCFF811LAE), and CTCF (ENCFF002COQ) sites as BED files. We subtracted the promoters from the enhancers, and we subtracted both the promoters and enhancers from the CTCF sites. We formed a background set by shuffling these annotations within the 131-kb regions for which we computed scores and removing instances that overlapped true elements. We computed a 2 statistics and significance calls . In a first analysis, we fit regression models individually for each tissue with LD score and |SED-LD| to the χ2 statistics and considered the significance of coefficients assigned to |SED-LD| across all variants. In a second analysis, we added the TSS-LD variables to the regression.Finally, we pruned the set of variants down to ensure that there were no pairs of variants in strong LD (Supplemental Material and from https://www.github.com/calico/basenji.The code to preprocess data, train models, and perform the analyses described is available in the D.R.K. is employed by Calico LLC. D.B., C.Y.M., J.S., and M.B. are employed by Google Inc.


D. melanogaster that we used here. We investigated the power of existing approaches for quality checking mRNA-seq data and explored additional, quantitative quality checks. To accommodate nested, multi-level experimental designs, we incorporated sample layout into our analyses. We employed a subsampling without replacement-based normalization and an identification of DE that accounted for the hierarchy and amplitude of effect sizes within samples, then evaluated the resulting differential expression call in comparison to existing approaches. In a final step to test for broader applicability, we applied our approaches to a published set of H. sapiens mRNA-seq samples, The dataset-tailored methods improved sample comparability and delivered a robust prediction of subtle gene expression changes. The proposed approaches have the potential to improve key steps in the analysis of RNA-seq data by incorporating the structure and characteristics of biological experiments.RNA sequencing (RNA-seq) is widely used for RNA quantification in the environmental, biological and medical sciences. It enables the description of genome-wide patterns of expression and the identification of regulatory interactions and networks. The aim of RNA-seq data analyses is to achieve rigorous quantification of genes/transcripts to allow a reliable prediction of differential expression (DE), despite variation in levels of noise and inherent biases in sequencing data. This can be especially challenging for datasets in which gene expression differences are subtle, as in the behavioural transcriptomics test dataset from  RNA sequencing (RNA-seq) has revolutionized the field of transcriptomics , 2, giviA key step in the analysis of RNA-seq data consists of sample quality checks and the identification, characterization and potential exclusion of sample outliers, e.g. those samples that are compromised due to technical issues . Tools sQuantitative analysis of sequencing output currently considers measures such as yield, coverage, 3’/5’ bias, number of detectable transcripts, strand specificity and read distribution across the genome . Additior2 ∈  are considered acceptable. If the PCC falls below 0.9 the suggestion is to exclude the problematic replicates/samples  which is the sum of abundances of fragments incident with position i. The point-to-point PCC was computed as the standard PCC on the corresponding vectors from the two samples which were compared.For the mRNA-seq samples, the QC consisted of two stages. Stage 1 comprised of previously described methods  includinion bias , (iv) stectively  and (v) ectively , with maectively  on full ectively . We examectively  on the tD. melanogaster data, to render the distributions fully comparable. For the analysis of the D melanogaster and H. sapiens datasets the quantile correction was employed. If the distributions of abundances, after the subsampling, are already perfectly aligned, then the quantile correction would not be necessary. The pseudocode is presented in We adapted a normalization procedure based on subsampling (without replacement) . The conExisting procedures which were used for the comparison of the new normalization methods were: scaling normalization , for whiExisting methods for the DE call are often based on comparing the variability between replicates with the difference between the treatments. However, coefficient of variance (or CV) based on a small number of points may often not reflect the true variance of the given gene/transcript –57. MoreD. melanogaster transcriptome data. (i) Identification of levels for the hierarchical differential expression and the constituent internal classes. For the D. melanogaster data one ‘level’ was body part  and the other was treatment . (ii) The ordering of the hierarchical levels based on the amplitude of differential expression. This was quantified by the width/ spread of the distribution of DE in terms of mean/ median, IQR and min/max values. The amplitude of DE in descending order provided the correct ordering of the levels for the hierarchical DE. (iii) The DE analysis on the proximal ends of the CIs, using OFC [DE was calculated using a hierarchical approach and by applying an offset fold change (OFC) method . There were 3 steps to the hierarchical analysis used for the analysis of the sing OFC . The pseThe two-step DE procedure (using OFC) consisted of (i) calculation of the list of genes showing DE between body parts, followed by (ii) calculation of the DE between genes in the ± rivals treatment comparisons. Step (i) was conducted on the summed expression levels in the ± rivals pairs . The genes were then separated into genes expressed only in HT, only in A, and in both the HT and A. Step (ii) of the DE was then applied on the resulting 3 categories  using the ± rival condition. The DE call was conducted on the +/- rivals condition, using maximal CIs created on the normalized replicates. As outlined above, we called DE genes with more than 1.5 fold between the proximal ends of the maximal CIs. The DE call as determined by edgeR and DESeq2 were calculated using the default functions and parameters.mRNA samples: (a) D melanogaster: males of D melanogaster exposed to conspecific rivals (or not) for 3 time periods (GSE55930). (b) H sapiens: For the mRNA Human samples, we chose 5 samples from the Pickrell et al. 2010 [al. 2010  study (GS1 MethodsA description with details for (1) Incremental subsampling and bootstrapping check for consistency of a sample, and (2) Subsampling to a fixed total.(PDF)Click here for additional data file.S2 MethodsA description with technical details for the two step  DE, including the identification of levels in the hierarchy.(PDF)Click here for additional data file.S1 TableFrom the study described in Mohorianu et al. 2017, RNA 23:1048–1059 , we sele(XLSX)Click here for additional data file.S2 TableReplicate 1 samples 02-A, 02+A, 02-H and 02+H  were used to illustrate the proportion of reads mapping simultaneously to pairwise groups of CDSs, exons, 5’ and 3’ UTRs, introns and intergenic regions; on the diagonal, the proportion is 1.00 since each category is compared to itself; the matrix is not symmetric because each proportion is calculated relative to the number of reads present in the category indicated by the column. We observe that a high proportion of reads is incident with protein coding genes, with few reads showing multiple matching to protein coding-genes and introns or intergenic regions; overall, these results suggest a high quality of the data. In the main study we computed the expression levels using gene mapping reads.(XLSX)Click here for additional data file.S3 TableShown is a 12 by 12 matrix of all the original samples compared with each other. Samples are labelled by time point (2h), by ± rivals treatment, by body part (A or HT) and then by replicate number. Each sample tested against itself along the diagonal is therefore 100% similar and shares the top 1000 most abundant genes in common. A to A comparisons are shaded in purple, HT to HT comparisons in peach. Samples drawn from the same body parts shared > 90% similarity, and between body parts the similarity dropped to ~50%. Similarity between the ± rivals treatments tended to be higher than between replicates. Two illustrative examples are highlighted, in which ± rivals indices (in red bold) were generally higher than replicate to replicate similarity (blue bold). This highlighted the need for adapted normalization methods.(XLSX)Click here for additional data file.S4 TableD. melanogaster, v 6.11). As a consequence of the incremental subsampling, the complexity, defined as the ratio of non-redundant to redundant reads, increased, and became comparable to the complexity of the other replicates, when subsampled at the same sequencing depth. A replicate was accepted if it exhibited a similar complexity (and distribution of per-gene complexities) with the other replicates for the same type of sample.For sample 02-H2  we present the incremental subsampling, without replacement, from 99% to 40% of the data. To judge whether a sample is consistent, and to determine the consistency threshold, we used the proportion of redundant reads matching to the reference genome ((XLSX)Click here for additional data file.S5 TableDrosophila melanogaster qRT-PCR 'validated' gene set from Mohorianu et al. 2017 (RNA 23:1048–1059).Results from (A) DEseq2 and (B) edgeR analyses of the  For the validations we used 3 reference genes and validated 15 A genes and 6 HT genes based on the DE selection using subsampling normalization and hierarchical DE. We investigated whether these genes were called DE by either (A) DESeq2 or (B) edgeR. In S5A Table we present the results for DESeq2, in S5B Table the results for edgeR. For each of the three categories of genes  we show the average of normalized abundances (baseMean for DESeq2 and logCPM for edgeR), the fold change between treatments (log2 FoldChange for DESeq2 and log2FC for edgeR) and the DE P-value and adjusted P-value .(XLSX)Click here for additional data file.S1 FigRequired inputs  and the six main steps of the analysis are shown in a workflow diagram, following Conesa et al. 2016 . The steps, for which additional details are included, are: Quality check (QC), alignment, normalization of gene abundances, identification of DE, functional enrichment and finally low-throughput validation.(PDF)Click here for additional data file.S2 FigCorrelation analyses (Pearson (PCC), Spearman (SCC) and Kendall correlation coefficients (KCC)) between the gene expression levels for the D. melanogaster data for (A) all samples, (B) HT samples, (C) A samples. A1, B1, C1 show the PCC; A2, B2, C2 show the SCC; A3, B3 and C3 show the KCC. Each panel shows the distributions of correlation coefficients for all pairwise comparisons. For example, in panel A.1, sample 1 on the x-axis shows the distribution of the n = 35 correlation coefficients calculated between the gene expressions in sample 1 compared with gene expressions in all other 35 samples, using the PCC. The results are presented as a standard boxplots i.e. the box indicates the inter-quartile range, the middle line is the median and the whiskers extend to 5% and 95%; the outliers are represented with circles. All three approaches supported the same conclusion i.e. the A and H gene expression levels in the A and H samples, respectively, correlated very well , whereas, if we compared between A and H samples (A panels), the minimum correlation dropped to 0.5.(PDF)Click here for additional data file.S3 FigPanel A shows replicate 1 vs 2, B replicate 1 vs 3, and C replicate 2 vs 3. Shown are the raw data, prior to normalization. For all replicate comparisons, more variability was consistently observed at lower abundances and derived mostly from the small number or scattered incident reads. For the higher abundance genes, the p2pPCC was tight, indicating a high reproducibility of the expression profile and a low incidence of alternative splicing events. This analysis formed the basis of the offset identification i.e. the offset was selected to be the value for which, for all replicates, for all samples, the median of the p2pPCC was above 0.5.(PDF)Click here for additional data file.S4 Fig2) and on the y-axis the distribution of point-to-point PCCs calculated for each expressed gene.To indicate the consistency during the subsampling, without replacement, the plots show the point-to-point PCC between the original and incrementally subsampled (from 40% to 95%) data . On the x-axis is the gene abundance (log(PDF)Click here for additional data file.S5 Fig2 scale). Panel A shows the replicate-replicate DE (blue) and the with/without rivals DE (red) for the abdomen (A) samples. Panel B shows the corresponding data for the HT (H) body part. Panel C shows the distribution of DE for the with/without rivals treatments (blue for HT and green for A samples) and the DE between H and A (orange). The DE distribution for the treatment  was overlapping with the DE distribution between the replicates (blue), indicating a subtle DE signature. However, the DE distribution between body parts (orange) showed a good separation between HT- and AB- specific genes. These DE distributions supported the choice of a hierarchical design for the identification of DE genes between the treatments.Frequency density plots were used to show the distribution of DE between samples Click here for additional data file.S6 Fig2 scale), on the y-axis the FC . Although the quantile and the DESeq2 produced very similar distributions of abundances between replicates/samples, the former did not produce a tight MA plot when variability in expression at gene level was assessed. For the comparable results for the subsampling (without replacement) approach, see main . On the x-axis we represent the average abundance between replicates (log(PDF)Click here for additional data file.S7 FigEdgeR only genes shown in (PDF)Click here for additional data file.S8 Fig2 scale, on the y-axis the coefficient of variation (CV)—the ratio between the standard deviation and the mean. For clarity, the distributions are represented as standard boxplots. The upper panels  show the CV for the original data for A samples without rivals (A), A samples with rivals (B), HT samples without rivals (C) and HT samples with rivals (D), respectively. The lower panels  give the CV for the same samples, after the subsampling normalization (without replacement). The red horizontal lines indicate 0.5 and 0,25 CV. It is clear that the subsampling normalization reduced the variance between the replicates to < 0.25 CV across most abundances (panels E-H), whereas the CV was much higher across all abundances for the raw data (panels A-D).On the x-axis is the abundance in log(PDF)Click here for additional data file.S9 FigIn the upper plots we show the coefficient of variation (CV), y-axis vs the average abundance, x-axis, obtained after the subsampling normalization (without replacement) for 5 sequencing pairs . The F1 and F2 plots show the CV for the combined pairs for the Yale and Argonne replicates, respectively. For individual comparisons we achieved lower CVs in comparison to Zhou et al. 2014  analysis of these sample data. For the sets of lab replicates, Yale and Argonne, respectively, the results using subsampling without replacement are in line with the edgeR results . Based on these distributions we concluded that the samples from the different laboratories could be rendered comparable using our subsampling approach . In the lower panels, we present the MA plots, after the subsampling normalization, for the same pairs of samples. The tightness of these plots  supported the conclusion that the subsampling made these samples derived from sequencing in different laboratories highly comparable.(PDF)Click here for additional data file.