Understanding the regulation of gene expression, including transcription start site usage, alternative splicing, and polyadenylation, requires accurate quantification of expression levels down to the level of individual transcript isoforms. To comparatively evaluate the accuracy of the many methods that have been proposed for estimating transcript isoform abundance from RNA sequencing data, we have used both synthetic data as well as an independent experimental method for quantifying the abundance of transcript ends at the genome-wide level.We found that many tools have good accuracy and yield better estimates of gene-level expression compared to commonly used count-based approaches, but they vary widely in memory and runtime requirements. Nucleotide composition and intron/exon structure have comparatively little influence on the accuracy of expression estimates, which correlates most strongly with transcript/gene expression levels. To facilitate the reproduction and further extension of our study, we provide datasets, source code, and an online analysis tool on a companion website, where developers can upload expression estimates obtained with their own tool to compare them to those inferred by the methods assessed here.As many methods for quantifying isoform abundance with comparable accuracy are available, a user’s choice will likely be determined by factors such as the memory and runtime requirements, as well as the availability of methods for downstream analyses. Sequencing-based methods to quantify the abundance of specific transcript regions could complement validation schemes based on synthetic data and quantitative PCR in future or ongoing assessments of RNA-seq analysis methods.The online version of this article (doi:10.1186/s13059-015-0702-5) contains supplementary material, which is available to authorized users. The general availability of high-throughput sequencing technologies greatly facilitated the detection and quantification of RNA species, including protein-coding RNAs, long non-coding RNAs, and microRNAs, in many different systems. In higher eukaryotes, the vast majority of protein-coding genes express multiple transcript isoforms –3. Altho4 transcripts, which does not allow accurate quantification of transcript abundance. Furthermore, the error rates are relatively high, making the transcript identification non-trivial. Thus, accurate and cost-effective quantification of the complete repertoire of full-length expressed transcripts, which are in the range of hundreds of thousands per cell [Analysis of expression pattern is often one of the first steps towards understanding a gene’s function. However, transcript isoform abundance is almost always quantified indirectly; most of the sequencing technologies that are currently used yield reads that are short (≤200 nt) relative to the length of eukaryotic transcripts   and thusper cell , remainsAs RNA sequencing (RNA-seq) has become commonplace in molecular biology laboratories, a variety of computational approaches has been proposed for isoform reconstruction from short read sequencing data . SimilaIn this study we carried out a systematic evaluation of a large number of methods for isoform quantification from RNA-seq data. We used not only synthetic, but also genome-wide experimental datasets. We took advantage of newly developed protocols for quantifying the abundance of distinct RNA 3′ ends, which result from the use of alternative 3′ end processing sites. These protocols allow a comprehensive surveillance of 3′ end processing site usage, with a method that is distinct from RNA-seq –25. FromOur results indicate that many of the available methods have comparable accuracy, and that the abundance of highly expressed isoforms is more accurately inferred than the abundance of isoforms with low expression levels. We further found that even the quantification of gene expression is more accurate when gene expression levels are computed by cumulating the levels of transcript isoforms than when ignoring the transcript structures. Given that many methods are available that differ little in accuracy, a user’s choice will likely be determined by factors such as the memory and runtime requirements, as well as the availability of methods for downstream analyses such as differential gene/transcript expression.We initially performed an extensive literature survey to identify tools that were developed for inferring the abundance of transcript isoforms from RNA-seq data. Although we tried to include as many of these as possible, our study setup required that tools are able to quantify a set of transcripts that we provided as input, thereby separating the problem of transcript reconstruction from that of abundance quantification. To be able to interpret the results, we further focused on methods that have been duly described in the literature. Lastly, we thought that ease of use would be critical for the adoption of the tool by the user community and we did not pursue methods which we were unable to implement within a reasonable amount of time. Table in silico-generated reads range between approximately 7 min (IsoEM) and more than 1 week (TIGAR2) when a single processor is used, and between about 5 min (IsoEM) and 8 h (RSEM) when 16 cores are available for the tools that support multi-threading (TIGAR2 does not). With the exception of Sailfish, runtimes strictly increased with the number of processed read alignments. Assuming that a method-specific, but largely sample size-independent time span is required to index the supplied transcriptome, time complexities for most of the quantification algorithms appear to be approximately linear. Sailfish’s runtimes seem to be the highest for the smallest dataset, presumably because the convergence of estimation is slow for small datasets, when the vast majority of transcripts are sparsely covered. Notably, Sailfish computes abundances based on raw read sequences rather than alignments. Thus, whenever alignments are dispensable, a considerable amount of time  can be saved on sample pre-processing compared to all other methods  . As expected, correlations generally improved with increasing library sizes, in a monotonic fashion and asymptotically towards saturation. For most methods, estimation accuracies reached a plateau at or around a read depth of 30 million reads, indicating that further increases in read depth are unlikely to significantly improve their results. In particular, Spearman correlation coefficients peaked at above 0.95 for six of the methods  and above 0.9 for a further three methods . Both Cufflinks and Scripture performed considerably worse than all other methods, with the corresponding correlation coefficients barely surpassing 0.75. The influence of the library size on accuracy varied somewhat between methods, with the total gain from the sparsest to the richest dataset ranging from approximately 0.01 (Cufflinks) to approximately 0.08 (BitSeq). Out of the nine most accurate methods, MMSEQ appears to be the least sensitive to the influence of read depth (approximately 0.04 gain in accuracy). In order to rule out that our chosen metric for measuring accuracy is prone to producing idiosyncratic results, we have compared it with both the Pearson correlation coefficient and the root mean square error . To avoid the situation that our results are dominated by how different methods handle transcripts that are essentially not expressed, we initially restricted our initial analysis to the set of expressed transcripts. These were those for which the simulation software assumed non-zero expression values. When comparing the abundances of these transcripts as inferred by each method with the ‘ground truth’  accuracy. In contrast, the count-based methods only achieved moderate accuracy (maximum rs = 0.89 and rs = 0.86 for the ‘union exon’ and ‘transcript’ methods). As suggested by the scatter plots in Additional file Gene expression levels are typically derived from RNA-seq-based data by intersecting the genome coordinates of ‘uniquely-mapped’ reads with the loci of annotated genes and taking into account the length of the transcript that is expressed from a given locus. As may be immediately apparent, this procedure has several limitations. The first is that it is generally unclear what transcript to consider for each locus, when correcting for transcript length. What is typically used is the total length of the ‘union exons’, which is clearly incorrect when the gene expresses multiple isoforms with different relative abundances and different sequences of exons. A second drawback is that the proportion of reads that are discarded depends on the repeat content of the gene with an unknown impact on the accuracy of gene expression estimates. Finally, reads that map across splice boundaries and are informative particularly for estimating the expression of individual isoforms, may be discarded by the simple counting procedure. This problem will preferentially affect expression estimates for genes with a large number of exons and isoforms. Thus, one expects that even gene-level estimates of abundance are improved by the appropriate treatment of transcript isoforms. To test how accurately gene expression levels could be estimated by the benchmarked methods compared to count-based methods, we implemented two variants of count-based gene expression level estimation , but drop most strongly for the bin with the least expressed genes (mean rs = approximately 0.68). Thus, our analysis confirms the expectation that low abundance and, consequently, sparse transcript coverage leads to noisier estimates of expression. However, for genes whose expression levels are in the top three quartiles, the estimates provided by the tools agree very well with the ‘true’ expression levels.Higher transcript coverage by reads is expected to increase the accuracy with which transcript abundance is estimated. The coverage depends on both the depth of sequencing as well as on the transcript abundance, and indeed we found that the size of the read library has a positive influence on the accuracy of expression estimates. To evaluate the extent to which ‘true’ abundance influences the accuracy of transcript abundance estimates, we grouped both expressed transcripts and genes by their ‘ground truth’ expression into four equally sized bins: low (log2 TPM <0 or 1.1), medium-low (0 or 1.1 < log2 TPM <3 or 4.1), medium-high (3 or 4.1 < log2 TPM <5.5 or 6.2) and high abundance (log2 TPM >5.5 or 6.2), with the first and second numbers referring to the ranges for transcripts and genes, respectively. The overall ranking of tools in terms of their accuracy within expression level bins Fig.  largely Because different methods appear to handle quite differently transcripts with very low abundance, we sought to further investigate their accuracy in this expression range in particular. More specifically, we determined the rates at which: (1) transcripts or genes that are not expressed are estimated to have non-zero expression ; and (2) transcripts or genes that are expressed and are also inferred by a tool to have non-zero expression levels (true positive rate). It should be noted that when dealing with real rather than synthetic datasets, one does not know whether a specific transcript truly had a copy number of 0 in the sample or not. When no evidence of expression is found, some of the Bayesian methods (BitSeq and MMSEQ) strictly assign non-zero ‘prior’ expression probabilities to transcripts, and thus they do not, strictly speaking, produce any ‘false negatives’. Nevertheless, even for these methods it may be relevant to determine how well very limited evidence of expression is handled, and whether transcripts with no such evidence really get assigned ‘prior’ expression values. Thus, after consulting the developers, we have assigned transcripts with an expression estimate which was essentially the method-specific prior value an estimate of zero see , and thes = 0.05 or rs = 0.03 on the transcript- and gene-level, respectively. On the other hand, CEM produced slightly more accurate results when supplied with TopHat-aligned reads, particularly when considering all features (gain of rs = approximately 0.08). Correspondence with CEM’s developers revealed that the program requires the TopHat-specific SAM/BAM tag ‘XA’, which encodes information about the strand of the transcript to which a read aligns, to correctly parse multi-fragment reads. Because this tag was not supplied in our input alignment files, CEM was unable to properly parse alignments that covered splice junctions and therefore produced less accurate estimates when supplied with our alignments.Some of the surveyed methods strongly recommend the use of a specific short-read alignment program. By default, RSEM even calls such an aligner (Bowtie) internally. Thus, we asked whether the choice of alignment program impacts the accuracy of isoform abundance estimates that are produced by these methods. Surprisingly, we found that the aligner has a relatively small impact on estimation accuracy, regardless of whether one considers transcripts or genes, and only expressed or all features . In contrast, the estimates produced by IsoEM seemed to slightly improve upon enabling the bias correction, but only when all transcripts were considered (rs gain = approximately 0.02). In all other cases, no appreciable differences were observed when executing programs with or without bias correction.A subset of the methods  also attempt to correct various biases that occur during sample preparation, such as positional , sequencing (depending on the nucleotide composition of the reads), or mapping (sequencing errors and multi-mapping reads) biases , and the number of exons of which they are composed. Likewise, we sorted genes by the number of annotated transcript isoforms. Reasoning that the influence of gene structural features on estimation accuracy is likely to be small compared to that of expression level differences, we concentrated on transcripts with mid-range expression, where differences should be most clearly apparent. For each of the structural features, we then defined non-overlapping bins containing comparable numbers of transcripts or genes. Additional file P values of the Kolmogorov-Smirnov’s goodness of fit tests carried out for the log-ratio of estimated and expected levels for genes/transcripts in specific bins compared to the entire set of genes/transcripts with moderate expression level  transcripts in the two human or mouse replicates, was generally high. At the gene level, rs ranged from approximately 0.82 for both human (Cufflinks) and mouse (MMSEQ) to approximately 0.91  and 0.90 . In contrast, at the transcript level, the agreement was much lower and varied considerably between tools, in the range of approximately 0.62 (TIGAR2) and 0.60 (MMSEQ) to approximately 0.95 and 0.91 (both Scripture) for human and mouse , likely because a large proportion of isoforms are expressed at low levels or not at all. In a few cases, differences between replicate agreement on the gene and transcript level exceed 0.2 in at least one species . On the other side of the spectrum, Scripture exhibits a slightly higher agreement between its transcript than its gene level estimates across both organisms . These behaviors likely reflect differences in the models underlying different methods, particularly with regard to how they treat low abundance transcripts and how readily they assign reads to the minor and major isoforms of a given gene.A basic test for any inference method is whether they produce similar results when supplied with similar data. For isoform quantification, reproducibility was generally tested on data that was generated synthetically. To investigate this aspect, here we have also prepared RNA-seq libraries from two batches of cells of two cellular systems, the murine fibroblast cell line NIH/3T3 and the human T cell line Jurkat. We then supplied the tools for inferring transcript isoform abundances with the resulting short reads (Sailfish) or alignments . The replicate agreement, defined as the Spearman correlation coefficient rWhile the tools for inferring isoform abundance have been quite extensively tested on simulated data, obtaining independent and comprehensive experimental reference data is not trivial. Quantitative PCR (qPCR) is the experimental method of choice for the quantification of transcript abundance. However, despite recent technological advances allowing qPCR experiments on a large-scale level, these methods are still cost- and resource-intensive. We therefore applied our A-seq-2 protocol  to prepas >0.97), suggesting that gene expression and 3′ end processing site usage are highly similar in the replicates that we obtained from both human and mouse cells.To assess the quality of these data we first quantified and compared the usage of annotated 3′ end processing sites that overlap the ends of GENCODE-annotated transcripts see  between Because in constructing the catalog of 3′ end processing sites from published data we applied stringent validation criteria, the set of ‘known’ sites is probably biased towards those that are used in relatively abundant transcripts. We therefore wondered whether the agreement between biological replicates is higher when one focuses only on the GENCODE transcripts that end in a ‘known’, annotated 3′ end processing site and that are likely to be polyadenylated. This was the case for 46,801 human and 26,821 mouse transcripts  and RSEM (rs = approximately 0.88) delivered the most accurate estimates at the gene level for human and mouse data, respectively. Importantly, we found that even when estimating gene-level abundance from biological data, isoform-aware methods yield more accurate results than the broadly used count-based methods. Across all methods, level of coarse-graining, and organisms, the second replicate yields estimates that are slightly more accurate, likely reflecting a batch effect pertaining to the preparation of RNA-seq and A-seq-2 sequencing libraries. On all levels, differences in accuracy between most methods are rather small, similar to what we observed on synthetic data. Also similarly, enabling or disabling bias correction in those methods that provide such an option also did not substantially alter the accuracy of estimates on experimental datasets , and ‘antisense’ in both human and mouse. For human, the 12,513 protein coding genes amenable to quantification by A-seq-2 are considerably more accurately quantified than lincRNA (739) and antisense genes (739), with Spearman correlation coefficients reaching values of up to approximately >0.85, 0.8, and 0.7, respectively, for the different gene classes Fig. . The absAccurate quantification of gene expression is essential for the understanding of gene regulatory processes in health and disease. Due to its large dynamic range, high reproducibility, and the ability to detect previously unknown transcripts, RNA sequencing has become the method of choice for global expression profiling. However, despite the digital nature of the resulting data, technical limitations (limited read length and non-uniform transcript coverage) render their analysis challenging, especially when large and complex genomes of higher eukaryotes, with frequent repeats and overlapping gene structures, are involved. Accurate computational methods for RNA-seq data analysis therefore remain in high demand. This is reflected in the large number of computational methods for estimating transcript isoform abundance that were developed over the course of the last 6 years. Naturally, the question arises which method should best be used in a particular context. Here we have tried to address this question in depth, using not only synthetic data, as is typically done when the computational methods are developed, but also using estimates that were obtained with an independent experimental method for a specific type of isoforms, namely those that arise from alternative polyadenylation. This is because methods for global quantification of 3′ end site usage distinct from RNA-seq are available , 29–31 aExpecting that most users would – at least initially – run the methods ‘out-of-the-box’, we sought to apply the surveyed methods with default settings, and departed from this general rule only to test the influence of specific options that the developers of the methods proposed. Although we found that neither the use of recommended short read aligners nor the activation of bias correction generally improved estimation accuracy, it is likely that the developers of the individual methods or experienced users would be able to improve the performance of individual tools in specific settings. During the course of this study we discovered a number of assumptions that the programs tacitly made and that affected the interpretation of the results. Therefore, a specific recommendation that we can make to developers is to ensure that sufficiently detailed information on input requirements, potential pitfalls and the implication of specific options  is provided.Encouragingly, we found that most of the methods that are currently used to estimate transcript isoform abundance produce quite comparable and accurate results, both on synthetic and experimental data. As a general trend, methods such as Scripture and Cufflinks, whose main objective is to assemble/reconstruct transcript isoforms but that have also been co-opted for estimating isoform abundance, perform poorer than methods specifically designed for the latter purpose. However, such methods could be part of the initial assembly of a comprehensive set of transcripts whose expression can be subsequently quantified with a different approach . CufflinNext to a general assessment of the accuracy of expression estimates produced by the tools, we also studied the impact of several transcript properties on the accuracy of expression estimation. We found that parameters that directly influence the coverage of a transcript or gene by reads, such as sequencing depth and true expression level, have a positive influence on estimation accuracy, as has been observed before . On syntTo ensure the widest applicability of our findings, we have based our study on single-end, short read (50 nt) data. Illumina’s paired-end sequencing technology, which has been employed in previous comparisons of isoform abundance estimation methods , 33, proWhile most methods produce comparable and fairly accurate estimates of transcript isoform abundance, they differ more strongly in their computing needs. In some cases, speed comes at the cost of increased memory requirements, which is evident for example with IsoEM, which is extremely fast, but uses tens of GB of memory. Nonetheless, with the increase in the number and size of the datasets that one typically analyzes, speed and scalability of processing become very important considerations for the utility of a program. The recently developed Sailfish is of particular interest in this regard because its running times scale well within the tested range of sequencing depths, while the memory footprint remains reasonable. Moreover, its alignment-free k-mer-based approach disposes of the time-consuming step of aligning reads to a reference genome or transcriptome. For typical datasets of approximately 100 million reads, most programs use 1–20 GB of memory and run for 1–2 h. An exception is TIGAR2, which produces highly accurate expression estimates that come at the cost of both high running times and high memory use.One important aspect that was beyond the scope of the current study is that in many studies, the interest is the identification of transcript isoforms that are differentially expressed between two conditions, rather than the quantification of isoform abundance in a specific condition. The estimates of isoform abundance inferred with the methods that we tested here can in principle be used in subsequent statistical tests for differential expression, but the issue of the underlying model has not been entirely addressed. If sufficient replicates are available, two-sample parametric or non-parametric tests can be used. However, due to the high costs of RNA-seq experiments, the availability of more than a few replicates is very rare. Instead, when the number of replicates is small, accurately accounting for the different sources of variability in the data is important. Differential expression analysis based on RNA-seq data is frequently done with programs such as baySeq , DEGSeq In summary, several methods for the inference of transcript isoform abundance can accurately quantify expressed transcripts even from relatively small short-read libraries and should thus be adequate for the analysis of both past and present RNA-seq datasets. Their performance is largely not affected by structural features  of the genes/transcripts, although, as expected, abundant transcripts are quantified more accurately compared to rare transcripts. Importantly, our analysis indicates that the explicit quantification of transcript isoforms leads to more accurate estimates of gene expression levels compared to the ‘count-based’ methods that are broadly used currently. Given the wealth of tools available, the user can largely base his choice of method on criteria related to usability, available processing and memory capacities, compatibility with pre-existing data processing pipelines, and the desired downstream analyses (see Table The hg19 (human) and mm10 (mouse) genome assemblies were obtained from UCSC Genome Bioinformatics, University of California, Santa Cruz . Haplotyin silico reads, we have used the Flux Simulator software [To generate software , with thsoftware . Flux Sisoftware . Finallysoftware  by speciHuman Jurkat T lymphocytes (ATCC TIB-152)  and NIH/Potential 3′ adapter and poly(A)-tail fragments were sequentially removed from FASTQ-formatted short reads sequences with two iterations of cutadapt , specifyin silico-generated read subsets (FASTA-formatted), were aligned to the genome and transcriptome of the respective species with segemehl 0.1.7 [The experimentally obtained sequence sets, as well as the five hl 0.1.7 , with dein silico-generated reads, which did not have such scores associated, strings of ‘I’ characters that match the length of the read sequence (SEQ field) were used to denote maximum quality scores (according to the Sanger FASTQ format). In the case of the experimental RNA-seq libraries, we used the quality scores that were provided in the initial FASTQ files that were obtained from the sequencing facility. The data processing was automated with the help of the Anduril [in silico generated reads with Bowtie (version 1.0.0) [During the course of the study, we have noticed that the transcript isoform quantification methods that we evaluated make certain assumptions about the format of the input alignment files and that in some cases these assumptions only hold for certain short read aligners or for outdated file formats. We therefore implemented additional post-processing steps to ensure that the information required by individual programs is present in the alignment file is the appropriate form. (1) We ‘uncollapsed’ the reads: across all alignment files, alignments corresponding to collapsed reads were ‘cloned’, but a randomized QNAME name was assigned to each individual read that was only re-used for additional alignments of the same read. (2) To avoid misinterpretation of tag fields, all custom segemehl tags were removed. (3) Reads aligning to more than one reference locus are reported by segemehl as individual alignment records with identical read names (QNAME field). In accordance with the SAM specifications , we have Anduril  data anan 1.0.0)  and TopHn 1.0.0) . The outThe reads obtained with the A-seq-2 protocol for 3′ end sequencing have a particular structure: they are the reverse complement of 3′ end RNA fragments and further have the sequence AAANNNN downstream of the actual 3′ end  for detaWith the exception of Sailfish (see below), all of the programs compared in this study use alignments of reads to either the transcriptome or the genome. We used the samtools  suite toScripture and CEM require annotation files in a BED-based format which supports multiple fragments  per entry and is known as BED12 or BED12+3. These were generated from the GENCODE-provided GTF annotation files with the help of the R/Bioconductor package rtracklayer . BecauseBitSeq , 68 usesparseAlignment \ <alignments_transcriptome> --trSeqFile \<sequences.fa> --outFile \ <out_prefix.prob> --trInfoFile \<out_prefix.trx> --uniform \The first step in BitSeq is to parse the alignment file to calculate probabilities of individual reads originating from individual transcripts:estimateVBExpression <out_prefix.prob> \ --outPrefix <out_prefix> \--outType RPKM --trInfoFile \ <out_prefix.trx> --samples 1000 --seed 1 \Then the mean transcript expression is estimated with a Variational Bayes inference algorithm:By default, when no read alignments are assigned to a given transcript, BitSeq sets the expression estimate of the transcript to a ‘prior’ that depends on the effective transcript length and the sequencing depth. When indicated and in communication with the developers, we have identified these cases by finding transcripts for which the expected read count  equals exactly 1 and replaced their RPKM estimates with zeros.python runcem.py --annotation \ <annotations.bed12> --forceref \--prefix <out_prefix> \ <alignments_genome.bam> \CEM  takes asWhere indicated, we have set the --usebias option to evaluate CEM’s built-in bias correction functionality, which attempts to correct for positional, sequencing, and mappability biases.cufflinks --GTF <annotations.gtf> \--library-type fr-secondstrand \--frag-len-mean <fragment_length_mean> \ --frag-len-std-dev \<fragment_length_sd> --multi-read-correct --output-dir <out_dir> \ <alignments_genome.bam> \Cufflinks  takes asOnly expression estimates with ‘fpkm_status’ ‘OK’ were considered. All other estimates were set to zero.express --no-update-check --f-stranded \ --frag-len-mean \<fragment_length_mean> --frag-len-stddev <fragment_length_sd> \--output-dir <out_dir> <sequences.fa> \<alignments_transcriptome.bam> \eXpress  takes asAs eXpress is correcting for biases introduced during library preparation  by default, we have set the --no-bias-correct option when evaluating the performance of methods without bias correction.isoem --GTF <annotations.gtf>--\fragment-mean \<fragment_length_mean> --fragment-std-dev \<fragment_length_standard_deviation> \--directional -o <out_file> \<alignments_genome.sam> \IsoEM  takes asIsoEM also attempts to correct for fragment sampling biases resulting from random hexamer priming during reverse transcription and to evaluate this functionality, we have generated isoform abundance estimates with the -b option.bam2hits <sequences.fa> \ <alignments_transcriptome.bam> > <hits> \MMSEQ   values, we first exponentiated them for our analyses. Similar to BitSeq, MMSEQ defaults to assigning ‘prior’ expression estimates to those transcripts for which no read/alignment evidence can be found. Where indicated, and in communication with the developers, we have identified such cases by substituting in MMSEQ’s output the log μ estimates for all transcripts or genes with a ‘unique_hits’ count of 0 with ‘NA’.rsem-prepare-reference --no-polyA--transcript-to-gene-map <gene_id_transcript_id_table> <sequences.fa> <index_prefix> \RSEM , 35 , we purged the alignment file of any entries that contained disallowed characters in their CIGAR string fields . After recalculating read length distributions across the resulting alignment files, we estimated maximum likelihood expression levels as follows:To evaluate RSEM’s built-in bias correction functionality, which attempts to correct protocol-specific 5′ or 3′ positional biases, we have set the --estimate-rspd (read start position distribution) option where indicated.rseq expression_analysis <sequences.fa> \<alignments_transcriptome.sam>rSeq  takes assailfish index -t <sequences.fa> -o \ <index> -k 20 \Sailfish   for the human and mouse samples to a standard library size of 1 million reads. We refer to these measures as transcripts per million transcripts (TPM) and processing regions per million processing regions (PPM), respectively. Since the benchmarked methods already supplied estimates in normalized expression units, no further processing of these values was required. In particular, we have used the reads/fragments per kilobase of exon model per million mapped reads (RPKM/FPKM) units wherever available, thus accounting not only for differences in library sizes but also for differences in transcript lengths. The latter is necessary because the number of fragments obtained from a given RNA during library preparation, and thus the read count for that transcript, is proportional to its length . Only inin silico-generated data) or the filtered set of transcripts  to the BED12 / BED12+3 format, a tabular format able to encode gaps. We then intersected the genomic alignments for each dataset with the corresponding annotation file using bedtools mode intersect [l for which overlaps were reported for a given read. For each of these, we then added i. For each library, the counts produced by each of these counting methods were then converted to RPKM by dividing by (1) the total number of reads that could be successfully aligned to the genome and (2) the total length (in nucleotides) of the ‘union exons’ (see above) of the considered transcripts, followed by multiplication by 1 billion.Although our primary interest was to assess the accuracy of methods for isoform expression profiling, a lot of studies rather limit themselves to gene-level estimates of gene expression. The question then arises of how the methods that are used for obtaining gene-level estimates compare with those that are specifically designed for estimating isoform abundance but can be co-opted for the estimation of gene-level expression levels as well. One method for estimating gene-level expression is ‘union exon’-based counting. To implement this method we have selected the exon entries from the GENCODE annotation files, grouped them by the ENSEMBL gene identifier, and merged overlapping exons for each gene. When analyzing human or mouse data, we have discarded the exons of transcripts that do not end in annotated poly(A)-processing regions or that are unlikely to be polyadenylated, analogous to the filtering that we applied to transcripts used in the benchmarking (see above). We then generated per-gene counts by intersecting the genomic alignments of the different datasets with the resulting ‘pseudoexons’, using the function summarizeOverlaps of the R/Bioconductor package GenomicAlignments  with optntersect  such thaWe assessed the accuracy of the methods in terms of Spearman correlation coefficients between the known (simulated data) or independently estimated abundances (A-seq-2) and the abundances inferred with the individual methods. Depending on the type of data and analysis, we applied this procedure on the level of transcripts, poly(A)-processing regions, and/or genes, either considering all features or subsets thereof, grouped by common features . Where indicated, we have further computed the Pearson correlation coefficient and the root mean square error. In these cases, we have first set all expression levels (true or estimated) below 0.03125 (the log2 of which is −5) to that value and log2-transformed the resulting ‘pseudocount’-adjusted values.in silico-generated read files are available in the Sequence Read Archive (SRA) [Raw sequencing (RNA-seq and A-seq-2) and ve (SRA)  repositove (SRA) . As the ve (SRA) . The pag


High-quality expression data are required to investigate the biological effects of microRNAs (miRNAs). The goal of this study was, first, to assess the quality of miRNA expression data based on microarray technologies and, second, to consolidate it by applying a novel normalization method. Indeed, because of significant differences in platform designs, miRNA raw data cannot be normalized blindly with standard methods developed for gene expression. This fundamental observation motivated the development of a novel multi-array normalization method based on controllable assumptions, which uses the spike-in control probes to adjust the measured intensities across arrays.Raw expression data were obtained with the Exiqon dual-channel miRCURY LNA™ platform in the “common reference design” and processed as “pseudo-single-channel”. They were used to apply several quality metrics based on the coefficient of variation and to test the novel spike-in controls based normalization method. Most of the considerations presented here could be applied to raw data obtained with other platforms. To assess the normalization method, it was compared with 13 other available approaches from both data quality and biological outcome perspectives. The results showed that the novel multi-array normalization method reduced the data variability in the most consistent way. Further, the reliability of the obtained differential expression values was confirmed based on a quantitative reverse transcription–polymerase chain reaction experiment performed for a subset of miRNAs. The results reported here support the applicability of the novel normalization method, in particular to datasets that display global decreases in miRNA expression similarly to the cigarette smoke-exposed mouse lung dataset considered in this study.R software package called ExiMiR and deposited in the Bioconductor repository.Quality metrics to assess between-array variability were used to confirm that the novel spike-in controls based normalization method provided high-quality miRNA expression data suitable for reliable downstream analysis. The multi-array miRNA raw data normalization method was implemented in an  MicroRNAs (miRNAs) are short (on average 22 nucleotides long) non-coding RNA molecules that are expressed in almost all eukaryotes. They post-transcriptionally repress the expression of their target genes using the cellular RNA interference mechanism . The latExpression profiling frequently serves as the basis of investigations involving miRNAs. In recent years, high-density oligonucleotide arrays have become the most popular profiling technology because they allow the simultaneous quantification of hundreds of miRNAs, similar to the way mRNAs are quantified in transcriptomics. This technology is currently less expensive than high-throughput sequencing and faster than quantitative reverse transcription–polymerase chain reaction (RT-qPCR); however, it is also noisier than RT-qPCR because of the high-throughput approach . CommercThe raw data generated by all of these miRNA profiling platforms must be normalized before the results from any single hybridized array can be compared with the results from another. The normalization step involves correcting for the experimentally unavoidable biases associated with each hybridized array, which would otherwise compromise the reliability of the subsequent comparisons. Several computational approaches to address this task have been developed. These approaches differ in various aspects such as the signal intensity dependence of the correction, or the single- or multiple-array design -14. In ret al. used the signal intensities of the control probes that targeted the spike-in RNA as the calibration quantities. These molecules constitute an ideal calibration means because they are designed to exhibit the same concentration in all the samples that are hybridized on the arrays. A further feature is the hybridization strategy of the RNA samples, which follows the single- or dual-channel design inherent in the profiling technology. The dual-channel Exiqon platform that was used in the present study can lead to complications in the case of multi-factorial experimental designs. A simplifying strategy is the “common reference design”, which hybridizes the same reference sample on the first channel across the whole experiment and, at the same time, hybridizes the actual samples on the second channel. In this case, a “pseudo-single-channel” normalization approach is possible, as it has already been tested for mRNA array data  interval. Because SCN and SCVSN both use the intensities of spike-in control probes to perform the raw data normalization, relativeCVspike is expected to be more advantageous for them. Figure relativeCVspike distributions need to be considered together with the relative medians to describe the results properly. The lowest relative medians of SCVSN, least-variant set (LVS), and invariant set (IS) were indeed accompanied by large IQRs. Inversely, SCN displayed a low relative median and the overall smallest IQR value. Rank-invariant normalization from the LUMI package (LRIN), median normalization (MEDIAN), and QN displayed larger relative medians and intermediate IQR values compared with LVS and IS, the other two members of the “stronger” group . Overall, the results shown in Figure r) was used to quantify the similarity of the values obtained for the 24 selected miRNAs, because this statistic is appropriate when comparing values obtained with different technologies that are related in a non-linear manner. The results supported the global decrease in miRNA expression observed in Figure r value of approximately 0.8 compared with the Affymetrix-based AQN approach, which achieved an r value of approximately 0.5. To compare the Exiqon-based pipelines in a meaningful way, the robustness of Spearman’s coefficient was assessed by estimating the 95% confidence intervals using a simple leave-one-out re-sampling approach. The results confirmed the results that QN was slightly separate from the other Exiqon-based approaches  and GC content RMA (GCRMA), which are widely used for gene expression analysis, assume that the distribution of the probe intensities is the same across all the arrays. The normalized intensity values for each probe are then obtained using this common distribution. For miRNA arrays, other invariant features have been used to normalize the data − for example, the “nonchanging” or “least-variant” probes ,17. In tn method ,25. A sin method , which, Various types of array-specific biases that can affect the raw data should be considered before examining the validity of the four assumptions A0–A3 that were applied in developing the SCN approach. The biases can be grouped under the following three categories based on their source:1. Biases that originate from the “biological” RNA, which affect only the miRNA probes. These biases arise from both biological variability, as observed in biologically replicated samples, and treatment-induced effects, as observed in differentially treated samples.2. Biases that originate from the “synthetic” spiked-in RNA, which affect only the spike-in control probes.3. Biases that appear only after the “biological” and “synthetic” RNAs are mixed, which affect both the spike-in control probes and the miRNA probes.ΔE from Figure ΔE .Simply stated, assumption A0 affirms that, in the context of normalization, the effect of bias (3) dominates the effects of biases 1) and (2), which indicates that the correction function  and (2),a posteriori by the results shown in Figure CVbetween, the variability in probe set intensities observed over biological replicates with the SCN approach was similar to the variability in intensities with alternative approaches, except for the methods  that formed the “stronger” group, which, however, performed less well on the other metrics  could be disregarded was confirmed  could beΔE illustrated in Figure Assumptions A1–A3 are aimed at the possibility of providing a practical correction for effect of bias (3). Unlike assumption A0, these three assumptions can be verified explicitly based on the raw data. The features tested, such as the concentrations of the synthetic spiked-in RNAs, depend more on the experimental procedures than on the actual array technology. Assumption A1 explicitly tests the coherence of the deviations across intensities and arrays shown in Figure a” and the noisier “f” and “j” probe sets. The residuals below 30%, as well as the variability among probe sets, can be attributed to uncontrolled factors such as the partial hybridization of the spike-in control probes with the “biological” RNA. The extent of this partial hybridization depends on the specific spike-in control probe sequences and on the variable abundances of the expressed miRNA sequences.The variance ratios in Figure ExiMiR was developed as an R package and deposited in the Bioconductor repository. ExiMiR executes the SCN pipeline for input raw data in various formats and compliant with assumption A0, and tests assumptions A1–A3 to ensure the applicability of the method. If the assumptions are not met, the MEDIAN is used instead of SCN, as discussed above. The “pseudo single-channel” preprocessing approach based on the “common reference design” is fully exploited in ExiMiR. For example, both raw and normalized data from the dual-channel Exiqon arrays are stored as single-channel “AffyBatch” and “ExpressionSet” R objects, so that the data processing is almost the same as for single-channel Affymetrix arrays.Because the validity of the SCN method for the “common reference design” strategy for dual-channel Exiqon arrays has been established, its practical implementation could be addressed. Therefore, for the implementation of SCN, a software tool called The outcomes of the SCN preprocessing pipeline were compared with the outcomes from the alternative approaches for normalizing miRNA microarray data that are listed in Additional file CVbetween, where it was intermediate , in perfect agreement with its design. The relativeCVspike distributions of the SCVSN method produced interquartile ranges that were significantly larger than the distributions observed for SCN. The second steps of SCVSN and SCN are very similar  were not altered by the normalization procedure. Therefore, although the RT-qPCR results for the selected differentially expressed miRNAs  yielded distributions that were lower overall than the SCN distributions. These results apparently suggest that the other methods produced better results than SCN. This finding is examined in more detail in the next two paragraphs.The results of the extensive comparisons between the SCN method and the alternative approaches showed that SCN performed very well, except when measured using ods” and ), which s Figure  showed ts Figure . The onlCVbetween distributions, and also yielded comparable treatment-induced responses  is used extensively in gene expression analyses, it is worth examining its performance in some detail. The underlying assumption of an invariant distribution of the probe intensities between arrays is not guaranteed to be fulfilled for miRNA arrays, as explained in the “Background”. The effects of this on the results are now examined. First, although quantile normalization preserves the relative ranks of the miRNA differential expression values, their signs were strongly affected, as shown by the results in Figure e Figure D revealst Figure C. These t Figure C and by ΔE used in SCN can become, at worst, an intensity-independent constant  method uses a more sophisticated non-linear transformation of the raw data, its performance is very similarly to MEDIAN, except when measured by relativeCVspike. The low number of measured probes on a miRNA array compared with the number on an mRNA array, may make it difficult for VSN to work efficiently. A similar result was observed for SCVSN, where the variance stabilization produced a broad relativeCVspike distribution when applied only to the spike-in controls , it constitutes a priori a sounder choice than approaches such as the quantile normalization that were originally developed for gene expression arrays. To characterize its features concretely and to compare it with alternative normalization methods, several quality metrics were introduced to measure the variability of the raw and normalized expression data at different levels . These metrics showed that compared with the other tested approaches, the novel SCN method best preserved the treatment-induced signal and reduced the variability between replicated measurements in a consistent and stable manner. The metrics also led to a deeper understanding of the features and relationships among the tested preprocessing pipelines for miRNA expression data. These findings have provided important information that will help researchers make informed choices regarding the best miRNA preprocessing pipeline to use in a particular study. The array-based results were supported by RT-qPCR measurements, which confirmed the value of the novel SCN method and the suboptimal performance of a quantile normalization approach. The RT-qPCR results also revealed a global decrease of miRNA expression in response to cigarette smoke exposure for which the novel SCN method is an appropriate approach. Further, the biological interpretation of the observed differential expression of four miRNAs revealed the possible relevance of the identified miRNAs.A novel multi-array method for normalizing miRNA raw data called the SCN method was assessed in this study. This method uses spike-in control probes to adjust the intensity values across arrays. It is implemented in an Male A/J mice , 9 to 13 weeks of age, were exposed in whole-body exposure chambers to conditioned fresh air (“sham”) or to diluted cigarette mainstream smoke from the reference cigarette 3R4F  at a target concentration of 75 (“low”), 150 (“medium”), or 300 (“high”) μg total particulate matter (TPM)/liter for 6 hours/day, 5 days/week. For the medium and high groups, exposure was started with a dose adaptation period of 2 hours/day on day 1, and gradually increased to reach the target concentrations by day 17.http://www.aalas.org).These exposure experiments were part of a larger study , which wBlood samples and lung tissue from mice exposed for either 1 day or 5 months were collected between 1 and 3.5 hours after the last exposure or after 24 hours (1 day post-exposure) from four mice per group. Lung tissue was snap frozen in liquid N2 immediately after dissection and stored at −70°C. Blood was collected directly upon dissection using the Mouse RiboPure™-Blood RNA Isolation Kit .Total RNA (including miRNA) was extracted from frozen left lung lobes and from whole blood using a mirVana™ miRNA Isolation Kit . The quality and quantity of the extracted RNA were analyzed using an Agilent 2100 BioAnalyzer  and a NanoDrop ND-1000 , respectively.In this study, the Exiqon  and Affymetrix  miRNA platforms were used.a”, “b”, “c”, “d”, “e”, “f”, “g”, “h”, “i”, and “j” were spiked into the labeling reaction at various concentrations. Hybridization was performed according to the miRCURY LNA™ array manual. Following hybridization and processing, the arrays were scanned using the Agilent G2565BA Microarray Scanner System  and image analysis was performed using the ImaGene 8.0 software  to quantify the signals. The raw data were provided by Exiqon as ImaGene text files.The experiments using the Exiqon platform were performed by Exiqon . Briefly, total RNA samples (0.75 μg) and reference RNA samples were labeled with Hy3™ and Hy5™ fluorescent labels, respectively, using the miRCURY LNA™ Array power labeling kit (Exiqon), following the procedures described by the manufacturer. No dye swap experiments were performed. Each Hy3™-labeled sample was mixed with the Hy5™-labeled reference RNA sample and then hybridized on the miRCURY LNA™ miRNA Arrays (v.11.0) following the “common reference design” . For eacTotal RNA (1 μg) was labeled using the FlashTag™ Biotin RNA Labeling kit  according to the supplier’s manual. The RNA of spike-in controls  were processed along with each sample RNA. The biotin-labeled RNA was hybridized on GeneChip® miRNA Arrays (Affymetrix). The wash and stain procedures were performed according to Genisphere’s suggestions on Affymetrix protocols . The arrays were processed using the automated washing and staining protocol specified in the Fluidics Script FS450_03 (Affymetrix). After washing, the array was scanned using the GeneChip Scanner 3000 7G and the raw image data were saved in DAT files. Command Console Software (Affymetrix) was used to automatically grid the DAT files and to create the CEL files (probe cell intensity data).normexp background correction, a within-array lowess normalization and a median summarization . The range of the CVbetween distribution is taken as the two extreme values within the interval , where Q1 and Q3 are the first and third quartiles. This interval corresponds to the black whiskers in the boxplots used in the figures. Because it is not necessary equal to the interval given by the smallest and the largest values of the CVbetween distributions, relativeCVspike may have values outside the  interval.The “relative spike-in” coefficient of variation Ejk of normalized expression defined over all samples k and the corresponding normalized dose vector Dk, a simple linear model is used to determine the treatment-induced differential expression for each probe set j. The corresponding (moderated) t-statistics were calculated with the Bioconductor limma package [Given the matrix  package .Ct-method. First, the average of the Ct values, which were determined in triplicate, were calculated and the ΔCt values were calculated as ΔCt = Ct(target miRNA) – Ct. The ΔCT data were used to calculate the relative miRNA differential expression in the samples from the smoke-exposed animals as -ΔΔCT = ΔCT(“high”) – ΔCT(“sham”). The following Applied Biosystems TaqMan® miRNA assays were used: mmu-miR-146a (ID 000468), mmu-miR-342-3p (ID 002260), mmu-let-7i (ID 002221), mmu-miR-31 (ID 000185), mmu-miR-221 (ID 000524), mmu-miR-135b (ID 002261), mmu-miR-147 (ID 002262), mmu-miR-146b (ID 002453), mmu-miR-21 (ID 002493), mmu-miR-144 (197375_mat), mmu-let-7b (ID 002619), mmu-let-7c (ID 000379), mmu-miR-30a (ID 000417), mmu-miR-26a (ID 000405), mmu-let-7a (ID 002478), mmu-miR-30c (ID 000419), mmu-let-7f (ID 000382), mmu-let-7d (ID 001178), mmu-let-7 g (ID 002492), mmu-miR-34c (ID 002584), mmu-miR-99a (ID 000435), mmu-miR-195 (ID 000494), mmu-miR-125b-5p (ID 000449), and mmu-miR-99b (ID 000436). snoRNA135 (ID 001230) was used as the internal control RNA. Pre-experiments revealed that the linear range of the RT-qPCR was at 5 ng RNA per reaction. However, for one assay (mmu-miR-99b), 2.5 ng RNA was determined to be in the linear range of the RT-qPCR.Applied Biosystems TaqMan® miRNA assays were used for RT-qPCR. Because of its very high sensitivity  and specificity, RT-qPCR is commonly used to confirm the results obtained with other platforms. TaqMan® MiRNA Assays use a stem-looped primer for reverse transcription and a sequence-specific TaqMan® assay. miRNA-specific cDNA was generated by reverse transcription with a Thermal Cycler . The qPCR reaction and signal detection were performed with an Applied Biosystems 7900HT instrument according to the manufacturer’s instructions. The relative expression of the miRNAs was determined by the comparative The authors declare that they have no competing financial or other interests in relation to the work described in this manuscript.R, and wrote the manuscript. SG wrote the ExiMiR package. UK, EV, and WH supervised the generation of the sample and raw data. AH supervised the generation of the raw data and wrote the biological interpretation. JH and MCP developed the overarching strategy and supervised the project. All authors read and approved the final manuscript.AS developed and applied the methodology, led its implementation in Table S1. miRNA raw data preprocessing pipelines used in this study.Click here for fileSupplementary results. Figure S1. Comparison of the raw intensities and of the “present”/”absent” calls. Figure S2. Inapplicability of the spike-in control based normalization for the Affymetrix lung dataset. Figure S3. Quality control metrics for the lung samples and all preprocessing pipelines. Figure S4. Quality control metrics for the blood samples and all preprocessing pipelines. Figure S5. Differential miRNA expression for the lung samples and all preprocessing pipelines.Click here for file


Principal components analysis (PCA) is a common unsupervised method for the analysis of gene expression microarray data, providing information on the overall structure of the analyzed dataset. In the recent years, it has been applied to very large datasets involving many different tissues and cell types, in order to create a low dimensional global map of human gene expression. Here, we reevaluate this approach and show that the linear intrinsic dimensionality of this global map is higher than previously reported. Furthermore, we analyze in which cases PCA fails to detect biologically relevant information and point the reader to methods that overcome these limitations. Our results refine the current understanding of the overall structure of gene expression spaces and show that PCA critically depends on the effect size of the biological signal as well as on the fraction of samples containing this signal. Principal components analysis (PCA) is frequently used in biological sciences for global analysis of omics datasets. It provides fully unsupervised information on the dominant directions of highest variability in the data and can therefore be used to investigate similarities between individual samples, or formation of clusterset al.In most of these studies, scientists focus on the first two to four principal components (PCs), assuming that higher-order components mainly contain irrelevant information or noiseet al.Another study by Lukk Finally, there is also some evidence from wet-lab biology that gene expression of a distinct cell type might be mainly controlled by only a few independent factors, due to the possibility to reprogram various adult somatic cell types by exogeneous expression of only four transcription factors into induced pluripotent stem cellsWhile this low-dimensionality in gene expression seems to be a common and prevalent observation, we show here that there are more than three to four biologically relevant linear dimensions in large heterogeneous gene expression datasets. Furthermore, we explain in which cases PCA fails to detect relevant embedded information and suggest methods that could complement PCA for a more comprehensive detection of underlying genetic or non-genetic variability in large-scale gene expression compendia.et al.et al.We first reproduced the results from Lukk et al.We assembled an additional larger dataset consisting of 7100 samples from the Affymetrix Human U133 Plus 2.0 microarray platform in order to compare the reported results to those obtained with a different dataset. Applying PCA to this dataset replicates the separation of hematopoietic cells, cell lines, and neural tissues from all other cells on the first three PCs  as reporet al., we employed a linear regression approach comparing each of the first three PCs of our PCA model using the first three PCs derived from the Lukk dataset. This results in surprisingly high R-squared values of 0.74, 0.66, and 0.70, indicating a strong concordance of our first three PCs with those reported by Lukk et al. In contrast, trying to match the fourth PC of the own dataset, which captures the liver dimension specific to our dataset, by the first 5 PCs of the Lukk dataset results in a non-significant R-squared value of 0.07. This measure can be increased to a still comparably low value of 0.50 when the first 10 PCs of the Lukk dataset are combined as regressors. Thus, the liver-specific component in our own dataset is incompletely represented in the first 10 PCs of the Lukk dataset.Thus, the first three PCs derived by Lukk and our own dataset seem to span similar spaces by separating samples mainly in hematopoietic, brain, and cell line related groups, although with rotated coordinate axes, while further components differ more strongly between the two datasets. In order to quantify the similarities and differences between our dataset specific PCA model and the model published by Lukk et al.Next, we set out to explore how the observed differences in the PCAs of the two datasets could be explained and assumed that different sample numbers for the various cell types in the respective datasets could have a major effect. The Lukk et al, revealing almost identical patterns of sample separation in PCs 1–3  to a specifically selected subset of 1394 samples in order to obtain a similar sample distribution as in the Lukk dataset in terms of the proportion of samples from the hematopoietic system, brain, cell lines, muscle, and incompletely differentiated cells . We then PCs 1–3 .As a second computational experiment, we went back to the full dataset of 7100 samples and varied only the number of liver (cancer) samples, exploring corresponding changes in the liver-specific PC 4. This analysis reveals that the direction of PC 4 changes significantly with increasing number of liver (cancer) samples . When thIn both, the Lukk and our own dataset, the first three PCs explain roughly 36% of the variability. The remaining 64% contain partly noise, but we were interested if there is also important biological information contained in these components.In order to investigate the information content in higher PCs, we decomposed the original dataset into two derived datasets with the same number of genes and samples as the original dataset. The “projected” dataset captures the information in the first three PCs, while the “residual” dataset represents the information in the fourth and all higher PCs.In a first analysis step, we compare the correlation between the genome-wide expression in different tissues in the original datasets  to that In order to formalize the measurement of tissue-specific information content in the residual space, we utilize a previously developed information ratio criterionWe could already show that PCA depends strongly on the specific choice of samples in the dataset, which is why we set out to investigate whether we can detect biologically relevant information in the residual space by applying PCA on a subset of the Lukk dataset. Therefore, we used two different subsets based on the biological annotation of the dataset: A “brain subset” consisting of all brain samples in the Lukk dataset, which we further subdivide according to the brain region into samples from caudate nucleus, hypothalamus, frontal cortex, and cerebellum and a “cancer subset” consisting of all colorectal cancer, ovarian cancer, and hepatocellular carcinoma samples in the Lukk dataset.et al.  and projected these residual subsets from the own dataset on the “residual subset PCs” detected based on the corresponding subset from the Lukk dataset before. This reveals a distinct separation of the three cancer subtypes as well as the cerebellum samples, while the hypothalamus samples showed a less clear distinction from cerebral cortex than before  insets. Having shown that there can be important information that is not properly found by performing a standard PCA on the complete dataset, we investigated more theoretically see  in whichOne case that was already exemplified above is due to sample size effects, e.g. when the number of samples from a specific tissue is small compared to the overall number of samples see . This raThe idea here is to project the data onto predefined directionsThe second case in which PCA fails to detect biological signals is due to a small effect size, meaning that the respective phenotype is associated with small overall differences in gene expression. In many cases, these phenotypic differences with a low overall effect size are not very stable, i.e. they are not well reproducible in independent studiesThis statement can be explained by the observation that differential expression is typically not restricted to a few genes onlyXIST gene, which lies on the X-Chromosome and is responsible for X-chromosome inactivation in females, are differentially expressed . Therefore, information on sex is often not contained in the first few PCs (at least in heterogeneous datasets), although it is very well reproducible. In xpressed .In order to check in which dimension the information about sex differences is contained, we calculated the IRet al. 2006), we can detect two genes on the y chromosome, RPS4Y1 and DDX3Y, that have the lowest and third lowest kurtosis of all genes, respectively and that are able to separate male from female samples.Due to the small effect size, it is typically not possible to directly separate cells with genetic differences based on a PCA on gene expression data. Therefore, it is necessary to focus on different approaches. A relatively simple approach that can identify single genes that separate two groups is based on bimodality measures . Using tHowever, distinguishing between male and female cells is a rather straightforward task. It becomes more complicated when the genetic differences are more subtle or if they are present in a small subset of samples only. For these cases, more advanced approaches need to be used, such as the recently proposed “functional genomic mRNA profiling” (FGM) methodPrincipal components analysis is frequently used in the analysis of whole-genome gene expression data, either for investigation of the overall structure of a dataset or for correction of confounding effects. Here, we focused on the former approach and investigated in which cases PCA fails to properly identify relevant biological information in the leading principle components. We distinguished between two different cases that are either due to a small effect size of the signal or due to a small number of samples containing the signal and gave examples of their practical relevance. It is important to note that the susceptibility of PCA to these two effects depends on the dataset under consideration. Very heterogeneous datasets with many different cell types or tissues, such as those analyzed by us, are strongly affected by these two effects, while more homogeneous datasets are generally less affected.From a biological perspective, it seems to be helpful to distinguish between genetic and non-genetic biological signals. Purely genetic signals, e.g. copy number variations or sex differences, usually have a small effect size due to a limited number of affected genes and are therefore not represented in the leading PCset al.Based on our analyses, we cannot conclude on the actual linear intrinsic dimensionality of gene expression spaces, but we have shown that the linear dimensionality is definitely higher than previously suggestedEstimation of intrinsic dimensionality is a complex task. The approaches discussed so far focus on linear dimension reduction. Thus, any nonlinearity in the actual intrinsic dimensions can lead to erroneous results and require appropriate nonlinear methodsIn this respect it is also important to recognize that different cell or tissue types tend to cluster separately. Therefore, it is important to distinguish between the local (within cluster) and global (across cluster) dimensionality. The latter is especially problematic when considering nonlinear methods, since completely separate clusters can in principle be connected via a single nonlinear line. Therefore, clustered data make the notion of intrinsic dimensionality complicated on a global scale.Another potential issue is that the analyzed datasets might not cover the full range of all possible gene expression patterns, e.g. due to sample selection or generally understudied tissues or cell types. This can also directly affect any method for estimation of intrinsic dimensionality.From a more biological perspective, it is important to note that genes are connected to each other in a gene regulatory network, such that changes in the expression of one gene lead to changes in the expression of other genes. This restricts the set of states a network can potentially attain. Furthermore, many network states would not result in a viable cell, leading to additional restrictions. Therefore, it is biologically reasonable to assume that the intrinsic dimensionality of gene expression spaces is substantially lower than the number of genes in the dataset would allow. In fact, it has been observed that different perturbations of the gene regulatory network eventually converge to the same network stateIn conclusion, we have refined the current understanding of the global structure of gene expression spaces, showed that the linear intrinsic dimensionality of gene expression spaces is higher than previously reported and explained the results of these reports by sample-size effects, limiting the interpretability of PCA results in terms of intrinsic dimensionality estimates. We hypothesize a structural difference between genetic and non-genetic biological signals and point to existing methods for both kinds of signals that overcome the described limitations of PCA.http://www.combine.rwth-aachen.de/index.php/resources.html.A graphical representation of the performed analyses, depicting interrelations, is provided in http://www.ebi.ac.uk/arrayexpress/, accession number E-MTAB-62). The own dataset consisting of 7100 samples from the Affymetrix Human U133Plus 2.0 platform was compiled based on 108 public datasets from the Gene expression omnibus (GEO) database  using the robust multi-array average (RMA) normalization method. The preprocessed dataset can be downloaded from http://www.combine.rwth-aachen.de/index.php/resources.html. The sample annotation was performed manually based on the description in the GEO database. Cancer cell lines and tissues were classified according to their primary tissue, leading to a less detailed distinction as in the Lukk dataset. This explains the reduced number of groups (192 instead of 369) in the own dataset compared to the Lukk dataset. In the latter, different cell lines and histologically different cancer tissues from the same primary site are classified into separate groups. The own dataset contains 213 in vitro (trans-) differentiated or teratoma samples which were not associated with any of the 192 groups.The Lukk datasetprcomp function of the stats package. Subsequently, we compared the first three PCs of both datasets in order to determine whether they span similar spaces. For this analysis, we mapped the probes of the two platforms using the getBM function of the biomaRt package in R and performed linear regression analysis (R function lm of package stats) to explain the expression pattern of PCs 1 to 4 from the own dataset by the first three  PCs of the Lukk dataset. The resulting R2 values were reported as similarity measure of the two spaces.Principal components analysis of the Lukk and the own dataset were calculated in R version 3.1.2 using the The selection of a subset of samples from the own dataset with similar sample distribution as the Lukk dataset was performed in the following way. We used all 482 hematopoietic samples in the own dataset and randomly selected 74 brain, 163 cell line, 40 incompletely differentiated, 48 muscle, and 587 other samples. These numbers were chosen to match the proportion of samples in each of these large-scale groups to the respective proportion in the Lukk dataset.The investigation of the effect of reduced numbers of liver or liver cancer samples in the own dataset was performed on the complete dataset with only the number of liver (cancer) samples reduced. Thus, we used all 6825 non-liver samples together with the specified number of liver (cancer) samples and performed a PCA on the dataset. PCs 1–3 did not change significantly with increasing number of liver samples (data not shown). Therefore, we focused on the differences in the liver-specific PC 4.The Pearson correlation of gene expression patterns between the 369 groups of the Lukk dataset as well as the 192 groups of the own dataset  was calcWithin-group correlation was calculated in the same way between individual samples within one specific group. These calculations were performed for all groups that contain at least 10 samples. Depicted are the mean correlation values for each group .The information ratio is described in detail inFor the analysis of sample subsets we also used the decomposition into the projected and residual data matrices. We then concentrated on the residual data matrix and performed a PCA on the respective subset of data, i.e. the cancer subset or the brain subset. PCA successively identifies the direction of largest variability in the space orthogonal to the already identified directions. Therefore, it would not make sense to perform PCA on the complete residual matrix, since this would exactly identify the fourth, fifth, and all further PCs from the original dataset. However, performing PCA on a subset of data can reveal different directions. In our case, the first two PCs of this subset analyses had a clear biological meaning  as opposbiomaRt package in R. Afterwards, the mean value for each gene (from our own dataset) was subtracted and the data were orthogonally projected onto the three PCs of the Lukk dataset using scalar products between the loading vector of each PC and the gene expression vectors. In a second step, we concentrated on subsets of the own dataset that correspond to the two subsets from the Lukk dataset, i.e. consisting of colorectal, liver, and ovarian cancer samples, as well as hypothalamus, cerebral cortex, and cerebellum samples. For both of these subsets the residual expression matrix was determined by subtracting the information that is contained in the three PCs of the Lukk dataset. Afterwards, the residual expression vectors were projected onto the respective first two “residual subset PCs” that were identified based on the subset analysis of the Lukk dataset  were usedet al.Color-coding in in vitro differentiation for 5 or 14 days .


Missing values are commonly present in microarray data profiles. Instead of discarding genes or samples with incomplete expression level, missing values need to be properly imputed for accurate data analysis. The imputation methods can be roughly categorized as expression level-based and domain knowledge-based. The first type of methods only rely on expression data without the help of external data sources, while the second type incorporates available domain knowledge into expression data to improve imputation accuracy.In recent years, microRNA (miRNA) microarray has been largely developed and used for identifying miRNA biomarkers in complex human disease studies. Similar to mRNA profiles, miRNA expression profiles with missing values can be treated with the existing imputation methods. However, the domain knowledge-based methods are hard to be applied due to the lack of direct functional annotation for miRNAs. With the rapid accumulation of miRNA microarray data, it is increasingly needed to develop domain knowledge-based imputation algorithms specific to miRNA expression profiles to improve the quality of miRNA data analysis.We connect miRNAs with domain knowledge of Gene Ontology (GO) via their target genes, and define miRNA functional similarity based on the semantic similarity of GO terms in GO graphs. A new measure combining miRNA functional similarity and expression similarity is used in the imputation of missing values. The new measure is tested on two miRNA microarray datasets from breast cancer research and achieves improved performance compared with the expression-based method on both datasets.The experimental results demonstrate that the biological domain knowledge can benefit the estimation of missing values in miRNA profiles as well as mRNA profiles. Especially, functional similarity defined by GO terms annotated for the target genes of miRNAs can be useful complementary information for the expression-based method to improve the imputation accuracy of miRNA array data. Our method and data are available to the public upon request. Missing values are commonly present in microarray data due to various reasons, such as the limitation on detection sensitivity, contamination or error induced in experimental operations, and inappropriate data preprocessing. For the sake of accurate analysis on expression profiles, the estimation of missing values has been a basic and key issue in microarray data analysis.Up to now, a variety of methods have been proposed for imputing missing values in gene expression profiles, which mainly fall into two categories. The first type is based solely on expression data, such as KNNImpute , LLSimpuK-nearest neighbor (KNN) is the most widely used algorithm for missing value imputation, and many other methods implement various modified versions of KNN process. For example, assume that gene g1 whose value on the first sample is missing, then the Euclidean distances of expression levels between g1 and other genes are measured as Eq. g1 and gj. Gi1, and Gj,i are the expression values of g1 and gj at the ith sample, respectively. {s1,⋯,sn} are the indexes of n samples where both g1 and gj have valid values. It should be noted that g1 and gj may have more than one missing values at different samples, and only the samples that both genes have valid values are considered. Thus the squared differences of expression values are averaged over n, i.e. the number of samples considered.where g1 and all other genes have been computed, K nearest genes are selected and a weighted average of their expression values on the first sample is assigned to G1,1.Once the distances between In many cases, gene expression level alone cannot guarantee good imputation result because of noise, bias or a large proportion of missing values. Actually, the microarray experimental results are not simply numerical matrices. Instead, these expression data should be treated with the consideration of biological background, such as functional similarity, biological pathways that genes involved and interactions between gene products. If the external annotation data sources can be effectively incorporated into the algorithms, the imputation quality would be improved.Common annotation data include Gene Ontology (GO) , KEGG paIn the past two decades, many efforts have been put on studying the semantic similarity of GO terms. Early methods refer to related studies in natural language –13, and p(x) is the probability of an attribute described by a GO term x, Gx is the set of genes that are associated with GO term x , and Groot is the set of genes that are associated with the root term . Here, we use ‘associated with’ instead of ‘annotated with’ because in GO DAGs, parent nodes are generalized concept of children nodes, thus a gene annotated with some terms is also associated with their parent nodes. Recent methods [http://bioinformatics.clemson.edu/G-SESAME/), which considers both common ancestors of two GO terms and their relative location in the GO graph.where  methods –17 consi methods   annotation data, and proposed PPI-KNNimpute and PPI-LLSimpute methods [Tuikkala et al.  investig methods . All of In recent years, miRNA microarrays have become more and more common because miRNAs play important roles in many biological processes and the development of complex diseases. Especially, the correlation of miRNA expression profiles with clinical-pathological characteristics of cancer patients has been largely investigated. MiRNA expression data is of great help for tumor classification and can be effective biomarkers for prognosis even more precisely than protein-coding genes , 20. TheConsidering that miRNAs play their roles mainly by regulating the expression of target genes at post-transcription level, the functional similarity of miRNAs can be inferred based on GO semantic similarities of their target genes . IdentifWe combine miRNA functional similarity with Euclidean distance of expression profile and develop a new imputation method. The new method is evaluated on two miRNA datasets from breast cancer research, on which the new method achieves improved performance compared with the expression-based method. The experimental results demonstrate that the functional knowledge retrieved from target genes of miRNAs can help to improve the estimation of missing values for miRNA expression profiles.In this study, we used two public miRNA data sets from NCBI , namely p % (maximum missing rate). Obviously, the bigger p is, the more missing values are present in the generated data sets. When p is equal to 20, it can be calculated that the expected value of missing rate of the whole simulated data set is 5 %. By changing the value of p, we can get simulated sets with different missing rates. The generation of data sets was repeated 100 times. Different imputation methods were tested and evaluated on all the simulated data set, and the accuracies were averaged to obtain the final result.In order to evaluate the performance of the new method, the original data sets were preprocessed by removing miRNAs with missing values. Then new data sets with different percentages of missing values were simulated from the preprocessed complete data. Considering that missing values are generally not evenly distributed over the whole profile, we did not generate the data sets by randomly selecting missing elements. Instead, we randomly selected 50 % of the miRNAs which were assumed to have incomplete profiles, and then randomly assigned missing elements in each of these miRNAs’ profiles. The missing rate was set to be a random integer percentage between 1 and NRMSE). Let Gest be the estimated data matrix by the imputation algorithm, and Gtrue be the matrix of the original complete data matrix. The NRMSE is defined as: There are multiple criteria for evaluating imputation algorithms. A commonly used one is the normalized root mean square error . GO-based KNN has better performance than conventional KNN no matter what the value of K is. The results also suggest that K has great impact on the accuracy of imputation. The two data sets have a similar pattern of varying accuracies, and the optimal value of K is 4 or 5, which is much smaller than that used in the imputation for mRNA expression profiles.Based on semantic similarity retrieved from GO graph and BMA (best match average) strategy for calculating similarity of two sets, we obtain the GO-based functional similarity matrix KNN and LLS . The maximum missing rate, i.e. p, is an upper bound of the percentage of missing values. For example, when p equals 40 %, the missing rate is randomly assigned between 1 and 40 %. In GSE40525 data set which has 120 samples, a miRNA which is selected to have missing values can have 48 missing values at most and one missing value at least with equal probability. Such generation of missing values is more practical than completely random model with regard to the real distribution in microarray data. Here K is set to be 5. Solid lines denote conventional methods, and dotted lines denotes GO-based methods. These two figures show that incorporating GO knowledge can improve the imputation accuracy for both KNN and LLS. The improvement in LLS is not as obvious as in KNN, since we only use GO-based measure to select neighbors in LLS, thus the GO information has less impact on the performance of LLS than that of KNN. Generally, the LLS method has much less NRMSE while takes more computation time compared with KNN.Figure NRMSE of miRNA data sets using conventional imputation methods is much less than that of mRNA data. The possible reasons are: a) miRNA profiles with hundreds of miRNA have much smaller scale than mRNA profile with tens of thousands of probes; b) the expression levels of the same miRNA vary slightly across samples. In addition, we observe that higher maximum missing rate does not necessarily results in bigger NRMSE. Similar cases are also present in mRNA expression data sets [Compared with previous studies which utilize GO information for the imputation of missing values on mRNA expression data, the improvement on miRNA data is not that significant. Actually, we have found that the ata sets . And theThis study focuses on effective incorporation of biological knowledge into the imputation of missing values in miRNA microarray data. Unlike protein-coding genes, the functional annotation of miRNAs cannot be utilized directly because no such database is available. Since miRNAs play their roles mainly by suppressing or silencing their target genes’ expression, two miRNAs may have strong correlation on their function and expession pattern if their targets are involved in the same biological process or have similar function. By transferring GO-based pairwise similarity of genes to the similarity of two target sets, the functional similarity of two miRNAs can be indirectly inferred.Although the experiments have demonstrated the performance improvement, it is obvious that biological knowledge can only be treated as a data source complementary to expression levels. On one hand, function similarity does not directly correlate with expression similarity . SimilarFirstly, the GO-based semantic similarity does not accurately reflect gene function similarity. Although we retrieve semantic information in the Molecular Function DAG, and the G-SESAME method has considered structural information, there is still a lot of information lost here. For example, the method only considers common ancestors but neglects common descendants. Moreover, even though we have a perfect method which comprehensively utilizes GO graph information, this semantic similarity is not necessarily corresponds to real functional similarity because the annotation is incomplete.Secondly, in order to cover as many miRNAs in our data set as possible, we adopt computational tools for target gene identification, which may have unreliable results although we have set a threshold to control false positives and false negatives.In addition, some process steps within the whole workflow, which have great impact on the final results, need further studies. The first issue is the simulation strategy for generating the evaluation set with artificial missing values. Although the partial-random model used in this study seems more reasonable than complete random model, how to simulate missing values according to their real distribution in microarray profiles needs careful studies. The second issue is the integration strategy from pair-wise GO similarity to miRNA similarity. Here a two-step best match average method is conducted, while other strategies may be examined. For example, each miRNA can be directly regarded as a GO set with redundant GO terms from its targets. A pairwise similarity for such redundant set can be defined.As more and more miRNA functions have been revealed, available annotation for miRNAs increase rapidly, and other biological knowledge with regard to miRNAs can also be utilized. For example, Wang et al.  used micKNN and other imputation methods. The experiments have been conducted on two breast cancer miRNA expression profiles, and the normalized root mean square error (NRMSE) is compared under different values of K and maximum missing values. The GO-based new imputation method shows decreased NRMSE on both data sets.Biological domain knowledge has been demonstrated to be very helpful to improve the imputation of missing values in DNA microarray data profiles. However, because of the lack of function annotation for miRNAs, the domain knowledge-based imputation methods have rarely been studied for miRNA expression data. In this paper, we propose to use the pairwise functional similarity of miRNAs based on the semantic similarity of target genes in GO graph as a complementary information for expression data and define a new similarity measure. This new measure can work together with conventional Our method is applicable to not only human miRNAs but also other species as long as the target genes can be obtained. With the rapid accumulation of miRNA microarray data, the studies on domain knowledge-based function similarity for miRNAs are of great need to improve expression data quality and eventually benefit for the identification of novel miRNA biomarkers and novel potential functions of miRNAs.KNN  with the new similarity measure. The whole workflow can be summarized in the following steps: Identify target genes of miRNAs via computational prediction tools or public databasesSearch GO terms annotated for each target gene obtained in Step 1Compute similarity for each pair of GO terms obtained in Step 2Compute similarity for each pair of target genes from the similarity of their GO termsCompute similarity of each pair of miRNAs from the similairty of their target genesDefine new similarity measure by combining GO-based similarity and expression level-based distanceImpute missing values with the new measureAs shown in Fig. g1 and g2 from their GO annotation, we firstly retrieve GO term list of each target gene for miRNAs. Two genes correspond to two GO sets, i.e., In order to compute the functional similarity of two genes GOSemSim , and comk and s terms respectively, and similarity of each pair of GO terms, there are four basic methods to obtain gene similarity, i.e. the similarity of these two sets, namely max, avg, rcmax and BMA [t1,t2), where max strategy fails to consider the overall similarity of two sets. For example, it regards the similarity of two sets as 1 even if they have only one term in common. As for the ave strategy, even though the two sets are exactly the same, the similarity between two sets is not 1. The rcmax method works like following: ti and the terms in For each term k maximum values obtained for all terms in max1Get average value of the tj and the terms in For each term s maximum values obtained for all terms in max2Get average value of the max to the gene similarityAssign Given two sets of GO terms  and BMA . The forrcmax is often biased when the sizes of GO sets differ considerably. The last method BMA (best match average) has the same steps 1) and 3) of rcmax. The difference is that it calculates the overall average value of k maximum values of terms in s maximum values of terms in BMA has been used in many studies [BMA to compute gene similarity based on GO pairwise similarity. Since each miRNA corresponds to a set of target genes, miRNA similarity can be computed by gene similarity in the same manner. studies , 21. HerIn this study, we use computational prediction tools to identify targets of miRNAs, because the experimental-supported target databases, like TarBase, only house a small portion of miRNAs in the data sets. We have found that miRanda provides prediction results for the most miRNAs in our date sets than other tools. In order to avoid too many false positives, we set a threshold of mirSVR score ≤−1 to ensure the prediction accuracy. It should be noted that the naming of some miRNAs is inconsistent between databases and miRNA array. We refer to miRBase  to deal H and Euclidean distance matrix D, each element of the new distance matrix C is defined as Eq. In our method, we combine the GO-based functional similarity with the Euclidean distance obtained from expression profiles as a new measure. Here we adopt a similar strategy used in Tuikkala’s work . Given ti,j=1,⋯,n, and n is the number of miRNAs in the data set. α is a positive parameter of weight, denoting the impact of GO-based similarity in the combined measure. If α=0, C turns out to be D. The bigger α is, the more impact that GO information has on the selection of nearest neighbor and performance of imputation. The value of α is determined on an evaluation set extracted from the original complete data set.where C, the following steps are the same as conventional expression-based imputation algorithm. Assume that we want to estimate miRNA gi’s missing value at the tth sample, and the nearest K miRNAs according to the new measure are KNN, Given the new distance matrix