With the recent advances in high-throughput RNA sequencing (RNA-Seq), biologists are able to measure transcription with unprecedented precision. One problem that can now be tackled is that of isoform quantification: here one tries to reconstruct the abundances of isoforms of a gene. We have developed a statistical solution for this problem, based on analyzing a set of RNA-Seq reads, and a practical implementation, available from archive.gersteinlab.org/proj/rnaseq/IQSeq, in a tool we call IQSeq (Isoform Quantification in next-generation Sequencing). Here, we present theoretical results which IQSeq is based on, and then use both simulated and real datasets to illustrate various applications of the tool. In order to measure the accuracy of an isoform-quantification result, one would try to estimate the average variance of the estimated isoform abundances for each gene (based on resampling the RNA-seq reads), and IQSeq has a particularly fast algorithm (based on the Fisher Information Matrix) for calculating this, achieving a speedup of  The concepts of genes and isoforms have evolved and become more complex With the recent development of high-throughput RNA sequencing (RNA-Seq) technology, it is possible for biologists to measure transcription with unprecedented precision. One problem that can now be tackled is that of isoform quantification, where one tries to reconstruct the abundances of similar isoforms based on a set of RNA-Seq reads. Various methods have been developed to solve this problem. In previous work, researchers proposed different statistical frameworks to solve this problem. Xing et al. These studies, however, have not fully addressed the problem of isoform quantification in a couple of respects: First of all, they usually assume that only one sequencing technique is used in an experiment, and that the reads are uniformly sampled along the transcripts. These are not necessarily good approximations to real data. Second, while some theoretical results have been presented on estimating the accuracy (e.g. average variance) of quantification results, there does not yet exist a method to efficiently compute these measurements other than using brute-force simulation, which is computationally infeasible in large scale expriments involving tens of thousands of genes and millions of sequencing reads. On the other hand, fast estimation of quantification accuracy would not only enable researchers to better understand the analysis results being obtained, but also will be useful in RNA-Seq experiment design to optimally integrate different sequencing technologies in a cost-efficient way.In order to fill in these gaps, we have developed a generalized statistical solution for the problem of isoform quantification, and a practical implementation in a tool we call IQSeq (Isoform Quantification in next-generation SEQuencing). IQSeq has the following features which represent improvements over previous work in isoform quantification in the following aspects:It has a generalized statistical read generation function during the sequencing process (i.e. a customizable function describing how reads are randomly sampled from isoforms). This provide a flexible way to incorporate characteristics of different sequencing technologies (e.g. 3′ end sequencing bias of transcripts).It integrates the analysis of different sample sets generated from different sampling technologies (e.g. long and short reads).It has a fast algorithm for estimating the average variance of the results provided by our expectation maximization based solution.Given the estimated isoform abundance output, IQSeq also provides an information theoretical method to measure the overall transcriptome complexity.In this paper, we will first introduce a mathematical definition of the generalized partial sampling and distribution estimation problem (which IQSeq is based on), and provide a expectation maximization based iterative solution. Then we discuss in detail on how to estimate the performance of this solution using Fisher information based heuristics, and present fast algorithms that implement the computation of these heuristics. Finally, we show results of applying our methods to both simulated and real-world data, illustrating scenarios where such integrated analysis can be the most informative.First, we formally define the isoform quantification with multiple sequencing technologies as a generalized statistical partial sampling problem, and present a computational solution based on maximum likelihood estimation and expectation maximization. We then show both analytical results and practical fast algorithms to estimate the average variance of the solution on isoform quantification, and compare their computational complexity against brute-force methods. We present the main theoretical results in this section, and detailed derivations can be found in We start by defining the generalized process of batch partial sampling, which represents the sequencing process in RNA-Seq experiments, and the relationships between partial samples and the objects being sampled. Let Here we assume a two-step sampling process: First, a sampling method Given As shown in In the remaining part of this paper, we will use two notations to describe a partial sample Definition 2 does not give an explicit criterion for a “good” estimation of  Given By plugging in the partial samples In the next subsection, we demonstrate how this problem can be solved by introducing a hidden variable We define By performing an E step that computesThe iterative estimation in Equation 7 is intuitively consistent with the case of estimating a distribution based on full samples: consider the scenario in which for each Given As we will show later, although brute-force simulation can be performed to obtain a relatively accurate estimation of this measurement, it is may become computationally intractable when there are too many reads and genes to be considered. We thus propose to use a Fisher information based heuristic for estimating We first introduce the Fisher information matrix Let (Observed Fisher information matrix).(Expected Fisher information matrix).Let We then estimate This means that we only need In order to provide a single value measure for the expected performance of Maximum Likelihood estimation, we propose to use the following heuristic to estimate the average variance of This heuristic avoids the potential computational intensive and numerically unstable computation of the inverse of In order to compute this heuristic, all we need is First of all, we can decompose A brute-force algorithm for computing Ik) computations of In Algorithm 4, if In order to continue our discussion on faster algorithms to compute Two partial samples If , , then  and  are equivalent w.r.t. .A set of partial samples Given an isoform  and a sampling method , if we divide all its possible partial samples into  non-overlapping equivalent sample setsthen:In this subsection, we consider a simplified partial sample generation model:A simple shotgun sampling method Given the sample generation model  above, if two samples  and  generated by this method are compatible with the same set of isoforms, i.e. , then  and  are equivalent w.r.t. .Given the sample generation model  above, if two samples  and  generated by this method overlap with all the junctions in the same set of connected exons , then  and  are equivalent w.r.t. .For example, in Based on Definition 8 and Theorem 1, we can design the following algorithm for efficiently computing In Algorithm 2, We can further reduce the number of times of computing In Algorithm 3, Given a set of Suppose that on average an isoform can be divided into The sequencing technology being used in an RNA-Seq experiment is usually more complicated than the simplified When the assumption of uniform sample generation still holds, it is straightforward to handle samples with different lengths in FIM computation. We can treat one sampling method as a combination of multiple simplified methods as in Definition 8, with different sample lengths In the case of non-uniform sample generation along the isoform, if Here we present our results on a set of simulated datasets. In order to demonstrate the accuracy and efficiency of the methods we developed, we first use simulations to show the performance of our approach on simplified gene models and a real gene. These simulations are useful in designing optimal sequencing experiments for isoform quantification.Due to the complexity of real gene structure, we apply our methods to three artificially constructed genes with simplified isoform structures, so as to better illustrate how different characteristics of the gene structures can affect the outcome of the isoform quantification analysis.As shown in Not only can the FIM based heuristic correctly approximate how the performance of MLE changes with regard to different sampling technique combinations, it is also able to dramatically shorten the time of computation, as shown in We have developed a Fisher information matrix (FIM) based fast algorithm  and late embryo (LE), to briefly illustrate the fact that different genes have different isoform composition differences between stages. Here we use the following formula to measure the difference in isoform composition of the same gene in two different stages:Diff values. The reads are plotted below the isoforms, and the numbers associated with the isoforms are their estimated relative abundances based on MLE. Furthermore, if we compute such values for all the genes in these two stages, we can get a histogram of isoform composition differences as illustrated in Diff score was calculated for each gene in all 6 pairwise comparisons. Because isoform quantification is noisy for genes expressed at very low level, we plotted the distribution for genes that have at least an RPKM value of Diff score range, while error bars indicate the maximum and minimum numbers. Diff scores close to Diff score.) In http://archive.gersteinlab.org/proj/rnaseq/IQSeq) the genes with stage-wise isoform composition differences, ranked by their FIM based estimation variances, and with a thresholded on Diff score and RPKM at We also investigate how different isoform sets  will affect the MLE result, especially in terms of the maximized likelihood value. We pick gene No. As we can see from To illustrate how non-uniform Also, there have been some recent works In order to understand the performance of our method compared with other existing tools, we have conducted additional computational analysis by applying IQSeq and Cufflinks on n assay  .In this paper we explore the problem of integrating different sequencing techniques to quantify the relative abundance of different isoform transcripts, which can be generalized to the problem of estimating the distribution based on partial samples from different sampling techniques. We first introduce a statistical framework to model the generative process of the partial samples, using a “plugin-able” function In order to further investigate the problem involving partial samples, we introduce a heuristic based on the Fisher information matrix (FIM) to estimate the variance of the previously presented MLE solution. Also, in order to accelerate the computation of this measurement, we introduce the concept of equivalent partial samples and develop a fast algorithm, Algorithm 3, to accurately calculate FIM, achieving a speedup of What is more, by applying the MLE method to a worm RNA-Seq dataset, we illustrate how we can compare the differential isoform composition between different developmental stages, and how different isoform sets  will affect the MLE result, especially in terms of the maximized likelihood value, showing that the likelihood score is an effective tool for ranking the “fitness” of isoform sets for a particular gene.Since IQSeq estimates isoform quantity within a probabilistic framework, it does not directly determine the existence of a certain isoform transcript in the data, but rather gives probability measures Click here for additional data file.


We report on results from whole-exome sequencing (WES) of 1,039 subjects diagnosed with autism spectrum disorders (ASD) and 870 controls selected from the NIMH repository to be of similar ancestry to cases. The WES data came from two centers using different methods to produce sequence and to call variants from it. Therefore, an initial goal was to ensure the distribution of rare variation was similar for data from different centers. This proved straightforward by filtering called variants by fraction of missing data, read depth, and balance of alternative to reference reads. Results were evaluated using seven samples sequenced at both centers and by results from the association study. Next we addressed how the data and/or results from the centers should be combined. Gene-based analyses of association was an obvious choice, but should statistics for association be combined across centers  or should data be combined and then analyzed ? Because of the nature of many gene-based tests, we showed by theory and simulations that mega-analysis has better power than meta-analysis. Finally, before analyzing the data for association, we explored the impact of population structure on rare variant analysis in these data. Like other recent studies, we found evidence that population structure can confound case-control studies by the clustering of rare variants in ancestry space; yet, unlike some recent studies, for these data we found that principal component-based analyses were sufficient to control for ancestry and produce test statistics with appropriate distributions. After using a variety of gene-based tests and both meta- and mega-analysis, we found no new risk genes for ASD in this sample. Our results suggest that standard gene-based tests will require much larger samples of cases and controls before being effective for gene discovery, even for a disorder like ASD. This study evaluates association of rare variants and autism spectrum disorders (ASD) in case and control samples sequenced by two centers. Before doing association analyses, we studied how to combine information across studies. We first harmonized the whole-exome sequence (WES) data, across centers, in terms of the distribution of rare variation. Key features included filtering called variants by fraction of missing data, read depth, and balance of alternative to reference reads. After filtering, the vast majority of variants calls from seven samples sequenced at both centers matched. We also evaluated whether one should combine summary statistics from data from each center  or combine data and analyze it together . For many gene-based tests, we showed that mega-analysis yields more power. After quality control of data from 1,039 ASD cases and 870 controls and a range of analyses, no gene showed exome-wide evidence of significant association. Our results comport with recent results demonstrating that hundreds of genes affect risk for ASD; they suggest that rare risk variants are scattered across these many genes, and thus larger samples will be required to identify those genes. Common and rare variants are important constituents of the genetic architecture of Autism Spectrum Disorders (ASD) For rare variants, even if effects are strong, single variant tests typically have little power. Rare variants have to be combined in some way, such as within a gene or across genes, for an association test to reach sufficient power. Hence statistical tests examine the cumulative effects over the observed rare variants in the target set. A number of statistical methods to test for association with rare variants are now available. Several of these tests fall into the category of burden tests in that they assess association with a “super-variant” Power of the test is determined by the number of causal variants in the gene, the size of the corresponding effects, and the sample size. Assuming that the rarest variants are likely to have the largest effects, it is challenging to amass substantial evidence for association without a large sample size. Based on extrapolation of effect sizes and frequencies from published studies In this ARRA autism sequencing consortium (AASC) study, data have been produced by two sequencing centers (Baylor College of Medicine and Broad Institute) and by different exome capture methods, different sequencing platforms and different pre-processing alignment and variant calling methods. Therefore the coverage and quality of these data sets varies. Nonetheless, as we show in the sequel, these data can be harmonized using standard filtering criteria. Given the distinct data sources, the most effective way of testing for association is unclear. Following in the tradition of association studies, meta-analysis is a natural option known ASD genes. We conclude that rare variants affecting risk are not clustering in a small number of genes, supporting recent results from de novo single nucleotide and copy number studies showing that hundreds of genes in the genome affect risk for ASD After quality control and controlling for ancestry, analysis of AASC data reveals no clear-cut associations, including associations in The AASC whole-exome sequencing data included 1039 ASD subjects of European ancestry and 870 controls of similar ancestry. Approximately half of the samples were sequenced using the Solid platform and called with AtlasSNP 2 We considered 6 filters to make these data sets more similar in terms of the distribution of variants in the exome. Filters were sequential in their stringency for including a variant: Filter PASS included variants that pass the baseline filter of GATK; Filter MISS excluded any variant with more than 10% missingness; Four additional filters placed increasingly stringent requirements on depth and balance of reference and alternative allele calls see . If not Seven control samples were sequenced by both centers, facilitating an independent comparison of cross platform calls and an evaluation of the filtering process. To do so, we identified all rare ; however, there are only three ways that 2 alleles can be partitioned between cases and controls. Thus with a larger sample, it is possible for rare alleles to obtain more unusual configurations. As variants become extremely rare the situation becomes more unfavorable to meta-analysis. Unless the sample is very large, most samples will draw only one copy of the rare allele and in this scenario neither of the two case-control configurations is unusual. With singleton variants SKAT can only gain information about association if the rare variants are grouped to form a super-allele.Alternatively, mega-analysis also has advantages when considering rare alleles with no effect. If, for a particular variant, we observe half of the rare alleles in cases and half in controls in the combined sample, but all of the alleles are in cases in the first sample and all are in controls in the second sample, then the evidence for association is appropriately diminished by considering the full sample simultaneously  over exonic CVs. From These findings suggest that the distribution of common variants might function as a proxy for the distribution of rare variants. Next we look to see if these descriptive analyses support the use of an eigen-map to control for confounding in rare variant tests due to ancestry. To test for association between ASD and rare variants in the AASC sample, we apply burden tests and SKAT to the filtered version of the data sets and obtain the p-values of genes in the Baylor, Broad and combined datasets. We investigate the effects of population structure by calculating the genomic control inflation factor Before comparing choices of eigenvectors, we investigate the behavior of the genomic control statistic, quantile . A similquantile . This isThese insights into the null distribution of the rare variant test statistics lead us to calculate ture see . To compatistics . Notice Next we examine the effect of adjusting for ancestry (using CVs) on the rare variant test statistics. Notice that while vidually ; in the We explore this further by contrasting the results obtained when applying no correction versus correction based on eigen-vectors derived from CVs, LFVs and CVs+LFVs and find that the corrected results are nearly indistinguishable regardless of the scenario  for 3 filters ranging from lenient (Filter PASS) to stringent  that has been demonstrated to be an ASD gene based on 3 recurrent de novo loss of function mutations While filtering is beneficial to remove false positives, it has the potential to remove real signals as well. We explore the effect of filtering on a particular gene  the gene has 15 variants, each with MAFAs with GWAS, to assimilate large samples and gain power, multiple studies must be combined. In the analysis of samples from multiple studies, meta-analysis, based on Z-scores, has become the norm for most genetic investigations. This form of meta-analysis has power equal to mega-analysis for single variant tests A drawback of mega-analysis is that we encounter challenges when combining datasets collected across multiple studies, which can differ in many respects due to the use of different sequencing platforms and protocols. For instance, these differences lead to differential coverage by exon and different alignment errors. Even the best laboratory process has measurement error and these errors are exacerbated when they differ across batches of samples, particularly if they differ between cases and controls. For these reasons caution must be exercised if one is to reap the benefits of mega-analysis. Indeed, even after careful filtering, heterogeneity between sites could account for the modest inflation in the associate test statistics and the genomic control factor after combining sites via meta- and mega-analysis.In this study we construct extra filters to ensure that the distribution of rare variation of the WES data is similar for the two centers. We find good results filtering called variants by fraction of missing data, read depth, and balance of alternative to reference reads. Ideally a filter is tuned by measuring some individuals on multiple platforms. We tune our filters using subjects measured twice. If such data are unavailable, however, we find that another promising approach is to compare minor allele counts (based on rare variants) per gene. A good filter is one that aims to equilibrate these quantities.Even with the most minimal filtering we observe no excess of positive signals for association within the individual data sets, but for mega-analysis we observe a great number of positive associations. These false discoveries are diminished, however, after filtering. Likewise mega-analysis is more susceptible than meta-analysis to the impact of differential missingness across platforms and across case/control status. Indeed, without filtering, mega-analysis has many false discoveries but meta-analysis did not. However, using filtered data we find that mega-analysis is quite robust to differences in missingness rates across platforms and case/control status, although we recognize that this robustness could fail for more extreme heterogeneity of missingness. Still our study has some differences in missingness and yet does not produce detectable false discoveries. From our analyses we conjecture that filtering that removes variants with When combining data sets the effects of population substructure on association is also a concern due to clustering of rare variants in ancestry space In conclusion we find that WES data on nearly 2000 samples collected for a case-control study are insufficient to discover novel liability genes for ASD, even after applying efficient methods like mega-analysis and controlling for ancestry effectively. These results demonstrate that much larger samples will be required for effective gene discovery and lend further support to the prediction that there are hundreds of genes that impact ASD liability in the human genome.www.nimhgenetics.org) to be of similar ancestry to cases  The Broad cases included probands only from trios. These trios were previously analyzed for de novo variants De novo variants were included in these analysis.The AASC whole-exome sequencing data includes 1039 independent subjects diagnosed with autism spectrum disorders (ASD). Subjects were selected to be of European ancestry, based on genetic (eigen-vector) analysis and European origin. Samples were selected from the Autism Genetic Resource Exchange , the Autism Simplex Collection TASC , NationaTo evaluate sequence quality, 7 controls were sequenced at both centers. The capture/enrichment assays used were Nimblegen (Baylor) and Agilent (Broad). The Baylor samples were sequenced using the Solid platform and called with AtlasSNP 2 de novo variants In general, the MAF of SNVs matched well for the majority of the SNVs in the two data sets, but some differed considerably . One souTwo rounds of filtering were performed on called indels. First, for each data set, we excluded indels with MAF greater than For We want to test the null hypothesis in model  are replThere are drawbacks to a burden test. It assumes that all rare variants in the gene have the same direction and magnitude of association. In reality, variants can be damaging, protective, or have no effect, potentially reducing the power of the test. To overcome these drawbacks, the C-alpha test Suppose we have samples from two (or more) datasets. To fix ideas, consider two data sets, We formally consider the SKAT test statistics in meta- and mega-analysis by deriving a closed form expression for the power of meta- and mega-analysis under restricted conditions. In the Results we show via simulations that the results hold more generally. Analysis is greatly simplified by choosing weights ar model , and(5)fic model .It follows that the mega-SKAT statistic In a comparison of the power of meta- and mega-analysis we assume data sets To gain more analytical insight, consider a gene for which each sample has sufficient coverage to detect all rare variants and that a total of In the Figure S1PCA from common variants, low frequency variants and both type of variants for Baylor samples. Eigen-vectors are obtained by applying PCA to all common variants that have no missingness  (A), all low frequency variants that have no missingness (8783 variants) (B), and both type of variants (C). The colors are obtained by clustering individuals based on their coordinates in panel (A) using model based clustering (TIF)Click here for additional data file.Figure S2PCA of Baylor and Broad samples together. first eigen-vector versus second eigen-vector for Broad and Baylor samples.(TIF)Click here for additional data file.Figure S3Histogram of p-values for SKAT and Burden Test. (A) and (B) are SKAT p-values for Broad and Baylor samples, respectively. (C) and (D) are Burden test p-values for Broad and Baylor samples, respectively. Green vertical lines are the 25%, 50% and 75% quantiles of p-values.(TIF)Click here for additional data file.Figure S4Distribution of the genomic control factor (TIF)Click here for additional data file.Figure S5P-values versus Missingness. We used 5500 genes to make this plot. For each gene, we calculate the -log 10 p-values and the odds ratio of missingness in case and control. The red line is the fitted line of these 5500 observations.(TIF)Click here for additional data file.Figure S6MAF Comparison: Baylor versus Broad. We compare the MAF for 72,758 shared non-synonymous variants in the two data sets.(TIF)Click here for additional data file.Figure S7Depth Comparison: Baylor versus Broad. We compare the average sample depth for all non-synonymous variants in the two data sets.(TIF)Click here for additional data file.Table S1Comparison of seven individuals called by both Baylor and Broad under different filters.(PDF)Click here for additional data file.Table S2Genomic control (PDF)Click here for additional data file.Table S3Genes with p-value(XLSX)Click here for additional data file.Table S4de novo nonsense or missense mutations as reported in The p-values of genes which have two or more (XLSX)Click here for additional data file.Table S5The p-values of 114 ASD genes.(XLSX)Click here for additional data file.Table S6The required sample sizes by applying meta- and mega-analysis.(PDF)Click here for additional data file.Table S7Classification tree results for heterozygote calls.(PDF)Click here for additional data file.Text S1Additional Information Regarding Methods. Part A gives additional information about sequencing, including data generation and quality control. Part B gives the mathematical exposition of mega- and meta-analysis. Part C provides details for association analysis.(PDF)Click here for additional data file.


Cancer outlier profile analysis (COPA) has proven to be an effective approach to analyzing cancer expression data, leading to the discovery of the TMPRSS2 and ETS family gene fusion events in prostate cancer. However, the original COPA algorithm did not identify down-regulated outliers, and the currently available R package implementing the method is similarly restricted to the analysis of over-expressed outliers. Here we present a modified outlier detection method, mCOPA, which contains refinements to the outlier-detection algorithm, identifies both over- and under-expressed outliers, is freely available, and can be applied to any expression dataset.We compare our method to other feature-selection approaches, and demonstrate that mCOPA frequently selects more-informative features than do differential expression or variance-based feature selection approaches, and is able to recover observed clinical subtypes more consistently. We demonstrate the application of mCOPA to prostate cancer expression data, and explore the use of outliers in clustering, pathway analysis, and the identification of tumour suppressors. We analyse the under-expressed outliers to identify known and novel prostate cancer tumour suppressor genes, validating these against data in Oncomine and the Cancer Gene Index. We also demonstrate how a combination of outlier analysis and pathway analysis can identify molecular mechanisms disrupted in individual tumours.We demonstrate that mCOPA offers advantages, compared to differential expression or variance, in selecting outlier features, and that the features so selected are better able to assign samples to clinically annotated subtypes. Further, we show that the biology explored by outlier analysis differs from that uncovered in differential expression or variance analysis. mCOPA is an important new tool for the exploration of cancer datasets and the discovery of new cancer subtypes, and can be combined with pathway and functional analysis approaches to discover mechanisms underpinning heterogeneity in cancers. Until recently, statistical analysis of expression-microarray data typically focused on the recognition of molecular subtypes and discovery of characteristic biomarkers; instances of within-subtype heterogeneity (outliers) were either removed from the analysis, or more rarely explored as a source of information on rare events. Cancer outlier profile analysis (COPA) was developed to identify transcripts up-regulated in only a small subset of cancer samples[Within a type of cancer, tumours are frequently heterogeneous at the molecular level. Some of this diversity may describe cancer subtypes, but even within a subtype, individual primary and metastatic lesions often differ from one another. Modern microarrays can measure the expression of >10The original outlier detection method implemented a data transformation in which the median of each expression feature across all samples is scaled to 0.0 and the mean absolute deviation to 1.0. Features are then ranked based on their value at the 75th, 90th or 95th percentiles. The COPA method has been integrated into Oncomine, but itsWithin our research program on cancer networks-6 we ideHere we present a modified COPA analysis program, mCOPA, which incorporates statistical refinements to outlier detection, including improvements in the calculation of percentiles,8 and thTo assess the performance of mCOPA as a feature-selection algorithm applicable to microarray datasets, we evaluated the utility of features selected by mCOPA in separating clinically defined cancer subtypes represented in expression microarray data. We evaluated the quality of features selected by mCOPA and by three other algorithms  analysis, and variance analysis) on 12 publicly available datasets. Quality was assessed based on the ability of feature sets to cluster samples into recognised subtypes. As it has been shown that clustering performance varies greatly depending on the dataset and the clustering method, we evalet al.[We also perform a detailed analysis of one of these 12 datasets , demonsmCOPA takes as input a matrix of preprocessed microarray data, with rows representing features and columns containing sample data. The first columns contain data from the normal samples, followed by the tumour samples. Example input and output files are included with the application, and a flowchart of the workflow is provided in the user manual. The COPA transformation is the fWe define over-expressed outliers as features that have a COPA-transformed value greater than the 75th percentile plus 1.5 times the inter-quartile value . Under-expressed outliers are defined as features with a COPA-transformed value less than the 25th percentile minus 1.5 times the inter-quartile value . This pr1. We require the feature to be an outlier in at least one tumour sample;i.e. for up-regulated outlier features, it cannot be an up-regulated outlier in any of the normal samples, but it can be a down-regulated outlier in normal samples); and2. We require that the feature is not an outlier of the same direction (up or down) in any normal samples . Non-normalised datasets were converted to ExpressionSet objects using Biobase. We applied a log2 transform and then normalised using the ge in R.. Finallyge in R. with an The features selected by the four methods were then passed to four clustering methods: K-means (KM), PAM, and Clues with the CH strength index or the Silhouette (Sil) strength index. The Stats package from R was used for K-means clustering, with the settings of 20 repeats and “MacQueen” algorithm. The Cluster package of R wasThe quality of the clustering was measured by the Adjusted Rand Index (ARI). The ARIij nij represents the number of sample pairs in the same cluster in K and in the same cluster in L. ∑i ni represents the number of samples in each cluster in K, and ∑j nj the number of samples in each cluster in L. Each of the four feature selection methods was combined with each of the four clustering methods, resulting in 16 ARI scores for each of the 12 data sets. Since normal distribution of these data cannot be assumed, we applied the parameter-free Kruskal-Wallis test[where ∑lis test to identMclust package[pgirmess package[The  package for R wa package of R waset al. 2007[ImputeMissingValuesKNN module from GenePattern, with k set to 10.mCOPA was applied to the dataset of Tomlins al. 2007 to general. 2007 was downtoptable command of the Limma package in R was used for output analysis results. An adjusted p-value of 0.01 was set as the threshold. All probes in the toptable output file with an adjusted p-value of less than 0.01 were chosen as DE probes, and the corresponding expression data were then extracted and used as input for the following clustering analysis. 90th and 10th percentile values were used in the mCOPA up- and down-regulated outlier selection, and columns of the output lists containing the expression values of up- and down-regulated outlier probe lists were extracted and merged into a single file for the following clustering step. The HierarchicalClustering module of the GenePattern package[A matrix file with 16572 rows and 87 columns was used in the analysis (DemoInput.txt available from the mCOPA website). The mi package was usedFollowing the clustering of the samples based on the selected features, the metastatic cluster was selected and separate up- and down-regulated outlier feature lists were generated using the getSubtypeProbes.pl function of the mCOPA package. These two feature lists were then converted into gene lists based on the array annotation file. Outlier profiles for the samples from the metastatic subtype were then used to generate sample-specific outlier lists. Pathway analysis in the software package MetaCore from GeneGo Inc. was then used to analyse these outlier lists.http://www.bioinformatics.org.au/mCOPA. The mCOPA method takes normalised expression values in matrix format as input, with each line representing a feature, each column representing a sample, and normal samples followed by tumor samples. The user also needs to indicate the number of normal samples (n) in the dataset (present as the first n columns of samples), and nominate the percentile values used for up- and down-regulated outlier detection  dataset), functionality absent from the earlier R implementation, which outputs only mutually exclusive outlier pairs. Additionally, the earlier R implementation of COPA does not use established definitions of outliers . Instead, it applies a hard-coded threshold which calls any feature an outlier in samples in which its COPA score exceeds 5, regardless of the distribution of these scores across all the samples, whereas our method applies sensitive, feature specific criteria to determine if features are outliers in a given sample. The COPA implementation in the Oncomine database only ranks genes, with no threshold applied to clearly define which genes are outliers. The additional restriction that it can only be applied to datasets within Oncomine reduces its use as a general analytical tool.mCOPA has been designed to select outliers as features that may be used in subsequent downstream analysis  or tWhile the ARI provides an objective index for comparing clusters with known clinical groups, there are some problems with this approach. Some datasets do not cluster well and score very low values; in particular, GSE7410 , GSE15824 (Brain) and GSE29431 (Breast) have very low ARI scores and the highest score for these datasets is lower than the lowest score across the many of the other datasets .We next asked whether the different feature sets correspond to unique biology. After mapping features to GO terms using DAVID,42, we oet al.[To explore the performance of mCOPA in more detail, we applied our method to the prostate cancer dataset of Tomlins et al.. We examBoth the mCOPA and DE feature sets , or (ii) pathway disruption is specific to one or more samples within the subtype .1. Outlier lists for subtypes can be used to perform pathway enrichment, identifying pathways disrupted by outliers in that subtype; outliers in each pathway can then be mapped back to the samples in which they occur to determine if (i) pathway disruption is general within the subtype . PTEN is a known tumour suppressor, and loss of PTEN function is associated with cancer progression,52. PTENFollowing the second approach outlined above, an outlier list was extracted for each metastatic prostate cancer sample in the Tomlins dataset. Samples show high variability in the number of over- and under-expressed outliers they contain (ranging from 4 to 196 outliers). While 40% of all outlier features are present in only one metastatic tumour sample, fewer than 2% of the outliers are present in five or more samples  showed that four of these 12 genes are known tumour suppressors in prostate cancer , while five  have been reported to be tumour suppressors in other cancers. The remaining three genes  represent potentially novel tumour suppressors.The original COPA method was used in conjunction with a set of known oncogenes to identify up-regulated oncogenic outliers. Because our extension to the method identifies down-regulated outliers as well, we investigated whether mCOPA might identify tumour suppressors. A set of 727 cell cycle regulators containing many known and potential tumour suppressors was extracted from the Gene Ontology Database. Of 223 We reviewed the annotation available for these three genes in Uniprot, and examined their expression profiles in Oncomine. The Fizzy-related protein homolog Fzr encoded by FZR1 plays a role in the degradation of positive regulators of cell cycle, and prevents entry into mitosis following DNA damage. TCGA datasets in Oncomine reveal that FZR1 has a significant loss of copy number in prostate, ovarian, lung, gastric, endometrial and breast cancers. Its expression is significantly reduced in 46 experiments. A similar trend is seen with the HEXIM1 gene, which encodes protein HEXIM1, a transcriptional regulator that acts as a general transcription inhibitor. This gene has significant copy number loss in prostate, ovarian, breast, colorectal and endometrial cancers, and is significantly under-expressed in 101 experiments collected in Oncomine. The third candidate tumour suppressor we uncovered, CDC14B, codes for the protein Dual-specificity protein phosphatase CDC14B, an essential regulator of the G2 DNA damage checkpoint. It does not show significant loss of copy number in TGCA prostate cancer data, but does show significant loss in breast, ovarian, renal, lung and endometrial cancers, and is significantly under-expressed in 84 experiments. Together these analyses demonstrate that these three genes are credible as potential tumour suppressors; they are subject to copy-number loss in a wide range of cancers, and are significantly under-expressed in a large number of microarray experiments.Here we have shown how mCOPA-derived cancer outlier profiles can be used to interpret cancer microarray data. We evaluated outlier profiles as a feature-selection method for clustering clinically defined cancer subtypes, and compared the performance of mCOPA to three other outlier selection approaches. mCOPA consistently selects features that are more informative. We hypothesise that this is because of the properties of outlier expression profiles, which capture the different molecular processes and networks disrupted in individual tumour samples. Approaches such as differential expression analysis, which identify features that are consistently different across cancer samples compared with normals, do not reveal this biological heterogeneity. Given the lack of overlap between genes and corresponding biology targeted by the feature selection methods we examined, we propose that researchers should explore multiple complimentary approaches, including mCOPA, in analysing high-throughput data, so as to exploit more fully the range of biology to which these approaches give privileged access.et al. dataset[Application of our method to the Tomlins  dataset highlighmCOPA provides a new tool for the understanding of cancer heterogeneity and individual differences as captured in expression array experiments. Additionally, as most existing microarray studies have used differential expression analysis, the opportunity exists to use outlier tools such as this to reanalyze and reinterpret existing data with far greater granularity. Sample-specific analysis requires new ways of interpreting results, and the integrated methods we apply here demonstrate such new approaches. In combination with well-structured experimental design and clinical annotation, sample-specific analysis creates an opportunity to identify the mechanisms underlying rare disease subtypes and map these variations to individual differences in etiology and treatment response. mCOPA provides insight into the unique transcriptional landscape and molecular networks of individual patients or samples, and represents one of a new breed of bioinformatics tools designed to provide the analytical capability required for computational analysis in personalized medicine.DE: Differential expression; COPA: Cancer outlier profile analysis; mCOPA: Modified cancer outlier profile analysis; ARI: Adjusted Rand index.The authors declare that they have no competing interests.CW developed the algorithm and implemented the software. CW and MJD performed analysis on the Tomlins dataset, and interpreted the results. AT and SRM performed the comparative evaluation of clustering performance using public datasets. CCN and MAR provided guidance and resources for the project. CW, AT, SM and MJD contributed text to the manuscript. All authors read and approved the final draft.PublicDatasets. Details of 12 datasets used in the evaluation.Click here for fileARIScoresInEachExperiment. Analysis of the significant differences of each method in each experimental dataset.Click here for fileARIScoreDistributions. Boxplots showing the distribution of ARI scores for each experiment.Click here for fileAnalysisOfFeatures. Comparison of the features selected by each method.Click here for fileAnalysisOfGOterms. Comparison of the GO terms selected by each method.Click here for fileTomlinsFeatures. Outlier features identified in the Tomlins dataset.Click here for filePathwayAnalysis. Detailed outlier-based pathway analysis of metastatic samples from the Tomlins data.Click here for file


Both molecular marker and gene expression data were considered alone as well as jointly to serve as additive predictors for two pathogen-activity-phenotypes in real recombinant inbred lines of soybean. For unobserved phenotype prediction, we used a Bayesian hierarchical regression modeling, where the number of possible predictors in the model was controlled by different selection strategies tested. Our initial findings were submitted for DREAM5  and were judged to be the best in sub-challenge B3 wherein both functional genomic and genetic data were used to predict the phenotypes. In this work we further improve upon this previous work by considering various predictor selection strategies and cross-validation was used to measure accuracy of in-data and out-data predictions. The results from various model choices indicate that for this data use of both data types  simultaneously improves out-data prediction accuracy. Adequate goodness-of-fit can be easily achieved with more complex models for both phenotypes, since the number of potential predictors is large and the sample size is not small. We also further studied gene-set enrichment (for continuous phenotype) in the biological process in question and chromosomal enrichment of the gene set. The methodological contribution of this paper is in exploration of variable selection techniques to alleviate the problem of over-fitting. Different strategies based on the nature of covariates were explored and all methods were implemented under the Bayesian hierarchical modeling framework with indicator-based covariate selection. All the models based in careful variable selection procedure were found to produce significant results based on permutation test. The simple statistical screening methods to find phenotype-genotype association or phenotype-expression association are still much used in practice because high dimensionality of the genomic data prevents use of more advanced statistical variable selection methods due to their computational demands. This state-of-the-art data combined with use of outdated statistical tools controversy have made a question of applicability of dimension reduction techniques very acute. Thus, several statistical initial screening methods have been developed which can filter large sets of predictors to the smaller sized sets so that more advanced methods can be applied to the selected predictors in the subsequent stage. By reducing size of the predictor set, the variable selection problem becomes less ill-posed as number of predictors is starting to exceed the number of individuals. Also a technique called “preconditioning” which can reduce noise from the variable selection experiments have received substantial attention recently among statisticians see . HoweverIn this paper, we have applied Bayesian methods in the context of association mapping to predict unobserved phenotype values of the soy plants with single feature polymorphism (SFP) genotype data, gene expression data and also both of these data types combined. We will loosely refer to the genomic locations like SFPs and genes as “markers”. Prediction is done using Bayesian models which perform simultaneous variable selection and parameter estimation. Our predictions were ranked as the best performing results for sub-challenge B3 of DREAM5 . The further improvements of these models are presented here and their accuracies are assessed in comparison to our original B3 models.http://wiki.c2b2.columbia.edu/dream/index.php/Challenges and in The data explored here is based on genetic and functional genomic studies on soybean provided as a part of DREAM5. The data sets along with descriptions are available from Recombinant Inbred Lines (RILs) were produced by many generations of selfing starting with two distinctly different inbred lines with substantial difference in susceptibility towards a major pathogen. Genotype measurements, in the form of single feature polymorphism (SFP), were available for 941 locations on the genome. Gene expression measurements on 28395 genes were also available. In all, SFP measurements were available on 260 plants and gene expression measurements were available for an overlapping set of 260 plants. There were 230 common individuals for whom both SFP and gene expression measurements were available.The response variables in this data were two different measurements on pathogen activity, which will be referred to as “phenotype(s)”. Both the phenotypes represent measures of amount of pathogen in the infected tissue sample and relate to severity of infection. The first phenotype (denoted as phenotype-1) is measured as ‘percent present’ and the second (denoted as phenotype-2) is captured as ‘scale factor’. Interestingly both the phenotypes were measured on continuous scale and their distributions were approximately normal. Because of this, we use parametric Pearson correlation coefficient instead of Spearman correlation for accuracy assessment in following if not stated otherwise.The original data sets came with suggested learning and test sets (given split-sample data) for carrying out various predictive exercises. The original requirement of the challenge was to analyse the split-sample data provided and the best results were obtained by us and these were 0.28 (0.19) and 0.24 (0.18) for the two phenotypes according to Spearman (Pearson) correlation. However initial exploratory analyses of the corresponding subsets of phenotype data suggested that these sets unfortunately may differ from each other in their underlying statistical distributions, although the sets were created by random sub-sampling . Thus, pk-fold cross-validation method k-(near) equal subsets and prediction algorithm is trained on the data separately k-times each time leaving one subset out. Prediction is then assessed by measuring the predicted outcome/response of the unseen part of the data not used for training. Ideally training set should be as large as possible; however this also means that the size of test set will be small. In order to exhaust the entire data covering by small test sets would imply many repetitions of the training and prediction procedure, increasing the computing time linearly. As a compromise between learning set size and computation time we chose k = 5. Accordingly learning sets were created randomly, however post checks were made to ensure the all the subsets of phenotypes (the main variables of interest) have similar distribution on all k-sets  only SFP data, (2) only gene expression values, or (3) both SFP and gene expression data jointly. We also carried out different strategies to select important covariates into the regression models. Summary of these are presented in M is the set of all markers (i.e. SFPs and/or gene expressions), X = th entry corresponding to that of plant i and marker l. In case of gene expression data, these entries contain values of transcription abundances and in SFPs, they are genotype codes with numbering depending on the parameterization of the model; for example, value zero  of an individual plant a-priori that there is only a small subset of important markers that are useful to predict the phenotype. In these predictive models, subset selection of important marker effects to the predictive model (after pre-selection) is based on use of random indicator variables to be estimated  are also assumed to be Gaussian with marker and marker type specific distributional choice for the variance parameters. Typically in indicator-based variable selection models the prior probability for each SFP to be involved in the model is Prior specification is intrinsically subjective and specifying prior that will satisfy everyone and/or every aspect might be un-achievable. We adopt the method where priors reflect our intuitive knowledge but are also useful in avoiding some potential pitfalls and helping to reduce the computational burden. Typically, we assume Gaussian errors with unknown variance under inverse-Gamma distributed prior. We assume a standard Normal prior distribution for the intercept parameter The models were implemented in WinBUGS It is known that including only the most associated subset of trait loci  from the genome-wide association studies to the predictive model suffers from low predictive power The kind of data that we have used here is becoming more commonly available and it has both the above aspects, that is studying association aspect is of importance as well as putative covariates from high throughput techniques are available where actual sample size is comparatively not large. Thus exploring appropriate variable selection becomes essential for useful further application of models thus developed. The following alternative strategies were considered.a-priori assumption of approximately 100 SFPs to be effective in the model.The size of the original marker set was not large, covering 941 SFP measurements for each plant, and thus drastic initial steps to reduce size may not be necessary. In our initial attempt, a shrinkage parameter value of 1/10 was used as prior probability of inclusion into the regression model for each SFP individually (see model 1 above). This corresponds to t-statistic which was used to measure the marginal relevance of an SFP to predict a phenotype and this is repeated for each phenotype separately. The two phenotypes produced rather different rankings of the SFPs according to this criterion. The cumulatively chosen top subsets of SFPs were then used in the predictive model for each individual phenotype separately.Alternately SFPs are ranked according to their (absolute) value of the t-statistic described above is only measuring marginal predictive performance and therefore a subset of SFPs selected based on their marginal significance may result including some redundant predictors into the model. Thus in addition to the t-statistics a further reduction or control on inclusion can be achieved by assigning indicator variables to control inclusion of markers in the model. The indicator variables were assumed to have inclusion probabilities distributed as Uniform distribution. Thus if a subset of size k top SFPs are included in the model then prior inclusion probability further assumes a-priori that only k/2 markers are effective in the model.The The genes are ranked according to their (absolute) correlation between transcript abundance and phenotype measurements over the plants. Information on all 260 plants, with expression data, were used for this purpose and top 10% genes were selected as a primary set of genes for all expression data related analyses. Once again the lists were specific for each phenotype.a-priori independent) degenerate prior distribution for inclusion of a gene in the model was used with prior shrinkage probability of 0.01 for gene expression data.Similar to the SFP data shrinkage parameter based on  was derived. A surprising amount of variability was noticed amongst the top genes thus listed. Only 237 genes for phenotype-1 and 180 genes for phenotype-2 were common in all 5 subsets amongst the top 5% genes. Then top genes common in all 5-lists were cumulatively taken to form subsets . Alternatively, bootstrap re-sampling technique may be used to estimate the variability in phenotype/expression correlation of genes. In this method, the selection procedure would be independent of choice of specific k-fold stratification of the data and genes found to have high variation among the (correlation-wise) top genes may be avoided for further analysis. This strategy is somewhat related to so called “stability selection” of The Similar to marginal testing of SFPs, subset selection based on the highest phenotype/expression correlation will also result in some redundancy among selected predictors in the model. Similar to that applied to the SFPs, further stringency on inclusion in the model is attempted for gene expressions by introducing suitable indicator variables with vague prior probabilities.Principal Component Analysis (PCA) was carried out for the expression measurements of the top 10% genes first selected based on the highest marginal phenotype/expression correlation. The top principal components explaining majority of the phenotypic variation are then used as predictors in the model. This method is known as supervised PCA t-statistics of the SFPs and phenotype/expression correlations of the genes have no evident relationship/connection with each other in two phenotypes  correlation between transcript abundance and phenotype over the re-sampled plant data sets of each gene. This effort indicated the likely independence between the mean and the standard deviation of such correlations. That is a gene can have on an average high correlation with phenotype but could have any possible variation (across re-sampled data sets) and similarly the most stable genes (across re-sampled data sets) can have any degree of predictive ability  and non-indicator models  with different choices of parameters. It appeared that if the effective numbers of covariates are kept same in indicator and non-indicator model the indicator model produces better in-data prediction and comparable out-data prediction. This is not surprising since to make the effective number of covariates comparable in the two models the potential set of SFP and/genes for indicator model will be larger allowing the model to explore more complex models yielding better in-data prediction. To remove this added advantage of the indicator models and make the comparison more stringent we applied indicator models on the same subsets of SFPs and/genes that produced the best results (among the ones explored here) with the non-indicator model. It should be noted that thus the indicator models effectively uses approximately only half the covariates due to the added prior on indicator probability. Surprisingly the indicator models produced equally good or comparable results for out-data prediction (After applying ediction .It is expected that since the indicator-based model is able to use and explore larger set of prospective predictors compared to (fixed/pre-specified) non-indicator model it would be able to provide insight in choice of preferable sets of covariates. Unfortunately from this aspect the results on this particular data were not helpful.t-test/(absolute) correlation  and posterior estimates of relevance from joint distribution based on vague priors (i.e. joint estimates) based on indicator models were carried out. The relevance measures used here are weighted genetic variation for SFPs However comparison of variable selection measures as given by Most of the published works on gene-set enrichment analysis are based on situations where phenotypes have distinct categories of outcome, like cancer types, or treatment/control etc viz. SFP and expression data, made it further difficult to provide a consolidated enrichment picture of the underlying processes.Thus, we studied chromosomal level enrichment of genes and enrichment of the gene sets involved in different biological processes Therefore we summarized the enrichment of these genes from two complimentary contributions by them. Based on the genetic data the summary measure reflects criticality of the average genetic variation in a particular gene-sets in prediction of phenotype. This is measured by the average weighted genetic variation of the SFPs involved in the process. The expression data on the other hand enables us capture the functional variability of the genes involved in a process. The coefficient of a particular gene in the predictive model captures individual contribution of that gene in the prediction. However these when averaged over several genes involved in a pathway might present altogether a different picture. This is measured by the weighted coefficients for the genes involved in the process and was calculated jointly for the gene set.Firstly note that both these measures include probability of enrichment as well as magnitude of enrichment. Secondly the contribution of any gene/marker could very well be affected by presence or absence of other genes and/or markers in the model. As a result, as we will see in subsequent exploration, results and interpretations may very well have to be context specific.For biological processes, the nature of the inclusion coefficients (corresponding to gene expressions) remained broadly the same in presence and in absence of SFPs in the model and this seems to the case for both phenotypes. (Average) weighted genetic variations of the SFPs however do differ with and without expression data in model. Also this departure is not similar for the two phenotypes . For pheThe comparative pattern of chromosomal level enrichment picture based on SFPs is somewhat opposite to what was observed for the biological processes. That is, the presence or absence of gene expression information in the predictive model affected to the magnitude of the weighted genetic variation for phenotype-2, whereas the estimates for phenotype-1 under the two models appeared to be roughly mirror images of each other .t-test, 2) weighted genetic variation estimated from (indicator) model with SFPs only and 3) weighted genetic variation estimated from (indicator) model with SFPs and expression data. Similarly for gene expression data three different relevance measures were considered 1) correlation, 2) weighted inclusion coefficient estimated from (indicator) model with expression data only and 3) weighted inclusion coefficient estimated from (indicator) model with SFPs and expression data. SFPs in top ten according to any one of these measures were considered as top in the overall list. Also the biological processes annotated in the top covariates SFPs were also obtained  predictive results for all data types considered in the illustration.With increasing availability of genomic high-throughput data along with the shift in our objective of simple biomarker identification to phenotype prediction, the problems (in predictive ability of the model) as experienced here is expected to occur in other situations as well. That is we believe that these problems are not specific to the data or problem at hand. Thus we felt it would be useful to explore different intuitive and rigorous variable selection and/or data reduction techniques. Keeping in mind that there is also the added complexity of differing data sources/types it was essential to make these investigations context specific. Attempt was made to strike a balance in exploring different methods and use of different sources of information.a-priori Normal distribution was assumed which in combination of prior distributions of coefficients and other parameters in the model constituted a Student's t-like distribution with heavier tails. It is known that although t-type distributions allow more mass to the tails than the Normal distribution they are symmetric. However the first phenotype appears to be asymmetric when compared to the second phenotype measured from the same subjects. This could very well be the reason why predictive performance for the first phenotype is consistently poorer than the second phenotype in all the different data and variable selection efforts explored here. Thus, use of skewed distributions (like Gamma) might be worth exploring in the future.Certain peculiarities in the phenotype distributions were also noticed, in particular for the first phenotype. The first phenotype seemed to have heavier right tail. For the current analyses an Lack of stability among selected variables in high-dimensional data could influence the performance of any predictive model. Because a degree of variability (across samples) in data type like gene expression might be high, it is unrealistic to expect a finite  learning set to truly capture/reflect the underlying variability. Thus more often than not test sets elements would fall outside the data domain  were taken from the 260 plants. Correlation between each gene's expression and phenotype was calculated for each such subsample. Then for each gene mean and stander deviation (std.) over 100 such bootstrap subsamples were computed. Below top row pertains to phenotype-1 and bottom row pertains to phenotype-2. Note that the horizontal axis with negative values represents mean and the horizontal axis with positive values only represent the standard deviation. Left panel: Equidistant bin points were identified for mean and std., (%) frequency of genes in these categories were cross tabulated and plotted. Right panel: Equal percentile points were identified for mean and std., (%) frequency of genes in these categories were cross tabulated and plotted.(TIF)Click here for additional data file.Figure S2Correlations between out-of-sample predictions and corresponding observed values of phenotypes were computed for a wide range of models under two different learning set creation schemes. The two schemes for learning and testing creations are as follows. In the first scheme k( = 5) equal sized folds (of 52 samples) of the total sample (of 260) were created focusing on homogeneity of phenotypes only across the folds. In another scheme attempts were made to assure that the samples in test set are not in the region on of extrapolation of the learning set. This was done by partial homogenization of the samples using PCA. In this method the learning set was of size 208 and the test set was of 52. For data reduction and predictive model formation, Supervised principal component analysis (SPCA) followed by multiple regression was carried. Input variables in the regression model were selected based on 48 different cut-off values on correlation and 34 different choices on number of principal components to use as predictors. In all 1632 models were attempted for each scheme of test set creation. For both K-fold and split sample method, no information from test set was used to carry out data reduction or to form the predictive model. Top row-1 presents results for phenotype-1 and the bottom row those for phenotype-2. Note that the left pane presents results corresponding to 5 fold validation and the right panel presents those for split sample analysis.(TIF)Click here for additional data file.Table S1Gene Ontological information on top SFPs selected based on joint and marginal estimates of relevance. For each phenotype separately three different relevance measures were considered 1) marginal t-test, 2) weighted genetic variation estimated from (indicator) model with SFPs only and 3) weighted genetic variation estimated (indicator) model with SFPs and expression data. SFPs in top ten according to any one of these measures were considered as top in the overall list.(DOC)Click here for additional data file.Table S2The biological processes annotated in the top SFPs (see (DOC)Click here for additional data file.Table S3Gene Ontological information on top genes selected based on joint and marginal estimates of relevance. For each phenotype separately three different relevance measures were considered 1) correlation, 2) weighted inclusion coefficient estimated from (indicator) model with expression data only and 3) weighted inclusion coefficient estimated (indicator) model with SFPs and expression data. Genes in top twenty according to any one of these measures were considered as top in the overall list.(DOC)Click here for additional data file.Table S4The biological processes annotated in the top genes (see (DOC)Click here for additional data file.Text S1Contains further notes on supplementary (DOC)Click here for additional data file.


The quality of gene expression data can vary dramatically from platform to platform, study to study, and sample to sample. As reliable statistical analysis rests on reliable data, determining such quality is of the utmost importance. Quality measures to spot problematic samples exist, but they are platform-specific, and cannot be used to compare studies.As a proxy for quality, we propose a signal-to-noise ratio for microarray data, the “Signal-to-Noise Applied to Gene Expression Experiments”, or SNAGEE. SNAGEE is based on the consistency of gene-gene correlations. We applied SNAGEE to a compendium of 80 large datasets on 37 platforms, for a total of 24,380 samples, and assessed the signal-to-noise ratio of studies and samples. This allowed us to discover serious issues with three studies. We show that signal-to-noise ratios of both studies and samples are linked to the statistical significance of the biological results.We showed that SNAGEE is an effective way to measure data quality for most types of gene expression studies, and that it often outperforms existing techniques. Furthermore, SNAGEE is platform-independent and does not require raw data files. The SNAGEE R package is available in BioConductor. High-throughput gene expression data have been generated by microarrays for over ten years. The technology has, however, remained plagued with quality issues, to the point that a consortium MAQC  was creaWhile quality control metrics are available, they are platform-specific, and are meant to compare the relative quality of samples. There are currently no methods to compare different studies or different platforms. A number of R packages implement those quality metrics: Affyexpress, simpleaffy A different approach was proposed This shows that there is a need for a quality measure that is more motivated by the underlying biology. Quality is a vague term, we propose to define here data quality as the statistical strength of the biological conclusions the data support. This could be expressed as the amount of biological signal relative to the amount of noise, that is a measure of the biological signal-to-noise ratio (SNR). As microarray studies can be used to answer many biological questions, possibly unrelated to the ones treated in the original publication, the strength of the biological signal is not determined based on sample annotations, but directly using the values of the microarray experiments. Whether SNR is a reliable estimate of data quality, in the sense of well-made experiments, depends on the study type. The signal-to-noise ratio depends not only on the amount of noise, which could be seen as a direct measure of quality, but also on the amount of signal. For instance, a high-quality study comparing a cell line in two different conditions could have very little variability, and so a low SNR. However, we show in this paper that SNR is a good proxy of quality for studies that comprise a large number of diverse samples, like for instance large studies on cancer tissues, and can reliably be used to rate comparable studies. It can also be used to flag problematic samples inside a study.It has been shown The use of gene correlations to assess gene reproducibility has already been explored The study SNR was calculated on all studies in our database. This allowed to single out several studies having serious issues. We also show that studies with lower SNR lead to less significant biological results. The SNR measure for samples was studied in several datasets. We show that first it properly flags problematic samples, and second that statistically less significant results are obtained on those samples.We estimate the SNR of a study using correlation of gene correlations. A study here is a set of hybridizations done as a group on one platform—if the data relative to a publication consists of more than one platform, then there are as many studies as there are platforms.The expected matrix of gene-gene correlations Specifically, data were put on a log scale and normalized, and multiple probes corresponding to the same gene were averaged (see the Data section for details). From the data matrix The error on the measure of gene-gene correlations decreases for large studies. If the data are normally distributed, their We estimate the SNR of a sample as the difference between the SNR of its study and the SNR of its study when the sample is removed. Removing a low SNR sample should increase the SNR of its study. The SNR of a study with a sample removed can be calculated efficiently since a sample has a simple additive effect on the mean, variance and covariance. We used a slightly modified SNR measure for this comparison:The SNR All calculations were done on the 2000 genes that were present among the most platforms in our database. More precisely, for each gene the number Most datasets were downloaded from GEO The correlations between genes were calculated for each platform and each study, giving as many correlation matrices as there are study/platform pairs, that is 101 in our case. The median of those matrices was taken, leading to a single The relevance of our SNR measure was shown using methods detailed below.Four NCI60 studies were taken , leading to a total of 9 study/platform pairs. We considered the 58 cell lines in common between those studies. SNR were not disattenuated as all data sets have the same number of samples. For each tissue type, the fraction of differentially expressed probes at e.g. background), control spots and the behaviour of specific genes. Bad quality arrays are found by MDQC as outliers in terms of those metrics.SNAGEE was compared with simpleaffy, NUSE, RLE, GNUSE and MDQC. Simpleaffy is an R package that can be used to calculate quality metrics on an array. Those metrics are based on array-wide measures  was recorded see . There iA specific study on the NCI60 cell lines was taken (GSE5720 on U133A—similar results were obtained with other datasets), and increasing amounts of Gaussian noise were added. This was done by fixing a level of noise  of DEGs , even thin vivo tissues The SNR is a measure of the amount of biological signal divided by the amount of noise. As such, it is increased by the presence of biological outliers. This stands in contrast with other quality measures, which search for outliers and so risk flagging biological outliers as low quality lculated . The addWe took a study of breast cancers (GSE6532 on U133A) as another illustration, with less striking outliers. An important feature of those cancers is the status of the estrogen receptor. An ESR− sample is an outlier in a study of only ESR+ samples. The SNR of studies consisting of 150 randomly drawn samples, a small number of ESR−, the rest being ESR+, was calculated . The SNRSNR was calculated on random small subsets of the expO dataset , using disattenuation (4) or not. The results  show thaExperiments measuring smaller biological signals are assigned a lower SNR. However, the SNR remains meaningful if there is some biological variability in addition to the signal. For instance, GSE1322 GNUSE allows to determine the quality of individual samples. By counting the number of such problematic samples, a study quality can be computed. We compared this study quality with the SNR determined by SNAGEE. Quite surprisingly, we found very little overlap between the two, albeit the five studies with 100% problematic samples, as determined by GNUSE, had indeed a slightly lower SNR than average  we found that they were very targeted: they all consisted of a single cell line cultivated in a single medium, with different treatments. Such design, meant to highlight a single biological effect, leads to a low biological SNR.A contrario, some studies were given a high SNR while GNUSE gave them a low quality. The authors of GNUSE could not pinpoint why some studies had a low quality. Looking at the list of studies that had over half bad samples from GNUSE but an SNR over 0.3, we found that half (6/12) used RNA amplification. This hints that the difference in the RNA preparation leads to sizeable differences in the raw probe intensities. SNAGEE, being based on gene correlation, is largely immune to those effects. Another study flagged by GNUSE but not SNAGEE is GSE9716, a study of tumor xenografts. It is likely that those tumors contain a mix of cells from mouse and human. Hybridization efficiency is expected to be very different between the two species. However, we found that gene correlations were largely conserved. We did not find a plausible explanation for the remaining 5 studies.The SNR assigned by SNAGEE to each sample was compared to the quality measure of RLE, NUSE, GNUSE as well as the metrics given by simpleaffy and MDQC. We first used the breast cancer dataset from MAQC-II (GSE20194). Problematic samples can decrease the quality of the analysis performed. To verify that, we tried to classify samples with a naive Bayes classifier. Since bad quality samples should not fit the classification well, the posterior probabilities can be used as a proxy for sample qualities: bad quality samples should have a lower probability than average see . HoweverFirst, we took the breast cancer dataset from MAQC-II Second, a similar analysis was performed on another breast cancer dataset (GSE4922). Low SNR samples did not fit the classification well and had lower posterior probabilities . Based oAs another illustration, we calculated the qualities of the samples from a study of muscle diseases In the analysis of the relationship between statistical significance and study SNR , statistWe studied the behavior of SNAGEE on biological outliers using the study on muscle diseases (GSE3307). The SNR for each sample was first measured. Each disease was then taken in turn, and all samples from that disease were removed but one. That sample was then in the situation of being a biological outlier. The SNR of that sample was calculated. As can be seen For a more dramatic effect, we used the study of normal and diseased tissues, plus cell lines (GSE7307). The same analysis as with the muscle diseases was made, using the cell lines as the biological outliers. As shown We have shown on the NCI60 datasets that SNAGEE ranked the studies in a statistically relevant manner, in that more significant results were found on studies with a higher SNR. This was the case even though p-values were calculated at the probe, not the gene, level, and did not take gene annotations into account. We have also shown in a controlled setting that SNAGEE gives a direct measure of the added noise. This shows that the study SNR measure of SNAGEE is useful to compare studies with similar samples.SNAGEE is a signal-to-noise ratio. As such, it increases if the signal—the variation in the data—is increased. This fits the concept that higher SNR studies hold more information, but must be kept in mind when comparing studies with different experimental contents. This is why we studied extensively the behavior of SNAGEE relative to biological outliers. SNAGEE gave them consistently a higher SNR than warranted. This means that the measure of SNR for a sample is not a measure of quality, but a measure of the added biological diversity brought by a sample to its study. A biologically plausible sample  will be given a high SNR even if it has moderate technical issues, while an average sample with moderate technical issues will be given a low SNR. This is in contrast with results from MDQC, which gave a low quality to a category of biological outliers in the muscle disease study. SNAGEE also gave a higher SNR to complex studies, with different types of samples, than to more focused studies, all other things being equal.A main limitation of SNAGEE is that it is dependent on the biological variation between samples. The signal-to-noise ratio at the core of SNAGEE is a proxy for quality only if there is some biological variablity between the samples. Very focused studies, for instance comparing only two cell lines, or one cell line treated with small stimuli under very controlled conditions, have often very low biological variability, and may be assigned a low SNR. In this case, the low SNR is telling of the experimental conditions and not of study quality. Similarly, studies with only few different conditions cannot be reliably assigned an SNR, since gene-gene correlations have to be calculated on a fair number of different samples. In particular, it was not possible to apply SNAGEE to MAQC, the benchmark for quality control of microarrays, as it consists only of two cell lines measured in many replicated ways. Typically, SNAGEE gave reasonable estimate of the quality of studies above 20 samples, unless those samples are replicated measures of a few well-controlled conditions, like MAQC.The SNR given by SNAGEE is an estimate of the amount of biological information available in the gene expression measure of a study. A higher SNR does not necessarily means that a study is more adequate to answer a given biological question than another study. This depends largely on the samples measured and the quality of their annotations, which are beyond the scope of SNAGEE. For instance, in the case of cancer tissues, a study with carefully dissected samples could have less biological variability than another study with samples contaminated by adjacent tissues. The first study may be more informative about the state of the cancer, while the latter would have a higher SNR. Practically, however, for similar studies the variation in SNR is mostly due to the noise, not the signal, which explains why SNAGEE gave a reasonable estimate of study quality in the cases presented.The study quality measure being dependent on the study type, and the natural variation between its samples, it should only be used to compare similar studies. The exception for this is studies where problems are suspected, as a very low SNR (below 5%) is telling of serious quality problems, and SNR much below those usual for the platform may be telling of systematic issues, like for instance the normalization by batches that was shown.A strength of SNAGEE compared to available quality metrics is that it works across platforms. However, if the goal is to compare studies on the same platform, or to compare samples, the use of a platform-specific gene-gene correlation matrix could be an improvement. This would improve the accuracy of SNR determination by getting rid of the probe annotation problem. Since a large number of studies are necessary to reliably determine the gene-gene correlation matrix, this could only be done on popular platforms.SNAGEE was compared to other methods for the detection of problematic samples. Some samples, among which the most problematic ones, were flagged similarly by all methods, while other samples were ranked differently. We used the posterior probability of a naive Bayes classifier to assess the validity of those rankings. On the examples shown, SNAGEE was shown to outperform the existing methods. It was shown in particular that all methods can fail if the experimental setup interferes with the quality metrics, as in the muscle disease where GAPDH is regulated for MDQC, or mRNA amplification for GNUSE, or very simplified experimental setup for SNAGEE. This highlights the danger of using only one method for quality control. Furthermore, those methods cannot be used blindly, and their limitations must be taken into account. Hence, the three methods are complementary on Affymetrix slides. SNAGEE is however the only one available across platforms.We have shown that SNAGEE flags samples that are harder to classify and so are of bad quality. It could be reasoned that samples can be harder to classify because they are biological outliers, but as SNAGEE gives biological outliers a high SNR this is not the case. Samples flagged by SNAGEE, and other methods are not always the same, though, which further shows that the techniques are complementary.It could come as a surprise that average gene-gene correlations allow for the determination of SNR, since those correlations are expected to vary from study to study. We believe it works because gene-gene correlations have a distribution which is meaningful for any study, being largely based on genes that belong to the main cellular processes . Biological variability, independantly of the experimental protocol, slightly up- or down-regulates those processes in every sample. This is sufficient to ensure the presence of gene correlations. Of course, some genes behave differently in each biological system, but we showed that this does not prevent the successful use of SNAGEE.In conclusion, we have presented a new biology-motivated signal-to-noise ratio measure for studies and samples. It is the first signal-to-noise ratio measure for complete gene expression studies. It was shown to work for a wide range of platforms. It was also shown to be resilient to changes in the experimental design, like dye-swapping, and in the number of samples.Figure S1Values of the two probes for gene CRH in study GSE6532 platform U133A, using (A) data from GEO or (B) renormalized data. A batch effect is present is (A), but not in (B).(PDF)Click here for additional data file.Figure S2Clustering of the NCI60 from GSE7947 (black), GSE5949  and GSE5720 , using Ward linkage and correlation. Cell lines duplicated between the three studies cluster together, with the exception of A498 (red) which clusters with LOXIMVI (blue) only in GSE7947.(PDF)Click here for additional data file.Table S1List of platform / study pairs. Platforms ID are from GEO. Studies of the form GSExyz were downloaded from GEO. Others were downloaded from SGDI or the author web site.(XLSX)Click here for additional data file.


RNA sequencing (RNA-Seq) is rapidly replacing microarrays for profiling gene expression with much improved accuracy and sensitivity. One of the most common questions in a typical gene profiling experiment is how to identify a set of transcripts that are differentially expressed between different experimental conditions. Some of the statistical methods developed for microarray data analysis can be applied to RNA-Seq data with or without modifications. Recently several additional methods have been developed specifically for RNA-Seq data sets. This review attempts to give an in-depth review of these statistical methods, with the goal of providing a comprehensive guide when choosing appropriate metrics for RNA-Seq statistical analyses. Transcriptomics holds the key to understanding how the information encoded in the genome is translated into cellular functions, and how this translation process responds to the changing environment. Given a transcriptome, or the collection of all the transcripts including both protein coding mRNAs and noncoding RNAs, one of the outstanding questions in transcriptomics is to accurately quantify the abundance of each transcript within different tissues and time points, and to correlate changes in abundance to genetic and environmental perturbation in order to understand genome function and adaptation.Transcriptome profiling, or gene expression profiling, is the technology used to determine the steady state abundance of each transcript within a transcriptome. Transcriptome profiling is traditionally done using either quantitative RT-PCR (reviewed in ) to inteThere are both shared and unique aspects in the experimental design and data generation phases of expression microarrays and RNA-Seq technologies, and these attributes are compared elsewhere ,8,10,11.Over the past decade, various statistical analysis tools have been developed to analyze expression profiling data generated by microarrays (Reviewed in ). BeforeAs mentioned previously, the expression levels measured by RNA-Seq experiments are represented by the number of reads derived from each transcript in a transcriptome. We will not discuss the problem of resolving the expression levels of alternatively spliced transcript isoforms from a single gene, since this is still a challenge and undergoing active research ,18-20. Hp × n) matrix (M), with p representing the number of genes and n the number of samples. We use phenotype to refer to an experimental condition, treatment, tissue, or time point. Typically the number of genes is far greater than the number of samples (p ≫ n). Another dimension often present in RNA-Seq datasets is the number of replicates. Since the RNA-Seq protocol is highly reproducible [nk biological  replicates under the kth phenotype, k = 1, 2, ⋯, t. As a result, a typical RNA-Seq data set has a series of two-dimensional sub-matrices containing non-negative integers (counts), with each sub-matrix being derived from a specific phenotype. Therefore, we have mijk) is useded reads . RPKM-baM are raw read counts without normalization. Unlike microarray data analysis which often requires sophisticated normalization procedures to compensate for biases introduced from sample loading, imaging, and other technical or biological factors [In this review we assume that the data in the matrix  factors , RNA-Seq factors .In the next section we will discuss the statistical methods that have been developed to address whether or not a gene is differentially expressed among a group of time points or conditions. In the case of t = 2, this problem reduces to the two-group (pair-wise) comparison.R/Bioconductor packages , with μik). For the imatrix M , Table 2. For themi1(1) follows a hyper-geometric distribution. The p value is the total of the hyper-geometric probabilities for outcomes at least as favorable to the alternative hypothesis (the gene expression in phenotype 1 is lower than that in phenotype 2 (the odds ratio is < 1), or vice versa) as the observed outcome [The Fisher’s exact test is to test whether or not there exists a significant association between the gene expression and the phenotype, in other words, whether or not the odds ratio is significantly greater or less than 1. This test is based on the fact that with the assumption of Poisson sampling and fixed marginal totals, the count  outcome . A simplwill give the p value of the test.mi1(1) = 10, mi1(2) = 20, L1(1) = 1e + 6, L1(2) = 2e + 6. The estimated odds ratio is 1 (no association by Fisher’s exact test), but the fold change is 2 . Furthermore, though Fisher’s exact test was designed for analyzing datasets without replicates, if there are replicates and Poisson sampling holds true, the test can still be applied – one simply sums up the replicates under the same condition to form the 2×2 contingency table.Since the null hypothesis of independence in Fisher’s exact test is equivalent to the null hypothesis that the odds ratio is equal to 1, one can avoid a potential false positive due to the difference in the sequencing depths. As an example, consider the case: The p value obtained above is for a single gene. As in the analysis of expression data from microarray experiments, there are thousands of genes in one RNA-Seq experiment and thus we need to consider the problem of an inflated false positive rate due to multiple hypothesis testing. This problem can be addressed by directly adjusting p values or calculating q values ,33. Manymijk)( = Ljk)(νik), where νik), and then used the likelihood ratio test to identify the differentially expressed genes [Ljk) parameter into the Poisson mean is to reduce the variation in sequencing depth. If we look into the significance of differential expressions of genes on a gene-by-gene basis, the likelihood function and maximum likelihood estimations are easy to obtain. For the two-sided alternative hypothesis, by applying simple algebraic operations we have the likelihood ratio test statistic,Marioni et al. assumed that the gene count ed genes . The purvi(1) > vi(2) the p value is half of the above right-tailed probability of the Chi-squared distribution if the unconstrained maximum likelihood estimates of vi(1), vi(2) satisfy: And the p value for an individual gene can be calculated as the right tailed probability of a Chi-squared distribution with 1 degree of freedom. For the one-sided alternative hypothesis, therwise . AdjustePoisson modeling is an appropriate fit not only for sequencing data with technical replicates , but alsGiven a sampling distribution, over-dispersion occurs if the observed variance is greater than the assumed variance. In the Poisson model, over-dispersion occurs if the sample variance is greater than the sample mean. There could be several sources that cause over-dispersion in RNA-Seq data, including the variability in biological replicates due to heterogeneity within a population of cells, possible correlation between gene expressions due to regulation, and other uncontrolled variations. The existence of over-dispersion in real data was observed in several previous studies ,30. Popumijk)(ϕ and the shape parameter μik)( and the variance μik)((1 + μik)(ϕ) [ϕ > 0 and reduces to the Poisson distribution as ϕ → 0. The R/Bioconductor package “edgeR” applies this model to detect the differentially expressed genes in RNA-Seq data [μik)( = Ljk)(νik) to adjust for the difference in sequencing depths across the samples. The ith gene is differentially expressed if the parameters vik)ϕ) . The NegSeq data , where tkelihood ,37, and kelihood , and theWhile the relationship between the Negative Binomial mean and variance simplifies the estimation of these parameters, it may result in some variation unexplained by the model and thus potentially introduce selection biases in the differentially expressed genes. Anders and Huber extended the method in edgeR by hierarchically modeling this mean-variance relationship . Their mAnother modification to edgeR is given by Hardcastle and Kelly . Their mhttp://www.bios.unc.edu/research/genomic_software/BBSeq/. Two R functions, free.estimate and constrained.estimate, can generate the raw p values for genes in pair-wise comparisons. However, no function in the package directly gives the p values for multi-group comparisons.Zhou, Xia and Wright model the gene count in a sample with a Beta-Binomial distribution . AssuminThe Beta-Binomial model has been widely used to model the over-dispersed, discrete count data. For example, it was applied to analyze tag count data derived from the Serial Analysis of Gene Expression (SAGE) ; tag couF-distribution. The built-in R functions "glm" and "deviance" can be used to obtain the likelihood ratio statistics. The detailed R code for p values can be downloaded from the author’s website (http://www.stat.purdue.edu/~doerge/software/TSPM.R), last accessed on 03/07/2012.Auer and Doerge  proposedUnder the model assumptions, the authors demonstrated that the Two-Stage Poisson Model is a powerful tool for detecting differentially expressed genes. However, if other confounding factors exist such that the levels of transcription within a phenotype are not stable, this method may not control the false positive rate well. Furthermore, as pointed out by the authors, this method requires a relatively higher degree of freedom (the difference between the sample size and the number of phenotypes) in order to be effective.Next-generation sequencing technologies are revolutionizing genomic/proteomic studies, providing high-throughput datasets with unprecedented precision and accuracy. The technology for profiling gene expressions and composition (RNA-Seq) has been widely applied in biological/medical research. Appropriate and powerful statistical analysis using RNA-Seq data is essential to the research.R function rnegbin, but are modeled by a Poisson distribution with the probability mass:Generating an accurate list of differentially expressed genes is the basis for pathway or gene set enrichment analysis. A gene set with a large number of false positives will compromise these analyses. The parametric approaches discussed here are preferable to nonparametric ones in order to increase the power of detection. However, the false positive rate may be dramatically increased if the assumptions of the model distribution are violated. In Figure λ = 10. Those in the right panel are generated from a Poisson distribution with mean 10 using the R function rpois, but are modeled by a Negative Binomial distribution with the probability mass:where the mean Γ represents the gamma function and the mean μ = 10, the over-dispersion parameter ϕ = 0.5. The values for the small circles in the fitted models are calculated as the product of 1000 and p(k). The differences in both cases are not negligible, indicating the seriousness of wrong assumptions about the model.where To our knowledge, there is no paper in the literature which studies the efficacy of these methods when the model assumptions do not hold. Given the limitation of small sample sizes in RNA-Seq experiments, robust test procedures which safeguard against the departure of model assumptions are necessary.Most of the proposed methods produce raw p values for genes based upon the approximate/asymptotic null distribution. This approximation performs well for highly expressed genes but performs poorly for lowly expressed genes. This may create bias during the selection of differentially expressed genes. Some authors simply filter out lowly expressed genes. This is very subjective, and some important genes, which are expressed in one condition but not in another, may be excluded from the result. New testing approaches, which are powerful and effective for both highly and lowly expressed genes, are still needed.The authors declare no competing interests.ZF, JM and ZW wrote the manuscript. All authors read and approved the final manuscript.