Tiling arrays have been the tool of choice for probing an organism's transcriptome without prior assumptions about the transcribed regions, but RNA-Seq is becoming a viable alternative as the costs of sequencing continue to decrease. Understanding the relative merits of these technologies will help researchers select the appropriate technology for their needs.C. elegans. We find that the raw signals from these two technologies are reasonably well correlated but that RNA-Seq outperforms tiling arrays in several respects, notably in exon boundary detection and dynamic range of expression. By exploring the accuracy of sequencing as a function of depth of coverage, we found that about 4 million reads are required to match the sensitivity of two tiling array replicates. The effects of cross-hybridization were analyzed using a "nearest neighbor" classifier applied to array probes; we describe a method for determining potential "black list" regions whose signals are unreliable. Finally, we propose a strategy for using RNA-Seq data as a gold standard set to calibrate tiling array data. All tiling array and RNA-Seq data sets have been submitted to the modENCODE Data Coordinating Center.Here, we compare these two platforms using a matched sample of poly(A)-enriched RNA isolated from the second larval stage of Tiling arrays effectively detect transcript expression levels at a low cost for many species while RNA-Seq provides greater accuracy in several regards. Researchers will need to carefully select the technology appropriate to the biological investigations they are undertaking. It will also be important to reconsider a comparison such as ours as sequencing technologies continue to evolve. Unbiased, high-throughput analytical methods are essential tools for identifying novel RNAs, discerning alternative splicing isoforms, and determining gene expression levels. Tiling arrays have been the investigative tool of choice and continue to lead to novel discoveries. They effectively identify novel transcribed regions -4 and quapriori knowledge of the genome, while tiling arrays exhibit cross-hybridization and have a limited dynamic range of detection  represent the list of N TARs in descending order by rank, so that the first TAR is the least confident one. Then, we compute the FPR against the annotation for this entire list, and call this the marginal FPR of TAR1. Next, we remove TAR1, compute the FPR for the list , and assign this as the marginal FPR of TAR2. Then, TAR2 is removed and the process is continued for each subset of TARs.Let [TARThe FPRs above are computed using the annotation. This procedure is then repeated, except using the matched RNA-Seq TARs as the gold standard set, leading to an alternative marginal FPR for each TAR. The difference between the two marginal FPRs is notated ΔFPR and can be used as an adjustment for TARs with the corresponding rank score.AA and DK led the analysis and prepared the bulk of the manuscript. AS and LH performed the nearest-neighbor and exon boundary analyses. AS and JR designed the rank score and marginal FPR calculations. RS assisted in defining the TAR calling and differential expression algorithms as well as generating the pseudogene set. LWH and RHW led the RNA-Seq experimental efforts, and LWH conducted the analysis to provide RNA-Seq signals. VR led the tiling array experimental efforts. JR, RHW, and MG oversaw the project and provided general guidance. All authors reviewed the manuscript.Supplementary materialDetailed descriptions of experimental and analysis methods, and additional results.Click here for fileBlack list regions. List of regions deemed unreliable, for the tiling array used in this work, based on nearest neighbor analysis.Click here for file


Modern high-throughput measurement technologies such as DNA microarrays and next generation sequencers produce extensive datasets. With large datasets the emphasis has been moving from traditional statistical tests to new data mining methods that are capable of detecting complex patterns, such as clusters, regulatory networks, or time series periodicity. Study of periodic gene expression is an interesting research question that also is a good example of challenges involved in the analysis of high-throughput data in general. Unlike for classical statistical tests, the distribution of test statistic for data mining methods cannot be derived analytically.We describe the randomization based approach to significance testing, and show how it can be applied to detect periodically expressed genes. We present four randomization methods, three of which have previously been used for gene cycle data. We propose a new method for testing significance of periodicity in gene expression short time series data, such as from gene cycle and circadian clock studies. We argue that the underlying assumptions behind existing significance testing approaches are problematic and some of them unrealistic. We analyze the theoretical properties of the existing and proposed methods, showing how our method can be robustly used to detect genes with exceptionally high periodicity. We also demonstrate the large differences in the number of significant results depending on the chosen randomization methods and parameters of the testing framework.By reanalyzing gene cycle data from various sources, we show how previous estimates on the number of gene cycle controlled genes are not supported by the data. Our randomization approach combined with widely adopted Benjamini-Hochberg multiple testing method yields better predictive power and produces more accurate null distributions than previous methods.Existing methods for testing significance of periodic gene expression patterns are simplistic and optimistic. Our testing framework allows strict levels of statistical significance with more realistic underlying assumptions, without losing predictive power. As DNA microarrays have now become mainstream and new high-throughput methods are rapidly being adopted, we argue that not only there will be need for data mining methods capable of coping with immense datasets, but there will also be need for solid methods for significance testing. Randomization methods are techniques for significance testing that are based on generating data that shares some of the same properties with the real data, but lacks the structure of interest. For example, if we are interested in predicting a target variable on the basis of some explanatory variables, then we can randomize the target variable to remove any real connection between the explanatory and target variables. The prediction method is run on randomized data, and the accuracy of the resulting classifier is noted. This is repeated for, say, 10000 randomizations, and the accuracy of the classifier obtained on real data is compared with the results on randomized data to obtain an empirical p-value. See ,,expectedp-values, but instead use the raw periodicity scores produced by the randomization methods. Equation (3) below defines the Futschik-Herzel empirical false discovery rate FH(f) for threshold f. The equation is based on periodicity scores in randomized data F and on periodicity scores in the original data F(g). Count of scores greater than or equal to f are calculated over all genes g ∈ {1, ..., n} and randomized matrices r ∈ {1, ..., R}, where R is the number of randomized matrices produced.Futschik and Herzel use an alternative approach to significance testing . They doP is true, and zero otherwise. By using Equation (3) it is trivial to select f so that FH(f) = q, where q is the desired false discovery rate. This approach is similar to threshold selection in [f, but instead select q and derive the value of f by using the equation. They do not discuss the statistical justifications of the method, or specify the underlying assumptions or the exact nature of the provided control for "empirical false discovery rate".Here ction in ,16, but p-values provide a way of using a null distribution of gene expression matrices to assess the significance of periodic signals in the original measured data. To produce null distributions of expression matrices, we describe four techniques for generating randomized n × m matrices E'; that share certain characteristics with the gene expression matrix E. The methods either use the original dataset as the data to be randomized or calculate parameters of a data generating model from the original dataset [Empirical  dataset . The metE. The result E, but does not maintain, e.g., the row and column distributions.Randomized matrix et al., where they argue that an upper bound for false positive count can be achieved with permutation randomization [It is obvious that permutations are a drastic randomization method and do not preserve much of the structure in the dataset. They are used in periodicity studies, as in the seminal paper by Spellman mization .E. The result E.Randomized matrix et al. argue that an upper bound for false positive count can be achieved with randomization by permuting within rows [P and R. Our results indicate that both of the methods should be considered optimistic: They produce only loose lower bounds for the number of false positives.Spellman hin rows , so the autoregressive AR(1) process for which the value of expression at time t depends linearly on expression at time t - 1 up to a normally distributed random variable [t X= a1 · tX-1 + tZ, where t Xis the time dependent random variable, i.e., gene expression, while a1 is the first order autocorrelation of t Xand t Zis the independent normally distributed random variable.Cell cycle time series data often exhibits autocorrelation due to the procedure employed to bring cells to a synchronized state in the beginning of the time series. In a method proposed by Futschik and Herzel gene expression is modeled as an variable . Specifit Zis a random variable, we can easily calculate a1 and the variance of t Zfrom the original data matrix E and use them to generate a random matrix tZ. Yang and Su have suggested using multiple methods simultaneously for estimating autocorrelation coefficients of circadian gene expression data [As ion data . In our ion data .tZ. With datasets of this study, we typically see variances less than the original in randomized data. When variance is normalized after randomization, the data is effectively scaled up on average. As Fourier score is used for periodicity detection, also the periodicity scores are scaled up. Unlike the other methods, One aspect of the Futschik and Herzel approach is that variance of the whole row is normalized, but the data generating model preserves only variance of the normally distributed random variable -package ,32.i.e., each suffix is randomly assigned to follow a prefix.We propose a new randomization method that takes a gene cycle and splits it into two equal sized parts referred to as the prefix and the suffix. Then we simply randomize the connection between prefixes and suffixes, g in the randomized matrix m time points is generated asMore formally, assuming there is only one gene cycle, a row h is chosen at random without replacement from {1, ..., m}. In the formula, we combine rows g and h in the original data to produce a randomized row. When there are multiple gene cycles, we split each cycle into prefixes and suffixes. Then again, we combine the prefixes of row g with suffixes of row h to create a new row. Each row in the randomized data contains values from two rows in the original data, except for the rare case that g = h, when the randomized row is actually equal to a one row in the original data.Here index We motivate our randomization method by interpreting the methods as null models that produce random samples from a null distribution. What we would like to get are samples from a situation where the same genes are measured under the same circumstances, but without a treatment that is assumed to induce cyclic expression.P can be interpreted as sampling each expression value from a global null distribution. The strength of the method is that there are a large amount of individual values in a microarray experiment and thus we have a robust sampling distribution. However by sampling each value independently we completely eliminate dependencies between values of a single gene. In time series data consecutive values are naturally strongly dependent and a realistic null model cannot be produced without taking these dependencies into account.The randomization method R improves on P in this aspect: now we sample each gene from its own distribution, preserving differences between average levels of expression of the genes. Unfortunately, there are very few values per gene and building a sampling distribution from, say, 20 values is not robust. Besides that, the distribution of values for induced and non-induced genes can be very different, and permuting expression values of an induced gene does not necessarily produce an expression profile that a typical non-induced gene might have. Dependencies between values of consecutive time points for a single gene are not taken into consideration with R, either.In expression data there are differences between the average level of expression of different genes, but complete permutation makes data uniform. The method A takes into account some of the patterns by adding AR(1) autocorrelation to the model. AR(1) autocorrelation can capture certain stress response patterns that the treatment induces and are argued to be a major feature of gene cycle data [A does not, however, preserve the average level of expression for each gene, but the level is mostly decided by the rather arbitrary first measurement value. Also it is not known how well stress response is modelled by AR(1) autocorrelation: The model can only capture a linear pattern that covers the whole timeseries, but the actual stress response pattern might be non-linear and only cover the first timepoints of the dataset. An open question is also what other patterns of non-induced expression should be taken into account.It is possible to sample genes by taking their average level of expression and adding normally distributed noise, with the same variance than in original data. This approach assumes that non-induced genes produce constant level of expression, excluding the noise. However processes not related to cyclicity can produce varying expression patterns that are not captured in this model. The randomization method cle data . Formulap-values is to assume the absence of the effect or pattern of interest (null hypothesis), to gather evidence to invalidate null hypothesis in favor of an alternative hypothesis and to assess the probability that the evidence was a product of luck. As null hypothesis is the starting assumption and we are producing evidence against it, data should be analyzed in a conservative way. If there is a bias in our methods, it should be towards conservative interpretation. A null model that exaggerates the randomness of the null distribution clearly contradicts with the statistical framework.As can be seen, producing a general null model of gene expression is very difficult. And what is more important, development of null models has been progressing from simple to more advanced. Each new feature that is taken into account can make the model more realistic, but we do not know when the model is realistic enough. The idea of significance testing with S maintains the original value distribution, value distributions of each column, and also the correlations between columns inside prefixes and inside suffixes. The method is based on the assumption that many of the genes are non-induced and hence the distribution of each column is close to the distribution of a non-treated sample. We take into consideration that data can contain global patterns such as stress response. Values are not mixed across timepoints and hence global stress response patterns are not removed, but experiment wide cyclic structures are. The method can produce a small amount of cyclic genes to the randomized data by combining two cyclic genes, with matching phases, in the original data (or a cyclic gene with itself), which produces a slight conservative bias.Method S does not take into account the known fact that there are correlations between genes. However our model is a conservative null model for gene periodicity detection: Correlations do not play a role when the periodicity of each gene is measured independently. If we would be assessing the significance of, say, clustering, then correlations should be taken into account.We point out that method This section analyzes the theoretical properties of the randomization methods. Reading this section is not necessary for understanding the rest of the article.To better understand results produced by different randomization methods, we investigate the theoretical properties of the periodicity scores produced by them. This analysis shows clearly how the commonly used randomization methods cannot take the time component or the sampling frequency of periodicity data into account even though the methods are used for analyzing exactly the time-based structure of the data.t = 0,1,2, ..., m - 1; let k c= . Denoting by E = (gte)n × m a dataset, and by g Ethe gth row of E, let g · ck Ebe the dot product of the vectors g Eand kc. Consider the slightly modified kth periodicity score for the gth row g Eof E defined byA perfect cyclical gene has values k cis We use this modified version of the periodicity score to make the results more easily understandable while still retaining all the important characteristics of the metric. The periodicity score of By straightforward calculations based on these definitions it is possible to analyze the periodicity scores given by each of the randomization methods. In the following we express the expectations and variances of periodicity score distributions. Details for the results can be found in the Additional file P method that permutes all the data elements at random the expected value and variance of the periodicity score for any gene in the randomized data areFor the E's elements raised to power 2 and 4 respectively. The statistics of the values generated by the P method do not have any connection to the time component of the data, but only to the value distribution of data. This can be restated so that the method simply calculates the whole signal energy , with noise included, contained in the data set. However only the fluctuation corresponding to some cyclical structure should be accounted.In the equations m grows, both the statistics become smaller and smaller. This is true, however, only if the data contains no cyclical components. In crude terms, any actual cyclical structure in data supports the periodicity score and prevents it from sinking significantly lower than some given level as m grows. Therefore, as seen in Figure p-values for data with cyclical structure move consistently toward zero as m grows.It can also be seen that as the number of samples in data R method that randomizes each row separately we reach similar results. The statistics of the periodicity scores of the randomized data areFor the g.) Here again, the statistics do not depend on anything except the value distribution of the data without any regard to the order or placement of the elements. These results on the theoretical properties of the P and R methods make them questionable if any consideration should be given to the actual dataset or the purpose of the experiments.m 1 × are generated as values of a martingale: X0 = Z0, X1 = αX0 + Z1, X2 = αX1 + Z2, ..., mX-1 = αXm-2 + Zm-1. We require that |α| < 1 and the t Zare i.i.d. samples from some distribution i.e., as a sum of different cyclical sine signals In the randomization method of  the valuk αgive the amount of cyclicity the signal has for each different frequency. The expectation and variance for periodicity scores of data samples generated with this autocorrelation method of [Here the values ethod of  can be sf(m).In the equations the notation α. To further simplify the equations one can note that for a strongly periodic gene its α is practically equal to for k.The periodicity scores of the generated samples depend on m the same way as for methods P and R, decreasing as the number of samples grows, because for higher sampling frequencies the data fluctuates rather slowly (with α ≈ 1) and the high frequency noise is more easily distinguished. This also exemplifies how A measures the significance of gene cyclicity against a background of pure noise. Figure p-values with increasing sample counts.The statistics depend on gk αthe kth Fourier coefficient of the gth gene, i.e., the gth gene has Fourier decomposition We now make an assumption that cyclical signals are cyclical in all parts of the signal, without turning into pure noise at any point. If we denote by k. On the contrary to the case of methods P, R and A, the statistics do not diminish with longer sample sizes or higher sampling frequencies. This happens, because the S method focuses on detecting genes that are exceptionally cyclical in the context of all the genes in the given dataset, not against an absolute background of pure noise as the other methods do.These values indicate that the cyclicity of the genes on each of the frequencies gets randomized separately, since the statistics depend only on the current value of S is the only method using the background distribution of the analyzed genes instead of a pure noise background. We also showed how the behavior of P, R and A changes with an increase in the sample count. Methods P, R and A suffer from high noise levels in data whereas S dissects the data and considers each frequency separately, which is also how the actual test score is calculated. Finally the P and R methods cannot even be seen as actual periodicity tests since they ignore any periodicity in the data and redistribute only the signal energy of the data, no matter whether it arises from periodicity, other meaningful patterns or from random noise.We illustrated how S has sound theoretical foundation for assessing the significance of periodicity. In this section we will back analytical results with experiments. We will also compare two different multiple hypothesis testing approaches and discuss their differences.In the previous section we analyzed the four different randomization methods discussed in this manuscript. The analysis showed that the method http://www.cs.helsinki.fi/u/akallio/periodicity/.All methods were implemented with Matlab and experiments were run with a standard workstation PC. Matlab code is available from a companion web site at Results in rest of this section are based on gene expression data from well-known yeast gene cycle studies ,16. The LPHA2C has been produced by bringing the whole yeast cell culture to a hypothetically synchronized state using alpha-factor arrest. The dataset ALPHA1C is a version of ALPHA2C with only the first cycle included (restricted to first 9 time points), mimicking a very low quality dataset. Datasets CDC15 and CDC28 are based on temperature arrest of cdc15 and cdc28 temperature sensitive mutants. Contrary to the other yeast datasets, the dataset ELU is based on elutriation where instead of external treatment a subset of hypothetically synchronized cells was collected from the population. The dataset MALARIA is based on a synchronized in vitro culture.The dataset Aadjc, where 0 <adj c≤ 2c and c is the hypothetical number of cycles. We iterate with 0.01 increments and choose adj cso that it maximizes the sum of periodicity scores for all genes. Chosen values of adj care reported in column Detected cycles of Table We could use the hypothetical numbers of gene cycles reported for each of the datasets, but the accuracy of periodicity calculation can be improved by detecting the number of cycles from the dataset . For thas δso that a fraction of s = 0.05, s = 0.10, or s = 0.20 of the genes have scores higher than sδ.As our initial test of the methods, we studied the behavior of the periodicity score on randomized data. We computed the scores for each row in each dataset, and selected for each dataset three thresholds s δshould be about s. On the other hand, if the data has strong periodic structure, the distribution of the scores of the randomized rows should be clearly different from the distribution of the scores in the real data, and the fraction of genes with scores above s δshould be less than s. The ratio of randomized genes labeled as periodic is reported in Table For each dataset and each method we then generated 100 randomized datasets, and computed the fraction of randomized data rows that have periodicity scores higher than the threshold. If there is no periodic structure in the data, the distribution of the periodicity scores in the randomized data should be about the same as in the original data, and hence the fraction of randomized rows with scores above P, R, and A consider that the fraction of genes in the randomized data with high periodicity scores is much lower than in the real data, for all datasets. That is, these methods would indicate that all datasets have significant periodic structure.Table S are different for different datasets: for datasets ALPHA1C ELU, and MALARIA the fraction of genes with high scores is the same in the randomized data and in the real data, while for ALPHA2C, CDC15, and CDC28 the randomized datasets have fewer genes with high scores. Thus method S indicates that only in the datasets ALPHA2C, CDC15, and CDC28 there is clear evidence of periodicity. These assessments produced by method S are consistent with the previous results. ELU was found to be low quality already by the original authors [LPHA1C was artificially constructed to be an example of a dataset with too few timepoints for reliable periodicity detection. The dataset MALARIA can be seen to be a special case, and it is discussed next.On the other hand, results for method ALARIA dataset is an example of an atypical microarray dataset and it is not a good fit for randomization methods presented in this study. This is because we are measuring here the count of genes that have significantly higher periodicity scores than given by the null model. MALARIA dataset contains mostly periodic genes, so it does not contain much information about gene expression of other types than of the synchronized periodic pattern. Simple permutations P and R do not preserve much of the structure of the original dataset and compare strongly periodic original dataset to noise mostly, resulting in no periodic genes to be detected in the randomized data. Method A is based on the assumption of Gaussian expression with AR(1) autocorrelation, and does not preserve much of the structure in the original data either. As was previously shown in the analysis, method S is different in this regard. It uses the dataset to build a model of all expression patterns without other major assumptions about the data, and as the dataset is very biased, so is the model. S reports that there is no periodicity in the data that cannot be explained by chance alone.The MS can be used to detect genes that are exceptionally periodic. However, the assumption about large numbers of non-induced genes is not true for MALARIA. For that reason we do not consider any of the results for MALARIA realistic, but hold the view that the recommended solution would have been to conduct more measurements to get data on typical expression patterns in the case of no treatment. If such data is not available, then additional assumptions have to be done to roughly approximate the null distribution of the periodicity score. Assumptions behind the method A are not realistic in this case. Level of autocorrelation and variance of Gaussian distribution are calculated from data, which now contains mostly induced genes. Therefore calculated parameters are not necessarily realistic for non-induced genes.The advantage of microarray experiments is that if we are studying the behavior of maybe some dozens of genes, the dataset will contain hundredfold of non-induced genes to compare to . As was p-values and the method of Futschik and Herzel on periodicity scores. For validation we use a benchmark gene list from [et al. and Cho et al.. The first benchmark set comprises of 113 genes that were confirmed to be periodic in small scale laboratory experiments. The other two sets are based on less decisive methods, so we restrict our comparison to the first set only.To estimate the number of periodic genes in the datasets, we use the above randomization methods to produce randomized samples and apply both the Benjamini-Hochberg procedure on empirical ist from  that preFor comparing the methods we use the positive predictive value (PPV), which was also used in . For a gTP is the count of true positives, i.e., the number of genes that the method considers periodic and that are found from the small scale experiment benchmark set. FP is the count of false positives, i.e., the number genes that the method reports as periodic, but which do not occur in the benchmark set. It is important to bear in mind that the benchmark set does not contain all periodic genes, so the absolute value of PPV has little meaning, but the relative difference between values indicates a difference between the quality of the two candidate gene lists. Also for this reason only one variable should be changed at a time when doing the comparison. The PPV and total count of positives is reported in Table PPV is not robust when the number of positives is small, and undefined when there are no positives, we omit the value when there are less than 10 positives.Here PPV values followed by the corresponding number of significant periodic genes in parentheses. We first compare our results to earlier results by Spellman et al. and Futschik and Herzel. Spellman et al. reported 400 to 1000 periodic genes, based on randomization methods P and R. Our results for the corresponding datasets are in the range of 300 to 450 with commonly used Benjamini-Hochberg FDR 0.05, i.e., much lower. As discussed earlier, Spellman et al. used a manually tuned threshold for periodicity scores. Their threshold does not account for the number of genes that are tested. Our framework accounts for multiple testing and produces smaller numbers of positives. If multiple hypothesis correction is omitted, our analysis pipeline with methods P and R produces approximately 1000 significant results, matching the results by Spellman et al..Table et al.. For the Cdc28 dataset they are the same and hence our results are also practically identical to the Futschik and Herzel study. The small differences are due to different KNN-imputation implementations and slight difference in wavelenght that is not visible due to rounding. For Cdc15 the detected wavelength differs and that yields differences especially for the stricter thresholds; At FDR 0.10 they are already quite close (132 here versus 126 in Futschik and Herzel study).When comparing results with those of Futschik and Herzel, we need to remember that there are differences in wavelengths that are used for Fourier calculations. We detected the number of gene cycles (or in other words wavelength) from the data, whereas Futschik and Herzel used the wavelength from the original study by Spellman PPV values, while the more relaxed FDR 0.10 is already too forgiving, at least when compared to the validation data. The FDR threshold 0.01 is very strict for this kind of study and that can be seen from the results also. There are no differences between the FDR control methods: The Futschik and Herzel method produces practically identical results to the Benjamini-Hochberg method.When looking at FDR control parameters, the typical FDR threshold 0.05 produces the best LPHA1C is identified by both A and S methods, but not by the simpler permutation methods. That same deficiency is even more exaggerated for the ELU dataset that was found bad in the previous section and also by Spellman et al. [P and R report over hundred significant genes already at FDR 0.01 and scale up to thousands at higher FDR levels. Judging by this, these two methods cannot be recommended even for simple sanity checking of the data.The poor quality of the artificially constructed bad dataset An et al. : The simLPHA2C and CDC15, both A and S give more conservative results that have better PPV values. The method S is systematically more conservative than A, but also gives a better PPV value in every case.For the good quality datasets ADC15. The vertical line shows threshold of significance as decided by the Benjamini-Hochberg method at FDR 0.05. The method S produces a null distribution that very closely follows the distribution in original data, except naturally for the bump at the high end, i.e., where alternative distribution of periodic genes mixes with the non-periodic ones. Method A produces a null distribution that is biased towards the low periodicity scores, as the peak of the distribution is too far in the low end. In the score range 6 to 10 the null distribution drops faster than the original. This probably cannot be explained by the periodic distribution mixing with the non-periodic distribution, because it seems highly unlikely that non-trivial portions of the periodic distribution extends to that range. As the null distribution is systematically off the mark for the low and mid ranges, it does not seem plausible that A could be used to reliably separate the periodic and non-periodic distributions in the high range. A threshold decision based on the method S is the one that is supported by Figure We can examine these differences closer by looking at null distribution of periodicity scores produced by the two methods, when compared to original distribution from data. Figure A and S disagree on the CDC28 dataset. Method S asserts only two genes to be significant even on the high FDR levels, while the method A gives more optimistic estimates. The case is analogous to dataset ALPHA2C, where both give zero significants at FDR 0.01. The only difference is that CDC28 is given more conservative assessment, to the point that S does not accept more than two positives at FDR 0.10. However at FDR 0.20 also method S would have already given 90 positives and we could more easily observe similar trend than in ALPHA2C. To examine this further, we again plot the distributions produced by the two methods in Figure DC15, the bump of periodic genes is not pronounced at all and it seems that the area where significant scores are located is populated mostly by genes from the non-periodic distribution. In this dataset method S still produces a distribution that more closely follows the original one, though the difference to method A is not as large. The peak of the distribution is somewhat too far in the low end for A, and perhaps slightly for S also. When looking at the null distribution produced by S, it is easy to understand why only two genes were declared significant. The two distributions of induced and non-induced genes are so mixed that reliably separating them with a threshold does not seem possible. Method A paints a more optimistic picture and admittedly it is not far off the mark in the distribution graphs, but still an optimistic bias is visible. It casts doubts on the much larger numbers of positives reported by A.The methods P and R give overly optimistic results, and secondly, how also the more recent and more advanced autocorrelation method A has optimistic tendencies and better results can be obtained by the more conservative method S. The two multiple testing control methods had identical performance. Hence we can recommend the Benjamini-Hochberg method, as it is established and has a well documented theoretical basis, unlike the method used by Futschik and Herzel.As a summary, we demonstrated firstly how previously used permutation methods i.e., what type of patterns we expect to see by chance. Typically most of the attention is given to careful formulation of the pattern we are looking for, such as formulation of a periodicity score. However, equal amount of attention should be given to formulation of the null model, i.e., defining what we expect to see when there are no patterns that we are interested in. Construction of traditional parametric statistical models for periodicity studies or any other complex and novel area of bioinformatics is very difficult. Randomization methods are an viable alternative for analytical significance testing.Microarray data is commonly noisy and can be plagued by effects not related to the biological question under study. It is therefore crucial to focus on null models of gene expression, However, simple randomization tests, as simple permutations applied to gene cycle data, do not produce meaningful information. Comparing any real data with noise will assess patterns in the data significant. So it is more important for a randomization test to retain the realistic general structure of the data than to remove all structure under study. Otherwise the result will be too optimistic.Originally it was estimated that for 400 to 1000 genes the periodic signal is strong enough to be considered statistically significant . In our To stress the point, the seminal results of gene periodicity studies were reported in 1998, and now more than 10 years later studies are still made that change the interpretation of those results. Biosciences are a rapidly developing area of research with new data and instruments coming out at a fast pace, and therefore more attention should be placed on significance testing.When gene expression data is mined for complex patterns, little if no effort is made to test results for statistical significance. With randomization, significance testing can be extended from simple classical statistical tests to complex pattern mining. Existing methods for testing significance of periodic gene expression patterns were found too simplistic and optimistic. We introduced significance testing framework that accounts for multiple testing and allows simple yet more realistic null models to be crafted with randomization algorithms. As a result, a much smaller number of genes showed significant periodicity.As DNA microarrays have now become mainstream and new high-throughput methods are rapidly being adopted, we argue that not only there will be need for data mining methods capable of coping with immense datasets, but there will also be need for a new approach to significance testing. We need methods that do not require demanding mathematical analytics and that can be easily modified and adopted to different situations. Randomization algorithms are intuitive and can be readily constructed by the growing numbers of computer literate bioresearchers. Future of biosciences seems to be characterized by large developments in data production capability and that development must be met with appropriate tools to guard against false interpretations.AK implemented the method and carried out experiments. NV carried out the mathematical analysis. AK and NV drafted the manuscript. The design and development of the methods was done by all authors. All authors have read and approved the final manuscript.Appendix: Periodicity score calculations. Additional file Click here for file


DESeq, as an R/Bioconductor package.High-throughput sequencing assays such as RNA-Seq, ChIP-Seq or barcode counting provide quantitative readouts in the form of count data. To infer differential signal in such data correctly and with good statistical power, estimation of data variability throughout the dynamic range and a suitable error model are required. We propose a method based on the negative binomial distribution, with variance and mean linked by local regression and present an implementation,  The PedgeR package for analysis of SAGE and RNA-Seq [μ and variance σ2. However, the number of replicates in data sets of interest is often too small to estimate both parameters, mean and variance, reliably for each gene. For edgeR, Robinson and Smyth assumed [σ2 = μ + αμ2, with a single proportionality constant α that is the same throughout the experiment and that can be estimated from the data. Hence, only one parameter needs to be estimated for each gene, allowing application to experiments with small numbers of replicates.To address this so-called overdispersion problem, it has been proposed to model count data with negative binomial (NB) distributions , and thi RNA-Seq ,10. The  assumed  that meaModel). As a result, more balanced selection of differentially expressed genes throughout the dynamic range of the data can be obtained . We demonstrate the method by applying it to four data sets (Section Applications) and discuss how it compares to alternative approaches (Section Conclusions).In this paper, we extend this model by allowing more general, data-driven relationships of variance and mean, provide an effective algorithm for fitting the model to data, and show that it provides better fits  distribution,We assume that the number of reads in sample ij μand the variance ij Kare non-negative integers. The probabilities of the distribution are given in Supplementary Note A.  (where ρ(j) is the experimental condition of sample j) and a size factor sj,First, the mean parameter qi,ρ(j) is proportional to the expectation value of the true (but unknown) concentration of fragments from gene i under condition ρ(j). The size factor sj represents the coverage, or sampling depth, of library j, and we will use the term common scale for quantities, such as qi, ρ(j), that are adjusted for coverage by dividing by sj.shot noise term and a raw variance term,Second, the variance vi, ρ is a smooth function of qi, ρ,Third, we assume that the per-gene raw variance parameter i from just the data available for this gene. This assumption allows us to pool the data from genes with similar expression strength for the purpose of variance estimation.This assumption is needed because the number of replicates is typically too low to get a precise estimate of the variance for gene i in sample j is proportional to a random variable ijR, such that the rate that fragments from gene i are sequenced is jrijs. For each gene i and all samples j of condition ρ, the ij Rare i.i.d. with mean iρ qand variance iρv. Thus, the count value ijK, conditioned on ij R= ijr, is Poisson distributed with rate jrijs. The marginal distribution of ij K- when allowing for variation in ij R- has the mean ij μand  the variance given in Equation (3). Furthermore, if the higher moments of ij Rare modeled according to a gamma distribution, the marginal distribution of ij Kis NB  is motivated by the following hierarchical model: We assume that the actual concentration of fragments from gene example, , Sectionn × m table of counts, ijk, where i = 1,..., n indexes the genes, and j = 1,..., m indexes the samples. The model has three sets of parameters:We now describe how the model can be fitted to data. The data are an m size factors js; the expectation values of all counts from sample j are proportional to js.(i) ρ, n expression strength parameters iρq; they reflect the expected abundance of fragments from gene i under condition ρ, that is, expectation values of counts for gene i are proportional to iρq.(ii) for each experimental condition ρ v: ℝ+ → ℝ+; for each condition ρ, ρ vmodels the dependence of the raw variance iρ von the expected mean iρq.(iii) The smooth functions j sis to render counts from different samples, which may have been sequenced to different depths, comparable. Hence, the ratios (ijK)/(ij' K) of expected counts for the same gene i in different samples j and j' should be equal to the size ratio js/j' sif gene i is not differentially expressed or samples j and j' are replicates. The total number of reads, Σi ijk, may seem to be a good measure of sequencing depth and hence a reasonable choice for js. Experience with real data, however, shows this not always to be the case, because a few highly and differentially expressed genes may have strong influence on the total read count, causing the ratio of total read counts not to be a good estimate for the ratio of expected counts.The purpose of the size factors Hence, to estimate the size factors, we take the median of the ratios of observed counts. Generalizing the procedure just outlined to the case of more than two samples, we use:j-th sample's counts to those of the pseudo-reference. .In Supplementary Note B in Additional file ρm, as is typically the case in applications, the values iρ ware highly variable, and iρ w- iρ zwould not be a useful variance estimator for statistical inference. Instead, we use local regression [ρw(q), withHowever, for small numbers of replicates, gression  on the gas our estimate for the raw variance.iρ wis a sum of squared random variables, and the residuals locfit package [Some attention is needed to avoid estimation biases in the local regression. ferences , Chapterferences , Section package .A mreplicate samples for biological condition A and B msamples for condition B. For each gene i, we would like to weigh the evidence in the data for differential expression of that gene between the two conditions. In particular, we would like to test the null hypothesis iA q= iBq, where iA qis the expression strength parameter for the samples of condition A, and qiB for condition B. To this end, we define, as test statistic, the total counts in each condition,Suppose that we have iS K= iA K+ iBK. From the error model described in the previous Section, we show below that - under the null hypothesis - we can compute the probabilities of the events iA K= a and iB K= b for any pair of numbers a and b. We denote this probability by p. The P value of a pair of observed count sums  is then the sum of all probabilities less or equal to p, given that the overall sum is iSk:and their overall sum a and b in the above sums take the values 0,..., kiS. The approach presented so far follows that of Robinson and Smyth [The variables nd Smyth  and is and Smyth , ChapterComputation of p. First, assume that, under the null hypothesis, counts from different samples are independent. Then, p = Pr(KiA = a) Pr(KiB = b). The problem thus is computing the probability of the event KiA = a, and, analogously, of KiB = b. The random variable KiA is the sum of mAKij. To this end, we first compute the pooled mean estimate from the counts of both conditions,NB-distributed random variables. We approximate its distribution by a NB distribution whose parameters we obtain from those of the qiA = qiB. The summed mean and variance for condition A arewhich accounts for the fact that the null hypothesis stipulates that KiA can be determined from KiB are obtained analogously.Supplementary Note C in Additional file Supplementary Note D in Additional file We present results based on the following data sets:B. Wilczynski, Y.-H. Liu, N. Delhomme and E. Furlong have conducted RNA-Seq experiments in fly embryos and kindly shared part of their data with us ahead of publication. In each sample of this data set, a gene was engineered to be over-expressed, and we compare two biological replicates each of two such conditions, in the following denoted as 'A' and 'B'.et al. [Engström et al.  performeet al.  for tisset al. [Saccharomyces cerevisiae cultures. They tested two library preparation protocols, dT and RH, and obtained three sequencing runs for each protocol, such that for the first run of each protocol, they had one further technical replicate  and one further biological replicate (different culture).Nagalakshmi et al.  performeet al. [Kasowski et al.  comparedwiρ (Equation (7)) plotted against the means A in the fly RNA-Seq data. Also shown is the local regression fit ρw(q) and the shot noise We start by demonstrating the variance estimation. Figure The many data points in Figure DESeq maintains control of type-I error, we contrasted one of the replicates for condition A in the fly data against the other one, using for both samples the variance function estimated from the two replicates. Figure P values obtained from this comparison. To control type-I error, the proportion of P values below a threshold α has to be ≤ α, that is, the ECDF curve (blue line) should not get above the diagonal (gray line). As the figure indicates, type-I error is controlled by edgeR and DESeq, but not by a Poisson-based χ2 test. The latter underestimates the variability of the data and would thus make many false positive rejections. In addition to this evaluation on real data, we also verified DESeq's type-I error control on simulated data that were generated from the error model described above; see Supplementary Note G in Additional file A samples against the two B samples. Using the procedure described in the previous Section, we computed a P value for each gene. Figure P values. 12% of the P values were below 5%. Adjustment for multiple-testing with the procedure of Benjamini and Hochberg [In order to verify that Hochberg  yielded edgeR . edgeR's single-value dispersion estimate of the variance is lower than that of DESeq for weakly expressed genes and higher for strongly expressed genes. As a consequence, as we have seen in Figure 2edgeR is anti-conservative for lowly expressed genes. However, it compensates for this by being more conservative with strongly expressed genes, so that, on average, type-I error control is maintained.Going back to Figure edgeR. While overall the sensitivity of both methods seemed comparable , DESeq produced results which were more balanced over the dynamic range.Nevertheless, in a test between different conditions, this behavior can result in a bias in the list of discoveries; for the present data, as Figure Similar results were obtained with the neural stem cell data, a data set with a different biological background and different noise characteristics  were recovered (with two new hits). These observations are explained by the fact that in the neural cell data, the variability between replicates was not much smaller than between conditions, making the latter a usable surrogate for the former. On the other hand, for the fly data, the variability between replicates was much smaller than between the conditions, indicating that the replication provided important and otherwise not available information on the experimental variation in the data .w(q) estimated by DESeq, a VST is given byGiven a variance-mean dependence, a variance-stabilizing transformation (VST) is a monotonous mapping such that for the transformed values, the variance is (approximately) independent of the mean. Using the variance-mean dependence τ to the common-scale count data, ijk/js, yields values whose variances are approximately the same throughout the dynamic range. One application of VST is sample clustering, as in Figure Applying the transformation DESeq can also be used to analyze comparative ChIP-Seq assays. Kasowski et al. [DESeq. No region was significant at 10% FDR using Benjamini-Hochberg adjustment. Differential occupation was found, however, when contrasting the two individuals, with 4,460 of 19,028 regions significant when only two replicates each were used and 8,442 when four replicates were used (upper right two panels).i et al.  analyzedet al. fitted generalized linear models (GLMs) of the Poisson family. This  bias. Kasowski et al. addressed this and adjusted for the bias by using additional criteria for calling differential occupation.Using an alternative approach, Kasowski Why is it necessary to develop new statistical methodology for sequence count data? If large numbers of replicates were available, questions of data distribution could be avoided by using non-parametric methods, such as rank-based or permutation tests. However, it is desirable (and possible) to consider experiments with smaller numbers of replicates per condition. In order to compare an observed difference with an expected random variation, we can improve our picture of the latter in two ways: first, we can use distribution families, such as normal, Poisson and negative binomial distributions, in order to determine the higher moments, and hence the tail behavior, of statistics for differential expression, based on observed low order moments such as mean and variance. Second, we can share information, for instance, distributional parameters, between genes, based on the notion that data from different genes follow similar patterns of variability. Here, we have described an instance of such an approach, and we will now discuss the choices we have made.While for large counts, normal distributions might provide a good approximation of between-replicate variability, this is not the case for lower count values, whose discreteness and skewness mean that probability estimates computed from a normal approximation would be inadequate.et al. [technical reproducibility of RNA-Seq. They extracted total RNA from two tissue samples, one from the liver and one from the kidneys of the same individual. From each RNA sample they took seven aliquots, prepared a library from each aliquot according to the protocol recommended by Illumina and sampled each library on one lane of a Solexa genome analyzer. For each gene, they then calculated the variance of the seven counts from the same tissue sample and found very good agreement with the variance predicted by a Poisson model. In line with our arguments in Section Model, Poisson shot noise is the minimum amount of variation to expect in a counting process. Thus, Marioni et al. concluded that the technical reproducibility of RNA-Seq is excellent, and that the variation between technical replicates is close to the shot noise limit. From this vantage point, Marioni et al.  suggestFigure DESeq to obtain variance estimates for the data of Nagalakshmi et al. [This is confirmed by comparison of technical with biological replicates . In Figui et al. . The anai et al. ,20,22,23edgeR [DESeq owes its basic idea to edgeR, yet differs in several aspects.Consequently, it is preferable to use a model that allows for overdispersion. While for the Poisson distribution, variance and mean are equal, the negative binomial distribution is a generalization that allow for the variance to be larger. The most advanced of the published methods using this distribution is likely edgeR . DESeq oet al. [First, we discovered that the use of total read counts as estimates of sequencing depth, and hence for the adjustment of observed counts between samples ; in fact, edgeR's power increases when it is supplied with those size estimates instead. EdgeR addresses this problem by estimating a single common dispersion parameter. In our method, we make use of the possibility to estimate a more flexible, mean-dependent local regression. The amount of data available in typical experiments is large enough to allow for sufficiently precise local estimation of the dispersion. Over the large dynamic range that is typical for RNA-Seq, the raw SCV often appears to change noticeably, and taking this into account allows DESeq to avoid bias towards certain areas of the dynamic range in its differential-expression calls  and (3)), suggesting that quantile adjustment is an unnecessary complication. The price we pay for this is that we need to make the approximation that the sum of NB variables in Equation (10) is itself NB distributed. While it seems that neither the quantile adjustment nor our approximation pose reason for concern in practice, DESeq's approach is computationally faster and, perhaps, conceptually simpler.Third, we have suggested a simple and robust way of estimating the raw variance from the data. Robinson and Smyth  employedFourth, our approach provides useful diagnostics. Plots such as Supplementary Figure S3 in Additional file CountDataSet, which is derived from eSet, Bioconductor's standard data type for table-like data. The package provides high-level functions to perform analyses such as shown in Section Application with only a few commands, allowing researchers with little knowledge of R to use it. This is demonstrated with examples in the documentation provided with the package (the package vignette). Furthermore, lower-level functions are supplied for advanced users who wish to deviate from the standard work flow. A typical calculation, such as the analyses shown in Section Applications, takes a few minutes of time on a personal computer.We implemented the method as a package for the statistical environment R  and distDESeq. Readers wishing to examine them in detail will find a Sweave document with the commented R code of the analysis code as Additional file All the analyses presented here have been performed with DESeq is available as a Bioconductor package from the Bioconductor repository [pository  and frompository .ChIP-Seq: (high-throughput) sequencing of immunoprecipitated chromatin; ECDF: empirical cumulative distribution function; FDR: false-discovery rate; GLM: generalized linear model; RNA-Seq: (high-throughput) sequencing of RNA; SCV: squared coefficient of variation; NB: negative-binomial (distribution); VST: variance-stabilizing transformation.SA and WH developed the method and wrote the manuscript. SA implemented the method and performed the analyses.Supplement. Contains all Supplementary Notes and Supplementary Figures.Click here for fileSupplement II. PDF file presenting the source code of all the analyses presented in this paper, with comments, as a Sweave document.Click here for fileRaw data. Tarball containing the raw data for the presented analyses.Click here for file


Transcriptional data covering multiple platforms and species is collected and processed into a searchable platform independent expression database (SPIED). SPIED consists of over 100,000 expression fold profiles defined independently of control/treatment assignment and mapped to non-redundant gene lists. The database is thus searchable with query profiles defined over genes alone. The motivation behind SPIED is that transcriptional profiles can be quantitatively compared and ranked and thus serve as effective surrogates for comparing the underlying biological states across multiple experiments.Drug perturbation, cancer and neurodegenerative disease derived transcriptional profiles are shown to be effective descriptors of the underlying biology as they return related drugs and pathologies from SPIED. In the case of Alzheimer's disease there is high transcriptional overlap with other neurodegenerative conditions and rodent models of neurodegeneration and nerve injury. Combining the query signature with correlating profiles allows for the definition of a tight neurodegeneration signature that successfully highlights many neuroprotective drugs in the Broad connectivity map.Quantitative querying of expression data from across the totality of deposited experiments is an effective way of discovering connections between different biological systems and in particular that between drug action and biological disease state. Examples in cancer and neurodegenerative conditions validate the utility of SPIED. However, these analyses do not allow for a direct quantitative comparison between separate expression studies and therefore a lot of the information contained in the experiment is effectively lost.There is a wealth of deposited gene expression data available for downloading and/or online interrogation. For example, the NCBI gene expression omnibus (GEO)  hosts ovet al [et al [The idea that transcriptional change profiles can be directly compared to asses drug target specificity was demonstrated in yeast systems by Marton et al  and latel [et al . The conl [et al . In thisl [et al . The remIf the CMAP methodology is accepted as a useful discovery tool then it is natural to look for ways of extending it to incorporate expression data from a wider set of experiments. There are obvious advantages to having this kind of database, for example it will open up a large number of different samples and treatment conditions for direct interrogation. This was the idea behind GEM-TREND , where 2http://www.ncbi.nlm.nih.gov), consisting of 24,764 genes and these were matched to probe IDs by inspection of the given platform annotation files. The final form of SPIED consists of individual files for each chip platform and these files are formatted starting with a gene list followed by the sample ID and corresponding EF profiles. This format lends itself to rapid searching in an analogous fashion to FASTA formatted sequence databases. In contrast to the KS query score scheme, which requires generating random reference gene list data, we adopted a simple regression scoring scheme with corresponding statistic. Searches can be performed on a standard desktop PC and take ~10 minutes per query. Although, the present database consisting of expression data for over 100,000 samples from five platforms covering three species  is all from Affymetrix expression array chips, the methodology is truly platform independent and it is a straight forward matter to include data based on other array technologies. Other species and platform technologies will be added to SPIED in the future. For the present study Affymetrix was chosen because of the relatively large number of available samples. Further details are presented in the methods section below.In compiling the expression database SPIED we sought to loosen the restraints inherent in previous treatments and thereby open up a larger set of data for interrogation. In many expression series sets there is no clear control/treatment assignment or there could be multiple alternative reference profile definitions. To address this problem of generating fold change profiles without reference to a defined control, an effective fold (EF) has been introduced corresponding to the expression level relative to the experimental series average. In this way, data can be compiled automatically without the need for manual inspection. In cases where the experimental series consists of well defined multiple treatment and control samples the fold profiles are usually given by the ratio of the average treatment to average control values. In general this fold profile will have high positive correlation with the EF profiles from the treatment set and high negative correlations with the control set. In cases where there is no obvious way of separating samples into control and treatment sets, as with samples from multiple organ types or cell types, the EF representation can be viewed as a normalized expression value. In searching SPIED with a query profile one is not deriving any biological significance for non-correlating profiles as lack of correlation can be attributed to multiple factors such as bad experimental data or genuine lack of biological relevance. Rather significantly correlating or anti-correlating profiles are posited as having biological significance. The next objective was to reduce the expression profiles to non-redundant EF gene profiles by associating each gene with just one probe ID, so that the database can then be searched with gene set data alone. Here, for a given chip platform the distribution of each probe ID EF value across the totality of series was compiled and each gene was then assigned to the probe having the highest average fold magnitude. The gene names were unambiguously associated with the Entrez human gene list correlated expression change profiles. The objective is to see to what extent the CMAP transcriptional signatures correlate with transcriptional responses assimilated within our platform independent database of over 100,000 microarrays deposited by a very large number of groups to the public domain.r = 0.90 N = 3565, rapamycin v wotmannin r = 0.91 N = 1849, LY-294002 v wotmannin r = 0.88 N = 2217, where r is the Pearson correlation coefficient and N is the number of shared genes with significant fold values), see additional file The CMAP is well populated with drugs that target the same or different steps in the PI3K-mTOR signalling cascade. In this context the results for LY-294002 (PI3K inhibitor) (61 instances), rapamycin (mTOR antagonist) (44 instances) and wortmannin (PI3K/mTOR antagonist) (18 instances) showed a high degree of overlap  inhibitor profiles in the CMAP series. These are vorinistat, trichostatin A and valporic acid. In Table Next we consider profiles derived from disease states. For brevity we focus on two unrelated pathologies: cancer and neurodegeneration.p < 0.05, see additional file r = 0.94), see additional file p of 20 or more resistant samples in the top 25 correlations is less than 9 × 10-7. The K-S significance score can be calculated by counting the number of times a random rearrangement of the samples gives a better enrichment, we find p < 3 × 10-6. The enrichment plot is given in Figure p < 5 × 10-5. The K-S significance is p < 2 × 10-4. The enrichment for positive correlations with the corticosteroid resistance profile in the cancer cases is shown in Figure p < 2 × 10-6 and a K-S significance score of p < 3 × 10-7. The enrichment curve is shown in Figure The class of diseases with the most extensive repository of expression data is cancer and therefore a cancer disease profile search of SPIED will be an ideal testing ground for the methodology. The original CMAP disease application implicated mTOR inhibition as a target for imparting sensitivity to dexamethasone treatment resistant ALL . We searp < 10-8, based on the Fisher test as above). Pooling the samples from the different brain regions results in significant correlations for 5 out of the 6 brain regions, see Figure The analysis of gene expression changes associated with neurodegenerative disease has been hampered by the difficulty of extracting high quality RNA from post-mortem tissue ,31. One p < 6 × 10-3. However, when the different brain regions are considered separately, we get significant regression scores in each region. The results are tabulated in Table In addition to AD correlations we found high scoring correlations with samples derived from Huntington's disease (HD), Down's syndrome (DS), Parkinson's disease (PD) and bipolar disorder (BD) brains. In this sense the profile cannot be considered to distinguish AD pathology from other degenerative diseases. However, it is of interest to examine in greater detail these cross-disease similarities. In particular, the severe AD query had a high correlation with severe stage HD caudate nucleus (CN) samples. The HD study consisted of 404 samples split across two platforms (201 samples on GPL96 and 203 on GPL97) in three brain regions from control and HD individuals . The higr = 0.88; lateral substatia nigra (LSN) r = 0.77; medial substantia nigra (MSN) r = 0.82, see Table The PD correlation was with a study of 94 samples from three different regions of diseased and normal brains . Poolingr = 0.58. Interestingly, the correlation is higher with the moderate AD profile, with r = 0.68, see Table The chromosome 21 trisomy underlying DS leads to the development of many of the characteristics of AD pathology ,37. TherThe first transcriptional profiling of BD brains pointed to the down regulation of synaptic and mitochondrial proteins in the orbital frontal cortex . This syhttp://pepr.cnmcresearch.org) and the AD profile correlation emerges at 72 hours post injury, see Table Not surprisingly the high correlations are dominated by experiments on human samples. Perhaps of greater interest to the biologist are animal models of neurodegeneration. There has indeed been a debate as to the relevance of animal models of neurodegeneration to drug discovery, as age-related neurodegenerative conditions are rare in nonhumans . In thisIt might be thought that we are getting further away from the specific pathology, in this case AD, and losing transcriptional information that could be of use in the hunt for a therapy. This is however not the case as can be seen when we search the CMAP with a profile composed of genes whose sense change is conserved across the rodent disease models. Combining the severe AD profile and the four rodent neurodegenerative disease model profiles we get a set of 24 genes whose sense change is conserved. This consists of 10 up regulated and 14 down regulated genes, which can be thought of as a binary signature for neuropathology, where +1 is assigned to up-regulated genes and -1 to down-regulated genes, see Table We have collected transcriptional data from diverse platform architectures corresponding to various species. By processing the data into effective fold profiles, with the expression levels factored by the average level over the experimental series and defined over a non-redundant gene list, we can directly compare transcriptional profiles from arbitrary sources. The fundamental principal underlying the utility of this approach is that biological effects can be compared through the corresponding transcriptional changes. This idea underlies the CMAP initiative for matching drug to phenotype by querying a database of drug induced transcriptional profiles with a profile defining the phenotype. We have extended this methodology to include potentially all available transcriptional data. In its current version SPIED contains transcriptional profiles for 106,101 arrays covering five platform architectures and three species. This can be easily extended to include other platforms and species. The results largely confirm the hypothesis that high scoring correlations correspond to similar biological processes. We have presented SPIED results for drug perturbagen induced profile queries and queries derived from disease states. For brevity we focussed on three sets of drug treatment profiles corresponding to mTOR/PI3K, estrogen and HDAC inhibitors. SPIED searches with these queries showed correlations with other drug treatments belonging to the same classes and in the case of the mTOR antagonist rapamycin we found high anti-correlations with the profile of a cancer inducing fusion transformation, suggesting a novel indication for rapamycin. Also, for brevity of exposition we focussed on two completely unrelated classes of pathology: cancer and neurodegeneration. In the case of leukaemia we show that a corticosteroid resistance signature derived from leukaemia cell cultures shows significant correlation with a lung cancer predisposition profile and a pancreatic cancer profile. Thereby implicating glucocorticoid resistance in these two pathologies. To illustrate the application of SPIED to neurodegenerative pathology we constructed a severe stage AD profile from a published study. Interestingly, querying SPIED resulted in high correlations with other neuropathological conditions indicating a common feature of synaptic loss and mitochondrial dysfunction. Restricting our searches to the rodent subset of SPIED returned expression profiles from animal models of neurodegeneration and neuronal injury. Combining the human and rodent signatures we obtained a core signature that we probed against CMAP for neuroprotective agents. Remarkably, we found at least 9 neuroprotective agents in the top 22 anti-correlating CMAP hits. These results motivate the extension of SPIED and the extension of the CMAP to include other cell types, for example a neuronal cell lineage will be more appropriate for generating drug profiles for neurological diseases. The correlation query scores maybe insensitive to a radical reduction in the number of probes and this should motivate the design of reduced and more cost effective arrays for more extensive data generation.Microarray sample files, GSM files, were downloaded form the NCBI GEO database. Individual GSM files were assigned to GSE series and log scaled values scaled to linear and low level responders dropped. EF profiles were then generated based on ratio of individual condition to the average across the series. Expression data from five Affymetrix GeneChip platforms corresponding to three species were collected. These were: all samples from two Human array platforms corresponding to Human Genome U133 Array Set HG-U133A GPL96  and U133 Plus 2.0 Array GPL570 ; all samples from the Mouse Genome 430 2.0 Array GPL1261  chip; all samples from two Rat chips corresponding to Rat Genome U34 Array GPL85  and Rat Genome 230 2.0 Array GPL1355 . The database thus totals 106,101 samples. Of course, this can always be extended to include more platforms from the same species and/or other species.k sis the expression level of the kth probe set and The individual GSM sample file expression values were transformed into EF values corresponding to the expression relative to the series mean. Expression values that have been logarithmically transformed are transformed back to a linear scale and low expression values dropped, that is are set to zero and don't contribute to the fold profile. We found that the results were relatively insensitive to the cut-off value and we set this to be 10% of the average expression value. All sample expression profiles within a series were scaled to the same average. The fold vales are defined ask-s¯sk+s¯, where sr. The experiments are ranked according to the significance. The significance is measured by scaling the correlation to the normal by a Fisher transformation and measuring the number of standard deviations from the mean. The Fisher transformation is r is the Pearson correlation coefficient and N is the number of genes making up the correlation. The final ranking score is The query profile is a statistically thresholded non-redundant list of genes and associated fold values. Statistical significance is assigned to a fold change based on a simple Student's t-test between multiple control and treatment sample expression values. This is compared to each profile in the database by means of a simple Pearson regression analysis, with a correlation coefficient i ris the rank of the th igene and min/max are the minimal/maximal ranks. Remembering that the highest rank corresponds to the most up-regulated gene. The SPIED was searched with CMAP profiles corresponding to folds with a p < 0.05 threshold and with at least three replicates. This left 1,218 separate perturbagen probes. We sought to cluster the perturbagens based on predicted target and response profile similarity. The profiles are given in the additional file The CMAP contains ranked lists of probes for 6,100 separate perturbagen treatments of four different human cell lines, with the ranking based on response level relative to control. The treatments are various multiples of 1,306 different drug-like compounds. To generate responder sets that can be used to search SPIED we combined rankings for each separate compound treatment and converted these into pseudo-fold values with associated statistics. The pseudo-fold value is defined byThe SPIED database and associated executables are available for download from ftp://ftp.hostedftp.com/~GWftpFILES/SPIED/. The download consists of the SPIED database together with executables for searching SPIED. Source code files to generate the database and perform query searches are provided together with the executables. Documentation on the database, the executables and source code files is also included.GW is the sole author of the present study.Expression profiles derived from CMAP and published AD studies. Expression profile signatures derived from pooling CMAP compound treatment replicates, form corticosteroid resistance studies and from various AD stages referenced to age matched controls. These profiles are used to query SPIED. The data is given as excel spreadsheets.Click here for fileTop scoring correlations for various expression signature queries against SPIED. The results upon querying SPIED with expression profiles derived from the signatures in additional file 1. The data is given as excel spreadsheets.Click here for file


The prognosis of cancer recurrence is an important research area in bioinformatics and is challenging due to the small sample sizes compared to the vast number of genes. There have been several attempts to predict cancer recurrence. Most studies employed a supervised approach, which uses only a few labeled samples. Semi-supervised learning can be a great alternative to solve this problem. There have been few attempts based on manifold assumptions to reveal the detailed roles of identified cancer genes in recurrence.In order to predict cancer recurrence, we proposed a novel semi-supervised learning algorithm based on a graph regularization approach. We transformed the gene expression data into a graph structure for semi-supervised learning and integrated protein interaction data with the gene expression data to select functionally-related gene pairs. Then, we predicted the recurrence of cancer by applying a regularization approach to the constructed graph containing both labeled and unlabeled nodes.http://embio.yonsei.ac.kr/~Park/ssl.php.The average improvement rate of accuracy for three different cancer datasets was 24.9% compared to existing supervised and semi-supervised methods. We performed functional enrichment on the gene networks used for learning. We identified that those gene networks are significantly associated with cancer-recurrence-related biological functions. Our algorithm was developed with standard C++ and is available in Linux and MS Windows formats in the STL library. The executable program is freely available at:  The use of accurate cancer biomarkers can help to determine the appropriate therapy based on patient status. These biomarkers can be presented as a list of genes or gene network structure. Microarray based gene expression has been used to identify these biomarkers et al. mentioned that obtaining microarray data with clinical follow-up information is time consuming, expensive, and limited by sample availability Although gene expression and interactome data are very useful for cancer research, the relatively small number of samples compared to the number of genes leads to challenges in analysis One approach to supplementing the small quantities of labeled data is semi-supervised learning, which is a combination of super-vised and unsupervised methods. Semi-supervised learning combines labeled and unlabeled data to construct a learning model with improved accuracy et al. developed a graph-based semi-supervised learning classifier that can predict pairwise synthetic genetic interactions et al. proposed a semi-supervised learning based method to predict genes involved in disease by inferring both disease genes and their neighbors through protein interaction networks et al. proposed using both available clinical data and gene expression data to identify the subset of the genes used to perform semi-supervised clustering et al. used gene expression profiles to identify a gene classifier associated with a high risk of metastasis and death from colon cancer Recently, semi-supervised learning based approaches have been widely applied to biological data analysis including genetic interactions. You et al. proposed a semi-supervised classifier based on low density separation that can identify high-risk and low-risk patients As mentioned above, semi-supervised approaches can supplement the limitations of gene expression data analysis, such as lack of an assigned clinical class for each patient. Shi In this article, we used graph regularization and integration of transcriptome and interactome data to build a novel semi-supervised learning-based classifier for human cancer, and constructed a cancer-specific gene network. The graph regularization is based on the ‘manifold assumption,’ where the construction of graph models is an important phase. In design of the graph model for classification, we constructed the graph using labeled and unlabeled samples as nodes. The connection between two samples was calculated using the selected informative gene pairs. In selecting useful gene pairs, we integrated Protein-Protein Interaction (PPI) data with gene expression data. PPI data provided information about the functional relationship among proteins and was applied to genes connected by PPIs. After selecting gene pairs, we applied a scoring scheme proposed in a previous paper We demonstrated that (i) the proposed semi-supervised learning based classification enhanced prediction performance compared with existing methods, including TSVM, which is a semi-supervised learning version of SVM, (ii) the proposed method was applicable to different cancers, (iii) the proposed method was robust regardless of the class label ratio and (iv) the cancer-specific gene network derived from the classifier was biologically meaningful, and the cancer-specific genes of this network played a role as members of complex biological processes.To build a semi-supervised learning classifier, we first integrated gene expression data with PPI and identified informative gene pairs with the labeled samples. Second, we constructed a sample based graph model using selected informative genes in order to build a classifier.We downloaded the gene expression datasets of three cancers from the Gene Expression Omnibus (GEO) database. We also downloaded 194,988 human PPIs from the I2D database, which included known, experimental, and predicted PPIs. Because the proteins in these PPIs were mapped into gene symbols using Universal Protein Resource (UniPROT), we obtained 108,544 PPIs after removing duplicated PPIs and PPIs that contained proteins that were not mapped to a gene symbol.This section describes a novel graph-based semi-supervised learning algorithm for cancer prognosis. The graph consists of nodes and edges corresponding to samples and interactions between two samples, respectively. The graph is constructed with both labeled and unlabeled samples of gene expression data, and the unlabeled samples were subsequently labeled based on the geometry of the graph structure. Therefore, it is very important to generate a sample-based graph from the given dataset. We propose a novel graph construction method that is specialized for a microarray dataset. Based on this graph construction method, we developed a semi-supervised learning algorithm that uses graph regularization.In this approach, the graph itself is a classifier. Thus, the parameters for constructing the graph imply that they are the key factors of the classifier. The classification results are dependent on the parameters. Semi-supervised learning generally utilizes the feature or underlying information of unlabeled data. This approach assumes that unlabeled data is able to enhance the classification performance. According to this distinguishing feature of semi-supervised learning, we take advantage of unlabeled data for building a classifier.k-fold cross validation. After this phase, we construct the graph with both labeled and unlabeled samples. Then, we identify whether the classification results from graph regularization are changed or converged. If they are changed, we regard the classified unlabeled data as newly labeled data and use them to determine the optimal candidate parameters. In this iterative process, the information of unlabeled samples is provided. The previous semi-supervised learning method proposed in The proposed method has two phases. The first phase is to determine the candidate optimal parameters for graph regularization varying the parameter ranges in The details of the semi-supervised learning module in this workflow are described in the following sections. This module consists of the following three core steps: (1) identification of informative gene pairs, (2) construction of sample graphs with selected genes, and (3) regularization of the graph and prediction of the labels of the unlabeled samples. The workflow of the semi-supervised learning module is shown in There are tens of thousands of genes in microarray datasets, and only some of them are specific to the classification of the sample. Informative gene pairs indicate interactions that are diacritical in the two contrary classes of labeled samples. We adopted and modified our previously proposed scheme for identifying interactions in the gene expression dataset iCg1 and iCg2 are vectors of the mRNA expression value of gene i on class 1 and class 2 samples, respectively, and jCg1 and jCg2 are vectors of the mRNA expression value of gene j on class 1 and class 2 samples. Only the gene pairs with a scoring value greater than gthreshold are regarded as being significantly different between two classes. This scoring scheme is performed only with the labeled samples in the gene expression dataset. A simple example of calculating Score values is shown in Figure S1 in The measurement of changes in interactions can be regarded as identification of the degree of dependency between two genes. A large correlation value between two genes as a degree of change indicates that there is strong dependency between the two genes. Based on this rationale, we propose a scoring scheme to calculate the strength of the connection between two genes that are connected by PPIs. Using this measure, we can facilitate the selection of informative interactions from gene expression datasets, since the cancer specific network was constructed based on a similar scoring function. In other words, we can choose the interactions specified for tumor recurrence using the proposed scoring scheme. The score of two genes is calculated by the following equation:*iS and *jS are vectors of the mRNA expression value of sample i and sample j, respectively, of the selected gene pairs with values larger than sthreshold. We assume that there is a significant relationship between two samples when they are highly associated with each other with a positive or negative pattern. We can transform the gene expression dataset into a graph structure that can be regularized. A simple example of calculation of the Weight value is shown in Figure S1 in We constructed a sample-based graph for regularization. The weight of a sample pair is calculated by the Pearson Correlation Coefficient (PCC) between two sample vectors that are composed of the genes as elements, where the genes are obtained from informative gene pairs. Both labeled and unlabeled samples are used in the graph. The weight function is as follows:y and ŷ respectively indicate the initial labels and the estimated labels for both labeled and unlabeled data. ijW indicates the weight between node i and node j. The total number of both labeled and unlabeled nodes is n, and the number of labeled nodes is l. In our problem, y indicates labeled and unlabeled samples of the cancer dataset, and ijW is obtained using the weight function defined in the above chapter. Using the cost function, we measure the consistency with the initial labeling using the first term, and we assign a penalty for regularization using the second term. Using the second term, we calculate the weighted difference between two nodes without consideration of whether or not they are labeled. The major purpose of this cost function is to minimize the weighted difference among all nodes. This process refers to regularization and is equivalent to the label propagation algorithm. In our case, it is unnecessary to reassign labels to the labeled data because they have already been clinically verified. Therefore, in the first term of the cost function, iŷ is constrained to be equal to iy. As a result, the cost function can be transformed into the following function with a graph Laplacian.Based on the sample-based graph structure derived from the method mentioned above, labels are assigned to the unlabeled nodes. To achieve this, we employ a basic regularization approach. For the regularization of the graph, we estimate a regularization framework based on the manifold assumptions. The cost function for regularization is as follows:L is the un-normalized graph Laplacian and D is a diagonal matrix of weight matrix W. This function penalizes rapid label changes in ŷ between two close data points according to the given weight matrix. Various approximations have been proposed to minimize this function over uŷ, where uŷ indicates the estimated label for unlabeled data and lŷ indicates the labeled data. Minimizing the function with respect to uŷ converts it into the following function.We predict the labels for the unlabeled data using this calculation. Since we do not focus on development of novel semi-supervised learning algorithm, we employ a general regularization approach for the weighted sample graph, and it is sufficient to apply the general approach to our problem.We performed experiments to obtain the optimal combination of two thresholds for the score of a gene pair and the weight of the sample based graph. We then compared our method with several existing methods in order to assess its performance. Finally, we analyzed the network derived from our method with the known cancer related gene list.k-fold cross validation by varying these two parameters. We changed the gthreshold value from 0.15 to 0.6 in intervals of 0.05 and the sthreshold value from 0.72 to 0.9 in intervals of 0.02. Overall, we performed 100 different experiments, varying these two thresholds and measuring the accuracy of each experiment by averaging the k accuracies generated during k-fold cross validation. Figure S2 in We used two parameters to both identify informative gene pairs and assign weights to sample pairs. To find optimal combinations of these two parameters, we measured the accuracy of the proposed classification model using i if the calculated value after regularization is greater than 0.5, and label 0 otherwise. However, this decision approach is only effective when the classes are well separated. Since gene expression data do not always have the same number of samples for each class, we adopted CMN to identify the final class label. CMN adjusts the criterion for determining the class label according to the ratio of the mass of classes.To determine the classification of unlabeled samples, we applied a heuristic method called Class Mass Normalization (CMN) proposed by k-fold cross validation and averaged the k accuracies. The purpose of this process was to compare the accuracy of classification on 100 different experiments. We also carried out the same experiments with an adjusted dataset, which had the same number of samples for both recurrence and non-recurrence groups since different proportions of class labels can affect the performance of the classifier. Our method uses semi-supervised learning-based graph regularization, which is influenced by the geometric structure of the graph, to classify the label. If the relative ratios of two classes differ considerably, the labels of a small number of samples may not be propagated through the graph. This can affect classification performance. All of the chosen cancer datasets were divided into original and adjusted sample groups. In the remainder of this article, we describe an experiment conducted with these two groups. We obtained two optimal threshold values at maximal accuracy for each dataset, as shown in k value of cross validation. The experimental results of k = 5 and k = 20 are described in Table S5 in The experimental results obtained from varying parameters are shown in We compared the proposed method with three typical supervised classification algorithms implemented in Weka 3.6.8, namely Support Vector Machine (SVM) We compared the accuracies, including the sensitivities and specificities, of the proposed method and other methods using 10-fold cross validation. We divided the dataset into two groups as mentioned above, and repeated the experiment 15 times each for three cancer types. We calculated the average values of accuracy, sensitivity, and specificity for each dataset in the adjusted group. The sensitivity and specificity of TSVM could not to be calculated since TSVM of SVM-light provided accuracy, precision, and recall. The average accuracy increased 24.9% compared to the four existing methods. For example, as shown in In addition, we performed an independent test where we applied relief-F to select informative genes instead of PPI. We also carried out a statistical analysis of significant difference in accuracy for comparison among methods. The detailed experimental results are described in the supporting information of Table S1, Table S3, and Table S4 in The performance of a classification method is influenced by the proportion of training data in each class. The computational contribution of the proposed method is determination of the coherent accuracy of the differences in class proportion. This is advantageous since the number of samples for each class cannot be adjusted during independent testing. In addition, though classification based on semi-supervised learning has been applied to microarray datasets, the results of the proposed method demonstrate that the approach based on the ‘smoothness assumption’ was sufficient for clinical application.To reduce the dimension of the microarray data, we selected gene sets with strong biological interactions. Therefore, the sample-based graph of regularization was constructed based on biological knowledge. The selected gene set can be referred to as a recurrence-specific gene network. Our analysis demonstrated that this gene network was biologically meaningful in regard to cancer recurrence. To analyze the cancer-recurrence-specific gene network, we enriched the informative gene set derived from the optimal parameter set using the Gene Ontology (GO) database and BiNGO et al. reported an association between breast cancer risk and survival based on CCND1 polymorphisms et al. concluded that activation of STAT1 plays an important role in the death of tumor cells and the activation of apoptotic genes The proposed method identified the sub-gene network composed of BRCA1, CCND1, STAT1, and CCNB1, shown in In this study, we proposed a novel semi-supervised learning method based on graph regularization in order to predict cancer recurrence. We also showed that the recurrence-specific gene networks derived from the proposed method contain many recurrence-related genes. We integrated the PPI data with the gene expression data to produce an informative gene set and to analyze the biological process related to recurrence. We also used a graph regularization approach and semi-supervised learning methods to predict the labels of unknown samples. We confirmed that the performance of the proposed method was better than that of several preexisting classification methods for colorectal and colon datasets, which had the same proportion of class labels. In the case of the breast cancer dataset, the proposed method showed outstanding performance compared to the existing method with both the original and the adjusted samples. Also, the proposed method was superior to TSVM for three cancer datasets. It is necessary to utilize the unlabeled samples because labeling medically many samples is time consuming and expensive. In a medical dataset, however, it is difficult to obtain a dataset with a balanced number of samples. We plan to focus on solving this problem of the semi-supervised learning-based method in future work. Last, we identified the functional relationships among recurrence related genes by constructing gene networks. We concluded that the proposed method, which uses many data points without class labels, is suitable for prognosis prediction and analysis of the biological roles of genes related to cancer recurrence.File S1File S1 contains four supplemental figures and, five supplemental tables.(DOC)Click here for additional data file.