It is desirable in genomic studies to select biomarkers that differentiate between normal and diseased populations based on related data sets from different platforms, including microarray expression and proteomic data. Most recently developed integration methods focus on correlation analyses between gene and protein expression profiles. The correlation methods select biomarkers with concordant behavior across two platforms but do not directly select differentially expressed biomarkers. Other integration methods have been proposed to combine statistical evidence in terms of ranks and p-values, but they do not account for the dependency relationships among the data across platforms.In this paper, we propose an integration method to perform hypothesis testing and biomarkers selection based on multi-platform data sets observed from normal and diseased populations. The types of test statistics can vary across the platforms and their marginal distributions can be different. The observed test statistics are aggregated across different data platforms in a weighted scheme, where the weights take into account different variabilities possessed by test statistics. The overall decision is based on the empirical distribution of the aggregated statistic obtained through random permutations.In both simulation studies and real biological data analyses, our proposed method of multi-platform integration has better control over false discovery rates and higher positive selection rates than the uncombined method. The proposed method is also shown to be more powerful than rank aggregation method. In gene expression experiments, the expression levels of thousands of genes are simultaneously monitored to study the underlying biological process. In proteomic data, the protein levels or protein counts are measured for thousands of genes simultaneously. In addition, there are other types of genomic data with different sizes, formats and structures. Each distinct data type, such as gene expression, protein counts, or single nucleotide polymorphisms, provide potentially valuable and complementary information regarding the involvement of a given gene in a biological process. Many biomarkers that play important roles in biological processes behave differently in treatment versus control groups; this phenomenon can be observed consistently across various data platforms. Therefore, integrating related data sets from different sources is crucial to correctly identify the significant underlying biomarkers. Integrative analysis of multiple data types would improve the identification of biomarkers of clinical end points . HoweverThe practice of combining different data sources to perform classification analysis has been considered in the literature. Efforts to integrate data and improve classification accuracy are widely seen in recent studies -5. In coThe problem of how to reliably combine data from different experiment platforms to identify significant biomarkers has recently received considerable attention in the bioinformatics literature. The rank aggregation method  has beenThe proposed method is similar in spirit to a meta-analysis. Both methods combine statistical evidence across multiple data sets. However, in meta-analysis different data sets are based on the same type of experiments or observational studies, and therefore the measurements are the same variables. Across different data sets, the quality of the data may vary. The goal of meta-analysis is to fully utilize all the information from different data sets and construct a weighted estimate of the effect size. Different weighting schemes are available depending on the statistical models . On the The aim of our multi-platform integration method is to select a set of significant biomarkers that are involved in a biological process and thus behave differently in the treatment group and the control group. In order to combine statistical evidence across different platforms, our method requires that analogous hypotheses based on the features being measured are formulated for each platform. Each null analogous hypothesis specifies the unrelatedness of the biomarker in that particular experimental setting, but all of them infer the unrelatedness of the biomarker to the biological process being investigated. Based on the set of Q analogous hypotheses for Q data sources, we construct a set of Q corresponding test statistics for each type of data. The test statistics can be different and tailored to the specific experimental settings. For example, if the microarray experiment has a multifactorial design, the appropriate test statistic can be an F statistic based on an ANOVA test. If the proteomics experiment generates counting data for diseased versus normal groups, the appropriate test statistic can be a nonparametric Wilcoxon rank sum test. A vector of observed statistics across multi-platforms is obtained. We then randomly permute data across diseased and control groups. All measurements from different platforms are permuted. In this way, we obtain an empirical null distribution of the vector of test statistics. In order to pool the randomized values of the statistics across the biomarkers to form the empirical null distribution, we assume data from different biomarkers are independent or have an exchangeable correlation structure. For the validity of the randomization procedure, we assume an exchangeable covariance structure for the measurements within each platform. Finally, we construct a weighted sum of the test statistics across different platforms with the weights being the inverse of the empirical standard deviation of each statistic. We determine a set of significant biomarkers based on the aggregated test statistic.l1 diseased samples and l2 control samples and the second having proteomic data measured on m1 diseases samples and m2 control samples. The objective is to find biomarkers significantly involved in disease development. In the following, we demonstrate our method by integrating microarray expression data and proteomic data as an example. We consider two experiments, the first having microarray expression data measured on H01: the gene’s mRNA level is the same in diseased and normal populations; for proteomic data, the null hypothesis would be H02: the protein level is the same in diseased and normal populations.Step 1): Define two analogous null hypotheses. For microarray data, the null hypothesis would be tm and tp, tailored to each type of data. Consequently, we obtain a vector of two observed statistics ′ across two data platforms. The test statistics can be of any type as long as they summarize information from the data and can be used to assess the statistical significance of the data toward the hypotheses. Let l1 gene expression measurements in the disease group, l2 gene expression measurements in the control group, m1 protein measurements in the disease group and m2 protein measurements in the control group, Step 2): Based on the hypotheses, construct two test statistics,  and s2 denotes the sample variance. The test statistics should be formulated so that a larger test statistic in the positive direction indicates more evidence towards the alternative hypotheses. For example, if Student’s t-statistic is used, then a one-sided alternative hypothesis corresponds to a one-sided t-statistic, whereas the two-sided alternative leads to the absolute value of the t-statistic. Consider n genes being measured in the experiments and we obtain n vectors of test statistics ′, i = 1,…,n, from the data sets. where tm,tp)′.Step 3): The samples are randomly permuted across diseased and control groups. If the same sample is being measured across different platforms, all the measurements from the different platform are permuted simultaneously. The simultaneous permutation preserves the dependency relationship among the measurements from different platforms. Based on random permutation, we obtain an empirical null distribution of the vector (Step 4): The aggregated test statistic will be: tm and tp based on the empirical null distribution, and tm and tp are the observed t-statistics or the absolute values of the t-statistics based on the direction of the alternative hypotheses. At significance level α, we choose a threshold Cα, such that Cα is the 100(1−α)% percentile of tA, which can be obtained from the empirical null distribution. Construct a decision line that separates selected significant biomarkers and nonsignificant biomarkers. The resulting separation line is:  where tm,tp) above the separation line will be declared as significantly involved in the disease development.All the biomarkers with ′. From random permutation, we obtain the joint empirical distribution of this vector of test statistics under the global null hypothesis. Let In the more general case, suppose we have Q data platforms with the observed test statistics % percentile of tA. Any biomarker with tA > Cα will be selected as behaving significantly differently between the diseased group and control group. where Cα is determined based on the empirical null distribution of the aggregated test statistics, which implicitly takes into account the dependency relationships among the test statistics. Furthermore, our method can deal with different data types and formats generated by various experimental settings.Our method aggregates actual values of the test statistics across different data platforms, which preserves more information compared to the rank aggregation method. Moreover, our method assigns different weights to each data set according to the variability of the test statistics: larger the variation in the test statistic, the smaller the weight assigned to it, and vice versa. The threshold α∗, then the individual level α = α∗/n, where n is the total number of biomarkers. When n is large, the Bonferroni correction leads to very stringent tests with α being very small. Alternatively, we can control the number of false discoveries. To set the number of false discoveries to be equal to or less than f , then α.There are two major ways to perform the multiplicity adjustment. The first is the Bonferroni correction. If we wish to control the familywise type I error rate at H10: mRNA is increasing in normal group; H20: antibody count is decreasing in normal group. Even though the actual measurements from two platforms are negatively correlated, we can construct the test statistics t1 and t2 so that the positive value of the statistics supports the alternative hypotheses and the weighted average can be used as combined evidence of the involvement of the biomarker in the process.Different platforms can be used to test different sub-hypothesis. All of these sub-hypotheses should be concordant in supporting the overall biological hypothesis. For example, the involvement of a gene in disease development can be supported by both mRNA expression level changes and proteomic level changes. In most cases, changes in measurements from different platforms are expected to occur in the same direction. However, our method is also applicable even if the changes are in different directions, as long as the statistical evidence from both sources can be combined. For example, consider Q different platforms. The number Q is set to be either 2 or 5. For the qth experiment, the data set is denoted as Xq. For each data set, we assume that n different biomarkers are measured, Xq = ith biomarker, Xqi= Xqi1 denotes data from the control group with mean μqi1 and Xqi2 denotes data from the diseased group with mean μqi2. The total number of biomarkers is set to be n = 1000. Among the n biomarkers, let g denote the number of biomarkers that are related to the biological process of interest, i.e. μqi1 ≠ μqi2. The number g of differentially expressed (DE) biomarkers is set to be 200. The number of measurements for each biomarker obtained from each platform is set to be 10, in which 5 are from the control group and the other 5 are from the disease group. We also consider different effect sizes. For continuous data, we generate ρ. The correlation ρ is set to be either 0 or 0.5. For differentially expressed markers, μqi1 = 0 × 1m, μqi2 = e × 1m, where e is the effect size and m = 5 is number of measurements. Discrete data Xqiis generated from a Poisson(λ) distribution, where λqi1 = μqi1 for the control group and μqi2 = μqi1 + efor the diseased group. The g differentially expressed markers are divided into two groups with g1 = 100 and g2 = 100. Each group is assigned a different effect size e. For each platform, the alternative hypothesis can be either left-sided, right-sided or two-sided. The number of permutation is 100. All of the permuted values from the n biomarkers are pooled together to form the empirical null distribution. The results are summarized for 100 simulated data sets.In this section, we examine the performance of our proposed method by examining its positive selection rates and false discovery rates under various testing scenarios. We simulate data sets from To compare our multi-platform integration method with the individual platform analysis method, the positive selection rate (PSR) and false discovery rate (FDR) are calculated to assess the performance of each method for selecting the differentially expressed biomarkers:  and α = 0.05 significance level. From the results, we can see that our multi-platform integration method has the highest PSR and the lowest FDR with the smallest variance compared to all other individual platform analyses in all scenarios. In addition, such advantage is consistently observed regardless of whether or not there is correlation among the measurements obtained for each biomarkers. Table Tables Figure α, we obtain the realized false discovery rate FP denotes the number of false positive biomarkers, π is the proportion of non-differentially expressed genes, and FP; the FDR as well. Under the simulation setting of scenario 2 left-sided case in Table π = 0.8 and α = 0.005, FDR of 0.0214. This demonstrates the integrative analysis yields satisfactory control of false discovery rate, which is improved compared to individual platform analyses.As we examine a large number of biomarkers, we need to investigate the control of the false discovery rate of the proposed method with regards to multiple hypothesis testing . Given aStreptomyces coelicolor provided by Jayapal et al. . To be consistent with the protein data, mRNA abundance data from different samples were processed to calculate log2[cDNAi/cDNAhr7] for each sample with respect to the first time point sample. Only gene expression values with protein values (894 genes) were analyzed. To deal with missing values, we deleted genes that had no values for mRNA at all or had at least five missing values in the protein data set. The rest of the missing values for genes were imputed by using R package MICE. In total, the number of genes suitable for the subsequent integrative analysis was 886. Based on the growth curve, time points were divided into two groups; those from 7, 11, 14 and 16 h represented the growth phase and those from 22, 26, 34 and 38 h represented the stationary phase.For microarray data, total mRNA from the same eight time point samples were isolated and a spotted DNA microarray experiment was conducted. Hybridization was performed using genomic DNA (gDNA) as a reference. The mRNA abundance was obtained using H0: the mRNA expression level is the same between the two phases. Similarly, for protein data, the null hypothesis is formulated as H0 : the protein ratio is the same between the two phases. For both mRNA data and protein data, two-sided alternatives are considered in the analysis. For each platform, we use Student’s t-statistics to summarize the statistical evidence, which are denoted as tm and tp. To obtain the multivariate null distribution, 100 permutations are conducted. The overall correlation between tm and tp is 0.2787. The variances of tm and tp are 3.0489 and 3.6411, respectively. Based on the decision line constructed at the significance level α = 0.05, our method detects 172 differential expressed genes with an estimated The objective of our analysis is now to select the biomarkers that are differentially expressed between the two phases. We apply our multi-platform integration method to identify differentially expressed biomarkers. For the mRNA data, we formulate the null hypothesis as Nine differentially expressed genes are identified by our method but not by the other two methods. Among these, we identify biosynthetic enzymes  involved in actinorhodin production. These genes are up-regulated only at late stages of the culture and produce antibiotics during the stationary phase. Expression of two genes encoding malate oxidoreductase (SCO2951) and translation elongation factor G (SCO4661) have been found to be depressed during the stationary phase compared with the growth phase . Table 5An ongoing problem in proteomics is that extremely small sample sizes often occur, largely due to biological reasons. To investigate the performance of our method in such situations, we consider a case for each platform wherein the control and the diseased groups each have only two measurements. Our method is applied and the simulation results shown in Table n platforms have a multivariate distribution and the data from the diseased group are independent of those from the control group. The new simulation results are summarized in Table We also consider the situation in which data on the same biomarker from tm and tp as long as they provide summarized statistical evidence for that platform. The Student’s t-statistic is adopted in the paper simply for illustration purpose. Alternatively, we can simply use the unstandardized differences: tm and tp and form a weighted linear sum statistic. To compare the empirical performance of the standardized versus unstandardized versions, we conduct simulations under the setting 1 of Table tm and tp has a slightly higher PSR and a slightly lower FDR.The proposed method allows different ways of constructing An alternative way of combining test statistics across different platforms is to form a multivariate quadratic statistic. Given two platforms, for example, we consider an alternative test statistic tm,tp) obtained from the empirical null distribution. Such multivariate statistic can be used to test the overall null hypothesis against two-sided alternatives, while the weighted linear statistic that we propose can be used to test one-sided alternatives or two-sided alternatives. Thus, our method is more broadly applicable. We further conduct simulations to compare the multivariate quadratic form with our proposed weighted linear statistic for two-sided tests under the setting of scenario 2, Table  where Finally, we compare our method with the existing robust rank aggregation method  with resWith the advent of various types of genomic technologies, it is imperative to develop a method that can integrate different types of genomic data to solve biological questions. We develop a general framework for data integration across multiple data platforms. For each data set, a test statistic is formed to summarize the statistic evidence toward the specific null hypothesis tailored to the data platform. The types of test statistics can vary and their marginal distributions can be different. The observed test statistics can then be aggregated across different data platforms. The overall decision is based on the empirical distribution of the aggregated statistic obtained through random permutations. Our method can accommodate different experimental designs and various data types across platforms.The authors declare that they have no competing interests.SW, XG, YX, XW and ZF developed the algorithm, SW and YX implemented the algorithm, YX, ZF, and XY performed data analysis; and XG supervised the project. All authors read and approved the final manuscript.


High-throughput RNA sequencing (RNA-seq) offers unprecedented power to capture the real dynamics of gene expression. Experimental designs with extensive biological replication present a unique opportunity to exploit this feature and distinguish expression profiles with higher resolution. RNA-seq data analysis methods so far have been mostly applied to data sets with few replicates and their default settings try to provide the best performance under this constraint. These methods are based on two well-known count data distributions: the Poisson and the negative binomial. The way to properly calibrate them with large RNA-seq data sets is not trivial for the non-expert bioinformatics user.tweeDEseq implementing a new test for differential expression based on the Poisson-Tweedie family. Using simulations on synthetic and real RNA-seq data we show that tweeDEseq yields P-values that are equally or more accurate than competing methods under different configuration parameters. By surveying the tiny fraction of sex-specific gene expression changes in human lymphoblastoid cell lines, we also show that tweeDEseq accurately detects differentially expressed genes in a real large RNA-seq data set with improved performance and reproducibility over the previously compared methodologies. Finally, we compared the results with those obtained from microarrays in order to check for reproducibility.Here we show that expression profiles produced by extensively-replicated RNA-seq experiments lead to a rich diversity of count data distributions beyond the Poisson and the negative binomial, such as Poisson-Inverse Gaussian or Pólya-Aeppli, which can be captured by a more general family of count data distributions called the Poisson-Tweedie. The flexibility of the Poisson-Tweedie family enables a direct fitting of emerging features of large expression profiles, such as heavy-tails or zero-inflation, without the need to alter a single configuration parameter. We provide a software package for R called tweeDEseq package forms part of the Bioconductor project and it is available for download at http://www.bioconductor.org.RNA-seq data with many replicates leads to a handful of count data distributions which can be accurately estimated with the statistical model illustrated in this paper. This method provides a better fit to the underlying biological variability; this may be critical when comparing groups of RNA-seq samples with markedly different count data distributions. The  High-throughput gene expression profiling across samples constitutes one of the primary tools for characterizing phenotypes at molecular level. One of the main advantages of the rapidly evolving massive scale cDNA sequencing assay for this purpose (RNA-seq ), over tFor DE analysis, after some pre-processing steps that include the alignment of the sequenced reads to a reference genome and their summarization into features of interest , raw RNA-seq data is transformed into an initial table of counts. This table should then be normalized -4 in ordDetection of DE genes using RNA-seq data was firstly based on using models assuming a Poisson distribution  with onen=69 unrelated Nigerian (YRI) individuals [P-values that are equally or more accurate than NB-based competing methods calibrated with a variety of configuration parameters. Finally, by surveying the tiny fraction of sex-specific gene expression changes in LCL samples, we approach the problem of assessing accuracy in DE analysis with real RNA-seq data and show that, in the context of extensively replicated RNA-seq experiments, tweeDEseq yields better performance than competing NB-based methods without the need to make an informed decision on the configuration of parameters. This improvement is reported in terms of precision and recall of DE genes and reproducibility of the significance levels with respect to matching microarray experiments.In this paper we propose to approach this problem by using other count data distributions that fit expression profiles better than the NB without the need to alter configuration parameters. The rest of the paper is organized as follows. Using a large RNA-seq data set of HapMap lymphoblastoid cell lines (LCLs) derived from ividuals , we starn=69 samples from unrelated YRI individuals, with our own pipeline (see Methods). The resulting table of counts consists of 38,415 genes by 69 samples. We filtered out genes with very low expression levels and used different normalization methods [The results we provide in this paper are based on data from a previously published large RNA-seq experiment  and on o methods ,4  [NLGN4Y (ENSG00000165246), a gene located in the male-specific region of chromosome Y and reported to have sex-specific expression, which shows remarkably different count data distributions between male and female samples. Permutation tests are also underpowered since distribution tails are not well estimated (due to the large dynamic range), which is important when correcting for multiple testing.There is currently a large body of literature on DE analysis methods for RNA-seq data -11,14, n. (2010)  argued t. (2010)  capturededgeR[DESeq[edgeR and DESEq, assume that expression profiles from RNA-seq data follow an NB distribution and borrow information across genes to first estimate a common dispersion parameter. Then, for each gene, they estimate its genewise dispersion and moderate it towards the common one. The way in which this moderation takes place depends on the method and its configuration parameters. DESeq[sharingMode=~fit-only~) and genewise (sharingMode=~gene-est-only~) dispersions. It provides a straightforward strategy  to choose between common and genewise dispersions by taking the largest value for each gene. edgeR allows one to calibrate, using the prior.df parameter, the transition from a purely genewise dispersion estimate  to the common one  by using an empirical Bayes approach. By default prior.df=20 which implies that a large weight is given to the common dispersion. However, according to the documentation, if the number of samples is large, the common dispersion becomes less important in the moderation step. Additional options in DESeq and edgeR that may be relevant in the context of large RNA-seq data sets are, in the case of DESeq, whether dispersions are estimated from the entire pool of samples  or separately per sample group (method=~per-condition~). In the case of edgeR, whether the DE test is performed using a likelihood ratio test (glmLRT function) or a quasi-likelihood F-test [glmQLFTest function), after dispersions are estimated. Table In this paper we will focus our comparisons on the two most widely used methods for DE analysis of RNA-seq data, edgeR,8,10  by means of quantile-quantile (Q-Q) plots Figure  and abouThis result suggests that NB distributions may be too restrictive for an important fraction of expression profiles in large RNA-seq data sets. Among the possible causes underlying the lack of fit of those genes to an NB distribution, a clear candidate is that the presence of many samples can potentially introduce features such as zero-inflation or heavy-tails , a dispersion (ϕ) and a shape (a) parameter (see Methods) and include Poisson and NB distributions, among others, as particular cases [tweeDEseq package through the functions mlePoissonTweedie, dPT and rPT.We propose to address this problem by adopting the Poisson-Tweedie (PT) family of distributions  to modelar cases . An impoar cases . We haveEEF1A2 (ENSG00000101210), SCT (ENSG00000070031) and NLGN4Y (ENSG00000165246), respectively. Among these three genes, only NLGN4Y has been reported in the literature to have sex specific expresssion, while the other two are likely to lack such property since EEF1A2 is a housekeeping gene and SCT is an endocrine hormone peptide in chromosome 11 that controls secretions in the duodenum. Each plot shows the empirical cumulative distribution of observed counts as well as the parametric cumulative distributions obtained through the estimation of parameters of the methods compared in this paper under different configurations. Note that the estimated dispersion parameter ϕ is identical between the two sample groups for edgeR and DESeq (pooled) as these approaches estimate ϕ irrespective from the sample groups. The P-value for testing whether the data follow an NB distribution (H0:a=0), indicated above the legend, reveals that in several sample groups  this hypothesis is rejected (P<0.05). In those cases, methods based on the NB distribution produce dispersion parameters that do not fit the data as accurately as the PT distribution. More concretely, heavy-tails present in panels a,c severely hamper the estimation of the pure NB and the common dispersion. These can be improved using a parameter configuration more suited to large sample sizes. However, this results in a poor estimate of zero-inflation in panels c-e.Figure a of the PT distribution which provides further flexibility (see Methods). Using the LCL data processed with different normalization methods, we show in Figure a for every gene as function of its mean expression level, illustrating the huge variability of this parameter in RNA-seq count data. This wide range of values involves distinct possible distributional assumptions [cqn normalization method seems to make the largest impact on count data and, in this case, on the shape parameter.The main difference between the PT and NB distributions lies in the additional “shape” parameter umptions  beyond PP>0.2 and as clear-cut non-NB genes those with P<2−16. By mapping all these genes to the Gene Expression Barcode catalog [We have investigated whether this diversity of count distributions underlying RNA-seq data is related to different expression dynamics in genes. Using the test for the goodness of fit to an NB distribution (see Methods) we have considered as NB those genes that do not reject the null hypothesis at  catalog  (see MetP<1.24−6) for every normalization method  fail to do so when data are not simulated from NB distributions. As expected, these methods perform correctly when data are generated under an NB model  implemented through the function P-value distribution under the null hypothesis of no differential expression with real RNA-seq data by making repeatedly two-sample group comparisons within males and within females samples such that we recreate the null hypothesis of no DE with real RNA-seq data and no DE gene should be expected to be found. The raw P-value distributions from such analysis should ideally be uniform.In the second simulation study we have first assessed the accuracy of the P-value distribution by means of Q-Q plots displayed in Figure P-values should lie along the diagonal of the Q-Q plot. The figure, however, shows large discrepancies to this criterion by some of the methods and configuration parameters, indicating that they may not be adequate for large RNA-seq data sets.We have formally tested this hypothesis for every gene by means of a Kolmogorov-Smirnov (KS) goodness-of-fit test to a uniform distribution and examine the resulting tweeDEseq, is consistently closer to the diagonal than every other method throughout the two male and female comparisons and the two normalization methods. More informally, the visual inspection of the histograms of raw P-values given in Additional file tweeDEseq provides P-value distributions closer to the uniform under the null hypothesis of no DE simulated from extensively replicated real RNA-seq data.The method introduced in this paper, P-values under the null hypothesis can substantially affect FDR estimates of DE genes. We have also assessed the calibration of P-values and false discovery rates (FDR) with synthetic count data of similar dimensions to the RNA-seq LCL data set, concretely with p=20,000 genes and n=70 samples. Working with this type of data allows to assess FDR estimates for a known subset of DE genes under a variety of simulated scenarios, which we defined by considering the combination of three different amounts of DE genes  and three different magnitudes of fold-change . Similarly to [As other authors have shown, in the context of analysis of RNA-seq data with very limited sample size , small dlarly to , data weP-values for the two-sample DE test were obtained with each method and configuration parameters. Using the qvalue Bioconductor package [q-values and the fraction of DE genes from each P-value distribution. Q-values provide a nominal estimation of the FDR for each gene and in Figures q-values for the simulations with constant and variable library factors, respectively. The dashed diagonal line indicates a correct calibration of P-values whose nominal FDR equals the observed eFDR. Lines above the diagonal correspond to liberal DE analysis methodologies and below to conservative ones.From every simulated data set, raw  package  we estimtweeDEseq provides the best calibrated P-values in most of the simulated data sets.To facilitate the comparison of methods across all simulated data sets we have calculated the mean squared error (MSE) between the eFDR and the nominal FDR and ranked the methods by increasing MSE. In Tables q-values with the qvalue package [p being the total number of genes. In principle, more precise P-values both under the null and the alternative hypotheses should provide more accurate estimates of the number of DE genes. We show such an assessment for the previous simulations in Additional file tweeDEseq produces P-values that lead to the most accurate estimation of the number of DE genes, closely followed by edgeR with prior.df=1 when library factors are not held constant. In both settings, DESeq leads to extremely conservative estimates of the number of DE genes.The previous calculations of  package  provide tweeDEseq with real RNA-seq data, we have searched for genes changing significantly their expression between female and male individuals of the RNA-seq experiments on LCLs analyzed in this paper. Again, we have compared different normalization procedures and parameter configurations of edgeR and DESeq. Next to considering the raw un-normalized data and the data normalized with cqn, TMM normalization was used for edgeR and tweeDEseq, while DESeq was used with its own normalization method. We have used a single significance cutoff of FDR <0.1 at which genes were called DE. Since LCLs come from a non-sexually dimorphic tissue and are outside their original biological context, the fraction of sex-specific expression changes we could expect should be rather small.Assessing performance of DE analysis methods without using simulated data is a challenging problem due to the difficulty of knowing or ensuring the exact differential concentration of RNA molecules in the analysed samples. In this respect, sex-specific expression constitutes a useful system to assess the accuracy of DE detection methods due to the vast literature on genes contributing to gender-specific traits. For this reason, in order to illustrate the accuracy of F-measure which summarizes the trade-off between these two diagnostics.In an attempt to verify the accuracy of these lists of DE genes between female and male individuals, we searched for genes reported in the literature to be potential contributors to sexually dimorphic traits. This list of genes with documented sex-specific expression was obtained from genes in chromosome X that escape X-inactivation  and fromtweeDEseq provides better performance than the other competing methods under different parameter configurations. The improvement is small with respect to the second best-performing method and parameter configuration but we would like to emphasize that tweeDEseq does not require any informed decision on a parameter configuration, as opposed to edgeR and DESeq. To assess the robustness of this figure, we have run this comparative assessment with a more stringent filter on lowly expressed genes and, as Additional file tweeDEseq keeps performing better than the other methods, this time however only when data are normalized.In Figure tweeDEseq from the data normalized with cqn, which is when it yields the best precision-recall tradeoff. More than a half of genes in this list (32) are located in either the X or Y chromosomes and where the first 10 with largest fold-change contain 7 from the gold-standard set of MSY and XiE genes. Among the other 3, we find TTTY15, a testis-specific non-coding transcript from the Y chromosome and the other two lack functional annotation in Ensembl release 63.In Additional file P-values in DE genes should lead to higher reproducibility of significance levels of DE genes.The YRI LCL samples we have analyzed have been previously assayed using microarray chips  and thislimma[P-values from DE genes called in RNA-seq by each DE detection method to the − log10P-value units from genes called DE by limma. In Additional file limma in microarray data or by the other compared method in RNA-seq data. Although we can observe a significant linear relationship between P-values in every compared method, the low fraction of variability explained by the fitted linear model (R2<0.25) in every of those comparisons indicates a rather poor level of reproducibility for every method.With this purpose, we applied limma on the mx or the y-axis). There may be many reasons, unrelated to the DE detection method itself, why a gene is not called simultaneously DE in two completely independent RNA-seq and microarray experiments on the same biological material, such as different isoforms being probed in the microarray and summarized in RNA-seq or differences in sample preparation. Therefore, for our current goal of assessing reproducibility of DE detection methods, we believe it makes sense to restrict this comparison to those genes that are called DE by both, limma in microrray data and the corresponding method in RNA-seq data.A closer look to genes in that figure indicates that the lack of reproducibility mostly comes from genes called DE by one method and technology but not by the other (dots close to zero in either the tweeDEseq attains a significant (P<0.05) linear fit with respect to the P-values from limma with a level of reproduciblity (R2=0.6) substantially larger (46% increase) than the second best method (DESeq - PgO) with R2=0.41.We can find this restricted comparison in Figure tweeDEseq in RNA-seq data with the entire output DE genes obtained with limma in microarray data. In Figure XIST gene which shows the largest fold-change and significance level and corresponds to the X-inactive specific non-coding RNA gene which acts as one of the key regulators in silencing one of the copies of chromosome X in females. Blue and red circles denote MSY and XiE genes, respectively. As expected, all MSY and XiE DE genes report significantly higher expression in males and females, respectively, except for the XiE gene NLGN4X in RNA-seq, likely due to low expression from the inactive X chromosome in female samples [limma on this microarray data set is able to detect a few more such genes than tweeDEseq on RNA-seq data. Last, but not least, an important difference between the volcano plots of Figure Finally, we have carried out a comparison between the entire output of DE genes obtained with  samples . SurpristweeDEseq, for detecting DE genes and demonstrated with simulations that produces more accurate P-value distributions that lead to better calibrated q-values and FDR estimates.The increased amount of biological variability revealed by extensive replication in RNA-seq experiments brings new challenges to the task of identifying genes whose change in expression is both, biologically and statistically significant. In microarray data, large fold-changes derived from large data sets were nearly synonymous of statistical significance. The volcano plots in Figure tweeDEseq provides a better precision-recall tradeoff than the compared NB-based methods  analysis, could potentially benefit of models based on the PT-distribution. It is our perception that richer count data models of this kind will become increasingly necessary to draw accurate conclusions from data as technology brings us closer the actual biology of the cell.et al. (2010) [http://eqtl.uchicago.edu/RNA_Seq_data/unmapped_reads and pre-processed using the GRAPE pipeline [tweeDEseqCountData available at http://www.bioconductor.org under the name pickrell1.We have analyzed data from Pickrell . (2010)  that seq. (2010)  individupipeline . This pipipeline . Second,pipeline  with EnsedgeR-TMM was applied and when cqn was applied then 27,438 were kept  throughout the samples. The results shown in Additional file edgeR[cqn[DESeq method for DE analysis in the LCL samples, the TMM normalization procedure was replaced by its own normalization procedure.Three approaches to normalizing the table of counts from the LCL data have been considered. The first one is to work with the initial table of raw counts without any kind of normalization, the second one is to apply TMM  normalizedgeR package,edgeR[cqn BiocondutweeDEseq. In the case of DESeq raw counts were transformed into normalized counts only when used with the cqn normalization method.Raw counts were transformed into filtered and normalized counts for the purpose of producing MA-plots Figure , assessiedgeR-TMM normalization, counts were transformed following the steps that the function exactTest in edgeR takes: calculate normalization factors with the TMM method , estimate effective library sizes and adjust counts to effective library sizes obtaining non-integer normalized pseudocounts  which were subtracted by 0.5 and then raised to the smallest integers not less than these pseudocounts (ceiling). These steps are written together in the function normalizeCounts from the tweeDEseq package.In the case of cqn, normalization offsets are calculated by the function cqn as log2 RPMs, which are added to original raw log2 RPMs. These are rolled back to absolute numbers and “unlogged” obtaining non-integer normalized pseudocounts which, analogously to the edgeR-TMM case, were subtracted by 0.5 and then raised to the smallest integers not less than these pseudocounts (ceiling). The rationale behind subtracting 0.5 to the pseudocounts instead of directly truncating or raising to the next integer value, is to try to approach as much as possible the correct proportion of zero counts in the normalized data.In the case of edgeR, or with DESeq and its own normalization procedure, the specific recommendations made by the corresponding software authors were followed. More concretely, raw counts were not transformed in order to preserve their sampling properties and normalization adjustments entered the DE analysis through the corresponding normalization factors and offsets arguments within the functions that test for DE  and discarding those samples that either of the two reported diagnostics was considered below a minimum quality threshold. Third, using the RMA algorithm  implemented in the rma function from the oligo package with argument target=~core~, expression values were background corrected, normalized and summarized into Affymetrix transcript clusters. Fourth, most samples formed part of family trios and only samples belonging to father or mother were kept. Fifth, using the getNetAffx function from the oligo package, Ensembl Transcript identifiers well obtained for each Affymetrix transcript cluster identifier. Sixth, using the bioconductor package biomaRt, Ensembl Transcript identifiers were translated into Ensembl Gene identifiers, resolving multiple assignments by keeping the Ensembl Gene identifier that had a match in the Ensembl Gene identifiers forming the table of counts of the [The microarray LCL data from  was procs of the  RNA-seq removeBatchEffect function from the Bioconductor package limma[At this point an expression data matrix of 16,323 Ensembl Genes by 74 samples was obtained and using the scanning date of each CEL file, samples were grouped into 5 batches, out of which one containing only three male samples was discarded leaving a total of 71 samples distributed into 4 balanced batches across gender. Batch effect was removed by using the QR-decomposition method implemented in the age limma while kecqn package.To perform the analyses summarized in Figure To perform the analyses summarized in Figure http://jun2011.archive.ensembl.org with R and the biomaRt Bioconductor package. Gene length and G+C content annotations, used with the cqn normalization method, were obtained by downloading all human cDNAs from http://ftp.ensembl.org/pub/release-63/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh37.63.cdna.all.fa.gz and calculating the length and G+C content of the longest cDNA for each Ensembl gene.Functional annotations for Ensembl genes forming the tables of counts, were retrieved from The gold-standard list of genes with sex-specific expression was built with genes reported in the literature that, in one hand, escape chromosome X inactivation  and, on hgu133plus2.db Bioconductor annotation package, leading to 16,292 genes with expression breadth values. When two or more probesets mapped to the same gene, the maximum value was taken for that gene.The list of housekeeping genes was retrieved from the literature  and mapptweeDEseqCountData available at http://www.bioconductor.org under the keywords annotEnsembl63, genderGenes and hkGenes.All these functional data are included in the experimental data package et al. (2011) [R package tweeDEseq we have developed a fast implementation, written in the C programming language, of this recursive algorithm. We briefly describe here the PT family of distributions as well as how we have used it to analyze RNA-seq count data in the context of a differential expression (DE) analysis.Poisson-Tweedie (PT) distributions have been studied by several authors -34 and uet al. (2011) [Y∼PT be a PT random variable with vector of parameters θ=T defined in the domain Following El-Shaarawi . (2011) , let Y∼PY has a probability generating function (pgf) of the form: The PT random variable a≠0, while when a=0, then: when Using this parameterization, the following recursive algorithm can be used to compute the PT probability distribution function : where pi denotes the probability of observing i counts.and θ= to θ=, where μ denotes the mean, ϕ=σ2/μ is the dispersion index (σ2 is the variance), and a the shape parameter that is used to define some count data distributions that are particular cases of PT such as Poisson or negative binomial. The relationship between both parameterizations is the following: For the sake of interpretability, we reparameterize a=1) and negative binomial (NB) (a=0) but also other distributions that have been used to analyze count data such as Poisson-Inverse Gaussian (PIG) (a=−1) or Neyman type A (a→−∞). Therefore, the PT distribution family unifies several diverse count data distributions, including different overdispersed distributions such as NB or PIG. These distributions can model different scenarios as, for instance, a RNA-seq expression profile with a wide dynamic range leading to a heavy tail in the distribution. In such a case, PIG has a heavier tail than NB and this would make it more appropriate for such a gene. Note that an extremely heavy tail implies overdispersion, but the converse does not hold; hence the NB distribution is not adequate to model RNA-seq expression profiles of genes with a wide dynamic range due to their intrinsic biological variability [The PT model includes not only Poisson  [Given a certain parameterization Kokonendji . (2004)  prove thp is the shape parameter of that specific parameterization. It follows that, whereas the NB distribution is only able to capture a quadratic mean-variance relationship, the PT family is able to generalize this relationship to any order. As a result, it is more convenient to use the PT model when dealing with count data which presents variable overdispersion.where ygk be the number of counts for gene g in sample k, derived from pre-processing RNA-seq data. We assume that ygk follows the PT distribution: We need to estimate the parameter vector θg=μg,ϕg,ag, but we can estimate them from data by maximum likelihood when the sample size is sufficiently large so that it guarantees the desirable large sample properties of unbiasedness and minimum variance of the maximum likelihood estimate (MLE). In the Additional file In practice, we do not know the parameters optim function in R. In order to guarantee good convergence, we consider as initial parameters the moment estimates of μg and ϕg, and ag=0. We choose this value for ag because it corresponds to an NB model that is the natural cut-point of PT’s parameter space.We obtained the MLE H0:a=0 versus Ha:a≠0. Using a likelihood ratio test (LRT), the testing statistic is [In the framework of PT distributions we can formulate a test of the goodness of fit to an NB distribution by considering istic is n grows large, and it can be used to decide whether count data follow a NB distribution by means of a Q-Q plot . El-Shaarawi et al. (2011) [θ is an interior point of the parameter space θ, asymptotic normality of μ parameter we have that where . (2011)  indicateμA and μB, a two-sample PT-test for the mean with null hypothesis H0: log(μA)= log(μB), can be built by calculating the PT-statistic: Consequently, if we are interested in comparing the mean counts for two sample groups, denoted by T, follows a standard normal distribution under the null hypothesis. Therefore, the (1−α)% percentile of a N distribution is used to determine whether the observed differences between the two groups are statistically significant or not by providing a corresponding P-value that can be later on corrected for multiple testing using, for instance, Benjamini-Hochberg’s FDR [The PT-statitic, rg’s FDR .cqn package [P-value distribution for any given gene throughout the 100 bootstrapped data sets should be uniform.The results shown in Figure  package . In prinϕg for every gene g at random from a gamma distribution Gamma and means according to three different fold-changes  where half of the genes were up-regulated and the other half down-regulated. The λgi Poisson parameter for every gene g and sample i was drawn at random from a gamma distribution Gamma) with a=fμgk/(ϕ−1) and f≈N corresponding to library factor which was either constant (σ=0) or variable (σ=0.5). Counts were simulated for each gene g from the resulting mixture gamma-Poisson distribution with parameters λgi for each sample i. Note that the resulting marginal distribution from the gamma-Poisson is a negative-binomial.The simulations shown in Figures Project name:tweeDEseq•Project home page: http://www.bioconductor.org/packages/release/bioc/html/tweeDEseq.html•Operating system(s): Platform independent•Programming language:R and C•Other requirements:R 3.0.0•Licence: GNU GPL••Any restrictions tu use by non-academics: no restrictionsThe authors declare that they have no competing interests.JRG and PP conceived the idea of modelling RNA-seq count data using PT family of distributions. ME programmed the recursive algorithm to compute PT probability distribution, performed simulation studies, and created the R package jointly with JRG and RC. PP and JRG proposed the statistical test for detecting DE genes. DG preprocessed the RNA-seq data. RC, ME and JRG analysed the data and wrote the paper. The project was supervised by JRG. All authors read and approved the final manuscript.Scripts. ZIP file (.zip) containing all scripts, in the form of Sweave vignettes, to reproduce the results shown in this paper, including one copy of the resulting PDF file. Please read first through the README file contained in this tar ball in order to understand how to run the scripts.Click here for fileSupplementary materials. PDF file including supplementary figures and tables.Click here for file


With an abundant amount of microarray gene expression data sets available through public repositories, new possibilities lie in combining multiple existing data sets. In this new context, analysis itself is no longer the problem, but retrieving and consistently integrating all this data before delivering it to the wide variety of existing analysis tools becomes the new bottleneck.inSilicoMerging R/Bioconductor package which, together with the earlier released inSilicoDb R/Bioconductor package, allows consistent retrieval, integration and analysis of publicly available microarray gene expression data sets. Inside the inSilicoMerging package a set of five visual and six quantitative validation measures are available as well.We present the newly released https://insilicodb.org/app/].By providing (i) access to uniformly curated and preprocessed data, (ii) a collection of techniques to remove the batch effects between data sets from different sources, and (iii) several validation tools enabling the inspection of the integration process, these packages enable researchers to fully explore the potential of combining gene expression data for downstream analysis. The power of using both packages is demonstrated by programmatically retrieving and integrating gene expression studies from the InSilico DB repository [ An increasing amount of gene expression data sets is becoming available through public repositories , NONE and XPN , COMBATal Bayes ), DWD (Dmination ), GENENOlization ). For a lization .COMBAT was based on the orginal R code made available by the authors on their website http://www.bu.edu/jlab/wp-assets/ComBat/Download∖_files/ComBat.R. The implementation of DWD uses the kDWD function from the corresponding package , providing a clear quantification of the batch effect or of the quality of the data integration process.where inSilicoMerging package based on an example analysis of two lung cancer data sets assayed on different platforms. We will discuss a complete workflow consisting of search and retrieval of data, followed by merging and corresponding validation.In the following sections we demonstrate the ease and power of the inSilicoDb package to search for data sets containing lung cancer samples, assayed on two different platforms: Affymetrix Human Genome U133A Array (GPL96) and Affymetrix Human Genome U133 Plus 2.0 Array (GPL570). We restrict our search to data sets with the number of samples between 100 and 150 (the select function). All data sets retrieved from the InSilico DB database are consistently preprocessed using frozen RMA );control lung cancer4958> table(pData(eset96));control lung cancer6060Disease annotation label and were selected for this example because they are almost equally divided in both disease subdivisions as can be seen in the previous code example.Both studies contain normal and tumor samples with a common inSilicoMerging package can now all be applied on the retrieved ExpressionSets by calling a single function merge. For example, simple combination without batch effect removal can be done as follows: > merge;with esets a list of ExpressionSet objects. The method parameter can be one of the options described above: BMC, COMBAT, DWD, GENENORM, NONE or XPN.The different merging methods implemented in the NONE) is applied and we also build a merged set using the COMBAT method, hence creating two different ExpressionSet objects containing merged data set:Next to the merged set where no transformation (> library(inSilicoMerging);> esets = list;> eset_NONE = merge;> eset_COMBAT = merge;> table(pData(eset_NONE))control lung cancer109118plotMDS function.Five visual validation methods are provided by the package in order to immediately enable a first inspection of the merged data sets. For this example, a first global visual inspection of the two merging approaches applied is demonstrated via the > par);> plotMDS");> plotMDS;targetAnnot and batchAnnot arguments, respectively. No extra manual step was required.In Figure COMBAT on the right part of Figure We can observe the effect of using plotGeneWiseDensity function for a specific gene.Similarly, on a local scale we demonstrate the use of the > par);> plotGeneWiseDensity");> plotGeneWiseDensity;MYL4 was arbitrary selected and its estimated density across both studies is shown in Figure COMBAT method makes the distributions of this specific gene across the two studies more similar to each other. Note that the magnitude of the batch effect at the gene level is very dependent on the specific gene selected.Gene R code illustrating the five visual inspection methods is provided in the Additional file inSilicoMerging, together with inSilicoDb, on three axes: data access, data merging and validation algorithms.We compare GEOQuery  is an exinSilicoMerging package have been implemented prior to this package in different programming and/or script languages . A major advantage of inSilicoMerging is the consistent interface for all merging methods and its integration in the R/Bioconductor framework. We will briefly discuss the added value of our package compared to those already available in R.Several of the batch effect removal techniques included in the COMBAT method provide an implementation on their website which allows the input of known covariates (besides batch id). They work with several matrices containing the numerical data, and relevant annotations and covariates. Within our package, all this information is bundled and encoded in the ExpressionSet structure.The authors of the DWD method recently have released the DWD R/Bioconductor package [inSilicoMerging package in order to obtain a merged data set as result.The authors of the  package . This paCONOR package which is available through the CRAN repository [The pository , most clEvaluating and validating the results of batch effect removal methods is perhaps as important and difficult as the batch effect removal process itself. Without good and reliable evaluation tools, these methods could result in an increased distortion of the data, introducing serious errors in the results of any downstream analysis performed.inSilicoMerging package.To the best of our knowledge no explicit software has been developed for validating the quality of the batch effect removal process of microarray gene expression data sets. The collection of both qualitative and quantitative validation methods is therefore a unique property of the inSilicoMerging R/Bioconductor package enables the integration of batch adjustment in any workflow dealing with large-scale analysis of gene expression data, paving the road towards unlocking the potential of the massive amount of publicly available microarray data. Moreover, its seamless integration with the R/Bioconductor framework for further analysis and the inSillicoDb package for retrieval of consistent data, minimizes the need for manual intervention. As a result transparency, trackability and reproducibility of large-scale gene expression analysis is significantly increased.The Project name:inSilicoMergingProject home page:http://www.bioconductor.org/packages/release/bioc/html/inSilicoMerging.htmlOperating system(s): Platform independentProgramming language: ROther requirements: BioconductorLicense: GPL-2Any restrictions to use by non-academics: none1http://www.insilico.ulb.ac.beThe authors declare that they have no competing interests.JT, SM and CL designed and implemented the inSilicoMerging package and drafted the manuscript. JT, DS and RD designed and implemented the inSilicoDb package. AC, CM and DWS were involved in the development of the back-end software. VdS provided the expert-curated annotation data of the underlying database tool. HB and AN conceived the study and participated in its design and coordination. All authors read and approved the final manuscript.code_example.R R script using both inSilicoDb and inSilicoMerging packages generating Figure R script. Click here for file


Gene expression profiling has gradually become a routine procedure for disease diagnosis and classification. In the past decade, many computational methods have been proposed, resulting in great improvements on various levels, including feature selection and algorithms for classification and clustering. In this study, we present iPcc, a novel method from the feature extraction perspective to further propel gene expression profiling technologies from bench to bedside. We define ‘correlation feature space’ for samples based on the gene expression profiles by iterative employment of Pearson’s correlation coefficient. Numerical experiments on both simulated and real gene expression data sets demonstrate that iPcc can greatly highlight the latent patterns underlying noisy gene expression data and thus greatly improve the robustness and accuracy of the algorithms currently available for disease diagnosis and classification based on gene expression profiles. With the rapid development of high-throughput technologies, gene expression profiling based on microarrays or next-generation sequencing techniques have been widely applied in clinical research . The bigTo surmount the difficulties caused by the curse of dimensionality problem, various approaches have been proposed and developed on different levels of the experimental and analytical protocol. On the experimental level, increasing the number of samples involved in the studies to enhance the statistical power and measuring the expression levels of a set of selected genes involved in a certain signaling pathway are two possible solutions. However, increasing the number of samples is generally difficult because of eligible sample limitations and the huge cost of high-throughput measurements. Furthermore, measuring genes involved in only selected pathways can introduce artificial biases. In fact, the rapid development of high-throughput technologies produces more and more information about samples, allowing unbiased investigation of the molecular truth of various biomedical phenomena. For example, RNA-Seq (deep sequencing the transcriptomes of samples) can detect expression levels of novel genes that are not annotated in the reference genomes, compared with the traditional gene expression profiling microarrays .k-means whei and i. Generally, we can define the t-order correlation features of sample i as follows:We name All the features i and t; (ii) i, j and t; and (iii) i, j and t. Below we compose three toy gene expression data sets of six genes (for example) to demonstrate numerically other properties of sample correlation features in an oversimplified setting.Because the correlation features are defined based on Pearson correlation coefficients, we named the computational method iterative Pearson correlation coefficients (iPcc). It can be seen that correlation features have several obvious properties: (i) In toy Data set 1, three samples are composed in which samples 1 and 2 (s1 and s2) represent two different classes, whereas sample 3 (s3) is more similar to sample 1 . By calcIn toy Data set 1, the initial feature profile of s3 is closer to that of s1, and the high-order correlation feature profile of s3 converges to the same as that of s1, suggesting that correlation features can enlarge weak pattern underlying the raw data set. In toy Data set 2, we demonstrate that correlation features can also preserve independent relationships among samples. We compose an initial feature profile for s3 that has equal distance to those of s1 and s2. The first-order correlation feature vector of s3 is , which is vertical to both of the feature vectors of s1 and s2. The second-order or higher correlation feature vectors of s3 are the same as the first-order correlation feature vector of s3, suggesting that the independence of s3 from s1 and s2 is conserved during the iterative correlation feature extraction.In toy Data set 3, we conceive two different classes of unbalanced class sizes . We further conceive s4 that has equal distance from s1, s2 and s3. The first-order correlation feature vector of s4 is , suggesting that s4 forms an independent class from s1, s2 and s3. However, the second-order correlation feature vector of s4 indicates weak similarity to that of s1. The higher-order correlation feature vectors of s4 highlight the similarity of s4 to s1 until their Pearson correlation coefficient converges to 1. Thus, correlation features not only can enlarge the weak patterns underlying the raw data set and preserve the balanced independent relationships, but also can highlight those patterns hidden in the higher feature space.σ from 0.1 to 0.5. We simulated 0-, 10-, 50- and 100-fold irrelevant genes according to a normal distribution with mean 0 and the same standard deviation as those six (2 × 3) relevant genes and added into the data sets. For each simulated data set, we applied our method and demonstrated the sample relationship by visualizing the sample similarity matrices in We evaluated our method on both simulated and real gene expression data sets. First, we simulated a series of artificial gene expression data sets to show the properties and performance of the correlation features against the magnitude of uncertainty of gene expression measurements and the ratio of irrelevant genes to relevant genes. We designed three different sample classes. For each sample class, we generated two relevant genes with expression levels randomly sampled from a normal distribution with mean 1 and given a standard deviation To evaluate the advantage of correlation features to real-world applications, we examined the clustering and classification performance of a series of widely used algorithms on three real gene expression data sets. The first real gene expression data set was the gene expression data set of leukemia  in whichThe second data set was a prostate cancer gene expression data set  downloadThe third data set contains two series of gene expression profiles of normal and psoriasis samples, downloaded from the NCBI database with accession numbers GSE13355 and GSE14905 ,40. Therk-means to implement disease class discovery. k-means is a classic clustering algorithm that partition n samples into k clusters in which each sample belongs to the cluster with the nearest mean , the original features could only reveal weak correlations between samples from the same classes at observed . Howeverk-means and the naïve Bayes classifier on the simulated data sets. For all four simulation data sets with 0-, 10-, 50- and 100-fold noisy genes, k-means generated more accurate clustering results (average accuracy in 1000 runs) in general based on the correlation features than those based on the original features .We tested the clustering and classification algorithms features . When σ σ = 0.1, 0.2 or 0.3, the prediction accuracy approached 1 on both the original features and correlation features for all four simulated data sets. When σ = 0.4 or 0.5, the prediction accuracy increased from 86 to 90% when 10-fold noisy genes were added, from 88 to 90% when 50-fold noisy genes were added and from 71 to 80% when 100-fold noisy genes were added.We tested the performance of the naïve Bayes classifier by leave-one-out cross-validation evaluations on the four simulated data sets. The naïve Bayes classifier generated more accurate classification results on the correlation features (with order k-means) and classification (by the naïve Bayes classifier) accuracy on both the original features and the correlation features. We applied the Student’s t-test to those accuracy values and found that no significant differences were observed between the original features and the correlation features for both clustering and classification.The performance of iPcc on the completely random data sets suggested that the clustering and classification accuracy based on those correlation features generated by iPcc was similar to that on the original features. For each type of statistical distributions, we generated 1000 completely random data sets. For each data set, we calculated the clustering  by iPcc. The sample similarity matrix revealed by the original 3571 features weakly suggest the existence of two big groups: the AML group (samples 1–24) and the ALL group (samples 25–72), in which sample 25 was wrongly indicated and is in fact an AML sample . The ALLHowever, the sample similarity matrix constructed based on the first-order correlation features by iPcc strongly indicated the existence of the AML and ALL groups . In the k-means. We ran k-means (implemented in Matlab R2011b) 1000 times on the original features to cluster the samples into two groups. Compared with the true disease classes, we found that 357 runs produced accuracy of <60%, 70 runs produced accuracy of between 60 and 90% and 573 runs produced accuracy of >90%. However, 1000 runs of k-means on the first-order correlation features constructed by iPcc produced accuracy of >90%  and inter-class  sample similarity values based on the original features was 0.0702. However, the differences on the first-, second-, third-, fourth- and fifth-order correlation features were 0.5002, 0.9755, 1.2217, 1.3316 and 1.3567, respectively. In 1000 k-means on both the original and correlation features of the GSE13355 data set obtained 99% accuracy. For GSE14905, the clustering accuracy by k-means was 98% on the original, the first-, second-, fourth- and fifth-order correlation features and 99% on the third-order correlation features. The leave-one-out prediction accuracy of the naïve Bayes classifier for the GSE13355 data set was 96% on the original features, 99% on the first- and second-order correlation features and 98% on the third-, fourth- and fifth-order correlation features. The leave-one-out prediction accuracy of the naïve Bayes classifier for the GSE14905 data set was 96% on the original, first- and second-order correlation features, and 98% on the third-, fourth- and fifth-order correlation features.We applied iPcc on the psoriasis data sets to show its performance in situations with batch effects. First, we applied iPcc to examine the sample relationships for GSE13355 and GSE14905 individually. The results suggested that those psoriasis samples were weakly discriminated from the normal samples based on the whole 54 676 original features for both of the two data sets. On greatly reducing the size of the correlation features, the distinctiveness between psoriasis from normal samples was further underlined and confirmed. Clustering by k-means stabilized at 59%. On the second-, third-, fourth- and fifth-order correlation features, the clustering accuracy by k-means stabilized at 81%. The leave-one-out prediction accuracy by the naïve Bayes classifier on the original features was 80%, similar to the clustering accuracy. However, on the first-order correlation features, the leave-one-out prediction accuracy increased to 97%. On the second-order correlation features, the accuracy further increased to 98%. On the third-, fourth- and fifth-order correlation features, the accuracy decreased to 96, 91 and 91%, respectively.However, we observed that the original features suggested that the sample relationship of the combined data set  was dominated by three clusters: the normal samples in GSE13355, the psoriasis samples in GSE13355 and all the samples  in GSE14905. Correlation features constructed by iPcc confirmed the observation based on the original features, and further suggested that the psoriasis samples in GSE13355 were more similar to those samples in GSE14905 . We clusi and j were done in the same laboratory, otherwise setting i and j were done in different laboratories. We generated the first-order correlation features k-means on the third-, fourth-, fifth- and sixth-order updated correlation features obtained stable 86, 94, 97 and 98% accuracy, significantly larger than the clustering accuracy on the original features or the un-updated correlation features. The leave-one-out prediction accuracy by the naïve Bayes classifier also stabilized at 98% on the fourth-, fifth- and sixth-order updated correlation features.Although the gene-wise normalization step tried to remove batch effects, the above observation suggested that batch effects still influenced the global sample relationship. Here, we show how iPcc removes the batch effects based on correlation features. Because we already knew the experiments were done in two different laboratories, we constructed a sample similarity matrix ratories . One thok-means on the original features produced 490 runs with accuracy of 98% and 510 runs with accuracy of 81%. The leave-one-out prediction accuracy of the naïve Bayes classifier on the original features was 98%. Thus, our iPcc-based batch-effect-removal method outperforms, or at least is comparable with, the performance of F-test-based feature selection methods.We compared the batch-effect-removal method by iPcc with feature selection methods, which are frequently used to improve the quality of clustering and prediction. We selected the top 1000 informative genes by F-test and evaluated sample relationship, clustering accuracy and prediction accuracy. The average difference between the intra- and inter-class sample similarities was 0.8592 on the original features. One thousand runs of k-means on the first-order correlation features yielded 598 runs with an accuracy of 98% and 402 runs with an accuracy of 84%. On the second-order correlation features, the accuracy of 1000 runs of k-means stabilized at 97%. The leave-one-out prediction accuracy of the naïve Bayes classifier on the original features and all the correlation features was 98%. Thus, iPcc can also enhance the power of feature-selection methods.By applying iPcc on the 1000 genes selected by F-test, we obtained the intra-class versus inter-class difference values on the first-, second-, third-, fourth- and fifth-order correlation features of 1.2800, 1.3629, 1.4095, 1.4271 and 1.4251, respectively, larger than that on the originally selected genes. One thousand runs of With the development of high-throughput technologies, thousands of genes can be measured simultaneously by microarrays or next-generation sequencing technologies. This facilitates comprehensive characterization of the biomedical states of samples but also introduces much irrelevant information. In this study, we proposed a novel feature extraction method, named iPcc, to extract the underlying patterns from noisy data sets through introducing the ‘correlation feature’ concept with iterative Pearson correlation coefficients. Simulations and evaluations on real data sets demonstrate that iPcc greatly improves the disease class discovery and prediction based on the gene expression profiles.The effectiveness of iPcc may partially originate from reduced dimensionality. In the high-throughput experiments, the number of samples is generally much lower than the number of measured genes, resulting in the ‘curse of dimensionality’. This feature extraction method reduced the dimensionality from thousands of genes to tens of samples. The reduced dimensionality allows rapid and accurate computation of the global optimum of many clustering and classification algorithms and thus improves accuracy. The effectiveness of iPcc may also originate partially from the introduced ‘correlation features’. Without rephrasing the gene expression levels of samples, correlation features introduced by iPcc used the similarity between samples, which is independent of the precise gene expression levels but is related to the trends of the profiles. These features are similar to clinicians’ clinical experience, which is helpful in diagnosis and prognosis. The effectiveness of iPcc is dependent on the measure of Pearson correlation coefficients. We evaluated those ‘correlation features’ computed by the Spearman correlation coefficients, and found that ‘iScc’ does not have the same effect. We will further investigate the differences between Pearson correlation coefficients and Spearman correlation coefficients in the iterative computation setting. The effectiveness of iPcc also relies on rapid convergence ,43. For Because the computation of iPcc does not require sample class information, it is essentially unsupervised. When confounding factors exist, iPcc can also enlarge the biases embedded within the gene expression data set instead of the subject information, just like the batch effects in the psoriasis data set. Thus, it works best on those data sets with irregular noise. However, if the confounding factors are known, iPcc provides an efficient means to remove the effects of those factors. However, iPcc is not an independent algorithm for disease class discovery or prediction. It provides an effective means to underpin the underlying patterns embedded within the gene expression data sets from the feature extraction perspective. Therefore, it can be used in combination with other clustering, classification, feature selection and feature extraction algorithms, as demonstrated in the results section.We also observed that multiple sample classes could converge to two classes in the high-order correlation feature space. For example, the B-cell ALLs and T-cell ALLs became undistinguishable in the high-order correlation feature space. This is closely related to the class relationship. Because T-cell ALLs are more similar to B-cell ALLs than to AMLs, iPcc merges T-cell ALLs with B-cell ALLs when the iteration number is adequately large. Because iPcc is unsupervised, it would also enlarge noise in high order of correlation features. Therefore, the optimal iteration number of iPcc is dependent on the specific noise level and the information users needed. Because iPcc generates correlation features, a certain order of features can also be jointly used with the original and other-order correlation features.In conclusion, we proposed a novel feature extraction method that is noise-resistant, i.e. iPcc, based on iterative Pearson correlation coefficients for disease class discovery and prediction based on high-throughput gene expression profiles. Evaluations on both simulated and real gene expression data sets suggest that iPcc not only can highlight the patterns underlying high-dimension gene expression files but also can greatly improve the accuracy of disease class discovery and prediction based on clustering and classification algorithms. This feature extraction method is expected to be a useful tool to boost the development of clinical diagnosis and prognosis.National Natural Science Foundation of China .Funding for open access charge: Conflict of interest statement. None declared.


Folsomia candida, a species often used for ecotoxicological testing. Animals were exposed to two chemically distinct compounds, being a metal (cadmium) and a polycyclic aromatic hydrocarbon (phenanthrene). We investigated the affected molecular responses resulting from either treatment and developed and validated 44 qPCR assays for their responses using a high throughput nano-liter RT-qPCR platform for the analysis of the samples.To incorporate genomics data into environmental assessments a mechanistic perspective of interactions between chemicals and induced biological processes needs to be developed. Since chemical compounds with structural similarity often induce comparable biological responses in exposed animals, gene expression signatures can serve as a starting point for the assessment of chemicals and their toxicity, but only when relevant and stable gene panels are available. To design such a panel, we isolated differentially expressed gene fragments from the soil arthropod 2s between 0.82 and 0.97) using gene expressions of 25 important qPCR assays correctly predicted the nature of exposures of test samples.Suppressive subtractive hybridization (SSH) was used to retrieve stress-related gene fragments. SSH libraries revealed pathways involved in mitochondrial dysfunction and protein degradation for cadmium and biotransformation for phenanthrene to be overrepresented. Amongst a small cluster of SSH-derived cadmium responsive markers were an inflammatory response protein and an endo-glucanase. Conversely, cytochrome P450 family 6 or 9 was specifically induced by phenanthrene. Differential expressions of these candidate biomarkers were also highly significant in the independently generated test sample set. Toxicity levels in different training samples were not reflected by any of the markers' intensity of expressions. Though, a model based on partial least squares differential analysis (PLS-DA) (with RMSEPs between 9 and 22% and RFor the application of molecular bio-indication in environmental assessments, multivariate analyses obviously have an added value over univariate methods. Our results suggest that compound discrimination can be achieved by PLS-DA, based on a hard classification of the within-class rankings of samples from a test set. This study clearly shows that the use of high throughput RT-qPCR could be a valuable tool in ecotoxicology combining high throughput with analytical sensitivity. In the field of environmental sciences a high throughput molecular research often called 'ecogenomics' ,2 has evTraditionally, the ecotoxicological approach is focused on the effects of different concentration levels of chemical compounds on organisms, rather than on the molecular and cellular mechanisms underlying these effects. In contrast, ecogenomics aims at studying genome-wide molecular biological processes in relation to toxicity and hence has a more mechanistic approach. Such an approach at the level of affected cellular processes and genetic response pathways may give new insights into the main hazards to human and environmental health, and may support the classification by hazard and the authorization of new and existing chemicals. In turn, this may aid the design of new highly selective environmental chemicals less hazardous for non-target species.It is suggested that molecular biomarkers based on gene induction, in combination with conventional endpoints, can provide robust insight of the dose responses and mechanistic underlying effects of unknown chemical compounds . StudiesFolsomia candida (Collembola: Isotomidae). RT-qPCR has a higher dynamic range than microarrays, which also suffer from inaccurate mRNA quantification due to the fact that detection relies on probe hybridization. The Fluidigm/Biomark Dynamic Array is an innovative platform exceeding the possibilities of sample throughput of microarrays, with much more accurate mRNA quantification  phenanthrene, using the ecotoxicological model organism ion  and theTo acquire an initial collection of candidate markers, the 'open' technique of suppressive subtractive hybridization (SSH)  was usedF. candida exposed to different concentrations of cadmium and phenanthrene in soil. We verified the analytical performance and applicability of the high throughput platform for eco-toxicogenomic application. Moreover, flexibility in test setup and low amounts of required input cDNA made this application very convenient for the analysis of our experiments.We performed initial testing and validation of 44 markers in EV473060 - EV481745) or Collembase [-5); these levels are commonly observed for SSH libraries  into unllembase ) shows tllembase . Annotatg 56% in ; 55% in g 56% in  and 30% g 56% in ).GO terms could be assigned to approximately 35% of all clusters using the Annot8r_blast2GO script  hierarchical maps for each root class, the results for the two stresses could be compared easily in a visually attractive way , which is a measure for the relative abundance of that particular term in the given dataset compared to a random situation. Also included are any terms that were interesting because they showed high difference in occurrence ratio between the phenanthrene and cadmium libraries . This explorative approach shows the trends  of gene classes that are changed most in at least one of the SSH libraries, which is actually a simplified representation of the DAG maps.To compare the relative abundance between the two SSH libraries, significantly enriched p < 0.01) GO terms from the GOEAST analysis, were again imported in CateGOrizer for a counting within each collection; the phenanthrene, cadmium and the normalized unstressed set. Table  GO termsTable 2 odds ratio 1.7, Additional file Genes related to membranes were found to be enriched in the category 'cellular component' in both the cadmium and phenanthrene sets. However, in the cadmium set this term points towards vacuolar membranes, the cis Golgi and the mitochondrial inner membrane, while in phenanthrene it most likely concerns the  plasma membrane and the endomembrane structures of the endoplasmic reticulum (ER)  . Other eQuantitative PCR markers were developed Table  for diff2 (2) = 40.97, p < 0.0001, followed by Dunn's multiple comparison test), implying that the enrichment for upregulated genes was successful  = 225.1, followed by Bonferroni post hoc testing p < 0.05, Figure When looking at the markers individually . Gene expression data was therefore correlated to the treatments by the multivariate method of partial least squares discriminant analysis (PLS-DA) , using t2 values  of the correlations between predicted and known (measured) exposure conditions were respectively 0.90 for cadmium, 0.81 for phenanthrene and 0.79 for the control group.Uncertainties  reflected by the root mean square error (RMSEP) were 16% for cadmium, 23% for phenanthrene, and 23% for the control group. RFor each treatment a list of 'important variables' (VIPs) (significant in the uncertainty testing) is calculated from the loading weights and regression coefficients of the x-variables. In the correlation loadings plot Figure  all 25 V2: 9%, 0.97; cadmium, 22%, 0.82; phenanthrene, 20%, 0.84; control). Along with a control treatment, these samples included cadmium and phenanthrene treatments which concentrations equaled to the EC50 values for reproduction. For phenanthrene, this concentration was lower than those of the samples used for building the model. For cadmium, the concentrations were similar, but two different exposure times (2-days versus 4-days) were tested. Prediction intervals of the predicted value ± RMSEP were used to classify samples. For cadmium, the model was quantitatively able to predict two of the cadmium samples and one of the control samples. No samples were attributed to the phenanthrene class. An alternative approach is to use a 'hard' classification, which assigns each sample to the best fitting class based on the ranking of within class values. This approach resulted in eight out of ten correctly predicted controls, four out of six phenanthrene samples, as well as eight out of ten cadmium samples; the exposure time of the cadmium samples did not affect the ranking of a sample.A test set of samples from a second experiment  was used to evaluate the performance of a PLSR in predicting exposure conditions of independent samples. The model was slightly modified and only the VIPs were included in the calculation, whereas the rest of the markers were included passively, by giving them a very low weight to remove the influence on the model while still showing the correlations to other variables. This did improve the (2PC) model's fit slightly  groups in proteins . One of 2+) transporters and also vesicular transmembrane processes are mediated by exocytic pathway transporters [NIPSNAP1 homolog, which is known to be an inhibitor of the specific Ca2+ transporter TRPV6 [NIPSNAP homolog was identified in an SSH enrichment study conducted for the sister species Orchesella cincta [Orchesella cincta, a metallothionein is induced by cadmium [F. candida. Earlier experiments by our group using degenerated primers neither succeeded in the isolation of a metallothionein gene for F. candida. However, very recently Nakamori et al. [F. candida, which appears to be very different from the other collembolan metallothioneins isolated so far.We found several ABC transporters and copper pumps  induced by cadmium. In yeast, cadmium uptake is provoked by calcium  which were linked to GO terms for cellular events of mitochondrial distribution and inheritance. Cannino et al. [In our SSH gene libraries, several mitochondrion related ESTs were enriched . We . We O. ced e.g. ,51). Lik. LikO. ced . This is believed to be the result of reversible and non-specific disturbance of membrane integrity and function resulting from the partitioning of the chemical into biological membranes ,54. MetaThe majority of enriched GO terms were related to the biotransformation process. For example many retrieved genes relate to the Endoplasmic Reticulum (ER), which is the organelle where the main steps of this process occur. Different glutathione S-transferases involved in chelation of oxygenized metabolites to glutathione, reduce the free flow of reactive metabolites, which is called phase II biotransformation . The lasCYP1A, the vertebrate homolog of the insect xenobiotic biotransformation cytochrome P450 9, was found by Søfteland et al. [The aim of the study was to find biomarkers capable of discriminating the nature of the chemical treatments, as well as the concentration levels of the exposures. The latter was unrealizable within the conditions of this experiment. Further research will therefore aim for the identification of dose related responses, by studying inductions of the markers in response to lower, environmentally relevant, concentration ranges. In acquiring treatment-specific gene fragments SSH proved to be a valuable method. On average, the set of qPCR markers developed from fragments in the SSH libraries were induced by the treatments. A small cluster of markers was found to be cadmium-specific, including an inflammatory response protein and an endo-glucanase. Phenanthrene-specific was cytochrome P450 from family 6 or 9. Lee et al.  summarizd et al.  after PCUsing a high throughput qPCR system, made it feasible to measure a set of 44 qPCR markers and perform a multivariate analysis on the expression levels using PLSR. A similar approach was undertaken successfully by Wang et al. in designing a qPCR based application for the prediction of progression of bladder cancer, where a panel of 57 genes resulted in a clinically feasible test [We evaluated the diagnostic power of our set of genetic markers by use of a new test set of samples. With an uncertainty level of nearly 20%, the model lacked the capacity to predict particularly the lower concentration phenanthrene treated samples. With validation parameters comparable to the controls, we assume that this was primarily the effect of the concentration difference, rather than the uncertainty level. Molecular methods for ecotoxicological applications in soil must be able to handle the relatively high intrinsic variation in expression data caused by the heterogenic nature of soils and therefore in exposure conditions in test setups. Enhanced predictive power may be acquired when dose response related markers can be included, and the model is trained with samples that include different concentration levels. Noteworthy is that the goodness of fit of the cadmium samples in the test set was not correlated with a difference in exposure time (2 or 4 days). Unfortunately, the effect of exposure time was not tested for phenanthrene.The possibilities and added value of a multi-marker approach are nicely demonstrated in a study of Ståhlberg et al. , who meaF. candida was kept in plastic containers with a water-saturated plaster of Paris bottom containing 10% charcoal at 20°C in a 12:12 light dark regime. The animals were fed dried baker's yeast (Dr. Oetker) ad libitum. For all exposures, adult animals of approximately one month of age were used.50;7d), corresponding to a concentration of 840 mmol/kg d.w. LUFA 2.2 reference soil). Surviving animals exposed to this concentration for 7 days were used for the molecular analysis. For cadmium the level of effect was intended not to intervene with the normal biological functioning of the organism on the longer term [50 pore water;28d, for reproduction [For the SSH library construction, 40-50 exposed adult animals  were used. The level severity of the SSH treatments was to cause reversible and irreversible cell injury and therefore to trigger a diversity of stress controlling pathways. For phenanthrene a high level of effect was chosen, 50% survival over the exposure duration of 7 days (LCger term . Therefooduction ). ControTotal RNA  was extracted according manufacturer's protocol and pooled for reverse transcription in the SMART cDNA synthesis procedure (Clontech SMART PCR cDNA Synthesis Kit). SSH EST libraries ,40 were -5), followed by GO term assignment when possible .From the SSH libraries of phenanthrene and cadmium responsive gene clusters, RT-qPCR assays were designed for clusters of sufficient length  and 0.42 mmol/kg d.w.  for this experiment. Exposure times were 48h and 96h. For phenanthrene the concentration was lower than the training experiment; 0.22 mmol/kg d.s.  phenant[50;28d, ) and 1.0 and Mol  between oduction ), with aTotal RNA  of ten exposed adult animals was pooled and used for reverse transcription with approximately 0.3 μg of total RNA input, using the M-MLV reverse transcriptase (Promega) and oligo d-T primer. cDNA was diluted 5×.YWHAZ, SDHA)  assays and two reference genes  were mixed with 2.5 μL DA assay loading reagents (Fluidigm). QPCR reactions were performed in quadruplets for each sample-assay combination, using a 3-step PCR program . Assays were validated for this platform before use, by correlating expression levels of samples run on both an Opticon qPCR system  and the Biomark qPCR-chip-platform and visual inspection of melting curve analyses and fluorescence curves.Two slots were left empty as no template control. Preamplification was done according to the manufacturer's protocol (Fluidigm) and diluted 5 times. 1.5 μL preamplified cDNA was mixed together with 4.5 μL sample mix. Sample mix was prepared using 0.6 μL PCR-buffer 20 mM, (Roche); 0.12 μL dNTPs, 10 mM; 0.12 μL FastStart Taq polymerase (Roche); 0.3 μL DA sample loading reagents (Fluidigm); 0.12 μL 50× Rox (Invitrogen); 0.6 μL 10× EvaGreen (Biotum); 2.64 μL ddHAnnotation of the SSH libraries was performed using GO terms generated by Partigene, Collembase . Followi2 transformation of the data. Assays that did not perform well in melting curve analysis or showed failure of performance for multiple samples, as well as data with high Ct-values (> 30) were excluded from analysis. Significance of expression levels was determined by ANOVA, using Bonferroni post hoc testing.For all quantitative analyses, Genex Light v4.3.5  was used2 data. We used the PLS-DA methods with 24 missing data points filled in during analysis, using the standard setting of the software . The PLS2 model was tested by full cross validation, which involves predicting a portion of the dataset using information from the remainder of the samples [Multivariate analyses were performed with The Unscrambler statistical package v9.8 . Mean c samples  and at t samples  for the F. candida treatments and conducted the molecular analysis. SB helped in performing the SSH and the differential screening. MJTNT and DR amplified SSH libraries for high throughput EST sequencing and MJTNT processed the raw sequencer trace files. JTD facilitated high throughput qPCR procedure. All authors helped shaping and approved the final manuscript.MEB, JE, DR and NMS conceived the study, set up its design and drafted the manuscript. MEB performed the DAG graph Cellular Compartment; a multiGOEAST analysis of the cadmium and phenanthrene SSH librariesClick here for fileDAG graph Biological Process; a multiGOEAST analysis of the cadmium and phenanthrene SSH librariesClick here for fileDAG graph Molecular Function; a multiGOEAST analysis of the cadmium and phenanthrene SSH librariesClick here for fileDetails of the qPCR assays used in this study; primer sequences and efficiencies; associated GO termsClick here for file