Simultaneous measurement of gene expression on a genomic scale can be accomplished using microarray technology or by sequencing based methods. Researchers who perform high throughput gene expression assays often deposit their data in public databases, but heterogeneity of measurement platforms leads to challenges for the combination and comparison of data sets. Researchers wishing to perform cross platform normalization face two major obstacles. First, a choice must be made about which method or methods to employ. Nine are currently available, and no rigorous comparison exists. Second, software for the selected method must be obtained and incorporated into a data analysis workflow.Using two publicly available cross-platform testing data sets, cross-platform normalization methods are compared based on inter-platform concordance and on the consistency of gene lists obtained with transformed data. Scatter and ROC-like plots are produced and new statistics based on those plots are introduced to measure the effectiveness of each method. Bootstrapping is employed to obtain distributions for those statistics. The consistency of platform effects across studies is explored theoretically and with respect to the testing data sets.http://alborz.sdsu.edu/conor and is available from CRAN.Our comparisons indicate that four methods, DWD, EB, GQ, and XPN, are generally effective, while the remaining methods do not adequately correct for platform effects. Of the four successful methods, XPN generally shows the highest inter-platform concordance when treatment groups are equally sized, while DWD is most robust to differently sized treatment groups and consistently shows the smallest loss in gene detection. We provide an R package, CONOR, capable of performing the nine cross-platform normalization methods considered. The package can be downloaded at  Simultaneous measurement of gene expression on a genomic scale can be accomplished using microarray technology or by sequencing based methods -3. Many Modern microarrays are commercially produced, and one-color hybridization schemes are often employed. Several companies have emerged as leading manufacturers, each using different manufacturing techniques, labeling methods, hybridization protocols, probe lengths, and probe sequences. Table Chemiluminescence provides greater sensitivity for low levels of expression compared to fluorescence, but at the risk of saturation for highly expressed genes. The Applied Biosystems scanning procedure attempts to mitigate scanner saturation by using both a short and a long exposure to extend the dynamic range of its expression measurements. Probe sequences affect the binding constants between probes and target and non-target molecules, and therefore the sensitivity and specificity of each probe depends partially on its sequence. Salinity and composition of the hybridization solution, temperature, and incubation time of hybridization may also affect sensitivity and specificity. Data from two microarrays are directly comparable only if those microarrays are identical in all design parameters including probe sequences and have been subjected to similar hybridization conditions. Because no two platforms share the same set of probe sequences, no two platforms produce data that are directly comparable, even if all other variables are the same. For experimenters this restriction is not major. They need only ensure that all experiments are conducted using the same array platform and protocol. However, platform effects pose a significant problem for the re-analysis of data from multiple microarray studies.Researchers who perform high throughput gene expression assays often deposit their data in public databases such as ArrayExpress  and GeneCross-platform normalization differs from meta-analysis; the former involves direct comparison between expression measurements obtained from different platforms while the latter combines the results of intra-platform comparisons at a higher level. While meta-analysis techniques are extremely useful tools, they are limited to combining the results of studies that have tested the same hypothesis or compared the same treatments, and can not easily be applied to the investigation of new hypotheses from existing data.Cross-platform normalization methods have been developed for the combination of data sets collected using different microarray platforms. These methods are the Cross-Platform Normalization (XPN) method of Shabalin et al., DistancCross-platform normalization could be a valuable resource to researchers. While several studies have employed it for microarray analysis -6,43, itIn this paper we provide a comparison of available methods based on the MicroArray Quality Control (MAQC) project  data sety = x under ideal circumstances. The closeness of the points to that line provides a measurement of inter-platform concordance. Throughout this work, we have used the squared Pearson correlation between the x and y values of these plots, denoted r2, as a statistical measure of inter-platform concordance.As an initial evaluation of the nine cross-platform normalization techniques, we applied each to a subset of the MAQC data set. Seven assays were selected at random from each of the four treatment groups, A, B, C, and D, included in the MAQC experiment from the Illumina (ILM) and Affymetrix (AFX) platform groups. The resulting reduced data set was then subjected to cross-platform normalization. To visually assess the effectiveness of each normalization technique, mean-mean expression scatter plots (or sunflower plots for methods with discrete output) were produced for each treatment group and each normalization method. Figure r2 value as follows: XPN, EB, DWD, GQ, QN, MRS, DisTran, QD, and NorDi. There is a substantial gap between the fourth and fifth methods in the ranking, and another between the eighth and ninth. To better interpret these plots it is necessary to know the concordance that can be expected for technical replicates  within a single platform and for technical replicates between two platforms. Ideally, the best cross-platform normalization methods will result in concordance levels similar to those obtained by technical replicates within a single platform. Figure Based on Figure A trivial transformation could be devised to produce perfect concordance between platforms by the removal of all platform and treatment effects. However, such a transformation would be undesirable for cross-platform analysis because of the loss of treatment effects. To assess the degree to which treatment effects were retained during cross-platform normalization, we plotted Receiver Operating Characteristic (ROC)-like curves for seven of nine methods for treatment groups A and B from the reduced MAQC data set Figure . NorDi ao and u, respectively, of differentially expressed genes. These area statistics have the advantage of not depending on any arbitrary FDR cut-off, and to our knowledge have not been used previously.By comparing these curves we obtain statistics for over and under-detection of differentially expressed genes after cross-platform normalization as follows. Over-detection can be assessed by observing the difference between the ROC-like curve for genes detected with the cross-platform data set (the red curve) and the curve for the intersection of the cross-platform genes and the union of the single platform genes (the blue curve), which gives the genes detected using the cross-platform data set but not using either single platform data set. Under-detection can be assessed by observing the difference between the curve for the intersection of the two single platform gene sets (the green curve) and the curve for the intersection of all three gene sets (the violet curve), which gives the proportion of genes detected by both platforms individually but not by the cross-platform data set. In our further analyses, the areas between these two pairs of curves are used as statistics to measure over and under-detection, denoted o or u described above. For example, a method may bring o to nearly zero by removing all treatment effects. Such a method would not be desirable, and would result in a large u. Each of the statistics o and u guards against unrestricted optimization of the other. The o and u statistics are related to false positive and false negative rates, respectively, and in practice a good cross-platform normalization should strike a balance between the two.It is not possible to rank cross-platform normalization methods by either one of the statistics Sorted from least to greatest under-detection, the methods are ordered: DWD, EB, GQ, XPN, MRS, QN, DisTran. Again, there is a major jump between the fourth and fifth ranked methods. When sorted instead by over-detection, the order is: MRS, QN, DisTran, XPN, DWD, GQ, EB, with a major discrepancy between the third and fourth ranked methods. Inspection of the ROC-like curves shows that the lower levels of over-detection among MRS, DisTran, and QN can be understood in terms of lower total detection of differentially expressed genes. By all three measurements, the seven methods cluster into two groups. The first group, made up of DWD, EB, GQ, and XPN, is characterized by higher concordance and over-detection and lower under-detection, while the second group, consisting of DisTran, MRS, and QN, is characterized by lower concordance and over-detection and higher under-detection. The ROC-like plots for the two groups are qualitatively different Figure . In the o, u, and r2, defined above, for cross-platform normalization between the AFX and ILM MAQC data. Additionally, we included MAQC data from the Applied Biosystems (ABI) and Agilent one color (AG1) platforms, obtaining bootstrap distributions for all three statistics and all six combinations of the four platforms. For all bootstraps, a sample size of 15 assays was fixed for each treatment group for a total of 60 assays per bootstrap per platform. Figures r2, o, and u, for the seven non-discrete normalization methods. The XPN method includes a clustering step, and normalizations were performed with different clustering options. For comparison, distributions are also shown for data that have not been normalized (no.norm) and for resampled data from each individual platform (resample.1 and resample.2). The resampling trials simulate the result of performing an identical experiment again using each each platform separately. The resampled and non-normalized distributions serve as positive and negative controls, respectively, for successful cross-platform normalization.Figures e - 20.In terms of concordance Figure , resultse - 20 level.The MAQC data set is unusual in that it contains a large number of technical replicates. The human sperm data set contains more biological than technical replicates and is more representative of data sets encountered in biological research. We performed the same bootstrapping analysis on the AFX and ILM human sperm data Figure . Again, Over all, some results were consistent for all data sets tested. DWD, EB, GQ, and XPN outperform MRS, QN, and DisTran in both concordance and under-detection, with DWD showing the lowest over-detection and XPN the highest concordance in all cases. Over detection never exceeded levels observed in resampling controls, even for the non-normalized control, which in fact showed no over-detection. Because all treatment groups were the same size for both platforms, platform effects would be expected to cancel out when comparing treatment groups. High variance within each treatment group due to platform effects explains the under-detection seen in the negative control. The performance of MRS, QN, and DisTran varied with the data sets tested. MRS, QN, and DisTran generally performed poorly. However, they still outperformed the negative control in the sperm data set, which is consistent with past successful applications of those methods.Four cross-platform normalization methods, DWD, EB, GQ, and XPN, showed satisfactory performance when evaluated using data sets with balanced treatment group sizes. Applications in which treatment group sizes are not consistent across platforms can easily be envisioned. We evaluated the four successful cross-platform normalization methods again using the MAQC data set with the same bootstrapping procedure. This time only data from treatment groups A and B were included, and the number of A and B samples was not necessarily equal between the two platforms. Five trials were performed for each method using different combinations of sample sizes: 15 A, 5 B for AFX and 5 A, 15 B for ILM; 13 A, 7 B for AFX and 7 A, 13 B for ILM; 10 A, 10 B for AFX and 10 A, 10 B for ILM; 7 A, 13 B for AFX and 13 A, 7 B for ILM; and 5 A, 15 B for AFX and 15 A, 5 B for ILM. Results indicate that DWD is the most robust to sample size differences, but that such differences have some effect on all methods Figure . The undAll cross-platform normalization methods studied share a common strategy. First, a set of parameters is determined from the data. Those parameters are then used to transform the data to remove platform effects. The parameters fitted from the data provide an estimate, in one form or another, of the platform effects present in the data set. Cross-platform normalization is only possible when all treatment groups are represented on both platforms. Otherwise, it becomes impossible to distinguish platform effects from treatment effects. Relaxation of this restriction would improve the usefulness of cross-platform normalization to researchers wishing to apply it. It may be possible to overcome this limitation by determining platform effects ahead of time using a separate data set, and then applying those parameters to the data set one wishes to transform. This possibility depends on the consistency of platform effects across different studies and treatment groups. To evaluate whether such consistency exists, we attempted to use parameters derived from the MAQC data set to perform DWD on the human sperm data set, first removing all N assays from the AFX data and all T assays from the ILM data, and then the reverse. DWD was selected for this experiment because of its success in previous trials and because of the simplicity of its model While concordance statistics could not be produced for this data set because data from each treatment group were represented on only one platform, we were able to evaluate both over-detection and under-detection using the same bootstrapping procedure as before Figure . Sample The inconsistency between platform effects in the MAQC and human sperm data sets may be due to differences in data processing or experimental protocols. Because of the age of the human sperm data set, its authors were unable to give any details regarding their exact procedures. To further explore the consistency of platform effects for different treatments, we tried using the MAQC A data to transform the MAQC B data and using the human sperm N data to transform the T data. Again we used the same bootstrapping procedure with 15 assays in each treatment group. Results Figure  show redFigures The four best performing methods, DWD, EB, GQ, and XPN, model platform effects using location parameters, while the other five methods do not include location shifts. The representation of platform effect as a location parameter is somewhat consistent with a physical interpretation of the microarray hybridization process. The hybridization process can be coarsely described by the chemical equationp, g, and h are the probe, the gene or transcript target, and the hybridized transcript, respectively, and k+ and k- are rate constants. For relatively large target concentrations, the signal for a hybridized probe at equilibrium iswhere S = Ih is measured signal, g0 is the initial concentration of target RNA or DNA, p0 is related to the initial number of available probe sites, and I is related to the intensity of the marking mechanism associated with each target DNA or RNA molecule. The parameter I should be thought of as the intensity of the label on each target molecule. The number of target molecules present at the probe site multiplied by the intensity of each gives the total intensity at that site. Model (2) has previously been applied to Affymetrix GeneChip© data [where ip© data ,50. Undek+, k-, I, and p0 are determined by the microarray platform and experimental conditions such as hybridization temperature. Only g0 is related to gene expression. Assuming experimental conditions are about the same for all users of a given platform, a statistical model of log signal intensity for a particular probe might beThe parameters ijk yis the log signal value for gene i, treatment j, and repetition k; iT, jP, and ij Care treatment, platform, and treatment-platform interaction effects, respectively; and ijk εis a random variable associated with repetition. Equation (4) is more general than, but fully consistent with, equation (3). The presence of a treatment-platform interaction term can be tested by analysis of variance (ANOVA). We performed ANOVA using model (4) for each gene separately on ABI, AFX, AG1, and ILM data with treatment groups A, B, C, and D from the MAQC data set, using the lm function of the R stats package [p-values for the null hypothesis (H0) that ij C= C for all treatments and platforms. Using the qvalue package [π0, for which H0 is true, and subsequently the proportion of genes, π1, for which H0 is false from the distribution of p-values using the method of Story and Tibshirani [P and C terms can be removed by a single location shift. For larger numbers of samples, the interaction term cannot be fully removed by a single location shift. An optimal location shift, in the least squares sense, for the m treatment case iswhere  package . Figure  package , we werebshirani . Resultsj ηis the location shift used to correct for platform effects. The values ij nare the number of repetitions for treatment i and platform j, and m is the total number of different treatment groups. It is assumed that the true values of the effects are known. In practice, j ηwill be an estimate based on the data.where None of the methods studied here make explicit use of a least squares optimal location shift to remove platform effects. Rather, this optimal location shift is used to illustrate the limitations of modeling platform effects as location parameters under model (1). The prospect of determining a location shift for a particular pair of platforms using one data set and applying that shift to another data set is complicated by the presence of interaction effects in equation (6). By equations (5) and (6), the difference between two treatment group averages in a two platform data set after the application of a location shift derived from another two-platform data set, assuming equally sized treatment groups, isij Cand where Uniformity of platform effects across different experiments is consistent with the model (1). Additivity is not, although empirically it appears to be a useful approximation, as indicated by the relative success of the location based methods. The accuracy of that approximation may be reduced, however, when employing a separate training set to estimate platform effects. This reduction was observed in our analyses, in particular in Figures http://alborz.sdsu.edu/conor and is available from CRAN.To facilitate the application of cross-platform normalization by other researchers, we packaged all the implementations used in this work, including those obtained from other authors, using the R package mechanism. Our package, CONOR, includes documentation and provides a common interface for all methods, along with reasonable defaults for user-selectable parameters. The package can be downloaded at Of the four methods capable of successful cross-platform normalization, DWD showed the least loss of treatment information and XPN showed the greatest inter-platform concordance, although the latter was sometimes in excess of resample controls and might be interpreted as a slight over-correction. DWD was the most robust to variations in treatment group sizes between the two platforms. This result is somewhat surprising because XPN incorporates an assay clustering step designed to correct for such variations . Our cluOur experiment in normalizing the human sperm data set using parameters derived from the MAQC data set shows that there may be some consistency in platform effects across different treatments. Larger differences between platform effects in the MAQC and human sperm studies may have resulted from differences in protocols between the two studies. However, they may also have resulted from larger disparity between gene expression patterns in sperm and those of other human tissues, in which case a more complicated model of platform effects might resolve the difference. Ideally, data sets like the MAQC project's could be used as "Rosetta Stones" for gene expression platforms, allowing data collected on one platform to be translated to be comparable with data from another platform regardless of treatment group disparities. This work has shown that a model including treatment-platform interaction terms will be required for such a system to be effective, and further investigation is required before such a system can be realized.In cases in which false positives are to be preferred over false negatives, or in which treatment group sizes are not equal for the various single platform data sets, DWD is the recommended cross-platform normalization method because of its lower under-detection and improved robustness. In cases in which false negatives are to be preferred to false positives, and for use with classifiers, XPN is recommended because of its superior cross-platform concordance. If there is doubt as to which situation applies, DWD is recommended, but there is no reason that multiple techniques could not be employed and the results compared. The existence of the CONOR package will make the latter option particularly painless. In cases in which treatment groups are missing from one platform or another, the DWD-based procedure described here is the only currently available method, but care must be taken to ensure that training data have sufficiently similar transcription profiles to the data being transformed.Next generation sequencing technology is replacing microarrays for the measurement of gene expression. The types of platform effects present in microarray data are probably not relevant for sequencing-based expression data. Nevertheless, existing databases, as well as microarray experiments that may be performed in the near future, represent a substantial resource. Cross-platform normalization has the potential to become a valuable tool for gene expression research by allowing researchers to combine and analyze existing data together with new data or in new contexts. While at least nine methods are currently available, we've shown that only four of those methods provide reasonable results on the MAQC and human sperm data sets. Although researchers are encouraged to draw their own conclusions based on their particular needs, two methods emerged as most effective. Cross-platform normalization of large or complex datasets in which treatment groups are missing will require improved models of treatment-platform interaction effects. Future work should include the modification of XPN to improve robustness and allow for the use of a separate training set, as well as the investigation of other models of platform effects. This study has not examined every platform available. In particular, no cDNA arrays were included, and the effectiveness of cross-platform normalization with such arrays should be investigated before those methods are applied. While the types of platform biases present in microarrays are not relevant to sequencing based assays, it is worth investigating the effects that different extraction, amplification, and sequencing methodologies have of such measurements, as well as how data from sequencing-based assays might be combined with microarray data. While the availability of an R package does much to make these methods accessible, there are some researchers who are not comfortable with the R command-style interface. For those researchers, a GUI application or web interface integrating all of those methods may be beneficial, although web interfaces are already available for some methods. If the transfer of platform parameters between data sets is improved, a web-based database of such platform parameters that could be integrated into cross-platform analyses would be extremely useful.©, the Agilent Whole Genome Oligo Microarray G4112A, and the Illumina Human-6 BeadChip 48k v1.0. These platforms were selected based on the availability of data and the variety of probe lengths, manufacturing techniques, and detection methods employed  and natural log transformation. For the experiments involving both MAQC and human sperm data, both sets were quantile normalized together. For all other experiments, the MAQC and sperm data were subjected to separate quantile normalization.The MAQC publication  providesx and y coordinates of each point in the mean-mean scatter plot was used as a statistic to measure inter-platform concordance. It has been pointed out by the editors that the concordance correlation coefficient is a more appropriate statistic for these purposes [y = x, and secondly, that the conclusions drawn are supported by the other statistics used.Mean-mean expression scatter plots were produced by plotting the average expression value for each gene and treatment group on one platform against the average expression for the corresponding gene and treatment group of another platform. Treatment groups for MAQC data are equivalent to sets of technical replicates , whereas treatment groups in the human sperm data set  represent biological replicates, although some technical replicates are included. Squared Pearson correlation between the purposes . Unfortuq-value for a particular feature (or gene) is defined as the proportion of false positives expected when calling all features on a list up to and including that one significant, where here the list in question is the list of p-values obtained from the t-tests [Differential expression was assessed using the p-value of a two-sided Welch's t-test as a statistic. A p-value was obtained for each gene, and the resulting list of p-values was transformed into a list of q-values using the q-value package , availab t-tests . The emp t-tests .The union and intersection curves were obtained by taking the empirical cdf of the gene-wise minimum and maximum, respectively, of the two q-value lists. Areas between curves were obtained by numerical integration using the trapezoid method, as implemented in the caTools package  availablA smoothed bootstrapping procedure was used to obtain distributions for the concordance, over-detection, and under-detection statistics. The smoothing was accomplished through the addition of zero-centered Gaussian noise with a 0.1 standard deviation. Resampling was restricted by treatment groups. That is, for the MAQC data set, every bootstrap maintained the same proportion of data from samples A, B, C, and D. For the human sperm data set, the proportion of data from N and T samples was kept constant. Data used to produce the native ROC-like curves for each platform (and subsequent union and intersection curves) were produced by the same resampling and smoothing procedure, but with the sample size doubled for the relevant treatment groups in order to maintain the same total sample size for each ROC-like curve produced. Data for resampling (positive) control methods was generated in the same manner as the data for the native ROC-like curves, sampled independently to simulate repetition of the experiment on each platform separately. Computations were performed using R version 2.9.0 2009-04-17) x86_64-redhat-linux-gnu on a Rocks 5.3 computing cluster. Bootstrapping results are available as additional file -17 x86_6k-means clustering step was added to estimate that information. The correct number of treatment groups was always used as the value of k. Implementation of DWD and XPN was based on the Matlab implementations of those methods provided by their authors. Both methods were tested on toy data sets to ensure agreement with the original implementations. For XPN, agreement was approximate due to the randomness inherent in the initial clustering step. DWD relies on the solution to a second order cone program (SOCP). No appropriate SOCP solver existed in R prior to this work, so a recent smoothing Newton method algorithm was implemented and employed [k-means clustering of assays was performed on data from both platforms together. If any cluster contained data from only one platform, clustering was simply repeated with a different set of initial centroids. To speed up the clustering process, an option to cluster data from the two platforms separately and match clusters based on the correlation of cluster centroids was added. XPN trials designated as "modified" or "mod" made use of this modified method.R code was obtained for all methods if available. R functions for GQ, MRS, and QD were taken from the source code for WebArrayDB. Code for EB and NorDi was obtained from the original authors of those methods. QN was available from the preprocessCore package of Bioconductor. To the best of our knowledge, no R implementations of DWD, DisTran, or XPN existed prior to this work. Implementation of DisTran was based on a description of the method . Becauseemployed . Additiop-value and alpha parameters, which were set at 0.01 and 0.05, respectively, in agreement with that method's use by its authors.Some methods require the user to select one or more parameters manually. XPN and DisTran both require the user to select how many assay clusters are to be used. In all cases, the actual number of treatment groups was used. XPN also requires the user to select the number of gene clusters and the number of iterations to perform. Values of 3, 6, and 9 were used for gene clusters as indicated in the results section. Where no value is indicated, three gene clusters were used. For all XPN trials, 30 iterations were performed as recommended by the original authors . The EB A detailed description of each method can be found in the references, with the exception of GQ. GQ is included as part of the WebArrayDB service  and was The EB method is capable of taking treatment group or other covariate information into account . BecauseJR located data, performed all programming and analyses, and wrote the manuscript. FV supervised all research, guided decision making with regard to methods, and edited the manuscript. Both authors contributed to the conception and design of the project. All authors read and approved the final manuscript.Mean-mean plots for MAQC treatment group B ILM and AFX data.Click here for fileMean-mean plots for MAQC treatment group C ILM and AFX data.Click here for fileMean-mean plots for MAQC treatment group D ILM and AFX data.Click here for fileRaw bootstrap results for all statistics presented.Click here for file


Gene expression data can provide a very rich source of information for elucidating the biological function on the pathway level if the experimental design considers the needs of the statistical analysis methods. The purpose of this paper is to provide a comparative analysis of statistical methods for detecting the differentially expression of pathways (DEP). In contrast to many other studies conducted so far, we use three novel simulation types, producing a more realistic correlation structure than previous simulation methods. This includes also the generation of surrogate data from two large-scale microarray experiments from prostate cancer and ALL. As a result from our comprehensive analysis of  The functional analysis of high-throughput data is a challenging but promising direction in the post-genomics era. It is challenging because genome-wide data are high-dimensional and noisy, but promising due to its potential to reveal knowledge about the systemic working mechanisms of biological information processing within cells, which we are currently lacking competitive, whereas a test comparing the same gene set for two different phenotypes is called self-contained.In the context of expression data the interest shifted in recent years from approaches focusing on the analysis of individual genes, detecting their differentially expression gene set’ associated with the above methods implies that the choice for defining gene sets by populating them with specific genes is not constraint. However, in the present study, we are considering only gene sets that have been defined by using biological information regarding their association with specific pathways extracted, e.g., from the gene ontology database pathway-based methodsThe name ‘sum of t-square and Hotelling's sum of t-square), multivariate  for a pathway.Permutation of the gene-labels to estimate the significance of the enrichment score  for the pathway.The hypotheses tested by GSEA are:The Hotelling Suppose we have two groups with sum of t-square test is an univariate test based on t-scores, The Again, the significance of In order to assess the performance of the statistical methods we use three principally different algorithms to simulate expression data.For this method we, first, define different non-overlapping pathways of varying sizes including a total of First, we generate a matrix Markov with respect to an undirected network Gaussian graphical model. This means that all conditional independence relations that can be found in For random variables that are from a p-dimensional multivariate normal distribution, i.e., We use the following two algorithms to obtain a covariance matrix for a given network structure The algorithm of The algorithm of Biogrid database observed interaction structures among genes and gene product and, hence, provide a more realistic structure than artificially generated networks, e.g., by using the preferential attachment model to generate scale-free networks As input network We would like to note that due to the properties of the Gaussian graphical model, as discussed above, this model has been used to infer gene regulatory networks from expression data competitive testexcluding the pathway under investigation, as background data.For our analysis we are using four different simulation types (ST) based on the algorithms described above. Because GSEA is a without expression difference between the treatment and control group, i.e., For this type of simulation we generate simulated expressed data for all Here we generate simulated data separately for each pathway, as described in methods section ‘Simulation of correlated data’. In contrast to ST I, ST II generates a correlation among the genes within a pathway. We use an overall set of gene ontology database biological process for level For this ST we generate simulated expressed data by sampling from a p-dimensional Gaussian graphical model. In order to obtain a more realistic correlation structure we use two different algorithms, as described in methods section ‘Simulation of network-like correlation structure’ in combination with a protein interaction network and a transcriptional regulatory network. We used the false positive rate (FPR) have been obtained by setting We would like to note that in the results section, the estimates for the biological process of level four in the gene ontology database To assess the power of the three pathway-based methods for microarray data we use two different large-scale data sets based on Affymetrix chips. The first is a prostate cancer data set consisting of true positives and the nonsignificant pathways as true negatives. In the second step, we construct Our analysis of these data consists of two steps. In the first step, we generate a reference list by testing the significance of pathways for the total number of samples For the following simulations, we use a significance level of sum of t-square (blue curves) for simulation type I and II and different parameter settings. Here by the power we mean the probability that the statistical hypothesis test is rejected when the null hypothesis is truly false In sum of t-square and GSEA than on Hotelling's sum of t-square and GSEA strongest resulting in a considerable loss in power for sum of t-square test has by far the worst power. For ST II , GSEA and sum of t-square and Hotelling's sum of t-square test and, more importantly, it is more robust with respect to DC values less than sum of t-square the FPR is in general controlled, except for ST IV and The significant reduction of the power for GSEA for simulation type I compared to simulation type II to IV can be explained by the background data that have been generated for simulation type I, but not for the other simulation types. Due to the fact that GSEA is a competitive hypothesis test which estimates a test statistics w.r.t the background, the background data have a prominent influence on the power of GSEA whereas a large background dataset without expression changes in the conditions increases the sensitivity of this test.In truely expressed pathways. Similarly, the interpretation for the FPR and the number of significant pathways. On the first sight, all methods seem to perform similarly, although, GSEA has for high sample sizes the lowest power. However, Hotelling's sum of t-square declare for all studied sample sizes many more pathways as significant than GSEA. Similar to the simulation studies, GSEA controls the FPR well whereas the other two methods assume larger values.The results for prostate cancer (left) and ALL (right) are shown in total sample sizes are too small for the employed test statistics, because otherwise we would observe a stable plateau as in An interesting observation of the power is its rapid decay for an even slightly reduced sample size. More precisely, for prostate cancer the total sample size is distance to convergence, as found from the comparison with our simulation results in initial step size of the power corresponds to the power for optimal refers to the minimal sample size for a method to become invariant against the removal of a small number of samples.In order to quantify this observation, we performed a linear regression analysis. For this analysis we use the size of the microarray experiments as predictor variable and the initial step size of the power curves  as outcosum of t-square test (red crosses in For the regression, we obtain F-statistics (each pathway, we obtain an entirely different result. In A central topic of this paper is the investigation of the influence of the correlation strength and its structure on the identification of differentially expressed pathways. In the introduction we presented arguments supporting the need for such an analysis. Now we add quantitative evidence, directly extracted from the used expression data from prostate cancer and ALL. As discussed in the methods section ‘Surrogate data: ALL and prostate cancer’, both microarray data sets were normalized. Estimating the average correlation among all genes from the normalized data results in For comparison with the simulated data, we include in We would like to point out that results about the range of the correlation values is only one indicator that should be met by simulated data. In addition, the structure among the genes is another important characteristics. Due to the fact that the data for ST III and ST IV are generated in a way that the inverse of the covariance matrix does reflect the independence relations that can be found in a protein network, this is another crucial difference to previous studies equipping our approach with a more realistic correlation structure.Finally, we present results about the biological distribution of DC values in prostate cancer and ALL. Using SAM sum of t-square test has for almost all studied cases the highest power if sum of t-square test controls in general the FPR well, except for ST IV and inappropriate background data, which is out of the control of the experimenter. On the other hand, GSEA has a good control of the FPR for all conditions.For our power analysis of simulated data, we assumed a maximum sample size of filter them in order to eliminate conditions for which a method is more likely to cause problems. We suggest to filter the pathways according to the following easy to check criteria: Hotelling's sum of t-square test should only be used for pathways with coincide with clustering methods, hints, that DEP analysis methods are not just the sum of individual gene test statistics. If one perceives this problem from a biological perspective, this correspondence becomes more plausible because the clustering of genes is frequently used to reveal functional relations between genes corresponding to biological pathways Taking all this into account our findings do not suggest to apply a method unconditionally to all pathways in a given data set, but to The underlying rationale for our power analysis, which provides statistical estimates of the true positive rates of tests, is to study the unbiased performance of the methods. In contrast, by conducting multiple hypotheses tests one would be obligated to apply a multiple testing procedure to control a selected error measure GeneNetWeaverSynTReNThe optimal experimental design for microarray experiments with respect to the identification of the DEP is an important topic that is currently still under debate. From our comprehensive analysis of simulated and experimental gene expression data of over 3 million pathways, we obtain three major results. First, we find that the heterogeneity of different biological conditions and the sensitivity of the statistical methods suggest a selective application to definite pathways. That means, it is not advisable to apply a method to all accessible pathways but only to selected ones. Second, future gene expression experiments aiming to detect the DEP should be conducted with an increased number of samples in order to avoid non-robust and underpowered studies. From our study, we find method-specific recommendations constituting lower bounds for minimal sample sizes. Specifically, we suggest sample sizes between For future studies of DEP methods, simulations based on our approach using ST III and ST IV can be very useful to investigate, e.g., the influence of different gene network structures, the effect of overlapping pathways or the influence of heterogeneous effect sizes. For example, one could compare protein networks and transcriptional regulatory networks for different organisms or compare them with gene regulatory networks. Here by gene regulatory networks we mean networks inferred from gene expression data The studied methods in this paper are expected to be also useful for the analysis of RNA-seq data


The current strategy for identifying the carcinogenicity of drugs involves the 2-year bioassay in male and female rats and mice. As this assay is cost-intensive and time-consuming there is a high interest in developing approaches for the screening and prioritization of drug candidates in preclinical safety evaluations. Predictive models based on toxicogenomics investigations after short-term exposure have shown their potential for assessing the carcinogenic risk. In this study, we investigated a novel method for the evaluation of toxicogenomics data based on ensemble feature selection in conjunction with bootstrapping for the purpose to derive reproducible and characteristic multi-gene signatures. This method was evaluated on a microarray dataset containing global gene expression data from liver samples of both male and female mice. The dataset was generated by the IMI MARCAR consortium and included gene expression profiles of genotoxic and nongenotoxic hepatocarcinogens obtained after treatment of CD-1 mice for 3 or 14 days. We developed predictive models based on gene expression data of both sexes and the models were employed for predicting the carcinogenic class of diverse compounds. Comparing the predictivity of our multi-gene signatures against signatures from literature, we demonstrated that by incorporating our gene sets as features slightly higher accuracy is on average achieved by a representative set of state-of-the art supervised learning methods. The constructed models were also used for the classification of Cyproterone acetate (CPA), Wy-14643 (WY) and Thioacetamid (TAA), whose primary mechanism of carcinogenicity is controversially discussed. Based on the extracted mouse liver gene expression patterns, CPA would be predicted as a nongenotoxic compound. In contrast, both WY and TAA would be classified as genotoxic mouse hepatocarcinogens. A crucial part of the drug development pipeline is the assessment of the carcinogenic potential of compounds, which is currently performed on the basis of the 2-year rodent bioassay. However, as these assays are associated with high costs as well as long study times, and require a high number of animals (more than 800 mice and rats) In general, carcinogens (C) can be subdivided into two groups depending on their mechanism. Genotoxic carcinogens (GC) are characterized by reactivity with DNA or by the formation of DNA-reactive metabolites which may lead to tumor initiation. Nongenotoxic carcinogens (NGC) do not directly cause DNA modifications, but mediate tumor initiation or promotion by secondary mechanisms, which are not completely discovered so far. In recent years, a wide variety of mechanisms causing nongenotoxic carcinogenesis in liver were proposed, such as oxidative stress, chronic cell injury, immunosuppression, induction of peroxisome proliferation, modulation of specific cytochrome P450 enzymes, increased secretion of growth-stimulating hormones, or the perturbance of specific signaling pathways in vitro genotoxicity tests  is routinely applied during the early phase of drug development For the detection of GCs a battery of three et al.) Compounds such as TAA and WY which show a negative Ames test but a positive mouse lymphoma test are controversially classified by different authors as nongenotoxic or genotoxic carcinogens  t-test, genetic algorithms, etc.), a pooled gene list was generated, from which a short-list of 6 signature genes was extracted using an exhaustive enumeration method. More recently, Uehara et al. published a rat liver signature comprising 112 genes which was inferred using prediction analysis for microarrays (PAM) et al., who recommended longer exposure periods, another signature comprising 82 probesets was published by Uehara et al.et al.In a pioneering study Nie et al. investigated the presence of characteristic, reproducible transcript expression patterns in diverse murine target organs In contrast to the majority of recent toxicogenomics studies, which focused on elucidating NGC mechanisms in rat liver, Jonker Here, we propose a new methodology, which employs ensemble feature selection in conjunction with bootstrapping for the extraction of mRNA signatures from transcriptomic profiles of GC-, NGC-, or NC-treated mice. Besides evaluating the average classification performance achieved by different algorithms for probeset selection, we also assess their robustness against small variations of the training data. When compared to recently published signatures for the prediction of hepatocarcinogenicity in mice, we found that our signatures are equally good or slightly better, depending on the employed classifier and evaluated prediction task.In contrast to previous toxicogenomics studies, we profiled global gene expression in both male and female mice, which is also done on a routinely basis in chronic toxicity and carcinogenicity studies. As some of the general pathways and biological processes leading to cancer  may be influenced by androgens and estrogens, we consider it relevant to include both sexes in the development of biomarkers for hepatocarcinogenesis. Ultimately, this study design facilitated capturing characteristic changes in gene expression, which are present in both sexes upon treatment with GCs and NGCs, respectively.The animal experimental work under this study was subject to the Danish Executive Order No. 1306 of 23 November 2007. The protocol was reviewed by the veterinarians responsible for animal welfare in the testing facility and conducted under animal license number 2009.561.1593 issued by the Danish Animal Inspectorate. For necropsy the animals were anaesthetized in isoflurane and exsanguinated from the heart. All animals were checked for mortality/ill health twice daily and animals in extremis were sacrificed after consultation with the veterinarian in charge of animal health and welfare. All animal procedures described in the present paper have been approved by the ethical committee for animal experimentation of UCB Pharma S.A. and were in accordance with the most recent European and Belgian legislation on the use of laboratory animals  as well as with the Guide for the Care and Use of Laboratory Animals .http://potency.berkeley.edu/) In this study, we employed the Affymetrix GeneChip Mouse Genome 430 2.0 Array to monitor the changes in the gene expression pattern induced by diverse compounds in mouse liver samples. The generated microarray data were deposited at Gene Expression Omnibus (GEO accession: GSE44783). Treatment groups of 6 male and female CD-1® mice, respectively, were dosed by oral gavage with a heterogeneous set of compounds over a time period of 3 or 14 days. Time matched control groups were treated with the corresponding vehicles, carboxymethyl cellulose or corn oil. The compounds, their CAS numbers, vehicles, doses and the group ID are all listed in Samples from 5–6 animals per treatment group were subjected to mRNA expression analysis. The snap-frozen liver samples were homogenized with Qiazol . The cRNA synthesis, target hybridization, probe array washing, staining and subsequent probe array scanning were done according to the standard protocol 3′IVT Express Kit User Manual (Affymetrix). Microarray analysis was performed by using GeneChip Mouse Genome 430 2.0 Arrays from Affymetrix.arrayQualityMetrics were used to assess the quality of the raw data After importing the Affymetrix raw data  into the R programming language and environment for statistical computing, different metrics and statistics implemented in the package One of the main research objectives of this study was the selection of mRNA signatures, i.e., sets of informative marker genes, which are consistently differentially expressed between compound classes, and thus can be utilized to reliably predict the type of carcinogenicity of a given compound. For this purpose, we implemented an analysis pipeline for the inference and evaluation of gene signatures from mRNA expression data. The central element of this pipeline is a two-stage method, encompassing statistical and supervised learning methods for marker gene selection (Stage 1) and expression profile-based compound classification (Stage 2).et al., which measures the correlation between gene expression states and compound class labels In the first stage, we employed linear Support Vector Machines (SVM), SVM-based Recursive Feature Elimination (SVM-RFE), Prediction Analysis for Microarrays (PAM) and a signal-to-noise ratio (Golub-Ratio) proposed by Golub m = 4 methods on n = 25 randomly drawn bootstraps, each containing 90% of the training samples. For each bootstrap the remaining 10% of the training samples (out-of-bag samples) were used to assess the classification performance, achieved with the selected marker genes, in terms of area under the Receiver Operating Characteristic (ROC) curve.For optimization of the classification performance, we used a grid search approach to tune the individual parameters of the machine learning methods. For this purpose, we employed a 3×3 nested cross-validation procedure, which ensures unbiased parameter tuning, while maintaining independent test sets. The robustness of the signature was increased by performing ensemble feature selection with k = 10 different probeset numbers between 2 and 100. Subsequently, the optimal number of features corresponding to the analytical maximum area under the ROC was estimated by fitting splines (implementation from package splines for R).In order to determine the optimal number of informative genes, the ROC scores were computed for X with |X| = n and two feature subsets A, B with 0<|A| = |B| = k<n and |A ∩ B| = r the Kuncheva index is defined as Next, for each feature selection method the robustness of the selected signatures, i.e., the correspondence of the informative gene sets inferred on different bootstraps, was measured based on the Kuncheva stability index On the basis of the signatures inferred on different bootstraps using different methods, we generated consensus signatures by computing an average rank for each gene. For this purpose a consensus ranking was generated by computing the rank sums across all methods and bootstraps. Subsequently, the genes were sorted in ascending order by the rank sums.k-Nearest Neighbor (KNN), PAM, Naïve Bayes, Random Forest, and Weighted Voting z-scores knnflex, pamr, klaR and randomForestIn the second stage, the signatures inferred on the training data were incorporated as predictive features for compound class prediction by diverse classifiers, namely SVM, Using an ensemble of statistical and machine learning-based feature selection techniques in conjunction with state-of-the-art methods for supervised classification, we constructed accurate and robust models for carcinogenic compound class prediction based on characteristic gene expression profiles. Our analysis workflow  was emplet al. .www.imi-marcar.eu), we are currently assessing the mechanistic relevance of such genes, which are specifically deregulated upon treatment with GCs and NGCs in order to gain a more profound understanding of the induced cancer types. Among the probesets which were specifically deregulated upon treatment with GCs and showed either inverse or unaltered patterns for NGCs were, for instance, Vim, Armcx4, Raet1e, Marco, SerpinA7, Egr1 and Gpr98. In response to nongenotoxic carcinogenic exposure an upregulation of Dio1, Ces2a, Akr1b7, S100a8, Orm3, Cyp2b10 and Cyp2c55 was measured  for dosing, which can be determined in a subchronic study 50 rate, which induced tumors in half of the tested animals.The choice of the right dose is crucial for the selection of signature genes which are specifically deregulated in response to the application of carcinogenic substances. Overdosing compounds may mask the specific changes in gene expression which are related to cancer initiation or progression, due to a huge amount of deregulated genes orchestrating an unspecific stress response et al. reported that the treatment time is also a critical factor, which may highly impact the success of a toxicogenomics approach et al. also reported that a higher accuracy of the prediction models could be achieved for longer exposure durations et al. proposed to represent each compound by expression data from multiple time points, at which a sufficient number of deregulated genes could be observed In a recent study, Auerbach et al., who published a signature consisting of 9 Affymetrix probesets (7 different genes), which was shown to discriminate NGC from NC in rat liver with reasonable accuracy When assessing prediction accuracy depending on the signature size, we found that predictions with tolerable reliability are possible, based on the expression status of ten informative genes . This fiPathway enrichment analysis of the 66 genes from the 14-day C vs. NC signature  against Cyp2c50, Cyp2c55, Cyp2b10, S100a8, Gsta1, and Akr1b7  and show class-specific expression patterns at multiple time points , we determined for each gene in how many of the 8 evaluated settings it was selected as an informative gene . The mosd Akr1b7 . HoweverXpa/p53+/− knockout mice were shown to improve the prediction accuracy of bioassays while at the same time reducing the required treatment duration As the susceptibility to carcinogenic compounds as well as the triggered changes in gene expression may vary between mouse strains, future analysis should be conducted in order to validate to what extent the inferred gene expression signatures can be reproduced in other strains. Especially, the applicability of our proposed signatures to transgenic mouse strains should be assessed in future studies, as for instance Recent toxicogenomics studies are mostly focused on profiling the RNA expression of rodents after single or repeated exposure to carcinogens As a wide variety of mechanisms was proposed for NGC, it is hardly feasible to extract a highly specific and at the same time universal signature capturing the gene expression changes caused by NGC treatment In conclusion, the here proposed methodology was shown to facilitate the extraction of robust gene expression signatures which were incorporated into generalizable models for the prediction of the carcinogenic class of compounds. These prediction models may accelerate the screening for promising candidates, and hence increase the efficiency and cost-effectiveness of preclinical drug development.Figure S1Accuracy and stability of signatures for compound classification. (A) The line plots depict the mean performance of GC vs. NC classification after 3 days of repeated dosing, which was achieved based on gene sets extracted with different feature selection methods. Each curve corresponds to a feature selection method and the performance was assessed depending on the number of genes selected as informative features. The prediction accuracy was assessed on the samples left out from 25 random subsamplings of the dataset (bootstraps), each containing 90% of the data, and measured in terms of area under the ROC curve. The inset bar plot depicts the ROC scores averaged across bootstraps and signature sizes. (B) Performance of GC vs. NC classification after 14 days of treatment illustrated as in (A). (C) The correspondence of the extracted GC vs. NC gene sets across 25 bootstraps was assessed based on the Kuncheva stability index (KI) for each of the 4 employed feature selection methods. The KI was then for each method plotted against the number of selected signature genes. (D) Robustness of signatures for GC vs. NC classification after 14 days of treatment illustrated as in (C).  Prediction accuracy achieved with signatures for NGC vs. NC classification after (E) 3 days and (F) 14 days of repeated dosing, respectively, depicted as in (A).  Similar illustration as in (C) showing robustness of signatures for NGC vs. NC classification after (G) 3 days and (H) 14 days of administration, respectively.(PDF)Click here for additional data file.Figure S2Classification results obtained for different signatures. (A) The four heatmaps show the predictions resulting from diverse binary classifiers for the discrimination of GC from NC after 3 days (left two heatmaps) or 14 days (right two heatmaps) of repeated dosing. For each dosing time two heatmaps are depicted which correspond to male and female mice, respectively. The rows correspond to different classifiers and the columns to different treatment groups. The continuous prediction scores, returned from the classifiers, were transformed to confidence scores between 0 and 1, which provide an estimate of the probability of class GC. The colorbar on top shows the true class annotation. The black vertical lines separate the test samples from the three folds of cross-validation. (B) Heatmaps illustrating the classification outcome of diverse predictors to distinguish NGC from NC based on characteristic gene expression profiles observed in male and female mouse liver samples after 3 or 14 days of administration. KNN, K-Nearest Neighbor; SVM, Support Vector Machine; PAM, Prediction Analysis for Microarrays.(PDF)Click here for additional data file.Figure S3Classification results obtained for different ensembles of feature selection methods. (A) The grouped bar plots depict the area under the ROC curves achieved with signatures which were inferred based on three different ensembles of feature selection methods and applied for the separation of C and NC after 3 days of treatment. The first ensemble contains the four methods SVM, SVM-RFE, Golub-Ratio and PAM, the second one is limited to SVM and SVM-RFE and the third one comprises PCA and PLS-DA. The performance of the signatures was determined based on the prediction outcomes obtained from six different classifiers which were evaluated in a 3-fold cross-validation. In the plot depicted above each bar corresponds to a certain classifier (see legend) and each group of bars refers to a certain ensemble of feature selection methods. The horizontal dashed line indicates the performance that would have been achieved by random guessing. The adjacent Venn diagrams illustrate informative genes common between signatures. (B) Same plots as in (A), but for C vs. NC classification after 14 days of repeated dosing. (C) Mean ROC scores and signature overlaps for GC vs. NGC classification after 3 days of treatment. (D) Same plots as in (C), but for GC vs. NGC classification after 14 days of repeated dosing.(PDF)Click here for additional data file.Figure S4Selection frequency of signature genes. The figure depicts a histogram showing the number of genes selected twice or more times as informative genes for compound class prediction. For the generation of this figure, we considered 8 predicted signatures for solving 4 binary classification tasks  after 3 days and 14 days of repeated dosing, respectively. The genes selected 4 and 5 times, respectively, are listed next to the histogram.(PDF)Click here for additional data file.Figure S5Expression profiles of most frequently selected signature genes. The heatmap shows the expression profiles of all candidate biomarker genes which are contained in at least 3 of the 8 predicted signatures. These signatures were proposed for 4 different classification tasks . For each classification task two treatment durations (3 days and 14 days) were considered, independently.(PDF)Click here for additional data file.Table S1et al. (Golub-Ratio). If possible the corresponding gene symbol, EntrezGene ID and description are provided for each probeset.mRNA signatures extracted with SVM, SVM-RFE, PAM and Golub-Ratio. The Excel sheets contain the Affymetrix IDs of the probesets selected by the following ensemble of feature selection methods: Support Vector Machines (SVM), SVM-based Recursive Feature Elimination (SVM-RFE), Prediction Analysis for Microarrays (PAM) and the statistical ratio proposed by Golub (XLS)Click here for additional data file.Table S2mRNA signatures extracted with PCA and PLS-DA. The Excel sheets contain the Affymetrix IDs of the probesets selected by the following ensemble of feature selection methods: Principal Component Analysis (PCA) and Partial Least Squares Discriminant Analysis (PLS-DA). If possible the corresponding gene symbol, EntrezGene ID and description are provided for each probeset.(XLS)Click here for additional data file.Table S3mRNA signatures extracted with SVM and SVM-RFE. The Excel sheets contain the Affymetrix IDs of the probesets selected by the following ensemble of feature selection methods: Support Vector Machines (SVM) and SVM-based Recursive Feature Elimination (SVM-RFE). If possible the corresponding gene symbol, EntrezGene ID and description are provided for each probeset.(XLS)Click here for additional data file.Table S4Published mRNA signatures for the detection of hepatocarcinogenicity in mice. The Excel sheets contain the Affymetrix IDs of published signatures for the toxicogenomics-based detection of carcinogens in mouse liver. If possible the corresponding gene symbol, EntrezGene ID and description are provided for each probeset. For signatures which were not derived from an Affymetrix platform, the original probeset IDs are listed in an additional column.(XLS)Click here for additional data file.


Machine learning is a powerful approach for describing and predicting classes in microarray data. Although several comparative studies have investigated the relative performance of various machine learning methods, these often do not account for the fact that performance (e.g. error rate) is a result of a series of analysis steps of which the most important are data normalization, gene selection and machine learning.In this study, we used seven previously published cancer-related microarray data sets to compare the effects on classification performance of five normalization methods, three gene selection methods with 21 different numbers of selected genes and eight machine learning methods. Performance in term of error rate was rigorously estimated by repeatedly employing a double cross validation approach. Since performance varies greatly between data sets, we devised an analysis method that first compares methods within individual data sets and then visualizes the comparisons across data sets. We discovered both well performing individual methods and synergies between different methods.Support Vector Machines with a radial basis kernel, linear kernel or polynomial kernel of degree 2 all performed consistently well across data sets. We show that there is a synergistic relationship between these methods and gene selection based on the T-test and the selection of a relatively high number of genes. Also, we find that these methods benefit significantly from using normalized data, although it is hard to draw general conclusions about the relative performance of different normalization procedures. Machine learning methods have found many applications in gene expression data analysis, and are commonly used to classify patient samples into classes, corresponding to for example cancer sub-type, based on gene expression profiles. Supervised learning is a powerful tool in these studies since it can be used both to establish whether the classes of interest can be predicted from expression profiles and to provide an explanation as to what genes underlie the differences between classes. The expression data in such studies typically undergo an analysis pipeline in which the most important steps are data normalization, gene selection and machine learning. Although there are several comparative studies of methods for normalization, gene selection and machine learning, none have studied how all of these analysis steps influence each other and the final model performance.A wealth of methods exists for microarray normalization, gene selection and machine learning. Normalization of microarray data involves several possible steps , includiet al. [et al. [et al. [et al. [et al. [et al. [Since gene selection is a necessary step in machine learning-based analysis of microarray patient data, all existing comparative studies have investigated the effect of gene selection and machine learning methods on classification performance. Most of these studies considered tumor classification. However, to the best of our knowledge, no study has also taken data normalization methods into account. Pirooznia et al.  studied  [et al.  investig [et al.  performe [et al.  investig [et al.  applied  [et al. -18. In p [et al.  used sevThe previous comparative studies have shown that classification performance varies a great deal from data set to data set. This is a challenge in comparative studies since it limits our ability to find general trends in terms of methods and combinations of methods that perform best across data sets. Here we approach this problem by comparing methods and pairs of methods on individual data sets, and then by visualizing trends across data set using heat maps. Thus we are able to draw general conclusions both about the individual effect of normalization, gene selection and machine learning on classification performance, and also to say something about synergistic effects that occur when these methods are used in combination. Our approach to studying synergy between methods, and the fact that we study the effect of normalization as well as gene selection and machine learning, makes our study unique. Our main conclusions are that Support Vector Machines (SVMs) with a radial basis kernel, linear kernel or polynomial kernel of degree 2 perform best across data sets. We show that these methods exhibit a synergistic relationship with gene selection based on the T-test and the selection of a relatively high number of genes. All these methods perform better on normalized than on non-normalized data, however, while the radial basis kernel and the linear kernel perform best when the data is not background corrected, the polynomial kernel benefit from background corrected data.We evaluated classification models induced from seven different two-channel microarray expression data sets with two known classes Table . Each clCombinations of methods were validated by employing a double cross validation (CV) approach where the inner loop was used to find good parameter settings ) and the outer loop was used to estimate the predictive power of the final model in terms of average error rate Figure . NormaliThe average error rate obtained by evaluating all the different combinations of methods on the seven data sets was 0.1512 with a standard deviation of 0.1195  have the best classification capabilities followed by Artificial Neural Networks (NNs), while Decision Trees (DTs) have the worst performance Figure . HoweverOne of the main questions we ask in this study is to what degree we can observe synergistic relationships between methods. In order to answer this question we need to study combinations of methods. To have enough data to claim statistical significance, we limited our study of synergy to comparing all method-pairs of the same type, i.e., machine learning-normalization (M-No), machine learning-number of genes (M-N), machine learning-gene selection (M-G), gene selection-normalization (G-No), gene selection-number of genes (G-N) and normalization-number of genes (No-N) , thus methods that do not significantly deteriorate the performance of other individually well-performing methods are of interest as well as methods that decrease performance of other methods and therefore represent combinations that should be avoided. Method-pairs that contain the number of selected genes (N) are generally exhibiting the pattern already seen for single methods; too few genes severely reduce the performance of most methods while more than 200 genes do not significantly improve performance. Thus we will mainly focus on pairs consisting of methods for machine learning, normalization and gene selection . However, the picture is rather complex. For example, the third best method, SVM Poly 2, works best with background corrected normalizations (No 2 and No 4), while normalizations without background correction is no better than non-normalized data. A clear trend is that the T-test is the best gene selection partner to all machine learning methods, although the actual improvement sometimes is rather small. SVM Rb combined with the T-test is the best performing method-pair, and T-test is also the best gene selection partner for SVM Linear and SVM Poly 2. Almost all pairs of gene selection and normalization methods are significantly better than almost all other pairs of this type in at least two data sets. The lack of clear trends is maybe not so surprising considering the complex behavior of these methods when studied individually. Surprisingly, there is a synergy between the T-test and non-normalized data, which is the best pair of gene selection and normalization. If we look at the performance of the normalization methods for each gene selection method separately, we again see the trend from our analysis of individual methods; three to four data sets result in significantly better models when normalized while at least two other data sets actually result in better models when not normalized.The comparison of method-pairs confirms the strong performance of the individually best performing machine learning methods . These methods require at least 150 genes to achieve their best classification performance. A trend is that SVM Rb is somewhat more robust with respect to normalization, gene selection and number of selected genes than the other well-performing machine learning methods. For example, both SVM Linear and Poly 2 perform worse when not using gene selection based on the T-test, while SVM Rb performs almost as well with one of the two other methods. This example illustrates a very important trend in our study; although some methods perform well by themselves, their performance can be severely hampered by unfortunate choices for the other methods. In this context, data normalization is shown to be very important; the wrong normalization method can severely reduce the performance of many of the best machine learning methods. Also, these best methods perform better on normalized than non-normalized data. There is also a very clear positive effect of using normalized data with DT methods, while NN methods actually perform best on non-normalized data.Of particular interest are synergistic effects where two methods perform significantly better together than any of the two methods do individually. We specifically searched for such patterns  showed a synergistic relationship with normalizations not using background correction (i.e. No 1 and No 3).The data sets in this study result in machine learning models that perform rather differently both with respect to error rate and also with respect to relative performance between methods. Although all data sets are from two-channel microarray experiments related to cancer, the classes we have used to train and evaluate our models are defined somewhat differently. However, all classes are based on clinical observations . We were rather surprised to see that all data sets upheld predictive performance even when 1000 genes were selected from the gene selection methods, and that the error rate actually improved for three data sets. Interestingly, the three data sets for which performance continuously improved when more genes were selected, were the same three data sets that have the fewest number of discriminatory genes Table ; Ye, SørThe gene selection methods considered in this study were all filter-based methods. These methods select genes prior to machine learning by typically ranking genes based on their individual ability to separate classes . UnfortuIn this study, we selected methods that are commonly used to analyze gene expression data. Thus, although we see rather small difference between, for example, gene selection methods, this does not mean that gene selection is not important. Initially, we also included a gene selection method that rank genes based on variance, however, the consistently poor performance of this unsupervised method spurred us to omit it from further analyzes.No 1, G T-test, N 100, M SVM Linear is compared to No 2, G T-test, N 100, M SVM Linear). Such paired tests are sensitive, and reveals interesting significant differences even when average performances are rather similar  with a radial basis kernel followed by SVMs with a Linear kernel and SVMs with a polynomial kernel of degree 2. We showed that there is a positive, synergistic relationship between these methods and gene selection based on the T-test and the selection of at least 150 genes.We used seven previously published 2-channel microarray experiments with a common reference design . All datSince the aim of this study was to investigate how the performance of various machine learning methods is affected by the choice of normalization method and gene selection, we must have true class partitionings to compare with the predicted classes from the models. To make it easier to compare the results for different models and data sets, we choose to divide data into two distinct classes:Alizadeh: 133 samples corresponding to patients with different lymphomas, including patients with diffuse large B-cell lymphoma (DLBCL) [ (DLBCL) . SamplesFinak: 66 samples profiled using Agilent microarrays [roarrays . We defiGalland: 40 samples profiled using Agilent microarrays [roarrays . The claHerschkowitz: 106 samples. We define the two classes according to the level of ER expression; high (59) or low (47) [low (47) .Jones: 91 samples. We define the classes as the two most distinctive ones in the data set; the cancerous (72) and the non-cancerous (19) samples [ samples .Sørlie: 73 samples. The classes are defined according to the level of ER expression; high (55) or low (18) [low (18) .Ye: 87 samples. We define the two classes as in the original study [al study ; metasta2-ratios between the treated channel and the reference channel .Expression values in this study are the logThe normalization methods we investigated were the four possible combinations of two dye-normalization methods, global MA-loess  and prinNo 0: Raw data No 1: Print-tip  MA-loess, no background correctionNo 2: Print-tip  MA-loess, background correctionNo 3: Global MA-loess, no background correctionNo 4: Global MA-loess, background correctionNo 1 and 2 were not performed for the data sets Galland and Finak, since print-tip normalization is not relevant for the Agilent microarrays.k most significant eigengenes to estimate missing values [svdImpute in the package pcaMethods to impute missing values.In each of the seven data sets there were spots flagged by the scanner or the experimentalist, and some spots also had a lower signal than the background (in the cases where background correction was adopted). These were all marked as missing values. A discussion on how to treat flagged spots can be found in . We chosg values ,38. Accog values , these tIn addition to the above described filtration and missing value imputation, we also remove some samples with incomplete annotations (i.e. class information). To avoid including duplicate genes we choose to compute the mean value of these.In this study we perform double cross validations; a 10-fold inner CV was used to optimize parameters in the methods and a 5-fold outer CV was used to estimate the final classification performance .In order to reduce the number of genes used by the machine learning methods, we always performed gene selection. There are several different types of gene selection techniques , but we T-test: A two sided, two sample t-test was used [t.test from the package stats in R.was used ,39 and gRelief: The Relief-algorithm assigns a value to each gene based on how well it separates nearest neighbors with different class labels [relief from the package dprep in R. The algorithm was modified so that all observations are chosen once rather than being sampled with replacement.s labels . We implPaired distance: This is, to the best of our knowledge, a new gene selection approach based on investigating pairs of genes. We first chose the 1000 genes with the highest variance and the 1000 genes with the highest absolute mean value, and retained the union of these genes. For each of the chosen genes we then calculated two medians, one for those observations belonging to the first class and one for those observations belonging to the second class. In the two dimensional space spanned by the gene-pairs, we then calculated the Euclidian distance between the two corresponding pairs of medians. Finally, we ranked pairs of genes according to these distances.The Decision trees (see Machine learning methods) also used an embedded technique for gene selection. Initially, we also used variance as an unsupervised gene selection method, but this strategy was later dropped due to its inferior performance.Given the ranked lists from the gene selection methods, we selected 21 different numbers of genes to be used for inducing machine learning models , Artificial neural networks (NNs) and Decision trees (DTs). Most machine learning methods have several parameters that need to be set before inducing the model. To avoid overfitting the models, some of the most important parameters were optimized using an inner cross validation. Parameters not specifically mentioned were set to default values.n dimensional space where the classes are separated by a hyperplane constructed to create the largest possible margin between the classes [ksvm from the package kernlab in R to build SVM models (see [ksvm function), before machine learning methods were applied. After some experiments with the different types of SVM classifications  we decided to use nu-scv for all the SVM models. We also experimented with different kernels and decided to use the following four kernels; linear, polynomial with degree 2, polynomial with degree 3 and radial basis kernel. The parameter nu was set to 0.2, which is the default value in ksvm. The inverse kernel width, sigma, for the radial basis kernel function was optimized using a grid search by choosing the value with the lowest cross validation error rate obtained from the ksvm function.The idea behind SVMs is to use an implicit kernel function to map the data into an  classes ,41. We uels  that transform the input values to an output value based on a threshold function applied to the weighted sum of the input values ,43,44. WN models . We set rpart from the package rpart in R to build the DT models. The trees were pruned by choosing the number of splits from the tree with the lowest cross-validated error [DTs are built by iteratively splitting the data using the most separating gene, thus forming a tree with nodes and leaves ,43,44. Wed error .Including settings (such as kernels in SVMs), we investigated the performance of the following eight machine learning methods:DT Gini:                Decision tree with Gini index as splitting criteriaDT Information:   Decision tree with Information index as splitting criteriaNN One layer:      Artificial Neural Network with one hidden layerNN No layer:        Artificial Neural Network with no hidden layerSVM Linear:         Support Vector Machine with a linear kernelSVM Poly2:          Support Vector Machine with a polynomial kernel of degree 2SVM Poly3:          Support Vector Machine with a polynomial kernel of degree 3SVM Rb:               Support Vector Machine with a radial basis kernellm and anova  in the package stats in R to conduct the regression analyses. Test for significant predicting variables was done using a Bonferroni corrected threshold (0.05 divided by the number of variables: 10).The possible combinations of the different methods described above gave in total 14685 models with calculated error rates and corresponding standard deviations. To analyze these results, we used multiple linear regression models with error rate as response and Data set (S), Normalization (No), Gene selection method (G), Number of selected genes (N) and Machine learning method (M)  in the heat maps equal the the number of data sets where method (or pair of methods) i is significantly better than method (or pair of methods) j. The number of significant data sets for combinations including No 1 or No 2 was adjusted by multiplying by 7/5 since these methods were not applied to two data sets. The heat map was created with the function levelplot in the package lattice in R.We used the Wilcoxon signed rank test to determine whether there was any significant difference between error rates obtained by different individual methods Figure  and betwJÖ was involved in developing the ideas presented in this article, implemented and conducted the machine learning, analyzed and interpreted the results and drafted the manuscript. EF collected the data sets, implemented filtration and missing value imputation and contributed in the general discussions. ML implemented and conducted the normalizations of the data. PR was involved in developing the ideas presented in this paper and interpreted the results. TRH was involved in developing the ideas presented in this article, analyzed and interpreted the results and wrote parts of the manuscript. All authors read and approved the final manuscript.Error rates obtained for all combinations of methods and data sets. A table of the 14 686 different combinations of methods and data sets. Columns include Data set (D), Machine learning method (M), Gene selection (G), Number of selected genes (N), Normalization method (No), Error rate (E), Standard deviation of error rate (Esd), Adjusted error rate (Eadj) and Standard deviation of adjusted error rate (Esdadj).Click here for fileThe predictive performance of individual methods across data sets. Comparisons of the predictive performance of individual methods .Click here for fileThe predictive performance of pairs of methods across data sets. Comparisons of the predictive performance of pairs of methods .Click here for fileSynergistic relationships between pairs of methods across data sets. The matrix show the number of data sets in which pairs of methods (given by one row and one column) performed significantly better  or significantly worse  than the best/worst method did individually. The Wilcoxon signed-rank test was used to compare the error rates of all combinations containing the pair against the error rates of all combinations containing the best/worst single method. Significance was determined using a p-value threshold of 0.05.Click here for file