Information extraction from microarrays has not yet been widely used in diagnostic or prognostic decision-support systems, due to the diversity of results produced by the available techniques, their instability on different data sets and the inability to relate statistical significance with biological relevance. Thus, there is an urgent need to address the statistical framework of microarray analysis and identify its drawbacks and limitations, which will enable us to thoroughly compare methodologies under the same experimental set-up and associate results with confidence intervals meaningful to clinicians. In this study we consider gene-selection algorithms with the aim to reveal inefficiencies in performance evaluation and address aspects that can reduce uncertainty in algorithmic validation.A computational study is performed related to the performance of several gene selection methodologies on publicly available microarray data. Three basic types of experimental scenarios are evaluated, i.e. the independent test-set and the 10-fold cross-validation (CV) using maximum and average performance measures. Feature selection methods behave differently under different validation strategies. The performance results from CV do not mach well those from the independent test-set, except for the support vector machines (SVM) and the least squares SVM methods. However, these wrapper methods achieve variable (often low) performance, whereas the hybrid methods attain consistently higher accuracies. The use of an independent test-set within CV is important for the evaluation of the predictive power of algorithms. The optimal size of the selected gene-set also appears to be dependent on the evaluation scheme. The consistency of selected genes over variation of the training-set is another aspect important in reducing uncertainty in the evaluation of the derived gene signature. In all cases the presence of outlier samples can seriously affect algorithmic performance.Multiple parameters can influence the selection of a gene-signature and its predictive power, thus possible biases in validation methods must always be accounted for. This paper illustrates that independent test-set evaluation reduces the bias of CV, and case-specific measures reveal stability characteristics of the gene-signature over changes of the training set. Moreover, frequency measures on gene selection address the algorithmic consistency in selecting the same gene signature under different training conditions. These issues contribute to the development of an objective evaluation framework and aid the derivation of statistically consistent gene signatures that could eventually be correlated with biological relevance. The benefits of the proposed framework are supported by the evaluation results and methodological comparisons performed for several gene-selection algorithms on three publicly available datasets. Modern biological and biomedical research has been challenged by the relatively new high-throughput methods of genomic, proteomic and metabolomic analysis , such asDespite the plethora of methods that have been developed for information extraction from microarrays, such information has not yet been widely used in diagnostic or prognostic decision-support systems . This isMany studies have addressed the different issues involved in data analysis from different microarray studies -14. HereAdditionally, various statistical issues can potentially affect the results of a study. For example, the bootstrapping strategy for the generation of random folds for training and testing is an issue of particular importance for performance comparison of gene selection approaches. Moreover, cross-validation (CV) can induce a certain bias due to mixing of training and testing samples . Leave-oThrough our study we aim to reveal shortcomings in the evaluation of gene selection approaches and address methodological issues that may lead to more objective evaluation schemes. Furthermore, we consider sample-specific effects on the performance and stability of prediction algorithms. For that purpose, we have selected three well known publicly available data sets, namely the Van't Veer et al. , the GolOur ultimate objective, besides presenting and highlighting the advantages and disadvantages of the tested methodologies, is to explore fundamental characteristics in the analysis of microarray data sets and identify methodological aspects that may influence the evaluation of algorithms. As a performance validation scheme we consider ten-fold CV repeated ten times, i.e. 100 iterations in total. It is worth recalling that CV with multiple gene-sets or different splits of data sets always presents some bias threats ,24. In oThree well known and publicly available data sets first published at, ,15 and  were conThe breast cancer (BC) data set  containsThe leukemia data set in  consistsThe colon cancer (CC) data set in  consistsTo highlight differences between the three data sets, a three-dimensional principal component analysis (PCA) depicted in Figure Three basic types of experimental scenarios are conducted in order to estimate the performance of tested methodologies:independent test-set scenario the algorithms are trained using the complete training set together with backward feature elimination process. Thus, a varying number of genes are recursively eliminated until 100 surviving genes are left and subsequently one gene is eliminated from this point onwards  defines a descent-ordered list of gene frequencies and Fi is the frequency rate of gene i defined as its number of occurrences (Oi) over the total number of runs (R). Furthermore, S is the total number of genes in the signature of each algorithm, defined on the basis of best performance on the independent test-set (Tables where [t Tables , 2, 3.BC: Breast Cancer; CC: Colon Cancer; CI: Confidence Interval; CV: Cross Validation; RFE-FLDA: Recursive Feature Elimination based on Fisher's Linear Discriminant Analysis; RFE-FSVs-7DK: Recursive Feature Elimination based on Fisher's ratio of Support Vectors using a 7 Degree polynomial Kernel; RFE-LNW1, RFE-LNW2: Recursive Feature Elimination based on Linear Neuron Weights; RFE-LNW-GD: Recursive Feature Elimination based on Linear Neuron Weights using Gradient Descent; RFE-LSSVM: Recursive Feature Elimination based on Least Square Support Vector Machines; RFE-RR: Recursive Feature Elimination based on Ridge Regression; RFE-SVM: Recursive Feature Elimination based on Support Vector Machines.MZ and MEB contributed equally to the work. MZ drafted most of the manuscript, designed the evaluation on a per run and a per-subject basis and performed data analysis on this part. MEB run the experiments, wrote all MatLab codes and contributed to data analysis for the independent test-set and 10-fold cross validation scenarios, while he assisted in drafting the manuscript. DK, VD, GT and MT assisted with the study design, data analysis and drafting of the manuscript. GT assisted with the statistical analysis of results. All authors read and approved the final manuscript.


This review provides a focused summary of the implications of high-dimensional data spaces produced by gene expression microarrays for building better models of cancer diagnosis, prognosis, and therapeutics. We identify the unique challenges posed by high dimensionality to highlight methodological problems and discuss recent methods in predictive classification, unsupervised subclass discovery, and marker identification. Gene expression microarrays provide a wealth of information on gene expression patterns and cancer pathways with potential for (1) cancer diagnosis, prognosis, and prediction of therapeutic responsiveness ; (2) disn) with typically at most several hundred clinical samples. A rule of thumb is to have at least 10 training samples per feature dimension  the best test error was achieved using a finite number of features; (2) using an infinite number of features, test error degrades to the accuracy of random guessing; and (3) the optimal dimensionality increases with increasing sample size. These observations are consistent with the ‘bias/variance dilemma’  searches, which combine forward and backward steps, and more complex simulated annealing and genetic algorithms, can also be applied . FinallyIn microarray data analysis, unsupervised clustering must be cautiously applied and may be unnecessary when samples come with appropriate and reliable supervising labels  features. Thus, many noisy/irrelevant features will dominate the  relevant features in determining how data points are partitioned, for example, many invariantly expressed genes used for microarray normalisation are irrelevant to classification or clustering. Rather than clustering samples using all genes, a practical alternative is to embed gene selection within unsupervised clustering – removal of noisy features improves clustering accuracy, which, in turn, guides a more accurate round of feature selection. Methods have been proposed along these lines , togetheAnother major challenge for clustering in high dimensions is estimating the number of clusters. Standard methods choose cluster number by best fitting the data while incurring least model complexity. However, under the widely used Bayesian information criterion . Being mMarker identification aims to discover those genes and their complex interaction effects that have statistically significant correlations with cancer phenotypes. As it is currently largely unclear how molecular variants and their interactions determine cancer pathogenesis and propensity, marker identification is valuable for improving understanding of the molecular mechanisms of cancers and for suggesting novel drug targets. Discovered markers may also define a subset of networked causal genes that regulate disease phenotype. A review of the current state of this effort is discussed by The objectives of feature selection for predictive classification and marker identification bear close resemblance. Although it is tempting to view these two problems as ‘one and the same’, this is often inappropriate. Inclusion of some true cancer markers in a feature set for cancer classification may provide negligible improvement in classification accuracy even though these markers are significantly associated with the cancer outcome of interest. A trivial example is where two markers are perfectly correlated, in which case only one of the two needs to be included in a predictive feature subset. A more interesting example is the one in which, even though two markers are only partially correlated, a classification model will not perceive any benefit from using both markers. This is illustrated below:A and B take on one of four possible discrete values and suppose the ground-truth statistics on class label C are Prob[C=‘cancer’∣A=1]=1.0; Prob[C=‘cancer’∣A=i]=0.5, i=2,3,4; Prob[C=‘cancer’∣B=3]=1.0; and Prob[C=‘cancer’∣B=j]=0.5, j=1,2,4. Suppose Prob[A=1]=0.1; Prob[B=3]=0.7; and Prob[B=3∣A=1]=0.5. Thus, A and B are both informative about the disease , and these variables are only partially correlated. However, in a small training set, it is quite possible that each time A=1, B=3 also occurs, even though Prob[B=3∣A=1] is much less than one. In this case, while association-based marker discovery might include both A and B, classification-based marker discovery would only include B, because the training set suggests no predictive benefit from including A.Let More generally, whether predictive gene selection will include a gene that possesses some predictive benefit will depend on the sensitivity of the criterion function used. For example, a predictive model may achieve the same estimated classification error rate using several different feature subsets, even if there is a unique true marker subset, with greatest class discrimination power. Another limitation of predictive gene selection is that most classification models lack interpretability, that is, they do not allow easy discernment of the underlying interactions between the identified markers. The sole focus of most predictive feature selection techniques is on defeating the curse of dimensionality. Exceptions to this include decision trees (if not too large) and Bayesian networks  on the discovered markers .k-fold cross-validation from the same data set, with the final, selected markers the ones with highest average ranking (and perhaps low variance/uncertainty). Nevertheless, the cost of increased stability in such approaches is an increase in computation.Ultimately, discovered cancer markers or subtypes must be validated against definitive biomedical ground-truth. However, the cost of such validation demands a high degree of confidence in the knowledge extracted from microarray data by marker identification and clustering algorithms. Specifically, such knowledge extraction should not strongly depend on the particular random sample of data used or on variable aspects of the algorithms. Many clustering algorithms find only locally optimal solutions whose quality depends on the pseudorandomly chosen initial cluster parameter values . Also, g


As part of its broad and ambitious mission, the MicroArray Quality Control (MAQC) project reported the results of experiments using External RNA Controls (ERCs) on five microarray platforms. For most platforms, several different methods of data processing were considered. However, there was no similar consideration of different methods for processing the data from the Agilent two-color platform. While this omission is understandable given the scale of the project, it can create the false impression that there is consensus about the best way to process Agilent two-color data. It is also important to consider whether ERCs are representative of all the probes on a microarray.A comparison of different methods of processing Agilent two-color data shows substantial differences among methods for low-intensity genes. The sensitivity and specificity for detecting differentially expressed genes varies substantially for different methods. Analysis also reveals that the ERCs in the MAQC data only span the upper half of the intensity range, and therefore cannot be representative of all genes on the microarray.Although ERCs demonstrate good agreement between observed and expected log-ratios on the Agilent two-color platform, such an analysis is incomplete. Simple loess normalization outperformed data processing with Agilent's Feature Extraction software for accurate identification of differentially expressed genes. Results from studies using ERCs should not be over-generalized when ERCs are not representative of all probes on a microarray. Recently, the MicroArray Quality Control (MAQC) Consortium published a series of papers on an important effort to address ongoing issues concerning the reliability of microarray data -6. Some ERCs are extremely valuable for quality control because their true concentrations are known by design. Since one knows what the microarray measurement should be, one can examine how well the microarray gives the right answer. One aspect of the study reported by Tong et al  was to lUnfortunately, no similar evaluation of data processing methods was presented for the Agilent two-color data in . While tThis paper examines six Agilent two-color MACQ datasets. Datasets were produced by three sites  with two different RNAs (A and B).2(10) ≈ ± 3.32; ± log2(3) ≈ ± 1.59; or log2(1) = 0. Tong et al present a figure  that shFigure The datasets considered here have the same RNA in the red and green channels. That is, other than ERCs, all spots have true log-ratio = 0. The true log-ratio is therefore known for every probe on the array, so these arrays are informative about the effectiveness of data processing methods. The bottom panel of Figure One of the most common uses for microarray data is to detect differentially expressed genes. In the MAQC datasets, one hopes that the ERCs with true log-ratio 10, 3, 1/3, or 1/10 can be detected among the remaining genes with true-log-ratio 0. When detection is the scientific goal of a study, the most appropriate way to judge accuracy is with the sensitivity and specificity of detection. Similar to  and 14]14], threDetection was superior using the mean or the SAM-statistic compared to the t-statistic, corroborating the finding of  for anotFigure The analysis methods and findings here are very similar to the study by Zahurak et al . The conThe three different ways of processing two-color data that were considered here  produce nearly identical curves for the observed log ratios plotted against expected log ratios [see Figure 4 from  and AddiTong et al  are careMicroarray data with ERCs are extremely valuable for understanding the behaviour of the microarray signal and the operating characteristics of data processing methodologies. However, ERC probes may not be representative of all probes on a microarray, as seen here. Moreover, a single ERC experiment cannot be representative of all real microarray experiments, since different experiments will exhibit different patterns of differential expression. As a specific but important example, datasets in which only a handful of genes, the ERCs, are differentially expressed are extremely well-suited to the assumptions of loess normalization. Therefore, such datasets cannot be used to evaluate the effectiveness of loess normalization for data with lots of differential expression.Clearly, the major difference among processing methods is the behavior of low intensity genes. One method for handling highly-variable low intensity genes is to simply discard them. However, Kerr et al  showed tThe results here show an advantage for alternative processing of the data over processing by the Feature Extraction software. Clearly, the FE data have greater variability at low intensities. This leads to worsened specificity of detection because some low-intensity genes with true log-ratio equal to zero exhibit large log-ratios. Zahurak et al  offer soIn the alternative methods of data processing, which out-performed FE, there was no compelling evidence to favor or disfavor background adjustment (BA). However, Zahurak et al  identifiChoosing a data processing method is an important step in the analysis of microarray data. The MAQC datasets considered together with previous spike-in datasets  disfavouth percentile of t-statistic denominators. Scripts for ROC curves and AUC calculation were downloaded from [There were six datasets from Sites 1, 2, 3 and RNAs A and B. All datasets had 5 replicates except  and  had 4 replicates due to failed assays. For each dataset, spots with any measurement in any replicate that were flagged as saturated were removed from further analysis. The median pixel intensity was used as the spot signal. For the BA data, the median background intensity was used as the local measurement of background and subtracted from spot signal. For loess normalization, the span was 4000 datapoints, or about 10% of the data. Note that each ERC was represented by 30 spots on the arrays and these were treated as separate "genes" in ROC analysis. The SAM-statistic is the classical t-statistic with a constant δ added to the denominator. In this analysis δ was set equal to the 90ded from .The author(s) declares that there are no competing interests. KFK analyzed the data and wrote this report. The author read and approved the final manuscript.Ratio-intensity plots for all arrays. The plots show log-ratios compared to signal intensity for three versions of the data.Click here for fileROC curves for the mean, t-statistic, and SAM statistic. The plots summarize the sensitivity and specificity for detecting differentially expressed genes using three different test statistics applied to three versions of the data.Click here for fileTest statistic values plotted against average spot intensity. The plots show the behavior of test statistics as a function of signal intensity for three versions of the data.Click here for fileObserved log-ratios compared to expected log-ratios. The plots show observed log-ratios compared to expected log-ratios for three versions of every array.Click here for file


Hierarchical clustering is a widely applied tool in the analysis of microarray gene expression data. The assessment of cluster stability is a major challenge in clustering procedures. Statistical methods are required to distinguish between real and random clusters. Several methods for assessing cluster stability have been published, including resampling methods such as the bootstrap.We propose a new resampling method based on continuous weights to assess the stability of clusters in hierarchical clustering. While in bootstrapping approximately one third of the original items is lost, continuous weights avoid zero elements and instead allow non integer diagonal elements, which leads to retention of the full dimensionality of space, i.e. each variable of the original data set is represented in the resampling sample.Comparison of continuous weights and bootstrapping using real datasets and simulation studies reveals the advantage of continuous weights especially when the dataset has only few observations, few differentially expressed genes and the fold change of differentially expressed genes is low.We recommend the use of continuous weights in small as well as in large datasets, because according to our results they produce at least the same results as conventional bootstrapping and in some cases they surpass it. Cluster analysis is a widely used tool for interpretation of gene expression experiments. It allows to group genes as well as (tissue) samples in classes (clusters) of similar characteristic profiles. Class assignment results from applying a similarity measure (i.e. distance measure or correlation) and a selected method to calculate the distance of an object to a class . The algorithms are well-defined and reproducible, however the choice of different similarity measures and cluster methods leads to different results .Algorithms for hierarchical agglomerative classification exist for a long time -6. They A problem in cluster analysis is to discriminate between real and random clusters. The latter arise from random variation of gene expression measurements due to technical variation and biological variability. A measure of cluster stability is needed to solve this problem.Several methods for validating clusters internally have been described -19. The Our proposed method is similar to the bootstrap. Only a few applications of the bootstrap method in cluster analysis over arrays can be found in the literature. Zhang and Zhao  use the Also Bhattacharjee et al.  use bootKerr and Churchill  use the Dudoit and Fridlyand  use baggn)n [As bootstrapping is a drawing with replacement and the size of the bootstrap sample is the same as the original data size, some observations are omitted. The expected proportion of points in the original sample absent from the bootstrap sample is given by (1 - 1/n)n , which cSeveral methods for combining individual dendrograms to a consensus tree exist. The majority rule consensus tree , which oTwo partitions can be compared by application of a similarity measure such as the Rand index ,21. In cHere we report of a new resampling method that is based on continuous weights. The creation of resampled datasets based on weighted sampling is similar to the creation of bootstrap samples but instead of drawing whole observations, random floating-point numbers larger than zero are drawn and the observations are weighted by these numbers so that each observation is represented in the resampled dataset. We compare this method to the conventional bootstrap to show where it is advantageous.We compared continuous weights and the conventional bootstrap using real microarray gene expression data as well as simulated data. Real data were preprocessed as specified in the corresponding papers. Majority rule consensus trees were generated from the individual dendrograms derived from continuous weights or bootstrap. For each combination original dendrogram/consensus tree obtained by weight matrix or bootstrap the weighted Rand index was calculated.We used two real datasets for evaluating our new method and comparing it to the bootstrap. The first dataset was the uveal melanoma dataset of Tschentscher et al. . In thisComparing continuous weights and bootstrap revealed for these datasets that in three cases  the clustering was exactly the same .Next, continuous weights and bootstrap were compared using a dataset where seven features, i.e. the maximal age of death and some birth and pregnancy data, were measured in 22 primates ,27. HierCebus capucinus), and their separation from the half-and-half apes and the new world monkeys. The other families and subfamilies are not replicated. The failure of bootstrap again could result from the small number of features in the data set.Results are shown in Figure Simulation studies uncovered relationships between the cluster behavior and the number of differentially expressed genes, the number of observations, the size of the groups, the fold change and the number of groups respectively.In the first simulation Figure  data forIn the next simulation Figure  the settIn another simulation the number of genes was kept constant at 100 genes, the fold change was constantly 9 and the number of differentially expressed genes was varied. The number of variables was varied between 10 and 25 per group with both groups having the same size resulting in 20 to 50 variables Figure . As in tIn yet another simulation the number of groups was increased to three or four groups with an equal number of variables in each group Figure . The numHierarchical clustering is an important explorative tool in microarray data analysis. It is often applied to get a first impression of the data structure of microarray gene expression experiments. It is important to assess the reliability of the clusters because random clusters may lead to a false interpretation of the data. Bootstrapping is one of the methods, which is used to determine the cluster stability. Nevertheless sometimes the results of bootstrapping are rather uninformative especially if the number of features is small. As about one third of the original observations is omitted in a bootstrap sample this finding is not surprising. Therefore we developed an alternative method similar to the bootstrap based on continuous weights, which have the advantage that every observation of the original dataset is retained in the resampled dataset. We hoped that this fact should improve the results of the conventional bootstrap. We evaluated our new method by real and simulated data and compared it to the bootstrap. The weighted Rand Index was applied to compare two partitions obtained by hierarchical clustering. By means of real data we could show that the continuous variant leads to more meaningful results than the conventional bootstrap when the number of features is small, and fares comparably when the number of features is large (as in many data sets obtained from microarray experiments).Analysis of the 24 datasets of the uveal melanoma revealed that in most cases the same or nearly the same consensus trees are obtained using continuous weights and bootstrap. Sometimes the frequency of the clusters in the samples is higher when weights are used.In some cases continuous weights led to more accurate information about cluster membership of individual observations. In the case of chromosome Y we made use of biological knowledge, i.e. the sex determines to which cluster an observation belongs, to confirm the true classification. We could show that only clustering methods based on weighted correlation distances are able to detect this.Also with the primate dataset continuous weights outperformed the bootstrap. Again this result was expected because the dataset is very small.The size of the dataset seems to be one of the important criteria for the advantage of weights over bootstrap. Especially in small datasets it is very important to consider every observation.Simulation studies confirmed the benefits of continuous weights over bootstrapping. The new method is especially advantageous the smaller the number of genes and, above a threshold, the smaller the fold change. Of course the fold changes we used for simulation studies are not at all realistic for microarray gene expression data where already a fold change of two denotes differentially expressed genes. Yet in our settings we had to use such high fold changes to see any differences between continuous weights and bootstrapping. In real microarray gene expression data there are other factors such as the dependence and high correlation of microarray data that make continuous weights act better than bootstrapping.Our results indicate that more simulation studies would be helpful to characterize the merits of continuous weights compared to the bootstrap. Simulated datasets should mimic microarray data sets more realistically to better understand the advantages of continuous weights.Computing times are comparable as both bootstrapping and sampling weights have O(n) computing times, as have the calculation of the correlation coefficient in the weighted and unweighted variant, and every subsequent step was carried out in the same fashion with either method.Nevertheless the use of continuous weights is strongly recommended because they perform at least as well as the bootstrap and in some cases they even surpass it.It may be promising to study if in methods, which use the bootstrap as a part of it, a substitution of the bootstrap by our proposed method could improve the results. Methods coming into consideration are those from Kerr and Churchill  or DudoiUp to now we can only apply continuous weights in combination with the Pearson correlation. We plan to adapt the spearman correlation accordingly.Furthermore we want to extend the application of continuous weights to other fields where bootstrap is employed such as k-means.For comparing the new method and the bootstrap real and simulated dataset were used. Real data were normalized according to the methods described in the corresponding papers. The uveal melanoma dataset of Tschentscher et al.  was diviThe creation of random continuous weights requires the generation of a suitable probability distribution function for the weights. The following considerations are useful for this purpose: if a procedure analogue to the bootstrap is applied on the level of variables it is equivalent to the usage of a weight vector for the original data whose elements are realizations of a multinomial distribution. An alternative procedure for bootstrap is the application of a weight vector with non-zero elements permitting non-integer diagonal elements. Thus the full dimensionality of space is maintained.2 with μ and σ as mean and standard deviation of the variable's logarithm. We used this distribution with μ = -log2 and σ2 = 2*log2 deliberately as basis for drawing of the weights to attain the desired mean and variance. For each resampled dataset each observation from the original dataset was assigned a random weight, a correlation matrix was computed using the weighted Pearson correlation as similarity measure, i.e. the correlation coefficient was obtained by the formulaThe underlying distribution of a drawing with replacement is the binomial distribution. This approximates to the Poisson distribution for a large number of observations and a small probability of success. Thus in a bootstrap sample the asymptotic distribution of the number of drawings per variable is Poisson(1) i.e. the expected value and variance is 1. To attain comparability with the bootstrap, the distribution from which weights for the weighted sampling are drawn has to have similar characteristics as the Poisson distribution but has to be continuous to allow every value for the diagonal elements. No continuous generalization of the Poisson distribution exists. Thus the distribution in demand has to have an expected value and variance of one and has to be positive. This is true for e.g. the lognormal distribution where -2μ = σxik denotes the kth feature on the ith specimen.) and the distance matrix was generated using the transformation dij = 1 - rij. Resampled datasets were clustered hierarchically using average or complete linkage, where all patients had initially the same weight (note that the weighting was used for genes and not patients). Individual dendrograms were summarized in a majority rule consensus tree according to published methods [.IGD participated in the design of the study, carried out the studies and drafted the manuscript. TB contributed to the design of the study. JH conceived of the study and participated in its design. MN participated in the simulation studies. KHJ participated in the design and coordination of the study. All authors read and approved the final manuscript.


Affymetrix high density oligonucleotide expression arrays are widely used across all fields of biological research for measuring genome-wide gene expression. An important step in processing oligonucleotide microarray data is to produce a single value for the gene expression level of an RNA transcript using one of a growing number of statistical methods. The challenge for the researcher is to decide on the most appropriate method to use to address a specific biological question with a given dataset. Although several research efforts have focused on assessing performance of a few methods in evaluating gene expression from RNA hybridization experiments with different datasets, the relative merits of the methods currently available in the literature for evaluating genome-wide gene expression from Affymetrix microarray data collected from real biological experiments remain actively debated.cis-acting expression regulators. The PDNN method demonstrated superiority over all other methods in every comparison, whilst the default Affymetrix MAS5.0 method was clearly inferior.The present study reports a comprehensive survey of the performance of all seven commonly used methods in evaluating genome-wide gene expression from a well-designed experiment using Affymetrix microarrays. The experiment profiled eight genetically divergent barley cultivars each with three biological replicates. The dataset so obtained confers a balanced and idealized structure for the present analysis. The methods were evaluated on their sensitivity for detecting differentially expressed genes, reproducibility of expression values across replicates, and consistency in calling differentially expressed genes. The number of genes detected as differentially expressed among methods differed by a factor of two or more at a given false discovery rate (FDR) level. Moreover, we propose the use of genes containing single feature polymorphisms (SFPs) as an empirical test for comparison among methods for the ability to detect true differential gene expression on the basis that SFPs largely correspond to A comprehensive assessment of seven commonly used data extraction methods based on an extensive barley Affymetrix gene expression dataset has shown that the PDNN method has superior performance for the detection of differentially expressed genes. The MM probe is designed to distinguish noise caused by non-specific hybridization from the specific hybridization signal.Affymetrix GeneChip microarrays are the most popular high density oligonucleotide gene expression arrays and have become an invaluable tool in genomics studies worldwide. Each gene on an Affymetrix microarray GeneChip is typically represented by a probe set consisting of 11 different pairs of 25-bp oligos covering features of the transcribed region of that gene. Each pair consists of a perfect match (PM) and a mismatch (MM) oligonucleotide. The PM probe exactly matches the sequence of a particular standard genotype, often one parent of a cross, while the MM differs in a single substitution in the central, 13Affymetrix microarrays inevitably introduce many sources of variation . NormaliThe first data extraction method provided as the Affymetrix default was the Average Difference (AD), a linear scale measure that relied upon the difference measure PM-MM to correct for non-specific binding. This measurement was superseded by the current standard MAS5.0 , which uCalibration datasets derived from mixture experiments , spike-iStudies utilizing benchmark datasets have typically observed a large effect of the normalization method on the outcome of the expression analyses -17. HoweMore recently, Harr and Schlotterer  introduca priori knowledge of which genes are differentially expressed. To address this challenge we used a novel strategy based on genes in which we detected single feature polymorphisms (SFPs). SFPs are genetic polymorphisms in observed expression within one particular feature (oligonucleotide probe) of a probe set (11 PM and MM probes) on the array [cis-acting regulators [In this article we present a comparison of the influence of seven commonly used data extraction methods on the detection of differentially expressed genes using a genome-wide gene expression dataset from eight genetically divergent barley lines. The major challenge arising from the use of this dataset is that one has no he array . Using tgulators . On thisThe present study implements seven methods commonly used in the literature to calculate expression indices from Affymetrix microarray gene expression data, which was collected from a well-designed genome-wide microarray hybridization experiment with eight genetically divergent barley cultivars. These methods are summarized in Table rijk . rijk represents the correlation coefficient between 22,840 corresponding pairs of gene expression indices calculated by methods i and j from the kth microarray sample. The lower triangle shows the overall correlation coefficients between all pairs of 22,840 gene expression indices calculated from methods i and j across all 24 samples . It is clear that the seven methods may be separated into two groups  according to the correlation coefficients. The coefficient of correlation is greater than 90% within each of the groups but becomes less than 80% between the two groups. The same pattern of correlation in gene expression estimate between these seven methods was also recovered in the analysis of gene expression profiles on two yeast strains. Notably, all of the methods in the first group were based on use of both PM and MM values (with the exception of MBEI PM), while the methods in the second group were based on PM value only. However, the average correlation coefficient between MBEI PM and MBEI PM-MM was as high as 0.988, therefore the division of the seven methods into two groups was unlikely to be caused by using either the PM-MM model or the PM only model.To explore the consistency of the 22,840 barley gene expression indices estimated from the seven different methods, we calculated Pearson's Product Moment Correlation coefficients in the expression estimates and the correlation analyses are summarized in Table p-value < 10-5, Mann-Whitney U-test), suggesting that the different methods have a profound effect that goes beyond the variance observed across the biological replicates, in support of previous findings [The diagonal elements in Table findings ,17.p-value < 0.0001, Mann-Whitney U-test), while the poorest performers were the GCRMA and MAS5.0 methods . The standard deviation obtained using the PDNN method was significantly lower than all other methods , except MBEI PM for which a similar standard deviation was obtained. In the analysis of the yeast dataset, the PDNN method also gave a superior performance over several of the other methods  as shown in Figure 2a [see Additional file We compared the ability of each method to calculate consistent gene expression values between biological replicates of a given barley variety using the intra-class correlation coefficients. The box plot in Figure To explain the different performances of the methods illustrated above, we investigated the effect of each step in processing the microarray datasets on estimates of the expression indices in the barley dataset. We tested use of different background correction methods but the same normalization and summarization steps in estimating the genome-wide gene expression indices, and calculated the correlation coefficient for each pair-wise comparison of background correction methods. The correlation coefficients for the MAS5.0 and RMA methods based on different background corrections were greater than 99% and 90%, respectively, which is greater than the correlation between different methods (60%–80%). Therefore the background correction methods did not have a significant effect on the correlation between methods. Similarly, the correlations for the AD and MBEI-PM methods based on different normalization methods were greater than 97% and 99% respectively, showing the normalization methods did not have a detectable effect on the correlation between methods either , while the worst two methods were MAS5.0 and GCRMA  analyses. It should be noted that the SFP analysis does not involve any of the methods under investigation here for quantifying gene expression. Thus, the SFP prediction provides an independent source of information for assessing performance of the methods in detecting differentially expressed genes.An important objective was to compare the ability of each method to identify genuine differential expression. To this end, we used a recently identified set of over 7000 barley genes containing single feature polymorphisms that largely represent gene expression markers (GEMs) corresponding to a combination of mainly gulators . On thisThe development of pre-processing methods for Affymetrix oligonucleotide gene expression data has been an area of active research and has led to the availability of a large and growing toolbox of statistical methods for data extraction. This presents a significant challenge for a researcher wanting to identify the most appropriate method to analyze her/his datasets. The present study examined the effect of different data extraction methods on the detection of differentially expressed genes in a barley Affymetrix oligonucleotide microarray dataset. Seven commonly used data extraction methods were used exactly as recommended by their developers, providing a directly relevant comparison of the methods as they will be used in practice by the majority of users of the software, and thus avoiding the well-known over-training problem associated with calibration datasets. The analysis exploits an extensive genome-wide gene expression dataset from eight barley varieties showing extensive variation at phenotypic, transcriptional and genotypic levels. The presence of three replicates for each variety gave a perfectly balanced experimental design and ideal data structure for the main aims of the present research as well as a high power to detect differentially expressed genes by the analysis of variance.It is clear from the present study that evaluation of the gene expression index is strongly affected by the data extraction method and this in turn has a strong influence on the ability to detect differential gene expression confidently. The seven commonly used methods can be divided into two groups according to the correlation structure in expression indices. Neither the use of different background correction nor normalization procedures could explain the marked variation in expression values estimated from the different methods, as shown previously . Therefop-value < 0.0001, Mann-Whitney U-test).Several studies have systematically compared different data extraction methods using tightly controlled calibration datasets, but in doing so, have restricted the comparison to limited amounts of data generated using a limited number of species and platforms ,12,13. Oa priori whether or not a given gene is truly differentially expressed. Therefore in comparing the sensitivity of each of the seven methods to detect differential gene expression, care and attention must be paid to ensure that detected differences in sensitivity among methods are not due to other factors. The Benjamini and Hochberg [The major statistical challenge in using real biological experimental datasets arises from the fact that one cannot know Hochberg  false dip-value threshold required by this method to declare differential expression for a given FDR level. This explains the excellent agreement observed for the differentially expressed genes with all of the other methods. The reproducibility of results from microarray experiments is a critical issue for data analysis methods. The seven data extraction methods showed varying sensitivities to the inherent biological variation expected within the system; the PDNN method produced the most consistent results across biological replicates, whilst MAS5.0 and GCRMA produced the poorest results.The seven data extraction methods were explored from several angles, including sensitivity, reproducibility and mutual agreement for the identity of differentially expressed genes. Across a range of FDR levels, the PDNN method had the highest sensitivity to detect differentially expressed genes and this was directly related to the less stringent In the absence of an expected outcome, detection of differential expression within those genes with single feature polymorphism was used to further assess the ability of each method to detect genuine differential gene expression. The set of differentially expressed genes identified by the PDNN method was significantly enriched for SFP genes compared to all other methods, reflecting the fact that the method incorporated the sequence information into its calculation of expression indices. The PDNN method may have the highest accuracy in detecting genuine differential gene expression compared to the other six data extraction methods. The GCRMA and MAS5.0 methods called only half the fraction of differentially expressed genes called by PDNN; however, their caution is unlikely to reflect improved prediction of genuine differentially expressed genes.Taken together, all comparisons suggest that the PDNN method is superior to its rivals for the detection of differentially expressed genes in the current dataset. In contrast, Shedden et al.  showed uWe have only used a parametric ANOVA to detect differentially expressed genes. However, variation due to the use of different test statistics is smaller than variation due to different processing methods ,17 so weThe reason for superior performance of the PDNN method based on the present dataset may lie in its use of the free energy statistical model to detect both the specific and non-specific bindings between probes and their corresponding target transcripts, which may accurately model the physical and chemical aspects of probe binding on Affymetrix microarray chips. This may be considered somewhat surprising given findings that positional dependent effects, but not interactions between bases that are physically close, add significant predictive power for specific signal probe effects .The question arising naturally from the present analysis is that of which is the best method for analyzing Affymetrix gene expression data with a view to identifying differentially expressed genes. However, the present study has considered a selection of highly distinguished approaches for data extraction as applied to a barley genome-wide gene expression dataset and recognizes that a greater number of datasets from both controlled experiments and calibration data will be necessary to answer this question. The method chosen will depend on the particular scientific question the study is designed to address and the priorities involved. For example, given the high number of differentially expressed genes detected in a typical microarray experiment, specificity may be a higher priority than sensitivity and influence the method(s) chosen to analyse the results.n = 1,..., 3) from each of eight genetically divergent barley varieties  known as Barke, Golden Promise, Haruna Nijo, Morex, Optic, OWB_D, OWB_R and Steptoe . Total mRNA was extracted from the plant leaves, and then hybridized to a Barley 1.0 Affymetrix microarray GeneChip, which consists of 22,840 probe sets , at the Iowa State University transcriptomics facility. A distributed probe set format array was used to prevent potential local image contamination from completely destroying the data of an entire probe set (PM and MM).The microarray data consisted of three biological replicates  from each of the two yeast strains , a high performance strain designated PHO, and a low performance strain designated PLO .The two yeast strains, whose gene expression data was analysed here, were the two haploid parental lines reported in our previous experimental analysis for the genetic dissection of quantitative trait loci affecting ethanol tolerance in budding yeast . The strThe raw signal intensities for each probe set (contained in the CEL files) were analysed by the seven most commonly used data extraction methods  implemented in the R statistical environment. The relevant software was downloaded from the Bioconductor website  to produk correlation coefficients, rijk,  were calculated between 22,840 (barley) or 5,814 (yeast) corresponding pairs of gene expression indices calculated by methods i and j from the kth microarray sample. The within method correlation was calculated as rimn, corresponding to correlation coefficients calculated from each possible pairing of the n = 3 (barley) or n = 4 (yeast) replicates for the mth barley cultivar  or yeast strain  using method i. The correlation coefficients (between methods and within method) were compared using the Mann-Whitney U-test.For the between method comparison i and j across all k = 24 (barley) or k = 8 (yeast) samples . The genome-wide gene expression indices and the correlation coefficients were then computed under the scenarios of changing a single step in the three-step normalization procedure  whilst maintaining the other two steps the same.Pearson's correlation coefficient was calculated between all pairs of 22,840 (barley) or 5,814 (yeast) gene expression indices calculated from methods Each of 22,840 barley genes  was tested for differential expression by one-way analysis of variance by partitioning the total variation in gene expression level into variation between groups (8 barley varieties or 2 yeast strains), denoted by p-value obtained. The false discovery rate (FDR) was controlled according to the standard method of Benjamini and Hochberg [and the associated Hochberg .r) according to the standard method of Snedecor and Cochran [For each of the 22,840 barley genes , we calculated the intraclass correlation coefficient ( Cochran  asn = 3 (barley) or 4 (yeast) replicates.for We implemented the method proposed in our previous paper  to detecZWL conceived of and designed the study. NJ analyzed the data and performed the statistical analyses. LJL contributed to the data analysis and wrote the manuscript. MJK and TJ participated in the design of the study and data analysis. XH provided the yeast dataset. EP, RW and AD were involved in generating the barley microarray data. All authors read and approved of the final manuscript.Pearson's Product Moment Correlation Coefficients among yeast gene expression indices calculated from seven different methods.Click here for filep-values across FDR levels.Statistical properties of estimated yeast gene expression indices from seven data extraction methods. (a) Intraclass correlation coefficients between biological replicates of the estimated expression indices for 5,814 genes; (b) Sensitivity for detecting differentially expressed genes; and (c) Calibration Click here for filePair-wise Pearson correlation coefficients between all pairs of 24 sets (8 barley cultivars × 3 replicates) of 22,840 gene expression indices calculated from the MAS5.0 and RMA methods with different background correction steps and for the AD and MBEI methods with different normalization steps.Click here for fileMutual predictability of the number of yeast genes declared differentially expressed from seven data extraction methods.Click here for file


Histopathology, clinical chemistry, hematology and gene expression data were collected from the rat liver and blood after treatment with eight known hepatotoxins. This report details the standardized experimental design and the different data streams that were collected  and a bio-available tissue (blood)) after treatment with eight known hepatotoxicants . The results of the study demonstrate the classification of histopathological differences, likely reflecting differences in mechanisms of cell-specific toxicity, using either liver tissue or blood transcriptomic data. The use of genomic approaches to better understand the adverse effects of environmental and xenobiotic exposures on human injury and disease processes engendered a great deal of early enthusiasm and excitement. This research initially focused on using gene expression alterations as measured by microarray analyses and is often referred to as 'toxicogenomics' . Quite eGene expression microarrays have developed over the past decade into a powerful tool for investigating biological, mechanistic, and disease processes in addition to developing genomic classifiers. Recent standardization efforts by the Microarray Quality Control Consortium, the Toxicogenomics Resource Consortium as well as other groups have clearly demonstrated the reproducibility of transcript level data generated using these approaches -9. HowevLinking gene expression changes with more traditional toxicological measurements of adverse biological responses to toxicants , referred to as 'phenotypic anchoring', allowed investigators to gain new insight into the processes involved in the adverse effects on target tissues -15. In aWhile much progress has been made in the application of toxicogenomics to the classification of toxicants and the investigation of mechanisms of toxicity, a full realization of its potential in a systems biology context, sometimes referred to as 'systems toxicology' , has yetThe eight compounds are all acute hepatotoxins that cause hepatocellular necrosis following a single administration at a suitable dose. Most of the compounds target hepatocytes; however, monocrotaline targets endothelial cells leading to hemorrhage and activation of the coagulation system. The toxicity of bromobenzene, 1,2- and 1,4-dichlorobenzene, N-nitrosomorpholine, monocrotaline, and thioacetamide require metabolic activation by various cytochrome P450s to reactive intermediates -29. SincThe work was performed in its entirety using standardized procedures for the in-life work and for the generation of gene expression microarray data . Doses that ranged from 'sub-toxic' to 'toxic' exposures were selected. Additionally, gene expression profiling was performed on two commercially available platforms (Agilent and Affymetrix) for one of the tissues (liver), thereby providing an opportunity to corroborate findings across platforms. In-life observations were recorded and clinical chemistry, hematology, and liver histopathology were also assessed for all 426 animals. These additional data facilitate the phenotypic anchoring of the gene expression data and confirm that the goal of the study to evoke different types of liver injury was achieved. Gene expression levels in whole blood were also assayed to evaluate whether biomarkers of different types of liver damage could be identified in this readily accessible bio-fluid.The analyses presented here further demonstrate the utility of using microarrays as a tool for gene expression profiling to address biological questions. Specifically, this experiment was designed to provide a means to generate mechanistic and predictive measures of toxicity by integrating multiple data streams that were recorded within this standardized study. These model chemical exposure studies also provide a comprehensive data set with a well defined phenotypic anchor (liver injury) that is needed to assess the utility of gene expression profiling from whole blood samples. These data can now serve as a rich resource to the scientific community to test and validate these as well as other hypotheses, and have been made freely available in a public repository -34. WhilThe motivation for this study was the need for a rich, public dataset consisting of multiple types of data  that could be interrogated to test hypotheses regarding the application of gene expression profiling data generated from microarrays. We therefore designed a study with multiple compounds (n = 8) that would induce a common phenotype (liver injury), while also differing in the extent and/or severity of the injury and/or the co-occurrence of other noted histopathological observations, such as the type of inflammatory cell infiltrates, the presence of bile duct hyperplasia, and so on Table . The selThe expected result of exposure to both the moderately toxic and toxic doses of these compounds  was necrosis within the liver lobule. For most compounds it was expected that the necrosis would be localized to the centrilobular region, whereas with diquat the damage was expected to extend into the midzonal region, and with galactosamine it was expected to be randomly distributed throughout the liver. The histopathology data confirmed these expectations; however, some individual animal variability was observed . A support vector machine (SVM) approach was used to identify compound-specific classifiers within each dose/time group for each tissue type (blood and liver). The resulting classifiers  were then used to group samples based on principal component analyses and hierarchical clustering to determine whether or not the classifiers functioned well at separating all of the animals within a dose/time group into different compound groups.An SVM identified 160 transcripts for the classification of the 8 compounds within the liver profile data of animals treated with the medium ('toxic') dose and sacrificed at the 6 hour time point. When the expression level data for these transcripts were used to cluster the medium dose samples at the 6 hour time point, the classifiers separated the animals into the eight different compound groups, with the exception of three individual animals  and one 1,2-dichlorobenzene animal ; Figure When one considers the variability of the presence and severity of necrosis and the infiltration of inflammatory cells, it is not surprising that gene expression data from a target organ would not accurately group all of the samples within a given compound. For this reason, the gene expression data generated from whole blood were examined to determine if these data could more accurately group samples based on compound. Blood is a highly accessible bio-fluid, yet is not the primary target tissue for compound toxicity. When the liver data were examined, the SVM-identified classifiers for each of the nine dose/time groups were not able to accurately group all of the individuals into the eight compound groups. In contrast, the classifiers that were generated using the blood data were able to accurately group all of the individuals according to the eight compounds in only one of the nine dose/time groups, but performed better than the liver data at grouping individuals into compound groups in six of the nine dose/time groups. Table per se. Taken together, these results indicate that the details of the injury resulting from compound application are a more appropriate grouping than are the doses and time points of individual compounds when using gene expression data from the primary site of injury - in this case the liver - or when using gene expression data from a surrogate source such as whole blood.Transcript data from blood, however, were also unable to group individual animals in some instances in which the compound manifested different phenotypic responses across the animals of that particular dose/time group. At the high dose/48 hour time point, three of the compounds  demonstrated variability in regards to the extent of hepatocellular necrosis, the amount of inflammatory cell infiltrates and/or indications of hepatocellular repair (mitosis). Consistent with this observation, these are also the three compounds that did not separate perfectly in the dendrogram resulting from the hierarchical clustering of the blood gene expression data for the 60 SVM-identified classifiers for this dose/time group Figure , suggestAnother illuminating trend in both liver and blood gene expression data was the grouping of galactosamine and thioacetamide samples. For example, using the 160 SVM-identified classifiers generated from the blood data for the low dose/6 hour samples, most of the samples within a given compound group clustered together Figure . It was p-values for all GO categories that had at least five members measured by the microarray are detailed in Additional data file 3. The GO categories that were found to be enriched in multiple tissue/time point/dose groups revealed some interesting aspects of the biology underlying the molecular response to the exposure to these acute hepatotoxicants. Not surprisingly, categories like 'response to chemical stimulus' and 'response to wounding' were found to be enriched in many of these groups. 'Taxis' and 'chemotaxis' were the most prevalently enriched GO categories in both tissue types, which likely reflects the differential response with regards to the inflammatory cell infiltrates across the compounds. 'Wound healing' was significantly enriched in many of the time point/dose groups in the liver, except at the time points associated with the high dose exposure, which is consistent with the histopathological observations with the sub-toxic and moderately toxic doses. Interestingly, this category was enriched at the 24 and 48 hour time points for the high dose exposures in blood. Also surprising was the observation that classifiers annotated to 'wound healing' were different for each of these time points in blood, potentially suggesting that the differences in the identity of the genes may represent some of the temporal responses to the healing and repair processes that occurred between the 24 and 48 hour time points.In order to assess the biological themes within each set of classifiers, a Gene Ontology (GO) enrichment analysis was performed on the classifiers for each tissue/time point/dose group using GoMiner ,38. The Cumulatively, these results support the previous findings that toxicants can be classified and differentiated based on gene expression profiling data from the target organ ,3,35,36,p-value ≤ 1 × 10-7) across the factor levels. Interestingly, over 60% of these transcripts (n = 19) overlap with the set of 3,659 transcripts that were identified using the same ANOVA approach with the liver gene expression data [Having demonstrated the utility of performing gene expression profiling on whole blood samples as an effective surrogate for the target organ to classify compound exposures, we sought to further explore the utility of this readily available sample source by establishing whether blood transcript data could provide insights into the presence and severity of injury within a target organ upon toxicant exposure. In previously published work using these same in-life studies, we detailed an approach to compile histopathological diagnoses into common biological themes and to appropriately weigh and score these observations . Using tion data . The genWe have generated a data set that will be a unique and valuable resource for the environmental health and toxicological research community. The results of the study presented here demonstrate that, using this data set, differences in the phenotypic manifestation of diverse hepatotoxicants can be resolved by microarray analysis. It was possible to classify histopathological differences, most likely reflecting differences in mechanisms of cell-specific toxicity, using either liver tissue or blood. Such data may allow for mechanistic inference and lead to a more precise definition of the potential hepatotoxicity of new compounds. In addition, the availability of public access to the data derived from these studies provides a resource to other investigators pursuing these and related issues.ad libitum.Male Fischer 344/N rats approximately 10-12 weeks old were obtained from Taconic laboratory animals . The animals were acclimated for 14 days and observed for absence of disease. All studies were conducted at Integrated Laboratory Systems, Inc. . The rats were kept two per polycarbonate cage  with Sani-Chips bedding . The animal rooms were maintained at 17-25°C and 33-96% relative humidity with a 12 hour dark-light cycle and 10 room air changes per hour. Cages were changed twice per week and NTP-2000 diet and tap water were provided ad libitum. After 6, 24 or 48 hours of treatment, animals were anesthetized by isoflurane anesthesia. Whole blood was collected via retro-orbital plexus for hematology and gene expression analysis. Serum was collected for clinical chemistry. The animals were euthanized while under anesthesia by cervical dislocation and necropsied within 4 minutes of sacrifice. The left liver lobe was promptly removed and two defined cross sections were placed in 10% neutral buffered formalin for subsequent histopathology. The remainder of the left lobe was flash frozen for subsequent RNA isolation based on established methods [Hepatotoxicants were identified based on published literature and chosen due to their ability to injure different cell types and regions of the liver. Doses were determined with pilot experiments. For each chemical, doses that elicited a sub-toxic ('low'), a moderately toxic ('medium') or a overtly toxic ('high') response 24 hours after treatment were selected. The doses, vehicle, method for delivery, and primary site of liver damage for each of the eight compounds are detailed in Table  methods . Experim methods . Additio methods .The cross sections that were fixed by immersion in 10% neutral buffered formalin (see 'In-life studies' section above) were transferred to histology grade alcohol for 18-24 hours after necropsy. Tissues were then processed, embedded in paraffin, sectioned at 5 microns, and stained with hematoxylin and eosin. A pathologist completed the initial microscopic evaluation of each liver sample for a given compound. Grading criteria for each diagnosis are detailed in Additional data file 5. An independent pathology quality assessment review was performed by a second pathologist on the initial histopathological findings. A third pathologist reviewed the quality assessment report and the slides and convened a consistent group of four to five pathologists (including the two mentioned above) to resolve any inconsistencies or discrepancies in diagnoses and grading for all studies. Additional data file 1 details the histopathological diagnoses and their associated severity scores for each animal in the study. These data are also available in CEBS under the CEBS ID# 001-00001-0020-000-4 .® Tubes, BD, Franklin Lakes, NJ, USA) and serum was separated. Clinical chemistry analysis was performed on all rats using a COBAS MIRA  using commercially available reagents from Equal Diagnostics  for the following assays: total protein, albumin, cholesterol, creatinine, direct bilirubin, total bilirubin, total bile acid, blood urea nitrogen and triglycerides. In addition, an enzymatic activity for the following proteins was measured: alanine aminotransferase, alkaline phosphatase, aspartate aminotransferase, lactate dehydrogenase, and sorbitol dehydrogenase. Additional data file 1 details the clinical chemistry value for each of these parameters for every animal. These data are also available in CEBS under the CEBS ID# 001-00001-0020-000-4 [At sacrifice, blood was collected in clotting tubes (BD Microtainer20-000-4 .® Tubes). The samples were assayed using the Technicon H*1 hematology analyzer . The instrument performs white blood cell (WBC) counts by two independent methods. The values must have agreed within 10%. Manual counts and smear estimates were used to confirm values, when necessary. To monitor red blood cell (RBC) parameters, spun microhematocrits were performed for each specimen for comparison with the automated hematocrit . Control products were assayed after every tenth specimen. For WBC differential data, smears were Wright-Giemsa stained using an Ames Hema-Tek II automated slide stainer  and manual WBC differential counts were performed on the first sample in each group. For reticulocyte counts, equal amounts of whole blood and New Methylene Blue stain were allowed to incubate at room temperature for at least 15 minutes. Smears were prepared and read to determine the percent reticulocytes per 1,000 RBCs. The percent value is multiplied by the RBC count to determine the absolute number of reticulocytes per microliter. Additional data file 1 details the values for each of these parameters for every animal. These data are also available in CEBS under the CEBS ID# 001-00001-0020-000-4 [Blood was collected in EDTA tubes  according to the manufacturer's protocol and also from whole blood using the PaxGene Blood RNA Kit (Qiagen). The quantity and purity of the extracted RNA was evaluated using a NanoDrop ND-1000 spectrophotometer  and its integrity measured using an Agilent Bioanalyzer. For microarray hybridizations performed on the Agilent platform, 1 μg of total RNA from the left liver lobe or whole blood from either an individual rat or a pooled sample representing time-matched, vehicle control animals was amplified and labeled with a fluorescent dye (either Cy3 or Cy5) using the Low RNA Input Linear Amplification Labeling kit  following the manufacturer's protocol. The amount and quality of the fluorescently labeled cRNA was assessed using a NanoDrop ND-1000 spectrophotometer and an Agilent Bioanalyzer. Equal amounts of Cy3- or Cy5-labeled cRNA were hybridized to the Agilent Rat Oligo Microarray (Agilent Technologies) for 17 hours, prior to washing and scanning. Data were extracted from scanned images using Agilent's Feature Extraction Software (Agilent Technologies). Fluorophore reversal hybridizations were performed for 318 treated rats against time-matched control pools for both tissue types (blood and liver) for a total of 1,272 hybridizations. For microarray hybridizations performed on the Affymetrix platform, 1 μg of total RNA from the left liver lobe of each individual rat was labeled using the GeneChip One-Cycle Target Labeling kit  following the manufacturer's protocol. The amount and quality of the cRNA was assessed using a NanoDrop ND-1000 spectrophotometer and an Agilent Bioanalyzer. The cRNA was then fragmented and hybridized to the Rat Genome 230 2.0 Array  for 17 hours, prior to washing and scanning. Data were extracted from scanned images using GeneChip Operating Software . Hybridizations were performed on all treated and control animals for each compound in addition to pooled control samples for two of the compounds for a total of 432 hybridizations. These data are also available in CEBS under the CEBS ID# 001-00001-0020-000-4 .® Gene Expression Analysis System version 6.0.0.0.311. Fluorophore reversal hybridization data generated using the Agilent platform were combined using an error-weighted average for each individual animal. Data generated using the Affymetrix platform were loaded using the Rosetta error model for gene expression analysis [Gene expression data were loaded into the Rosetta Resolveranalysis . Rosettaanalysis ,38.ANOVA, analysis of variance; CEBS, Chemical Effects in Biological Systems; GO, Gene Ontology; PCA, principal component analysis; RBC, red blood cell; SVM, support vector machine; WBC, white blood cell.RSP, RWT, PH, EKL, KG, CJT, RDI, GAB, JTA, MLC and ANH conceived, designed, and supervised various aspects of the study. PEB, SOS and MLC participated in the in-life work/tissue collection and RNA purification. EKL and PH made substantial contributions to microarray processing. EKL, PH, PEB, SMW, RSP, PRB, KG, JMF, JTA, REW, BAM, DEM, and GAB contributed to the analysis and interpretation of the data. EKL drafted the manuscript. PH, RSP, PRB, KG, SOS, PH, JMF, BAM, DEM critically reviewed the manuscript. All authors read and approved the final manuscript.p-values that resulted from performing a GO enrichment analysis on the list of classifiers for each dose/time/tissue group. GO categories must have had at least five associated genes measured on the microarray in order to be included. Additional data file The following additional data are available with the online version of this paper. Additional data file Histopathological scores as well as the clinical chemistry and hematology data that were generated for each animal in this project.Click here for fileAgilent probe identifiers that resulted from the SVM analyses detailed in the Results section.Click here for filep-values that resulted from performing a GO enrichment analysis on the list of classifiers for each dose/time/tissue group. GO categories must have had at least five associated genes measured on the microarray in order to be included.Enrichment Click here for fileObservations and measurements made during the in-life portion of the project.Click here for fileScoring method used for the histopathological observations made in this project.Click here for file


In a previously published pilot study we explored the performance of microarrays in predicting clinical behaviour of ovarian tumours. For this purpose we performed microarray analysis on 20 patients and estimated that we could predict advanced stage disease with 100% accuracy and the response to platin-based chemotherapy with 76.92% accuracy using leave-one-out cross validation techniques in combination with Least Squares Support Vector Machines (LS-SVMs).In the current study we evaluate whether tumour characteristics in an independent set of 49 patients can be predicted using the pilot data set with principal component analysis or LS-SVMs.The results of the principal component analysis suggest that the gene expression data from stage I, platin-sensitive advanced stage and platin-resistant advanced stage tumours in the independent data set did not correspond to their respective classes in the pilot study. Additionally, LS-SVM models built using the data from the pilot study – although they only misclassified one of four stage I tumours and correctly classified all 45 advanced stage tumours – were not able to predict resistance to platin-based chemotherapy. Furthermore, models based on the pilot data and on previously published gene sets related to ovarian cancer outcomes, did not perform significantly better than our models.We discuss possible reasons for failure of the model for predicting response to platin-based chemotherapy and conclude that existing results based on gene expression patterns of ovarian tumours need to be thoroughly scrutinized before these results can be accepted to reflect the true performance of microarray technology. Ovarian cancer ranks fifth when considering cancer mortality in women . UnfortuPreviously, we performed a pilot study consisting of microarray analysis on three groups of patients: seven stage I without recurrence, seven platin-sensitive advanced stage and six platin-resistant advanced stage ovarian tumours . We inveIn the current study, we describe results of an independent evaluation of models for predicting disease stage and response to platin-based chemotherapy built on the data of the pilot. Our goal was to evaluate whether an independent study could confirm the applicability of microarrays for the clinical management of ovarian cancer. This independent evaluation was carried out on a set of 49 new tumour samples which were subjected to the same experimental protocol. This data set was used as a test set to estimate the performance when predicting the difference between stage I and advanced stage disease, and between platin-sensitive and platin-resistant disease using models trained on the pilot data set. After presenting the results, we discuss the generalization performance on this independent data set and compare with models based on previously published gene sets.s and Ar respectively. The patient and tumour characteristics are shown in table Tissue collection and analysis were approved by the local ethical committee. After obtaining informed consent, tumour biopsies were sampled and immediately frozen in liquid nitrogen during primary surgery and were taken from three groups of patients: 4 from patients with stage I disease, 30 from patients with platin-sensitive advanced stage disease and 15 from patients with platin-resistant advanced stage disease [Microarray procedures were similar to our pilot study . BrieflyThe gene expression data were analysed using MATLAB 7 (R2006b). Pre-processing was done similarly as in our pilot study. Briefly, each microarray in the independent data set was analysed separately in the following order: the intensities were background-corrected, log-transformed and finally normalised using the intensity dependent Lowess fit procedure. The mean of the replicate and normalised log ratios was used as a measure for expression. After pre-processing, first PCA and secondly LS-SVM were used to analyse the data. PCA was used for visualisation of the data while LS-SVMs were used for building classification models. A p-value is considered statistically significant if smaller than 0.05. All statistical tests were two-sided unless mentioned otherwise. Exact bionomial confidence intervals were calculated using SAS 9.1.3 statistical software.The procedure followed during PCA analysis can be found in Figure Next, we used the pilot data set to build an LS-SVM to predict disease stage and an LS-SVM to predict the response to platin-based chemotherapy . In tTo assess the performance of models based on our data we compared them with the performance of models based on published gene sets that predict a broad range of outcomes in ovarian cancer. It is difficult to directly apply the published models on our data since multiple different microarray platforms  or two-channel custom arrays (cDNA)) have been used to derive these gene sets. Therefore we adopted the strategy visualized in Figure In this study we describe the results of the evaluation of models developed based on the data from our previously published pilot study  using PCSecondly, we used LS-SVMs to assess if a supervised classification model can discriminate between the stage I and advanced stage disease, and between platin-sensitive and platin resistant disease. This resulted in a classification accuracy of 97.96% (CI 19%–99%) for the distinction between stage I and advanced stage disease which corresponds to one stage I tumour out of four that was classified as an advanced stage tumour. Next, a classification accuracy of 51.11% was obtained for the distinction between platin-sensitive and platin-resistant disease. This corresponds to five platin-resistant and eighteen platin-sensitive tumours that were misclassified, corresponding to a sensitivity of 67% (CI 38%–0.88%) and specificity of 40% (CI 23%–0.59%) when considering a platin resistant patient as a positiveTable Recently, several studies have investigated the use of microarrays to predict several clinically relevant outcomes of ovarian cancer ,13,15,21Although more data should be gathered on stage I patients, the results presented in this paper indicate that predicting the response to platin-based chemotherapy is not straightforward and more subtle than predicting advanced stage disease. Furthermore, since most published studies lack a proper independent evaluation, their results should be cautiously interpreted. We advocated the use of microarrays based on the results from our pilot study, but warned for overestimating the generalization performance, as these results were based on a cross validation technique instead of using an independent data set. Additionally, since the pilot study performance for predicting the response to platin-based chemotherapy was not statistically significant, we searched for confirmation on an independent test set. Therefore, we carried out a new study to estimate the performance of models based on independently gathered microarray data in an unbiased way. The present results, both the PCA analysis and the performance of the LS-SVM models, show that the independent evaluation is disappointing. Only the LS-SVM stage model performed well and was able to distinguish early stage and advanced stage disease on the independent data set. The PCA analysis however demonstrated that, for the three classes, the independent data did not cluster to their corresponding class in the pilot study. Additionally, the LS-SVM platin model was not able to perform better than a random predictor. Therefore, we argue that a gene expression study should be validated on independently gathered data before the results can be considered for clinical use. Independently gathered data can be influenced by subtle changes in sample preparation, sample analysis and sample hybridizations, which can deteriorate model performance. Even the techniques used by the same lab might undergo subtle changes throughout time, causing a drop in model performance when the model is applied on new patient samples. It is unclear whether published models are robust against these influences.Additionally, ovarian cancer represents an immense variation in histological structure and biological behaviour which complicates microarray based modelling. A large number of samples is required to correctly represent the complete microscopic spectrum. It is not unlikely that an independent data set contains a different mix of tumour samples with slightly different histological characteristics compared to the pilot study, complicating independent evaluation. Moreover, the quality of the samples has a major effect on the ability to detect true differential expression and subsequent model building. However in most cases, including ours, only a limited number of samples with sufficient follow-up is available which limits our ability to obtain a similar distribution of histopathology in the pilot and independent data set, and also forces us to use archival samples instead of new ones.The comparison of the LS-SVM stage and LS-SVM platin model with published genes sets confirmed that predicting disease stage is easier than predicting response to platin-based chemotherapy. For predicting disease stage many previously developed gene sets are able to distinguish both classes indicating that many genes change when a tumour progresses from early to advance stage disease. Predicting the response to platin based chemotherapy is more challenging. None of the previously developed gene set models related to the response to platin based chemotherapy are able to predict this outcome significantly better than chance. This indicates that these gene sets do not generalize to our independently gathered data set. Only the 27-gene model by Lancaster and colleagues , which dOur results show that an independent evaluation of models based on gene expression data is necessary to validate models before considering subsequent steps to make microarray analysis clinically available. Previously published studies should be critically reviewed, in light of the current results, to assess if the reported model performance is not overestimated by inappropriate use of a test set and, if this is not the case, to consider if an independent study would confirm the reported model performance. Finally, prospective validation in multi-centre trials is necessary before microarray technology can move to clinical practice.The author(s) declare that they have no competing interests.FDS, OG, NP, FA, BDM, DT and IV conceived the study and provided clinical and mathematical background. TVG looked up patient records in the database and tissue samples in the tumour bank, performed sample annotation and gathered follow-up of patients. KE performed pre-processing of the data sets. OG, FDS and NP performed the PCA and LS-SVM analysis, and the comparison with published gene set analysis. All authors contributed to the manuscript and approved it.The pre-publication history for this paper can be accessed here: