A critical step in processing oligonucleotide microarray data is combining the information in multiple probes to produce a single number that best captures the expression level of a RNA transcript. Several systematic studies comparing multiple methods for array processing have used tightly controlled calibration data sets as the basis for comparison. Here we compare performances for seven processing methods using two data sets originally collected for disease profiling studies. An emphasis is placed on understanding sensitivity for detecting differentially expressed genes in terms of two key statistical determinants: test statistic variability for non-differentially expressed genes, and test statistic size for truly differentially expressed genes.In the two data sets considered here, up to seven-fold variation across the processing methods was found in the number of genes detected at a given false discovery rate (FDR). The best performing methods called up to 90% of the same genes differentially expressed, had less variable test statistics under randomization, and had a greater number of large test statistics in the experimental data. Poor performance of one method was directly tied to a tendency to produce highly variable test statistic values under randomization. Based on an overall measure of performance, two of the seven methods (Dchip and a trimmed mean approach) are superior in the two data sets considered here. Two other methods (MAS5 and GCRMA-EB) are inferior, while results for the other three methods are mixed.Choice of processing method has a major impact on differential expression analysis of microarray data. Previously reported performance analyses using tightly controlled calibration data sets are not highly consistent with results reported here using data from human tissue samples. Performance of array processing methods in disease profiling and other realistic biological studies should be given greater consideration when comparing Affymetrix processing methods. Affymetrix microarrays are high throughput assays for measuring the expression levels of thousands of gene transcripts simultaneously. This type of microarray measures the expression of each transcript multiple times through a set of "probe pairs". Since the advent of the Affymetrix microarray, numerous methods have been proposed for producing numerical expression summaries for each transcript based on the probe pair data. Several systematic studies have appeared comparing a number of methods on a common basis e.g. -5). Thes. Thes5])]. While not every popular method is included in our study, several highly distinctive and original approaches are studied. For example, Dchip was one of the first approaches to attempt to learn probe weights directly from the probe data, and RMA pioneered the approach of disregarding the control mismatch probes. PDNN uses physical modeling to determine probe weights, while the two GCRMA methods use GC content of the probe sequences to reduce variance in the mismatch (control) probe levels. The MAS5 method is the current default method provided by Affymetrix.The following seven methods are considered here: Dchip , GCRMA-EIn addition to the six methods cited previously, we also include a method designated TM (trimmed mean). This is a simple method that has been used in a number of published investigations e.g. ,7), but , but 7])An important feature of this study is the use of False Discovery Rate (FDR) to quantify the sensitivity of a processing method in terms of its ability to distinguish differentially expressed genes from genes having invariant expression. This is a highly relevant property, as differential expression analysis is the most common application of microarray data. A key advantage of using FDR to compare processing methods is that FDR values can be calculated accurately using real disease profiling data where the identities of differentially expressed genes are uncertain. In contrast, most previous systematic comparisons of array processing methods have focused on calibration data sets in which concentrations of certain genes were experimentally manipulated.When it is highly likely that at least one gene is differentially expressed, false discovery rate may be defined as the expected ratio of the number of false positive calls to the total number of positive calls in a differential expression analysis between two groups of samples . If the A small FDR is due either to a small numerator, a large denominator, or both. The denominator of the FDR depends on the actual data distribution, so variation in this value may be due to factors such as accuracy in modeling the physical and chemical nature of probe binding. Variation in the FDR numerator, however, depends only on the distribution of values produced for randomized data, a purely statistical quantity reflecting the tendency of the method to incorrectly produce test statistic outliers. Our results suggest that both factors are important in determining sensitivity. The best methods produce many large test statistic values in the actual data, and also produce consistently small test statistic values for randomized data. Poor performance of one method can be directly explained by the tendency of the method to produce outlier expression values, leading to greater numbers of incorrectly large test statistics.strongly outperform the first. Based on this comparison, two of the methods considered are clearly favored, two are inferior, and results for the other three methods are mixed.For overall comparison, we evaluated every pair of methods on the basis of whether the first method is expected to call at least one truly differentially expressed gene that is not also called by the second method. If this is not expected to occur, the second method is said to 0 level, where FDR0 is a rescaled FDR (see methods). Figure Our primary basis for comparison is sensitivity – the number of genes detected at a given FDR0 = 0.1 as a reference point, there is roughly 7-fold variation across the seven methods in the number of detected genes in colon data using t-test statistics, while for rank-sum statistics the range is roughly 2-fold. In the ovary data, the range is around 1.25-fold for both statistics. Also notable is that variation in sensitivity due to the choice of test statistic (t-test or rank-sum) is smaller than variation in sensitivity due to the seven processing methods.The more challenging colon set distinguishes the seven processing methods to a greater extent than the ovary set. Using FDRNo single method stands out as having the best or worst performance in every case. However some methods generally perform better than others. The Dchip and TM methods perform consistently well, while the GCRMA-EB and MAS5 methods consistently perform poorly. PDNN performs well on the ovary data, but poorly on the colon data, and results for the other methods are mixed.0 threshold vary across the methods. Figure 0 thresholds in k or more of the seven methods to the number of probe sets falling below the threshold for at least one method is plotted against the FDR0 threshold, for k = 3, 4, 5, 6, 7. In the ovary data there is a very high level of agreement in this measure. For the rank-sum analysis, almost 90% of called genes are called by at least four methods, and more than 70% of called genes are called by all seven methods. For the t-test analysis, the agreement is slightly higher yet. For the colon data, the methods are much more inconsistent. For the rank-sum analysis, three of the methods agree on up to 90% of genes, but all seven methods only agree on around 30% of genes. The t-test analysis is even worse, with only around 10% of genes common to all seven methods.Identities of probe sets falling below a given FDRTurning to pairwise agreement, Table Complementing comparison of the statistical tests, we also compared the expression levels produced by the seven processing methods. For each pair of methods, and for each pair of samples within one of the two data sets, we calculated Pearson correlation coefficients of expression levels over all genes. These values were summarized by taking the median over all pairs of samples within a data set, shown in Table Variation in FDR across the seven methods is due to two factors – variation in the number of transcripts with large test statistics, and variation in the expected number of transcripts with large test statistics when there is no real differential expression. Here we investigate the second factor, which is driven by the tendency of each method to produce outlier expression values.The numerator of the FDR aims to correct for variation in the number of false positives, so that a method claiming large numbers of differentially expressed genes is not considered superior unless it also produces relatively small numbers of false positives. This can be viewed as a calibration, in which for each method, the test statistic must reach a certain threshold in order that the proportion of false positives is no greater than a specified value.0 less than f was calculated, and plotted against f. For example, to achieve any FDR0 value between 0.05 and 0.1 in the colon rank-sum data, GCRMA-MLE requires the lowest test statistics, RMA requires a rank-sum statistic 0.15 units larger than that of GCRMA-MLE, and MAS5 requires a rank-sum statistic 0.3 units larger than that of GCRMA-MLE.Calibration results are summarized in Figure Figure  denote the log2 expression level of transcript i in sample j for method k, where k = 1, ..., 7 denotes the seven processing methods, and letSince calibration depends only on randomized data, it should be possible to trace variation in thresholds across the processing methods to statistical properties of the expression levels. For example, if one method produces expression levels with heavier tails, it is easier to get a large t-test statistic value by chance, particularly for the colon data with small sample sizes. This would necessitate a higher threshold. To quantify this, let  is the pth quantile of , and med is the median value. This is an affine-invariant measure of the size of the right tail of the expression values. Values of Bk for the seven methods and two data sets are shown in Table B value of 3.74 when the sample size is as in the ovary data, and 3.56 when the sample size is as in the colon data. The GCRMA-EB method is seen to have a much greater propensity for producing extreme expression values, explaining its low sensitivity, poor agreement with other methods, and conservative calibration.where t, the number of probe sets for which the observed test statistic value exceeds t was calculated and plotted against t. For example, in the colon rank-sum data, PDNN had the smallest test statistics, with MAS5 having around 500 more probe sets meeting a log test statistic threshold of 5 compared to PDNN. The Dchip and TM methods have over a thousand more probe sets meeting this threshold.In addition to calibration differences, FDR variation is also influenced by the observed test statistic values. This is summarized in Figure Variation in test statistic values across the methods is greater in colon than in ovary data, and generally tracks with sensitivity. However note that in the colon rank-sum data, Dchip has substantially larger test statistics than GCRMA-MLE, even while GCRMA-MLE has better sensitivity Figure , due to An interesting possibility that can not always be excluded is that the intergroup differences are so vast that nearly every gene is affected to a small degree. If this were the case, the FDR values for the t-test and Wilcoxon statistics would converge to zero for every gene as the number of samples grows, making FDR values difficult to interpret. To further investigate this issue, we repeated the analysis using t-statistics truncated to zero when the fold change is less than 1.5 as test statistics for FDR analysis. The corresponding FDR values remain bounded away from zero for genes having true fold change smaller than 1.5, while genes with true fold change exceeding 1.5 have FDR values converging to 0. Thus the statistic identifies a meaningful subset of genes even when all genes are differentially expressed to some degree.Results for this analysis are shown in Figure N1 and N2 genes respectively at a given FDR level. Then nk = (1 - p0FDR0)·Nk estimates the expected number of truly differentially expressed genes called by method k. Now suppose that I is the number of genes called by both methods. Then nk - I is an estimated lower bound for the expected number of genes correctly called by method k but not by the other method. We will say that method 1 strongly outperforms method 2 if n1 - I ≥ 0 but n2 - I < 0. This means that in terms of differential expression, method 2 is not expected to contribute any true positives that were not called by method 1.Thus far we have focused on sensitivity as a criterion for comparing methods. However even if one method is less sensitive than another, if the overlap in the called gene sets is not too great then the less sensitive method may still contribute to our understanding of which genes are differentially expressed. Suppose two methods denoted 1 and 2 give p0 = 1 and FDR0 = 0.05, showing the number of times that each method was strongly outperformed by other methods in our study. This analysis clearly favors the TM and Dchip methods, while the MAS5 and GCRMA-EB methods are nearly always found to be strongly outperformed by the other 5 methods. These results are not sensitive to choices of p0 between 0.5 and 1 .Table The choice of processing method for Affymetrix array data evidently has a major impact on the ability to confidently report the results of differential expression analysis. The effect is greater, for example, than the choice of using a robust or a non-robust analysis, even in the colon data where robust analysis results in substantial improvements. Differences among processing methods are much greater in the more challenging colon data set compared to the ovary data, yet it should be noted that the sample sizes in the colon data are not atypical in real investigations.While results from two data sets can never conclusively determine the optimal method, it is notable that across both data sets, using both t-statistic and rank-sum analyses, there is a high degree of similarity in the rank ordering of the methods from the best to the worst performer. The trimmed mean (TM) and Dchip methods consistently perform as well or better than any of the other methods. A possible explanation for this is that the weights used by the Dchip may tend to downweight the least and greatest PM-MM differences, just as the TM method excludes these differences.When comparing array processing methods using experimental data in which the identities of differentially expressed genes are unknown, great care must be taken to ensure that apparent differences in sensitivity are not due to other factors. One critical point is that the null distribution providing the expected number of false positives at a given test statistic threshold (the numerator of the FDR) must fairly reflect the statistical behavior of null genes. Permutation approaches have been extensively used to produce empirical p-values (e.g. ) and werWhile we have focused on FDR as the basis of comparison, the pursuit of small FDR values is not the only desirable operating characteristic of an array processing method, and other reports have also emphasized the accuracy of estimating the precise size of concentration differences. However to the extent that most actual studies seek to find differential expression between groups, the use of small FDR values seems more instrumental as the basis for judging methods.Although our primary aim was to investigate variation in sensitivity due to the seven processing methods, all analysis was carried out independently for two test statistics. The t-statistic is widely used in practice, but is well-known to be sensitive to outliers, particularly when the sample size is small. We found that certain processing methods, particularly EB-GCRMA, had a tendency to produce outlier expression values in the colon data set. Thus the combination of using the EB-GCRMA method with t-statistics in the colon data led to particularly poor performance.In practice, the approach used for array normalization and for forming log-transformed expression values may be equally or more influential than the method used for producing probe set summaries . In thisNevertheless it remains of interest whether these routine processing steps are the determining factor of performance. In a future study it will be important to investigate this question further by modifying the implementations of the processing methods so that uniform log transforms and array normalizations are applied.A key point that we advocate in this work is that false discovery rates in actual disease profiling data constitute a valuable complement to benchmarking results obtained from spike-in, dilution series, and mixture experiments e.g. ,5). The . The 5])In spite of the statistical challenges in using disease profiling data for benchmarking, we argue that these data sets also offer some unique advantages. Calibration data sets are relatively few in number and are not available for all platforms. Newer platforms in particular are under-represented. Therefore overtraining to the available calibration data through manipulation of the many tuning parameters in the more complicated processing methods is an unavoidable concern. In addition, the calibration data sets likely do not represent the same degree of challenge as disease profiling data in that reproducibility of fold changes for affected and unaffected genes is quite high compared to data from, say, human tissues where a large number of uncontrolled sources of variability are present.Performances of multiple array processing methods on disease profiling data sets vary widely across the seven methods studied here, but results are generally consistent between the two data sets studied. Results of our analysis generally do not parallel results obtained using calibration data sets ,5, suggeA major determinant of sensitivity is test statistic variability for randomized data. Such variability will affect false discovery rates as well as empirical p-values, which are an often-used alternative approach for identifying differentially expressed genes (e.g. ). TherefIn this comparative analysis we did not seek to draw definitive conclusions about the "best" or "worst" methods. Such conclusions may be made after investigating a greater number of data sets, including disease profiling data, data from controlled experiments, and calibration data. Moreover, it may be that the correct choice of method may depend on the scientific question being asked. The key message of this work is that the wide range of data sets collected in actual scientific investigations may be used for comparison of processing methods, and that in at least the two data sets considered here, similar results were obtained in the rank ordering of the methods.We used two data sets – one consisting of 79 ovary tumors and the other consisting of 47 colon tumors. Both sets were generated at the University of Michigan using Affymetrix HG_U133A arrays, which consist of 22283 probe-sets, each of which is designed to assay a RNA transcript. Each probe-set consists of a set of  probe-pairs, with each probe-pair comprising a "perfect match" (PM) probe which is a 25-base oligonucleotide complementary to the transcript, and a "mismatch" (MM) probe that is identical to the PM sequence except for alteration of the central base. The MM probe is intended as a control for nonspecific hybridization, so that the difference PM-MM measures only specific binding. However not all processing methods use the MM data in this way.For differential expression analysis, the 79 ovary samples were partitioned according to histological class into 38 endometrioid and 41 serous samples. The 47 colon samples were partitioned into 40 microsatellite stable (MSS) samples and 7 microsatellite instable samples (MSI). In both data sets, the partition is based on an independently measured biological characteristic, so there almost certainly are differentially expressed genes to be found. However in neither case are the two classes highly distinct, and numerous other sources of biological variation are undoubtedly present in the data.Array normalization refers to an adjustment of data distributions within each array in order to make the arrays more comparable. Each array processing method has been coupled with a normalization procedure by its developers (see references). We followed these method-specific normalization practices in our analysis. All methods other than MAS5 use some form of quantile normalization.All analysis was based on log-transformed data. Log-transformed values, including truncations where needed, were calculated in the manner recommended by the developers of each method (see references).T is always analyzed as |T|, and the rank-sum statistic R is standardized as , where m0, m1 are the numbers of samples in the two classes, and m = m0 + m1 is the total number of samples.We compared the seven methods based on their sensitivity in detecting differential expression at a fixed false discovery rate (FDR). For each method, two different two-sample test statistics were calculated for each gene – the standard two-sample t-statistic, and the Wilcoxon rank-sum statistic . The t-test statistic t, the FDR was estimated as follows. Randomized data sets were constructed by randomly reassigning the class identifiers to the samples. The average number of transcripts with test statistic value exceeding t was calculated over 1000 randomized data sets. This number was divided by the number of actual transcripts with test statistic value exceeding t to produce a value that we denote FDR0. In practice the value of FDR0 should be scaled by the proportion p0 of non-differentially expressed genes, giving FDR = p0FDR0. Although various estimates of p0 exist, we elected to ignore this factor since it is constant across the methods for a given data set, and any estimate of p0 would add an additional source of uncertainty to our results. Thus it should be noted that the reported FDR0 values, while comparable across methods, are somewhat larger than the usual estimates. Since p0 would generally be greater than 1/2, the bias is likely less than a factor of 2.Our FDR approach closely follows the "global estimate" of Efron et al. ( equationKS, JMGT, RK, and DG participated in all phases of design and analysis. WC and JM performed the data analysis. KRC, TJG, SBG, ERF, and SH assisted in study design and supervised data collection. All authors read and approved the final manuscript.


TEL-AML1], t[MLL-AF4], t[BCR-ABL], and t[E2A-PBX]. ALL has been studied at the molecular level through expression profiling resulting in un-validated expression correlates of these prognostic indices. To date, the great wealth of expression data, which has been generated in disparate institutions, representing an extremely large cohort of samples has not been combined to validate any of these analyses. The majority of this data has been generated on the Affymetrix platform, potentially making data integration and validation on independent sample sets a possibility. Unfortunately, because the array platform has been evolving over the past several years the arrays themselves have different probe sets, making direct comparisons difficult.Acute lymphoblastic leukemia (ALL) is the most common pediatric malignancy and has been the poster-child for improved therapeutics in cancer, with life time disease-free survival (LTDFS) rates improving from <10% in 1970 to >80% today. There are numerous known genetic prognostic variables in ALL, which include T cell ALL, the hyperdiploid karyotype and the translocations: t[To test the comparability between different array platforms, we have accumulated all Affymetrix ALL array data that is available in the public domain, as well as two sets of cDNA array data. In addition, we have supplemented this data pool by profiling additional diagnostic pediatric ALL samples in our lab. Lists of genes that are differentially expressed in the six major subclasses of ALL have previously been reported in the literature as possible predictors of the subclass.We validated the predictability of these gene lists on all of the independent datasets accumulated from various labs and generated on various array platforms, by blindly distinguishing the prognostic genetic variables of ALL. Cross-generation array validation was used successfully with high sensitivity and high specificity of gene predictors for prognostic variables. We have also been able to validate the gene predictors with high accuracy using an independent dataset generated on cDNA arrays.Interarray comparisons such as this one will further enhance the ability to integrate data from several generations of microarray experiments and will help to break down barriers to the assimilation of existing datasets into a comprehensive data pool. The advent of DNA microarrays has provided the science community with a tool to concurrently examine the expression of thousands of genes within a given cell or tissue type, thus providing a platform for future diagnoses and prognostic analyses of disease with gene-level specificity ,2. Microet al. generated a list of genes that have distinct expression levels for various karyotypic and phenotypic aberrations common to pediatric ALL , t[MLL-AF4], t[BCR-ABL], and t[E2A-PBX] [TEL-AML1 fusion gene have a better prognosis than patients in the other subclasses [ALL is the most common pediatric malignancy comprising over 75% of the annual diagnoses of leukemias in children . In the L-AF4], t;22[BCR-Abclasses . The iniML1], t4;[MLL-AF4]et al. (2002), for each of the aforementioned prognostic genetic variables. Cross-platform array validation was used successfully to ascertain the accuracy, sensitivity and specificity of the gene predictors for the prognostic variables. In addition, we have demonstrated the ability to compare datasets from different microarray platforms. To our knowledge, this is among the first known successful applications of this technique, along with the validation of the Yeoh et al pediatric ALL gene lists on adult ALL by Kohlmann, et al (2004) [In this study, we sought to determine if datasets from different microarray platforms could be compared in a useful manner. We chose to study pediatric ALL because there is already a substantial pool of datasets freely available in the public domain. First, we collected pediatric ALL array data and cDNA array data generated in experiments from various laboratories. In addition to the collection of public data, we supplemented the hyperdiploid karyotype and the T-cell lineage ALL subclasses by expression profiling additional diagnostic pediatric ALL samples from a tumor bank in Children's National Medical Center in Washington, D.C. We used these independent datasets, including the one generated in our lab, to validate the gene predictors, as defined by Yeoh l (2004) . InterarTo supplement the ALL subclasses that are under-represented in expression profiling thus far, we collected and extracted the total RNA from sixteen diagnostic bone marrow samples housed at Children's National Medical Center; seven of the hyperdiploid karyotype and nine with T-cell lineage. The extracts were then hybridized to Affymetrix U133A arrays and expression profiled as an independent training data set.et al. in 2002 [et al. used both a training and test dataset in their analysis to first uncover the subclass-specific gene expression profiles and then to test their predictability on independent samples [. We then accumulated the ALL diagnostic bone marrow array data available in the public domain , which were provided by the comprehensive training ALL sample set analyzed and published by Yeoh 0 genes, in Table .. We then used the discriminating gene list for each subclass to extract the appropriate probes and their intensity values from the expansive expression data for each sample of the validation datasets independently. The level of similarity between the probe sets of the two different array platforms was evident through the number of genes within the 40 discriminators that could be found within the validation data  was generated on the U95Av2 array platform [MLL-AF4, TEL-AML1, and hyperdiploid datasets. Similarly, the U133A arrays that were used to generate expression data in our lab for the hyperdiploid karyotype and the T-cell lineage ALL contained probes representing the majority of the 40 discriminators, with 38 and 35 genes, respectively. The HuGene FL arrays contain significantly fewer probe sets in common with the selected predictors (from the later-generation U95Av2 microarrays). Accordingly, of the 40 original predictor probes, only 25 were present in the TEL-AML1 dataset, 26 in the E2A-PBX1 dataset and 13 in the T-cell dataset. The difficult task of matching probes from the Affymetrix gene chips with cDNA arrays was illustrative of the disparities between the probe sets within these two platforms. For example, the five cDNA predictors built using the datasets produced by Moos et al. (2002) and Catchpoole et al. (unpublished data) contained data for only ten genes or less from the predictor set gene list.The independent datasets that we accumulated, including the one generated in our lab, spanned four different microarray platforms: Affymetrix HuGene FL, U95Av2, U133A and custom cDNA microarray platforms , using the aforementioned independent test datasets from various array platforms, we employed supervised learning methods using GeneCluster2 software. Prior to analysis, we formatted the discriminating gene expression values from the test datasets onto spreadsheets according to software instruction, and subsequently applied the data to the software. Genecluster2 then generated blinded predictions on the ALL samples of the test datasets through weighted voting with a leave-one-out methodology. This is accomplished by randomly removing one sample at a time from the test dataset of ALL samples and "training" a predictor gene profile to recognize similarities or disparities between the two classes based on the expression profiles of the samples for the genes of interest [To validate the gene predictors from Yeoh interest . In thiset al. 002, usinE2A-PBX1 predictor (96%) and the MLL-AF4 predictor (98%), the sensitivities were only 50%. In both cases there were only 2 samples out of the sample pool expressing the respective translocation and in both analyses one of the two was continuously classified incorrectly. This could be due to many factors, misdiagnosis or mislabelling of the sample, poor sample quality or differences in sample handling. It is difficult to draw a conclusion due to the fact that the samples were collected and processed in a laboratory outside of our own. Another problem with these two predictors may simply be the low sample number. Two samples may not provide enough strength to the classification by Genecluster2 simply due to the inability of such a low sample number to account for the biological variability that exists between patients that is independent of their subclass of ALL. The most surprising result was the high accuracy with which the gene lists could classify T-ALL, TEL-AML1 and MLL-AF4 from cDNA data considering the disparity between the probe sets of cDNA arrays and oligo arrays. The accuracies of the classifiers were: T-ALL (Catchpoole data), 100%; T-ALL (Moos data), 98%; MLL-AF4, 98%; and TEL-AML1, 87%. Therefore, it appears that the number of genes in the predictor gene list is much less of a factor in the predictor's classification accuracy than the number of samples representing the phenotype of interest, which supports the argument that being able to do cross-platform analyses to increase sample size is crucial for sensitive and specific class prediction using expression data. This is strongly illustrated by the cDNA predictors, which have few probes in common with the arrays used to generate the predictor gene list, but still classify the ALL samples with high sensitivity, specificity and accuracy. On the other hand the E2A-PBX1 predictor and the MLL-AF4 predictor had low sensitivities correlating to a low sample number in these groups. The high number of probes in common between the arrays used to generate these independent datasets and the arrays used to generate the predictor gene list, 26/40 and 40/40, respectively, were not able to rescue the low sensitivity of the predictor.We saw a high accuracy from the predictors employing data from both U95Av2 and U133A arrays, attesting to the fact that nearly all of the 40 discriminating genes were present in the datasets, thus maximizing the possible prediction strength. In the case of the et al. (2002) withstands validation by testing the predictors using a leave-one-out strategy on all publicly available datasets as well as a dataset generated in our own lab regardless of the array platform used. This meta-data analysis of over 200 arrays from diagnostic ALL samples with hyperdiploidy, T-cell lineage and translocation status (previously confirmed through gold standard techniques), shows that expression profiling as an integrated platform is robust and that ALL data, and presumably other disease models, can be interplatform comparable. By validating the comparability between data from distinct microarray platforms we have demonstrated a tool that can enhance the statistical power provided by large sample sets. Thus, we can potentially develop and validate sensitive diagnostic tools based on large training sample sets, to allow for the rapid assignment of individualized therapy to improve disease outcome in pediatric ALL and other diseases.Currently the vast majority of expression data from numerous labs is not being used to its highest potential as independent labs continue to move to more expansive array platforms rendering older datasets less informative in the context of new data. Increasingly, progressive technologies in genome databasing and chip construction are prompting this inevitable evolution of microarrays. Until data can be analyzed and directly compared across array platforms, the size of the data pools will remain small and isolated according to platform . Here weALL diagnostic bone marrow samples were housed in a tumor bank in Children's National Medical Center in Washington, D.C. Mononuclear cells from diagnostic bone marrow aspirates were separated using density centrifugation on Cappel Lymphocyte Separation Medium  and immediately flash frozen according to manufacturer's instruction. A total of sixteen samples were obtained with IRB approval; seven with a hyperdiploid karyotype and nine samples of a T-cell lineage as confirmed by immunophenotyping. The frozen samples were placed directly in TRIzol reagent for RNAse-free thawing for total RNA extraction. We extracted a 10 μg–20 μg pellet of total RNA from each sample by centrifugation following phenol-chloroform extraction. The integrity of the resultant total RNA from each sample was quantified by gel electrophoresis before it was considered to be of good quality for cDNA synthesis. Samples were re-extracted if ribosomal bands were not visible.et al. (2002) [et al. (2002) from Affymetrix U95Av2 microarrays. The 40 genes that showed the greatest mean difference in expression between the subclass of interest and the remaining subclasses was used as our predictor gene set. The gene lists and additional information, including the statistical metrics used to generate the gene list from the training set, can be viewed at: . To identify comparable data points between the gene lists from the training set (produced on the U95Av2 Affymetrix chip), and the expression values of samples provided by other public datasets on different generation Affymetrix arrays, we used the probe match function within NetAffx . Data for these probe pairs in the validation sets were extracted and expression values were linearly adjusted to fall within 1–45000 [ and GenBank accession number queries. Ratios were log-transformed prior to analysis. GeneCluster2  was used to perform blinded predictions on the validation dataset using weighted voting with a leave-one-out methodology. Accuracy, specificity and sensitivity values were then generated for each predictor, as a measure of the predictor's ability to correctly group the samples into their respective class in the validation sets.10 μg of the extracted RNA from each sample was labelled and hybridized to an Affymetrix U133A array  according to protocol as previously described [. (2002) . To creaSAM carried out the accumulation of datasets, preparation of the predictors, data analysis and drafted the manuscript. SAM with the help of KMB and MMH participated in the sample preparation and hybridization to arrays of the in-house gene expression data. KMB also trained SAM and participated in the preparation of the predictors with SAM and also provided essential mentor support. MM participated in finding and extracting the public data and aided in drafting the manuscript. MMH was a tremendous help in editing the final manuscript. BL provided statistical guidance throughout the project. DC provided us with cDNA array data and aided in the analysis of the cDNA expression data. DC also provided much appreciated critical input throughout the entire project. DAS initiated and crafted the idea and provided the necessary mentorship along the way.


Much of the microarray data published at Stanford is based on mouse and human arrays produced under controlled and monitored conditions at the Brown and Botstein laboratories and at the Stanford Functional Genomics Facility (SFGF). Nevertheless, as large datasets based on the Stanford Human array began to accumulate, a small but significant number of discrepancies were detected that required a serious attempt to track down the original source of error. Due to a controlled process environment, sufficient data was available to accurately track the entire process leading to up to the final expression data. In this paper, we describe our statistical methods to detect the inconsistencies in microarray data that arise from process errors, and discuss our technique to locate and fix these errors.To date, the Brown and Botstein laboratories and the Stanford Functional Genomics Facility have together produced 40,000 large-scale  cDNA microarrays. By applying the heuristic described here, we have been able to check most of these arrays for misidentified features, and have been able to confidently apply fixes to the data where needed. Out of the 265 million features checked in our database, problems were detected and corrected on 1.3 million of them.Process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis. We show the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays. Expression microarrays, with the capability to measure the mRNA expression level of tens of thousands of genes simultaneously, have found broad application in both clinical and basic research -7. With This example is from the analysis of experiments from an ovarian tumor dataset that firh and plate i and plate n and plate o for all four comparisons of the first PCR round to the second.With MuFu, a check using a 384-well plate partition of the data shows a discrepancy but no obvious plate rotation or misidentification. We then repartition and repeat our tests at the 96-well plate level. The results are shown in Table h and i are swapped, and n and o are swapped, most likely in the plates from the first PCR round. The ambiguity is broken by sequencing a small sampling of wells, which confirms that the misidentifications are in the print plates from the first round of PCR, and not the second. In Figure In all four comparisons, the two distributions resolve themselves well, as can be seen in Figure p, identified in both the rank and rotated-rank matrices across print batches is a clear candidate for a plate rotation. The data from the distance matrix comparisons is shown in Table We compare four arrays A1, A2, A3 and A4 from a particular print production batch A to four arrays B, C, D and E from four other distinct print production batches. The total number of 384 well microtiter plates in print batch A is 113. The results of the plate rotation test are shown in Table q has a poor self-match comparison. Indeed, when the distance matrix is examined, we see the plate matches instead plate r across the four array comparisons. We check the three criteria for plate misidentification and summarize these data in Table In this example we compare four arrays A1, A2, A3 and A4 from a particular print batch A to four other arrays B, C, D and E from four other distinct print batches. As seen in Table Out of the 265 million features checked in our database using MuFu, problems were detected and corrected on 1.3 million of them. That we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm. That so few problems existed overall (0.5%) reassures us as to the robustness of our process in general.Microarray data, by its sheer volume, presents interesting laboratory information management challenges. The data arrive at the investigators desk after a significant number of steps. We have found that the better you track production, quality control, and experimental steps, the better chance you have of uncovering the reasons behind discrepancies that may appear in the data. Often, statistical analyses look only at the data presented as final expression values or ratios, without taking into account relevant quality control data. In our effort to understand our errors and the source of large systematic discrepancies in our data we have found the MuFu algorithm a useful test of certain classes of process errors, and as a check for general problems with specific process steps or microtiter plates. We use MuFu to find, verify and fix problems that can be attributed to an error in plate processing. We also flag plates for which we can find no specific problem but we see yield inconsistent results. These may be plates that, at some point in the process, had a problem  that was not detected while the process step was being carried out. The fact that we can test the data, using our standard quality control hybridizations, for these types of quality issues is reassuring and has helped us gain confidence in our data.Obviously, there are many other classes of error that creep into microarray data. Aside from the gross process errors that are amenable to detection, as we have described here, there is also a large class of more subtle systematic errors that can contribute to the overall systematic error on the expression ratio measurement. Isolating the source of these individual errors is sometimes quite difficult. Properties of the microarray feature such as spot size, shape and uniformity can contribute, but the majority of systematic errors are introduced at the time the experiment is performed. Slide post processing, RNA quality, labelling, hybridization and washing all lend the possibility for introducing systematic errors. Improvements in protocols and hybridization apparatus have helped reduce these errors and should continue to do so in the future. As these sources of error are identified and eliminated, expression microarrays will continue to provide progressively more sensitive measures of gene expression..Process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis. We have shown the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays. We generalized our procedure using a simple heuristic, which found and fixed several problems with the proper assignment of gene identifier with physical microarray feature. We found thirteen print runs (9K arrays) that had four plates mistracked, six print runs with single 384 plate rotations, and one instance of a plate rotation at the 96 well plate level. One skipped plate was detected, as well as one plate printed twice. Out of the 265 million features checked in our database using MuFu, problems were detected and corrected on 1.3 million of them. That we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm. That so few problems existed overall (0.5%) reassures us as to the robustness of our process in general. A list of corrected arrays can be found at We follow the simple heuristic outlined here. The flowchart for the program is shown in Figure Aij be the intensity measurements of array A, subset i and gene index j. The measurements are usually of the channel (Cy3 or Cy5) used as a common reference. For example, if we partition the data by the 384-well plate each feature once occupied at some point in its process history, A is the array id, i is the plate id, and j is the well index between 1–384. Let Ai be the vector . In our vernacular this is the 384-well plate expression vector for plate i. Let Bij andBi be the similar definitions for array B. We also reverse the data vectors from array A and label it Ai' . In our notation, Ai' =  is the expression vector for a plate rotated by 180°.1. Begin by partitioning the gene expression data on an array into subsets according to plate. Other partitions can also be made but the plate level partition is the most useful for our purposes. Let Dmn, 1 ≤ m, n ≤ N} in which each element Dmn = 1 - corr. For the 384-well plate example, Am is the 384-well expression vector for plate m on array A, Bn is the 384-well expression vector for plate n on array B, and Dmn is the distance between the two vectors in this 384 dimensional space. We also generate the corresponding rotated-distance matrix {D'mn, 1 ≤ m, n ≤ N} for the plate rotation case, in which each element D'mn = 1 - corr. We tried several correlation functions including Euclidean and Pearson but found the Pearson to be best suited to this task.2. Generate a distance matrix {Rmn, 1 ≤ m, n ≤ N} by converting the distances to ranks such that the row m in the rank matrix is the order statistic of the row m of the distance matrix. We also generate the corresponding rotated-rank matrix {R'mn} for {D'mn}. Ideally, for the case where we are comparing identical subsets from two different arrays we expect the diagonal elements of the rank matrix to all be equal to one, which means that each subset of genes from the first array matches its corresponding subset in the second array the best. In general, due to the statistical variation in array data, the diagonal elements are not all equal to one, even if there are no misidentification errors. The examples show that this does not hinder us from making a clear distinction between real problems in the data and statistical fluctuations.3. Generate a rank matrix {A1, A2, A3, A4 in the example) from the print batch in question to a control sample of arrays  selected from several distinct print batches. By comparing the rank R, and rotated-rank R', matrix assignments for comparisons across array batch boundaries we can quickly flag possible rotated plates. A visualization of this test is shown in Table A) against all of the control batches  in the non-rotated case, we conclude that the corresponding flagged plate or partition from batch A may be misidentified. Note that in the limit that the partitions of the array are all uniform in expression ratios there is a 5% probability of a spurious flag. For this reason it is better to use high quality, highly variegated control arrays for such tests. Next, to better resolve ambiguous cases and to check our rank matrix determination we use the distance rather than rank matrix. If, for example, a plate x is to be considered a rotated plate, the following three criteria must be met.A plate rotation may have an affect on a single microarray batch if it occurs during array printing or may persist across many print batches if it happens during a 96-well (PCR) process step. In any case, the mismatch will persist across many array comparisons. To check for plate rotations in a print batch, we compare an array sample (D'xx <Dxx. The rotated-distance must be less than the non-rotated distance.1. D'xx is close to the mean of the distribution, {Dmm, 1 ≤ m ≤ N}, and Dxx is close to the mean of the distribution {D'mm, 1 ≤ m ≤ N}.2. D'xx is an outlier of the distribution {D'mm, 1 ≤ m ≤ N}, and Dxx is an outlier of the distribution {Dmm, 1 ≤ m ≤ N}.3. D'mm, 1 ≤ m ≤ N} and {Dmm, 1 ≤ m ≤ N} separate well. In Figure The second and third criteria above are valid if the distributions of {x for a match with plate y with the following criteria:If a plate has consistently poor self-match rankings and a plate rotation has been excluded as a possible source of error, this next step tests for a plate misidentification. From the rank matrix we can identify the best alternative match. We test a suspect plate Dxy <Dxx. The mismatch distance is shorter than the match distance.1. Dxy is close to the mean of the distribution {Dmm, 1 ≤ m ≤ N}, and Dxx is close to the mean of the distribution {Dmn, 1 ≤ m ≠ n ≤ N}.2. Dxx is an outlier of the distribution {Dmm, 1 ≤ m ≤ N}, and Dxy is an outlier of the distribution {Dmn, 1 ≤ m ≠ n ≤ N}.3. Dmm, 1 ≤ m ≤ N} and {Dmn, 1 ≤ m ≠ n ≤ N} are well resolved.Again, the second and third criteria above are valid if the distributions of {IT, MF developed the heuristic and carried out analysis and implementation.MS brought this problem to our attention and provided test data.MD provided advice and checked our work with an alternative technique.All authors read and approved the final manuscript.


Analysis of DNA microarray data usually begins with a normalization step where intensities of different arrays are adjusted to the same scale so that the intensity levels from different arrays can be compared with one other. Both simple total array intensity-based as well as more complex "local intensity level" dependent normalization methods have been developed, some of which are widely used. Much less developed methods for microarray data analysis include those that bypass the normalization step and therefore yield results that are not confounded by potential normalization errors.Instead of focusing on the raw intensity levels, we developed a new method for microarray data analysis that maps each gene's expression intensity level to a high dimensional space of SEDs (Signs of Expression Difference), the signs of the expression intensity difference between a given gene and every other gene on the array. Since SED are unchanged under any monotonic transformation of intensity levels, the SED based method is normalization free. When tested on a multi-class tumor classification problem, simple Naive Bayes and Nearest Neighbor methods using the SED approach gave results comparable with normalized intensity-based algorithms. Furthermore, a high percentage of classifiers based on a single gene's SED gave good classification results, suggesting that SED does capture essential information from the intensity levels.The results of testing this new method on multi-class tumor classification problems suggests that the SED-based, normalization-free method of microarray data analysis is feasible and promising. DNA microarray technology is now playing an increasingly important role in biomedical research. Microarray technology gives one the opportunity to measure gene expression levels of thousands to tens of thousands of genes simultaneously, in order to study the differential gene expression pattern between different developmental stages, diseases states and samples treated with drugs or other compounds. Before comparing data from different arrays to address these biological questions, however, a "much more mundane but indispensable " normalization step  is curreBecause of the slight difference in RNA quantities, imaging settings and other variables, even in very controlled experiments the intensity levels from different arrays are of different scales and need to be normalized before they can be compared with each other. Various normalization methods have been developed and some are widely used. The simplest method is total intensity based normalization ; this apFor most current applications, these normalization methods seem to be adequate. However, the residual left by a less than perfect normalization procedure is another source of non-biological variation that is usually non-desirable, especially when the differences in expression levels are expected to be small . In addietc. We decided to use multi-class tumor classification as a test case to illustrate the power of our approach. We compared our results for a multi-class tumor classification problem with more conventional approaches published by Ramaswamy et al. , we will use (sij) to compare gene expression differences between arrays. Since we expect measurement variations within an array will be less than those between arrays and we take the signs of relative expression differences to get SED, we expect the SED will be less "noisy". However, the value of any single sij may still vary between technical and biological replicates. One would expect more sij would change values randomly if the technical replicates was done on arrays that were fabricated in a different run than arrays from same run, for example. Biological variations are expected to be even more frequent. However, we hypothesize that we can perform statistical analysis on the SED, which contains tens of thousands of sij for a single gene, and minimize the impact of such noises.Instead of directly using the intensity levels x, and its derivatives such as the mean μ, the standard deviation σ and the signal to noise ratio (S2N) between two sample groups A and B, , since the NB method is sensitive to the extreme values of sp, and samples can be over-predicated without thresholds.Because of the uneven and relatively small sample sizes for each tumor class (mostly 8 but up to 24), extra care was taken in computing spIn addition, several alternatives were tested to demonstrate that our results were reasonably robust and not sensitive to the particular choices we made:i.e. only the first 8 samples were used to calculate spij with no significant change of results observed;1) To examine the influence of the sample size, in a separate analysis the sample size of ME, LE, CNS class was artificially reduced to 8, ij depends on the sample size n for each tumor class, we have applied a "sample replacement" strategy in addition to the usual "take-one-sample-out" approach for cross-validation, i.e. when one sample is taken out as the test sample, another sample from the same class is duplicated to take its place to keep the sample size constant. Essentially the same results were obtained. Results reported are from the sample replacement runs.2) Since spij between all 14 classes was calculated as:In Feature Filtering, the variance of sp2 = /14 - /14)2     (4)σwith the test sample taken out, and used as the criterion for feature exclusion.Feature filter was done as in the Naive Bayes Method.The analysis was done on a computer (Pentium M 1.5 GHz) operating under Microsoft XP. Both Naïve Bayes and Nearest Neighbor Classifiers are implemented in Java. Since SED can be easily calculated from the raw intensities only the later are kept in memory and SED are computed from the intensities on an as-needed bases. Memory needed to analysis the 144 samples is less than 64 MB.The most computationally intensive algorithm that we tried is the Nearest Neighbor method without any feature selections and it takes about 10 sec to calculate SED score for one pair of tumor samples with about 16000 genes.hw14@columbia.edu.A Java program named SED (including source code) to perform nearest neighbor analysis of microarray samples is freely available by contacting author at H.W. conceived of the SED study and performed implementations. H.H. refined the approach and provided additional statistical insight on SED. Both authors read and approved the final manuscript.


High-density oligonucleotide microarray technology enables the discovery of genes that are transcriptionally modulated in different biological samples due to physiology, disease or intervention. Methods for the identification of these so-called "differentially expressed genes" (DEG) would largely benefit from a deeper knowledge of the intrinsic measurement variability. Though it is clear that variance of repeated measures is highly dependent on the average expression level of a given gene, there is still a lack of consensus on how signal reproducibility is linked to signal intensity. The aim of this study was to empirically model the variance versus mean dependence in microarray data to improve the performance of existing methods for identifying DEG.In the present work we used data generated by our lab as well as publicly available data sets to show that dispersion of repeated measures depends on location of the measures themselves following a power law. This enables us to construct a power law global error model (PLGEM) that is applicable to various Affymetrix GeneChip data sets. A new DEG identification method is therefore proposed, consisting of a statistic designed to make explicit use of model-derived measurement spread estimates and a resampling-based hypothesis testing algorithm.The new method provides a control of the false positive rate, a good sensitivity vs. specificity trade-off and consistent results with varying number of replicates and even using single samples. DNA microarrays have become common tools for monitoring genome-wide expression in biological samples harvested under different physiological, pathological or pharmacological conditions. One of the most challenging problems in microarray data analysis is probably the identification of differentially expressed genes (DEG) when comparing distinct experimental conditions. In spite of its biological relevance, there is still no commonly accepted way to answer this question.An ideal DEG identification method should limit both false positives, i.e. genes wrongly called significant type 1 errors), and false negatives, i.e. genes wrongly called not significant (type 2 errors). To this end, understanding how gene expression values measured in replicated experiments are spread around the true expression level of each gene, would help to distinguish biologically relevant gene expression changes from fluctuations due to different sources of variability that are unrelated to the biological phenomenon under investigation. Measurement error estimates can be obtained in two ways: either by empirically inferring noise from highly replicated data or by deducing noise from a theoretical error model [ errors, et al. [et al. [The most widely used methods for identifying DEG range from purely empirical filtering techniques (e.g. selecting genes that show a fold change higher than a fixed threshold) to more sophisticated statistical tests such as the signal-to-noise ratio described by Golub et al.  or the S [et al. . While eet al. [et al. [et al. [et al. [In the past years various authors have proposed competing error models for microarray data from which discordant implications for the variance versus mean dependence can be deduced. Chen et al.  first pr [et al.  and Li a [et al.  introduc [et al.  proposed [et al.  proposed [et al.  empiricaet al. [Independently from the choice of the error model, another point that remains to be faced is on how to estimate residual error. A discussed by Wright et al.  the possThe aim of this study was to use highly replicated microarray data to empirically determine the true variance versus mean dependence that exists in this type of data. This knowledge enabled the proposal of PLGEM as a simple but powerful error model. We fitted the proposed model on various data sets without pre-filtering the data, deriving an improved test statistics and identifying DEG even in data sets with very little number of replicates.The relationship between measurement variability and average expression values was investigated by means of scatter plots where different measures of spread were displayed against different measures of location. For each gene absolute or relative standard deviation was plotted against the mean expression value in either linear or log-log plots using data from the 16iDC data set :where α and β can be estimated from linear regression coefficients in 1 in a straightforward way:Model parameters α = ec     eqn. 3β = k     eqn. 4et al. [et al. [Instead of performing a simple linear fit through the whole set of points, we preferred to implement a method that could provide improved model robustness by partitioning the data to gain local estimates of spread as in Mutch et al. . Most im [et al.  proposed) value and subdivide the overall expression range into a given number p of partitions containing an equal number of ranked genes.• Rank genes according to their ln values as the x-coordinate and q-th quantile of ln(s) values as the y-coordinate.• Choose a "modeling quantile" q and determine for all the genes contained in each partition a single "modeling point" with median of ln. The reason for not considering p > 500 was that above this number we empirically revealed a poorer modeling quality in terms of correlation coefficients (data not shown), most likely due to the decrease of the number of data points contained in each partition.Thus, for all possible combinations of p and q a slope kth-percentile to obtain a central tendency of standard deviation to be used for improving test statistics (see next section). Another application of this method could be to fit PLGEM at the 5th- and at the 95th-percentile of standard deviation, to consequently find the limits of the corresponding 90% empirical confidence interval of standard deviation.As a straightforward application of this modeling method, PLGEM could be fitted at the 50In order to verify the feasibility of the former application, fitting of PLGEM on real-life data as well as distribution properties of the random variable E were investigated by analyzing the residuals of the model, i.e. differences between observed and expected values:εg computed from the 16iDC data set and its dependency on the rank of mean expression values. Figure α and β was determined. In all of the three independent data sets measurement variability could be accurately modeled through equation 2, with a power coefficient β that was always between 0 and 1 and a random variable E that appeared to be normally distributed with zero-mean and constant standard deviation over the whole range of expression values. Of course, these findings were eventually expected only for q = 0.5, and their occurrence demonstrated a goodness of fit of PLGEM on a series of unrelated real-life data sets.Figure et al. [In order to identify DEG, we implemented the following general algorithm derived from the framework of statistical hypothesis testing, in which we test against the null hypothesis of non-differential expression. First of all, we chose to implement as the test statistic the signal-to-noise ratio (STN) already used by Golub et al. , becauseet al. : and  represent, respectively, the mean of the replicated expression measures of gene g in condition 1 and 2, whereas  and  are the corresponding standard deviations. Instead, we propose to use model-derived standard deviation estimates predicted by PLGEM in equation 2 for the corresponding signal mean, rather than data-derived standard deviation values calculated independently from the few data points that are usually available for every single gene.where in the original version a priori knowledge on the truly differentially expressed transcripts. ROC plots investigate the relationship between false positive rates (FPR) and false negative rates (FNR) at different significance levels; in this way the performance of the PLGEM-derived STN statistic (PLGEM-STN) has been compared with the original STN statistic (CLASSIC-STN) and the statistic implemented in the commonly accepted Significance Analysis of Microarrays (SAM) DEG identification method (SAM-STAT). To this purpose Exp01 of the Latin Square was taken as the baseline to which the remaining 13 experiments were compared. For each comparison absolute values of each statistic were ranked in decreasing order and first n genes selected (where n ranged from 5 to 200). Figure n (data not shown). In addition, the ROC curve of PLGEM-STN always had the shortest distance from origin, indicating that it resulted in the best trade-off between sensitivity and specificity. Interestingly, improved sensitivity was observed especially when the nominal fold change was particularly low (see Exp02 vs. Exp01 and Exp14 vs. Exp01).The improvement of the test statistic in ranking DEG was evaluated as done by Broberg  through Apart from discriminating between significant and not significant gene expression changes, an optimal test-statistic should additionally provide an accurate quantification of the actual degree of differential expression. Figure Though ranking of genes based on the absolute value of their test-statistic has been proven to be an effective method for selecting DEG, an even more useful way would be to compare the observed statistic with its null distribution , in order to control the FPR.A classic approach to empirically obtain the null distribution of a test-statistic is running a series of random permutations of the chip indexes of the full data set and re-computing the test-statistics at each permutation. Permutated test-statistics can then be pooled and significance thresholds  are found as specific quantiles of the null distribution.Nevertheless, we can foresee that the classic permutation strategy may not be optimal for estimating the actual FPR when the test-statistic makes use of a global error model such as PLGEM. We can in fact hypothesize that measurement spread of DEG may not be accurately described by means of a global error model that was designed to describe signal variability in absence of differential expression. To test this hypothesis we compared the correlation between the expected significance level and the observed FPR using PLGEM-STN and the classic permutation strategy either including or excluding DEG during the permutation step. To this end, data sets containing different percentages of DEG were obtained by merging the 62 known DEG of the Latin Square data set with differently sized random samples of not DEG extracted from the same data set. As predicted, the presence of DEG during the permutation step caused the significance level to be less correlated with the observed FPR and this correlation worsened with increasing percentages of DEG Table . This la1 replicates of condition A with n2 replicates of condition B:Since in real-life data sets true DEG are unknown in advance, we propose the following resampling-based method to obtain the null distribution of not DEG when comparing n1 indexes corresponding to the replicates of only one experimental condition. If available, chose the condition with the highest number of replicates;• Artificial condition A* is obtained by randomly sampling with replacement n2 values from the same set to obtain indexes of artificial condition B*;• Similarly sample n• Compute resampled test-statistics between A* and B* at each cycle.The previous resampling should be repeated a sufficiently large number of times – as large as possible compared to the total number of possible combinations and compatibly with available computational resources – and the resampled test-statistics finally pooled. In our opinion resampling the expression values from only one experimental condition, rather than permutating indexes of both conditions, makes more sense with this particular statistic, because in this way we avoid merging true and false null hypothesis. Note that when more than one condition  are to be compared to a common baseline, the distribution of resampled test-statistics needs to be determined only once, obviously providing a computational advantage. As a test of substantial equivalence between this resampling method and the classic permutation strategy (excluding DEG), we compared the distribution of the permutated and of the resampled PLGEM-STN test-statistics in Q-Q plots. The distribution of the PLGEM-STN resampled from Exp01 of the Latin Square data set was almost identical with the distributions of permutated PLGEM-STN obtained with the classic strategy from each comparison with the remaining 13 experimental conditions (data not shown). Figure In accordance with the previous observations, the ROC curve of the resampling method applied to the PLGEM-STN statistic was not significantly different from the ROC curve of the classic permutation strategy (excluding DEG) applied to the same statistic on the Latin Square data set (data not shown). Conversely, ROC curves of the classic permutation strategy (including DEG) applied to the CLASSIC-STN statistic and of the SAM method gave poorer performance similarly to the results in Figure Another appealing feature of an optimal DEG identification method is that it should provide consistent results when different replicates of a same data set or different numbers thereof are analyzed. We therefore compared the performance of our resampling approach applied to the PLGEM-STN statistic (method 1) with SAM (method 2) and with the classic permutation strategy applied to the CLASSIC-STN statistic (method 3). The number of available replicates for each experimental condition in the Latin Square data set was unfortunately too small to investigate this particular task. We therefore took advantage of the 16iDC+LPS data set, where the first sixteen columns can be considered as the baseline condition for the remaining four experimental replicates. We then constructed a series of reduced data sets in which the baseline columns were kept constant while all possible combinations of 1, 2 or 3 replicates of LPS-stimulated DC were systematically deleted from the 16iDC+LPS data set, reaching a total of fifteen distinct data sets including the original one. Since methods 2 and 3 are not applicable on the four reduced data sets containing single samples for the LPS experimental condition, only the eleven data sets with at least two replicates were used for comparison purposes. Since the sixteen baseline columns are identical in each reduced data set, PLGEM parameters were determined only once on this common baseline condition. Significance levels used by each method in all eleven data sets were empirically selected in order to achieve a similar number of significant genes (ca. 500 probe sets) in the full data set, i.e. the one containing all available replicates. Thus, for each method eleven lists of identified DEG were obtained and the consistency between these lists was evaluated by counting the number of times each probe set was selected, giving a probes set count between 1 and 11. In Figure We finally evaluated the possibility of applying our method also to data sets where one of the experimental conditions was investigated only with a single sample without replication. To this end, we used the remaining four reduced data sets that could not be used in the previous comparison. In this case, the same PLGEM parameters derived from the sixteen baseline columns were applied to each of the single LPS-treated DC sample to obtain an estimate of standard deviation associated to each gene expression value, treated here as if it was a mean value from a larger group of values. Interestingly, when results obtained through this procedure were compared to the previously described results a comparable number of DEG was identified and only one probe set was newly detected in comparison to the previously identified ones (data not shown), arguing for a good consistency of results.In the present work we described a new global error model for microarray gene expression data that describes measurement variability with the same degree of accuracy over the whole dynamic range of values and that can be fitted at any desired quantile of spread. PLGEM has proven to correctly model signal standard deviation, in spite of the presence of different sources of variability, e.g. biological variability as well as the use of different target preparation protocols or of different chips. Moreover, PLGEM has shown to be able to deal with the great variability that exists at low expression levels while at the same time considering the significant relative reproducibility of highly expressed genes. Previously proposed error models assumed that measurement spread depended on signal location following different mathematical relationships, but none of them was based on a power law thus far. Analysis of the residuals showed a good fit of PLGEM to a number of high-density oligonucleotide microarray data sets, with model parameters being very similar to each other even when dealing with RNA samples coming from completely different biological sources and analyzed on different array layouts. This suggests that PLGEM could represent a general Affymetrix GeneChip measurement noise model. Even though scaled MAS5 Signals gave satisfactory modeling results, a further improvement could be achieved by using other emerging gene expression indices ,14 or moβ was found to be quite stable and comprised between 0 and 1 in all analyzed data sets. It is noteworthy that for β ∈ (0:1) absolute variability increases with growing expression values, while relative variability decreases  values increases, thus leading to an underestimation (or overestimation) of the standard deviation and a consequent overestimation (or underestimation) of the test statistic, finally leading to false positives .Interestingly, when the performance of our method was compared on a data set of DC stimulated for 24 hours with LPS, SAM showed a decreased sensitivity in identifying down-regulated genes when the number of LPS replicates was low (data not shown). Under these experimental conditions DC undergo a process known as maturation, which is a specialized form of cellular differentiation, for which both up- and down-regulation of gene expression is expected ,18. We sThe gene selection method proposed in the present work does not provide a direct control on the FDR, but the significance level has been proven to be a direct estimate of the FPR. Thus, if a significance level of 0.001 is used and 12488 probe sets are displayed on the MG-U74Av2 chip, 12–13 genes are expected to be selected by chance in cases where all genes are in fact not differentially expressed. Therefore, a researcher can test how many genes would be selected over a range of different significance levels and chose the one that results in the most acceptable compromise between number of selected genes and estimated FPR.The proposed DEG identification method provides a direct control of the FPR and an indirect control of the FDR. Moreover, as tested on the Latin Square data set, our method improved the specificity vs. sensitivity trade-off in comparison to other commonly applied DEG selection techniques. It finally showed an increased robustness when different replicates or numbers thereof are analyzed, giving consistent results even in data sets containing single samples. In conclusion, the global error model presented here may facilitate the analysis of microarray gene expression data by discriminating information from noise, and thus possibly helping the formulation of new hypothesis concerning gene functions.RNA was harvested from ten biological samples of unstimulated immature mouse dendritic cells (DC), each extracted from an independent batch of cells. One operator prepared the biotin-labeled cRNA for hybridization from three of the ten RNA samples, a second operator prepared the remaining seven. While operator 1 applied the total RNA protocol to all of its three samples, operator 2 applied the purified mRNA protocol to five of its seven samples and the total RNA protocol to the remaining two. Two of the three cRNA samples prepared by operator 1 and four of the seven cRNA samples prepared by operator 2 have been hybridized twice; therefore, a total of 16 MG-U74Av2 GeneChips  have been employed.Eight RNA samples were harvested from human fibroblast cell lines each deriving from a distinct Leigh syndrome patient ,20 and iFour individual and two pooled RNA samples from human muscle biopsies of sixteen healthy young male donors were hybridized on six HG-U95Av2 GeneChips (Affymetrix). This data set was downloaded from , experimThis data set consists of 3 technical replicates of 14 separate hybridizations (named Exp01–14) of 42 spiked transcripts in a complex human background at concentrations ranging from 0.125 pM to 512 pM. Thirty of the spikes are isolated from a human cell line, four spikes are bacterial controls, and eight spikes are artificially engineered sequences believed to be unique in the human genome. Further details on the design of the Latin Square data set can be found at . ConsideThis data set consists of the same samples of the 16iDC data set, but includes additional four samples as a second experimental condition. To this end dendritic cells were stimulated to mature with lipopolysaccharide (LPS) for 24 hours. Two independent biological samples were harvested and individually processed by the same two operators that prepared the samples for the 16iDC data set: one applied the total RNA protocol, the other one applied the purified mRNA protocol. Each cRNA sample was hybridized twice, thus using a total of four Affymetrix MG-U74Av2 chips.All chips mentioned in the present study were hybridized and scanned following Affymetrix recommendations and MicroArray Suite 5.0 (MAS5) was used as the image acquisition and analysis software. All data sets used passed quality control tests and probe set signals were scaled so that the 4%-trimmed mean of all expression values of each chip was equal to a predefined reference intensity  following manufacturer's recommendations:TGT = 100 for MG-U74Av2 and HG-U133A chips and TGT = 500 for HG-U95Av2 chips.All procedures for fitting PLGEM, for calculating observed PLGEM-based signal-to-noise ratios (STN), for obtaining expected PLGEM-STN through the resampling-based approach and for comparing observed with expected STN values have been implemented as R functions  and willNP conceived the study and drafted the manuscript. MP wrote the software, participated in the design of the study and in the editing of the manuscript. CV performed the microarray experiments, participated in the design of the study and the editing of the manuscript. MC participated in the microarray experiments, AS participated in the design of the algorithms, FG and PRC coordinated the study. All authors read and approved the final manuscript.Performance of modeling method using different combinations of parameters p and q. The modeling method described in this study was tested on the 16iDC data set using different combinations of partitions , and quantiles . For all 77 analyzed combinations of p and q regression lines were fitted to the data as described in the text. Goodness of fit was evaluated from the resulting slope (panel A), intercept (panel B) and adjusted r2 (panel C).Click here for file