Recent research examining cross-platform correlation of gene expression intensities has yielded mixed results. In this study, we demonstrate use of a correction factor for estimating cross-platform correlations.X and Y are measured with error. Herein we demonstrate that correcting for measurement error by estimating the "disattenuated" correlation substantially improves cross-platform correlations.In this paper, three technical replicate microarrays were hybridized to each of three platforms. The three platforms were then analyzed to assess both intra- and cross-platform reproducibility. We present various methods for examining intra-platform reproducibility. We also examine cross-platform reproducibility using Pearson's correlation. Additionally, we previously developed a correction factor for Pearson's correlation which is applicable when When estimating cross-platform correlation, it is essential to thoroughly evaluate intra-platform reproducibility as a first step. In addition, since measurement error is present in microarray gene expression data, methods to correct for attenuation are useful in decreasing the bias in cross-platform correlation estimates. Previous microarray gene expression studies have examined within-platform reproducibility among different generations of the Affymetrix GeneChip ,2 and amX and Y to be microarray gene expression values from two different platforms, and ρXY is estimated. However, for microarray data, both random variables X and Y are subject to measurement error. It is well known that the flourescent intensities from the scanned microarray images are proxies for the true underlying gene expression values [X and Y, and demonstrate its use in measuring cross-platform correlation.Most of these studies estimated Pearson's correlation as a means of assessing cross-platform reproducibility. That is, we consider n values . Therefon values . In a ren values . We haven values  which acMicroarray hybridizations were performed using three different technologies, each in a different laboratory. The Affymetrix (Affy) HG-U133A GeneChip was utilized in the Virginia Commonwealth University's (VCU) Division of Molecular Diagnostics Laboratory. A custom-designed oligonucleotide microarray designed specifically to interrogate genes more commonly expressed in brain tissue was used in VCU's School of Engineering's Center for Bioelectronics, Biosensors and Biochips (C3B). The C3B microarray platform comprises 10,000 genes represented by 3' fifty-mer oligonucleotides (MWG Biotech) that were spotted in duplicate. Finally, a cDNA microarray spotted with full and partial length PCR probes (Research Genetics/Invitrogen) was used in George Mason University's (GMU) Center for Biomedical Genomics and Informatics.Each laboratory designed a small experiment to assess intra-platform quality control. Each laboratory used the same lot of reference RNA, the Stratagene Total Human RNA, for hybridizing a set of technical replicates for a process variability study. These 'self-self' hybridizations permit meaningful assessments of reproducibility since, under ideal circumstances such as that the same experimental conditions exist among platforms and that there are no probe-binding affinity effects, each gene across the set of chips should exhibit linearly related gene expression intensities across platforms. Although the RNA hybridized was from the same lot, the study designs and protocols differed from lab to lab. A description of of each experiment can be found in the Methods section of this paper.2 ratio for all genes is 1, so that no correlation is expected when comparing two arrays in terms of the log2 ratio. Therefore for two channel arrays, we restricted attention to intensities from one channel as well as to the post-normalized intensities from that same channel. For the Affymetrix GeneChip, intensities were highly correlated across the set of three technical replicates for all expression summary methods , these two resulting datasets were then merged by Affymetrix probe set ID, resulting in 1,288 common probe sets/spots among the three platforms.w) of the log transformed Affymetrix GeneChip expression and C3B array expression are reported in Table Not accounting for measurement error, the average Pearson correlations (w) of the log transformed Affymetrix GeneChip expression and GMU array expression are also reported in Table The average Pearson correlations . We applied various methods for examining within-platform reproducibility including Pearson's correlation, the weighted kappa, and percent of invariant genes. We also examine cross-platform reproducibility using Pearson's correlation. We previously demonstrated the effectiveness of applying a correlation correction factor via a small simulation study and demonstrated its application in estimating gene-specific correlations. In this paper we demonstrated its use in estimating cross-platform reproducibility. We note that correcting for measurement error by estimating the "disattenuated" correlation removes the bias or attenuation inherent in cross-platform correlation estimates. Specifically, to the extent that random intra-platform variation is present, the effect is degraded performance on the apparent cross-platform correlation. Therefore, by removing random intra-platform variation through measurement error methodology, the cross-platform correlation will go up.Due to the increased public availability of gene expression microarray data through Gene Expression Omnibus  and ArraA major application of DNA microarray technology is differential gene expression profiling, or the detection of the differences in expression levels of genes between two different types of samples. Some have argued that the consistency of the differences via fold-change or ratio is a more relevant metric for assessing cross-platform comparability than intensities from a single channel. However, to estimate the correlation between fold-changes from two platforms, two different samples are needed. We therefore plan to use data from the MAQC project to examine cross-platform fold-change correlations. In addition, it has been suggested that a more relevant metric is not agreement in the identification of individual differentially expressed genes, but rather whether consistent and accurate predictions of sample class is obtained from the platforms being compared . This mePrevious researchers demonstrated that single and two channel microarrays yield consistent results, and concluded that the selection of which technology to use is not necessarily a critical factor in the design of a microarray study . Here weOne limitation of this study is that platform is completely confounded with laboratory technician and protocol, that is, the platform-specific sequence of reactions, scanner, procedures and events involved in the production of microarray data. It was previously noted that there is a high positive correlation between technician experience and intra-platform correlation . This isIn calculating cross-platform correlation, we assumed that the correlation estimated using the using the 1288 matching probes across the three platforms are representative of expected correlation of genes in the human genome that could be represented on the plaforms. Examination of absolute tag counts for the Stratagene Total Human RNA obtained using Serial Analysis of Gene Expression data (available from GEO #GSM1734) revealed that the intensity distribution of the 1,288 genes in common among the three platforms is not representative of the range of expected values Figures , 6, 7. TWhen estimating cross-platform correlation, it is essential to thoroughly evaluate intra-platform reproducibility as a first step. We also note that the commonly invoked procedure of estimating cross-platform consistency using only probes in common to all platforms is demonstrated to suffer from bias related to genomic coverage and probe annotation. Future studies comparing commercially available and custom designed arrays need to take this into consideration. Moreover, to the extent that random intra-platform variation is present, the effect is degraded performace on the apparent cross-platform correlation. Therefore, by removing random intra-platform variation through measurement error methodology, the cross-platform correlation will go up. Methods to correct for attenuation, such as that presented, are thus useful in decreasing such a bias in cross-platform correlation estimates. Platform-specific attenuation estimates may subsequently be used as a platform-specific quality measure and incorporated into a meta-analytic framework.Previously, each laboratory designed a small experiment to assess intra-platform quality control. Each laboratory used the same lot of reference RNA, the Stratagene Total Human RNA, for hybridizing a set of technical replicates for a process variability study. These 'self-self' hybridizations permit meaningful assessments of reproducibility since, under ideal circumstances such as that the same experimental conditions exist among platforms and that there are no probe-binding affinity effects, each gene across the set of chips should exhibit linearly related gene expression intensities across platforms. Although the RNA hybridized was from the same lot, the study designs and protocols differed from lab to lab.μm resolution using the Agilent G2500A Technologies Gene Array scanner. The full set of 16 Affymetrix GeneChips is publicly available [The Affy platform was assessed using an unbalanced three-factor design using 16 technical replicates . The samvailable .μg) were labeled with Cy3 and Cy5 according the The Institute for Genomic Research protocol and hybridized to three Human I chips. For each chip, the Stratagene Total Human RNA served as both the experimental and reference sample [μm of pixel resolution was used. Images were aquired by ScanArray Express 2.0 software and processed with QuantArray software.At GMU, the RNA was amplified using the MessageAmp aRNA Kit (Ambion). The amplified RNA (aRNA) was quantified and its quality was monitored by agarose gel and average size by the Agilent 2100 Bioanalyzer. The same amount of aRNA  at 80% laser power, 70% PMT gain, and 5 μm scan resolution. Spot intensities were acquired from the images using QuantArray software.The C3B laboratory assessed quality of their fabricated microarray using a fractional factorial design. The factors investigated were cDNA labeling strategy , input total RNA concentration ratio , hybridization time (2 levels: 4 and 16 hours), hybridization buffer , and production lot (2 levels: lot 7 and 9). Due to the expense of microarray production and hybridization, a fractional factorial design, rather than the full factorial design, was used. Therefore, all combinations of experimental conditions were not included. Specifically, by assuming that high-order interactions are negligible, information regarding the main effects and low-order interactions may be obtained by running only a fraction of the complete factorial design. Since we were interested in examining the effects of hybridization buffer (3 levels), RNA input ratio (3 levels), labeling strategy (3 levels), hybridization time (2 levels), and lot (2 levels), we were initially interested in a 3The analyses conducted in the current study were restricted to an equal number of chips by platform to ensure one technology did not dominate the results simply because of having a larger sample size. Three arrays were hybridized at GMU, so a random sample of size 3 was taken from the 16 Affy hybridized samples. These three GeneChips were QAQC8.CEL (Day 1 Frozen), QAQC10.CEL (Day 2 Frozen), and QAQC13.CEL (Day 2 Frozen). The three replicates selected from the C3B fractional factorial study were chosen based on 'optimal' hybridization conditions identified from the fractional factorial experiment. Specifically, the number of genes found to be signficantly different from the analysis of variance model was used as the metric estimating the relative influence of each main and two-factor interaction term. The level of each factor having the smallest number of genes differentially expressed was considered optimal. The three C3B chips used in this study were hybridized using the same buffer (Amersham), ratio of input experimental and control samples (1:1), and labeling method . The chips differed with respect to lot number and hybridization time, though these factors were found to not significantly influence the resulting intensities in the larger study.Since single-channel arrays measure expression intensities on an absolute scale whereas two-channel arrays measure expression intensities on a ratio-metric scale, we first investigated intra-platform reproducibility using different methods for calculating gene expression to aid in our determination of how to best transform the intensities from the three platforms to a similar scale. In addition, since the objective included an assessment of platform-specific reproducibility across the set of available technical replicates, methods for within-array normalization rather than methods that simultaneously normalize the data across all arrays, were applied in a platform-specific fashion.i = 1, ..., G by fitting print-tip loess regression models to the Mi = log2(channel 1i/channel 2i) (log difference) on Ai = (log2(channel 1i) + log2(channel 2i))/2 (log average) [th and 20th percentile of intensities for a given spot. Since simple background subtraction has been demonstrated to increase spot-level variability [For the two-channel arrays, we employed a commonly used procedure of normalizing the spot-level intensities on the array using print-tip loess regression and the subsequently analyzing the normalized spot-level intensities . The useaverage) . Probe ig ratios . In addiaaa@43EB@. Backgroiability , no backThe Affymetrix GeneChip Operating System (GCOS) was used to calculate expression summaries with a target intensity of 100 using the Microarray Suite version 5.0 (MAS 5.0) method . For comThe RESOURCERER annotation and cross-reference database  was deveE <0.000001) for the same spot were retained as matches. This threshold was determined considering the breakdown bound of the Tukey biweight estimator used in the MAS 5.0 expression summary algorithm. M-estimators with symmetric ψ-function have breakdown bound close to 50%. Therefore, probe sets for which > 60% of its PM probes specifically interrogated the same RefSeqID were retained. For the C3B microarray, each RefSeqID is spotted two times on the array. For the intra-platform reliability study (Stratagene dataset), average spot intensity per RefSeqID was retained as C3B gene expression. For the Affymetrix GeneChips, when multiple probe sets interrogated the same transcript, first, that probe set with the maximum proportion of probes with E <0.000001 was retained; when two or more probe sets had the same proportion, then the most 3' probe set was retained, defined by the probe set with maximum stop query sequence location among probes within a GenBank ID; when both quantities were the same, the probe set was randomly selected.Therefore, probe sets and spots with common sequences to all three platforms were retained for analysis using the following method. First, the GCG program 'netfetch' was used to obtain the NCBI GenBank records for spot IDs on the GMU and C3B microarray platforms. The perfect match (PM) probe level sequence data for the Affymetrix HG-U133A GeneChip was downloaded from the Affymetrix website (06/14/2005). BLASTN (v2.2.10) was used to query the Affymetrix probe sequences against the C3B sequences. Thereafter, all probe sets for which at least 60% of the probes reported low e-scores values  - argminil(Rikl)). A gene was designated as 'invariant' for platform l using the indicator I(Δil/G ≤ 0.05). As an example, this would correspond to permitting the rank to shift by no more than 1,114 when 22,283 genes are spotted on the array. Statistical tests of hypothesis comparing the proportions of invariant genes across platforms were conducted using a chi-square test.It has been suggested that poor cross-platform correlation is likely a result of low intra-platform consistency . Therefoyi. A weighted kappa statistic was used to allow a smaller penalty of misclassification among closely related classes, where the weights were taken to be wrc = (1 - 0.1 × |r - c|) when |r - c| < 10 and 0 otherwise.Finally, the weighted kappa statistic was estimated by first grouping gene expression intensities into 25 approximately equal-sized classes based on their ranked intensities, When fitting a linear regression modelxi and yi on observations i = 1, ..., n, it is assumed xi ~ N, which is not corrected for measurement error, can be estimated asFrom the intra-platform results, it is clear that microarray gene expression data is subject to measurement error. When estimating cross-platform correlation, let 2 Affymetrix intensities and where whereσx, σu, σy, and σv were fit using the regression calibration rcal function in Stata version 9 [w, were estimated separately for the GMU and C3B platforms relative to Affymetrix.This estimate adjusts for the bias present in estimating the correlation when measurement error is present. Estimates for ersion 9 . In estin values ,40, therKJA performed the statistical analyses and drafted the manuscript. CID, AFG, and CTG designed and performed the MDX Affymetrix quality control study. GST and TGE designed and performed the C3B quality control study. GMG designed and performed the GMU quality control study. MDC performed the BLAST search and assisted with merging the cross-platform data. All authors read and approved the final manuscript.


The use of clustering methods for the discovery of cancer subtypes has drawn a great deal of attention in the scientific community. While bioinformaticians have proposed new clustering methods that take advantage of characteristics of the gene expression data, the medical community has a preference for using "classic" clustering methods. There have been no studies thus far performing a large-scale evaluation of different clustering methods in this context.k-means, exhibited the best performance in terms of recovering the true structure of the data sets. These methods also exhibited, on average, the smallest difference between the actual number of classes in the data sets and the best number of clusters as indicated by our validation criteria. Furthermore, hierarchical methods, which have been widely used by the medical community, exhibited a poorer recovery performance than that of the other methods evaluated. Moreover, as a stable basis for the assessment and comparison of different clustering methods for cancer gene expression data, this study provides a common group of data sets (benchmark data sets) to be shared among researchers and used for comparisons with new methods. The data sets analyzed in this study are available at .We present the first large-scale analysis of seven different clustering methods and four proximity measures for the analysis of 35 cancer gene expression data sets. Our results reveal that the finite mixture of Gaussians, followed closely by  Microarray technologies enable the measurement of molecular signatures of cancer cells. These data allow different types of analyses, such as 1) the identification of differentially expressed genes  the iden. Anotherk-means  or . The third procedure transforms the values of the attributes into a ranking. This type of transformation is more robust to outliers than the first two procedures  for all genes [Z3.The above procedures could be adversely affected by the presence of outliers in the genes. This is especially true for ll genes . For shoThirty five microarray data sets are included in our analysis Table . These dIn terms of the data sets, it is important to point out that microarray technology is generally available in two different types of platforms, single-channel microarrays  or double-channel microarrays  ,47,49. Nj (attribute), we compute the mean mj among the samples. However, in order to get rid of extreme values, we first discard the 10% largest and smallest values. Based on this mean, we transform every value i and sample j as follows:Moreover, for the case of Affymetrix data, we apply the following procedure to remove uninformative genes: for each gene l-fold in at least c samples from their mean expression level across the samples. With few exceptions, the parameters l and c were chosen so as to produce a filtered data set with at least 10% of the original number of genes (features). It is important to point out that the data transformed with the previous equation is only used in the filtering step.For further analysis, we then select genes with expression levels differing by at least A similar filter procedure was applied for the case of cDNA microarray, but without the need to transform the data. In the case of cDNA microarray data sets, attributes (genes) of which could present missing values, we discard those with more than 10% of missing values. The attributes that remain and still present missing values have the values replaced by the respective mean value of the attribute.Z0), standardized (Z1), normalized (Z2) and ranked (Z3). We also applied the data with these transformations for the case of FMG.The experiments compare seven different types of clustering algorithms: SL, AL, CL, KM, FMG, SPC and SNN. Each of these algorithms  was implemented with versions of four proximity measures: Pearson's Correlation coefficient (P), Cosine (C), Spearman's correlation coefficient (SP) and Euclidean Distance (E). As the implementation of the KM and SNN used in this work does not support Spearman's correlation coefficient, these algorithms were only tested with P, C and E. Furthermore, regarding the Euclidean distance version, experiments are performed with the samples in four forms, namely, original , topic  and merge . Preliminary experiments reveal that variations in the other parameters did not produce very different results. Thus, the default values were used for the parameter strong and the value 0 was used for the parameters noise and label . From the partitions created with such parameters values, we choose the partition with best cR and with k in the interval of interest for further analysis.For SNN, we execute the algorithm with several values for its parameters (2%, 5%, 10%, 20%, 30% and 40% for DA implemented the approach and performed the experiments. DA, MS and IC evaluated the results. MS, IC, TL and AS designed the study and wrote the manuscript. All authors read and approved the final manuscript.


Disease classification has been an important application of microarray technology. However, most microarray-based classifiers can only handle data generated within the same study, since microarray data generated by different laboratories or with different platforms can not be compared directly due to systematic variations. This issue has severely limited the practical use of microarray-based disease classification.In this study, we tested the feasibility of disease classification by integrating the large amount of heterogeneous microarray datasets from the public microarray repositories. Cross-platform data compatibility is created by deriving expression log-rank ratios within datasets. One may then compare vectors of log-rank ratios across datasets. In addition, we systematically map textual annotations of datasets to concepts in Unified Medical Language System (UMLS), permitting quantitative analysis of the phenotype "distance" between datasets and automated construction of disease classes. We design a new classification approach named ManiSVM, which integrates Manifold data transformation with SVM learning to exploit the data properties. Using the leave one dataset out cross validation, ManiSVM achieved the overall accuracy of 70.7%  with many disease classes achieving the accuracy higher than 80%.Our results not only demonstrated the feasibility of the integrated disease classification approach, but also showed that the classification accuracy increases with the number of homogenous training datasets. Thus, the power of the integrative approach will increase with the continuous accumulation of microarray data in public repositories. Our study shows that automated disease diagnosis can be an important and promising application of the enormous amount of costly to generate, yet freely available, public microarray data. Microarray technology provides a revolutionary tool for understanding human diseases. Golub et al.  demonstrRecently, several studies have tested the feasibility of disease classification on cross-platform microarray data -9. EmploOur goal is that given microarray data profiling two samples, one normal condition and another disease condition, the disease condition can be classified based on the phenotype annotations of datasets in the public microarray database. To approach this problem, we need three component tools: (1) a feature vector to describe a microarray profile pair  that is comparable among microarray data generated with different platforms; (2) disease classes built from cross-platform microarray data based on their associated phenotype information; and (3) a machine learning approach capable of assigning potential phenotypes to a queried sample pair based on its similarity to profiled pairs in known disease classes.For the first component, we derive the expression log-rank ratio for each gene in each profile pair. By first deriving the expression log-rank ratios between a disease and a normal profile as meta-information within the same dataset, and then comparing such ratio profiles across datasets, the results shall be comparable across datasets. Simply speaking, we compare cross-dataset signals by emphasizing on differentially expressed genes, which were shown to be relatively robust to platforms or laboratory settings. To compBy integrating the microarray data of major platforms in the NCBI Gene Expression Omnibus (GEO) database , we consIn this section, we will first give a brief introduction to the methods, followed by analysis results. Figure We collected 232 human microarray datasets from three major platforms of the NCBI GEO : U95, U1To systematically categorize the phenotype information associated with each microarray dataset, we mapped the MeSH headings and the GEO dataset summary of each dataset to the UMLS concepts. Any UMLS concepts associated with only one microarray dataset were filtered out, resulting in a vocabulary of 185 disease and phenotype concepts. More details on this phase of the analysis are given in the Methods section.term frequency-inverse document frequency (tf-idf) [phenotype distance score whose estimated p-value is less than 0.05 were kept for further analysis. This cut ensures that each class has a similar level of homogeneity in its associated UMLS concepts. Details of the tf-idf calculation and its associated p-value estimation are described in the Methods section.This step groups microarray datasets into disease classes. We first employed the frequent itemset mining (FIM) algorithm  to ident(tf-idf) . Only thIn this manner we constructed 117 classes, comprising 68 microarray datasets. Each class contains 3 to 12 datasets. The classes covered a wide spectrum of conditions: cardiovascular/heart diseases, "bacterial infections and mycoses", neoplasms, CNS disorders, skin disorders, and metabolic diseases. Table positive set for that class. However, the size of the datasets can vary widely. Large datasets may come to dominate the characteristics of their disease classes. In order to get an unbiased estimator of classification accuracy, we randomly selected 50 log-rank ratio vectors from each dataset if its total number of profile pairs is greater than 50. We built the negative set by randomly sampling an equal number of vectors from datasets not in the positive set.The datasets within each disease class naturally form a t-test p-value < 0.05) between the positive and negative training sets. In the second approach, we first constructed a Laplacian matrix to represent the gene expression data [We applied two approaches to training the disease classifiers. The first was direct application of the SVM algorithm with a linear kernel and C-support vector classification (C-SVC), using the LIBSVM package . Prior tion data , therebyWe performed disease classification with SVM and ManiSVM, and evaluated performance with LOOCV (see Methods for details). We performed LOOCV by leaving out one dataset from the positive set (within a disease class) and the equal number of expression log-rank-ratio vectors from the negative set as the testing positive and negative set, respectively. Then the remainder positive and negative set are training positive and negative set, respectively. Even though our classification unit is a single profile, we left out the entire positive dataset to avoid bias caused by replicates in the same dataset. We used the following measures to assess classification performance: precision = TP/(TP+FP); recall = TP/(TP+FN); and accuracy = (TP+TN)/(TP+TN+FP+FN), where TP stands for true positives, TN for true negatives, FP for false positives, and FN for false negatives. The accuracy is actually a way to summarize the precision and recall information. For each LOOCV procedure, we repeated 5 times by random sampling of different negative sets, and we averaged the result to assess the classification performance.Our results showed that ManiSVM achieved the overall accuracy of 70.7% outperforming SVM (58.8%) by the default hyperplane positions. Although SVM can achieve high classification precision (89.8%), its recall is rather low (19.8%). In the contrast, ManiSVM provides more balanced performance and yielded 68.6% precision and 76.9% recall with 12% disease classes achieving the accuracy higher than 80%. By further shifting the position of hyperplane via adjusting the threshold of SVM decision value, SVM achieved the maximum accuracy of 67.5%  and ManiSVM achieved the maximum accuracy of 75.6% . Again, ManiSVM outperformed SVM. The endowed classification power of ManiSVM is attributed to the fact that the data points can be better separated in the Manifold space in terms of their phenotype similarity based on the chosen data similarity metric.The performance of individual classes varies greatly. For example, ManiSVM achieved 83.4% accuracy for the disease class described by the UMLS concepts , but only 67.2% accuracy for the disease class described by . One reason for this difference is that the former contains 9 datasets, while the latter contains only 3 datasets. In general, a class with more datasets will be easier to classify . Figure Although the correlation between dataset number and classification performance is strong Figure , outlierp-values correspond to lower phenotype distance scores and higher degrees of within-class data homogeneity. Figure p-value of the phenotype distance score. As the p-value of the phenotype distance score increases from 10-6 to 10-2, the accuracy of the classifier decreases from 74.6% to 63.3% and from 63.7% to 54.0% for ManiSVM and SVM respectively. This analysis demonstrates conclusively that classification power increases with dataset homogeneity. Thus, integrating multiple datasets is only expected to enhance classification performance if they are sufficiently homogenous in the diseases being measured.To properly compare the average phenotype distances within disease classes of different sizes (3-12 datasets), we estimated the statistical significance of phenotype distance scores by random sampling (see the Methods section). In general, more significant p-value 0.008), but proved rather difficult to classify. A further investigation of individual datasets revealed that the dataset GDS1022 actually studied lung pneumocytes after infection with Pseudomonas aeruginosa, which is not related to digestive system disorders. The reason that GDS1022 was mapped to this disease class is because its dataset summary mentioned "Pseudomonas aeruginosa causes serious respiratory infections in cystic fibrosis patients," and "cystic fibrosis" was automatically mapped to the "digestive system disorders" by the UMLS system. Thus, although the object of this study (lung pneumocytes) is not related to the digestive system, GDS1022 was nonetheless classed among digestive system disorders by the UMLS concept of "cystic fibrosis." This mapping imprecision resulted in inaccurate phenotype distance scores, and led to a low classification accuracy of 60.2% and 59.1% for this disease class by ManiSVM and SVM, respectively. On the other hand, such failures prove that the performance of our method could be further enhanced by a more advanced UMLS text mining tool. As with all automated text mining methods, however, mapping imprecision cannot be fully avoided. But with the rapid accumulation of microarray data, we should be able to minimize or bypass the influence of UMLS mapping noise by imposing stricter homogeneity requirements on candidate disease classes.In the above analysis, we have assessed the homogeneity of a disease class by comparing dataset annotations that were mapped to the UMLS concepts. Our results show that such assessment generally reflects the true phenotype similarity between datasets. Despite a satisfactory overall performance, we have observed a few exceptions to this rule. For example, the disease class  contained 3 datasets {GDS858, GDS1321, GDS1022} and had a fairly small average phenotype distance score 0.67  = ni, j/, where ni, j denotes the number of occurrences of UMLS concept i in dataset j. Then by definition, the value of tfi, j) (indicates the level of occurrence frequency of UMLS concept i in dataset j.1. For the UMLS concept inverse document frequency idfi = log(D/Di), where D denotes the total number of datasets and Di is the number of datasets containing the UMLS concept i. A smaller idfi implies a higher popularity of UMLS concept i among the collected microarray datasets.2. We calculated the tf-idf score was defined by tf-idfi, j)     define adjacency matrix W as wij = 1 if the Pearson correlation of the vector xi and xj is greater than the threshold γ; and wij = 0 otherwise. Here we set γ to be 0.25.1. Constructing the adjacency matrix: (1) calculate Pearson correlation for each vector pair L = D - W where W is the adjacency matrix defined above and D is a diagonal matrix of the same size as W satisfying Dij = ∑j wji; (2) identify the eigenvalues and eigenvectors by solving the equation Le = λe for λ and e, and order the obtained k eigenvalues increasingly: λ1 ≤ λ2 ≤ ⋯ ≤λk. The p eigenvectors e1, e2, ⋯ ep that corresponds to the p smallest eigenvalues λ1, λ2, ⋯, λp are then used to represent x1, x2,⋯, xk in the manifold space, where xi ∈ ℜl  is mapped to (e1(i),...,ep(i)) with ej(i) being the ith component of the eigenvector ej. We note that all the eigenvalues are non-negative since L is symmetric and positive semi-definite.2. Singular value decomposition (SVD): (1) build laplacian matrix Elab denote the k × p matrix with e1, e2, ⋯, ep being the column vectors of Elab th entry of Elab is ej(i)). Then Elab represents the transformed data in the manifold space with a reduced dimension of p, as described above., SVM analysis is subsequently performed on these transformed data. The motivation for performing the manifold transformation comes from the following important mathematical property of Elab:Following the notations in Belkin and Niyogi , we let Elab, say f1,...,fp)(e1(i),...,ep(i))T , we haveFor any linear operation on Elab is essentially to perform a linear operation on Elab, such as wij's being the weights. We note that the labelling differences associated with larger wij's (corresponding to the pairs of objects with higher similarities) are having more weights. Hence, the smaller S is, the more likely that the objects with high similarities would have the same class label. Furthermore, from equation (3), our manifold data naturally leads to the smallest S for a given linear operation and p due to the use of the smallest eigenvalues/eigenvectors. In brief, the manifold transformation helps better distinguish between the positive and negative sets and thus further improves the classification results.Applying SVM with the linear kernel to C-SVC: C-support vector classification; FIM: Frequent Itemset Mining; GEO: Gene Expression Omnibus; LOOCV: Leave One dataset Out Cross Validation; MeSH: Medical Subject Heading; SVD: Singular Value Decomposition; SVM: Support Vector Machine; tf-idf: term frequency-inverse document frequency; UMLS: Unified Medical Language Systems.The authors declare that they have no competing interests.CCL, JH, HH, and XJZ designed the study. CCL, JH, and MK performed the study. CCL, HH, and XJZ analyzed the result. CCL, HH, and XJZ wrote the paper.


Identification of molecular classifiers from genome-wide gene expression analysis is an important practice for the investigation of biological systems in the post-genomic era - and one with great potential for near-term clinical impact. The 'Top-Scoring Pair' (TSP) classification method identifies pairs of genes whose relative expression correlates strongly with phenotype. In this study, we sought to assess the effectiveness of the TSP approach in the identification of diagnostic classifiers for a number of human diseases including bacterial and viral infection, cardiomyopathy, diabetes, Crohn's disease, and transformed ulcerative colitis. We examined transcriptional profiles from both solid tissues and blood-borne leukocytes.The algorithm identified multiple predictive gene pairs for each phenotype, with cross-validation accuracy ranging from 70 to nearly 100 percent, and high sensitivity and specificity observed in most classification tasks. Performance compared favourably with that of pre-existing transcription-based classifiers, and in some cases was comparable to the accuracy of current clinical diagnostic procedures. Several diseases of solid tissues could be reliably diagnosed through classifiers based on the blood-borne leukocyte transcriptome. The TSP classifier thus represents a simple yet robust method to differentiate between diverse phenotypic states based on gene expression profiles.Two-transcript classifiers have the potential to reliably classify diverse human diseases, through analysis of both local diseased tissue and the immunological response assayed through blood-borne leukocytes. The experimental simplicity of this method results in measurements that can be easily translated to clinical practice. The development of gene expression microarray technology has enabled genome-wide transcriptional profiling of human and other cells in diverse tissues and phenotypic contexts -5. AmongRecent investigations have generated quantitative classifiers that typically consider tens to hundreds of relevant genetic transcripts in the classification of different disease states or the analysis of pathogenic processes ,15. In pThe top-scoring pair (TSP) algorithm discriminates between binary phenotypic states using just two transcriptional measurements. First described by Geman and colleagues in 2004 , the TSPTSP classifiers have been previously shown to accurately classify a number of human cancers, including tumors of the colon, prostate, and lung ,24; a trWe acquired publicly-available microarray gene expression data representing a diverse spectrum of human pathologies from the Gene Expression Omnibus . These sWe sought to determine the degree to which TSP and k-TSP classifiers are sensitive to the transcriptional measurements available on microarray probe sets. In previous studies, highly accurate single- and multi-pair classifiers were generated in diverse cancer classification tasks. However, the extent to which other gene pairs are able to discriminate between phenotypic states, beyond these top-performing classifiers, has never been determined. We iteratively generated TSP and k-TSP classifiers on the same GIST/LMS microarray data originally used to derive the OBSCN/PRUNE2 classifier . This daThe top-scoring pair algorithm generated classifiers that discriminate between diverse phenotypic states with various degrees of apparent accuracy Table . We alsoSensitivity and specificity were found to vary with the dataset. Two cases exhibit markedly low sensitivity - cardiomyopathy and transformed colitis. This was likely due to the comparatively smaller number of "positive" than "negative" tissue samples present in these microarray datasets, which serves as an implicit 'weight' for the algorithm to selectively choose classifiers exhibiting correct 'negative' diagnoses over correct 'positive' diagnoses. Variability in cross-validation accuracy was observed as a function of the disease being examined, with limited correlation observed between sample size and classifier accuracy. Extremely low false-discovery rates were witnessed for all datasets as derived from comparing the distributions of classifier accuracies in unmodified and randomly permuted data, indicating high statistical significance of each classifier. Every classifier except the prediction of ulcerative colitis transformation had an estimated false discovery rate of well below 0.01 from 10 independent permutations; the lower performance in this dataset was likely due to the smaller number of experimental samples included therein.Selected classification tasks are shown in Figure We have shown that simple two-transcript gene expression classifiers can accurately classify a wide spectrum of human diseases. This algorithm is invariant to data normalization and generates robust, statistically significant biological classifiers even in the context of low sample sizes. Our results reveal that many pathological processes, even those not traditionally considered genetic in nature such as infections and inflammatory disorders, can be diagnosed through just two transcriptional measurements. Whereas previous work has shown the diagnostic value of gene expression perturbations, this study demonstrates that as few as two transcriptional measurements can reliably detect diverse human diseases.Transcriptional networks themselves can thus be seen to encode aspects of pathological phenotypes, with strong correlation observed between gene expression status and disease state. These transcriptional signatures were sufficiently robust to be detected even in tissue samples of possibly heterogeneous cell populations. The accuracies observed in these simple diagnostic modalities were comparable to pre-existing transcription-based classifiers that rely on more complex, multivariate measurements. For example, a 12-gene classifier generated against the same Crohn's disease dataset using a weighted-voting scheme exhibited a cross-validation accuracy of 94%, compared with equivalent TSP cross-validation performance of 87% . AdditioThe TSP method compared favourably to the estimated accuracy of standard clinical methods for the differentiation of viral and bacterial infection, as well as cardiomyopathy classification- conditions that present ongoing diagnostic challenges in the clinic. For example, a recently developed clinical prediction rule to discriminate between bacterial and viral pneumonia in children achieved positive predictive value of under 80%, in contrast to a TSP classifier cross-validation accuracy of 96.7% . AdditioInterestingly, the ability of the classifier to obtain an accurate diagnosis was significantly lower in the comparison of ischemic and idiopathic cardiomyopathies than in any other dataset we examined. This is likely due to the broad cellular and metabolic heterogeneity observed in these two closely related conditions. Both clinical and molecular differentiation of ischemic and idiopathic cardiomyopathies remains a significant challenge . Ischemia priori to be involved in pathological processes, the strong association held by many such transcripts with their cognate phenotypes demonstrates the biomolecular relevance of these classifiers.We observed that the genes present in highly accurate two-transcript classifiers were often associated with disease processes in previous literature reports. For example, PRUNE2 has been shown to inhibit certain forms of oncogenic transformation, which may correspond to its differential regulation in GIST and LMS as observed through the TSP method . The TSPIntriguingly, in this study it was found that analysis of transcription in circulating mononuclear cells provides a robust diagnostic platform for both the detection of invading cellular or viral pathogens, and the diagnosis of somatic medical conditions such as diabetes and Crohn's Disease. Of particular interest are the simplicity, robustness and accuracy of two-transcript classifiers using a data source that provides an easily accessed transcriptomic 'readout' from pathologies of disparate tissues. Recent studies have examined the utility of serum-borne mRNA in the prediction of diseases, with varying fidelity ,41. ThesTo implement the two-transcript classifiers, transcriptional measurements can be readily obtained in the clinic through routine PCR procedures . The sucWe have shown that the top-scoring pair algorithm is able to generate statistically significant and accurate gene expression classifiers from microarray data. These methods are insensitive to data normalization, and perform consistently when applied to novel experimental data. Furthermore, the method is able to detect diverse human diseases, even those not considered genetic in nature or cause. Ultimately, two-transcript classifiers obtained from microarray gene expression data present a robust analytical tool for clinical diagnostics.rank relative to all the gene expression values within the sample. This rank-based processing renders the algorithm invariant to monotonic data normalization. Importantly, this algorithm treats each probe within a microarray platform individually - such that, even when multiple probes are spotted independently for the same gene on a microarray, both probes are treated as independent, unrelated measurements.The input to the top-scoring pair (TSP) algorithm is a gene expression matrix from a microarray probe set corresponding to semi-quantitative transcriptional measurement, from multiple unique tissue samples. The algorithm first replaces the gene expression value within each sample by its corresponding A and B whereby their relative expression predicts phenotypic class (either class 1 or 2), employing a simple classification rule for any sample:The algorithm then assesses all possible pairs of genes Rank (gene A) > Rank (gene B), THEN Class = 1; ELSE Class = 2IF difference in rank between two genes is calculated in the event of ties between equivalently accurate classifiers as described previously [For each gene pair, the number of accurate class predictions is counted and each gene pair is then ranked according to the cumulative predictive accuracy across all samples. The most accurate transcript pairs are returned as top-scoring classifiers. The mean eviously . SensitiTo address the significant data-storage matters arising from assessing so many prospective classifier pairs, the algorithm employs a dynamic data analysis feature whereby only pairs that might possibly represent 'top-scoring' pairs are recorded for further analysis. Additionally, the algorithm employs this parsimonious library of highly accurate pairs to reduce the computational time required for the k-TSP algorithm and leave-one-out cross-validation analysis. With these optimizations, the algorithm is able to fully analyze even large microarray datasets within one day on a standard desktop computer, including cross-validation analysis and False Discovery Rate prediction.k individual TSP classifiers can be combined into a multi-pair 'k-TSP' classifier. In this approach, the TSP algorithm itself is performed, and all possible transcript pairs are ranked in order of their classification accuracy. The top k highest-ranked TSP pairs for a given classification task each represent one 'vote', with equal weight, for the class of each given sample; the final predicted class of each sample is the phenotype with the majority of votes. To avoid ties, k is restricted to odd numbers only; for this study the maximum value of k was held to 11. For each classification task, a leave-one-out cross-validation loop (described following) is employed to determine the optimal value of k.In an extension of the TSP algorithm, We employed TSP and k-TSP algorithms to determine the degree to which these methods can generate multiple unique gene expression-based classifiers. We first determined the optimal TSP and k-TSP classifiers against the previously mentioned GIST/LMS gene expression data. We then removed the top-scoring individual gene pair from the dataset, and repeated the algorithm on this reduced gene expression data. We iteratively performed this gene-pair excision, and recorded TSP and k-TSP classifier accuracies at each step. The value of k was held to a maximum of 11, and was determined in each iteration by an internal loop of leave-one-out cross-validation that established the optimal value of k for each classification task.N-1 samples is used to predict the left-out sample class. This cross-validation is performed iteratively for each of N samples, with the number of correct predictions out of N then averaged to determine LOOCV accuracy. Cross-validation sensitivity and specificity were also determined.To estimate algorithm performance on novel samples, we performed leave-one-out cross-validation (LOOCV), in which the top-scoring pair as determined by To estimate the statistical power of each classifier, we applied the algorithm to each dataset following random permutations of phenotypic class labels across all samples. We then compared the distribution of gene pair classification accuracy between actual and randomized data, and calculated the false discovery rate (FDR), corresponding to the likelihood of finding an apparently accurate classifier due to chance. Examples of these original and randomized distributions are shown in figure LBE was responsible for the design and execution of the study, implemented the TSP algorithm, collected and processed data, and assembled the manuscript; GT assisted with the collection of microarray data and data analysis; WZ and NDP conceived of the study. WZ, DG, and NDP provided guidance on the study and edited the manuscript. All contributors read and approved the final manuscript.Appendix 1. DatasetsClick here for file


Cancer diagnosis and clinical outcome prediction are among the most important emerging applications of gene expression microarray technology with several molecular signatures on their way toward clinical deployment. Use of the most accurate classification algorithms available for microarray gene expression data is a critical ingredient in order to develop the best possible molecular signatures for patient care. As suggested by a large body of literature to date, support vector machines can be considered "best of class" algorithms for classification of such data. Recent work, however, suggests that random forest classifiers may outperform support vector machines in this domain.In the present paper we identify methodological biases of prior work comparing random forests and support vector machines and conduct a new rigorous evaluation of the two algorithms that corrects these limitations. Our experiments use 22 diagnostic and prognostic datasets and show that support vector machines outperform random forests, often by a large margin. Our data also underlines the importance of sound research design in benchmarking and comparison of bioinformatics algorithms.We found that both on average and in the majority of microarray datasets, random forests are outperformed by support vector machines both in the settings when no gene selection is performed and when several popular gene selection methods are used. Gene expression microarrays are becoming increasingly promising for clinical decision support in the form of diagnosis and prediction of clinical outcomes of cancer and other complex diseases. In order to maximize benefits of this technology, researchers are continuously seeking to develop and apply the most accurate classification algorithms for the creation of gene expression patient profiles. Prior research suggests that among well-established and popular techniques for multicategory classification of microarray gene expression data, support vector machines (SVMs) have a predominant role, significantly outperforming k-nearest neighbours, backpropagation neural networks, probabilistic neural networks, weighted voting methods, and decision trees .In the last few years substantial interest has developed within the bioinformatics community in the random forest algorithm  for clasC that balances data fit with insensitivity to outliers. Fourth, the performance metric used in [However, closer inspection of this prior comparison  reveals  used in , proport used in -10. Fift used in  which is used in . FurtherWe hypothesize that these apparent methodological biases of prior work have compromised its conclusions and the question of whether random forests indeed outperform SVMs for classification of microarray gene expression data is not convincingly answered. In the present work we undertake a more methodologically rigorous comparison of the two algorithms to determine the relative errors when applied to a wide variety of datasets. We examine the algorithms both in the settings when no gene selection is performed and when several popular gene selection methods are used. To make our evaluation more relevant to practitioners, we focus not only on diagnostic datasets that are in general known to have strong predictive signals, but also include several outcome prediction datasets where the signals are weaker and larger gene sets are often required for optimal prediction.The performance results of classification prior to gene selection are shown in Figure Six classification performance estimates have been produced for each classifier and dataset (5 estimates corresponding to various gene selection methods and one estimate corresponding to using no gene selection). In Figure According to the results in Figure The number of genes selected on average across 10 cross-validation training sets is provided in Table The results presented in this paper illustrate that SVMs offer classification performance advantages compared to RFs in diagnostic and prognostic classification tasks based on microarray gene expression data. We emphasize that when it comes to clinical applications of such models, because the size of the patient populations is typically very large, even very modest differences in performance  can result in very substantial differences in total clinical outcomes  .The reasons for superior classification performance of one universal approximator classifier over the other in a domain where the generative functions are unknown are not trivial to decipher ,14. We pmtry  and ntree  even in the case of linear decision function when complicated decision surface modelling is not required. The experiment shows that the choice of RF parameters creates large variation in the classifier performance whereas the choice of the main SVM parameter has only minor effects on the error. In practical analysis of microarrays this means that finding the RFs with optimal error for the dataset may involve extensive model selection which in turn opens up the possibility for overfitting given the small sample sizes in validation datasets.The simulation experiment described in Additional File A second plausible explanation is that decision trees used as base learners in the RF algorithm cannot learn exactly many linear decision functions in the finite case. Specifically, if the generative linear decision function is not orthogonal to the coordinate axes, then a decision tree of infinite size is required to represent this function without error . The votWe note that regardless of the specific reasons why RFs may have larger error on average in this domain, it is still important to be aware of the empirical performance differences when considering which classifier to use for building molecular signatures. It may take several years before the precise reasons of differences in empirical error are thoroughly understood, and in the meantime the empirical advantages and disadvantages of methods should be noted first by practitioners.Data analysts should also be aware of a limitation of RFs imposed by its embedded random gene selection. In order for a RF classification model to overcome the trap of large variance, one has to use a large number of trees and build trees based on a large number of genes. The exact values of these parameters depend on both the complexity of the classification function and the number of genes in a microarray dataset. Therefore, in general, it is advisable to optimize these parameters by nested cross-validation that accounts for the variability of the random forest model .Finally, it is worthwhile to mention the work by Segal  who quesThe primary contribution of the present work is that we conducted the most comprehensive comparative benchmarking of random forests and support vector machines to date, using 22 diagnostic and outcome prediction datasets. Our hypothesis that in previously reported work, research design limitations may have biased the comparison of classifiers in favour of random forests, was verified. After removing these benchmarking limitations, we found that, both on average and in the majority of microarray datasets, random forests exhibit larger classification error than support vector machines both in the settings when no gene selection is performed and when several gene selection methods are used.The quest for high performance classifiers with microarray gene expression and other "omics" data is ongoing. Random forests have appealing theoretical and practical characteristics, however our experiments show that currently they do not exhibit "best of class" performance. Our data also points to methodological limitations of prior evaluations and thus emphasizes the importance of careful design of bioinformatics algorithm evaluation studies.[Gene expression microarray datasets used in the present work are described in Table  and from.We used 10-fold cross-validation to estimate the performance of the classification algorithms. In order to optimize algorithm parameters, we used another "nested" loop of cross-validation by further splitting each of the 10 original training sets into smaller training sets and validation sets. For each combination of the classifier parameters, we obtained cross-validation performance and selected the best performing parameters inside this inner loop of cross-validation. Next, we built a classification model with the best parameters on the original training set and applied this model to the original testing set. Details about the "nested cross-validation" procedure can be found in ,20. NotiSeveral theoretical reasons explain the superior empirical performance of SVMs in microarray data: e.g., they are robust to the high variable-to-sample ratio and large number of variables, they can learn efficiently complex classification functions, and they employ powerful regularization principles to avoid overfitting ,21,22. ElibSVM software library [ with polynomial kernel. Recall that the SVM polynomial kernel can be defined as: K = (γ·xTy + r)d, where x and y are samples with gene expression values and γ, r, d are kernel parameters. The parameters γ and r were set to default value 1. The kernel degree d together with the SVM penalty parameter C were optimized by nested cross-validation over d values {1, 2, 3} and C values {0.01, 1, 100}.We used SVM implementation in the  library  with polRandom forests (RF) is a classification algorithm that uses an ensemble of unpruned decision trees, each of which is built on a bootstrap sample of the training data using a randomly selected subset of variables . As mentrandomForest [, we considered different parameter configurations for the values of ntree = {500, 1000, 2000} (number of trees to build), mtryFactor ={0.5, 1, 2}  and selected the best-performing configuration by nested cross-validation. Note that the above parameter values are also consistent with the recommendations of the study [We employed the state-of-the-art implementation of RF available in the R package omForest . This imomForest ,25 and ,Even though both SVM and RF classifiers are fairly insensitive to very large number of irrelevant genes, we applied the following widely used gene selection methods in order to further improve classification performance:varSelRF implementation of the RFVS method developed by its inventors and applied it with the recommended parameters: ntree = 2000, mtryFactor = 1, nodesize = 1, fraction.dropped = 0.2 , and c.sd = 0 (a factor that multiplies the standard deviation of error for stopping iterations and choosing the best performing subset of genes). We refer to this method as "RFVS1."• Random forest-based backward elimination procedure RFVS : The RFVc.sd = 1 (denoted as "RFVS2"): This method differs from RFVS1 in that it performs statistical comparison to return the smallest subset of genes with performance statistically indistinguishable from the nominally best one.• RFVS procedure as described above, except for • SVM-based recursive feature elimination method RFE : This is• Backward elimination procedure based on univariate ranking of genes with "signal-to-noise" ratio ,21,28  which was computed from continuous outputs of the classifiers (distances from separating hyperplane for SVMs and outcome probabilities for RFs) . For mulWhen comparing two classifiers, it is important to assess whether the observed difference in classification performance is statistically significant or simply due to chance. We assessed significance of differences in classification performance in individual datasets or in all datasets on average using a non-parametric permutation test  based onConceived and designed the experiments: AS, LW, CFA. Performed the experiments: AS. Analyzed the results of experiments: AS, LW, CFA. Wrote the paper: AS, CFA. All authors read and approved the final manuscript.Results for all gene selection methods, classifiers, and datasets.Click here for fileSimulation experiment demonstrating sensitivity of random forests to input parameters.Click here for fileComplete information about microarray datasets used in the study.Click here for file


Non-biological experimental error routinely occurs in microarray data collected in different batches. It is often impossible to compare groups of samples from independent experiments because batch effects confound true gene expression differences. Existing methods can correct for batch effects only when samples from all biological groups are represented in every batch.In this report we describe a generalized empirical Bayes approach to correct for cross-experimental batch effects, allowing direct comparisons of gene expression between biological groups from independent experiments. The proposed experimental design uses identical reference samples in each batch in every experiment. These reference samples are from the same tissue as the experimental samples. This design with tissue matched reference samples allows a gene-by-gene correction to be performed using fewer arrays than currently available methods. We examine the effects of non-biological variation within a single experiment and between experiments.Batch correction has a significant impact on which genes are identified as differentially regulated. Using this method, gene expression in the blood of patients with Duchenne Muscular Dystrophy is shown to differ for hundreds of genes when compared to controls. The numbers of specific genes differ depending upon whether between experiment and/or between batch corrections are performed. Non-biological experimental variation is commonly observed in microarray data processed in different batches. A batch is defined as a set of microarrays that are processed together within a single experiment. In this report we define an experiment as an individual study conducted at one site at one time. An experiment often has many samples processed in multiple batches. Different batches are processed at different times or by different operators. Batch effects are caused by many factors such as the methods for RNA isolation, amplification and target labeling, and array processing and scanning. Several methods have been proposed that can adjust for batch effects provided a large number of samples > 25) are included in each batch ,2. More 5 are incThe aforementioned methods can adjust for batch effects provided that samples from each biological group are represented in every batch. The left panel of Figure In this report we present an experimental design that allows one to compare different biological groups drawn from independent experiments. The feature of our experimental design that allows samples to be compared even when control and experimental samples are in completely separate experiments is that identical replicate reference samples of pooled RNA are processed in each batch in every experiment. These reference samples are pooled RNA from the same tissue as are the disease and control experimental samples. In our study the experimental samples and reference samples are from human blood. Only a single reference sample is necessary to be included in each batch. Our experimental design is illustrated in Figure Simulations and data analyses are used to study the properties of the proposed methods. These methods are an extension of the empirical Bayes batch effect adjustment proposed by Johnson et al .We first show that the empirical Bayes method can correctly adjust for batch effects using two simulated data sets Figures  and 4. FFor the first simulation, analysis of unadjusted data identifies only 265 (44%) of truly differential expressed genes as being differentially expressed with FDR adjusted p-value ≤ 0.05 and fold change > 2.0. Because the batch effects lowered the fold change values, all of the remaining simulated genes had fold change values of less than 2.0 Figure . HoweverIn the second simulation, batch effects amplify the expression level differences between the two groups Figure . Not surWe next demonstrate on our experimental data set that the empirical Bayes method removes batch effects that occur between experiments. In this data there are two experiments, one that includes microarray data for four batches of control samples processed at the University of Cincinnati and a second that includes microarray data for patients with muscular dystrophy processed in five batches at UC Davis. We illustrate the successful removal of batch effects by comparing expression of differentially expressed genes before and after adjustment for batch effects.For the unadjusted GCRMA-summarized expression values, a total of 527 genes are differentially expressed in patients compared to controls. Using the expression values for these genes only, we applied agglomerative hierarchical clustering with Pearson correlation to the 98 samples , we identified 629 differentially expressed genes. Figure In this section we show that the empirical Bayes method successfully adjusts for within-experiment batch effects and that these can be of even greater magnitude than the between-experiment batch effects.Table There is one batch processed at UC Davis that clearly has the most pronounced effects. Out of the 527 genes that were differentially expressed in at least one batch for the unadjusted data set, nearly all (517) were differentially expressed in the same batch. For the remaining four batches, the numbers of differentially expressed genes were many fewer – ranging between 27 and 112 for the four batches processed at UC Davis and between 4 and 28 for those processed at Cincinnati. The lower color bar beneath the heat map in Figure The numbers of differentially expressed genes are shown in Table In this section we illustrate the advantages that the empirical Bayes method offers by comparing it to a more naïve approach which uses a t-test to filter out genes with significant differences in expression between reference arrays in the two experiments. Applying the t-test to the 527 genes identified as differentially expressed between patients and controls in the unadjusted data results in the removal of 254 genes with significant between-experiment batch effects. Out of the remaining 273 genes, while none show differential expression in at least one of the four batches processed in Cincinnati, 42 show differential gene expression in at least one of the five batches processed at UC Davis Table . This isFigure Figure In this report we have shown that with an appropriate experimental design and statistical methods it is possible to adjust for both within and between experiment batch effects and hence compare gene expression values between biological groups drawn from independent experiments. The unique feature of our experimental design that enables us to compare data from separate experiments is that identical replicate reference samples of pooled RNA  are included in each batch within each experiment. The inclusion of the identical reference samples from the same tissue in every batch in every experiment allows us to adjust for non-biological variation and hence be able to distinguish differences in gene expression due to the underlying disease biology from those due to confounding batch effects. This approach enables us to compare the gene expression of samples from the two groups that would otherwise be incomparable due to confounding batch effects. The importance and uniqueness of this method are best understood when viewed in the context of the other methods described below.Several computational methods have been developed to correct for non-biological variation. Alter et al.  used SinThe Bayesian approach developed by Johnson et al  is basedall samples, the multiplicative batch effect (i.e. the variance) can still be estimated even when there is only a single reference sample in each batch. However, an experimental design would ideally include more than one reference sample per batch as a preventative measure in case one of the reference samples was of poor quality. In this study we are able to justify the use of a single reference sample based on the results of an ANOVA analysis of the batch adjusted data. The bottom row of Table This Bayesian approach is particularly advantageous for the analysis of our data set which had a particularly limited number of reference samples – the minimum number of one reference sample per batch. Because in the first step of the parameter estimation procedure, the pooled variance for each gene is calculated across A related concern is that the estimate of the variance for many genes might be unusually large relative to the differences in the batch means because there was only one reference sample used. In this scenario the adjustment method would often times only add noise and reduce sensitivity. It is important to note that for the observed estimated parameter values in our data set, it is infrequent for the batch variances to be large relative to the differences in the batch means. Again, even though there is only one reference sample per batch, the variance estimate is likely stabilized by the pooling of samples.The methods described above have been applied in the context of adjusting batch effects within a single experiment. Existing methods for comparing microarray data across independent experiments include meta-analysis where sets of genes that are consistently differentially expressed across multiple experiments are identified. In this approach, statistical measures of differential expression generated from data sets derived from independent experiments are compared instead of the gene expression data matrices from different data sets. Rhodes et al.  developeThe meta-analysis approaches described above have been very useful in comparing independent microarray data sets-especially when the data sets were generated across different platforms. However, in these methods the gene expression data matrices for test samples from one experiment are not compared to the control samples from another experiment. In contrast, our experimental design enables such comparisons to be made between experiments using the same microarray platform and hence offers an enormous increase in the number of different gene expression comparisons that can be made between data sets drawn from independent microarray studies. This makes it possible to test many more hypotheses for differences in gene expression. For example, suppose two independent studies were performed, each comparing a different disease group to healthy controls. If identical reference samples were included in each of these experiments, it would be possible to directly compare gene expression values between the two disease groups and identify genomic signatures that distinguish one disease group from the other in addition to the genomic signatures that distinguish each disease group from the healthy individuals.The proposed experimental design has substantial impact for the use of microarrays in clinical studies. Because it is possible to compare gene expression values for disease and control samples drawn from separate experiments, it is possible to recycle expression data for one set of controls, comparing it to disease gene expression data sets for follow-up experiments. This would eliminate the burden and expense of reprocessing control samples each time an experiment is repeated. Furthermore with the ability to re-use a single control gene expression data set, it becomes possible to define a single standard control population for a disease study and establish a universal reference data set of gene expression values. This would dramatically increase the utility of microarrays in clinical diagnostics.Gene expression was measured using Affymetrix U133plus2 microarrays on 98 human blood samples in two separate experiments. This research in humans was carried out in compliance with the Helsinki Declaration. In one experiment, 66 arrays were processed at the University of California at Davis (UC Davis) in six batches. Fifty-one arrays probed independent samples from teenagers with Duchenne Muscular Dystrophy ('patients'). The other 15 arrays probed an identical reference sample generated at UC Davis which consists of pooled RNA isolated from four adults ('reference'). In five of the batches processed at UC Davis one reference array was included, and the sixth batch is composed entirely of reference arrays. In a separate experiment conducted at the University of Cincinnati, there were 32 arrays processed in four separate batches with each batch including between seven and nine arrays. 28 out of the 32 arrays probed samples from healthy teenagers ('controls'). Four arrays (one per batch) probed the same reference sample used at UC Davis.). Pre-processing involved non-linear background reduction utilizing probe DNA sequences, quantile normalization, and summarization by median polishing [Sample processing and array hybridization were performed according to standard Affymetrix protocols. Probe level data were summarized into a single expression value for each gene on each array using GCRMA in GENESPRING 7  of the adjustment for this batch and is assumed to follow a normal distribution. δig measures the multiplicative batch effect, which is assumed to follow an inverse gamma distribution. The error terms, εijg, are assumed to follow a normal distribution with expected value of zero and variance σg2.where We employ three versions of the above model to derive three different sets of empirical Bayes adjusted data: one adjusted for both within and between experiment non-biological variation, a second to adjust for only the variation that occurs between experiments, and a third to adjust only for the within experiment batch effects. We evaluate these three different formulations in order to assess and compare the relative effects of adjusting for the within and between experiment non-biological variation.g has parameters γig and δig for each of the ten distinct batches. There are a total of three regression coefficients in the vector of parameters for the biological group covariates, βg, one that specifies whether a sample is the reference, and two that specify disease state .Model 1: within and between experiment effects: In Equation 1 gene γig and δig per gene, one for each experiment. In this model, each experiment is considered as one batch and batch effects within each experiment are not modeled.Model 2: between experiment effects only: The model formulation is the same as above except that there are only two values for βg.Model 3: within experiment effects only: For the case where the data is not adjusted for between-experiment batch effects, the reference sample is not used and so there are only two coefficients in the vector αg, βg, and γig for each gene by least squares constraining The model parameters are estimated as detailed by Johnson et al . This prN is the total number of samples. Then, the standardized data iswhere i sample mean is estimated asIn the second step, for each gene, the batch i sample variance asand the batch γig and δig using the method of moments. Then, using the parameter estimates for these two prior distributions along with Bayes theorem, posterior distributions for each γig and δig are calculated. Final estimates for the batch effect parameters, γ*ig and δ*ig, are estimated as the expected values of the posterior distributions. This empirical Bayes estimation procedure allows information from all genes to be used to estimate batch effects for each gene, providing more stable estimates than the standard sample mean and sample variance.These estimates are used to estimate the parameters of the distributions of In the third step, the data is adjusted to remove batch effects. The empirical Bayes  batch ad. This software features diagnostic plots that allow one to check the validity of the model assumptions as well as an alternate nonparametric Bayesian method .We performed the above calculations using the COMBAT software developed by Johnson et al and written in R We performed unpaired two-sample t-tests in GENESPRING 7 to find genes with different mean expression between disease groups. First, subjects with muscular dystrophy were compared to the control subjects using unadjusted GCRMA-summarized gene expression data. The same t-test was repeated using the various empirical Bayes batch adjusted gene expression values. P-values were corrected for mulitple comparisons with the Benjamini-Hochberg False Discovery Rate (FDR) technique. An FDR corrected p-value ≤ 0.05 was considered significant, which means that no more than 5% of the declared different genes are expected to represent false positives. We further filtered each of the above lists removing all genes with average fold change < 2.0.Differentially expressed genes between batches within each experiment were identified using one-way ANOVA within GENESPRING with an FDR corrected p-value threshold of 0.05.We compare the proposed empirical Bayes approach to a simple method that filters out genes with significant cross-experimental batch effects based on a t-test comparing the 15 reference arrays from UC Davis to the four reference arrays processed at the University of Cincinnati. This t-test filter was applied to each gene that was differentially expressed between disease groups in the GCRMA-summarized unadjusted gene expression data set. Each gene that was found to be significantly different between experiments  was removed from the gene list because of the ambiguity regarding the reason for its differential expression . The loose p-value threshold of 0.2 for removing genes was chosen because of the limited accuracy of the t-test with one of the sample sizes only being four.We performed two simulations to assess the ability of the empirical Bayes method to accurately identify differentially expressed genes in the presence of confounding batch effects. In one simulation we tested the ability of the method to recover false negatives – genes whose differential expression is reduced due to batch effects. For the second simulated data set, we tested the ability of the method to remove false positives – genes whose differential expression is elevated due to batch effects.αg and βg to generate a set of genes with mean fold changes between the disease and healthy samples that range between 1.1 and 3.0 in increments of 0.1. We simulated 60 genes with each fold change to generate (1) 600 genes with fold changes greater than 2.0 , and (2) 600 additional genes with fold changes below 2.0, the threshold we selected for differentially expressed genes. Parameter values for αg and βg for the remaining 53,475 genes were selected so that there are no differences in mean expression between the two groups. The errors, εijg, for each gene were randomly simulated from a normal distribution with mean 0 and standard deviation σg, chosen to reflect the average standard deviation observed in the real data. Batch scale parameters, δig, were also selected to reflect a typical range of observed values.For each simulated data set, we generated gene expression values for 54,675 probe sets for 28 controls, 51 patients, and 19 identical reference samples. Model parameter values were selected to reflect trends observed in the muscular dystrophy data set. Specifically, we selected parameter values for γig values whose mean for the patients is lower than that for the controls by an amount that counter balances the average difference in mean expression between patients and controls for truly differentially expressed genes. For the second simulation we selected γig values so that the mean for the patients is higher than that for the controls by an amount that generates a difference in expression between patients and controls for truly non-differentially expressed genes equivalent to that seen in truly differentially expressed genes. In each simulation, we identify genes declared to be differentially expressed before and after batch adjustment and compare the numbers of genes that are correctly and incorrectly classified as being differentially expressed in each case.For the first simulation, we selected FDR: False Discovery rate; RNA: ribonucleic acid; GCRMA: GC Robust Multiarray Average; ANOVA: analysis of variance.WLW and FRS were responsible for designing the study and devising the batch correction approach presented here. KSP and CEM made suggestions about the modeling and comparison of the Bayes and t-test approaches. BW and DLG recruited the control and Duchenne Muscular Dystrophy patients that were used for the application of the methods developed here. IL and LL aided in writing the manuscript and evaluating the proposed methods. All of the authors read the manuscript and made suggestions for improvements.