With the popularity of DNA microarray technology, multiple groups of researchers have studied the gene expression of similar biological conditions. Different methods have been developed to integrate the results from various microarray studies, though most of them rely on distributional assumptions, such as the t-statistic based, mixed-effects model, or Bayesian model methods. However, often the sample size for each individual microarray experiment is small. Therefore, in this paper we present a non-parametric meta-analysis approach for combining data from independent microarray studies, and illustrate its application on two independent Affymetrix GeneChip studies that compared the gene expression of biopsies from kidney transplant recipients with chronic allograft nephropathy (CAN) to those with normal functioning allograft.The simulation study comparing the non-parametric meta-analysis approach to a commonly used t-statistic based approach shows that the non-parametric approach has better sensitivity and specificity. For the application on the two CAN studies, we identified 309 distinct genes that expressed differently in CAN. By applying Fisher's exact test to identify enriched KEGG pathways among those genes called differentially expressed, we found 6 KEGG pathways to be over-represented among the identified genes. We used the expression measurements of the identified genes as predictors to predict the class labels for 6 additional biopsy samples, and the predicted results all conformed to their pathologist diagnosed class labels.We present a new approach for combining data from multiple independent microarray studies. This approach is non-parametric and does not rely on any distributional assumptions. The rationale behind the approach is logically intuitive and can be easily understood by researchers not having advanced training in statistics. Some of the identified genes and pathways have been reported to be relevant to renal diseases. Further study on the identified genes and pathways may lead to better understanding of CAN at the molecular level. DNA microarray technology was launched in the early 90's. With its development and commercialization, it has been a popular tool for researchers to perform genome-wide analysis of gene expression profiles. One majMeta-analysis is the quantitative synthesis of a number of study results. A few groups of researchers have developed meta-analytic methods for combining results from multiple DNA microarray experiments -6. The bA statistical challenge in performing a meta-analysis of microarray studies is that often samples are hybridized to different microarray platforms, and the technical differences among platforms lead to fundamental differences in the nature of the gene expression measurements produced. For example, the data for a custom spotted cDNA array are usually expressed as ratios of the intensity values corresponding to an experimental sample to the intensity values of a co-hybridized reference sample; while the data from an Affymetrix high density oligonucleotide microarray are absolute intensity values for the single channel. Therefore, data from different microarray platforms are not directly comparable, and it is essential to use the original values of gene expression from each platform to derive an "effect size" estimate that is independent of platform, thus rendering the different platforms comparable. The term "effect size" commonly used in meta-analysis refers to a standardized index measuring the effect associated with a treatment or covariate, or the magnitude of difference in gene expression in microarray studies . Anotherg  and study i , they fit a simple linear regression model with tissue type as the covariate and expression measurement as the response. Then the ordinary least square estimate of the covariate coefficient divided by its standard error was used as the effect size measurement, which is equivalent to a t-statistic. Thus these gene-level effect size measurements are comparable among all the individual studies. To integrate these t-statistics across studies, a weighted average of t-statistics was calculated to obtain a global statistic for differential expression of each gene g, whereas the proportions of the sample sizes in study i to the overall sample size were used as the weights. To identify genes that were truly differentially expressed between the cancer and healthy tissues, the authors used permutation method by fitting linear models with permuted tissue labels to calculate the false discovery rate (FDR). Genes were declared differentially expressed corresponding to a specified FDR.To address these challenges, different effect size measurements and models for integrating microarray data have been proposed. In an early study , the autpg, i calculated from the permutation t-test for each gene g in each study i to serve as the effect size measurement. To integrate across all the studies, Fisher's method for combining p-values, g did not have differential expression between the two groups, Sg is chi-square distributed with degrees of freedom 2·I. Then the p-value for gene g based on the integral analysis of all the datasets can be calculated using the χ2df = 2I-distribution. Controlling the FDR at a certain level, differentially expressed genes could be identified as those with p-values less than a threshold determined by the FDR level.The same four prostate cancer microarray datasets were analyzed using an alternative method by Rhodes et al. . They usg in each study i, and the study effect was treated as a random effect. A z-statistic zg, i, the effect size measurement, was calculated to be the ratio of the estimated mean expression differences to its standard error. To integrate across studies, the average z-statistic Choi et al.  used a mg in each study i , they simply pooled the data from all I studies into one dataset and used univariate logistic regression technique to quantify genes relevance to breast cancer.Shen et al.  also useNevertheless, to some degree, all these methods rely on the adherence of the data to a specified parametric distribution, such as the Gaussian distribution for the t-statistic methods and the mixed-effects model, or the different forms of prior distributions in the Bayesian context. However, commonly within each individual microarray study, only a small sample size is available, thus the normality assumption may not hold well. Further, it may be even more difficult to test the validity of the assumptions on the prior distributions and their parameters for Bayesian models. Some of the aforementioned methods are also somewhat computationally complicated and might be difficult to be understood by clinical researchers not having advanced training in statistics.We obtained the data from two independent microarray experiments that compared the gene expression profiles between chronic allograft nephropathy (CAN) and normal functioning kidney allograft. Chronic Allograft Nephropathy is a major cause of graft loss and patient morbidity after kidney transplantation. The histopathology features of CAN are nonspecific and this makes it difficult to detect CAN before the occurrence of clinical manifestations. However, it is now well known that CAN may already be present in protocol biopsies before its clinical appearance . It may In this study, we sought to integrate the results from these two independent DNA microarray experiments. Herein we present a non-parametric approach for combining microarray data from various studies which does not suffer from the aforementioned limitations. This new method does not require any distributional assumption on the gene expression measurements, is logically intuitive, and is also easy to implement using statistical software. We will also report some of the biological findings from its application to integrate the two microarray studies.The Hotchkiss study used Affymetrix HG-U133A human GeneChip arrays to measure the gene expression of 16 biopsies from patients with CAN and 6 biopsies from patients with normal functioning allograft . The datUsing our proposed meta-analysis method on the two microarray datasets, we identified 330 probe sets representing 309 genes that were significantly relevant to CAN. Associated with each gene was a score that measured this gene's ability to discriminate between CAN and normal allograft. The lower the score value, the better discriminative ability the gene had. The definition of the score and the procedure to obtain it are elaborated in the Methods section. Table PVALB  has the most outstanding discriminative ability, or in other words, it has very distinct expression patterns between CAN and normal allograft, as shown in Figure The gene The Significant Analysis of Microarray (SAM) method  was applThe discordance among these results demonstrates that independent microarray experiments, even they use the same platform and study the same biological condition, may not be reproducible ,19. It aITGAE, the fifty third in the identified gene list. In Figure ITGAE expression values in the two studies are presented. It is clear in both plots that the expression patterns are different between the two groups, with the expression of normal allograft samples generally being lower than that of CAN samples. However, because of the presence of several normal samples that have relatively high expression, the standardized means are not significantly different between the two groups. As in reality, even if a gene has been biologically validated to be up-regulated (or down-regulated) in a disease, relatively low (or high) expression may still be observed in a few samples. The two-sided t-test for ITGAE in the Hotchkiss study yields a p-value 0.20, and 0.14 in the Mas study. Therefore, models based on t-statistic will not recognize this gene as having differential patterns between the two groups, although the expression patterns are indeed different enough to differentiate the samples.A unique advantage of our meta-analysis method is that it can discern the differential gene expression patterns between CAN and normal allograft that cannot easily be described by some measurement of standardized mean difference in expression values, such as the t-statistic, which is the basis of the SAM method. An example is illustrated using the expression values of probe set "205055_at" in both studies. This is gene α = 0.01, 2026 probe sets are identified to have significantly differential expression between the normal allograft and CAN samples. The majority of the 330 probe sets identified by our non-parametric meta-analysis are also identified by this parametric meta-analysis; however 73 out of the 330 probe sets are not identified by the parametric analysis, including the above mentioned probe set ''205055_at'', which has an adjusted p-value 0.41. Most of the 73 probe sets indeed have the similar differential expression pattern as seen in Figure We also performed a meta-analysis on the two CAN studies using a commonly used parametric approach that is based on t-statistic, and compared the result to that from our non-parametric meta-analysis approach. The t-statistic based method is derived from  and is dThe meta-analysis was performed on the individual gene level and assigned a score for each gene that quantified its difference in expression between CAN and normal allograft. However, genes do not express independently, especially when they are components of the same biological pathway. Therefore, we also examined the identified genes at the pathway level.α = 0.01 to test the null hypothesis of no difference against the alternative that significantly more genes were represented in the list of the identified genes of that pathway than would be expected by chance. Six pathways were found to be significantly over-represented. Table Among the 330 identified genes using our non-parametric approach, 129 have been annotated in KEGG (Kyoto Encyclopedia of Genes and Genomes) pathway database, and they are components of 114 pathways. For each pathway we used Fisher's exact test with As an illustration, we examined the most significantly over-represented pathway in the literature. Oxidative phosphorylation is a process of cellular respiration and consisted of five complexes located in the inner mitochondrial membrane. ATP synthesis is the final protein complex in the metabolic pathway. It has been reported that mitochondrial disorders can sometimes give rise to kidney dysfunction . The finPredictions were performed on 6 additional transplant kidney biopsies that were procured and hybridized to Affymetrix HG-U133A 2.0 GeneChip arrays from Mas et al. after their previous study. These 6 samples were all diagnosed as CAN by experienced pathologists based on the histological observations. Each identified gene was used as the single predictor to predict the class labels of the 6 samples, using classifiers derived from the Hotchkiss data and Mas data respectively. The weighted average of the error rates associated with each identified gene is recorded in the last column of Additional File When using the classifiers derived from Hotchkiss data and Mas data respectively with all the 330 identified genes expression measurements as predictors, the predicted classes all conformed to their true diagnostic classes.G = 2000 genes in each of two studies, with a percent p = 10% of genes to be truly differentially expressed between the disease group and normal group. The sample sizes were nd 1, = 4 or 8, nnorm 1, = 6 or 10 in study I for the disease and normal groups respectively; similarly, the samples sizes were nd 2, = 6 or 10, nnorm 2, = 15 or 15 in study II for the two groups respectively. These small samples sizes were used to mimic the CAN meta-analysis. To identify differentially expressed genes, the non-parametric meta-analysis approach and permutation t-test based approach were then applied on the simulated datasets. We note that because of the small sample sizes in the disease group, mixture-effects model based meta-analysis approach is not likely to be appropriate to use. The performance of the applied two approaches was evaluated by two statistics: sensitivity ; and specificity .We conducted a simulation study to investigate the generalizability of our non-parametric meta-analysis approach. Two datasets were generated independently to simulate the RMA expression summary data observed from the two Affymetrix GeneChip experiments in which CAN was studied. We simulated expression values for The intensity of a gene was generated using the following model:yg, i, j is the intensity for gene g in study i = 1, 2 and group j = 1  and 2 (disease); I is the indicator function; μg is the overall gene g effect; βi)g g g is the random error term nested within group, allowing different variability in different groups. The procedure used in the generation of the datasets is outlined in the following steps:where μ = ' ~ multivariate N, where μ0 = , where μ0~Uniform  with probability 0.9 and Uniform with probability 0.1; and Σ~[(n1 - 1)·Σ1 + (n2 - 1)·Σ2]/(n1 + n2 - 2), where Σ1 and Σ1 are the sample variance-covariance matrices of 2000 randomly selected genes in the Hotchkiss study and Mas study, respectively. Thus the correlation structure in the genes is preserved in the simulated data.1). Generate the gene effects vector βi)g, where μβ~Uniform  and σβ2~Uniform.2). For samples in Study II, generate the study effect for each gene: p = 10% of the genes to be truly differentially expressed genes. Additionally randomly select some samples to represent the diseased group. Generate the group effect for each of these genes in these disease samples: γj)g, where μγ~Uniform with probability 0.4, Uniform with probability 0.1, Uniform with probability 0.4, and Uniform with probability 0.1; σγ2~Uniform.3). Randomly select εi(j)g~N.4). Generate the random error for each gene in each sample: G = 2000 genes truly have non-differential expression.We ran 30 simulations and the means and standard deviations (SD) of the sensitivity and specificity statistics using the two approaches are reported in Table The two microarray experiments analyzed in this meta-analysis were both performed using Affymetrix platform, whereas the Hotchkiss study utilized the HG-U133A arrays and Mas et al. used HG-U133A 2.0 (version 2) arrays. Other than some differences in the control probe sets, the probe set IDs are identical between the two versions of HG-U133A arrays. Therefore, gene mapping between datasets was not a challenging step.Hu et al.  developeThe meta-analysis method we proposed is applicable to situations where multiple microarry platforms are involved. However, if the data are from the same platform, the same normalization method should be used. The Tumor Analysis Best Practices Working Group compared different probe set expression summary algorithms for Affymetrix GeneChip arrays and claimed "different probe set interpretation algorithms lead to different results" . They ofIt is well known that genes, especially genes in a common pathway, are correlated. We considered starting the meta-analysis from a pathway level, i.e. first identifying pathways that might be relevant to the progress of CAN; and then focusing on the individual genes in those pathways and finding out genes whose expression patterns were differential between CAN and normal allograft. However, since only less than 30% of the genes measured on the Affymetrix HG-U133A chips have been annotated with known KEGG pathway information, we decided to perform the analysis at the gene level. This may help researchers understand gene functions that are still unknown and avoid throwing away 70% of the available data.Chronic allograft nephropathy is a complex entity at both histological and molecular level. We identified 330 sequences whose differential expression patterns could distinguish between CAN and normal allograft. The functions of most of the identified genes are not well understood yet. More studies on these genes, especially on those at the top of the list, may lead to a better understanding of the progression of CAN at the molecular level. Furthermore, each gene is associated with a score that measures its degree of differential expression between the two groups. All the identified genes have scores below the pre-determined threshold 0.1737. By adjusting the threshold based on prior expertise knowledge about CAN, more or less genes can be identified for further study. To utilize a smaller set of genes for prognosis on kidney transplant recipients, the genes with the lowest score values can be selected, such as the 10 listed in Table In this paper, we present a new meta-analysis technique for combining DNA microarray studies by analyzing two independent microarray studies comparing the gene expression of CAN and normal allograft. This is a non-parametric approach that is statistically easy to understand, and can discern differential expression pattern that may not be detected by t-statistic based models and mixture-effects model. Although the new method is applied to combine two microarray studies of the same platform, its' use is by no means limited to a single platform and can be used to different platforms without difficulty.GAPDH, ISGF and β-actin. For both studies, all the ratios were less than 3, the threshold recommended by Affymetrix. Therefore, sample degradation did not seem to be a problem in both studies, and thus we regarded all samples as useful. Before combining the data across the two independent studies, all Affymetrix control probe sets were excluded, leaving 22,215 probe sets that were common to both studies.The probe level data from the two independent Affymetrix microarray studies were normalized and summarized using RMA method. To assess the sample quality, we calculated the 3' :5' ratios for three Affymetrix control probe sets corresponding to human genes We considered finding in each individual study for each gene an effect size measure that could quantify the gene's ability in discriminating CAN from normal allograft. This was essentially determined by the degree of difference in the gene's expression between the two groups, and reflected as how well its expression measurements could classify the samples. Conceptually, if a gene is irrelevant to CAN, using its expression to determine class membership would appear as random guessing. On the other hand, if a gene is an important predictor in distinguishing class, i.e. it has different expression patterns between CAN and normal allograft, we will expect to correctly classify most of the samples using its expression, and the misclassification error rate will be close to 0.i  we used the expression for each gene g  as the single predictor variable and applied the K-Nearest-Neighbor (KNN) classification method to develop a classifier for the ni samples in study i. Thereafter, the KNN classifier was used to predict class label for each sample. The predicted labels were compared with their corresponding true class labels and an unbiased estimate of the misclassification error rate, denoted as errg, i, was calculated. This error rate estimate measured this gene's discriminative ability and is defined as our effect size statistic.Therefore, within study Using KNN, we estimate directly at each observation the posterior probability of each class, given the observed predictor (gene expression), as the proportion of that class among the k nearest "neighbors" of the target observation. Then the classification for the target observation is the class which had the largest estimated posterior probability. The advantages of KNN include that it does not require any distributional assumptions, and it has reasonable performance comparing to other classification methods. A property of the large-sample behavior of KNN is described in the following theorem :E* denote the error rate of the Bayes rule in a C-class problem, i.e. the best possible error rate for the classification problem. Then the error rate of KNN converges in L1 as the size of the training set increases to a value E1 bounded below by E* and above by THEOREM: Let E* = E(min(P(class I|x), P(class II|x))) ≤ 0.5, where x is the vector of predictors. Thus the asymptotic upper bound of the error rate of KNN is E* × (2 - 2E*) ≤ 0.5, which means that KNN has asymptotic performance as good as the performance of the Bayes rule. When the sample size is small, as is often the case in microarray studies, KNN is also suitable to use as Ripley [For a two-class problem, s Ripley  indicatek (the number of neighbors) to avoid ties, and chose k = 3 considering in the Mas study, only 4 samples were available in the normal allograft group. For microarray studies with larger sample sizes, k can be determined using cross-validation and can be different for the individual studies. Since the data from both studies were summarized using RMA method and thus the scales of the data are similar, the Euclidean distance was used to quantify the similarity in the predictor gene's expression profile between samples and determine the ''nearest neighbors'' for each sample.We used an odd value of g in each individual study i, we generated B = 100 bootstrap resamples. For each bootstrap resample, we used the bootstrap sampled gene's expression measurements as predictor values and applied the KNN to develop the classifier. The classifier was used to predict the class labels for the bootstrap samples, as well as the original samples, respectively. If Rboot,(b), g, i denotes the misclassification error rate in the bth bootstrap samples, and Rori,(b), g, i denotes the misclassification rate in the original samples using the classifier built from the bth bootstrap samples; then errg, i is the apparent error rate in the original samples, which uses the classifier built from the original samples themselves, plus the optimism estimate from bootstrap samplings, i.e., errg, i = Rg, i + Ropt, g, i, where Rg, i denotes the apparent error rate and g = 1, 2,..., 22215; i = 1, 2.The apparent misclassification error rate, which is the number of misclassified observations in the training dataset divided by the total number of samples in the training dataset, tends to under-estimate the true misclassification error rate . TherefoIt is noteworthy to notice that this step can be carried out on microarray datasets from any platforms. Although different platforms may yield distinct scales of numerical measurements of gene expression, as long as there exists a relative different expression pattern between the two classes, the discrimination method can be applied to quantify each gene's association with the class label.errg, i) for each gene g, we calculated the weighted average across studies as the combined effect size (or score) of this gene. The weight corresponding to each study was taken to be the proportion of its sample size in the total combined sample size. The combined effect size for gene g across studies is After we obtained the study-specific effect size measurements  ≤ α. The Q-Q plot of N, the threshold T is the quantile such that P) = α, and estimating T by plugging in the moment-based estimates for μ and σ2. This is illustrated graphically in Figure α to identify either a larger or smaller number of genes.To identify genes capable of distinguishing between CAN and normal allograft, we wanted to identify the The entire meta-analysis method is summarized in Figure The analysis was conducted in the R 2.4.0 environment on a PC with Intel Core Duo CPU@2.0G × 2 and 2.0G RAM. The average calculation time needed for each gene is about 0.9 second. More efficient and faster performance of the algorithm can be realized when the implementation in the C language is available. Our R code for analyzing the two CAN studies can be found in Additional File G0. Among the remaining probe sets, we let GID denote the number of probe sets that were identified to be relevant to CAN, and let Gp 0, denote the number of probe sets that were components of a pathway p. Among the GID identified probe sets, GID, p probe sets were in the pathway p. Then we performed a Fisher's exact test [p was over-represented in the identified genes at a significant level 0.01.To identify the over-represented pathways associated with the identified genes, we first filtered the whole gene list by excluding probe sets that had not been annotated in KEGG pathway database, and denoted the number of remained probe sets as act test  on the 2PVALB was significantly differentially expressed between CAN and normal allograft in both studies; however, the measurements on CAN samples in the Hotchkiss study were in the range of , while in the Mas study they were in the range of . Therefore, a global shift on the RMA normalized measurements existed between the two independent studies. Because the six unpublished samples may have been processed by different technicians, the shift on the measurements may also be present between these samples and the earlier two studies.To predict the class labels of the 6 additional samples, we first normalized and summarized the *.CEL files using RMA method. It was noticed that although a gene's differential expression pattern might be similar in both studies, the numerical values between the studies took on different ranges. As can be seen in Figure In order to diminish the impact of the global shift and make the classifier derived from the previous two studies applicable to these unpublished samples, we centered the measurements of each identified gene by subtracting its median in each individual dataset. Next the KNN algorithm was run to build classifiers respectively with the centered data in the Hotchkiss study and Mas study. Then the two classifiers were applied respectively to classify the 6 samples and the predicted results were compared to their true classes.XK developed the method, performed the statistical analysis, wrote programming code, and drafted the manuscript. VM provided data, contributed to the scientific understanding of the clinical implications of CAN in renal transplant recipients, and contributed to revising the manuscript. KJA conceived the study and provided extensive and detailed comments on the statistical analysis and on the manuscript. All authors read and approved the final manuscript.The complete list of the identified 330 probe sets that were significantly relevant to chronic allograft nephropathy.Click here for fileThe R code to perform our non-parametric meta-analysis on the two CAN microarray studies.Click here for file


The increasing number of gene expression microarray studies represents an important resource in biomedical research. As a result, gene expression based diagnosis has entered clinical practice for patient stratification in breast cancer. However, the integration and combined analysis of microarray studies remains still a challenge. We assessed the potential benefit of data integration on the classification accuracy and systematically evaluated the generalization performance of selected methods on four breast cancer studies comprising almost 1000 independent samples. To this end, we introduced an evaluation framework which aims to establish good statistical practice and a graphical way to monitor differences. The classification goal was to correctly predict estrogen receptor status (negative/positive) and histological grade (low/high) of each tumor sample in an independent study which was not used for the training. For the classification we chose support vector machines (SVM), predictive analysis of microarrays (PAM), random forest (RF) and k-top scoring pairs (kTSP). Guided by considerations relevant for classification across studies we developed a generalization of kTSP which we evaluated in addition. Our derived version (DV) aims to improve the robustness of the intrinsic invariance of kTSP with respect to technologies and preprocessing.For each individual study the generalization error was benchmarked via complete cross-validation and was found to be similar for all classification methods. The misclassification rates were substantially higher in classification across studies, when each single study was used as an independent test set while all remaining studies were combined for the training of the classifier. However, with increasing number of independent microarray studies used in the training, the overall classification performance improved. DV performed better than the average and showed slightly less variance. In particular, the better predictive results of DV in across platform classification indicate higher robustness of the classifier when trained on single channel data and applied to gene expression ratios.We present a systematic evaluation of strategies for the integration of independent microarray studies in a classification task. Our findings in across studies classification may guide further research aiming on the construction of more robust and reliable methods for stratification and diagnosis in clinical practice. Transcriptional profiling studies of human diseases aim to identify causal molecular mechanisms as well as to improve diagnosis. For example, in breast cancer several prognostic gene signatures have been proposed -4. To thHere, we establish a systematic approach to assess the performance of the integration of independent gene expression microarray data sets for classification across studies Figure , proposeThe limited sample size in microarray studies always raises the question whether the sample under investigation is representative of the population. This is a major concern for any prediction task, and therefore it is highly desireable to consider all available data from related studies. Still then, it is unclear whether a meta-analysis increases the performance of a classifier, since technical and experimental settings vary between studies and introduce an additional layer of variation which might undo the benefits of a larger sample size. Therefore, any arbitrary integration of available data may not be the best choice. Nonetheless, an adequate strategy for the integration of heterogenous data sets may ideally help to identify those signals or genes which are not only most relevant for the classification task but also least variable with respect to the underlying experimental and technical differences. This helps to construct more robust gene signatures and ultimately leads to classifiers of increased predictive power.Experimental and technical differences of microarray gene expression studies hamper data integration. Gene expression microarray platforms are based upon the principle of hybridization. The most important difference between platforms are the sequences used to measure the mRNA of a given gene. Not only the length of the sequence varies but also the choice of the most representative sequence for any particular gene. Moreover, the choice of the sequence itself influences the intensity of the signal independent of the given mRNA abundance. For example, the number of physical positions which can bind a fluorescent dye varies. Besides that, the total of all sequences on the microarray as well as the experimental processing influence the cross-hybridization properties. Thus, it is not only difficult to decide which measurements respectively sequences correspond to the same mRNA, but even in the case of matching sequences the measurement characteristics still vary. For example, the offset, scale, dynamic range and behavior as well as the level of noise differs. Another level of complexity arises from the use of two-channel/two-color platforms and single channel platforms. The two-channel platforms mostly quantify all measurements relative to a common reference sample. As a consequence the offset of the measurement is completely dependent on the choice of the reference. Nonetheless, the influence of the reference approximately cancels out if one focuses on differences in mRNA abundance between classes. Here, we disregard all issues of sequence matching including exons, introns, splicing variants etc. and rely solely on the gene annotation of probes or probe sets. The gene symbol serves as basis for the comparison of annotated probes across platforms. Note that any analysis will be hampered by different, technology-specific probe affinities for the same gene, by measurement failures as well as by wrongly annotated probes.The major issue when focusing on the experimental and technical aspect of classification across studies can be summarized as follows: different gene expression profiling platforms or studies measure the expression of the same common gene with different precision and on a different scale. Nonetheless, a common way to represent the gene expression measurements does not only allow to directly combine microarray data sets, but also to readily apply the generated classifier on a new data set which is represented in the same manner. To this end, ,11 propoWe systematically evaluated the generalization performance of five selected methods SVM, PAM, RF, kTSP and DV on four breast cancer gene expression microarray studies almost comprising 1000 independent samples .First of all, we benchmarked the studies while using a complete cross-validation approach to estimate the misclassification rate. This was done separately for each study, classification method and task Figures , 3A. OveThe major classification results are visualized in Figure FOXA1, GATA3 and SLC39A6 which have been previously reported in a ten gene signature [FOXA1, GATA3 and ESR1 overlap with the ten gene signature. All three final classifiers contain several genes which are well known to be estrogen responders like AGR2, STC2, TFF1 or XBP1 . These findings confirm the relevance and predictivity of the gene signatures. The classifiers obtained by the methods SVM and RV contain 1000 genes since no further feature selection was applied. The final classifier for histological grade upon the DV and kTSP method was derived after the integration of three studies which contain grading annotation  leads to reduced cell migration and regulated various angiogenesis linked proteins in breast cancer cell lines [CENPA, GJA1, HMGB3, KIF13B, KIF2C, NAT1, QDPR, SCUBE2, STC2, UBE2C. This concordance between the signatures for histological grade across different methods confirms their relevance. Moreover, the observed partial overlap of the predictive signatures derived for estrogen receptor status and grade may relate to the aforementioned association of estrogen receptor status negative tumors and high grade. For example, in case of DV and kTSP the intersection of the corresponding signatures comprises the three genes NAT1, STC2 and SCUBE2 .The final classifier for estrogen receptor status which we obtained when integrating all four independent studies comprises 50 genes for the methods DV and kTSP  of kTSP, we tradeoff between robustness and performance by introducing an upper bound F that limits the number of classifiers that contain a common gene. Our derived version selects gene pairs by starting with the highest scoring gene pair and then successively adjoining the next highest scoring gene pair for which none of its two constituents is already contained in F previously selected gene pairs. Note that the parameter choice F = 1 results in the original kTSP method when only considering all pairs of genes in the set of two times k genes which have been selected by the kTSP classifier.The kTSP (k-top scoring pairs) method is a binary classifier which is composed of qgs), where qgs denote the relative rank of the expression value of gene g in sample s among all expression values of sample s . Let cs ∈ {± 1} be the class of sample s. Let nc be the number of samples in class c. The essential building block compares the expression values of two genes i, j and is given by the score |Δ ij|,More formally, let the measurements be given as a genes × samples matrix . The misclassification rate was estimated with cross-validation in each study separately. The average and the standard deviation across the four studies is visualized.Click here for fileAverage misclassification rate across studies. The figure displays the average misclassification rate on an independent breast cancer study for five classification methods. . The average is calculated across the different studies. The misclassification rate for each study itself is the average rate of all classifiers in which the study was not used in the training. The results are shown separately with respect to the number of contributing studies which formed the training set. The dotted lines indicate averages across all classification methods and visualize a tendency of a decreasing error rate with an increasing number of studies which were used for the training of the classifier.Click here for fileClassification results across studies for estrogen receptor status. The figure summarizes all classification results for estrogen receptor status. Samples correspond to rows and methods to columns. The estimates of the cross-validation approach are shown on the left separated by a vertical line from the results of the classification across studies on the right. For the latter the samples of the studies used for the training are marked in gray and the ones not used are shown in white. The cross-validation approach was run separately for each study. Misclassified samples are labelled in red and correctly classified ones in light yellow. The error estimates of the repeated cross-validation have been mapped to the range from red to light yellow. Samples are ordered by study, class, their average misclassification rate in the cross-validation and classification across studies. The color code at the bottom indicates the method , at the left the study , at the right the class .Click here for fileClassification results across studies for histological grade. The figure summarizes all classification results for the histological grade. Samples correspond to rows and methods to columns. The estimates of the cross-validation approach are shown on the left separated by a vertical line from the results of the classification across studies on the right. For the latter the samples of the studies used for the training are marked in gray and the ones not used are shown in white. The cross-validation approach was run separately for each study. Misclassified samples are labelled in red and correctly classified ones in light yellow. The error estimates of the repeated cross-validation have been mapped to the range from red to light yellow. Samples are ordered by study, class, their average misclassification rate in the cross-validation and classification across studies. The color code at the bottom indicates the method , at the left the study , at the right the histological grade . Study 4 is not included since the histological grade of the samples was not available.Click here for fileGene signatures. Gene signatures for the prediction of estrogen receptor status and histological grade derived from the complete data set.Click here for fileDescription of DV. Algorithmic description of DV.Click here for file


Here, we present data on 204 analyses from an international standardization programme that was conducted in 11 laboratories as a prephase to the Microarray Innovations in LEukemia (MILE) study. Each laboratory prepared two cell line samples, together with three replicate leukaemia patient lysates in two distinct stages: (i) a 5-d course of protocol training, and (ii) independent proficiency testing. Unsupervised, supervised, and r Several microarray studies have already demonstrated the identification of differentially expressed genes associated with distinct clinical and therapeutically relevant classes of leukaemias (http://www.leukemia-net.org) in 11 laboratories: seven from the ELN, three from the United States, and one in Singapore. The so-called Microarray Innovations in LEukemia (MILE) study programme will prospectively assess the clinical accuracy of gene expression profiles of 16 acute and chronic leukaemia subclasses, of myelodysplastic syndromes (MDS), and a “none of the target classes” control group, as compared to current routine diagnostic workup in over 3000 patients. As a first step representing a major effort to standardize the microarray analysis workflow in the participating centres, a prephase of the MILE study was performed. This report presents the results of the prephase, i.e., a standardization programme of the microarray procedure in the participating laboratories in order to ensure a robust gene expression profiling test performance before patient samples were analysed.Here we report data from an international study group formed around the European Leukemia Network  and HepG2 (liver carcinoma) cell line total RNA  with 1·0 μg and 5·0 μg input of total RNA, respectively, and four leukaemia patient sample lysates prepared from mononuclear cells obtained after Ficoll density purification. Patient lysates comprised cells of one chronic myeloid leukaemia (CML), one chronic lymphocytic leukaemia (CLL), and two replicate lysates of an AML patient sample , French-American-British (FAB) type M2). The total RNA from the patient lysates was extracted at each centre as part of the training programme, making these samples a test of the entire microarray process workflow post sample acquisition . Subsequently, after the training phase and for operator proficiency testing, each laboratory independently performed four microarray experiments each for MCF-7 and HepG2 cell lines with inputs of 1·5 μg, 3·0 μg, 5·0 μg, and 8·0 μg total RNA. In total, 204 microarray profiles were included in the analysis , the individual gene expression profiles grouped closely together with their corresponding biological sample types based on the underlying similarity, but not according to the centre where the microarray experiments were performed . The arrpe HepG2 , where mAs an example of inter-laboratory reproducibility of gene expression analyses, correlations between Centre 3 and all other ten laboratories are given . The degTaken together, this study demonstrated that standardizing experimental protocols for microarray analysis and performing a thorough operator training resulted in excellent comparability with respect to both data sets generated within a participating laboratory and across 11 different laboratories in three continents. This extends the observations of a recent across-platform comparison study from the Toxicogenomics Research Consortium (In conclusion, this standardization effort represented the prerequisite foundation of the first phase of the MILE study, wherein 1889 patients have, thus far, been analysed by whole genome expression microarrays (AK, LW, TH: design of the study and drafting the article; RL, WML, PMW: statistical analysis and interpretation of data; TJK, LZR, JRD, SAS, KIM, AFG, WKH, GB, MCDO, RF, SC, JDV, SR, PRP, JMH, EL, AEY, ESK: data acquisition, interpretation of data, and article revision. All authors approved the final version submitted for publication.


Discrete classification is common in Genomic Signal Processing applications, in particular in classification of discretized gene expression data, and in discrete gene expression prediction and the inference of boolean genomic regulatory networks. Once a discrete classifier is obtained from sample data, its performance must be evaluated through its classification error. In practice, error estimation methods must then be employed to obtain reliable estimates of the classification error based on the available data. Both classifier design and error estimation are complicated, in the case of Genomics, by the prevalence of small-sample data sets in such applications. This paper presents a broad review of the methodology of classification and error estimation for discrete data, in the context of Genomics, focusing on the study of performance in small sample scenarios, as well as asymptotic behavior. It does not depend on a particular set of training samples, but it gives the average classification error over all possible training data; therefore it is an intrinsic performance measure of the classification rule for the particular problem  and sample size n.Being a function of the random variables 4.nε has to be estimated from the sample data using an error estimator εnˆ. An error estimator is a function of the classification rule Ψn and the sample data nS. Therefore, it is a random variable through dependency on the random training data nS. If the error estimator depends on any additional random factors, sometimes called internal factors, it is called randomized, otherwise it is said to be nonrandomized. Examples of the latter include the apparent error or resubstitution  such that Pa≤εn≤bεnˆ>1−α, for 0 ≤ α ≤1, which give bounds on the true error corresponding to a given precision α, the observed error estimate, and the sample size. Confidence intervals express statistical power in error estimation --- more powerful methods will produce shorter confidence intervals for the true error at the same sample size. A very important fact is that all of the aforementioned performance metrics, and in fact any others, can be determined if one has knowledge of the joint sampling distribution of the vector of actual and estimated errors εn,εnˆ. Section 5 reviews exact analytical methods to compute these performance metrics, as well as complete enumeration methods that allow the computation of the joint sampling distribution of actual and estimated errors.conveniently combines both the bias and the deviation variance into a single measure, and is widely adopted for comparison of error estimator performance. Additional performance measures include the εnrˆ ≥E[εˆnr] showing that the leave-one-out estimator must be necessarily less optimistic than the resubstitution estimator. In fact, it is a general result (not restricted to discrete histogram classification), thatE[εˆnl]=Eεn−1 making this estimator almost unbiased. As it turns out, this bias reduction is accomplished at the expense of an increase in variance :O(n-1/2) bound is inferior to the exponential bound in (24), but it does guarantee a fixed rate of convergence that is independent of the distribution.This polynomial Eεˆnr to ε*, and again assuming random tie-breaking over cells, it is shown in , that the following exponential bound appliesRegarding convergence of provided that there is no cell over which c0ip = c1iq. Here, the constant c > 0 is the same as in (24). The presence of a cell where c0ip = c1iq invalidates the bound in (26) and slows down convergence; in fact, it is shown in [ε∗−Eεˆnr has both upper and lower bounds that are O(n-1/2), so that convergence cannot be exponential. Finally, observe that the bounds in (24) and (26) can be combined to bound the bias of resubstitution  Eεˆnr−εn. We can conclude, for example, that in case there are no cells over which c0ip = c1iq, convergence of the bias to zero is exponentially fast.ound in 2 and slowThe previous results on the discrete histogram rule concern expectation and bias. In ,  (for fixed b), whereas convergence of the RMS bound for the leave-one-out estimator is O(n-1/4), thus slower. Now, as remarked in [lower bound of kind O(n-1/4) on the RMS of leave-one-out. Therefore, in the worst case, the RMS of leave-one-out to zero is guaranteed to decrease as n-1/4, and therefore is certain to decrease slower than the RMS of resubstitution. Note that the bad RMS of leave-one-out is due almost entirely to its large variance, typical of the cross-validation approach, since this estimator is essentially unbiased.An important factor in the comparison of the resubstitution and leave-one-out error estimators for discrete histogram classification resides in the different speeds of convergence of the RMS to zero. Convergence of the RMS bound for the resubstitution estimator is arked in , p.419, 7.coefficient of determination (CoD) gives the relative decrease in unexplained variability when entering a variable X into the regression of the dependent variable Y, in comparison with the total unexplained variability when entering no variables:In classical regression analysis, the SSY and SSXY are the sums of squared errors associated with entering no variables and entering variable X to predict Y, respectively. The term SSY is proportional to the total variance σY2, which is the error around the mean µY (so that entering no variables in the regression corresponds to using the mean as the predictor).where In classification, a very similar concept was introduced in :εY∗=minPY=0,PY=1 is the Bayes error in the absence of any features, and εXY∗ is the Bayes error when using feature vector X to predict Y. By convention, one assumes 0/0 = 1 in the above definition. This binary coefficient of determination measures the relative decrease in prediction error of a target variable when using predictor variables, relative to using no predictor variables; notice the remarkable similarity between (30) and (31).where The binary CoD was perhaps the first predictive paradigm utilized in the context of microarray data, the goal being to provide a measure of nonlinear interaction among genes . Even thEven though the definition above employs Bayes errors, the CoD can be likewise defined in terms of the classification error of predictors designed from sample data, using for example the discrete histogram rule. In addition, the actual classification errors will typically need to be computed through error estimation techniques; e.g., one may speak of resubstitution and leave-one-out CoD estimates. All the issues discussed in previous sections regarding classification and error estimation for discrete data generally apply here.intrinsically multivariate predictive (IMP) genes using the binary CoD. Briefly, IMP genes are those the expression of which cannot be predicted well by any subset of binary predicting gene expressions, but is predicted very well by the entire set. In [canalizing genes, i.e., master genes that constrain  large gene-expression pathways [A recent paper  defined  set. In , the propathways .8.The importance of discrete classification in Genomics lies in its broad application in problems of phenotype classification based on panels of gene-expression biomarkers and inference of gene regulatory networks from gene-expression data, where data discretization is often employed for data efficiency and classification accuracy reasons. This paper presented a broad review of methods of classification and error estimation for discrete data, focusing for the most part on the discrete histogram rule, which is the classification rule most employed in practice for discrete data, due to its excellent properties, such as low complexity and small data requirement , and universal consistency. The most important criterion for performance is the classification error, which can be computed exactly only if the underlying distribution of the data is known. In practice, robust error estimation methods must be employed to obtain reliable estimates of the classification error based on available sample data. This paper reviewed analytical and empirical results concerning the performance of discrete classifiers (in terms of the classification error) as well as of error estimators for discrete classification. Those results were categorized into small-sample results --- small-sample data being prevalent in Genomics applications --- and large-sample  results. The binary Coefficient of Determination was also reviewed briefly; it provides a measure of nonlinear interaction among genes and is therefore very useful in the inference of gene regulatory networks. Progress in classification and error estimation for discrete data, particularly the analysis of performance in small-sample cases, has a clear potential to lead to genuine advances in Genomics and Medicine, and therefore the study of such methods is a topic of considerable research interest at present.


Several classification and feature selection methods have been studied for the identification of differentially expressed genes in microarray data. Classification methods such as SVM, RBF Neural Nets, MLP Neural Nets, Bayesian, Decision Tree and Random Forrest methods have been used in recent studies. The accuracy of these methods has been calculated with validation methods such as v-fold validation. However there is lack of comparison between these methods to find a better framework for classification, clustering and analysis of microarray gene expression results.In this study, we compared the efficiency of the classification methods including; SVM, RBF Neural Nets, MLP Neural Nets, Bayesian, Decision Tree and Random Forrest methods. The v-fold cross validation was used to calculate the accuracy of the classifiers. Some of the common clustering methods including K-means, DBC, and EM clustering were applied to the datasets and the efficiency of these methods have been analysed. Further the efficiency of the feature selection methods including support vector machine recursive feature elimination (SVM-RFE), Chi Squared, and CSF were compared. In each case these methods were applied to eight different binary (two class) microarray datasets. We evaluated the class prediction efficiency of each gene list in training and test cross-validation using supervised classifiers.We presented a study in which we compared some of the common used classification, clustering, and feature selection methods. We applied these methods to eight publicly available datasets, and compared how these methods performed in class prediction of test datasets. We reported that the choice of feature selection methods, the number of genes in the gene list, the number of cases (samples) substantially influence classification success. Based on features chosen by these methods, error rates and accuracy of several classification algorithms were obtained. Results revealed the importance of feature selection in accurately classifying new samples and how an integrated feature selection and classification algorithm is performing and is capable of identifying significant genes. Microarray technology allows scientists to monitor the expression of genes on a genomic scale. It increases the possibility of cancer classification and diagnosis at the gene expression level. Several classification methods such as RBF Neural Nets, MLP Neural Nets, Bayesian, Decision Tree and Random Forrest methods have been used in recent studies for the identification of differentially expressed genes in microarray data. However there is lack of comparison between these methods to find a better framework for classification, clustering and analysis of microarray gene expression.Another issue that might affect the outcome of the analysis is the huge number of genes included in the original data that some of them are irrelevant to analysis. Thus, reducing the number of genes by selecting those that are important is critical to improve the accuracy and speed of prediction systems. In this study, we compared the efficiency of the classification methods; SVM, RBF Neural Nets, MLP Neural Nets, Bayesian, Decision Tree and Random Forrest methods. We used v-fold cross validation methods to calculate the accuracy of the classifiers. We also applied some common clustering methods such as K-means, DBC, and EM clustering to our data and analysed the efficiency of these methods. Further we compared the efficiency of the feature selection methods; support vector machine recursive feature elimination (SVM-RFE) 2], Chi , Chi 2],Supervised classification, also called prediction or discrimination, involves developing algorithms to priori-defined categories. Algorithms are typically developed on a training dataset and then tested on an independent test data set to evaluate the accuracy of algorithms. Support vector machines are a group of related supervised learning methods used for classification and regression. The simplest type of support vector machines is linear classification which tries to draw a straight line that separates data with two dimensions. Many linear classifiers  are able to separate the data. However, only one achieves maximum separation. Vapnik in 1963 proposed a linear classifier as a original optimal hyperplane algorithm . Bag. Bag18].Cluster-analysis algorithms group objects on the basis of some sort of similarity metric that is computed for features. Genes can be grouped into classes on the basis of the similarity in their expression profiles across tissues, cases or conditions. Clustering methods divide the objects into a predetermined number of groups in a manner that maximizes a specific function. Cluster analysis always produces clustering, but whether a pattern observed in the sample data remains an open question and should be answered by methods such as resampling-based methods. The k-means algorithm, Farthest First Traversal Algorithm, Density-based clustering, Expectation Maximization (EM) Clustering are four common methods used in this study -26.wrapper model and the filter model . Str. StrCorrfeatures 29]. Lin. LinCorrvariable .Chi Squared: Another commonly used feature selection method is Chi-square statistic (χ2) method  or .-5, 2-3, … , 215 and γ = 2-15, 2-13, … , 23.We considered the RBF kernel and used cross-validation to find the best parameter C and γ. We used a “grid-search”  on C andThe classification methods were first applied to all datasets without performing any feature selection. Results of 10-fold cross validation have been shown in Figure The lowest accuracies are detected from Decision Tree algorithms (both J48 and ID3). As it is shown in Figure We applied two class clustering methods to the datasets that are illustrated in Figure Pairwise combinations of the feature selection and classification methods were examined for different samples as it is shown in table First we tested SVM-RFE, Correlation based, and Chi Squared methods on several gene numbers . Methods were mostly consistent when gene lists of the top genes 50, 100, or 200 were compared. We selected 50 genes because it performed well, consumed less processing time, and required less memory configurations comparing to others.Almost in all cases, the accuracy performance classifiers were improved after applying feature selections methods to the datasets. In all cases SVM-RFE performed very well when it applied with SVM classification methods.In lymphoma dataset SVM-RFE performed 100% in combination of SVM classification method. Bayesian classification method performed well for SVM-RFE and Chi Squared feature selection methods with 92% accuracy in both cases.CFS and Chi Squared also improved the accuracy of the classification. In breast cancer dataset the least improvement is observed from applying Chi Squared feature selection methods with no improvement over SVM, RBF and J48 classification methods with 97%, 84%, and 95% respectively.In ovarian cancer dataset all feature selection methods performed very close to each other. However the SVM-RFE had a slightly better performance comparing to other methods. We detected 100% accuracy with SVM-RFE feature selection with both SVM and RBF classification methods. We also observed high accuracies among MLP classification and all feature selection methods with 94%, 92%, and 92% for SVM-RFE, CFS, and Chi Squared respectively.In lung cancer datasets we can observe high accuracy in Decision Tree classification methods (both J48 and ID3) with all feature selection methods.Overall we have to repeat again that although it is obvious that applying feature selection method improves the accuracy and also particularly it reduces the processing time and memory usage, but finding the best combination of feature selection and classification method might vary in each case.The bioinformatics techniques studied in this paper are representative of general-purpose data-mining techniques. We presented an empirical study in which we compare some of the most commonly used classification, clustering, and feature selection methods. We apply these methods to eight publicly available datasets, and compare, how these methods perform in class prediction of test datasets. We report that the choice of feature selection method, the number of genes in the gene list, the number of cases (samples) and the noise in the dataset substantially influence classification success. Based on features chosen by these methods, error rates and accuracy of several classification algorithms were obtained. Results reveal the importance of feature selection in accurately classifying new samples. The integrated feature selection and classification algorithm is capable of identifying significant genes.Multi-layer perceptron (MLP): Error backpropagation neural network is a feedforward multilayer perceptron (MLP) that is applied in many fields due to its powerful and stable learning algorithm .,12.Δwji networks: RBF networks have 2 steps of processing. First, input is mapped in the hidden layer. The output layer is then a linear combination of hidden layer values representing mean predicted output. This output layer value is the same as a regression model in statistics [atistics . The outMoody and Darken  have proSupport Vector Machines (SVM): Given a training set of instance-label pairs . The algorithm starts with k random clusters, and then move objects between those clusters with the goal to 1) minimize variability within clusters and 2) maximize variability between clusters [k-means clustering analysis, the means for each cluster on each dimension would be examined to assess how distinct k clusters are. Obtaining very different means for most is perfect [clusters . In othe perfect .Farthest First: Farthest First Traversal Algorithm works as a fast simple approximate clustering model after Simple K-Means. To find k cluster centers, it randomly chooses one point as a first center, and then selects point with maximal min-distance to current centers as a next center [t center .Density Based Clustering (DBC): Density-based clustering has turned out to be one of the most successful traditional approaches to clustering. It can be extended to detect subspace clusters in high dimensional spaces. A cluster is defined as a maximal set of density-connected points. Correlation clusters are sets of points that fit to a common hyperplane of arbitrary dimensionality. Density-based clustering starts by estimating the density of each point to identify core, border and noise points. A core point is referred to as a point whose density is greater than a user-defined threshold. A noise point is referred to as a point whose density is less than a user-defined threshold. Noise points are usually discarded in the clustering process. A non-core, non-noise point is considered as a border point [er point .Expectation Maximization (EM) clustering: An expectation-maximization (EM) algorithm finds maximum likelihood estimates of parameters in probabilistic models. EM performs repeatedly between an expectation (E) step, an expectation of the likelihood of the observed variables, and maximization (M) step, which computes the maximum expected likelihood found on the E step. EM assigns a probability distribution to each instance which indicates the probability of it belonging to each of the clusters [clusters . By crosThe goal of EM clustering is to estimate the means and standard deviations for each cluster so as to maximize the likelihood of the observed data. The results of EM clustering are different from those computed by k-means clustering . K-meansCross validation: In order to perform to measure classification error, it is necessary to have test data samples independent of the learning dataset that was used to build a classifier. However, obtaining independent test data is difficult or expensive, and it is undesirable to hold back data from the learning dataset to use for a separate test because that weakens the learning dataset. V-fold cross validation technique performs independent tests without requiring separate test datasets and without reducing the data used to build the tree. The learning dataset is partitioned into some number of groups called “folds” [V-fold cross classification. 10 is the recommended and default number for “V”. It is also possible to apply the v-fold cross-validation method to a range of numbers of clusters in k-means or EM clustering, and observe the resulting average distance of the observations from their cluster centers. “folds” . The numLeave-one-out cross-validation involves using a single observation from the original sample as the validation data, and the remaining observations as the training data. This is repeated such that each observation in the sample is used once as the validation data .Financial support from Mississippi Computational Biology Consortium (MCBC) and Mississippi Functional Genomics Networks (MFGN) is gratefully acknowledged. The authors declare that they have no competing interests.MP and YD initiated the project. MP procured the necessary data and software, carried out the analyses, analyzed the results and drafted the manuscript. YD directed the design of the project and data analysis. JYY and MQY gave suggestions and helped to revise the manuscript. All authors read and approved the final manuscript.


SWISS uses Euclidean distance to determine which method does a better job of clustering the data elements based on a priori classifications. We apply SWISS to three different gene expression applications. The first application uses four different datasets to compare different experimental methods, normalizations, and gene sets. The second application, using data from the MicroArray Quality Control (MAQC) project, compares different microarray platforms. The third application compares different technologies: a single Agilent two-color microarray versus one lane of RNA-Seq. These applications give an indication of the variety of problems that SWISS can be helpful in solving. The SWISS analysis of one-color versus two-color microarrays provides investigators who use two-color arrays the opportunity to review their results in light of a single-channel analysis, with all of the associated benefits offered by this design. Analysis of the MACQ data shows differential intersite reproducibility by array platform. SWISS also shows that one lane of RNA-Seq clusters data by biological phenotypes as well as a single Agilent two-color microarray.Contemporary high dimensional biological assays, such as mRNA expression microarrays, regularly involve multiple data processing steps, such as experimental processing, computational processing, sample selection, or feature selection (i.e. gene selection), prior to deriving any biological conclusions. These steps can dramatically change the interpretation of an experiment. Evaluation of processing steps has received limited attention in the literature. It is not straightforward to evaluate different processing methods and investigators are often unsure of the best method. We present a simple statistical tool, Standardized WithIn class Sum of Squares (SWISS), that allows investigators to compare alternate data processing methods, such as different experimental methods, normalizations, or technologies, on a dataset in terms of how well they cluster  Suppose an investigator has a dataset that has a fixed number of samples designed to measure biological differences  and wants to process the data, but the optimal processing method is unknown. This processing may involve background correction, normalization, sample selection, or feature/gene selection. A central question is, “Which processing method works best on a given dataset?”et alt-statistics et alThere are a variety of papers in the literature which address the above question Many problems can arise when trying to evaluate two processing methods or compare different platforms. For instance, the best way to compare methods/platforms is not always clear when the data are on different scales or the methods have different (unknown) distributions. Also, investigators may not be interested in measuring phenotypes, but rather measuring the elements of the phenotypes. It is also important for investigators to select the optimal method independent of the results.Motivated by these problems, our goal is to develop a more generic approach to comparing processing methods or platforms. Our method, Standardized WithIn class Sum of Squares (SWISS), uses gene expression (Euclidean) distance to measure which processing method under investigation does a better job of clustering data into biological phenotypes . SWISS takes a multivariate approach to determining the best processing method. It tends to down-weight noise genes  while depending more on differentially expressed genes (genes with large variation between the classes). We also develop a permutation test based on the SWISS scores that allows an investigator to determine if one processing method is significantly better than another method.et alUsing the within class sum of squares to compare how well data are clustered has appeared before in the literature. For instance, Kaufman and Rousseeuw There are several advantages of SWISS. As previously mentioned, because we are standardizing the within class sum of squares by dividing by total sum of squares , SWISS can be used to compare methods that are on different scales. For example, different scales can arise from differing normalization methods or when comparing different platforms. Another advantage is that SWISS can be used to compare methods that have different dimensions. This can be useful when comparing the same biological samples, but using two different gene sets. Finally, because the permutation test reports a p-value, we are able to decide which processing method is preferred without relying on subjective evaluation.We will use SWISS to evaluate the one-color versus two-color microarray problem. Two-color gene expression array assays are among the most common genomic profiling tools currently in use The reference design, while powerful, has its disadvantages There are many aspects to comparing one-color versus two-color arrays. First, there is the underlying experimental question of whether it is more advantageous to use a one-color or two-color array. Second, there are questions of which normalization should be used when comparing different platforms. Finally, there is the decision of which samples and which genes should be included in the analysis. We will address the above issues using our SWISS method.The MicroArray Quality Control (MAQC) project was initiated to address concerns over the reliability of microarrays. In this project, gene expression levels were measured from two high-quality distinct RNA samples on seven microarray platforms . Each microarray platform was deployed at three independent test sites and five replicates were assayed at each site. This experimental design and the resulting datasets provide a unique opportunity to assess the repeatability of gene expression microarray data within a specific site, the reproducibility across multiple sites, and the comparability across multiple platforms We will use SWISS to address some of the issues raised by the MAQC project. Specifically, we will compare one-color Agilent microarrays, two-color Agilent microarrays, and Affymetrix microarrays by measuring how well replicates of the same samples cluster together. We will also compare Affymetrix pre-processing methods RMA Microarrays have been the technology of choice for large scale studies of gene expression. However, array technology has its limitations. First, each microarray can only provide information about the genes that are included on the array. Second, there are multiple sources of variability such as differences in arrays, dye labeling, efficiency in reverse transcription, and hybridization d-dimensional vector of covariates (such as gene expression) of the jth observation ith class/phenotype d-dimensional overall mean of all N samples, and i. Following classical statistical ANalysis Of VAriance (ANOVA) ideas, the Total Sum of Squares (SST) is defined to beLet Then the Standardized WithIn class Sum of Squares (SWISS), which is the proportion of variation unexplained by clustering, is defined asSuppose we have two processing methods  that we are interested in comparing on the same dataset with pre-defined classes or phenotypes . We consider Method A to be “better” than Method B if each of the classes of Method A have “tighter” clusters and/or have larger distances between the classes than Method B. When this occurs, SWISS will report a lower score for Method A. This is shown by a 2-dimensional toy example with two phenotypes in However, suppose the investigator has a preference for using Method A. For example, Method A may be easier to implement, or may be more cost effective, as is the case when considering whether one-color arrays perform as well as two-color arrays. To answer the question of whether the difference between the SWISS scores of Methods A and C is statistically significant, we developed a permutation test based on SWISS. This permutation test is described in detail in http://cancer.med.unc.edu/nhayes/pubs.html.R and Matlab code for calculating SWISS and performing the corresponding permutation test are available at We analyzed four cases of microarray experiments likely to represent the diversity of data presented in a typical microarray-intensive laboratory . The casFor the purpose of compatibility, both datasets I and II contained breast cancer samples and both datasets III and IV contained head and neck squamous cell carcinoma (HNSCC) samples. These sample sets were selected based on the availability of distinct phenotypes of approximately equal prevalence. The phenotypes were estrogen receptor positivity/negativity for the breast cancer samples and tumor/normal for the head and neck cancer samples. Datasets I-II were existing data previously reported https://genome.unc.edu/). Dataset II is also available under the GEO accession number GSE1992. The image files of dataset I were analyzed by using a ScanArray 3000  or a GenePix 4000  scanner and the primary data tables and the image files were originally stored in the Stanford Microarray Database (http://smd.stanford.edu//) The gene expression data of two breast cancer datasets I  were obtet alAs novel experiments in this study, we selected eight HNSCC tumor samples and eight normal tonsil samples collected on an IRB-approved protocol from patients treated at our institution. RNA isolation and microarray protocols were carried out as described in Hu et allimmaPrior to log-ratio transformation, signal intensities of Cy5-labeled experimental channel and Cy3-labeled reference channel were background corrected and loess normalized between the two channels for dye-bias correction In our lab, as in many labs, Cy5 has historically been used for the experimental channel and Cy3 for the reference channel. Therefore, in these retrospective analyses, Cy5, by default, is the single-channel signal. Many investigators will be familiar with research suggesting that Cy3 is preferable due to its more favorable stability properties and this should be considered for any single-channel experiments planned as a result of the retrospective analysis We first filtered the datasets down to approximately 10% of total genes  based on Median Absolute Deviation (MAD). We then clustered the datasets using Consensus clustering http://edkb.fda.gov/MAQC/MainStudy/upload/). Four different samples were assayed on these platforms. The two RNA sample types used were a Universal Human Reference RNA (UHRR) from Stratagene and a Human Brain Reference RNA (HBRR) from Ambion. Sample A was 100% UHRR; Sample B 100% HBRR; Sample C 75% UHRR and 25% HBRR; Sample D 25% UHRR and 75% HBRR. The two-color Agilent arrays only used samples A and B.The Agilent and Affymetrix array data (datasets V – VII) were obtained from the MAQC website  were carried out at six separate test sites (we only use data from the first two test sites) with each site processing five replicate assays on each of the four samples for a total of 40 microarrays. The Agilent one-color experiments (dataset VI) were carried out at three separate test sites (we only use test site 2) with each site processing five replicate assays on each of the four samples for a total of 20 microarrays. The Agilent two-color experiments (dataset VII) were carried out at three separate test sites (we only use test site 2) with each site processing five replicate arrays on each of the two samples (A and B) for a total of 10 microarrays. For more information about the experiments, see Shi For dataset VI, we obtained the normalized data, which was transformed by setting all measurements less than 5.0 to 5.0. All data points were median scaled to 1 using the median signal intensity value for data points labeled as present. For dataset VII, we obtained the normalized log ratio data, which defined the log ratio as CH1/CH2 (Cy3/Cy5) where the Cy3 and Cy5 channel intensities were background-subtracted. Data were extracted using Agilent's Feature Extraction software, version 8.5. Only MAQC samples A and B were used in Agilent two-color experiments.For dataset V, we obtained both the normalized and raw CEL files. For the comparison of Affymetrix and Agilent platforms, we used the normalized data from test site 2, which was normalized using PLIER Microarray experiments (dataset VIII) were performed with custom UNC Agilent two-channel microarrays. After hybridization, the arrays were scanned by Axon GenePix 4000B scanner . The images were analyzed using Gene Pix Pro 5.0 software . These data have been deposited into the Gene Expression Omnibus (GEO) under the accession number of GSE20234.https://genome.unc.edu/) to perform the filtering and pre-processing. The data matrix was gene median centered and sample standardized. Missing data were imputed with 10-nearest-neighbor imputation Gene expression values were quantified by the two based log ratio of red channel intensity (mean) vs. green channel intensity (mean), followed by loess normalization to remove the intensity-dependent dye bias th, 2009) on the isoform level using the equation RPKM  = 109× C/(NL), where: C is the number of reads that mapped to a transcript, N is total number of mappable reads in the experiment, and L is the length of the transcript. The average of RPKM for all isoforms within a gene locus was computed and used to represent the quantity of the genes expressed in the cell. Logarithm (base 2) transformation was applied on each RPKM value on the gene level. A link to this data has been provided in the Gene Expression Omnibus (GEO) under the accession number of GSE20234.mRNA was prepared from each experimental sample and the standard RNA-Seq sequencing process (dataset IX) was carried out with Illumina Genome Analyzer II. Raw short read (36 bp) sequence tags were pre-filtered according to manufacture recommended error rate. Pre-processed short read sequences were aligned using MAQ The 1541 genes used in this study came from four published intrinsic gene lists SWISS can be applied to compare different normalization techniques on a dataset. We compared loess and quantile normalizations along with the raw expression data for the single channel design after performing background correction. The results of the analysis for dataset II are shown in This does not show that loess normalization is the optimal normalization method for Agilent single channel array data. There are other single channel normalization methods not considered here, such as normalization based on principle component analysis For datasets I – III, we performed our SWISS permutation test on the full gene set to test whether there was a significant difference in the clustering capabilities of the reference design versus the single channel design. If both reported p-values were less than 0.05, we concluded that the design with the lower SWISS was significantly better. Otherwise, we concluded that there was no significant difference between the reference and single channel designs. For dataset I, we see that the reference design and single channel design have p-values of 0 and 0.01, respectively. Because the reference design has a smaller SWISS score of 0.94, we conclude that the reference design is significantly better at clustering the data of different ER status (positive vs. negative) than the single channel design. This is not surprising because dataset I used older generation spotted cDNA arrays. Thus, the two-color reference design is absolutely necessary in this case.For dataset II, which used Agilent 22K custom oligonucleotide arrays, the reference design has a lower SWISS score of 0.91. Its p-value is 0.02, and the single channel design has p-value 0.03. Because both p-values are less than 0.05, we conclude that the reference design is significantly better than the single channel. However, both p-values are larger than the p-values for dataset I. Therefore, the difference between the reference and single channel designs is not as strongly significant for dataset II as for dataset I.Dataset III used Agilent 4×44K oligonucleotide arrays. Because the p-values of the reference and single channel designs are both 0.26, we conclude that there is no significant advantage to using the reference design over the single channel design. Thus, we confirmed results suggested by other investigators that high-quality commercial two-color arrays are available that do not benefit in a significant manner from the normalization offered by a reference channel An important question regards the biological significance of a difference between SWISS scores. The answer will differ for each dataset, but one possible solution to this question is to introduce a perfect-feature gene. For each class, find the average of each gene across all samples within the specified class. Each sample in the specified class is assigned the maximum of these gene averages for its value of the perfect-feature gene, and each sample not in the specified class is assigned the minimum of the averages. There are as many perfect-feature genes as classes. Add all of the perfect-feature genes to the dataset then recalculate the SWISS score . A perfect-gene approaches the smallest biologic quantity for which there is clearly a biologic interest and we can calculate the impact of adding such a feature to the dataset on the SWISS score. We can then use the SWISS score with and without a perfect-feature gene to assess the biologic significance relative to any statistically significant comparisons observed in an experiment. For example, for dataset I, the original SWISS scores of the single channel and reference designs are 0.958 and 0.943, respectively shown in . The perIn each of the previous analyses, the classes used were “true” phenotype classes. We were also interested in seeing how using unsupervised methods to determine classes would change the results. We applied standard hierarchical clustering techniques. Supplementary For our next analysis, we used SWISS to evaluate the effect of different types of and amounts of gene filtering. For each dataset (I – IV), we filtered the gene set by keeping the genes with the largest variances. We calculated SWISS scores starting with only 16 genes and ending with all genes included. Note that for each dataset, the gene lists of the reference and single channel designs were decided independently, and thus the gene sets may not be identical. For dataset IV, we calculated SWISS scores for the self-self hybridization experimental Cy3 and Cy5 channels, along with the average of those two channels. For dataset I , we see For dataset IV , the selNotice that in Remember that in When comparing et alSimilar to comparing different normalizations of the single channel design in the first experimental application, we use SWISS to compare two different Affymetrix pre-processing methods: RMA and the Affymetrix Micro Array Suite 5.0 (MAS 5.0). Arrays from test site 1 (n = 20) were analyzed. This eliminated the need for any intrasite adjustment. The SWISS scores of RMA and MAS 5.0 processed data are 0.06 and 0.48, respectively. The difference between these scores is statistically significant. This result is consistent with the results of Millenaar d-dimensional space. There was only one site (test site 2) common between all three platforms under consideration, the U.S. Food and Drug Administration (FDA), and hence we only consider data from this site. Because only two of the four samples (samples A and B) were assayed on the Agilent two-color microarrays, we only consider all four samples when comparing Affymetrix with Agilent one-color microarrays.We now use SWISS to compare the intersite reproducibility of the Affymetrix, Agilent one-color, and Agilent two-color microarray platforms (datasets V – VII). For this example, a low value of SWISS would mean that the sample replicates lie very close to each other in et alWhen considering all four samples, Affymetrix has a SWISS score of 0.01, which is significantly lower than the Agilent one-color SWISS score of 0.36. When considering only samples A and B, the SWISS scores are 0.007 for Affymetrix, 0.18 for Agilent one-color, and 0.80 for Agilent two-color. All of the pair wise tests between the three platforms return significant p-values, as shown in Supplementary et alFor our final application, we use SWISS to compare the performance of a single Agilent two-color array with one lane of RNA-Seq (datasets VIII and IX). Samples were classified by their biological phenotype, either Basal or Luminal. The SWISS scores for RNA-Seq and the microarray data were both 0.63  is that comparisons across platforms are directly interpretable. Competing approaches to SWISS for cross-platform technology comparisons frequently rely on transformations of the data to render competing platforms similar in their component elements. Such decisions such as gene filtering and cross-platform gene annotation may influence the interpretation of the results, as we document in et alWe recognize that the examples we provide cover a wide range of quantitative problems, and that in some cases a competing method might be considered or may have previously been suggested. In general, however, we have found that such competing methods are either limited in the range of problems which they address or subjective in their interpretation. For example, in Experimental Applications II and III, our analyses based on SWISS agreed with results found in the literature, such as Affymetrix having the best intersite reproducibility Coefficient of Variation (CV) of the quantitative signal values between the intrasite replicates,Total CV of the quantitative signal, which included both the intrasite repeatability as well as variation due to intersite differences, andPercentage of the common genes with concordant detection calls between replicates of the same sample type.et alAlso, Marioni Counts of RNA-Seq with normalized microarray intensities,2 fold changes, andEstimated logOverlap between genes called as differentially expressed.Most of their analyses relied on subjective cutoffs determined by the author rather than on easily interpretable p-values. Additionally, we are able to draw the same conclusions with only one analysis based on our SWISS method (compared to three each).While stressing the broad applicability of SWISS to a range of analytical problems and the ease of its use, we would like to acknowledge some important weaknesses as well. First, high-dimensional data such as these array experiments are the product of complex protocols and depend on the quality of reagents and samples. Any change in upstream elements, such as a lab protocol or normalization method, might influence the resulting SWISS scores dramatically. For example, while we showed that the SWISS score of the Affymetrix arrays was significantly lower than the SWISS score of the Agilent arrays, this result is most interpretable in light of the complex set of protocols that generated the MAQC data and might not generalize to other labs or samples. Additionally SWISS scores may not represent the only criterion on which one method is preferred. For example, we showed that RMA gives more reproducible results than MAS 5.0. However, some investigators may prefer using MAS 5.0 because it is more conservative, gives positive output values, down-weights outliers, and minimizes bias We also acknowledge that while SWISS is convenient to implement across a broad set of analyses, there are likely cases where more dedicated methods would likely provide more nuanced insights. For example, SWISS should not be the only tool used when the investigator is interested in performing an in-depth analysis of competing methods or platforms, such as comparing a new normalization method with other established normalization methods. We should also acknowledge that the examples we have provided should be viewed with extreme caution in terms of the potential to introduce bias. We have shown examples in which one might be tempted to select what appears to be optimal gene sets based on SWISS scores. This is not the intention of the examples but rather to demonstrate the opposite: to document the impact on SWISS scores by varying gene lists. When approached with this caution in mind we feel that these concerns are offset by the broad scope of applicability that our SWISS method offers. Future work will address the concerns about bias adjustment.We hope investigators will see many uses for SWISS when considering competing processing methods or datasets for evaluating complex multidimensional problems, and will consider incorporating SWISS into their respective pipelines.Text S1Description of permutation-based SWISS hypothesis test.(0.05 MB PDF)Click here for additional data file.Table S1Consensus clustering results.(0.03 MB PDF)Click here for additional data file.Table S2Effect of clustering samples on SWISS scores for Experimental Application I.(0.01 MB PDF)Click here for additional data file.Figure S1SWISS permutation test results, Affymetrix and Agilent platforms.(0.06 MB PDF)Click here for additional data file.