Human heart failure is a complex disease that manifests from multiple genetic and environmental factors. Although ischemic and non-ischemic heart disease present clinically with many similar decreases in ventricular function, emerging work suggests that they are distinct diseases with different responses to therapy. The ability to distinguish between ischemic and non-ischemic heart failure may be essential to guide appropriate therapy and determine prognosis for successful treatment. In this paper we consider discriminating the etiologies of heart failure using gene expression libraries from two separate institutions.We apply five new statistical methods, including partial least squares, penalized partial least squares, LASSO, nearest shrunken centroids and random forest, to two real datasets and compare their performance for multiclass classification. It is found that the five statistical methods perform similarly on each of the two datasets: it is difficult to correctly distinguish the etiologies of heart failure in one dataset whereas it is easy for the other one. In a simulation study, it is confirmed that the five methods tend to have close performance, though the random forest seems to have a slight edge.For some gene expression data, several recently developed discriminant methods may perform similarly. More importantly, one must remain cautious when assessing the discriminating performance using gene expression profiles based on a small dataset; our analysis suggests the importance of utilizing multiple or larger datasets. Human heart failure is a complex disease diagnosed in over 500,000 American people every year, causing more than 250,000 deaths annually. It may arise from coronary atherosclerosis, exposure to toxins, infection, inflammation, valvular disease leading to volume/pressure overload, or an underlying genetic or idiopathic event -3. EmergA new approach to discriminating etiologies of heart failure is gene expression profiling using DNA microarray technology, which has been shown to be promising in the diagnosis of human diseases or subdiseases, especially in cancer -12. Recen, large p" problem [n is much smaller than the number of genes or predictors p. In our first dataset, we have n = 30 and p > 20000. Many new statistical methods have been developed or adapted to face the challenge. With more and more new methods emerging and existing methods being adapted, it becomes increasingly compelling for practitioners to compare and assess their performance, but there are few such comparative studies [Sample classification with gene expression data is statistically challenging due to the "small  problem : the num studies -20. Huan studies  compared studies , nearest studies , and a p studies ,20 have  studies  and rand studies . It is oMyocardial tissue samples from the left ventricular apex of patients with severe refractory heart failure were collected at the time of the left ventricular assist device (LVAD) placement at the University of Minnesota Medical School. A total of 30 tissue samples were processed for microarray analysis on the Affymetrix Human Genome U133A chip containing ~22,000 genes. The initial data analysis was completed using Affymetrix Microarray Suite (MAS 5.0). A more complete description on the data is provided in .The heart failure patients are divided into three classes according to the underlying etiology. Patients with clinical ECHO and EKG evidence, history of previous myocardial infarctions, and direct observation of the heart for confirmation of infarction at the time of LVAD implantation are defined as ischemic. Patients with an ischemic etiology were further divided into two classes: patients with ischemia but without acute myocardial infarctions (ischemic class) and patients with ischemia that have had an acute myocardial infarctions within ten days of LVAD implant (IM class), the remaining patients were assigned to the idiopathic class. Among the total 30 samples, 10 of them are ischemic, 7 are IM and 13 are idiopathic..The PGA data were obtained in another heart failure study conducted at the Cardio-Genomics PGA (Programs for Genomic Applications) at the Harvard Medical School. Myocardial samples were collected from patients undergoing heart transplantation whose failure arises from different etiologies  and from normal organ donors whose hearts were not used for transplants. The transcriptional profile of the mRNA in these samples was also measured with Affymetrix oligonucleatide microarray technology. HG-U133 plus 2 chips containing 54,675 probe sets were used and data were analyzed in MAS 5.0. In the PGA data set there were 11 normal samples, 11 ischemic samples and 14 idiopathic samples. The PGA dataset is publicly available at Genomics of Cardiovascular Development, Adaptation, and Remodelling. NHLBI Program for Genomic Applications, Harvard Medical School with URL: In order to make the results comparable to those based on the HG-U133A chips used in the Minnesota data, we matched the probe sets on a HG-U133 plus 2 chip with those on a HG-U133A chip. Only six probe sets on the HG-U133 plus 2 chip could not be found on a HG-U133A chip. Hence we used the remaining 22277 ( = 22283 - 6) probe sets in the following analyses with the PGA data.one-against-others two-class LOOCV misclassification errors  and one three-class LOOCV misclassification error. Note that throughout this article, three-class classification results for SC and RF were obtained by direct applications of SC and RF, rather than by combining multiple binary classifications.Table Based on Table In the two-class classification problem of ischemic vs. others, PPLS, PLS, SC, LASSO and RF yielded LOOCV errors range from 7 to 15, 6 to 13, 8 to 13, 9 to 15 and 10 to 12 respectively. The five methods obtained similar numbers of errors in all instances. In the two-class classification problems of IM vs. others and idiopathic vs. others, all five methods yielded similar numbers of errors. In the overall three-class classification, again all the five classification methods perform similarly. There is no clear evidence that one method is clearly superior to others.In order to assess whether a gene expression profile is affected by gender, we classified the 23 samples from male patients only. Among the 23 samples for male patients, 9 of them are ischemic, 7 are IM and 7 are idiopathic.Table one-against-others two-class classification problem of ischemic vs. others, PPLS made 10 errors when the top 400 genes are considered but the number of errors suddenly increases to 14 when the top 800 genes are used. Again there is no obvious relationship between the gene subset size and the five methods' performances.Based on Table one-against-others binary classifications.Comparing PPLS, PLS, SC, LASSO and RF, we find that the five methods perform very similarly in almost all instances. There is no evidence that any method is clearly the best. One thing we noticed about LASSO is that when the model contains many genes, say top 12800, then it gives more errors than PPLS, PLS, SC and RF in the three-class classification. This could happen by chance since we did not observe the same trend in the decomposed As in Table If we compare Table To see whether the high prediction errors are due to the presence of the three classes, we considered a binary classification problem. We applied all the five methods to the 10 ischemic and 13 idiopathic samples. We also assessed the classification accuracy on the male patients with 9 ischemic and 7 idiopathic samples. The classification results were shown in Table From Table In the previous classification problems, we only included linear terms of gene expression levels in a model. We also considered expanded models including squared terms of each gene's expression levels. The motivation is to possibly improve model fitting, for instance, to avoid masking in linear models . In thispair-wise approach. The results are included in Table one-against-others approach gives much smaller errors. This suggests that for this specific problem, the one-against-others approach is probably better. Again, we see the classification with male patients gives much larger LOOCV misclassification error rates.We repeated the three-class classification with PPLS via the one-against-others two-class LOOCV misclassification errors  and one three-class LOOCV misclassification error.Table Based on Table It may be argued that a classification involving normal samples should be much easier because the normal class is very different from the other two classes. Correct diagnosis between ischemic and idiopathic would be much more challenging. Hence we conducted a two-class classification with 11 ischemic and 14 idiopathic samples. The results are shown in Table n genes with n the number of the samples. It turned out that random forest also used many of the genes.We consider genes remaining in a final model for each method. To save space, we restrict attention to models starting with all the genes for binary discrimination between ischemic and idiopathic samples. Briefly, LOOCV was first used to select any tuning parameters in a method (e.g. number of components in a PLS model), then a model with the selected parameters was fitted using all the samples. Except that all the genes are used in a final PLS model, for any of the other methods there may be fewer genes remaining in the final model. In particular, LASSO can select at most Tables We also give the univariate ranks of the genes (based on the F-statistics for the two classes) in the above two tables. It shows that the two sets of genes  may not include some genes ranked high in the univariate ranking while including some ranked low, highlighting a possible limitation of solely depending on univariate ranks to select important genes.To further explore why the methods all work much better for the PGA data than for the Minnesota data, we drew some plots using the first and the second PLS components for binary classification of ischemic vs idiopathic groups. We found that, for both datasets, there was a clear separation between the two groups. However, in LOOCV, although again the two groups were separable for both datasets, the left-out sample was more likely to be closer to the other group than to its true group for the MInnesota data, leading to a high LOOCV error rate; Figure P-value is defined as the proportion of permuted datasets with misclassification errors fewer than that of the original data. To generate a randomly permuted data, we randomly permute the class labels of the original data. Because all the methods have similar performance, we consider the nearest shrunken centroid method with the Minnesota data. Tables P-values. This implies that, although there are relatively high misclassification error rates with the Minnesota data, the methods perform significantly better than a random guess.Because of high misclassification error rates with the Minnesota data, it is of interest to investigate whether there is any signal at all in the data. This can be accomplished by a permutation test that compares misclassification errors resulting from using the original data with that from randomly permuted data; a  is the fitted response value for sample i based on the original Minnesota data using PPLS or LASSO. Note that  is a real number without being dichotomized yet. Suppose that i Y= 1 or -1 is the class label of sample i in the original data, and . To generate a simulated data, we independently draw  from Normal distributions  for i = 1, ..., 23 and b = 1, ..., 50. Then we apply each method to a simulated dataset , where Xi is the gene expression profile of sample i in the original Minnesota data, and obtain fitted values ; the resulting misclassification error number for dataset b is .We did a simulation study to further evaluate the performance of the five classification methods. To mimic the real data, simulated data were generated from either a fitted PPLS or a fitted LASSO model to the Minnesota data comparing ischemic vs idiopathic, each containing top 50 genes in the initial model. Specifically, suppose that Table With more and more statistical methods being proposed for discriminant analysis for gene expression data, it has become increasing important to compare and evaluate their performances with real data, as it has been done in other contexts . Comparione-against-others approach and the pair-wise approach have been widely used in extending a binary classifier to multi-class settings. Our result suggests that, at least for the two datasets used here, the one-against-others approach is better, which was found to be true with support vector machines but in general should also depend on which binary classifier is used [Both the  is used . We alsoAn interesting finding of this work is that it is difficult to discriminate the different etiologies of human heart failure using one gene expression dataset, and at the same time, it is quite easy for the other dataset. A possible explanation is the different types of the microarray chips used: Affymetrix HG-U133A chips were used in the Minnesota study while Affymetrix HG-U133 plus 2 chips were used in the PGA study. Because the HG-U133 plus 2 chips contain more genes (or ESTs), to minimize the effects of using different genes, we only used the genes present in the Minnesota data and still yielded much better performance for the PGA data. In fact, we used all the genes in the PGA data and obtained similar results for the PGA data. Although we can say that the performance difference in the two datasets is not caused by different genes contained on a chip, we do not know whether the more recent HG-U133 plus 2 chips provide more reliable measurements on gene expression. In addition, quality control criteria for the inclusion of a chip were nearly identical between the two datasets. We would suspect that the performance difference may be the result of different patient populations and different study protocols (e.g. lack of clearly pre-specified patient inclusion/exclusion criteria). As discussed in , a key tMany studies have aimed to develop new statistical and machine learning methods for best sample discrimination. Our results suggest that, at least for some gene expression data, several existing methods may work almost equally well. More importantly, because of the quite different performances of the methods on the two datasets, one must remain cautious when assessing the performance of sample discrimination using a small gene expression dataset; it may be necessary to use larger or multiple datasets to draw a more reliable conclusion.We first briefly review the three binary classifiers, which was first designed for regression and can be directly applied to two-class classification, even when the number of covariates (i.e. genes here) is much larger than the sample size.Y = 1 for class 1 and Y = -1 for class 2. Suppose that xi is the expression level of gene i, i = 1, ..., p with p as the total number of the genes, and that we have n samples in the training data. A challenge is that we have n <<p.We code the response variable (i.e. class label) as  for j = 1, ..., m, then apply ordinary least squares (OLS) to regress Y on zj's to obtainThe main idea of partial least squares (PLS)  is to sezj's. It turns out thatwith β's as OLS estimates. The key of course is how to form linear components j = argmaxαCorr2 Var(Xα)α for l = 1,..., j - 1, where y is the vector of observed Y's (in the training data), X is the design matrix (i.e. matrix of observed x's), and S is the sample covariance matrix of x's [m has to be chosen, typically by a form of cross-validation, such as leave-one-out cross-validation (LOOCV), to minimize misclassification errors.with the constraints ||α|| = 1, x of x's . In pracPPLS is a penalized regression method in the framework of PLS ,32. SuppThen we shrink the PLS coefficients by soft-thresholding ,34sign(a) = 1 if a ≤ 0 and sign(-a) = -1 if a < 0, λ is a shrinkage parameter to be determined, and f+ = max. It is common that the shrinkage leads to many , effectively eliminating gene i from the model, thus gene selection is automatically accomplished. Next we construct a linear component . Finally a PPLS model is built by regressing Y on z using OLSwhere . The parameters involved in building a PPLS model, such as the shrinkage parameter λ and the number of PLS components, are estimated by LOOCV. The goal is to choose the largest shrinkage parameter and the smallest number of PLS components for which the LOOCV misclassification error estimate is minimized.which can be re-expressed as The LASSO estimates  in a linare obtained by, where Yi is the observed response for sample i and  is its LASSO estimate, i = 1, ..., n, and t can be chosen by LOOCV. The constraint can often force many , leading to gene selection.subject to Y is binary, but in any of the above binary classifiers, the response Y is treated as a continuous variable and the estimate  could be any real number. To predict the class of a new sample, we use sign: if the estimated response  is greater than or equal to 0, then we classify it into class 1; otherwise, class 2. In particular, this direct use of PLS for binary classification (as in [Note that the class label (1/-1) for the response n (as in ) is diffn  is built on a diagonalized linear discriminant analysis (DLDA) ,41. SuppSC is motivated from the observation that many of the genes will not be predictive of the class membership and should be eliminated from the above DLDA rule. Formally, definenk is the number of training samples in class k, and  is the overall mean expression level of gene i in all the training samples. Note that by the definition, we have . Letwhere i and k, where Δ is the shrinkage parameter to be chosen by LOOCV. Then substituting  in the DLDA rule by , we obtain a SC rulefor all x* is assigned to class k0 such that .The new sample , then  and thus gene i plays no role in classifying for class k. Hence SC effectively accomplishes gene selection by shrinkage.Note that, if Random forest (RF)  is an enK > 2) classification problem can be handled by a binary classifier, such as PLS, PPLS and LASSO. It is achieved by formulating a multi-class classification problem as multiple two-class classification problems. We consider two most popular approaches: one is to compare each class against all the others, and the other is to compare all possible pairs of classes. Applications of these two approaches can be found, among others, in [Here we describe how a multi-class  is predicted asfor .That is, the new sample is classified into the class maximizing pair-wise approach reduces a K-class classification to K(K - 1)/2 two-class classification problems [k1 and k2 (with 1 ≤ k1 <k2 ≤ K) is defined asThe problems . Specifi using only samples belonging to class k1 or k2, and denote the fitted response value for a new sample with expression profile x* as . As described earlier, we classify the new sample into class k1 or k2 according to the sign of . After this is done for any 1 ≤ k1 <k2 ≤ K, the final decision is to assign the new sample to the class that wins the most pairwise comparisons. In the case when there are multiple winning classes, we randomly pick one of the winning classes to be the final winning class. Comparing to the one-against-others approach, the pair-wise approach is computationally more expensive if K ≥ 4.We build a binary classifier with response xik is the gene expression level of the gene in sample i that is in class k, i = 1, ..., nk, and k = 1, ..., K, where nk is the total number of samples in class k and K is the total number of classes. Let  be the mean expression level of class-k samples,  be the overall mean  and  be the total number of samples. We can construct an F-statistic as the ratio of the mean sums of squares for between-class and within-class variations:To explore the effect of the number of genes a model starts with on the classification performance, we have a preliminary gene ranking using a usual F-statistic. This univariate ranking is used throughout, and obviously is by no means to be optimal. For the purpose of the presentation in this section, we only need to consider a given gene. Suppose F-statistics: a gene with a larger F-statistic indicates a stronger relationship between its expression levels and the class membership in the samples, and therefore has a higher rank as a potential predictor of the class. We started with various models by including different numbers of top ranked genes. We considered models starting from the top 50, 100, 200, 400, 800, 1600, 3200, 6400, 9200, 12800, 16000, 19200, and all (22283 and 22277 for the two datasets respectively) genes respectively.We can rank all the genes based on their corresponding It is an incorrect practice in microarray experiments to first select genes using all the samples and then perform cross-validation using the selected genes, which gives downward biased prediction error estimates ,52. HencTo facilitate the application of penalized regression (i.e. PPLS and LASSO) so that their regression coefficients are in the same unit and thus can be penalized using a global penalty parameter, the expression levels of each gene were scaled to have sample variance 1.In addition to PLS/PPLS, we will consider the shrunken centroids (SC) method, the LASSO, and the random forest (RF). SC, LASSO and RF have been implemented in R , and are by default, where p is the total number of variables (i.e. genes).We use the leave-one-out cross-validation (LOOCV) to estimate the prediction error for each of the methods. Within this first-level LOOCV, a second-level LOOCV is used to select tuning parameters for each method to minimize cross validation errors. Specifically, in PLS, the smallest number of PLS components is selected among the PLS models that give the minimum LOOCV error. In PPLS, among the models with minimum LOOCV error, we first pick the ones with the smallest number of PLS components, then pick the one with the largest shrinkage parameter. In the SC method, the largest shrinkage is selected among the models that minimize the LOOCV error. The number of candidate threshold values and the number of cross validation folds are both set to be default . In LASSO, the maximum fraction parameter of the models that minimize LOOCV error is selected while the number of the candidate fraction values is set to be 51  and the number of cross validation folds is set to be the total sample size. In RF, every parameter is set to be default. For example, the number of trees to grow is set to 500, and the number of candidate splitting variables considered at each split is set to Due to the small sample size (about 10 in each class) in each dataset, it is quite challenging to estimate the prediction error well. Although it is straightforward to apply LOOCV or other cross-validation methods, their performance may not be optimal. After submitting this work, we became aware of the recent work by Fu et al , where aXH downloaded the PGA data, did the analysis and simulations. WP conceived of and directed the study. SG cleaned and managed the Minnesota data. XH, YC, SJP, LWM and JH conducted the Minnesota study and generated the data. XH, WP and JH drafted the manuscript.PLS plots with the first two PLS components. PLS plot for the Minnesota data comparing is-chemic vs idiopathic groups.Click here for filePLS plots with the first two PLS components. PLS plot for the PGA data comparing ischemic vs idiopathic groups.Click here for filePLS plots with the first two PLS components. PLS plots for LOOCV for the Minnesota data comparing ischemic vs idiopathic groups.Click here for filePLS plots with the first two PLS components. PLS plots for LOOCV for the PGA data comparing ischemic vs idiopathic groups.Click here for fileSimulation results with various simulation set-ups.Click here for file


High-throughput systems for gene expression profiling have been developed and have matured rapidly through the past decade. Broadly, these can be divided into two categories: hybridization-based and sequencing-based approaches. With data from different technologies being accumulated, concerns and challenges are raised about the level of agreement across technologies. As part of an ongoing large-scale cross-platform data comparison framework, we report here a comparison based on identical samples between one-dye DNA microarray platforms and MPSS .The DNA microarray platforms generally provided highly correlated data, while moderate correlations between microarrays and MPSS were obtained. Disagreements between the two types of technologies can be attributed to limitations inherent to both technologies. The variation found between pooled biological replicates underlines the importance of exercising caution in identification of differential expression, especially for the purposes of biomarker discovery.Based on different principles, hybridization-based and sequencing-based technologies should be considered complementary to each other, rather than competitive alternatives for measuring gene expression, and currently, both are important tools for transcriptome profiling. During the last decade, a considerable number of high-throughput technologies for transcriptome profiling have been developed. These include hybridization-based technologies, such as DNA microarrays -3, and sFavorable features of hybridization-based approaches include a significantly lower workload and relatively low cost. However, the probe collection on a chip, which necessarily relies on the coverage and the accuracy of both genomic sequences and clone libraries, presents a hard constraint on its detection power. In contrast, the "tag-and-count"-based methodologies require more advanced instruments that are more cost- and labor-intensive, but their capability of exhaustive transcript sampling allows the potential identification of novel mRNAs.The present co-existence of various DNA microarray platforms and sequencing-based technologies offers biomedical research increased options for transcriptome profiling. It is, however, important to understand how to compare data generated by these different technologies. Recent studies indicate that performance of various microarray platforms, as measured by data consistency, have been shown to be comparable ,7. Sever® and MPSS data, 2) the inclusion of gene expression data of a second pool of mouse retina (MRP2) for all microarray platforms, and 3) the investigation of variation across biological replicates, as measured over two different pools of mouse retina samples (MRP1 and MRP2). It has been a consensus that the use of biological replicates is an important element to ensure the reliability of microarray results [In the present study, which is part of an ongoing cross-platform comparison framework , we comp results . To our  results . It remaIn summary, our results showed that there were moderate, yet significant, correlations between microarray data and MPSS data, while the data from microarray platforms, including the recently included Illumina arrays, generally were well correlated. The majority of discrepant measurements between the technologies based on hybridization versus sequencing were genes with low-abundance transcripts. Tag-to-gene mapping ambiguity and the absence of tagging enzyme recognition site could also explain some of the discrepancy between MPSS and microarrays. Using two-way ANOVA and SAM, we examined the microarray data for the magnitude of data variation introduced by biological replicates and technical replicates, and showed that biological variations are smaller than platform variations. The genes we found most susceptible to variations between biological replicates were likely to be associated with metabolic pathways, biosynthesis, and binding related pathways. Due to their vulnerability to sample variation, any changes observed in these pathways and related genes should be interpreted more cautiously in biomarker discovery applications. Furthermore, as demonstrated in our study, the relative affordability of array replicates makes them useful to corroborate MPSS experiments where technical replicates are seldom feasible due to higher costs. For comprehensive transcriptome profiling, we suggest that complementary use of hybridization-based and sequencing-based technologies is likely to provide a better solution than pursuing a single type of technology alone.MPSS libraries for MRP1 and MRP2 were generated using 'Signature' MPSS protocol. Two alternative sequencing reactions conducted independently, i.e. two-stepper and four-stepper sequencing, provided two read-outs of tag sequences for each sample, and referred to as MPSS 17-bp and MPSS 20-bp, respectively.In the 17-bp signature sequencing, a total number of 34,341 signatures were detected for MRP1, and 29,509 signatures for MRP2. The total number of signatures was obtained in the 20-bp signature sequencing was 34,424 and 30,967, for MRP1 and MRP2, respectively.The assignments of signature to gene were performed as described in the Methods section. Only the reliable MPSS signatures were kept in the downstream analysis. In the 17-bp signature libraries, 6,001 and 5,340 unique UniGene identifiers for MRP1 and MRP2, respectively, were identified. This corresponds to the sum of transcript copies as 615,944 and 615,771 tpm (transcript per million). For the 20-bp tag collections, the numbers of unique UniGene identifiers were slightly smaller: 5,793 and 5,125 for MRP1 and MRP2, respectively, which corresponds to the sum of transcript copies as 655,919 and 652,588 tpm.All platforms showed generally good consistency among technical replicates for all samples, both in terms of CVs and correlation coefficients . Filtering was applied but was not observed to increase the correlations between intensities, while the correlations between logThe inter-platform data agreement by measuring correlation coefficients is shown in Additional file Table The Pearson correlation coefficients of log10-transformed gene expression were used to measure the overall linearity between microarray and MPSS data, since it is well known gene expression data often have a distribution close to log-normal. As shown in Table Next, we wanted to examine how data correspondence, both in terms of binary presence call and correlation of expression, across the two technologies varied as a function of transcript abundance. In MPSS, a gene was defined as present if the copy number was higher than a given tag count threshold, otherwise, it was considered absent. These calls were combined with microarray present/absence calls to perform a chi-square test to measure the agreement of present and absent calls between the technologies. When the threshold of tag count changed from 1 to 100 at a step of 1 (tpm), we obtained the binary call agreements as a function of tag abundances  plots  see Fig, we obseFurthermore, we found that 305 genes (286 in MRP1 and 272 in MRP2) were present across five microarray platforms, but were not detected by MPSS, neither from the 17-bp library nor the 20-bp library. For both libraries, about 64% of these genes were filtered out in the tag-to-gene mapping procedure. A total of 97 genes were not detected by MPSS at all (any sample or library), among which 11 were labeled "GATC"-negative. Using percentile-transformed data, we found that 67.8% of the remaining genes (60 of 86) were detected as having low expressions . This indicates that low-end sensitivity is the major reason for data discrepancy.Taking the whole MPSS libraries as gold standard, we defined false negative detections by each microarray as the number of genes that were detected by MPSS, as well as printed on the chip, but were not detected by the microarray. We found that the proportion of false negatives in the five microarray platforms ranged from 12.99%  to 2.18% .Based on the data from the microarrays where we had technical replicates, we used SAM to compare the detection of differential expression from MRP1::MC versus MRP2::MC, i.e., differential expression was investigated between the two mouse retina replicates based on a common reference of mouse cortex. As shown in Figure 2ratios (MRP1::MC and MRP2::MC) after normalization matched by both RS (RefSeq) and RSEXON (RefSeq ID and exon) across all four platforms where technical replicates were available. Only those genes that did not have missing values across the 10 measurements were used in this analysis. In RS- and RSEXON-based matching, 563 and 305 genes, respectively, satisfied the criterion. The threshold of statistical significance was chosen as p < 0.001.Two-way ANOVA analysis was performed to identify the contribution of platforms and samples on data consistency. We examined filtered logThe majority of genes did not exhibit any bias induced by neither sample replicate nor by platform. For the two matching options 52.0 % 293 of 563) and 55.9% (170 of 304) of the genes for RS-based and RSEXON-based matching, respectively, were robust both in sample and platform variability. Based on Table  of 563 aFrom the four microarray platforms with technical replicates, we used one-way ANOVA to identify genes with differential expression in MRP1 versus MRP2. These results were then compared with the Z-test statistic commonly used for detecting differential expression in MPSS data. The microarray data used for this analysis was restricted to genes that had valid measurements for all ten technical replicates. We also excluded measurements from probes with ambiguous UniGene mapping, which was used to match microarray data with MPSS data. For the four microarray platforms, Affymetrix, Amersham, Mergen, and ABI, we obtained 4,783, 5,987, 3,610, and 8,460 UniGene identifiers with corresponding p-values from one-way ANOVA, respectively.Taking a threshold p-value of 0.001, we summarized the concordant and discordant calls between the microarray one-way ANOVA test and the MPSS Z-test, as represented in Table For each microarray platform, we applied GSEA to generate hypotheses regarding which biological processes or pathways might be responsible for the differences between the two replicate sample pools. Only the genes (RSs) which had valid values across all 10 chips were used, that is, 2,490 for Affymetrix, 4,899 for Amersham, 3,132 for Mergen, and 7,556 for ABI. In GSEA, the GO hierarchy level was set to 4, and the permutation times as 1000. Several biological themes resulted from GSEA with enrichment score, but none were statistically significant. Although different microarray platforms revealed different degrees of variation between MRP1 and MRP2, the GSEA analyses on the affected gene sets were similar among platforms. When considering enriched themes found in all four platforms, there were 15 GO terms from the "biological process" category enriched in MRP1 compared to MRP2 and 12 GO terms from the "molecular function" category. There were no terms found to be enriched in MRP2 common to all platforms [see Additional file With microarray technology being rapidly developed and advanced in the past decade, it has become an important tool for studying gene expression patterns. In parallel, the technologies based on the "tag-and-count" principle, such as SAGE and MPSS, are also being used for exploring the full transcriptome. Although some efforts in designing and adopting standards of microarray experiments (e.g. ERCC ) and datThe main observations made in this study were that across the microarray platforms, both intra- and inter-consistency of data was generally high, but the agreement between MPSS and any microarray platform was moderate, as also reported in previous studies ,18. The 2ratios). The low intensity signals generally corresponding to low-abundance transcripts are typically filtered as it is expected that the signal-to-noise ratio becomes too small. The comparisons between MPSS and microarrays indicate that the MPSS technology also has problems in detecting low-abundance transcripts. It appears that neither technology can reliably detect transcripts expressed at very low frequencies in an environment where the expression levels of all transcripts could span several orders of magnitude. This is a major problem as many transcripts have low-abundance, and there is currently a consensus that many of the corresponding genes are associated with critical regulatory roles in the cells [Data filtering is commonly applied both to microarray and MPSS data. Both in the present study and an earlier study, we have shown that data filtering considerably improves data consistency between microarray platforms, and in particular on relative expression  and Swiss-Webster post-natal day one (P1) mouse cortex (MC) (n = 19). Retinas were dissected, collected and stored in Trizol (one pair of retinas per eppendorf tube) at -80°C prior to pooling. During the RNA extraction process, two pools of adult mouse retina  were created (700 retinas per pool), aliquoted. All samples were stored at -80°C until being used in experiments. The animal experiments were approved by the Institutional Animal Care Facility at Harvard University.®, Amersham  CodeLink®, Mergen ExpressChip®, Applied Biosystems (ABI) microarrays, and Illumina BeadArray®. Microarray experiments are composed of sample preparation, hybridization, scanning and image quantitation, which are a series of integrative procedures being conducted at a laboratory, generally according to the manufacturer's recommended protocols. To obtain sufficient statistical confidence in the data analysis, for each biological replicate (MRP1 and MRP2), five technical replicates on each platform were obtained, with an exception on Illumina. We wanted to include Illumina in this study to examine the magnitude of data variation in MPSS experiments and microarray experiments when no technical replicates are performed. For details of the experimental protocols and laboratories, we refer to Kuo et al., [Whole-genome mouse gene expression arrays (one-dye oligonucleotide microarrays) were investigated in this study, including: Affymetrix GeneChip et al.,  except fet al. [2ratio calculation, as described.The raw data sets of 63 chips after image scanning and quantification in each platform were collected. For Illumina data, we set the filtering threshold as "Detections" = 0.9. Filtering for the other microarray platforms are described in Kuo 2ratios) or the median after percentile transformation (for intensities) of repeated measurements on each platform were used in calculating correlation coefficients.Data repeatability and reproducibility  are two Total RNA of MRP1 and MRP2, which were identical to those used in microarray experiments, was sent to Lynx Therapeutics, Inc.  for 'Signature'-based MPSS experiments. Following an RNA quality test on a Agilent 2100 BioAnalyzer , cDNA libraries were generated according to the Megaclone protocol ,43. SignThe abundance of each signature was converted to transcripts per million (tpm), and the MPSS signatures were mapped to UniGene clusters by Lynx Therapeutics, based on the mouse genome sequence   and the Two approaches to match probes across different microarray chips, annotation-based and sequence-based probe matching were used . Brieflyin silico constructed "virtual tags" library, as described above. Thus, the gene expression data measured by microarrays and by MPSS were paired up for comparisons via UniGene clusters.MPSS signatures were mapped to UniGene clusters, using an Two separate total mRNA extraction processes were conducted on mouse retina under the same experimental settings and protocols, which generated mouse retina pool 1 (MRP1) and pool 2 (MRP2). Note that Illumina data were not included in this part of analyses due to lack of technical replicates.To investigate the effects of biological variation and platform variation, two-way ANOVA  was performed. As concluded in our previous study , the seqet al., [A group of differentially expressed genes represents the desired result of most microarray users. SAM , proposed by Tusher et al.,  is a met2ratios underwent two sequential filtering steps: (1) filtering according to spot quality flags; and (2) filtering out the probes which had less than three valid measurements out of the five replicates. For each microarray platform and each experiment group, the number of "called" genes  and FDR were recorded for every threshold "delta", which steps from 0.1 to 4 at an interval of 0.1.In our study, for each platform, the five pairs of chips on which MRP1 and MC were hybridized respectively were considered as one experiment group, while similarly another experiment group consisted of MRP2 and MC data. Prior to SAM analysis, the normalized logGSEA  ,50 combiOne-way ANOVA was conducted for each microarray platform . One-way ANOVA uses those genes that have valid measurements  across all experiments as input, and assigns a p-value to each gene indicating whether this specific gene displayed significantly varied expression levels between two biological replicates.MPSS analyses were also performed on the two pools of retina sample independently. Due to its high cost, it is not feasible to conduct repeated MPSS experiments for each biological sample. The common practice for those who use MPSS to identify differentially expressed genes has been to apply statistical tests that are able to handle non-replicate data based on certain sampling assumptions. The Z-statistics test  was applThe microarray data and MPSS data for the manuscript has been submitted to the GEO Omnibus. The series record number is GSE6313.FL designed and performed the data analysis workflow, drafted the manuscript. TKJ, EH, and WPK participated in the data analysis. WPK, JT and CP participated in running the microarray experiments. WPK, CC, LOM, and TKJ conceived the study and participated in the study design. All co-authors contributed to manuscript revisions and approved the final manuscript.Evaluation of intra-platform and inter-platform data consistency . This file shows the CVs and correlation coefficients of both the within-platform data set and among the across-platform data.Click here for fileCAT (Correspondence At the Top) plots,. The CAT plots show cross-platform data correspondence when using one platform as the reference.Click here for fileList of GO terms derived from GSEA . This file provides a list of GO terms which are shown as being variable between the two biological replicate pools used in our study.Click here for file® Experimental Protocols. This is the detailed description of experimental protocols for Illumina BeadArray® chip.Illumina BeadArrayClick here for file


Serial Analysis of Gene Expression (SAGE) and microarrays have found awidespread application, but much ambiguity exists regarding the evaluation of these technologies. Cross-platform utilization of gene expression data from the SAGE and microarray technology could reduce the need for duplicate experiments and facilitate a more extensive exchange of data within the research community. This requires a measure for the correspondence of the different gene expression platforms. To date, a number of cross-platform evaluations (including a few studies using SAGE and Affymetrix GeneChips) have been conducted showing a variable, but overall low, concordance. This study evaluates these overall measures and introduces the between-ratio difference as a concordance measure pergene.In this study, gene expression measurements of Unigene clusters represented by both Affymetrix GeneChips HG-U133A and SAGE were compared using two independent RNA samples. After matching of the data sets the final comparison contains a small data set of 1094 unique Unigene clusters, which is unbiased with respect to expression level. Different overall correlation approaches, like Up/Down classification, contingency tables and correlation coefficients were used to compare both platforms. In addition, we introduce a novel approach to compare two platforms based on the calculation of differences between expression ratios observed in each platform for each individual transcript. This approach results in a concordance measure per gene , as opposed to the commonly used overall concordance measures between platforms.We can conclude that intra-platform correlations are generally good, but that overall agreement between the two platforms is modest. This might be due to the binomially distributed sampling variation in SAGE tag counts, SAGE annotation errors and the intensity variation between probe sets of a single gene in Affymetrix GeneChips. We cannot identify or advice which platform performs better since both have their (dis)-advantages. Therefore it is strongly recommended to perform follow-up studies of interesting genes using additional techniques. The newly introduced between-ratio difference is a filtering-independent measure for between-platform concordance. Moreover, the between-ratio difference per gene can be used to detect transcripts with similar regulation on both platforms. Methods for the analysis of gene expression profiles have gone through progressive development over recent years. Traditionally, the level of transcribed mRNA has been analyzed using methods such as Northern blots, quantitative RT-PCR, differential display ,2, repreSAGE is based on the high-throughput sequencing of concatemers of short  sequence tags that originate from a known position within a transcript and therefore theoretically contain sufficient information to identify a transcript . In contIn contrast to SAGE, DNA microarrays are used to measure relative expression levels between samples of thousands of known transcripts. Currently, three array variants are being used  i.e.At present, SAGE, oligo microarrays, cDNA microarrays and Affymetrix GeneChips are the most widely used techniques for determining gene expression levels and gene expression ratios in different disease states and in cells under different physiological conditions or environmental stimuli. Often these different gene-expression profiling platforms are being used in parallel and data generated with the different techniques need to be compared, and possibly interchanged, within and between laboratories. Due to the overall difference in platform design, transcript level estimation, and gene annotation, direct comparisons are difficult and only a few attempts have been made to compare these different platforms Figure . To deteIn the current study we have determined the similarity between SAGE- and Affymetrix GeneChips-generated gene expression profiles of two independent RNA samples. One RNA sample is isolated from a Wilms' tumor; the other is the Stratagene Universal reference RNA. These expression data were then used to evaluate the annotation problems when comparing different gene profiling platforms and the methods that can be used to compare two different platforms with respect to individual gene expression measurements and with respect to between-sample gene expression ratios. Finally, it is demonstrated that the between-ratio difference can be applied to select those transcripts that display similar expression changes in both platforms.MmeI instead of BsmF1) and elimination of a blunt-end ligation. The pattern of variation in figure In order to compare SAGE with other gene expression profiling techniques we created a SAGE library with 69792 tags from a Wilms' tumor sample. SAGE data (51954 tags) for the Stratagene Universal reference RNA  were obMicroarray experiments were performed using Wilms' tumor RNA and the Stratagene Reference RNA. Results of biological replicas of each sample, with independent cRNA synthesis and hybridizations, showed a good reproducibility (Pearson correlation coefficients of 0.982 (n = 11938) and 0.979 (n = 10489);both P < 0.01) using intensity values for all probe sets with a "present" signal  . AnnotaThe Affymetrix HG-U133A GeneChips contained probe sets for 13727 Unigene clusters that could be identified, whereas eight percent of the probe sets (i.e. 1795 probe sets) could not be linked to a Unigene cluster because these sequences are withdrawn or because these sequences are currently under revision. Figure vice versa . By matching SAGE and Affymetrix data, an unambiguous data set was generated. On average about 30% of the unambiguous genes were observed to be expressed by both SAGE and Affymetrix GeneChips and could be included in the final comparison. Although this comparison comprised only 13% of all SAGE Unigene clusters and only 8% of the Affymetrix Unigene clusters, it was demonstrated that this selection was unbiased with respect to gene expression levels in each of the platforms. This allows the extrapolation of the conclusions to the whole platform.We looked for the correspondence in gene expression results between the two techniques using Up/Down classification Figure , the conThe overall similarity between SAGE and Affymetrix GeneChips is modest when expression ratios are compared. The correspondence improves to 90% when only highly expressed transcripts are included which means that noise is filtered out for both platforms. The differences between SAGE and Affymetrix GeneChips were not caused by a biased selection of the final data set, differences in GC-content of the included transcripts or extreme ratios resulting from low gene expression values. The observed cross-platform differences, arise from intrinsic properties of the platforms themselves, differences in the principle of determining the expression levels, such as absolute (SAGE) versus quantitative (microarray) mRNA levels, and/or processing and analytical evaluation . These dFuture studies should be aimed on improving the efficiency of SAGE tag annotation and avoidance of systematic bias in microarray techniques. Only then, measurements of various technologies can be directly compared and transformed to a universal gene expression catalogue. SAGE has the advantage that a whole transcriptome is analyzed, but is limited to the analysis of a small number of samples. For screening of large sets of samples SAGE cannot be the favored choice and Affymetrix GeneChips might be a good alternative. Therefore, we think that the future lies in combining the data from SAGE with Affymetrix GeneChips, custom cDNA or oligo arrays. This gives the advantage of complete expression profiling using SAGE and high-throughput array screening of a larger panel of samples allowing rapid identification and for instance validation of clinical relevant genes involved in disease onset ,43. FinaThis paper evaluates several approaches for the comparison of different gene expression platforms, outlined using SAGE and Affymetrix GeneChips. We demonstrate that for both SAGE and Affymetrix GeneChips the intra-platform correlations are extremely good, but that the inter-platform agreement based on an unbiased selection of transcripts is modest. The agreement between platforms increases if only transcripts are included with high tag counts and high hybridisation intensities. It appears that the expression distributions are similar for each of the platforms, but that the correlation between platforms is modest due to intrinsic differences, like sensitivity, levels of noise, and gene annotation. Finally, we introduce a novel, filtering-independent approach for data analysis based on the calculation of differences between expression ratios observed in SAGE and Affymetrix GeneChips for each individual transcript. The statistical probability value that can be assigned to each individual betweenratio difference, allows the selection of individual transcripts that display similar regulation on both platforms.® approach.Wilms' tumor tissue was obtained from a single individual after resection of the tumor. Tissue was immediately frozen in liquid nitrogen. Informed consent to use this material for scientific research was obtained. After homogenization, total RNA was extracted using Trizol , dissolved in RNase free water and stored at -80°C. The Stratagene Universal reference RNA was obtained from Stratagene . Purity and integrity of the RNA samples was confirmed on the Agilent 2100 Bioanalyzer , using the LabChipBsmFI was replaced by MmeI, linkers were adapted for LongSAGE and ditags were created using sticky-end ligation. All sequence files were processed using the SAGE2000 software provided by Dr. K.W. Kinzler . A detailed protocol may be obtained as a free download . For Lonsee also ). The SAsee also ,20).th base from our original sequence files using the SAGE2000 software. This 11th base can be used to match against the deposited sequences  and in this way one may be able to exclude Unigene clusters that contain a different 11th base in their sequence and thereby minimize the number of multiple matches. In the final comparison tags matching to multiple Unigene clusters were excluded. For annotation of LongSAGE tags we used the data available at the CGAP site for Unigene Build 170 (July 2004). These annotations were not available for Unigene Build 160.Extracted SAGE tags were annotated based on the SAGE Genie principles  through 2 Log ratios), from which the geometric mean gene expression ratio between the two samples was calculated. Probe sets on the Affymetrix chips were matched with Unigene clusters .Affymetrix HG-U133A GeneChips were used and the hybridizations were performed according to the manufacturer's protocols and carried out at the Micro-array Department . For analysis, the MAS 5.0 software suite was used and comparisons between duplicate Wilms' tumor hybridizations and duplicate Stratagene Universal reference RNA hybridizations were made (data were deposited into the GEO under accession GSE1158). This gives four comparisons  per gene and a P-value for this standardized difference can be obtained from the normal distribution. This P-value can then serve as a measure for the position of each gene in the distribution of between-platform ratio differences. Note that this P-value should not be interpreted as a significance value for the ratio difference between platforms. Such a test requires a gene specific variance estimate in the denominator of the standardized difference, which cannot easily be derived from the available data.For each platform and each transcript that full-filled the matching criteria an expression ratio between Wilms' tumor and Stratagene Reference RNA was calculated. With these ratios the correspondence between platforms was estimated using the Pearson correlation coefficient, Up/Down classification and a contingency table Figure . BecauseMK, FB and FVR planned and designed the study. JMR and FVR analyzed the data, generated the figures and drafted the manuscript. MK and FB helped by editing the manuscript, providing overall technical guidance and coordination. LA, DAZ and FVR created the LongSAGE and SAGE libraries, and performed cloning and sequencing of the concatemers. JMR developed the new approach for the comparison of multiple platforms, performed calculations with FVR and provided guidance with the statistical analyses. GJS and FVR performed the annotation of SAGE tags. All authors read and approved the final manuscript.This work was supported by the Stichting Kindergeneeskundig Kankeronderzoek (SKK) and the Dutch Cancer Society


As gene expression profile data from DNA microarrays accumulate rapidly, there is a natural need to compare data across labs and platforms. Comparisons of microarray data can be quite challenging due to data complexity and variability. Different labs may adopt different technology platforms. One may ask about the degree of agreement we can expect from different labs and different platforms. To address this question, we conducted a study of inter-lab and inter-platform agreement of microarray data across three platforms and three labs. The statistical measures of consistency and agreement used in this paper are the Pearson correlation, intraclass correlation, kappa coefficients, and a measure of intra-transcript correlation. The three platforms used in the present paper were Affymetrix GeneChip, custom cDNA arrays, and custom oligo arrays. Using the within-platform variability as a benchmark, we found that these technology platforms exhibited an acceptable level of agreement, but the agreement between two technologies within the same lab was greater than that between two labs using the same technology. The consistency of replicates in each experiment varies from lab to lab. When there is high consistency among replicates, different technologies show good agreement within and across labs using the same RNA samples. On the other hand, the lab effect, especially when confounded with the RNA sample effect, plays a bigger role than the platform effect on data agreement. Diversity of microarray data poses some unique and interesting questions on cross-experiment comparisons and the analysis tools needed for such comparisons. Since the invention of the microarray technology in 1995 , many ofPossible incompatibility of results between similar microarray experiments is a major challenge that needs to be addressed, even though the data produced within a single experiment may be consistent and easy to analyze. Different labs produce microarray data in different ways using different technology platforms, such as Affymetrix GeneChip, spotted cDNA array, and spotted oligo array. Affymetrix GeneChip uses one fluorescent dye while the spotted array uses two fluorescent dyes in the experiments. Direct comparison of raw data obtained from different technologies may not be meaningful. Instead, the final form of the data is often presented as relative expression levels, mostly ratios of intensities, after some statistical treatments including filtering, normalization and model-based estimation. Experiments using different technologies require different protocols for analyzing the raw data to derive the ratios. Scientists have published microarray data in a variety of formats including raw intensities and ratios of intensities. Does it make a difference which technology platform is chosen? Can we make use of the studies from different platforms and labs? To answer these questions and provide some guidance for platform comparisons, we report on a comparative study of three different platforms. The experiment is a simple two-tissue comparison between mouse liver and spleen. We used previously published data sets from two different sources as well as new data sets produced in house. There are several similar studies published in recent years -8. A not. Two data sets were downloaded from the NCBI Gene Expression Omnibus , which were generated by Choi et al. at California Institute of Technology  using Agilent oligo (GEO accession: GSE334) and cDNA arrays (GEO accession: GSE330), respectively. Two other data sets were generated at the Functional Genomics Unit at the W. M. Keck Center for Comparative and Functional Genomics at the University of Illinois using an in-house printed cDNA mouse array and Affymetrix mouse expression set 430A, and the data sets are available at . Another data set was downloaded from  and it was generated using Affymetrix Murine Genome set U74Av2. The data set names  given in Table As summarized in Table One indication of data reliability is the consistency of replicates in a particular data set. We used kappa coefficients as well as the Pearson correlation coefficients and intraclass correlation coefficients on the replicates within each data set. Those measures set a benchmark against which the reliability of different platforms can be assessed; see Figure For the KC data, we can see that two pairwise comparisons gave slightly higher agreement than the other four. It is, we believe, due to the double spots of each gene on the array. The two comparisons with higher agreement are the comparisons between the replicates within the slides.Using the matched genes by common UniGene IDs, we compared different data sets in this study. Table From Figure In a recent study, Jarvinen et al.  comparedSince we only used a subset of genes in the study due to the gene matching problem, we asked whether the gene-to-gene variation was impacted by the choice of the subset. A comparison of the box plots for the full data set and for the subset used in the overall comparison showed that the variation in the subset of genes is similar to that of the full data set. For example, Figure UniGene has been widely used to match genes on different microarrays. Using UniGene and sequence similarity, Thompson, et al.  reportedFrom the present study, we showed that the GNF data set had the lowest agreement with the other data sets. This difference is compounded with the facts that the consistency of replicates within the GNF data set is the lowest and the sample used to generate the data is different from those used by other labs. We believe that the technology platform plays a relatively minor role in the disagreement, but the variation introduced by sample differences is one of the major factors. It has been shown that the expression level can vary significantly between genetically identical mice . VariatiIn this study, we also found that the lab effect can be greater than the platform effect. As shown in Figure Obviously, the present study has limitations. The results were generated from a very limited number of data sets. Using UniGene IDs for gene matching across data sets can also be questioned. Further research is clearly needed to address these limitations.In this paper, we aim to address several issues in comparing microarray data across different platforms and different labs. We demonstrated that the consistency of replicates in each experiment varied from lab to lab. With high consistency among replicates, different technologies seemed to show good agreement within and across labs using the same RNA samples. A closer look at the results indicated that the variability between two labs using the same technology was higher than that between two technologies within the same lab. The source of RNA samples can make a difference in microarray data, however in our present study we do not show conclusive results pertaining to possible sample or lab effects, because we did not have data collected from two different samples within one lab.For the spotted arrays , we used raw intensity data from both Cy5 and Cy3. We filtered out the non-expressive data points using median plus three times median absolute deviation  of the affy of Bioconductor using the average difference (AV) between Perfect Match (PM) and Mis-Match (MM) probe pairs, and the Robust Multi-Array Average (RMA) expression measure developed by Irizarry et al. [For the in-house Affymetrix data, we used three summarization methods to generate the probe set level signals. KAV and KRMA are based on the R package y et al. , and KLWy et al. . The GNFTo stabilize the variances of data across the full range of gene expressions, we also performed the generalized log transformation for all the data sets following Durbin et al. .There are five different data sets in this study. The origins of the genes vary in those datasets. In order to study inter-lab agreement, we have to first identify common genes represented in different arrays. Based on the annotation of each data set, we found that we could maximize the number of cross-matched genes using the mouse UniGene IDs (Build 107). Based on the common UniGene IDs, we found 551 common genes across all five different data sets. But in the pairwise comparisons, the number of common genes ranged from 1,374 to 5,018  cell, ni+ is the sum of the i th row, nj + is the sum of the j th column, and n is the total number of subjects.An intuitive measurement of agreement is to count the percentage of genes falling in the same categories . However, this percentage can be high even if the data obtained from different platforms are not so compatible. Usually the ratios for the great majority of genes do not show a two-fold change, and the percentage of agreement can be high just due to chance. To adjust for this excess agreement expected by chance, we prefer to use the kappa coefficient, which is a popular measure of inter-rater agreement in many other areas of science. The kappa coefficient was first proposed by Cohen  for analFor example, the kappa coefficient between KC and KAV is 0.523. Table , divided by the total variance. A high ICC (close to 1) means that the inter-rater ratios vary little relative to the overall variability in the data. In computing the ICC for the replicates,  equals , where  is the variance within UniGene IDs. If we consider lab as a random effect in the overall comparison, the total variance  will equal , where  is the variance between labs. The ICC incorporates both the association between raters and the rater differences, while the Pearson correlation is insensitive to the latter.We used three measures of correlation to compare the ratios from different data sets: Pearson correlation coefficient, intraclass correlation coefficient (ICC) and intra-transcript correlation coefficient (ITC). ICC measures the inter-rater reliability relative to the total variability of the ratios. Here, a rater could be a replicate or a technology platform. ICC is the variance of different ratios between UniGene IDs,i, we defined ρi to be the square root of the ratio of within dataset sum of squares (SSW) and the total sum of squares (TSS). A common SSW was used in comparing lab pairs to avoid the problem of having seemingly higher ITC's due to unusually large within-lab variability at some lab. We applied logit transformation to each ρi to get γi, and then calculated the average γ. Converting γ back to the correlation scale, we obtained .We introduced the ITC for pairwise comparisons as described below. For each gene HW and LL collected data from public resources. MB and CW generated in-house data. HW and LL processed the collected data. HW and XH did statistical analyses. All authors read and approved the manuscript.


There is a large amount of microarray data accumulating in public databases, providing various data waiting to be analyzed jointly. Powerful kernel-based methods are commonly used in microarray analyses with support vector machines (SVMs) to approach a wide range of classification problems. However, the standard vectorial data kernel family  that takes vectorial data as input, often fails in prediction if the data come from different platforms or laboratories, due to the low gene overlaps or consistencies between the different datasets.We introduce a new type of kernel called maximum entropy (ME) kernel, which has no pre-defined function but is generated by kernel entropy maximization with sample distance matrices as constraints, into the field of SVM classification of microarray data. We assessed the performance of the ME kernel with three different data: heterogeneous kidney carcinoma, noise-introduced leukemia, and heterogeneous oral cavity carcinoma metastasis data. The results clearly show that the ME kernel is very robust for heterogeneous data containing missing values and high-noise, and gives higher prediction accuracies than the standard kernels, namely, linear, polynomial and RBF.The results demonstrate its utility in effectively analyzing promiscuous microarray data of rare specimens, e.g., minor diseases or species, that present difficulty in compiling homogeneous data in a single laboratory. Microarray has become a standard tool in many biological studies. Typically, classification analyses, where gene expressions of distinct biological groups are compared and classified according to their gene expression characteristics, are frequently performed in various clinical situations such as tumor diagnosis ,2, anti-vectorial data kernel family, there is another family called structured data kernel family that has been studied in many other fields including bioinformatics and machine learning [Other than the above learning -12. The learning , the marlearning  for biollearning  and the In microarray analysis, one of the main issues that hamper accurate and realistic predictions is the lack of repeat experiments, often due to financial problems or rarity of specimens such as minor diseases. Utilization of public or old data together with one's current data could solve this problem; many studies combining several microarray datasets have been performed -15. HoweAmong the structured data kernels, the ME kernel can take any distance data as input, and is thus applicable to vectorial data as well when converted into the Euclidean or other types of distance relationships among vectors. Since the ME kernel increases the distances among different sample vectors (or samples hereafter), while keeping similar samples in close distance, discriminative boundaries may be found more explicitly than in the case of the vectorial kernels due to the sparse distribution of heterogeneous samples Figure . FurtherThis paper is constructed as follows. We first show how the ME kernel can effectively work in heterogeneous microarray data using the Euclidean distance among sample vectors. Then, we show the unique and powerful noise reduction ability of the ME kernel in microarray data. Finally, we demonstrate that the ME kernel performs better than the standard kernels in classifying practical microarray data, namely, squamous cell carcinoma metastasis in the oral cavity.k-nearest neighbor gene distance (kNND) since it violates the positive semidefiniteness of kernels.We describe herein the classification performance of the ME kernel, compared to that of the three standard kernels, linear, polynomial and RBF. We also test two types of distance-based kernels, EKM and Saigo ,17, for heterogeneous kidney carcinoma data to confirm the ME kernel's superior discrimination ability against a highly mixed heterogeneous dataset. Then, we demonstrate the ME kernel's interesting denoising ability based on kNND using homogeneous leukemia microarray data with artificial noise. Finally, we further apply the ME kernel with kNND denoising to a more practical problem, i.e., heterogeneous data of squamous cell carcinoma metastasis in the oral cavity, to assess its total performance.We first use Dij) are also generated from the same feature genes. If a sample contains missing values, we again adopt a simple imputation; we replace the one-dimensional Euclidean distance (xih- xjh)2 with 2 if xih or xjh is missing. The six kernels are tested with SVMs to analyze their classification performance with various numbers of feature genes and various parameters described in Methods. The maximum accuracy among the tested parameters for each number of feature genes is recorded as the accuracy for each kernel.Before testing the performance, all the data are properly normalized by being first log-transformed, and then scaled to mean 0 and standard deviation 1  in each sample and then each gene. All the normalized datasets are available for free at our Internet server . Also thThe human kidney data of normal tissues and renal clear carcinoma tissues are collected from the public gene expression database, GEO-Gene Expression Omnibus . This daClassification analysis is performed between normal and carcinoma data. The results of the LOOCV test of 100 samples against various numbers  of feature genes are plotted in Figure et al. [νadd × 100% of original data in a gene expression profile with artificial white noise, i.e., the noise is added according to a normal distribution model with a mean of 0 and a standard deviation of twice that of each gene value distribution in the original dataset. Then, we extract 50 feature genes from the training dataset for each iteration of the LOOCV test by standard t-statistics.Acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL) data for cancer subtype classification have been reported by Golub et al. . There akNND denoising method with the following noise level settings is applied. First, raw noise that is assumed to internally exist in the original data is arbitrarily set at νraw = 0.05. Then, we define the total noise level as the sum of the raw noise and the above artificially added noise, νadd. For example, if 10% noise is added, the total noise level is νraw + νadd = 0.05 + 0.1 = 0.15, and (1 - 0.15)2 × 100 = 72.3% of the nearest distance genes out of the feature gene set are considered in calculating the kNNDs between samples (see Methods).As the control experiments using linear and RBF kernels, the standard singular value decomposition (SVD) denoising method is applied to reduce noise immediately after the noise is introduced. In the SVD denoising, three levels of noise removals by dfferent cumulative proportions, 85, 90, and 95%, of eigenvalues are explored. For the ME kernel, the We repeat the above random noise-adding test ten times and average the highest accuracies among various parameter combinations. The results are shown in Figure kNND-distance-based methods show high accuracies; for example, the kNND-ME kernel has an accuracy of 97.8% even at 20% noise level and maintains high accuracies of 97.2 and 92.0% at 30–40% noise levels. The EKM and Saigo kernels using kNND-distance also show similar accuracies to the kNND-ME kernel. To verify our results, we extensively analyzed the same data with various parameters including many cumulative proportions in the SVD but obtained similar tendencies, confirming the superior denoising ability of the kNND-based method [Interestingly and surprisingly, the three d method .kNND-ME method with a more practical problem-heterogeneous oral cavity carcinoma metastasis data. The data consist of two GEO datasets (GSE2280 and GSE3524) from different authors [We further analyze the total performance of the  authors . One dat authors . Both arkNND denoising (kNND-ME), are shown in Figure kNND-ME kernel, five different noise levels, ν = 0 (no noise), 0.05, 0.1, 0.15, and 0.2 are evaluated. For comparison, we also classify the two datasets separately and average the accuracies % is observed for the kNND-ME kernel at two points, 7 and 15 feature genes, in the mixed dataset. Those accuracies are obtained with ν = 0 and 0.05 denoising parameters. The result also indicates that the kNND-ME kernel shows more stable and higher accuracies than the other kernels for large numbers of feature genes.The results of the LOOCV test of the 40 samples against various numbers  of feature genes with four different kernels, namely, linear, polynomial, RBF, and ME with s Figure . The resIncidentally, the top 15 feature genes that show the highest average ranks by t-statistics in the LOOCV test and that are considered to be associated with oral carcinoma metastasis are: HFE (AF150664), FLJ12529 (NM_024811), CXorf56 (NM_022101), HEATR1 (NM_018072), MGAM (NM_004668), APOL3 (NM_014349), PYY2 (NM_021093), RBP3 (J03912), UBE2V1 , KCNJ15 (U73191), GLS (AB020645), ARHGEF3 (NM_019555), MDM1 (NM_020128), ZC3H13 (AL136745), and C9orf16 (NM_024112).We further investigate the effect of the SVD denoising when applied to the mixed dataset before learning and classification. Table kNND-ME kernel in raw data is not improved by SVD denoising (data not shown).Although a sufficient number of genes  are used for SVD denoising, the denoised dataset does not significantly improve the raw accuracies in small numbers (≤ 30) of feature genes, where the overall maximum range accuracy (84.0%) exists. SVD denoising affects only large numbers (≥ 31) of feature genes. This is probably related to the property of SVD denoising that affects the ratio of information to noise content. Further analysis is needed to understand the full property of the SVD denoising method. In summary, the maximum accuracy (87.5%) of the Using kidney carcinoma data, we show that the ME kernel generally gives better classification results for heterogeneous microarray datasets than the three vectorial data kernels, linear, polynomial and RBF. As an alternative approach using vectorial data kernels, it is theoretically possible to train multiple SVMs for all distinct sub-data contained in the composite dataset. However, this approach has practical diffculties in that (i) there are too many heterogeneous sub-data, (ii) some sub-data contain only a few samples, and (iii) some sub-data contain all positive (or negative) samples. The SVMs cannot be trained properly with only a few samples or data with one-sided (positive or negative) labels. In addition, if we do not know the origin  of the test samples, it would be diffcult to determine which SVMs should be used for the classification. The ME kernel is much simpler yet quite flexible in this regard.kNND denoising method for the distance-based kernels, which show better performance than the linear, polynomial and RBF kernels for leukemia data, even though the data are pre-denoised by SVD. This is quite important in a situation where there are few or heterogeneous samples where SVD may not work properly for denoising because the quality of the eigenvalue decomposition depends on the number of homogeneous samples. Since the kNND denoising method only concerns the set of genes between sample pairs, it seems quite robust with regard to the number of samples or the degree of heterogeneity.Another remarkable property of our ME kernel is that the generated kernel matrices always hold positive semidefiniteness, even when the distance matrices for input to our optimization algorithm violate the triangle inequalities. This allows one to arbitrarily choose genes from among a set of feature genes to build the distance matrices in a distance-by-distance fashion. Utilizing this property, we devised the kNND distance data. From these observations, we can conclude that the entropy maximization process works favorably for 'heterogeneous' data and allows SVMs to find the discriminant boundaries more easily than the other two distance-based methods, EKM and Saigo.Furthermore, the results of kidney carcinoma and oral cavity carcinoma metastasis data in Figure kNND-ME kernel demonstrates higher and more robust classification performance than the linear, polynomial, and RBF kernels and even the other two distance-based kernels, regardless of SVD denoising.It is also important to point out that combining similar but distinct data in the microarray analysis may enhance the diagnosis of cancer or other diseases. As shown in our example of metastasis prediction for oral squamous cell carcinoma, each dataset contains only around 20 samples, which is not suitable for training of good SVM predictors, especially in the case of the vectorial data kernel family  and xj =  is represented by an inner product function such as K = ϕ(xi)·ϕ(xj). ϕ(·) means an arbitrary mapping of vectors to another space with generally different dimensions called 'reproducing kernel Hilbert space (RKHS)' [ϕ(·) for xi and xj, the inner product forms, i.e., K = ϕ(xi)·ϕ(xj) of the mappings of xi and xj are preferably used in practical calculation [K is called a kernel.Kernels are numerical expressions of similarity metrics between two samples. The basic form of kernels for two sample vectors with  (RKHS)' , which hculation . The funK = xi·xj, the polynomial kernel: K = (xi·xj + 1)D, and the RBF kernel: N samples as input, and we can fill all the  elements in the N × N kernel matrix with the specified kernel function.Three standard kernels popularly used in microarray studies are the linear kernel: positive semidefiniteness (see Appendix for details). If a kernel matrix is positive semidefinite, the mapped vectors ϕ(x) exist in the RKHS where the triangle inequalities among the mapped vectors are conserved. Our aim is to develop kernels that are robust to heterogeneous and noisy gene expression data. To this end, we first devise a new distance metric called kNND (detailed later) that can fulfill our requirements. However, unfortunately, the triangle inequalities are not conserved in the metric. To construct valid kernels from such distances, we introduce the following ME kernel algorithm.Any kernel matrix generated from such kernel functions possesses a necessary property for SVM learning called Dij, between samples, we obtain the ME kernel in matrix form, K, by basically solving the following optimization problem:The ME kernel was recently devised by Tsuda and Noble  to represubject to:H(K) = - tr(KlogK) is called von Neumann entropy of the kernel matrix K [K) = 1 is necessary to avoid unlimited divergence of matrix K. Constraints ||ϕ(xi) - ϕ(xj)||2 ≤ D ij are given as prior knowledge. For example, we can give the constraint such that a particular pair of the mapped samples in the RKHS must not be distant. Regardless of the constraint values, von Neumann entropy of a kernel is always maximized so that the kernel matrix holds positive semidefiniteness [Dij depending on i and j samples. The RBF kernel, in contrast, violates positive semidefiniteness if different gene sets are used to construct a kernel matrix using the above function, namely, by negating and exponentiating distance matrices Dij [Dij. Matrices K that resulted after fully maximizing H(K) can be used for further kernel analysis methods such as SVM classification and kernel principal component analysis [This optimization cannot be solved analytically; hence, we have implemented an efficient numerical algorithm for optimization . The fumatrix K . The firniteness . Thus, oices Dij . Intuitianalysis .M genes × N samples matrix) where a sample contains ν (×100)% of noisy genes on average. In such a case, only 1 - ν of genes in that sample contain no noise. Therefore, for any pair of samples, the ratio of common genes not containing noise is expected to be (1 - ν)2. Based on this observation, we compute the distance between two samples xi and xj as follows: First, we compute the one-dimensional Euclidean distances dh= (xih - xjh)2 for h = 1, ... , M genes. Then, we select k = (1 - ν)2 × M of one-dimensional Euclidean distances dh from the nearest  ones. Finally, we take the sum of the selected dhs as the distance between xi and xj. We refer to this method as k-nearest neighbor gene distance denoising (kNND denoising) method hereafter. For instance, if a sample with M = 100 feature genes contains ν = 15% of noisy values, k = (1 - 0.15)2 × 100 ≃ 72 of the nearest distance genes out of the 100 feature genes are only considered in calculating kNNDs between samples.The main issue addressed herein is how to handle missing or noisy values that exist in a large portion of a gene expression profile consisting of heterogeneous data. To effectively eliminate such spurious values without removing the entire gene, we devised the following simple method. Assume that we have a gene expression table  is generated. Note that since the samples use different gene sets in kNND metric, positive semidefiniteness will not hold when directly imported to the RBF kernel function. Instead, however, when our ME kernel algorithm is applied and those kNNDs are used as constraint, the resulting kernel matrix holds positive semidefiniteness as well as reflects similarities between samples. Subsequently, we will train SVMs for the optimized 'ME' kernel with sample labels.Multiplying the above Dij. Both approaches are originally devised for conversion of a non-positive-semidefinite similarity matrix S into a kernel matrix. The first approach is to take STS as a new kernel matrix. The kernel is sometimes called empirical kernel mapping (EKM) [S from its diagonal. We call it the Saigo kernel [Dij via Sij = exp(-Dij/σ 2).For comparison, we tested two approaches to obtain a kernel matrix from distance matrix ng (EKM) . The seco kernel . We obtaN expression values that are centered by means are reduced to the major q principal components by solving the eigenvectors of M genes × N samples matrix AM × N:As an alternative and conventional approach to noise reduction, singular value decomposition (SVD) is often used in many analytical studies including microarray analysis ,26. We uUM × N is M × N column-orthogonal matrix  and N × N orthogonal matrix . The matrix WN × N is N × N diagonal matrix and 1/(N - 1)W2 equals eigenvalues of the uncentered covariance matrix of A. We choose the largest q eigenvalues and replace other diagonal elements of W with 0, creating the Wq matrix. Finally, we obtain the denoised matrix, Aq, with Aq = U·Wq·VT. The vectorial data kernels are subsequently computed from the denoised matrix, Aq.where et al. [For actual analysis, software called SVDMAN developed by Wall et al.  is used.M-dimensional vectors or samples from different classes. The hyper-plane w·ϕ(xi) + b is obtained by solving the following optimization problem:The SVMs  are wellsubject toN samples where yi and C are the class label of i-th sample and a constant parameter, respectively. Optimization of this problem yields the hyper-plane that maximizes the margin between the two classes. This is called the SVM learning algorithm. The above optimization problem can be transformed into an equivalent form of the other equations in which only the kernel function K = ϕ(xi)·ϕ(xj) appears and mapped vectors ϕ(xj) are not explicitly described [K and mapped vectors ϕ(xj) are not necessary.for all escribed . Hence, N data and the SVM trained with the remaining N - 1 samples predicts the excluded one. All accuracies reported in this paper are calculated with the following formula:The classification accuracies for the evaluation of the ME kernel against other methods are estimated by a standard leave-one-out cross-validation (LOOCV) procedure where each sample is alternatively excluded from the TP, FP, TN and FN are true positive, false positive, true negative, and false negative frequencies, respectively, in the classification.where C = {10-3, 10-2, 10-1, 1,10, 102, 103}, are tested. For the polynomial kernel, D = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} are tested. For the RBF, EKM, and Saigo kernels, σ = {10-10, 10-9, 10-8,10-7, 10-6, 10-5,10-4, 10-3, 10-2,10-1, 1} are tested. In the ME kernel, we used only one parameter G that magnifies the distance constraints Dij to adjust the trade-off between over-learning and generalization of classification models  kernels tested here, seven SVM parameters, ils, see ). The pa problem . We testWF conceived of the study, carried out the analyses and drafted the manuscript. TK participated in the design of the study, wrote the program and helped to draft the manuscript. Both authors have read and approved the final manuscript.K, which guarantees the existence of mapping functions ϕ (·) for sample vectors, xi and xj, is defined as follows: A symmetric matrix K is said to be positive semidefinite if K holdsFormally, the positive semidefiniteness of a kernel matrix ci and cj.for any real numbers, K is positive semidefinite if and only if a mapped feature vector can be assigned to each sample such that each element of the matrix satisfies Kij = ϕ(xi)·ϕ(xj), where ϕ(xi) and ϕ(xj) are the mapped feature vectors of i-th sample and j-th sample, respectively [Practically, a symmetric matrix ectively . For exais positive semidefinite since we can assign mapped feature vectors as:ϕ(x1) = ⊤, ϕ(x2) = ⊤, ϕ(x3) = T,Kij = ϕ(xi)·ϕ(xj) indeed. A symmetric matrix,which hold Kij = ϕ(xi)·ϕ(xj).is not positive semidefinite since there exist no mapped feature vectors satisfying