Whole genome expression profiles are widely used to discover molecular subtypes of diseases. A remaining challenge is to identify the correspondence or commonality of subtypes found in multiple, independent data sets generated on various platforms. While model-based supervised learning is often used to make these connections, the models can be biased to the training data set and thus miss inherent, relevant substructure in the test data. Here we describe an unsupervised subclass mapping method (SubMap), which reveals common subtypes between independent data sets. The subtypes within a data set can be determined by unsupervised clustering or given by predetermined phenotypes before applying SubMap. We define a measure of correspondence for subtypes and evaluate its significance building on our previous work on gene set enrichment analysis. The strength of the SubMap method is that it does not impose the structure of one data set upon another, but rather uses a bi-directional approach to highlight the common substructures in both. We show how this method can reveal the correspondence between several cancer-related data sets. Notably, it identifies common subtypes of breast cancer associated with estrogen receptor status, and a subgroup of lymphoma patients who share similar survival patterns, thus improving the accuracy of a clinical outcome predictor. DNA microarray-based whole genome expression profiling is subject to poor reproducibility of discovered molecular disease subtypes and can lead to biomarkers that do not generalize Comparison and integration of the molecular disease subtypes, independently defined in different data sets, has been a highly challenging problem. Subtypes are often based on subtle differences in gene expression, which can be dominated by the measurement variation between different experiments and/or platforms. A widely used method to connect such independent data sets, supervised learning, does not completely solve this problem. Subtype models depend on one particular “training” data set with its own platform-specific data structure. This structure may not be present in new “test” data sets.Here we describe subclass mapping (SubMap), an unsupervised method that reveals common subtypes observed in independent data sets without relying on, or being biased by, one model data set. By bi-directionally evaluating association of predetermined subtypes between independent data sets, our method identifies more reliable molecular disease subtypes and subpopulations of cohorts that share similar clinical behavior. These results indicate the great potential of this method to maximize the use of the vast amount of accumulating genomics data and to improve current clinical practice through the development of better diagnostics and therapeutics.de novo class label to each sample). Because the subclass correspondence is evaluated for all pairs of subclasses, one subclass drawn from each data set, the number of subclasses or subtypes in the two data sets does not need to be the same or even similar.SubMap is an unsupervised method, which estimates the significance of an association between subclasses observed in two independent data sets. The subclass labels are predetermined as manually assigned phenotypes or by clustering prior to the application of the SubMap algorithm  for subclass A1, and similarly for A2, using any suitable subclass discrimination metric. Using the same metric, genes in data set B are rank-ordered according to their correlation with B1 vs. B2 to yield a gene list, ranking(B1). Association between A1 and B1 is evaluated by quantifying the over-representation of marker(A1) in the up-regulated end of the list ranking(B1) using Gene Set Enrichment Analysis (GSEA) as described in marker(B1) in ranking(A1). We can generalize this process straightforwardly when there are more than two candidate subclasses in either data set and compute a pair of nominal p-values for every possible pair-wise combination of subclasses. Thus if there are An subclasses in A and Bn subclasses in B, then we have mutual association information for each of the An*Bn pairs of candidate subclasses from the two independent data sets.We start with two independent data sets, A and B, with candidate subclasses independently determined in both. For simplicity, suppose we have two candidate subclasses in each data set: A1 and A2 in data set A; B1 and B2 in data set B. We define a set of marker genes, FF, we generate an appropriate null distribution by randomly picking ES scores from the null distributions corresponding to each direction of the enrichment analysis  in ranking(B1) and marker(B1) in ranking(A1)) and generating the corresponding Fisher inverse chi-square statistic.To summarize and estimate the significance of the mutual association information, i.e., the pair of nominal p-values for the ES, we use the Fisher inverse chi-square statistic, We use a Bonferonni adjustment to account for multiple hypotheses testing, and summarize the adjusted p-values in a matrix called the subclass association matrix (SA matrix). Two-way clustering of this matrix reveals general subclasses common to these data sets.We applied our method to four pairs of publicly available data sets . We firsFor a straightforward validation of subclass mapping, we used two data sets of multiple normal tissue types acquired on two different generations of the Affymetrix GeneChip oligonucleotide microarray platform, Multi-A In actual biological or clinical data sets that nominally include the same or similar samples, the entire range of sample diversity in one data set may not be represented in the other data sets, i.e., a subclass in one data set may not be represented in others. To assess whether such a situation would lead to false positive associations, we removed the samples corresponding to one tissue type in Multi-B and re-ran SubMap with the Multi-A . ImportaWe next investigated whether the choice of the number of marker genes to be mapped might affect the result of SubMap. The number of strongly differentially expressed genes for cross-dataset subclass pairs might differ. Since we are using the same number of marker genes for all comparisons we sought to confirm that the enrichment score was robust. We believed this would be the case based on the robustness of the GSEA score. We tested the effect of using 20, 50, and 200 ranked marker genes from a list of differentially expressed genes, as they represent the number of markers commonly used in classification and prediction studies for microarray data, as well as the level at which we might expect to begin to see noise. The resulting observed significance calls were quite stable for pairs of same tissue type irrespective of the number of the marker genes used .Here we show the superiority of SubMap in directly highlighting all three of the corresponding, common subtypes in two independent Diffuse Large B-Cell Lymphoma (DLBCL) data sets. We first compare with our previous work in We analyzed two DLBCL data sets, DLBCL-A In this precedent analysis, we associated these subtypes between the data sets by assessing the overlap of each subtype's marker genes in both data sets one-by-one using the Fisher test for a 2×2 table. However, unlike SubMap, this approach is highly sensitive to the criteria for marker gene selection and the total number of genes common to both platforms. Notably, initially only the HR clusters from the two datasets showed a significant association, and only after their removal could we identify the significant association for the OxPhos and BCR subtypes Next we compared SubMap with supervised methods for their ability to identify similar subtypes from distinct data sets using DLBCL-A and DLBCL-B. Supervised learning methods are widely used to associate two independent data sets; the model of the subtype is defined in a “training” data set, and assessed in a “test” data set. We employed two standard supervised methods, k-nearest neighbors  and support vector machines  and interchanged the roles of the two sets as train and test  and four  candidate subclasses in Breast-A and Breast-B, respectively . This result leads us to speculate that ER status may indeed be a significant factor in determining the subclasses and subclass associations.Both data sets have immunohistochemistry data of estrogen receptor (ER) status, which is known to play important role in breast cancer biology, e.g., BRCA1 mutation, lymphocytic infiltration, and early occurrence of distant metastasis 1 ∪ A2 and B1 ∪ B3), and evaluated the over-representation of the ER-related gene set  is larger than that of B3 . Although this difference is not statistically significant , this might suggest the existence of some biological heterogeneity among the ER-positive subclasses, and cause such “step-like” shape.The association between Anificant . In factIn this final example, we applied SubMap to see if identifying the corresponding subtypes in two diffuse large B-cell lymphoma data sets would enable us to distinguish subpopulations relevant to patient survival. We analyzed two independent DLBCL data sets, DLBCL-C (n = 58) 1, C2, C3, and C4 in DLBCL-C, and D1, D2, D3, and D4 in DLBCL-D). There was no statistically significant difference in the survival curves among these subclasses (data not shown). SubMap identified two significantly corresponding pairs of subclasses between these two data sets, “C3 and D2” and “C4, and D3” , and tested in their counterparts, D2 ∪ D3 (n = 61), the predictive model yielded better separation of survival than a model built using all of the samples (1 ∪ C2 (n = 33) did not work on D1 ∪ D4 (n = 68) This suggests that the predictive survival signature common to both of the data sets is carried by the “corresponding” cases, which our method identified.We next sought to determine whether we could improve the survival distinction between samples by restricting our consideration to the corresponding subclasses. Based on the sample labeling of poor prognosis in DLBCL-C defined in  samples  in spiteGenomic profiling is a powerful approach to discovering molecular disease subtypes. However, the uncovered subtypes may also reflect idiosyncrasies, technical variation, representational and measurement biases that limit the ability of the model and results to generalize to other studies or platforms. In this paper, we have shown how SubMap can address these issues and be an effective approach to recognizing subtypes across different data sets.In the four examples presented above the method clearly outperforms several more traditional, alternative approaches such as, i) using supervised models  to learn the subclasses obtained in one of the datasets and then predicting the subclass membership of the samples in the second set, ii) merging the two data sets and clustering samples to search for common subtypes, iii) associating the subtypes between the data sets by assessing the overlap of each subtype's marker genes in both data sets, or iv) creating an explicit model of subclassification in one data set, and evaluating it in another data set . All of these alternative methods have serious limitations and do not always produce reproducible results. While two different datasets may claim to represent the same biological phenotype, they often contain different molecular subtypes or biased representations and samplings with different dynamic ranges, measurement biases, probe efficiencies etc. These differences violate the common underlying assumption, of these other approaches-that two instances of the data, such as the train and test set, are samplings from the same probability distribution.SubMap associates the subclasses between datasets by using a more abstract, higher-level similarity measure . SubMap focuses on the concordant collective behavior of groups of genes rather than specific gene expression values, and does not rely on one particular data set to define a “model” like supervised learning methods. The result of this approach, which concurrently and bi-directionally evaluates similarity by treating the two input data sets equally within a self-contained unit, is that SubMap is able to provide solutions to the mapping problem that are more robust, more generalizable and more tolerant of the platform idiosyncrasies and biases.SubMap performs marker gene selection multiple times. For example, in the case where data set A and B have 3 and 4 subclasses, respectively, it performs maker gene selection 7 times, 3 times in data set A and 4 times in data set B. We cannot assume the same number of statistically “significant” marker genes for all of them, and to compute the significance for each round of marker selection by, e.g., permutation testing is highly computationally intensive. The GSEA enrichment score (ES) is weighted by the correlation between gene expression signals and the subclass. Therefore, genes having small correlation with phenotype  show negligible contribution in the computation of the ES. For this reason, there is no need to set significance thresholds to select the marker genes. The fixed number of marker gene set only needs to include sufficient significant marker genes to produce a robust ES see .Another important and useful property of SubMap is that it helps to identify an appropriate level of subclass granularity to select the best overall concordance. Typically in unsupervised clustering, the depth of splitting of the clustering solution is somewhat arbitrary. Even when a model selection method is used, there is the possibility of over-fitting to a dataset structure that represents batch effects or platform idiosyncrasies. As a consequence, a dominant structure in one data set might not necessarily show the best concordance with the other data set. A concrete example of this was shown in Example 3, where the second dominant splitting in Breast-B, rather than the first, showed the best concordance with Breast-A. The clustering of the subclass association matrix gives an indication of which candidate subclasses show the best concordance between the data sets.The use of more robust and generalized subclasses enables the better utilization of microarray data and the construction of better classification models. For example, the possibility of mapping the corresponding classes across a large collection of cross-platform or cross-laboratory datasets provides a method to increases the number of available samples related to a given phenotype. This translates to a better signal to noise ratio and thus improves the selection of biomarkers and the training of more accurate, robust, and generalizable supervised classifiers. Moreover, SubMap can provide a more precise stratification of the data for the application of predictive models. In Example 4, our method identified subpopulations in two independent data sets, where a clinical outcome predictor is more accurate than one modeled by all the samples. This additional resolution might help explain why an outcome predictor works only on a subset of the population. The integration and utilization of multiple datasets, representing a wider span of the biological space of interest, can both increase the accuracy of models as described above and provide models that are more interpretable, realistic, and biologically meaningful.a priori limit on the number of the candidate subclasses that can be associated as long as they have robust corresponding marker genes.The ability of SubMap to find common subtypes across data sets is, of course, dependent on the representation of the subtypes in those data sets, and the quality and granularity of their labeling. This is true whether the samples are labeled manually by, for example, a pathologist, or computationally by clustering. At a coarser granularity , a biological subclass of interest might be missed. On the other hand, a finer granularity  could yield an unbalanced comparison in selecting the marker genes. In addition, too fine a class splitting might yield multiple similar candidate subclasses, which might cause a failure to detect distinct marker genes resulting in lower sensitivity in detecting subclass association. However, we note here that, aside from the issue of increasing runtime, there is no SubMap is an exploratory method, and the optimal  number of the candidate subclasses would depend on the particular data sets with complex trade-offs. One possible approach is to define multiple possible candidate subclassifications in each data set, and evaluate the subclass association structure in all combinations of candidate subclassifications . To defiIn summary, SubMap is a method that provides more general molecular subclass identification and correspondence among a collection of microarray data sets. By allowing the integration of multiple data sets measured on different platforms, from different laboratories, it increases the possibility of extracting more meaningful biological subclasses and their corresponding biomarkers and classifiers. This has the potential to improve gene expression-based clinical classifiers where robustness, generalization, and reproducibility across platforms are paramount.http://www.broad.mit.edu/genepattern/.The entire methodology is implemented as the SubMap module of the GenePattern software and is freely available from http://www.broad.mit.edu/genepattern/).We started from data sets that were already normalized for their respective study without any additional normalization procedure to account for different platform derivation. For the signal intensity data generated by one-channel oligonucleotide microarrays, Affymetrix's GeneChip, we applied a lower threshold of 20U and a upper threshold of 16,000U. For the log2 transformed ratio data generated by cDNA microarrays, we first removed genes whose values were missing in more than 5% of the samples, and then imputed the missing values for the rest of the genes using a k-nearest neighbor algorithm Before marker gene selection, we used following gene filtering. For the oligonucleotide array data, only genes exhibiting at least 3-fold differential expression and an absolute difference of at least 100 units across the samples in the experiment were included. For the cDNA array data, only genes with an absolute log2 ratio greater than one and whose difference in log2 ratio across all the samples in the data set was greater than one were included.http://www.gene.ucl.ac.uk/nomenclature/), and multiple probe data corresponding to a single gene symbol was averaged. The number of genes remaining for our analyses of multiple tissue types, DLBCL, breast cancer, and DLBCL  data sets were 5565, 661, 1213, and 3795, respectively.Before applying the SubMap, each microarray probe ID was converted into its corresponding HUGO gene symbol . We determined subclass splitting by tracing from the root of the dendrogram. A subclass that contained at least 10% of the samples was considered as a possible candidate subclass. We set the maximum number of the candidate subclasses as four. Any other criteria could be used to define a candidate subclass.We emphasize here that SubMap is an approach to evaluate the relationship between otherwise determined subclass structures in independently generated data sets. Thus, any clustering method can be used to generate candidate subclass labels. Alternatively, biologically/clinically known phenotype information can be used as candidate subclass labels. For the specific examples described in the results section, we chose to use agglomerative hierarchical clustering to generate candidate subclass labels  [16]. WeAn and Bn candidate subclasses, respectively. First, we take the top 100 differentially expressed genes between a candidate subclass (CS) and the rest of the candidate subclasses (nonCS) the data set A. Such marker genes are identified for each candidate subclass in A. In data set B, genes are ordered according to their differential expression with respect to CS vs. nonCS. To evaluate differential gene expression between CS and nonCS, any measure, e.g., t-statistic, can be used. In this study, we used the signal-to-noise ratio (SNR), (CSμ-nonCSμ)/(CSσ+nonCSσ), where iμ and iσ denote, respectively, the sample mean and sample standard deviation within class CS or nonCSSuppose data set A and B have An marker gene sets and Bn gene rankings, class similarity is measured by calculating a gene set enrichment score (ES) and estimating its significance as previously described in j , a marker gene set of a candidate subclass Ai  is identified, and its enrichment or over-representation at the top of the list is measured by computing a ES (observed ESiBjA). The ES is essentially a Kolmogorov-Smirnov score weighted by the SNR.For every pair-wise combination of iBjA for 1000 random assignments of the subclass labels in Bj to generate a null distribution for ESiBjA. We sort the resulting ES scores in the null distribution, and estimate the significance of the observed ESiBjA from its rank in the list as a nominal p-value (piBjA). We repeat this process by interchanging the role of the data sets A and B. Thus, we compute piBjA and pjAiB.We recompute the ESiBjA and pjAiB in order to obtain combined mutual marker gene enrichment information as a measure of similarity of subclasses in the two input data sets. We use the Fisher inverse chi-square statistic, ij = -2F(log(AiBjp)+log(BjAip)) to summarize the two p-values . To correct for multiple hypothesis testing, a Bonferroni correction is applied. Alternatively, for cases with a larger number of candidate subclasses, we can estimate a FDR by multiplying each nominal p-value by An*Bn and dividing by the rank of the nominal p-value. Using these corrected p-values, we obtain a subclass association (SA) matrix, which represents the global association structure of the candidate subclasses in the two independent data sets. By applying two-way clustering on the SA matrix , it becomes possible to evaluate whether there are common subclasses that exist in both data sets.We now have For prediction analysis, we used k-nearest neighbor (k-NN) and support vector machine (SVM) modules from the GenePattern software. For k-NN, briefly, genes in the training data set were rank-ordered by the SNR. Using the selected marker genes, prediction was performed based upon a majority vote of the class membership of its k nearest neighbors in the training set weighted by the reciprocal of the cosine distance. The k and the number of marker genes were chosen so as to minimize the leave-one-out cross-validation (LOOCV) error rate in the training set.In Example 4, the prediction models were built based on LOOCV. Fractional use of each gene in the LOOCV models was calculated, and top 5 most frequently used genes were used for class prediction in the test data set .Kaplan-Meir analysis and the log-rank test were performed using GenePattern SurvivalCurve and SurvivalDifference modules, respectively.Table S1(0.02 MB XLS)Click here for additional data file.Table S2(0.02 MB XLS)Click here for additional data file.Table S3(0.01 MB XLS)Click here for additional data file.Figure S1Effect of the number of marker genes on the result of SubMap.(0.10 MB TIF)Click here for additional data file.Figure S2Two breast cancer data sets, Breast-A and Breast-B, are directly merged and clustered. (a) Each column was normalized by subtracting the column mean and divided by the column SD before clustering. (b) Gene expression data are converted to their rank in each column, and clustering was performed using the rank to compute the distance. Pearson correlation and the average linkage method were used for the hierarchical clustering.(0.08 MB TIF)Click here for additional data file.Figure S33 ∪ C4 in DLBCL-C as described in the Method section. Bar indicates fractional use of each gene in LOOCV models. Box indicates top 5 most frequently used genes in LOOCV that is used for prediction in D2 ∪ D3 in DLBCL-D. Genes shown in red are also included in a prediction model built using all samples.Genes used for the survival prediction in Example 4. Leave-one-out cross-validation (LOOCV) is performed using C(0.05 MB TIF)Click here for additional data file.Figure S41 and 2-A2) and three  candidate subclasses. In Breast-B, we defined sets of two (2-B1 and 2-B2), four , and six  candidate subclasses. SubMap was performed on all combinations of sets of the candidate subclasses. When the coarsest granularity  was assumed in Breast-B, we observed no significant subclass association (left heatmaps). When finer granularity was assumed for Breast-B (middle heatmaps), significant “two-class” correspondence started to appear, indicating the coarsest granularity in Breast-B was not appropriate to find significant subclass association. The finest granularity for Breast-A derived more significant associations (middle bottom heatmap). When the finest granularity was assumed in Breast-B, a small fraction of samples (6-B6) showed no association with any subclasses in Breast-A (right heatmaps), suggesting that this is too fine a granularity yielding weaker marker genes and lower sensitivity to capture a counterpart of 6-B6.Effect of granularity of the candidate subclasses on SubMap result. In Breast-A and Breast-B data sets in Example 3, the finest granularity  was defined as subclasses having at least 10% of the cohort. Each subclass was labeled by “number of subclasses”-“data set”-“subclass number”. In Breast-A, we defined sets of two (2-A(0.13 MB TIF)Click here for additional data file.Box S1Algorithm to generate a SA matrix.(0.29 MB DOC)Click here for additional data file.


A large-scale survey using single nucleotide polymorphism data from dbSNP provides insights into the evolutionary selection constraints on human proteins of different structural and functional categories. The rates of molecular evolution for protein-coding genes depend on the stringency of functional or structural constraints. The Ka/Ks ratio has been commonly used as an indicator of selective constraints and is typically calculated from interspecies alignments. Recent accumulation of single nucleotide polymorphism (SNP) data has enabled the derivation of Ka/Ks ratios for polymorphism (SNP A/S ratios).Using data from the dbSNP database, we conducted the first large-scale survey of SNP A/S ratios for different structural and functional properties. We confirmed that the SNP A/S ratio is largely correlated with Ka/Ks for divergence. We observed stronger selective constraints for proteins that have high mRNA expression levels or broad expression patterns, have no paralogs, arose earlier in evolution, have natively disordered regions, are located in cytoplasm and nucleus, or are related to human diseases. On the residue level, we found higher degrees of variation for residues that are exposed to solvent, are in a loop conformation, natively disordered regions or low complexity regions, or are in the signal peptides of secreted proteins. Our analysis also revealed that histones and protein kinases are among the protein families that are under the strongest selective constraints, whereas olfactory and taste receptors are among the most variable groups.Our study suggests that the SNP A/S ratio is a robust measure for selective constraints. The correlations between SNP A/S ratios and other variables provide valuable insights into the natural selection of various structural or functional properties, particularly for human-specific genes and constraints within the human lineage. It is well established that there are tremendous variations in rates of evolution among protein-coding genes. A central problem in molecular evolution is to identify factors that determine the rate of protein evolution. One widely accepted principle is that a major force governing the rate of amino acid substitution is the stringency of functional or structural constraints. Proteins with rigorous functional or structural requirements are subject to strong purifying (negative) selective pressure, resulting in smaller numbers of amino acid changes. Therefore, these proteins tend to evolve slower than proteins with weaker constraints. A classic measure for selective pressure on protein-coding genes is the Ka/Ks ratio , that isIn the past few years, advances in sequencing technology have led to a rapid accumulation of DNA variation data for human populations, including copy number variations and single nucleotide polymorphisms (SNPs). Currently, the dbSNP database  at the NIn the present study, we conducted a large-scale survey of SNP A/S ratios using SNP data from dbSNP. We first confirmed that the SNP A/S ratio is a good measure for selective pressure by showing its correlation with Ka/Ks from inter-species alignments and protein alignment conservation. We then obtained a variety of structural and functional properties from either database annotations or computational prediction methods and analyzed SNP A/S ratios for different classes of proteins and residues in an attempt to study the natural selection of these properties from the SNP perspective. Our comprehensive analysis provides: valuable insight into some features that have not been examined previously; independent confirmation of some previously established results; and additional data for areas where previous studies have had contradictory findings.We collected 13,686 human genes that have at least one validated coding SNP according to dbSNP. The analysis was limited to validated SNPs to ensure data quality. Overall, 45,538 coding-region SNPs and 1,529,119 intronic SNPs were identified in these genes, corresponding to SNP densities of 2.0 and 2.4 SNPs, respectively, per 1,000 nucleotides. The number of non-synonymous coding SNPs per non-synonymous site (A) is 0.00123, the number of synonymous coding SNPs per synonymous site (S) is 0.00439, and the A/S ratio is 0.28. The values of A and S are both two times more than what have been reported in a small study [p-value < 1e-04), which is in agreement with the neutral theory of molecular evolution. Analysis of data from chimpanzee and Old World monkey (Macaca mulatta) led to similar conclusions, although the Ka/Ks values may need to be corrected to subtract the contribution of SNPs due to relatively short evolutionary distance.To assess whether SNP A/S ratios from the current large-scale SNP data set provide a good measure for selective constraints, we first compared them with Ka/Ks ratios derived from inter-species alignments. We collected 9,759 human proteins with both validated coding-region SNPs and available human-mouse Ka/Ks data from Ensemble , binned We next investigated whether the conservation in protein sequences correlates with the SNP A/S ratio under the assumption that both the conservation at the protein sequence level and the SNP A/S ratio at the nucleotide level are indications for selective constraints. Using the position-specific alignment entropy (a measure for conservation) from PSI-BLAST profiles , we calcMany studies have been published addressing the correlation between evolutionary constraints and other variables, most of which were based on relatively small data sets. Having established the SNP A/S ratio as a good measure for selective constraints, we attempted to use the large-scale human SNP data set to revisit some of the features in the earlier studies, and also to investigate several protein properties that had not been examined before.p-value < 1e-04), but only marginally with tissue specificity . Since these four expression parameters correlate strongly with each other, we carried out partial correlation analysis [p-value = 0.499; Table S1 in Additional data file 1) or expression breadth . Expression breadth and mean expression level both remain significantly correlated with the A/S ratio when controlling one for the other . Peak expression level is highly correlated with mean expression level and its partial correlation patterns largely resemble those of mean expression level. It has recently been recognized that it is critical to control for expression when studying the statistical relevance of other variables as predictors for evolutionary rates, since many previously reported correlations became insignificant after this control. As expression breadth appeared to have the strongest correlation with the SNP A/S ratio in our data set among the four parameters, we chose to control for it in the following correlation analysis between selective constraints and other variables. The results did not change qualitatively when controlling for mean expression level instead.Until a few years ago, the prevalent theory in molecular evolution was that evolutionary rate is largely dependent on structural and functional constraints. Recently, increasingly more evidence suggests that there is a strong correlation between evolutionary rate and gene expression. It has been observed that highly expressed genes evolve slowly in bacteria , yeast , and mamanalysis  to identp-value < 1e-04; Figure p-value = 0.507). The difference in the A/S ratio did not change significantly when the expression breadth was controlled by Monte Carlo sampling  have the highest A/S ratios (0.76 for human and 0.66 for primates), whereas proteins traceable to all animals or other eukaryotes have much lower ratios of about 0.25 Figure . This isp-value < 1e-04). The correlation did not change upon controlling for expression breadth. Our analysis also showed that this correlation is only prominent for proteins shorter than 500 residues, and disappears for longer proteins  or numeric relative accessibility values. This underscores higher selective constraints on buried residues, possibly due to their importance in maintaining protein stability.Solvent accessibility measures the degree of an amino acid residue's exposure to the surrounding solvent. There have been a number of studies about the effect of mutations on solvent accessibility and its implication in human diseases; most of them were based on relatively small collections of SNPs in known protein structures. The general consensus was that buried residues are less likely to vary and their mutations are more likely to cause disease ,39. We oWe also investigated selective constraints upon different protein structure conformations. We first grouped all residues into different secondary structure conformations  according to predictions by PSIPRED . Signifip-value < 1e-04; Figure S3a in Additional data file 1). There is no difference between membrane proteins and non-membrane proteins, membrane segments and non-membrane segments according to TMHMM [Subcellular localization is an important aspect of protein function. There have been conflicting reports about the correlation between protein subcellular localization and evolutionary rate. While a previous survey of human SNPs in 2002 did not find a significant correlation of selective pressure against deleterious non-synonymous SNPs with localization , a more to TMHMM  predictito TMHMM , a hieraWe next studied the variation in SNP distribution of functional categories based on GO annotations. A/S ratios were calculated for 176 GO biological process categories and 152 molecular function categories that have at least 20 genes in our data set. As expected, there are dramatic differences in selective constraints among different categories: A/S ratios range from 0.72 for 'sensory perception of smell' to 0.07 for 'protein kinase C activation' Table . In termWe also sought to quantify the selective pressure on protein families. Of the 13,686 proteins in our data set, 10,629 can be assigned to at least one Pfam  family up-value < 1e-04; Figure et al. study [Knowledge about the degree of selection for disease-related genes can help us understand the etiology of human diseases. An early study found that human disease genes evolve faster at both synonymous and non-synonymous sites than non-disease genes, and Ka/Ks ratios of disease genes are 24% higher . Althougl. study  includedl. study . This isSaccharomyces cerevisiae. After an initial report that yeast proteins with more interaction partners evolve slowly [p-value < 1e-04), and it dropped to -0.106  after controlling for both mean expression level and expression breadth. The correlation between protein abundance and high connectivity in the interaction network could be either a real biological phenomenon or experimental bias; for example, mass spectrometry-based protein complex pulldown experiments are more likely to identify interaction partners for abundant proteins. When we included only yeast two-hybrid interactions in our analysis, which are supposedly less biased with respect to intrinsic expression levels, the correlation between connectivity and abundance largely disappeared, except for the proteins with no interaction partners in the database  after controlling for expression. Nevertheless, the partial correlations were still statistically significant in both the yeast two-hybrid interaction set and the all interaction set. Our analysis supports the idea that the correlation between evolutionary rate and connectivity in the interaction network can, in part, be explained by protein abundance and that some of the correlation may result from experimental bias. Similar to all the conflicting studies in yeast, it is likely that this result is inconclusive and may vary from data set to data set.It is still debatable whether there is any correlation between protein-protein interaction and selective pressure. Most studies so far have been based on data from the budding yeast e slowly , severale slowly , or thate slowly . The late slowly . We obtae slowly  and examp-value < 1e-04).A recent study suggested that protein evolution is strongly affected by mRNA splicing, in addition to the biology of the protein . Based oThe average SNP density in our data set is about 2 SNPs per 1,000 nucleotides. Although we have limited our analysis to proteins with at least one validated coding SNP, many proteins in our set still have no non-synonymous SNPs or synonymous SNPs. Therefore, it is neither practical nor reliable to measure the selective constraints on individual proteins using the SNP A/S ratio. Nevertheless, we have demonstrated that when a group of proteins (or residues) are measured together, the measure can be quite robust and often in good agreement with Ka/Ks for divergence. For example, the SNP A/S ratio for proteins with paralogs in our data set is 0.47, very close to the Ka/Ks ratio of 0.45 for duplicated mammalian genes reported earlier . AlthougThere are two unique advantages of using SNP A/S ratios to measure selective constraints. First, for human-specific genes and genes whose ortholog relationships can not be determined reliably, inter-species Ka/Ks ratios can not be estimated. For example, among the 13,686 genes in our data set, human-mouse Ka/Ks ratios for 3,937 genes are not available through Ensemble. In those cases, SNP A/S ratios provide an alternative way of measuring selective pressure by the same evolutionary principle. Second, the SNP A/S ratio is a direct measure of the selective constraints specific to the human lineage, which can not be obtained from Ka/Ks for species divergence.In comparison with other simpler measures used in some earlier SNP studies , such aTo address the issue of determinants of molecular evolution, many studies have been published examining the correlation between variables that characterize the evolution, expression and function of genes. However, as noted in the Results section, there are many controversies among those reports. It is clear that the validity and significance of correlations depend on many factors, including feature variables, data sets, and the correlation measure. Here, we provide the first large-scale study of SNP A/S ratios, and reported correlations between the ratio and a number of variables; not surprisingly, some are not in agreement with previous studies. Although further studies are still needed to draw definitive conclusions about the major determinants, we agree with many others that it is likely that many of these variables correlate with each other in some way, and some of them are secondary effects rather than primary determinants . SeveralMolecular evolution is a dynamic process, with strong implications for both differences between species and variations within a given species. SNPs in the human genome perhaps capture a glimpse of this dynamic process and, therefore, could offer key insights missed by conventional cross-species comparisons. Here, we first established the SNP A/S ratio as a reliable metric for studying the selective constraints for molecular evolution, and then used this metric to systematically investigate a large number of protein features that contribute to differences in molecular evolution rate. Our study provided the first such large-scale survey based on SNP A/S ratios, leading to novel insights into features that have not been examined before and clarification of findings that were contradictory in the literature.We first filtered the human entries in the NCBI's Entrez Gene database  by the pmRNA expression data were obtained from Gene Expression Atlas ,67. Onlyn = 54 is the number of human tissues examined here, Sj is the expression level in each tissue, and Smax is the highest expression level of the probe set across all tissues. When a gene has multiple probe sets, its expression levels and tissue specificity were represented by the probe set with the highest mean expression level.where Affymetrix present/absent calls were used to calculate the breadth of the expression. A probe set was considered 'present' in a tissue if it had 'present' calls in no less than half of the samples in that tissue, and the expression breadth of a probe set was defined as the number of tissues in which the probe set was 'present'. When a gene has multiple probe sets, its expression breadth was represented by the probe set with the highest value of breadth.Ka/Ks ratios from human-mouse alignments were downloaded from Ensembl . The infWe obtained protein structure features by the following computational methods using their default parameters: two-state (exposed or buried) solvent accessibility by PROFacc , signal GO annotations  were extSequences within 70 bp of an exon-intron junction were collected in the same way as described in . BrieflyThe features we considered can also be divided into two classes: protein-level features and residue-level features. A protein-level feature describes the property of an entire protein, for example, whether a protein has a transmembrane helix or not; in contrast, a residue-level feature describes the property of a subset of residues within a protein, for example, whether a residue resides in the transmembrane helix or not.The SNP A/S ratio, also known as the Ka/Ks ratio for polymorphism, is defined as the ratio of the number of non-synonymous SNPs per non-synonymous site to the number of synonymous SNPs per synonymous site. The numbers of synonymous sites and non-synonymous sites were calculated using the method of Miyata and Yasunaga . The ratp-values of the differences were obtained from those 10,000 sets of A/S ratios. For protein-level feature groups, re-samplings were performed within each group, so that those re-sampled data sets had the same number of proteins in each group as the original data set had. For residue-level feature groups, the re-sampled data sets were constructed by re-sampling the 13,686 proteins in our entire data set.We performed bootstrap re-sampling analysis to assess the statistical significance of the differences observed in A/S ratios between different groups. We obtained 10,000 bootstrap replicates by re-sampling with replacement from the original data set. A/S ratios for different groups were calculated for each replicate, and confidence intervals and Monte Carlo samplings were used to assess the differences in A/S ratios between groups when controlling for expression parameters. Briefly, the distribution of expression breadth (or expression level) for the group with the smallest number of genes was set as the target distribution; genes in all other groups were then sampled without replacement using Monte Carlo simulation so that all groups had the same distribution of the expression breadth (or level). The effectiveness of Monte Carlo samplings was confirmed by Kolmogorov-Smirnov tests. An example is shown in Figure S1 in Additional data file 1. For each group, 100 samplings were performed, and the A/S ratio for the group was taken as the mean of A/S ratios from the 100 Monte Carlo samples.p-values were used throughout the study. When controlling for expression parameters, Kendall's partial correlation between x and y controlling for z was calculated as:Since the average SNP density in our data set is about 2 SNPs per 1,000 nucleotides, and many proteins in our data set have either no non-synonymous SNPs or no synonymous SNPs, it is not possible to reliably calculate the correlation between the SNP A/S ratio and other continuous variables using each protein as a data point. We chose to randomly group every six proteins together as a data point so that, on average, each data point had roughly the same number of SNPs as the reported number of single-nucleotide substitutions (1.23%) between the human and chimpanzee genomes . Non-parThe random grouping was performed 100 times. The correlation and partial correlation coefficients were computed from these 100 samples, and the medians of those 100 sets of coefficients were reported.CGC, Cancer Gene Census; COSMIC, Catalogue of Somatic Mutations in Cancer; GO, Gene Ontology; GPCR, G protein-coupled receptor; Ka, non-synonymous substitutions per non-synonymous site; Ks, synonymous substitutions per synonymous site; OMIM, Online Mendelian Inheritance in Man; SNP, single nucleotide polymorphism.ZZ and JL designed the study. JL and YZ collected the data and performed the data analysis. XL participated in the statistical analysis. JL and ZZ drafted the manuscript. All authors read and approved the final manuscript.The following additional data are available. Additional data file Table S1 presents correlation and partial correlations between SNP A/S ratios and expression parameters. Figure S1 shows an example of using Monte Carlo sampling to get the same distributions of expression breadth for different groups of proteins. Figure S2 and S3 demonstrate that for most variables in our study, the differences in A/S ratios between groups do not change qualitatively after controlling for expression breadth.Click here for file


However, non-linear variation associated with the technology can make data interpretation unreliable. Therefore, methods to correct this kind of technical variation are critical. Here we consider a method to reduce this type of variation applied after three common procedures for processing microarray data: MAS 5.0, RMA, and dChip® processed data. Datasets with unbalanced numbers of up and down regulated genes seem to be particularly susceptible to this type of intensity-dependent technical variation. Unbalanced gene regulation is common when studying cancer samples or genetically manipulated animal models and preservation of this biologically relevant information, while removing technical variation has not been well addressed in the literature. We propose a method based on using rank-invariant, endogenous transcripts as reference points for normalization (GRSN). While the use of rank-invariant transcripts has been described previously, we have added to this concept by the creation of a global rank-invariant set of transcripts used to generate a robust average reference that is used to normalize all samples within a dataset. The global rank-invariant set is selected in an iterative manner so as to preserve unbalanced gene expression. Moreover, our method works well as an overlay that can be applied to data already processed with other probe set summary methods. We demonstrate that this additional normalization step at the "probe set level" effectively corrects a specific type of technical variation that often distorts samples in datasets.We commonly observe intensity-dependent technical variation between samples in a single microarray experiment. This is most common when MAS 5.0 is used to process probe level data, but we also see this type of technical variation with RMA and dChipWe have developed a simple post-processing tool to help detect and correct non-linear technical variation in microarray data and demonstrate how it can reduce technical variation and improve the results of downstream statistical gene selection and pathway identification methods. These redundant probes are called a "probe set" and summarizing each probe set to arrive at a robust value representing the abundance of the associated transcript is one of the first steps in any analysis . This report focuses on data from the Affymetrix®, but should work on other operating systems which are supported by the "R" statistical platform, including MacOS and Linux. Multiple "R" versions have been tested, including version 2.4.0. Future updates and additional information will be provided at: Sears Lab when available.This tool is available for free for non-commercial use [see Additional file . Affymetrix® CEL files were processed using the MAS 5.0 and RMA algorithms implemented in the BioConductor package: affy Version 1.5.8 [® 2006 . Gene Set Enrichment Analysis (GSEA) was performed using the author's R language implementation version GSEA.1.0.R [All statistical analysis, method implementation, and graphing for figures, was performed using the open-source programming language R, Versions 2.0.1 – 2.4.0 , and Bioon 1.5.8   scaled, no values will be less than negative 2.0. The data is then log2 scaled. Next, a rank matrix of the same dimension as the input matrix is calculated. For each column of the input matrix, the rank (order in a sorted list) of each transcript in that column is calculated and used to populate the rank matrix. The variance of each row of the rank matrix is calculated and the rows are then ranked based on their variance. The rows with the highest variance are discarded. A subset of the original log2 scaled matrix is produced, containing only the rows corresponding to the remaining rows in the rank matrix. Using this new matrix, another rank matrix is calculated. This process is repeated in an iterative manner four (4) times.  for the Global Rank-invariant Set (GRiS) from the original row count and dividing the result by four.) The remaining transcripts are considered the GRiS. Next, reference values for each of the transcripts in the GRiS are obtained by calculating the trimmed mean from all samples for each of the respective transcripts. A calibration curve is then calculated for each sample by comparing the GRiS values from the given sample to the GRiS reference values using an M vs. A plot. The plotted points representing the GRiS should be centered about the horizontal line where M = 0. The calibration curve is obtained by drawing a smoothed line through these points using lowess [At the heart of the GRSN method is the selection of a global rank-invariant set. Microarray data is passed to GRSN as a matrix of expression values with rows representing transcripts (probe sets) and columns representing samples. The Affymetrixg lowess  as impleg lowess .Quality control graphs are plotted for each sample as it is normalized. For each sample, three M vs. A scatter plots are generated: The first is pre GRSN, and compares the given sample to the virtual reference sample . The second shows the GRiS transcripts only, both before and after adjustment, and includes the lowess smoothed calibration curve. The third shows the given sample, after adjustment, compared to the virtual reference sample.GRSN was evaluated using simulated data produced by adding simulated gene expression and simulated non-linear artifacts (skew) to a cell culture dataset containing 10 replicated controls . The 10 CP developed the GRSN application, wrote all of the software, created the application web site, prepared and analyzed all of the data presented in this study, and wrote the manuscript. RS supported and helped CP with all of the data analysis in this study, figure preparation, and with writing of the manuscript. GB and MK-M provided unpublished data for analysis and scientific discussion of the GRSN algorithm.Additional examples of GRSN. Scatter plots showing the results for selected samples when applying GRSN to various datasets.Click here for fileAdditional examples of a global skew corrected by GRSN. Scatter plots showing global skew between experimental conditions before and after correction with GRSN.Click here for fileMini website with code and instructions for GRSN method. This file contains a zipped directory structure to provide a mini website with code and directions which provide an implementation of the GRSN method using the open source “R” environment.Click here for file


We consider both univariate- and multivariate-based feature selection for the problem of binary classification with microarray data. The idea is to determine whether the more sophisticated multivariate approach leads to better misclassification error rates because of the potential to consider jointly significant subsets of genes (but without overfitting the data).We present an empirical study in which 10-fold cross-validation is applied externally to both a univariate-based and two multivariate- -) based feature selection processes. These procedures are applied with respect to three supervised learning algorithms and six published two-class microarray datasets.Considering all datasets, and learning algorithms, the average 10-fold external cross-validation error rates for the univariate-, single-stage GA-, and two-stage GA-based processes are 14.2%, 14.6%, and 14.2%, respectively. We also find that the optimism bias estimates from the GA analyses were half that of the univariate approach, but the selection bias estimates from the GA analyses were 2.5 times that of the univariate results.We find that the 10-fold external cross-validation misclassification error rates were very comparable. Further, we find that a two-stage GA approach did not demonstrate a significant advantage over a 1-stage approach. We also find that the univariate approach had higher optimism bias and lower selection bias compared to both GA approaches. DNA microarray technology has greatly influenced the realms of biomedical research, with the hopes of significantly impacting the diagnosis and treatment of diseases. Microarrays have the ability to measure the expression levels of thousands of genes simultaneously. They measure how much a given type of messenger RNA (mRNA) is present in a tissue sample at a given moment. The wealth of gene expression data that has become available for microarray data analysis has introduced a number of statistical questions to tackle. Some questions are targeted towards various preprocessing stages of a microarray experiment such as RNA hybridization to arrays, image processing, and normalization, while others are geared towards assessing differential expression and identifying profiles for classification and prediction. Within the framework of tumor classification, the types of goals that have been explored include discovering or identifying previously unknown tumor classes, classifying tumors into previously known classes, and identifying “marker genes” that characterize various tumor classes.N is usually much larger than the number of feature variables p. However, in the context of microarrays, the number of tissue samples N is usually between 10 and 100, significantly smaller than the thousands of genes considered in a typical microarray analysis. This presents a number of problems to a prediction rule in a discriminant analysis setting. The prediction rule may not even be able to be formed using all p variables, as is the case with Fisher’s linear discriminant analysis , i = 1, ..., N and j = 1, ..., p). Each value xij corresponds to the expression level for gene j in sample i. Each sample would have associated with it a gene expression profile xi =  ε Rp, along with its class designation yi. This variable serves as the response, or dependent variable, and can take one of two predefined values from {0, 1}. Using the observed measurements X, a classifier for two classes is thus a mapping G : Rp → {0, 1}, where G(x) denotes the predicted class, ypred = c, c ε{0, 1}, for a sample with feature vector x.Gene expression data for L = {, , . . . , }, constitute the training (or learning) set. The training set is used to construct a classifier, which is then used to predict the classes of an independent set of samples . This way, the class yi pred,  predictions for each test set expression profile xi can be made. Of course, with the true classes yi,  of the test set known, a misclassification error rate (MER) can then be computed.The samples already known to belong to certain classes, In general, feature (variable) selection is an important aspect of classification problems, since the features selected are used to build the classifier. Careful consideration should be given to the problem of feature subset selection with high-dimensional data. With respect to microarray data, this of course amounts to reducing the number of genes used to construct a prediction rule for a given learning algorithm. There are several reasons for performing feature reduction. Whereas two variables could be considered good predictors individually, there could be little to gain by combining the two variables together in a feature vector. It has been reported that as model complexity is increased with more genes added to a given model, the proportion of training samples (tissues) misclassified may decrease, but the misclassification rate of new samples  would eventually begin to increase; this latter effect being the product of overfitting the model with the training data . Furthert-test used to measure the degree of gene expression difference between two types of samples (groups). The basic idea with univariate feature selection is to create a ranked-list of genes based on their individual scores . With respect to multivariate feature selection, the basic idea is to consider the predictive ability of groups of genes together. This study implements a particular type of multivariate feature selection technique–an evolutionary algorithm known as the genetic algorithm (GA), the basics of which are introduced in the following section.Feature selection can be performed in a univariate or multivariate fashion . The most common approach to univariate feature selection, which has been used extensively for years in the context of binary classification problems, is the simple selection, which parallels the “survival of the fittest” notion of evolution. In each generation, the fitness of every solution in the population is evaluated, multiple individuals are stochastically selected from the current population (based on their fitness). The “most fit” members of the population survive to be candidates for the next generation of solutions (next population), whereas the “least fit” members are dismissed. The fitness function can be any of a number of metrics or supervised learning algorithms . The process of selection is also what drives the selection of the initial population of solutions from the original gene pool. Selected solutions then become part of the process of crossover, which parallels the crossover of DNA strands that occurs during reproduction in living organisms. This genetic operation seeks to combine elements of existing solutions together and form new and potentially better solutions (“offspring”) with some features from each “parent.” These new solutions comprise the subsequent generation (population) of solutions. The third basic process of a GA is mutation, which is used to maintain genetic diversity of solutions from one generation to those of the next generation. This process parallels the role of biological mutation within an organism’s DNA in natural evolution. The mutation operation in a GA usually involves a very small probability that an arbitrary gene from some newly-created solutions will be replaced with a different gene from the original gene pool. Ultimately, through these three genetic operations, optimal  candidate solutions are generated after evolving through a pre-specified number of generations. More details on the actual parameterization of the GA used in this study, including a schematic of the basic GA process, are provided in Methods.The basic premise of a GA is to apply the principles of natural evolution and selection as a means of determining an optimal solution to a feature subset selection problem. Genetic algorithms begin with an initial population of randomly chosen candidate solutions  to use for classification of a given sample as one of two types. Each solution is of a pre-specified length . Through several basic processes, these solutions then evolve toward better solutions . One of the basic processes used is that of K-fold CV” heading. Here, the training set of samples is divided into K non-overlapping subsets of (roughly) the same size. One of the K subsets is “held-out” for testing, the prediction rule is trained on the remaining K − 1 subsets, and an estimate of the error rate can then be obtained from applying each stage’s prediction rule to its corresponding test set. This process repeats K times, such that each subset is treated once as the test set, and the average of the resulting K error rate estimates forms the K-fold CV error rate. The whole K-fold CV process could be repeated multiple times, using different partitions of the data each run and averaging the results, to obtain more reliable estimates. Leave-one-out CV (LOO CV) represents the extreme case of K-fold CV. This type of CV occurs when K = N, the number of samples. In this case, each sample serves as its own test set. Although it is nearly unbiased, it is generally highly variable and requires considerable computation time. At the expense of increased computation cost, repeated-(10-) run CV has been recommended as the procedure of choice for assessing predictive accuracy of the classification of microarray data  is very often used, especially when one does not have the luxury of withholding part of a dataset as an independent test set and possibly even another part as a validation set . Further, the repeatability of results on new data can be assessed with this approach. In general, all CV approaches can fall under the “ray data . With miray data .Although the intention of CV is to provide accurate estimates of classification error rates, using CV in this manner means that any inference would be made with respect to the classifier building process only. Leaving out feature selection from the cross-validation process will inevitably lead to selection bias, as the feature selection would not be based on the particular training set for each CV run. Hence, overly optimistic error rates would be obtained. To prevent this selection bias from occurring, an “external”, or “honest”, cross-validation process should be implemented following the feature selection at each CV stage  for 2000 genes across 62 samples. The total intensities for the genes were obtained using the mean filtered (perfect match–mean match), or PM-MM, intensity. To compensate for variations between arrays, the intensity of each gene on an array was divided by the mean intensity of all genes on the array and multiplied by a nominal average intensity of 50. The binary classes used for analysis are normal (22 samples) and tumor (40 samples). As discussed in , five coThis dataset consists of gene expression levels from Affymetrix chips (HuGeneFl). The oligonucleotide arrays have 7129 probe sets over 72 samples. The binary classes used for analysis are acute myeloid leukemia  and acute lymphoblastic leukemia . Intensity values were re-scaled such that overall intensities for each chip were equivalent. This re-scaling was done by fitting a linear regression model using the intensities of all genes with “P” (present) calls in both the first sample (baseline) and each of the other samples. See  for moreThis dataset consists of gene expression levels measured from Affymetrix high-density oligonucleotide chips (U95Av2) using the GeneChip software. Each array contains 12625 probe sets over 50 samples. The binary classes used for analysis are glioblastoma (28 samples) and anaplastic oligodendroglioma (22 samples). The downloaded raw expression values were previously normalized by linear scaling such that the mean array intensity for active (“present”) genes was identical for all the scans. See  for moreThis dataset consists of gene expression levels measured from Affymetrix high-density oligonucleotide chips (HuGeneFl) using the GeneChip software. Each chip contains 7129 probe sets. To facilitate the binary classification framework, dataset ’A2’ from the project website was used, in which 60 medulloblastoma (MD) samples formed one class and the remaining 30 samples classified as “Other” for the second class , 10 atypical teratoid/rhaboid tumor (AT/RT), 6 supratentorial primitive neuroectodermal tumors (PNET), and 4 normal cerebellum samples). See  for moreThis dataset consists of gene expression levels measured from Affymetrix chips (HuGeneFL) using the GeneChip software. Each oligonucleotide array contained 7129 probe sets over 77 samples. The two classes used for analysis are diffuse large B-cell lymphoma  and follicular lymphoma . See  for moreThis dataset consists of gene expression levels measured from Affymetrix chips (HU95Av2) using the GeneChip software. The number of arrays available for analysis was 102, with each containing 12600 probe sets. The two classes used for analysis are normal (50 samples) and prostate cancer (52 samples). See  for moreFirst of all, to get an idea of how effective the two GA-based feature selection processes were at selecting genes that would otherwise not be considered “top genes” from a univariate screening procedure, all three feature selection approaches were implemented in a resubstitution setting, in which all samples were used for each dataset. From Both 10-fold external and internal CV were performed, using both univariate- and GA-based feature subset selection. The idea was to consider the problem of how best to evaluate prediction rules formed from models such that the effects of optimism bias, selection bias, and “total” bias are properly taken into account, where these bias estimates are defined as follows:In these analyses, the empirical results were based on the same six datasets and same three learning algorithms as this study. Considering all datasets, learning algorithms, and gene subset sizes together, we found that for the results based on univariate feature selection, the average optimism, selection, and total bias estimates were only 4%, 3%, and 7%, respectively. The average optimism, selection, and total bias estimates for the GA-based results were 2%, 8%, and 10%, respectively, and those for the “GA-GA”-based results were 2%, 7.5%, and 10%, respectively. Hence, the optimism bias, incurred from using the same data to both train the classifier and estimate the classifier’s performance, from each of the GA-based analyses was half that of the univariate-based results. However, the selection bias, incurred from using the same data to both select the gene subsets and estimate the classifier’s performance, from each of the GA-based analyses was 2.5 times that of the univariate-based results.The 10-run external CV results for each dataset and classifier combination, across a number of gene subset sizes, are shown in Finally, it is interesting to consider the means and standard deviations of the 10-fold external CV misclassification error rates averaged across all datasets and classifiers, for each of the three approaches to feature subset selection. A plot of these results is shown in k-NN. The gene selection method implemented was to select the p genes with largest ratio of between to within-sum-of-squares. In this study, repeated (150) runs of training/test set partitions were performed, with feature selection done only on each training set. The ratio of training to test set samples was 2:1. No cross-validation study was performed. More recently, univariate screening with both a simple t-test and a rank-based t-test (Wilcoxon Test) were used to analyze a couple of published two-class microarray datasets  as their learning algorithms. The gene selection method used was univariate in nature (t-tests). To assess the predictive ability of the prediction models built, the authors used K-fold CV, LOO CV, Monte Carlo CV (MCCV), and .632 + Boostrap , a more recent study that incorporates the GA as the feature selection algorithm is . Here, tFinally, the GA was also implemented as part of a comparative study in . Here, tk-NN, Mahalanobis distance, is employed in the GA algorithm implemented in this research. More details on the GA methods used in this study are provided in the Methods Section.The current research builds on the findings of the studies of , Xiong , Dudoit, Molina, and Wa, in the Repeated- (10-) run 10-fold external cross-validation was applied to each of six datasets, using each of three different learning algorithms. With the external CV approach, the feature selection was performed at each stage of the CV process, based only on the training set partitions of each stage and hence external to the test sets used to evaluate the models. The average error rates across all the classifiers and gene set sizes were very comparable among the univariate and two multivariate feature selection approaches, as they were all 14%. In terms of datasets, only the Nutt dataset had noticeably higher error rates across classifiers and subset sizes than those of the other datasets, as they were, on average, above 27% for each of the three feature selection approaches. Ultimately, the misclassification rates did not vary significantly by dataset, for five of the six datasets at least, suggesting that these results should generalize well to other clinical microarray datasets. The same generalization ability should hold with respect to classifiers, since the three classifiers used function in different ways, and since there is no clear reason to suspect that the results are connected to the method of classification.For each of the six datasets used in this study, we have shown that although a multivariate-based approach to feature subset selection may have greater potential to select combinations of genes that are jointly discriminatory than would a method that combines individually predictive genes, there was no clear advantage of the more computationally intensive GA approaches over the simpler univariate ones in estimating the prediction error for a classification rule constructed from a selected subset of genes from microarrays. Considering all classifiers, subset sizes, and learning algorithms, we found that the optimism bias estimates from the GA analyses were half that of the univariate approach, while the selection bias estimates from the GA analyses were 2.5 times that of the univariate results. This higher selection bias suggests that selecting genes in multivariate models using a GA may be more likely to select spurious genes than would be the case with a univariate-based approach. This finding makes sense in that since the selection bias measures the bias in the estimate of CV prediction error due to feature selection, one would suspect that it would be higher with the multivariate feature selection approach since this approach searches a much higher dimensional model space when finding the features. Thus, with the multivariate feature selection approach, it would naturally be more possible to include spurious genes in candidate models, which can be seen as overfitting the data. That is, considering the notion of overfitting to mean that too much flexibility is allowed in the model space, such that the models trace the data too closely, likely select spurious features of the given data set, and hence do not accurately generalize to independent validation data, it would be safe to say that the GA-based methods tend to overfit the data. Ultimately, whether a univariate or a GA-based feature selection approach is used, the presence of optimism and selection bias should be taken into account through the use of external CV.k-NN (k=3 in this study). For more details on each of these classifiers, the reader should refer to , DLDA, and refer to  and , was used to perform gene selection. For more details on the GA in general, the reader is referred to . Both a refer to .p genes of each dataset. The number of d-gene solutions (“chromosomes”), M, selected by each implementation of the single-stage GA is 1000, so over R = 10 iterations, a “superpopulation” of 10000 candidate solutions are obtained. The number of generations, G, to run for each iteration of the GA was set to 250, which we found was large enough to ensure convergence of the 1000 solutions. Of the 10000 solutions, there will be as many as 10 unique solutions that will have converged after 250 generations, corresponding to each of the 10 runs of the GA). Of these 10 converged solutions, the one with the best fitness score is retained for use in the classification algorithm, which in-turn preserved the actual solutions discovered by the GA . Note tp genes, as done in the standard GA process just described. However, for the second GA stage, the algorithm is applied to a reduced initial gene pool, based on the initial GA’s selection results. This new gene pool is composed of the union of all genes selected among the final generation’s population of 1000 d-gene solutions from the first stage of GA, for all 10 iterations of the GA. That is, the second stage’s GA procedure uses as its initial gene pool all genes that appeared at least once among the “superpopulation” of 10000 solutions obtained from the initial GA stage. This way, genes that may appear in just a small proportion of the 1000 final solutions of any given iteration, but still appear in multiple iterations of the GA, now have a better chance of being considered for use in building classifiers. Thus, the idea behind the second implementation of the GA would be to attempt to select the ’best of the best’ genes from a given training dataset. It should be noted that for the second stage GA, the number of generations remained at 250, but the number of d-gene solutions selected by each implementation of the GA was reduced to 500, since the initial gene pool was reduced considerably. Finally, note again that for the external 10-fold CV procedure, this process is again applied within each of the 10 stages of the 10-fold CV.For the two-stage approach (“GA-GA”), the first stage GA begins with all


Over the past decade, gene expression microarray studies have greatly expanded our knowledge of genetic mechanisms of human diseases. Meta-analysis of substantial amounts of accumulated data, by integrating valuable information from multiple studies, is becoming more important in microarray research. However, collecting data of special interest from public microarray repositories often present major practical problems. Moreover, including low-quality data may significantly reduce meta-analysis efficiency.2DB is a human curated microarray database designed for easy querying, based on clinical information and for interactive retrieval of either raw or uniformly pre-processed data, along with a set of quality-control metrics. The database contains more than 10,000 previously published Affymetrix GeneChip arrays, performed using human clinical specimens. M2DB allows online querying according to a flexible combination of five clinical annotations describing disease state and sampling location. These annotations were manually curated by controlled vocabularies, based on information obtained from GEO, ArrayExpress, and published papers. For array-based assessment control, the online query provides sets of QC metrics, generated using three available QC algorithms. Arrays with poor data quality can easily be excluded from the query interface. The query provides values from two algorithms for gene-based filtering, and raw data and three kinds of pre-processed data for downloading.M2DB utilizes a user-friendly interface for QC parameters, sample clinical annotations, and data formats to help users obtain clinical metadata. This database provides a lower entry threshold and an integrated process of meta-analysis. We hope that this research will promote further evolution of microarray meta-analysis.M Rapid accumulation of vast amounts of microarray data in public databases like Gene Expression Omnibus (GEO)  and ArraFirst, identifying suitable studies for meta-analysis is a time-consuming process because experimental information is often stored in a free-text format in a data warehouse. Although most microarray repositories have adopted the Minimum Information about a Microarray Experiment (MIAME)  standardSecond, re-processing raw data is of great importance for integrating data from multiple datasets ,10,11, bThird, using pre-processed data by different algorithms will introduce variations into the results of meta-analysis ,13. DiffFourth, several investigators have suggested considering data quality within the context of microarray meta-analysis ,11,13. R3D [A number of databases or web servers have recently been developed to tackle these problems. For example, Celsius  is a dat3D  has coll3D  has exte3D  re-annot2DB, an expert curated database, to solve microarray data retrieval, annotation, pre-processing, and quality-assessment problems. M2DB contains more than 10,000 previously published Affymetrix array data, re-annotated with controlled vocabularies from ontologies (most from NCI Thesaurus), according to available clinical information. Samples of interest can be easily queried, based on five clinical annotations: "Disease State," "Disease State Suppl.," "Disease Location," "Organism Part," and "Organism Part Subtype." Raw data were retrieved from HG-U133A and HG-U133 plus 2.0 and were uniformly pre-processed using Affymetrix Microarray Suite 5 (MAS5) [2DB web query interface. MAS5call and labeling efficiency values (LEVs) [2DB efficiently facilitate the search and retrieval process, as well as ensuring the reliability of human clinical microarray metadata. In summary, M2DB provides human curated annotations, raw data, uniformly preprocessed data, and sets of QC metrics, and significantly improves the quality and comparability of microarray metadata generated by different laboratories.This study develops M5 (MAS5) , robust 5 (MAS5) , and GC-5 (MAS5) . Quality5 (MAS5) , Mahalan5 (MAS5) , and Par5 (MAS5) , package5 (MAS5) , were prs (LEVs)  are supp2DB server.As illustrated in Figure 2DB contains more than 10,000 Affymetrix GeneChip arrays from 192 experiments in ArrayExpress . All microarray raw data were pre-processed using three different algorithms: MAS5, RMA, and GCRMA as implemented in the Bioconductor packages. RMA and GCRMA processed data on a multi-array basis; therefore all arrays of the same platform were uniformly pre-processed to reduce variance.Raw intensity data (CEL files) generated using Affymetrix HG-U133A and HG-U133 plus 2.0 platforms were retrieved from GEO and ArrayExpress. Arrays performed using samples other than human clinical specimens, such as cell lines, primary cells, and transformed cells, were excluded. In addition, datasets without links to publications were also excluded except Expression Project for Oncology (expO) (GSE2109). A total of 69% of published arrays which contain raw data were performed by human clinical samples. The other 31% of arrays were hybridized to cell lines, primary culture cells, transformed cells, etc. About 8% of clinical arrays were uploaded to repositories more than once. We removed the redundant arrays according to the generation date of CEL files. Eventually, M2DB, based on several factors. First, the Affymetrix platform provides a consistent and reliable system with a high level of reproducibility [2DB is providing relative "pure" data to reduce the variance of data processing in meta-analysis. For these reasons, we excluded non-Affymetrix platforms from M2DB.The Affymetrix microarray system is recognized as naturally suited for meta-analysis  and was cibility ,14,33. Scibility . On the cibility . MoreoveTo ensure annotation consistency and make the retrieval process more efficient, clinical information for each sample was manually curated, based on data obtained from GEO, ArrayExpress, and published papers. As shown in Figure 2DB for query because it is frequently not available.These five characteristics were derived from MGED ontology, but some were modified for easier querying. In order to obtain an accurate clinical annotation from the free-text descriptions in GEO, ArrayExpress, and related papers, terms used in annotations are controlled vocabularies from existing ontologies . This research manually extracted relative information from the free-text description of each sample, then identified accurate terms and unified the synonyms via the Bioportal , Ontolog2DB provides QC metrics generated by three R packages, SimpleAffy, MDQC, and PMVO. SimpleAffy provides a set of QC factors developed based on the Affymetrix QC report. PMVO and MDQC are multivariate approaches that used to evaluate the quality of an array. MDQC examines the "Mahalanobis distance" of its quality attributes from those of other arrays, while PMVO uses parametric multivariate outlier testing using a multivariate Gaussian model.For array-based quality control, Affymetrix recommends using a set of QC factors to describe hybridization performance and array quality. MFive QC factors are included on the query interface for custom definition: scale factor, average background, the 3'/5' intensity ratio of GAPDH, the 3'/5' intensity ratio of beta-actin, and the proportion of probes called present (percentpresent) as provided by SimpleAffy. The default cutoff values were selected according to the recommendations of Affymetrix, SimpleAffy, and Larson et al. : scale fFor gene-based filtering, the database provides MAS5call and LEVs. The MAS5 algorithm supplies the MAS5call, which annotates whether the gene is expressed in specimen. LEV accesses the effect of RNA-labeling efficiency and RNA quality. Research has proven that filtering out genes with highly variable LEV improves the comparability between different laboratories and the homogeneity of gene expression profiles within the same class of specimens .2DB web query interface consists of two parts. Part I provides "sample type," "platform," and "quality control" criteria for selection. "Sample type" indicates whether the RNA samples for hybridization are "individual," from a single individual, or "pooled," from multiple individuals. Only two platforms, HU133A and HU133 plus 2.0, are supported in M2DB, and users can choose either one or both. The "quality control" option provides SimpleAffy, PMVO, and MDQC QC factors. For SimpleAffy, users can choose the default value or define a customized value for each QC factor. For PMVO and MDQC, three distribution percentiles are offered in their respective algorithms as QC thresholds.As shown in Figure Part II provides a "flexible combination selection" of five clinical annotations for samples which passed the criteria in Part I. Users can define from one to five annotations for query. The types of annotation and their quantities can be visualized in real time according to users' combined selections. This combination selection helps users quickly reduce the number of items submitted for query. For example, there are more than 250 items in "Disease State Suppl." By applying the combination of "Cancer or Tumor" in "Disease State" and "uterine cervix" in "Disease Location", the number of items submitted for selection in "Disease State Suppl." reduces to less than five. On the other hand, users can define different combinations according to their demand. For example, users can select 222 tumors collected from the lung by applying the combination of "Cancer or Tumor" in "Disease State" and "lung" in "Organism Part". When adding the criterion: "lung" in "Disease Location", there are only 215/222 samples classified into lung cancer. In this case, 7/222 samples are classified into lung metastasis.The experiment information table contains simple descriptions of experiments and hyperlinks to the GEO, ArrayExpress, and PubMed web sites to obtain more complete information. The sample information table contains complete clinical annotations and QC factors from SimpleAffy, PMVO, and MDQC for each sample. Moreover, the database provides an "additional selection box" for each sample to enable making further selections. Users have an additional option to decide whether a sample is suitable for further analysis according to the complete information in the table.2DB offers four data types: one is raw data (CEL files), and the other three are normalized data . The database also provides MAS5call and LEVs as gene-base filtering. After entering an email address, the user submits the job to the server. The user receives two email alerts. One is a confirmation email informing the user that the job has been successfully submitted and is being processed. The other email with a download link is sent when the job is completed.After confirming the selected samples, users can define which type of data to download. M2DB, we created the Organism Part Subtype to assist users to define the sample location. For example, T lymphocyte samples can be derived from blood, bone marrow, or umbilical cord blood in the database. The two annotations, Organism Part and Organism Part Subtype, can be more accurate, efficient, and less complicated to define the sample location. According to our annotation categories, users can easily and quickly find samples defined in the selection via our web query interface. It provides instantaneous visualization and selective combination (up to five criteria) of the various quantities and types of items selected.To quickly and easily query samples, annotation categories adopt a flat list instead of a tree structure. For example, only two annotations, the Organism Part and the Organism Part Subtype, are used to describe the sample location in anatomic position. This choice is motivated by the increased complexity of the ontological tree structure in our web design. Additionally, the Organism Part is only used to describe the sample location in MGED ontology. In M2DB, the authors supplied elementary annotations that were manually curated according to free-text descriptions of the collected experiments. If researchers require further clinical information for advanced analysis, support from the authors of original published papers will be necessary. The authors therefore urge public microarray repositories to request microarray researchers for more detailed information, such as sex, age, disease-free survival...etc. This would greatly encourage microarray meta-analysis across different experiments.Detailed descriptions of experimental parameters and sample clinical information are necessary to make the metadata fully interpretable. However, complete descriptions are frequently not available or only partially available in either microarray repositories or published papers. Accordingly, in MThe uniform pre-processing eliminates the technical variance of data transformation, such as background correction, probe-set summarization, and normalization. Gagarin et al. demonstrated that two different summarizations of the same data may produce differential expression gene (DEG) lists that are only 30% concordant . However2DB were selected for the analysis. After submitting these clinical annotations for query, forty-nine samples from seven different datasets were identified by M2DB. The expression variation of each housekeeping gene is presented as C.V. of intensity as shown in the Additional file Housekeeping genes have been used for normalization in gene expression analysis, such as quantitative RT-PCR, northern blotting, and gene expression microarray -47. Furt2DB can be used by researchers to collect metadata for the following purposes: 1) Searching for biomarkers of prognosis or disease [2DB includes more than 1,800 normal samples from healthy individuals without diseases, abnormalities, or treatments according to the descriptions of the experiments. These data from normal samples can help researchers discover and address the differences between normal and diseased  specimens by cross-comparing different datasets.M disease -51. 2) U disease . 3) Inte disease . Further2DB does not put extra effort into constructing online analysis tools. Users can directly upload the M2DB's results to those analysis web servers, for example Expression Profiler [Many public microarray web servers have provided analysis tools such as differential expression, clustering, and supervised classification. Thus, MProfiler , GEPAS [Profiler , EzArrayProfiler , or ArraProfiler . Users wProfiler , CARMAweProfiler , ExpressProfiler , GEPAS [Profiler , and EzAProfiler , which a2DB is updated periodically to incorporate new experiments which provide raw intensity data. Newly incorporated microarray data will be re-annotated. It took six researchers about one month to curate ~20,000 arrays  and to annotate clinical arrays into five clinical characteristics. Finally, we selected 10,202 arrays into M2DB. In the future, when expending the dataset, the needed time will be proportional to the amount of new arrays. In addition, the entire set of raw data will be uniformly re-processed using normalization as well as QC algorithms when adding new chips into M2DB.MIAME 2.0 now requests authors to deposit their raw data files in public microarray depositories. This policy will greatly help in data integration and meta-analysis. M2DB to facilitate the search and retrieval process, as well as to ensure the reliability of human clinical microarray metadata. Providing raw data, uniformly pre-processed data, and several sets of QC metrics, M2DB can be used to significantly improve the quality and comparability of microarray metadata generated by different laboratories. The manually curated annotations with the "flexible combination selection" relieve time-consuming searching and help researchers easily find the clinical expression data they need. M2DB provides a lower entry threshold and an integrated process of meta-analysis. We hope that this database will promote further evolution of microarray meta-analysis.This research develops Mhttp://metadb.bmes.nthu.edu.tw/m2db/.The web-application is freely accessible at WC, MT, and IH wrote the manuscript. WC, CH, and CWC collected the datasets. WC, MT, TW, JH, CL, and CWC carried out sample annotation. YL and WS performed QC and statistical analysis. CRC and CWC carried out the data pre-process and database server construction. IH, WS, and CH contributed to the discussion and commented the manuscript. IH financially supported this study. All authors read and approved the final manuscript.Summary list of sample location. The excel file containing the summary list of sample location for 10202 samples of M2DB.Click here for fileSummary list of disease information. The excel file containing the summary list of disease information for 10202 samples of M2DB.Click here for fileThe intensity C.V. of 14 housekeeping genes in normal skeleton muscles. The expression variation of 14 housekeeping genes in normal skeleton muscles.Click here for file