Although high-throughput microarray based molecular diagnostic technologies show a great promise in cancer diagnosis, it is still far from a clinical application due to its low and instable sensitivities and specificities in cancer molecular pattern recognition. In fact, high-dimensional and heterogeneous tumor profiles challenge current machine learning methodologies for its small number of samples and large or even huge number of variables (genes). This naturally calls for the use of an effective feature selection in microarray data classification.We propose a novel feature selection method: multi-resolution independent component analysis (MICA) for large-scale gene expression data. This method overcomes the weak points of the widely used transform-based feature selection methods such as principal component analysis (PCA), independent component analysis (ICA), and nonnegative matrix factorization (NMF) by avoiding their global feature-selection mechanism. In addition to demonstrating the effectiveness of the multi-resolution independent component analysis in meaningful biomarker discovery, we present a multi-resolution independent component analysis based support vector machines (MICA-SVM) and linear discriminant analysis (MICA-LDA) to attain high-performance classifications in low-dimensional spaces.https://sites.google.com/site/heyaumapbc2011/.We have demonstrated the superiority and stability of our algorithms by performing comprehensive experimental comparisons with nine state-of-the-art algorithms on six high-dimensional heterogeneous profiles under cross validations. Our classification algorithms, especially, MICA-SVM, not only accomplish clinical or near-clinical level sensitivities and specificities, but also show strong performance stability over its peers in classification. Software that implements the major algorithm and data sets on which this paper focuses are freely available at This work suggests a new direction to accelerate microarray technologies into a clinical routine through building a high-performance classifier to attain clinical-level sensitivities and specificities by treating an input profile as a ‘profile-biomarker’. The multi-resolution data analysis based redundant global feature suppressing and effective local feature extraction also have a positive impact on large scale ‘omics’ data mining. With the rapid developments in genomics, high-throughput microarray pattern analysis shows a great potential in cancer diagnosis for its efficiency and cost-effectiveness . Howeverp×n matrix after preprocessing, each column of which represents gene expression values of all biological samples at a gene; each row of which represents gene expression values of a single biological sample across a genome. The total number of genes is in the order of 103~104, and the total number of biological samples is on the magnitude of tens or hundreds. Since the number of variables (genes) is much greater than the number of samples (observations), some traditional pattern recognition methods  may have instable solutions and lead to a low or poor classification performance. Alternatively, although there are a large number of genes in a profile, only a small portion of them have meaningful contributions to data variations. In addition, the high-dimensional data are not noise-free because preprocessing algorithms may not remove systematic noise contained in raw data completely. Obviously, the data redundancy and noise may inevitably affect the discriminative power of the classification algorithms applied to microarray data.A gene expression profile can be represented by a It is clear that feature selection play a critical role in gene expression analysis to decrease dimensionalities, remove noise, and extract meaningful features before performing classification. Feature selection algorithms usually can be categorized into three types: statistical test-based , wrapper-based  , and traHowever, these transform-based feature selection algorithms are generally good at selecting global features instead of local features. The global and local features contribute to the global and local characteristics of data and interpret global and local behavior of data respectively. Statistically, the global features consist of high-frequency signals and the local features consist of low-frequency signals. Unlike the global features, the local features are difficult to extract for most feature-selection algorithms, because the low-frequency signals have a lower likelihood to get involved in inferring the ‘new’ low-dimensional data, which are generally the linear combinations of all input variables, than the high-frequency signals. Finally, the low dimensional data obtained from the traditional feature selection methods may miss some local data characteristics described by the local features. For example, PCA is by-nature a global feature selection algorithm: each principal component contains some levels of global characteristics of data and receives contributions from all input variables in the linear combinations. In addition, changes in one variable will inevitably affect all loading vectors globally. However, local features may be a key to attaining high-performance gene expression pattern classification for its subtle data behavior capturing. For example, some benign tumor samples may display very similar global characteristics with malignant tumor samples but with different local characteristics. To attain high-performance diagnosis, it is essential to capture local data characteristics to distinguish these samples with similar global characteristics.The main reason for these algorithms’ global-feature selection mechanism is because they all are single-resolution feature selection methods, where all features are indistinguishably displayed in a single-resolution despite the nature of their frequencies. It inevitably causes global features more likely to be selected than local features and prevents effective local data-characteristics capturing. Mathematically, all variables of the input data are involved in the linear combinations to compute principal components in PCA, independent components in ICA, and basis vectors in NMF respectively. Such a global feature selection mechanism will prevent high-accuracy genomic pattern recognition in the following classification because only the features interpreting global characteristics are involved in training a learning machine . The redundant global features may inevitably decrease the generalization of the learning machine and increase the risk of misclassifications or over-fitting. Finally, the learning machines integrated with the global feature-selection algorithms will display instabilities in classifications.To avoid the global feature selection mechanism, it is desirable to distinguish  features according to their frequencies rather than treat them uniformly, which makes the high-frequency signals dominate the feature selection and the low-frequency signals lose opportunities. A discrete wavelet transform (DWT)  can hierIn this study, we propose a novel multi-resolution independent component analysis (MICA) to conduct effective feature selections for high dimensional heterogeneous gene expression data. Then, a multi-resolution independent component analysis based support vector machines (MICA-SVM) are proposed to achieve a high-performance gene expression pattern prediction. We demonstrate its superiority and stability by comparing it with existing state-of-the-art peers on six heterogeneous microarray profiles, in addition to extending MICA to linear discriminant analysis (MICA-LDA). We also develop a MICA-based filter-wrapper biomarker discovery algorithm to further demonstrate the novel feature selection algorithm’s effectiveness in biomarker capturing. Finally, we discuss potential extensions on the multi-resolution independent component analysis in microarray based molecular diagnosis and conclude this paper.Multi-resolution independent component analysis is based on the discrete wavelet transform (DWT) and independent component analysis (ICA). A discrete wavelet transform decomposes input data in a multi-resolution form by using a wavelet and scaling function. The coefficients at the coarse and fine levels describe the global and local behavior of data respectively. Mathematically, DWT is equivalent to multiplying the input data by a set of orthogonal matrices block-wisely. On the other hand, ICA seeks to represent input data as the linear combination of a set of statistically independent components by minimizing their mutual information. Theoretically, it is equivalent to inverting the central limit theorem (CLT) by searching maximally non-normal projections of the original data distribution. More information about the DWT and ICA methods can be found in ,9.The goal of the multi-resolution independent component analysis is to seek the statistically independent genomic patterns from a meta-profile computed by suppressing the coarse level coefficients  and maintaining the fine level coefficients  in the DWT of an input profile. As an approximation of the high dimensional input profile, the derived meta-profile captures almost all local features and keeps the most important global features. Unlike independent components in the classic ICA that are mainly retrieved from the global features for their high frequencies, the independent components calculated by our proposed MICA method are statistically independent signals, which contain contributions from almost all local features and the most important global features. As such, the latter is more representative in revealing the latent data structure than the former. Moreover, the redundant global feature suppressing brings MICA an automatic de-noising mechanism: since the coarse level coefficients  in DWT generally contain “contributions” from noise, suppressing coarse level coefficients not only filters unnecessary global features but also removes the noise. The MICA algorithm can be described as following steps.p samples across n genes L-level discrete wavelet transform for each sample to obtain a sequence of detail coefficient matrices Given a gene expression profile with jD to obtain its PC matrix: jD by using the first loading vector 1u in the PC matrix as jD by using the loading vectors rρ from those first r loading vectors is defined as iλ is the data variance from the thi loading vector. In the implementation, this step can be ‘lazily’ simplified as: keep all detail coefficient matrices A level threshold X:Conduct the corresponding inverse discrete wavelet transform using the updated coefficient matrices *X to obtain independent components and the mixing matrix: Conduct the classic independent component analysis for X* is the approximation of the original profile X by removing the redundant global features and retaining almost all local features by selecting features on behalf of their frequencies. It is easy to decompose each sample in the subspace spanned by all independent components ia is the thi row of the mixing matrix recording the coordinate values of the sample ix in the subspace. As a low dimensional vector, the meta-sample ia retains almost all local features and the most outstanding global features of the original high-dimensional sample ix. Thus it can be called as a data-locality preserved prototype of ix.The meta-profile τ = 3,4,6 with a Daubechies family wavelet ‘db8’, where the control and cancer samples are indicated by red and blue lines respectively. Interestingly, extracted local features and selected important global features make two types of samples display two distinct prototypes in the low-dimension subspace. With the increase of the level thresholds, the two groups of prototypes tend to show more capabilities to separate cancer and control samples. Moreover, two types of meta-samples demonstrate a “self-clustering” mechanism in that the meta-samples belonging to the same type show close spatial proximities. Obviously, the clear sample separation information conveyed by the self-clustering mechanism of the meta-samples is almost impossible to obtain from the original high-dimensional data directly, and the key discriminative features captured by our proposed MICA method would be able to facilitate the subsequent classification step and contribute to high-accuracy disease diagnosis.Figure n where n~103-104, MICA-SVM separates biological samples by constructing the maximum margin hyperplane in the spanned subspace k ≤ p ~ 102, using the meta-samples. If we assume the number of support vectors sN is much less than the training points l, the time complexity of the MICA-SVM is The MICA-based support vector machines apply the classic support vector machines (C-SVM)  to the mw and b.A way to solve (1) is through its Lagrangian dual that is also a quadratic programming problem, where x′, where rbf’: radial basis function kernels).The normal of the maximum-margin hyperplane is calculated as We have performed extensive experiments using six publicly available gene expression microarray profiles consisting of five oligonucleotide profiles -15 and ok-fold (k=10) and 100-trial of 50% holdout cross validations. The classification accuracy in the thi classification is the ratio of the correctly classified testing samples over total testing samples: tp (tn) is the number of positive (negative) targets correctly classified, and fp (fn) is the number of negative (positive) targets incorrectly classified respectively. In the 100-trial of 50% holdout cross validation (HOCV), all samples in the data set are pooled together and randomly divided into half to get training and testing data. Such a partition is repeated 100 times to get 100 sets of training and testing datasets. In the k-fold cross validation, an input dataset is partitioned into k disjoint equal or approximately equal proportions. One proportion is used for testing and the other k-1 proportions are used for training alternatively in the total k rounds of classifications. Compared with pre-specified training or testing data, the cross validations can decrease potential biases in algorithm performance evaluations.To address our algorithm’s superiority and reproducibility, we compare it with six comparison algorithms in terms of average classification rates, sensitivities, and specificities under the The existing six comparison algorithms can be categorized into two types. The first type consists of standard support vector machines (SVM)  and linedb8’ to conduct a 12-level discrete wavelet transform for each data set, and select a level threshold τ = 3 in MICA for all profiles. Although not an optimal level threshold for all data, it guarantees automatic de-noising and ‘fair’ algorithm comparisons. Moreover, we have found that the meta-samples obtained from MICA at τ = 3 can clearly distinguish two types of samples. Although other level threshold selections may be possible, any too ‘coarse’ (e.g.τ = 1) or too ‘fine’ (e.g.τ ≥ 9) level threshold selection may miss some important global or local features and affect following classifications.We employ the wavelet ‘exceptionally leading advantages over its peers in the three classification performance statistics for all datasets. For example, it achieves 98.26%, 99.04%, 99.69%, 98.76%, 98.30% and 97.23% average classification rates on the stroma, breast_1, prostate, glioma, HCC, and breast_2 data respectively under the 100 trials of 50% HOCV. In addition, MICA-SVM achieves 98.00%, 99.52%, 100.0%, 100.0%, 100.0%, and 99.00% for the stroma, breast_1, prostate, glioma, HCC, and breast_2 data respectively under the 10-fold CV. All these results indicate that MICA can effectively capture global/local features as well as eliminate the noisy features so that SVM can perform significantly better than the state-of-the-arts. Furthermore, unlike the other methods that display instabilities in classifications, our proposed MICA-SVM algorithm demonstrates a strong stability in attaining high-accuracy detections for all profiles. This observation is also supported by its lower standard deviations of the three classification measures than those of the others.Table Figure db8’ wavelet and set the level threshold τ = 3 in MICA. Interestingly, the MICA-LDA classifier is only secondary to the MICA-SVM classifier: it outperforms the other comparison algorithms on the five datasets except the prostate data in terms of the average performance under the 100 trials of HOCV and 10-fold CV. This further indicates that MICA’s effective feature selection and its contribution to subsequent classification methods. Figure We also apply MICA to linear discriminant analysis (LDA) to further explore its effectiveness. Similar to the MICA-SVM algorithm, the MICA-based linear discriminant analysis (MICA-LDA) applies the classic LDA to the meta-samples obtained from MICA to gain sample classifications . The MICA-LDA algorithm’s performance on the six profiles can be found in the Additional file Z in MICA to resolve it, where Smax and Smin are the maximum and minimum singular values of the matrix Z calculated from MICA. A smaller condition number indicates a more stable matrix that suggests a better status in global and local feature capturing. The level-threshold is counted ‘optimal’ if the condition number δ is the smallest. If the condition numbers from two level thresholds are same numerically, the lower level threshold (which is required to be > 1) is counted as the optimal one. For example, the smallest δ value is achieved at τ = 6 and τ = 7,8,9,10,11 respectively on the HCC data. We choose τ = 6 as the optimal threshold which is corresponding to the best average the average classification rate: 98.77% (STD: 2.26%) with average sensitivity: 99.44% (±2.11%) and specificity are 97.59% (±4.97%) respectively.A remaining question is how to determine the optimal level threshold in MICA so that the following SVM classifier achieves best performance. We employ the condition number δ values under the 100 trials of 50% HOCV on the ‘stroma’, ‘breast_1’, and ‘breast_2’, and ‘HCC’ data, as the level threshold values in MICA are selected from 1 to 11. Obviously, the optimal level threshold can be identified by finding the level threshold corresponding to the minimum condition number. Although the optimal threshold at τ = 8 corresponding to average classification rate 99.11% (±0.89%), which is slightly lower than the actual best average classification rate: 99.20% (±0.92%) achieved at τ = 6, it is ignorable due to possible numerical inaccuracy from the fixed point iteration in MICA. Furthermore, we have found that MICA-SVM has relatively low-level performance at too coarse level thresholds (e.g. τ = 1). Although δ values and MICA-SVM performance show some-level stability under some fine level thresholds, too fine level thresholds (e.g. τ ≥ 8) may decrease classification performance on some data . Also, the optimal level threshold selection method may bring some computing overhead in practical classification. In practice, we suggest the empirical level threshold as Figure db8’ is employed in our experiments, there is no other specific requirement in MICA-SVM for a wavelet except it should be orthogonal. To compare effects of different wavelet selections on the algorithm performance, we select four family wavelets: ‘db8’, ‘sym8’, ‘coif4’, and ‘bior4.4’, in the classifications on the six profiles at the level threshold τ = 3. It seems that there is no obvious classification advantage from one wavelet over the other under the 10-fold CV, because the robust prior knowledge and less number of trials may have larger impact factors on the algorithm performance than a wavelet selection. However, we have found that the wavelet ‘db8’ show some advantages over the others under the 100 trials of 50% HOCV. In addition, it is interesting to see that the wavelets ‘coif4’ and ‘sym8’ have almost same-level performance, but the wavelet ‘bior4.4’ has a relatively low performance for the six profiles.Although only wavelet ‘We further demonstrate the superiority of MICA-SVM by comparing it with three state-of-the-art partial least square (PLS) based regression methods, which can be found in the Additional file rbf’ kernel on the biomarker under leave-one-out cross validations. The order of the three biomarkers in Table Fos gene family. It is responsible for encoding leucine zipper proteins, which is able to dimerize with proteins of the JUN family, and form the transcription factor complex AP-1. As a regulator in cell proliferation, differentiation, and transformation, recent studies [ibc') and 34 non-inflammatory breast cancers ('non-ibc')) of the stroma data using the three biomarkers. It is interesting to see that two types of cancers are separated into two spatially disjoint sets clearly, though one ‘ibc’ sample is wired in the ‘non-ibc’ samples.In addition to classifying large scale heterogeneous tumor profiles with exceptional performance, multi-resolution independent component analysis can be also applied to capture biomarkers for microarray profiles. We present a MICA-based filter-wrapper biomarker capturing algorithm and apply it to the stroma data. The details of this algorithm can be found in the Additional file  studies ,20 have  studies . In addi studies . Figure rbf’), where each learning machine can only recognize the majority type samples of the training data in classification despite the testing sample type. Moreover, we have tried kernel ICA [inear’, otherwise, it encounters overfitting under the standard Gaussian kernel. In our ongoing project, in addition to further polishing our algorithm by comparing them with other state-of-the-art methods , we are SVM-RFE .omics’ data mining. In our future work, we plan to further explore MICA-SVM’s potential in other platform gene expression data, SNP, and protein expression data classification.In this study, we present a novel multi-resolution feature selection algorithm: multi-resolution independent component analysis for effective feature selection for high-dimensional heterogeneous gene expression profiles, propose a high-performance MICA-SVM classification algorithm, and demonstrate its superiority and stability by comparing it with the nine state-of-the-art algorithms. Our algorithm not only consistently demonstrates the high-accuracy or clinical-level cancer diagnosis by treating an input profile a whole biomarker but also shows effectiveness in meaningful biomarker discovery. It suggests a great potential to facilitate high-throughput microarray technology into a clinical routine, especially, current classification methods have relative low even poor performance on the gene expression data. In addition, the multi-resolution data analysis based redundant global feature suppressing and effective local feature extraction will have a positive impact on large scale ‘The authors declare that they have no competing interests.HEY collects and processes the data, designs algorithms, implements the methods, and drafts paper. LXL participates in discussion and provides help to polish the paper. HEY and LXL jointly finalize the paper.Implementations of the four comparison algorithms PCA-LDA, PCA-SVM, ICA-SVM, and NMF-SVM algorithm implementationsClick here for fileMICA-LDA performance MICA-LDA performance under 100 trials of 50% HOCV and 10-fold CVClick here for fileComparing MICA-SVM with PLS-based regression methods Comparisons MICA-SVM with three partial least square (PLS) based regression methodsClick here for fileAlgorithmic stability analysisClick here for fileMICA-based biomarker discovery algorithmsClick here for file


Lung cancer is the leading cause of cancer deaths in the world. The most common type of lung cancer is lung adenocarcinoma (AC). The genetic mechanisms of the early stages and lung AC progression steps are poorly understood. There is currently no clinically applicable gene test for the early diagnosis and AC aggressiveness. Among the major reasons for the lack of reliable diagnostic biomarkers are the extraordinary heterogeneity of the cancer cells, complex and poorly understudied interactions of the AC cells with adjacent tissue and immune system, gene variation across patient cohorts, measurement variability, small sample sizes and sub-optimal analytical methods. We suggest that gene expression profiling of the primary tumours and adjacent tissues (PT-AT) handled with a rational statistical and bioinformatics strategy of biomarker prediction and validation could provide significant progress in the identification of clinical biomarkers of AC. To minimise sample-to-sample variability, repeated multivariate measurements in the same object  across patients should be designed, but prediction and validation on the genome scale with small sample size is a great methodical challenge.To analyse PT-AT relationships efficiently in the statistical modelling, we propose an Extreme Class Discrimination (ECD) feature selection method that identifies a sub-set of the most discriminative variables (e.g. expressed genes). Our method consists of a paired Cross-normalization (CN) step followed by a modified sign Wilcoxon test with multivariate adjustment carried out for each variable. Using an Affymetrix U133A microarray paired dataset of 27 AC patients, we reviewed the global reprogramming of the transcriptome in human lung AC tissue versus normal lung tissue, which is associated with about 2,300 genes discriminating the tissues with 100% accuracy. Cluster analysis applied to these genes resulted in four distinct gene groups which we classified as associated with (i) up-regulated genes in the mitotic cell cycle lung AC, (ii) silenced/suppressed gene specific for normal lung tissue, (iii) cell communication and cell motility and (iv) the immune system features. The genes related to mutagenesis, specific lung cancers, early stage of AC development, tumour aggressiveness and metabolic pathway alterations and adaptations of cancer cells are strongly enriched in the AC PT-AT discriminative gene set. Two AC diagnostic biomarkers SPP1 and CENPA were successfully validated on RT-RCR tissue array. ECD method was systematically compared to several alternative methods and proved to be of better performance and as well as it was validated by comparison of the predicted gene set with literature meta-signature.We developed a method that identifies and selects highly discriminative variables from high dimensional data spaces of potential biomarkers based on a statistical analysis of paired samples when the number of samples is small. This method provides superior selection in comparison to conventional methods and can be widely used in different applications. Our method revealed at least 23 hundreds patho-biologically essential genes associated with the global transcriptional reprogramming of human lung epithelium cells and lung AC aggressiveness. This gene set includes many previously published AC biomarkers reflecting inherent disease complexity and specifies the mechanisms of carcinogenesis in the lung AC. SPP1, CENPA and many other PT-AT discriminative genes could be considered as the prospective diagnostic and prognostic biomarkers of lung AC. Lung cancer is the leading cause of cancer deaths in the world -5. It isPatients with lung AC have better survival rate when the localized tumour is removed at an early stage . HoweverThe currently available AC gene signatures demonstrate substantial differences in both the number of predicted genes and the specific genes they include, mostly due to differences in the study design, assay platforms, tumour histology and patient selection protocols. Only few of the studies in the field produced detailed clinical and research protocols that were statistically and biologically validated -9. AdditUsing microarray technology, several putative gene markers of lung AC have been intensely studied and evaluated in clinical trials, for example EGFR  and osteThe experimental design and the properties (or models) of the feature selection method for biomarkers prediction are two important decision steps preceding the actual data analysis. Their choice might significantly affect the sensitivity and specificity of the findings, given a high dimensionality of the data and the numerous error sources that affect them.vs. control tissue design) originating from the same test object (a patient)[In contrast to studies based on unpaired data (independent patient samples for case and control), paired data experiments have not been extensively applied in feature selection of transcriptome studies aiming to biomarker identification ,25,26. H patient),28. The Haney et al.  showed eWe suggest that analysis of gene expression data of the primary PT-AT handled with a rational computational strategy of biomarker prediction essentially facilitates personalized identification of a comprehensive set of specific AC signatures, disease-specific pathways and clinical biomarkers.Within this framework, the primary aim of this work was to design a feature selection (e.g. class discriminative genes) method that is capable not only of identifying conventional ‘differentially expressed genes’ -36, but We introduce a function to characterise the completely discriminative variables  which according to some additional statistical and biological criteria might be considered as the potential clinical biomarkers (see Introduction and Discussion).j, i = 1…, N; j = 1, …, M; X is normal class; Y is tumor class), are derived from N patients. The sub-set of N paired dependent signals from quantitatively distinct sample sets is called a completely discriminative signal (CDS) if:Let X and Y represent the normal (adjacent) and malignant paired dependent tissue samples , respectively. The paired signals for microarray identificator or1{.} is an indicator function with value 1 if the expression in brackets holds and 0 otherwise. When where Even if all paired samples are CDS in a given patient cohort, the increase of the number of patients (or sample size) should lead to failure of “completeness” of a given discriminative feature.http://www.ncbi.nlm.nih.gov/geo/) using the GEO accession number GSE7670 and [In order to select the prospective gene markers of AC and evaluate potential biomarker space of AC, we used 54 U133A Affymetrix microarray designed in pairs: lung AC versus surrounding lung tissue, which have been obtained from 27 lung AC patients . The MAS7670 and . Figure p, of having random classification on two groups (Methods). In particular, in the extreme case of CDS (i.e. no misclassification) for our 54 samples (27 paired samples) after Bonferroni adjustment we get the p-value estimate We used the sign symmetry test to estimate the p-value, In this work mostly CDS expression signals are considered as the prospective gene markers. However, a relatively weak relaxation of the extreme criteria (a few misclassifications) is also considered, but at specified statistical conditions (Methods and Discussion). In our feature selection method we used cross-normalization (CN) method followed by our modified Wilcoxon sign rank test (MWT) Figure , MethodsTo demonstrate a performance of CN and MWT with respect to its discrimination ability of the gene expression paired samples, we started with several well-defined gene markers of lung AC: EGFR -44, osteWe also re-analysed the five-gene lung AC diagnostic signature, which was identified from protein network analysis combined with the microarray gene expression data analysis . This sivs. the surrounding tissue samples for the initial  U133A probe sets and for that data after application of the mutual cross-normalization procedure. In some cases, the distinction between adjacent lung tissue and cancer tissue was quite strong (e.g. MYST3 (202423_at), but in general, the paired CN yielded smaller numbers of discrimination errors or even complete separation of the normal and cancer samples . Figure Figure Thus, CN provides a systematic improvement of the separation ability of the paired microarray signals for well characterized biomarkers of AC and for gene markers of the AC diagnostic signature  having aOur task here is to identify extremely discriminative gene expression signals  classifier derived in this work using data , which cThese data integration results demonstrate a good agreement between SAM U95A signature and ECD U133A signature: 604 Refseq genes are common  method  to the dTo further assess the feature selection performance of ECD, we decided to overlap the top 2,829 ECD-selected probe sets with the top 2,829 probe sets derived by EDGE, PAM, Wilcoxon and t-tests. The Venn diagram in Figure When the CN data were used for analysis of classification accuracy of the methods, the ECD set provides the highest, closely followed by the standard Wilcoxon sign ranked test, EDGE and the t-test. While none of 2,829 probe sets from the ECD had 2 or more misclassified sample pairs, the same-size top discriminative sets from the standard Wilcoxon sign ranked, EDGE, the t-test and PAM had 90 (3%), 111 (4%), 105(4%) and 335 (12%) probe sets with 2 or more misclassified sample pairs, respectively  was applied to 54 samples represented by the top-selected 2,829 probe sets, each of the feature selection method could clearly discriminate tumour samples from all normal samples ; 342 Refseq genes) and cluster 3  contain down-regulated genes, while cluster 2  and cluster 4  contain the genes that are up-regulated in the lung AC vs AT  on the 27 PT-AT paired lung AC sample U133A dataset (Dataset section in Methods), we catalogued approximately 100 references cited as either lung AC prognostic- or diagnostic- biomarkers and further manually selected 25 relevant publications and extracted set of 1,249 genes . These sites have been experimentally altered by mutagenesis. UniProt DB describes the effect of the experimental mutation of one or more amino acid(s) on the biological properties of the protein. It describes only those experiments in which a limited number of amino acid residues were altered: gross alterations in protein structure, such as the deletion of hundreds of amino acids are not described.Table vs. AT  were found with our ECD signature. Among these proteins, we found epidermal growth factor receptor , v-erb-b2 erythroblastic leukemia viral oncogene homolog 2, neuro/glioblastoma derived oncogene homolog (avian) (ERBB2: 216836_s_at), MAD1 mitotic arrest deficient-like 1 (yeast) , solute carrier family 22 (organic cation transporter), member 18  and tumour protein p73-like . Mutations in these well-known genes are associated with lung cancer, and specifically AC cells [Table To better assess the functional annotation of the gene markers enriched in our ECD classifiers, we will focus now on the statistical enrichment analysis of gene set with gene ontology terms (GO) .Interestingly, the term “signal peptide” is associated with the proteins which belong to a common fraction (112 gene symbols) Table . InteresAccording to David Bioinformatics software, a common fraction of our discriminative AC and AC Meta-signatures is also strongly enriched with the genes encoding proteins of cell cycle, mitosis, micro-tubular based processes, DNA replication, spindle organization, regulation of cell cycle process and other GO terms which are typically represented in many “proliferative signatures” reported for many cancers. However, unique subsets are represented by more diverse sub-sets enriched with the specific GO terms. For instance, AC Meta-signature is strongly over-represented by GO terms “protein amino acid phosphorylation”, “phosphate metabolic process”, “intracellular signalling cascade”, “regulation of apoptosis”, while the unique subset of discriminative AC signature is more than 2-times enriched with the GO term “vasculature development” and 4.8 times enriched with the GO term “protein amino acid O-linked glycosylation”. The genes representing these two terms were reported in the literature to be the key players in the early stage of cancer progression, tumour dormancy and risk of tumours metastasis.EF improves significantly when the early diagnostic biomarkers reported in the literature were exclusively compared with the best discriminative genes derived using ECD and the alternative signatures. For instance, Table Thus, among studied feature selection methods, ECD provides the most comprehensive lists of the genes which are strongly supported by the literature as the prospective genes for biomarker discovery.The size of the ECD-derived PT-AT discriminative signature can be extended to derive a classifier that provides better coverage of the AC Meta-signature. Many AC Meta-signature genes are well characterized as promising lung AC biomarkers  and its subset of AC early diagnostic gene markers Table  could beIn order to validate the diagnostic utility of our discriminative signature we selected two genes SPP1 and CENPA, both of which showed 100% separation with strong fold change of PT-AT pairs in the U133A expression dataset. SPP1 is a well-characterized gene which is considered as a promising prognostic and diagnostic biomarker of NSCLC . CENPA iThe results of QRT-PCR tissue arrays of genes SPP1 & CENPA) are completely concordant with the microarray analysis  better separate the samples of the two classes , (ii) reduce the number of misclassification errors (pattern recognition and machine learning issues) in comparison with initial data, (iii) provide a highly sensitive and ultra-specific feature selection procedure of the discriminative features  and (iv) select the most biologically reliable classifiers  representing the classes of interest (tumour and adjacent tissues). Our feature selection model uses consequently CN, MWT, and the bootstrap method and sign-rank tests figure . It seleWe compared the feature selection method based on ECD with the standard methods from the literature, using our aforementioned quality control approach, and then finally assess the efficiency of alternative classifiers and our method, using a comprehensive set of known gene markers of lung cancer. The efficiency of ECD classifier was successfully evaluated in several ways: (i) by the similarity in content between gene lists selected from the same data , such as EGFR , osteopoWe also identified a large number of cell adhesion and cell communication pathway genes, such as CDH3, SPP1, SPINK and CEACAM5  and adjacent lung tissue, we identified the global reprogramming of the transcriptional profile in human lung AC compared to normal lung tissue. Through the mutual CN of expression data belonging to dependent paired samples and using a separability score, calculated by the modified Wilcoxon signed-rank test, the method provides an essential improvement of the sensitivity of the feature selection at the highest specificity (100%) in a comparison to the alternative methods. Our method reveals expression shift of at least 2,428 genes discriminating lung AC from adjacent normal lung tissues with ~100% accuracy. Additionally, more than 2000 genes could discriminate the lung AC tumours from the normal lung tissues with ~93% accuracy.Systematic structural and functional study of the protein peptides found in the common subset of the Meta-signature and ECD signature (as well as SAM U95 signature and ECD signature) should be useful in our understanding of the basic interactions of the tumour with stroma, surrounding tissue, immune and endocrine systems. Specifically, we suggest that immunity and signal peptides gene signatures (and their sub-networks) could be used for early diagnostics of AC and for quantitative monitoring of the adjuvant therapy. Thus, we could suggest that these genes may represent promising targets for novel specific therapies.Non-small cell lung cancer (NSCLC) can be classified into the major subtypes, AC, and squamous cell carcinoma (SCC). A global gene expression profiling of 58 human NSCLC specimens revealed large differences between AC and SCC subtypes, with more than 1700 genes found to be differentially expressed . The assECD provides superior selection in comparison to conventional methods and can be widely used in different applications. However, the selection and classification performance and error estimation strongly depend on the data set size and dimensionality of the co-variate space. It is a challenging task to develop an exact model of microarray data taking into account paired dependent design and to compare the benefit of the model versus non-paired design models. Several ways to treat paired dependent data more efficiently in a parametric modelling framework have been proposed . Using uThe thresholds of our feature selection procedure could be fine-tuned and corrected . Despite the statistically reasonable cut-off of p-value and the 100%-correct prediction criteria, we found that a large proportion of the lung cancer gene markers reported in the literature were still filtered out (Methods). Some potentially important gene markers, such as MYC and BRCA1  , under the GEO accession number GSE7670. The lung tissue samples were obtained during the lung cancer patients’ surgery. The authors selected adjacent normal-tumour matched lung cancer pair samples for RNA extraction and hybridization of mRNA to Affymetrix U133A microarray platform  based on the manufacturer’s standard protocol. In total, 66 microarray samples were available . We used 54 pair-wise tissue samples, selected from 27 patients.The present study used Su et al.  microarrThe microarray hybridization signal intensity values of the 22,283 U133A probe sets were MAS5.0 calibrated, and the values were scaled by adjusting the mean signal intensity values to a target value of log500. For the present study, we used non-log transformed data .j, j , measured in the cancer (class1 or Y) and surrounding tissue samples (class2 or X) of N patients, i = 1, …, N, is discriminative if there is a significant difference between the expression levels of the two classes. Let i, probeset j and class Y. Similarly denote as The workflow of our discriminative feature selection method is given in Figure j across the N patients.In a classical univariate paired difference analysis, we would first subtract the non-disease value from the disease value for each measured variable of the same subject, then compare these distribution of the differences around zero. Mathematically, let j in paired samples is lower because there is positive correlation within each dependent pair.The mean difference between the groups does not depend on whether we organize the data as pairs. The mean difference is the same for the paired and unpaired statistics but their statistical significance levels can be very different, because of the estimation of the variance component. Below we show that the variance of DThe following statistical model is useful for understanding the paired difference test:ijY = μj + αi + εiji is a random effect that is shared between the two values in the pair, and εij is a random noise term that is independent across all data points. The constant values μ1, μ2 are the expected values of the two measurements being compared, and the test interest is in δ = μ2 − μ1.where αi capture "stable confounders" that have the same effect on the “normal” tissue and “abnormal” tissue measurements. In unpaired data analysis the αi cancels out, so it does not contribute to the variance. In paired data the within-pairs covariance is Cov(α)>0 which decreases the variance of δ differences and leads to better performance.In this model, the αMWT is a simple non-parametric separability test of paired data, which can be used to select the highly discriminative features (gene markers).Obtaining a sufficiently large number of sample pairs in clinical microarray studies is a very difficult task. Usually, the number of paired samples is smaller than that in studies of independent samples. To cope with these constraints, a nonparametric approach is appropriate. Specifically, the Wilcoxon signed (WT) rank test  can be uij has been defined above θ is the unknown ‘disease effect’, and eij are random errors, which are assumed to be mutually independent and symmetric about zero.where DTo test the null hypothesis of no disease effect is equivalent to test:Our modification of the model of WT (MWT) involves the Cross Normalization step and the Wilcoxon sign rank test as follows:1: Calculate the class average across samples: 2: Cross-normalize the average of the opposite classwhere 3: Calculate differences between cross-normalized vectors for each sample pair:4: Calculate the absolute value of the differences obtained from Step 3.R-/+).5: Rank the differences obtained from Step 4 .R-/+ (Step 6) with the following formula:7: Calculate the Z-value using the smaller value where:N is the total number of paired samples.And 8: Introduce the correction factor of 9: Calculate the p-value from the Z-table  and disr10: Calculate the separation error of each feature :+VE denote the number of patients that have a positive dj.Let t-VE denote the number of patients that have a negative djLet t+VE and t-VE over N  combines the properties of the parametric symmetry (zero mean the differences) test and the non-parametric (zero rank the differences) symmetry test.A sign test could be used to estimate the probability of misclassifications error in paired sample experimental design . For each microarray expression signal i, given the gene expression intensity values of 27 independent patients measured in two conditions, we can estimate the probability of observing less than s patients' misclassifications. This probability of observing an event that is more extreme than s  and can be estimated by the Binomial distribution. Assuming that the probability of positive and negative increments is p = 0.5 the probability of misclassification is the following:For example the probability of at most two misclassifications in 27 paired samples is calculated as:which after Bonferroni correction on the number of U133A probe sets becomes adjusted p-value = 0.062.Bootstrapping testing  was usedA program in C++ language was written for the MWT method to process the microarray Affymetrix data sets. The data, when processed through the program, generate an N x M table of cross-normalized data, where M is the number of the Affymetrix ID and N is the number of the Patient’s ID. Another N x M table contains 1-s and 0-s, where 1 represents positive signs and 0 represents negative signs of gene expression in the pair. This indicates the relative down- or up- regulation of a given hybridisation signal (corresponding to the expression level of the gene transcript) for a given object of the class Y (cancer tissue) in comparison to the paired object of the class X . Finally, the M U133A probe set IDs (representing the M columns of the table) are sorted based on the MWT p-values, and the total number of 1-s (and percentage) for each microarray probe set ID is tabulated. The program also calculates the ratio of the average expression signals of Class 2 to Class 1.Hypergeometric function (Fisher exact) test ,62 is usk: Number of elements in the overlapped region of the Venn diagramn: Total number of elements present in group 1m: Total number of elements present in group 2N: Total number of elements present in both groups.k or more elements intersect subsets of n and m members at random  in a universe set of size N, we summon over the right tail of a hypergeometric distribution:To find the probability that k ≥ l ; n, m, N) := p-valuePr in the intersection region of the Venn diagrams. In the case of Refseq genes non-redundantly presented on U133A microarray The standard Student’s t-test for paired data and the non-parametric Wilcoxon signed-rank test for paired data, implemented in the R statistical package as the ‘t.test’ and ‘wilcox.test’ functions, were used to compare the expression levels in pairs of normal and tumour samples. The resulting P-values were corrected with the FDR values using p-value adjustment module ‘p.adjust’, implemented in the R statistical package. The ensuing FDR values were used to rank the genes, and the desired number of genes was then selected for use.An optimal discovery procedure (ODP) method, implemented in the Extraction of Differential Gene Expression (EDGE)  program,Initially, we ran the PAM algorithm  with allHierarchical cluster analysis was performed with XCluster , using bDiavid Bioinformatics , ClueGO Primers were designed for validating 2 discriminative genes which perfectly classify lung tumours from surrounding normal tissues . Relative levels of mRNA of genes selected for validation were quantified using Tissue scan panel HLRT-02 which includes 24 lung cancer cDNA samples matched in pairs with surrounding lung  cDNA samples. This panel was provided by the OriGene Technologies, Rockville MD (manufacture). The panel contains pre-normalized 48 cDNA samples with various subtypes of lung cancer with information regarding TNM, tumour size, tumor type, stage, age and other clinical information. qRT-PCR experiments were carried out on Applied Biosystems 7500 Real Time PCR machine. The normal to tumour Ct-values of each gene were obtained and relatively quantified using ΔΔCt analysis . PrimersThe authors declare that they have no competing interests.The conception of analysis strategies was contextualized by VAK. He suggested the theory of the Extreme Class-Discrimination method and the strategy of the performing the bioinformatics and statistical analyses of lung adenocarcinoma data. SHT and PP were in charge of major part of the computational calculations and the statistical data analysis and involved in the generation of the specialized computer programs used in this work. They also were participated in writing the draft of the manuscript. EM was involved in the discussion of the statistical methods, the interpretation of the results and in manuscript correction and revision. SPY provided PCR validations of the prospective biomarkers and discussed the results of this study. KCK was involved in discussions of this work. VAK did data analysis and conceived the study, coordinated the project, and wrote the final version of the manuscript.Table S1. Master table annotated with gene symbols with gene symbols from Lung AC meta gene signatures, default and relaxed lung AC gene signatures and signatures derived by other methods cited in the literature.A: Characteristics of the data set and final results; B: Read me File C: Summary statistics. D: AC signature derived by SAM software from U95A data. S1e: Tables of the 604-gene PT-AT consensus signature, the results of GO analysis, the immunity and cell cycle gene signatures discriminating AC from normal lung tissues.Click here for fileTable S2. Distribution of the number of false classifications of the 27 paired samples. Comparison of the classification accuracy of the extremely discriminative 2,829 probe sets (ECD) with the top-level 2,829 probe sets identified by the standard Wilcoxon sign ranked test (WT), EDGE, PAM, and Student’s t-test. . While the ECD classifier was derived using the cross-normalized dataset as input, the classifiers derived using PAM, EDGE, WT and t-test used the original MAS5-normalized data as input. However, the classification accuracy, in terms of the number of probe sets with 2 or more anomalous fold-changes, was estimated using the MAS5-normalized dataset.Click here for fileComparison of different feature selection methods. Analysis of discriminative ability using the original MAS5 normalized data The two-way hierarchical cluster analysis of the 2,829 probe sets and the 27 pairs of normal-lung AC samples demonstrates the ability of the selected methods to separate lung AC from normal samples (Supplementary figures S1-S4). All of the methods, with the exception of the t-test and the Limma paired test, produced a near-perfect separation of the two classes, however, ECD provides more biologically reasonable grouping of the genes (see Results). Figure S1. Two-way hierarchical cluster analysis of the MAS5-normalized expression values of 2,829 probe sets identified by the standard Wilcoxon test. Figure S2. Two-way hierarchical cluster analysis of the cross-normalized expression values of the 2,829 probe sets identified by EDGE. Figure S3. Two-way hierarchical cluster analyses of (A) the MAS-normalized expression values and (B) the cross-normalized expression values of 2,829 probe sets identified using the Student’s t-test. Figure S4. Two-way hierarchical cluster analyses of (A) the MAS-normalized expression values and (B) the cross-normalized expression values of 2,829 probe sets identified using the Limma paired t-test .Click here for fileTable S3. Meta-gene signatures associated with Lung AC, curated from the literature with the corresponding references.Click here for file


Machine learning has increasingly been used with microarray gene expression data and for the development of classifiers using a variety of methods. However, method comparisons in cross-study datasets are very scarce. This study compares the performance of seven classification methods and the effect of voting for predicting metastasis outcome in breast cancer patients, in three situations: within the same dataset or across datasets on similar or dissimilar microarray platforms. Combining classification results from seven classifiers into one voting decision performed significantly better during internal validation as well as external validation in similar microarray platforms than the underlying classification methods. When validating between different microarray platforms, random forest, another voting-based method, proved to be the best performing method. We conclude that voting based classifiers provided an advantage with respect to classifying metastasis outcome in breast cancer patients. The analysis of high-dimensional gene expression datasets has posed new computational challenges. These datasets have, for example, in breast cancer research, been applied to develop classifiers predicting metastasis outcome, disease recurrence, or breast cancer survival. Some of the classification methods most frequently applied to microarray data are logistic regression , 2, suppThis study compares the performance of seven classification methods belonging to four different categories for predicting metastatic outcome in lymph negative breast cancer patients, which have not been treated with adjuvant systemic therapy. The classification methods used included an ensemble decision tree model (random forest), regression (logistic regression), four support vector machines and a neural network. To address various degrees of variation for such tasks, the comparisons were done either within the same dataset  or between different datasets . Within the same dataset model building and classification were performed using 10-fold cross-validation. Across datasets the comparisons were done in two ways. The first is in which the validations are conducted between studies using the same microarray platform , while the second encompasses validations across studies with different platforms . Furthermore, we examined the effect of combining the classification results on each sample by the seven methods into one final classification determined by majority voting, and performances compared by internal and external validation as well.The following eight datasets were used for either defining the gene features and or training purposes in the further study: samples from the studies –27 and srandomForest, glm, e1071, and nnet packages, respectively.The eight datasets above were downloaded and directly used for identification of rank-significant genes. Following this identification, the four datasets: AM, RO, TR, and MA were all standardized to have mean zero and standard deviation one. Calculations and classification were all conducted using the R free package. For random forest, logistic regression, support vector machines, and neural network we used the To determine which genes should be used to build gene expression classifiers, we used the eight publicly available datasets mentioned above, which were used in our two previous studies , 35. Thintree) from settings of 2000, 3000, 4000, and 5000 trees, and the number of subselected predictors for node splitting (mtry) with settings of: γ-parameter, controlling the spreading of samples in feature space, with the settings of 0.001, 0.01, 0.1, and 1, and for P-SVM also the polynomial degree using degrees of 2, 3, and 4.The features within each training dataset were ranked according to their random forest variable importance measure. For each feature, this value reports the standardized drop in prediction accuracy when the classlabels are permuted . For eacThe voting procedure can be regarded as a metamodel, where a sample is first fed to be predicted by each of the respective classification methods. These predictions are next fed to the final metamodel, combining each of these predictions into a final classification determined by majority voting. The voting procedure, at the level of internal dataset prediction, consisted of two steps. In the first step, each sample is classified ten times during 10 × 10 CV, meaning that each sample is given 10 votes for classification within each classification method. To prevent ties, the nine first votes were used for class decision. In the second step the final votes from each of the seven classification methods are combined into one vote, thus creating the cross-classification voting result.During external validation, every sample is classified once by each of the seven classification methods. The voting classification for each sample is determined by the winning class assigned by the seven classification methods (voters). We compared the bAcc, defined as the mean of sensitivity and specificity, of classifiers at two levels either internal or external. Internal performance was determined by the 10-times repeated 10-fold cross-validation classification accuracies. External performances obtained through transferring the trained classifier from the training sets to classify each of the independent samples are reported. In external validation, two different situations were examined: (1) between similar (RO on TR or MA) and (2) different microarray platforms (AM on TR or MA), covering Affymetrix-based classifiers validated on an Affymetrix dataset and Agilent-based classifiers validated on another Affymetrix dataset, respectively. The outcome is defined as metastasis after time of diagnosis. As this study addresses outcome classification, we did not consider the time-to-event component or censoring, due to the fact that survival analysis sometimes can be misleading when considering classification, and because transformation of time-to-event into a binary outcome can blur prediction of the classes .χ2-test; (4) the P-value of this test is stored. The steps 2 to 4 are repeated 1000 times; (5) from the 1000 tests, the median P value is reported as the statistical significance impact between the two compared methods.There is to our knowledge no standard statistics for comparing classifier performance on unbalanced datasets using the balanced accuracy as a performance measure. Therefore, in order to test the significance of the performance difference between the classification methods (defined as a significant difference between correct predictions using method A versus using method B), we used a repeated downsampled binomial test approach consisting of five steps. (1) The classifiers classification results upon the entire test data were initially converted into a balanced test result by downsampling. Downsampling obtains a class-balanced dataset from an imbalanced dataset by removing a subset of randomly selected samples from the majority class, where the number of samples removed equals the difference in sample size between the major and the minor class. In this study the majority class is the nonmetastasis class; (2) the number of samples correctly classified by one classification method but incorrectly by the other classification method and vice versa is counted; (3) the significance of the difference in these counts is determined using a binomial In this study the classifiers were developed to predict metastasis outcome using full follow-up time. To make the classifiers globally applicable and robust, we identified genes being significantly associated with outcomes across eight different studies using different microarray platforms and originating from different populations. These eight datasets are referred to as the “feature definers” (FD) . In the As a preliminary feature selection step, we identified genes being significantly associated with outcomes across eight different studies using a rank-based method , LR (P = 1.2e−5), L-SVM (P = 0.027), and P-SVM (P = 6.3e−4). NNET only borderline significantly outperformed R-SVM (P = 0.07) and S-SVM (P = 0.10). Furthermore, S-SVM also performed significantly better than RF (P = 0.049), LR (P = 9.0e−5), and P-SVM (P = 0.049), and R-SVM outperformed RF (P = 0.018). No significant performance difference was found when comparing the other classification methods.The performances within the AM and RO by each classification method were combined, and the mean performance calculated. The classifiers based on NNET had the best performance achieving a mean 10 × 10 CV bAcc of 78%, followed by S-SVM, L-SVM, R-SVM, P-SVM, RF, and LR achieving mean 10 × 10 CV bAcc of 74.1%, 72.1%, 71.6%, 70.9%, 69.4%, and 68.0%, respectively (P = 1.1e−19), LR (P = 1.7e−18), R-SVM (P = 4.2e−12), L-SVM (P = 1.4e−14), P-SVM (P = 8.9e−16), S-SVM (P = 5.8e−11), and NNET (P = 2.6e−4) , which covers the validation of RO-based classifiers on the TR and MA test data , which cP = 0.0085), L-SVM (P = 0.026), P-SVM (P = 0.0057), and NNET (P = 0.0012), and R-SVM also performed significantly better than LR (P = 0.0064), L-SVM (P = 0.025), P-SVM (P = 0.013) and NNET (P = 0.0025)  . RF perf 0.0025) . P = 6.1e−5), LR (P = 1.5e−8), R-SVM (P = 0.00026), L-SVM (P = 6.0e−8), P-SVM (P = 2.1e−7), S-SVM (P = 1.9e−6), and NNET (P = 4.7e−7) (The voting procedure increased the performance to a 72.5% mean bAcc and significantly outperformed the seven underlying methods: RF (4.7e−7) . DetaileThe performance of the classifiers was finally validated in independent datasets (Affymetrix) based on a different microarray platform from the one used by training data (Agilent) and covers the validation of AM-based classifiers on the TR and MA test data. These classifiers contained 4 to 21 features . As in tP = 3e−4), R-SVM (P = 0.038), L-SVM (P = 0.0012), P-SVM (P = 9.8e−5), S-SVM (P = 0.0015), and NNET (P = 0.032) (P = 0.059)  . In cont= 0.059) . DetaileThis study compared seven classification methods and a voting procedure ability to predict metastasis outcome in lymph node-negative breast cancer patients. The results showed that during internal assessment and external validation—methods based on voting had the best performance. Our study first compared the internal performance within a single dataset and showed that NNET had the best performance followed by the support vector machines, while RF and LR had the worst performances. This implies that at least for prediction of metastasis outcome within the same dataset—NNET and support vector machines displays superiority. This finding agrees well with other studies using cross- or hold-out procedures for performance comparisons. For example, one study comparing the performance of eight different classification methods showed that NNET and SVMs in general perform better than the other six methods for predicting outcome in eight different cancer microarray datasets . SeveralCombining the classification results by each method into classification based on voting significantly increased the internal performance. The finding of voting superiority in the internal validations suggests that voting would be valuable when applied to datasets having a combination of limited technical variation (due to using same protocols and platforms) and biological heterogeneity. Although the patients in our study are limited to being node negative, they may still be very heterogeneous due to the existence of various breast cancer molecular subgroups and the disease etiology. Voting may therefore reduce the variation associated with this biological heterogeneity. This is in line with the above-mentioned study, showing that some classification methods are more suitable for prediction tasks in complex datasets .Our finding of voting superiority agrees with four other studies: one using multiple different feature extraction methods in combination with SVM for gene microarray classification showed that using a voting-based method across all the examined combinations achieved a better 10-fold cross-validated classification performance compared to any single combination ; a seconIn the second experiment, we investigated the classifiers performance when tested in an external dataset based on a similar microarray platform. In this setup, RF and R-SVM achieved the best performances, both significantly outperforming four of the five remaining classification methods. Furthermore, the voting procedure significantly outperformed the seven underlying classification methods. This suggests that every classifier in the voting committee agrees on most of the samples that are predicted correctly and that the majority of voters do not make the same misclassifications. The finding that voting and RF have the best performances could be explained by these methods' ability to reduce the cross-study prediction variance, without simultaneously increasing prediction bias . Only a In the third experiment, the trained classifiers were externally validated on datasets based on a different microarray platform. With this setup, the performances dropped dramatically. This suggests that the data distributions in the Agilent and Affymetrix datasets are dissimilar. This is likely caused by biological and technical variation. The fact that the training and test samples originate from two different patient populations could make the data distributions dissimilar. The technical variation may originate from several sources, for example, the size of the oligonucleotides used, probe coverage, labeling, cross-hybridization, and detection limits by the scanner. Furthermore, the two platforms use different strategies for measuring the same RNA quantity. On the Agilent platform, this quantity is measured as the ratio of fluorescence intensities between a sample and a reference at each spot on the array, while the Affymetrix platform uses single channel measurements for a collection of probe sets covering one gene, which are therefore not comparable. To circumvent this obstacle, we standardized the datasets. However, this standardization seemed not to be sufficient for avoiding a drop in performance by all the classification methods used. Therefore, it is likely that the data distributions of the training and test sets are very heterogeneous thus hampering the external application of the classifiers.Although all classification methods experience a drop in performance when validating between datasets measured on different platforms, the results showed that RF remained the strongest method and significantly outperformed the six other methods. Surprisingly, the voting procedure performed poorly when validated on data measured on a different microarray platform and was a borderline significantly outperformed by RF. This is probably due to randomness by each method/voter. The finding of RF performing better than the cross-method voting procedure suggests that when tested on datasets using a different microarray technology for gene expression measurement, voting procedures based on the same classification algorithm, in the case of random forest being a collection of decision trees, are more advantageous than voting procedures based on diverse classification methods. This implies that RF compared to voting is more capable of reducing the prediction variance associated with validation across studies and platforms. Therefore, in a situation when validating between different microarray platforms and where voting is outperformed, an approach called bagging might prove advantageous. Bagging uses voters consisting of multiple classifiers developed by bootstrap resamplings from the same dataset and based on the same classification method (decisions trees in the case of random forest) . Thus, bIn the literature, there has been a limited number of studies comparing the performance of multiple classification methods, applied to across the dataset and microarray platform validations. One study by Yoshida compared a nearest template prediction method (NTP) with CART (single decision tree method), weighted voting, SVM, and k-nearest neighbor classification (k-NN) across datasets using Agilent datasets for training and Affymetrix datasets for testing . For preOur results showed that when validation is applied between two datasets of similar or different microarray platforms, LR and NNET were among the poorest performing methods. The general poor performance of LR could be due to several factors. First, a strong LR model is frequently composed of predictors being highly univariate significant and remains significant in the multivariate model. The fact that the list of 519 rank-significant genes defined by the eight feature definer datasets was reduced to a pool of 283 genes could have led to the exclusion of some highly significant genes, due to the only reason that they were not shared by the AM, RO, TR, and MA datasets, thus impairing the possibility for development of a stronger model. This could explain the poor performance by LR classifiers developed by the individual AM and RO datasets; second, an LR model requires a large sample size for providing robust maximum likelihood parameter estimation. Although the training datasets contain 151 and 286 samples, these sample sizes may not be sufficient for developing a strong model if some of the highly discriminative genes are absent; third, LR models rely on the assumption that there is no colinearity between the variables, meaning that the variables/features should be independent from each other. This assumption may be violated if predictors in a logistic model consist of, for example, gene expression features, some of which could be coregulated, thus leading to colinearity and weakening the model; finally, LR is sensitive to outliers. As we have not removed any samples from our datasets, and the possibility of outlier presence thus could be evident, this could also hamper the predictive power of the LR models.tansig activation function and using single value decomposition for parameter estimation had a better performance compared to the three other methods and that this classifier outperformed support vector machines, principle component analysis classifiers, and Fisher discriminant analysis classifiers [The finding of NNET had a high internal 10-fold cross validation performance but a weak external validation performance could suggest that the NNET classifiers are not very generable. Another explanation could be that the transfer function used by the neural network was a sigmoid function, which is identical to that used in logistic regression, thus leading to some of the weaknesses observed in logistic regression, although the parameter estimation in neural networks is not conducted by maximum-likelihood but by a gradient descent algorithm. Interestingly, a study has compared the classification performance of four different single hidden layer feedforward neural networks on three microarray gene expression cancer dataset, showing that an SVD-neural classifier based on a ssifiers . This imIn conclusion, voting-based classifiers provided an advantage with respect to classifying metastasis outcome in breast cancer patients. When testing was performed within the same dataset or between datasets using similar microarray platforms, combining class decisions by multiple classification methods significantly increased the classification performance. Random forest, a voting-like method, proved to be the strongest method when testing was performed in datasets based on a different microarray platform.Supplementary Table 1 shows list of 283 rank-significant genes for classifier building, by gene symbol (left column) and by description (right column).Supplementary Table 2 shows the internal validation results within the AM and RO datasets and the mean of AM & RO. These results are shown with respect to sensitivity (Sen), specificity (Spe) and balanced accuracy (bAcc) based on the ten times repeated ten folds cross-validation by each of the classification methods used. The classification are as follows: random forest (RF), logistic regression (LR), support vector machines with a radial (R-SVM), linear (L-SVM), polynomial (P-SVM), sigmoid kernel (S-SVM), a neural network with a single hidden layer (NNET) or cross method voting (Voting).Supplementary Table 3 shows the testing results by classifiers developed in the RO dataset and validated in TR or MA and the combined mean performance in TR and MA. These results are shown with respect to sensitivity (Sen), specificity (Spe) and balanced accuracy (bAcc), by each of the classification methods used. The classification are as follows: random forest (RF), logistic regression (LR), support vector machines with a radial (R-SVM), linear (L-SVM), polynomial (P-SVM), sigmoid kernel (S-SVM), a neural network with a single hidden layer (NNET) or cross method voting (Voting).Supplementary Table 4 shows the testing results by classifiers developed in the AM dataset and validated in TR or MA and the combined mean performance in TR and MA. These results are shown with respect to sensitivity (Sen), specificity (Spe) and balanced accuracy (bAcc), by each of the classification methods used. The classification are as follows: random forest (RF), logistic regression (LR), support vector machines with a radial (R-SVM), linear (L-SVM), polynomial (P-SVM), sigmoid kernel (S-SVM), a neural network with a single hidden layer (NNET) or cross method voting (Voting).Click here for additional data file.


DNA methylation plays a very important role in the silencing of tumor suppressor genes in various tumor types. In order to gain a genome-wide understanding of how changes in methylation affect tumor growth, the differential methylation hybridization (DMH) protocol has been developed and large amounts of DMH microarray data have been generated. However, it is still unclear how to preprocess this type of microarray data and how different background correction and normalization methods used for two-color gene expression arrays perform for the methylation microarray data. In this paper, we demonstrate our discovery of a set of internal control probes that have log ratios (M) theoretically equal to zero according to this DMH protocol. With the aid of this set of control probes, we propose two LOESS  normalization methods that are novel and unique for DMH microarray data. Combining with other normalization methods , we compare four normalization methods. In addition, we compare five different background correction methods.We study 20 different preprocessing methods, which are the combination of five background correction methods and four normalization methods. In order to compare these 20 methods, we evaluate their performance of identifying known methylated and un-methylated housekeeping genes based on two statistics. Comparison details are illustrated using breast cancer cell line and ovarian cancer patient methylation microarray data. Our comparison results show that different background correction methods perform similarly; however, four normalization methods perform very differently. In particular, all three different LOESS normalization methods perform better than the one without any normalization.It is necessary to do within-array normalization, and the two LOESS normalization methods based on specific DMH internal control probes produce more stable and relatively better results than the global LOESS normalization method. Microarray technology has been used extensively in genetic and epigenetic studies over the last ten years. Several microarray platforms are available including the single-channel Affymetrix oligonucleotide arrays, the two-color (or two-channel) cDNA arrays, and Agilent two color arrays. In the two-color use, which is the focus of this paper, two samples (or target genes) are labeled using two different fluorophores  and hybridized simultaneously onto each probe (or spot) of the array (or chip). Then the arrays are laser-scanned and images are processed to obtain the data for analysis . In genePreprocessing gene expression microarray data obtained from two-color cDNA microarray and single-color Affymetrix array have been extensively studied -4. Howevf, Gf, Rb, and Gb respectively for the two channels . The foreground estimates (Rf and Gf) are the overall measurement of the intensity at each probe (spot) in each channel. The background measurements  are usually an estimate of the ambient signal around the round circle of each spot. This may be due to unequal distribution of hybridization solution, spatial bias . As used in , we let The basic rationale of normalization is to remove or adjust for artifacts caused by microarray technology rather than biological differences of the samples between printed probes. In order to do so, it is helpful to normalize the data with some known information, for example, some control probes that are known to be non-differentially methylated. In this section, we first identify the probes that are known to be not differentially methylated based on the DMH protocol, that is, probes with M = 0. DNA fragments are restricted by two MSREs, HinpI and HapII, which have the recognition cutting sites CGCG and CCGG respectively. If a DNA fragment contains at least one cutting site that is not methylated, it will be restricted , and will not be hybridized onto the microarray. If a DNA fragment does not have any cutting sites, it will not be digested by any MSREs and can be hybridized onto the array. If all the cutting sites of a DNA fragment are methylated, this fragment will be saved for hybridization onto the array. These three typical types of DNA fragments with examples are given in Table If there is not any recognition cutting site within a long region around a probe, the hybridization from two channels are supposed to be the same whether or not there is methylation on the DNA fragment that is hybridized onto this probe. Therefore, theoretically the log 2 ratio of this probe methylation signal will be 0 . In this paper, we identified the probes around which there are no recognition cutting sites within L = 900 base region. These probes are selected in the following way. For each probe, we check the regions that are L = 900 bases around the center of each probe. That is, there are L/2 bases on each side of the center of the probe. Then we check how many restriction cutting sites are around this probe within these L bases. If there are no cutting sites , we claim that this probe is a non-differential methylated internal control probe with M = 0. These internal control probes are important because we can make full use of them to do normalizations. Because the length of DNA fragments is about 400-500 bp, we use L = 900 bases assuming that DNA fragments can be hybridized onto the methylated probes and regions evenly. 199 probes, which have no recognition cutting sites around 900 bp, are identified.Similar to the LOESS normalization and composite LOESS normalization using control probes in the context of the gene expression microarray preprocessing, we introduce the control and composite LOESS normalization for DMH methylation microarray data. Combining with the standard LOESS normalization and the method without any normalization, there are four normalization methods. Details about these four normalization methods are described as below:new = Mobserved.1) None: no normalization is done, and Mnew = Mobserved -fLOESS(A).2) Global LOESS normalization: all the biological probes are used, and Mnew = M- fcontrol(A).3) Control LOESS normalization: We fit a LOESS curve only using the 199 control probes, and for each M value which corresponds to an A value, we have Mcontrol(A) + (1-a) fall(A), and Mnew = Mobserved - g(A), where fall and fcontrol are the global and control LOESS curve, and 'a' is defined as the proportion of genes less than a given intensity A value [4) Composite LOESS normalization: This is to let the normalization curve to be a weighted average of the global LOESS curve and the control probe LOESS curve. That is, at each specific average intensity level A, the new normalized estimate is g(A) = a* f A value .In Figure ap = arraya+probep + errorap, 0, where we let p0 = 0.05, 0.04, 0.03, 0.02 and 0.01. At each p-value threshold p0, we have an integer methylation score n for each CpG island. The range of n is from 0 to N, where N is 40 and 26 for breast cancer cell line data and ovarian cancer data respectively. There are in total of Nm and NHK methylation scores for known methylated genes/CpG islands  and housekeeping genes (NHK = 47) respectively.All 20 different preprocessing methods as described are implemented using the LIMMA package  of Bioco0 that ranges from 0 to N, the false positive rate is the ratio of the number of un-methylated housekeeping genes/CpG islands with methylation scores greater than or equal to C0 and the total number of housekeeping genes (NHK), and the true positive rate is the ratio of the number of known methylated genes/CpG islands with methylation scores greater than or equal to C0 and the total number of known methylated genes (Nm). For both T.stat and AUC, the larger a statistical measurement is, the better a processing method is.In order to see if known methylated genes and housekeeping genes are identified correctly, we use two different statistical measurements for known methylated genes and housekeeping genes. One is the statistics of mean difference of methylation scores of two groups of genes divided by their variance. That is, For each of the two data sets, we choose five p-value cutoffs, 0.05, 0.04, 0.03, 0.02, and 0.01. At each p value cutoff point, we calculate the two statistical measurements: "T.stat", that is, mean difference of methylation scores of two groups divided by their variance, and "AUC", that is, the AUC of a ROC curve, for each of the 20 preprocessing methods. The results are listed in Tables In order to further compare the performances of different normalization and background correction methods, at each p-value cutoff point we calculate the average of each statistical measurement for each normalization method (across five different background correction methods) and for each background correction method . The average scores are plotted in Figure In this paper, we compare four normalization and five background correction methods. There are more differences among normalization methods than background correction methods. Among four normalization methods, the result of no normalization performs the worst in that both statistical measurement scores are the smallest in both data sets. Therefore, it is necessary to do normalization. The control LOESS and composite LOESS normalization methods provide relatively stable results in both data sets when the p value threshold changes. However, the LOESS normalization results are more variable across different p-value cutoff points. On the other hand, the differences among background correction methods are relatively small. Our comparison results show that even though some background correction methods are slightly better than others, the differences are much smaller than the differences among normalization methods. With appropriate normalization, the need for background-corrected DMH methylation data might be obviated. This conclusion is consistent with the findings of , which aThe housekeeping genes used as non-methylated genes are selected from publicly available data  using thIn Table The effectiveness of a normalization method depends on whether or not its assumption is valid. The LOESS normalization assumes that each array has a larger number of probes (or genes) that are not differentially methylated (expressed), or there is an approximately equal number of positive and negative log ratios. It also requires that a certain number of probes with these characteristics should cover a full range of intensities . HoweverIn this paper, we did not compare with the Agilent Feature Extraction Software  because The authors declare that they have no competing interests.SS developed and implemented the models, performed all statistical analyses, drafted and revised the manuscript. PSY and YWH were involved in the data collection and helped in preparation of the manuscript. THMH oversaw the project and revised the manuscript. SL provided suggestions on the project and revised the manuscript. All authors have read and approved the final document.