ComBat) appears to be most suitable for small sample sizes and multiple batches.Batch effects due to sample preparation or array variation  may influence the results of microarray experiments and thus mask and/or confound true biological differences. Of the published approaches for batch correction, the algorithm “Combating Batch Effects When Combining Batches of Gene Expression Microarray Data” (RMA). Data were batch-corrected for different acquiry dates using ComBat and the efficacy of the correction was validated using hierarchical clustering.Synovial fibroblasts  were obtained from rheumatoid arthritis (RA) and osteoarthritis (OA) patients (n = 6 each) and stimulated with TNF-α or TGF-β1 for 0, 1, 2, 4, or 12 hours. Gene expression was analyzed using Affymetrix Human Genome U133 Plus 2.0 chips, an alternative chip definition file, and normalization by Robust Multi-Array Analysis , but also to all time points following stimulation except for the late 12 hour time point. Batch-corrected data then allowed the identification of differentially expressed genes discriminating between RA and OA. Batch correction only marginally modified the original data, as demonstrated by preservation of the main Gene Ontology (GO) categories of interest, and by minimally changed mean expression levels  or variances for all genes of interest. Eight genes from the GO category “extracellular matrix structural constituent”  were differentially expressed between RA and OA (RA > OA), both constitutively at time point 0, and at all time points following stimulation with either TNF-α or TGF-β1.Batch correction appears to be an extremely valuable tool to eliminate non-biological batch effects, and allows the identification of genes discriminating between different joint diseases. RA-SFB show an upregulated expression of extracellular matrix components, both constitutively following isolation from the synovial membrane and upon stimulation with disease-relevant cytokines or growth factors, suggesting an “imprinted” alteration of their phenotype. Gene expression microarray technology measures the expression of thousands of genes in one single array by using multiple probes to assay each transcript. It is a widely-used tool for identifying genes whose expression changes in response to specific treatments. There are several concerns with regard to the reliability of DNA microarray technology in the study of diseases.Microarray experiments are costly and time-consuming. Many studies use multiple arrays, with experiments performed at different times, on different array charges or even with different microarray platforms. The resulting gene expression data could thus be affected by non-biological variables. These systematic differences between the measurements are commonly referred to as “batch effects”. There are several causes for batch effects, as outlined below: • Ambient conditions during the sample preparation and handling, such as room temperature and ozone levels • Storage and shipment conditions of the biological samples and/or the arrayscRNA/cDNA synthesis • Amplification, labeling, and hybridization protocol: Use of reagents from different batches • Sites/laboratories: Different laboratories have different operating procedures • Chip type/charge/platform: The array quality varies from batch to batch • Washing conditions: Temperature, ionic strength • Scanner: Type, settings, and the calibration drift.Although some of these batch effects can be minimized or even prevented with appropriate precautions and experimental design, certain batch effects are unavoidable. Some studies require large sample sizes and have to be carried out over many months or sometimes years. In clinical experiments, experiments are often driven by the availability of the samples, which cannot be specifically controlled for in the original study design. Combining data from different batches without removing batch effects can give rise to misleading results, since the bias introduced by the non-biological nature of the batch effects can be strong enough to mask and/or confound true biological differences. Thus, there is a need to identify and remove these masking effects before further processing.MAS5, Robust Multi-Array Analysis (RMA), and dChip for Affymetrix chips, median scaling for GE-CodeLink microarrays, and LOWLESS-based methods for cDNA two-color microarrays. Common to all normalization methods is that they are not specifically designed to remove batch effects reflected by systematic differences between two or more groups of samples. Consequently, batch effects may often remain after normalization. However, of thousands of papers dealing with DNA microarrays published in the last 5 years , only few address the potential existence of batch effects and/or their correction. Of the 219 papers using microarray data published from January 1 to July 1, 2010, not even ten percent took this issue into account  calculated using GPower[g GPower.Click here for fileTable S3. Differentially expressed genes calculated with LIMMA for 20 permutations.Click here for fileFigure S4. Cluster plots for time point 0 on the basis of 20 permutations of the disease status (RA and OA).Click here for fileTable S4. Mann–Whitney U tests for the comparison between RA and OA in a), or for the comparison between TNF-α and TGF-β1 in b); +, significant difference; -, lack of significant difference; *, significance test is not applicable .Click here for file


Diagnosis techniques using urine are non-invasive, inexpensive, and easy to perform in clinical settings. The metabolites in urine, as the end products of cellular processes, are closely linked to phenotypes. Therefore, urine metabolome is very useful in marker discoveries and clinical applications. However, only univariate methods have been used in classification studies using urine metabolome. Since multiple genes or proteins would be involved in developments of complex diseases such as breast cancer, multiple compounds including metabolites would be related with the complex diseases, and multivariate methods would be needed to identify those multiple metabolite markers. Moreover, because combinatorial effects among the markers can seriously affect disease developments and there also exist individual differences in genetic makeup or heterogeneity in cancer progressions, single marker is not enough to identify cancers.We proposed classification models using multivariate classification techniques and developed an analysis procedure for classification studies using metabolome data. Through this strategy, we identified five potential urinary biomarkers for breast cancer with high accuracy, among which the four biomarker candidates were not identifiable by only univariate methods. We also proposed potential diagnosis rules to help in clinical decision making. Besides, we showed that combinatorial effects among multiple biomarkers can enhance discriminative power for breast cancer.In this study, we successfully showed that multivariate classifications are needed to precisely diagnose breast cancer. After further validation with independent cohorts and experimental confirmation, these marker candidates will likely lead to clinically applicable assays for earlier diagnoses of breast cancer. Breast cancer is currently the second most common type of cancer  after luOf the various types of samples, diagnostic techniques using urine are advantageous in terms of clinical application to real patients because these techniques are non-invasive, inexpensive, and easy to perform, likely leading to earlier detection for malignancies . In addiSince multiple genes or proteins would be involved in developments of complex diseases such as breast cancer, multiple compounds including metabolites would be related with the complex diseases, and multivariate methods would be needed to identify those multiple metabolite markers. Moreover, because combinatorial effects among the markers can seriously affect disease developments and there also exist individual differences in genetic makeup or heterogeneity in cancer progressions, single marker is not enough to identify cancers. Figure Therefore, in this study, we proposed classification models using multivariate classification techniques Figure  and deveUrine samples were collected from female breast cancer patients  and healthy subjects as normal controls  at the Samsung and Hanyang University Medical Centers . All study subjects underwent the same diagnostic procedures, i.e., a physical examination of the breasts, mammography, and ultrasonography as detailed by the American Joint Committee on Cancer staging. Breast cancer patients underwent either a modified radical mastectomy (MRM) or a lumpectomy with an auxiliary lymph node dissection. Both pre- and postoperative urine samples were collected, with the latter obtained 2 weeks after surgery. The sex- and age-matched controls had no evidence of benign or malignant breast disease. All of the urine samples were collected in the early mornings and kept frozen at −20°C until analysis. In this study, we used only pre-operative and normal samples to construct models to distinguish between breast cancer and normal samples.Urinary metabolites were prepared by extraction under four conditions. First, each urine sample (1 mL) was loaded into a Strata-X cartridge , washed with distilled water (1 mL), and extracted with 4 mL of methanol. The eluate was divided in half, and one half of the eluate (2 mL) was evaporated and dried in the desiccator over 30 min. For the second condition, the other half of the eluate was evaporated, dissolved in 1 mL of 0.2 M acetate buffer (pH 5.2), and hydrolyzed with β-glucuronidase/arylsufatase (50 μL) from Helix Pomatia  at 55°C for 3 hr. After cooling, urinary metabolites were extracted with 5 mL of diethyl ether by mechanical shaking for 20 min and centrifugation at 2500 rpm for 5 min. The separated organic layer was evaporated under nitrogen and kept in the desiccator over 30 min. For the third condition, the remaining aqueous layer was adjusted to pH 1-2 with 200 μL of 3 M HCl and extracted with 5 mL of diethyl ether. The separated organic extract was evaporated and dried. For the fourth condition, the remaining aqueous layer was adjusted to pH 10 – 11 with 0.73 g of K2CO3 and extracted with 5 mL of diethyl ether and dried. All dried extracts were derivatized by 50 μL of MSTFA/TMSI/TMCS  mixture at 60°C for 15 min and injected into a GC-MS system.All samples prepared were separated through a Ultra-1 capillary column  and analyzed by a Thermo Finnigan GC-MS system consisting of a Trace 2000 GC and a Polaris Q mass-selective detector in the scan range of m/z 50 – 800 .In this study, we have organized an analysis procedure to construct classification models based on metabolome datasets using various multivariate classification methods. The procedure consists of four stages: data standardization, preprocessing, feature selection, and classification. Figure 1) Data standardization stage: raw data from the mass spectrometry machine is converted into standard formats. The mzXML  and CDF 2) Preprocessing stage: multiple steps are used to preprocess raw data, including smoothing, peak detection, and peak alignment. For those purposes, MZmine software ,15 is su3) Feature selection stage: it is critical in the construction of classification models and in biomarker discovery to extract the meaningful variables from among thousands of variables . To reflect various types of distributions of data, we have used a variety of feature selection algorithms comprising univariate (t-test and chi-square), multivariate (the correlation-based feature selection (CFS) algorithm), and model-embedded methods (Decision tree  and Rand4) Classification stage: with selected feature sets, classification models are constructed. In this work, Decision tree and Random forest are used as tree-based non-parametric methods. Support vector machine (SVM) is used as a generalized linear discriminative method. An Artificial neural network (ANN) is not used since it is known that SVM outperforms ANN unless training datasets are sufficient , and ANNIn addition, for visualization of datasets the dimension reduction algorithms PCA and PLS, are used, allowing separablity of given datasets to be checked.We have constructed models to classify urine metabolome data into breast cancer and normal, and we have identified several potential biomarkers for breast cancer, which are detectable in urine samples, with the metabolome-data analysis procedure described above. First, a total of 26,306 features, which are m/z valued-peaks with retention time information, are standardized through our data converter and then preprocessed by MZmine .Next, to inspect overall distributions of the data, PLS analysis has been performed Figure . Two claAmong 26,306 peaks, feature selection has been performed to extract significantly meaningful metabolite peaks between two groups of people. After many trials using different feature selection methods, 10 feature sets have finally been selected by Decision Tree   in diffeUsing selected feature sets, classification models have been constructed through various classification algorithms in our analysis procedure with 10-fold cross-validation. In both of the two feature sets, overall accuracies were more than 90 percent. In the first set, performance, including both sensitivity and specificity, was more than 94 percent in all of the classifiers except the linear SVM. Table  and 3 FiLast, to confirm the multivariate classification’s power, a t-test, which is a univariate method, has been applied to the identified feature sets. A metabolite peak of 191.2261 m/z with 535.3876 retention time shows a p-value of 2.866e-06 in the t-test, and it seems that this peak (M191) can be identified by both the univariate and multivariate methods Figure . HoweverTo apply the proposed classification models and potential biomarkers to real clinical situations, the following analysis procedures are required:First, additional validations must be performed using independent datasets that may have different characteristics from the data used in this study so as to guarantee the generality of the proposed models; there have been a few studies -22 perfoHowever, although further work remains to be done for actual application in clinical settings, this study proposes several possibilities for classification and biomarker discovery research using the urine metabolome. The first possibility is that breast cancer can be recognized by analyzing urine metabolome samples. This becomes more obvious with the results of partial least square analysis (PLS), showing that urine samples have the information that can discriminate between normal and breast cancer groups. Second, this study shows the possibility that reliable diagnosis models and potential markers, whose performance were all better than 94 percent, can be constructed from GC-MS urine metabolome datasets. After experimental validation, the proposed marker candidates will likely lead to clinically applicable assays for earlier diagnosis of breast cancer. Furthermore, this study shows the possibility that multivariate methods can discover ‘hidden features’ that univariate methods cannot easily find. It seems that this capability is very important with complex, noisy data, such as urine metabolome data that may be affected by heterogeneity in cancer progression, individual differences of genetic makeup, or the averaging of multiple characteristic signals into undistinguishable signals.Diagnostic assays based on urine samples have several major advantages, including non-invasiveness, inexpensiveness, and ease of performance, that will likely lead to impacts in clinical settings including the earlier detection of malignancies. As the end products generated by an organism, metabolites are closely linked to phenotypes and can be diagnostic clues regarding abnormalities. In this study, we have proposed analysis procedures using multivariate classification to more precisely analyze these urine metabolome data. By using multivariate classification methods, we were able to more effectively analyze urine metabolome datasets for which univariate analysis is not powerful enough due to the data’s complexity. We have found in our experiments that this multivariate approach can identify features that are not recognizable by univariate methods. In conclusion, we have proposed classification models and five potential urinary biomarkers for breast cancer diagnosis. Our findings will be helpful in real clinical settings if additional validations and experimental confirmations are performed.The authors declare that they have no competing interests.YK designed metabolome data analysis procedures, implemented them, analyzed classification results, and wrote the manuscript. IK co-designed the analysis procedures and advised on statistical methods. BJ and BC produced metabolome profiles and analyzed them. DL conceived this research and critically reviewed the whole data analysis works and the manuscript.


We utilized abundant transcriptomic data for the primary classes of brain cancers to study the feasibility of separating all of these diseases simultaneously based on molecular data alone. These signatures were based on a new method reported herein – Identification of Structured Signatures and Classifiers (ISSAC) – that resulted in a brain cancer marker panel of 44 unique genes. Many of these genes have established relevance to the brain cancers examined herein, with others having known roles in cancer biology. Analyses on large-scale data from multiple sources must deal with significant challenges associated with heterogeneity between different published studies, for it was observed that the variation among individual studies often had a larger effect on the transcriptome than did phenotype differences, as is typical. For this reason, we restricted ourselves to studying only cases where we had at least two independent studies performed for each phenotype, and also reprocessed all the raw data from the studies using a unified pre-processing pipeline. We found that learning signatures across multiple datasets greatly enhanced reproducibility and accuracy in predictive performance on truly independent validation sets, even when keeping the size of the training set the same. This was most likely due to the meta-signature encompassing more of the heterogeneity across different sources and conditions, while amplifying signal from the repeated global characteristics of the phenotype. When molecular signatures of brain cancers were constructed from all currently available microarray data, 90% phenotype prediction accuracy, or the accuracy of identifying a particular brain cancer from the background of all phenotypes, was found. Looking forward, we discuss our approach in the context of the eventual development of organ-specific molecular signatures from peripheral fluids such as the blood. From a multi-study, integrated transcriptomic dataset, we identified a marker panel for differentiating major human brain cancers at the gene-expression level. The ISSAC molecular signatures for brain cancers, composed of 44 unique genes, are based on comparing expression levels of pairs of genes, and phenotype prediction follows a diagnostic hierarchy. We found that sufficient dataset integration across multiple studies greatly enhanced diagnostic performance on truly independent validation sets, whereas signatures learned from only one dataset typically led to high error rate. Molecular signatures of brain cancers, when obtained using all currently available gene-expression data, achieved 90% phenotype prediction accuracy. Thus, our integrative approach holds significant promise for developing organ-level, comprehensive, molecular signatures of disease. One important goal in systems medicine is to develop molecular diagnostics that can accurately and comprehensively report health and disease states of an organ system While many molecular signature studies have focused on identifying differences between case  and control , a more clinically relevant and challenging task is the multi-category classification problem. This task pertains especially to identifying signatures for molecular screening and monitoring purposes. Such signatures need to detect and stratify various pathological conditions simultaneously; they must therefore be highly specific for a particular disease as well as tissue of origin. The successful identification of more reliable and efficient molecular signatures will also be critical for the blood-based, organ-specific diagnostics envisioned for the future Data-driven, hierarchical approaches to multi-category classification have been investigated extensively in machine learning batch effects, which arise from differences or inconsistencies in experimental protocols, data quality, data-processing techniques, and laboratory conditions and personnel A significant drawback to the use of molecular signatures derived from high-throughput—particularly transcriptomic—data is limited reproducibility and performance accuracy, which is often observed across independent studies of what are considered the same disease phenotype. This drawback holds true for both binary and multi-category classification problems. The lack of robustness, even for promising results, can be attributed to molecular heterogeneity within tumors or other diseased tissue-samples A promising method to address some of these limitations in robustness is to accumulate and combine data from many independent studies into large meta-analyses In this study, we developed a computational approach called Identification of Structured Signatures And Classifiers (ISSAC) to identify molecular signatures that simultaneously distinguish major cancers of the human brain. From an integrated dataset of publicly available gene expression data, ISSAC provides a global diagnostic hierarchy and corresponding structured brain cancer signatures composed of sets of gene-pair classifiers. The signal in the transcriptomics data was sufficient to develop accurate, comprehensive signatures, as long as the training set was sampled from the same population as the validation set . In contrast, training on one dataset and testing against an independent set  generally failed to reach the same performance due to biological and technical sources of dataset variation. To address this issue, we found that integration of datasets from multiple studies enhanced the disease signal sufficiently to mitigate batch effects and greatly improve independent validation results for brain cancers.We compiled a multi-study, integrated dataset of brain cancer and normal transcriptomes Our marker panel consists of 39 total gene pairs and 44 unique genes (46 unique Affymetrix microarray probe IDs). Details on how the gene-pair sets were chosen as classifiers, and how they are used for phenotype prediction, can be found in the Materials and Methods section and Here, we summarize the overall method of ISSAC into three main steps ; a detaiall other phenotypes. For example, the classifiers of node 6 in PRPF40A and PURA in reversal of relative expression , as previously reported et al. is based on two genes  for distinguishing between two phenotypes : If the expression of gene i is greater than that of gene j for a given profile, then the phenotype is classified as class A; otherwise, class B. It has been shown that using such simple decision rules with only a small number of gene pairs can lead to highly accurate supervised classification of human cancers i and j in Second, ISSAC identifies gene-pair classifiers corresponding to the nodes and edges of the diagnostic hierarchy  and 3. Bcoarse-to-fine fashion: a classifier is executed if and only if all of its ancestor classifiers have been executed and have returned a positive response—i.e., predicted the phenotypes in each node. The cumulative outcome of the node classifiers for a given expression profile is the set of its candidate phenotypes, corresponding to all the leaves of the hierarchy that were reached and tested positively. This property means that it is possible to traverse multiple paths to multiple leaf nodes, and thus multiple diagnoses may be made in this step . For tie-breaking purposes, the decision-tree classifiers at the edges of the diagnostic hierarchy are used to reach a unique diagnosis.Overall, the collection of node classifiers represent a series of coarse-grained to fine-grained explanations of the hierarchical groupings and are used in diagnosis to screen for phenotype-specific expression patterns (described below). Thus, the hierarchy of binary predictors guides classification of an expression profile in a dynamic Finally, ISSAC uses the gene-pair classifiers for class prediction . Given aTo estimate the robustness of signature accuracy, it is best to test molecular signatures against datasets  that are truly independent of the training set . To study the effects of training across multiple studies, we used glioblastoma (GBM), where we had the highest number of transcriptomic datasets for the phenotype. We trained ISSAC on each of the five transcriptomic datasets  of GBM, coupled in each case to all the data from the other brain phenotypes. The full multi-class signatures were completely relearned (every step) with the only difference in each case being which single GBM dataset was included in the training stage. We then assessed the accuracy of correctly classifying GBM transcriptomes measured in the four held-out datasets from all other possible phenotypes. We term this evaluation method as “hold-one-lab-in validation”.The overall hold-one-lab-in validation performance, or the average of all classification accuracies in We found considerable discrepancy between the minimum and maximum validation set accuracies for training sets GSE4412  and GSE4290 (17% and 92%) . This inWe next analyzed how the multi-study integration approach affects performance robustness. One of each of the five datasets of GBM was sequentially withheld as the validation set, while all remaining gene expression data  were used for training. The GBM signature was then evaluated on the held-out validation set. We term this strategy as “leave-one-lab-out validation”. Classification accuracies using this approach ranged from 63%  to 100%  . The aveTo evaluate how multi-study dataset integration alone affects performance robustness independent of sample size, we performed hold-one-lab-in and leave-one-lab-out validations for the studies with the largest number of samples, GSE4412, GSE4271, and GSE4290  while training on the same number of samples for GBM. More specifically, the same steps in the analyses of The average performances of hold-one-lab-in and leave-one-lab-out validations were 47% and 70%, respectively. Overall, the results were consistent with our two aforementioned conclusions: 1) when a molecular signature is learned from an individual dataset, its ability to accurately and precisely represent phenotype features across a broad population highly varies depending on the particular dataset used for training ; and 2) Lastly, we used the results in Overall, we have shown that when a broader range of conditions within a particular phenotype is presented during the classifier-learning stage, ISSAC can better distinguish the true disease signal from noise prior to independent validation. However, single and/or smaller training sets that were used to define the classifiers might not be representative of, or generalizable to, larger populations – leading to poor validation results. Therefore, the utilization of all currently available datasets from various sources and conditions may be a promising approach to finding novel diagnostic markers, and eventually bringing the successful adaptation of genomic biomarkers into clinical practice. Also, prospective design of studies is generally best when they utilize multiple sites to avoid over-fitting to particular contexts.It is worth mentioning that in some cases, molecular signatures from a single source can have (or at least appear to have) superior performance, as demonstrated by the molecular signatures from GSE4271. Specifically, training on a single GSE4271 data set provided higher accuracy 87%,  than leaAs shown by our leave-one-lab-out validations, learning signatures across multiple datasets significantly improved average classification accuracy with concomitant reduction in performance variance. In this regard, the brain cancer marker-panel obtained using all currently available microarray data simultaneously  and 3 shThe classification performance of this comprehensive brain cancer marker-panel was evaluated by ten-fold cross-validation . Our maret al. analyzed gene expression of 20 anatomically distinct regions of the central nervous system Four other brain cancers  have estimated accuracies of at least 91%, suggesting clear differences between them and the other phenotypes at the transcriptomic level. The anatomical region specificity of these four cancers may have contributed toward their highly accurate separation, as there are regional areas of unique gene expression patterns. Roth Compared to the cross-validation accuracies of other phenotypes, lower performance was observed for GBM and oligodendroglioma (OLG) . This could have been mainly a consequence of the limited ability of the marker-panel to correctly differentiate these two cancers from each other. Indeed, the distinction of these two phenotypes from transcriptomics seems to be rather difficult in general, and our accuracies here are comparable to those reported previously in two-phenotype comparison studies NRCAM , was found to be over-expressed in high-grade gliomas and to promote tumor cell invasion FLNA was detected in the serum of high-grade astrocytoma (grade 3 and GBM) patients ANXA1, a gene that encodes an anti-inflammatory phospholipid binding protein, was implicated in astrocytoma progression FLNA and ANXA1 as two classifier genes expressed higher in GBM than in oligodendroglioma (nodes 12 and 13). The basic helix-loop-helix (bHLH) transcription factor OLIG2 is innately expressed in oligodendrocytes and was recently characterized as a key antagonist of p53 function in neural stem cells and malignant gliomas OLIG2 as an EPN classifier in this study (node 8), OLIG2 expression was used as a negative marker to differentiate EPN from other gliomas SEMA3E, one of several classifier genes for PA (node 11), has been reported to drive invasiveness of melanoma cells in mice IDH2 (node 4) in GBM is well known, with occurrence reported in 80% of secondary glioblastomas Several genes in our marker panel are strongly associated with brain cancers, suggesting putative relationships to the underlying pathophysiology of their corresponding phenotypes. One such gene is IDH2 and GMDS (node 4). IDH2 converts isocitrate to α-ketoglutarate within the TCA cycle. This reaction produces NADPH, which not only is an essential cofactor for many metabolic reactions, but also helps to protect the cell against oxidative damage GMDS aids the biosynthesis of GDP-fucose from GDP-mannose in mannose metabolism, in which NADPH is produced IDH2 and GMDS participate in the conversion between NADP+ and NADPH is interesting, considering the well-known alteration to cellular metabolism and deregulated redox balance in cancer IDH2 and/or GMDS, or changes in the regulatory network that controls the expression of these two genes, may affect cellular redox balance and functions of other metabolic enzymes.One of the classifier gene pairs involved in the differentiation between meningioma and the remaining five brain cancers  are two metabolic enzymes, TLE4 and OLIG2 gene pair is used to differentiate EPN from GBM, OLG, and PA (node 8). TLE4, a human homolog of the Drosophila Groucho protein, represses the Wnt and FGF developmental signaling pathways OLIG2 expression in zebrafish The i and gene-set j  showed that the genes reflect disease properties. Specifically, the genes that are in gene-set i, or those expressed higher in GBM compared to OLG, were the most enriched in the biological process of ‘Immunity and Defense’  on an exhaustive list of gene pairs discriminating GBM and OLG . Our staDefense’ ; this isThe work reported herein has focused on identifying a structured molecular signature that can separate major brain cancers simultaneously, as well as on evaluating issues related to reproducibility in molecular signatures. However, our long-term motivation for wanting molecular signatures of an organ system is ultimately to find corresponding signatures in the blood, where they can be assayed non-invasively. Blood bathes virtually all organs, which secrete proteins and nucleic acids. Subsets of these secreted biomolecules can potentially constitute disease signatures for molecular diagnostics, as measurement technologies mature. Moreover, the blood is easily accessible in contrast to biopsies of diseased organs for obtaining transcript or protein profiles. In this regard, the brain represents an organ system where a critical need exists to develop non-invasive techniques to monitor its health state through secreted proteins.Previously, organ-specific proteins have been detected in blood; when these proteins changed in concentration or chemical structure, the tissue origin of this change was identified APODCALUCD163CHI3L1CSF1EGFRIGFBP2NID1PDGFCPSG9PTNAs proof of concept and to provide candidates for targeted proteomics analysis, we performed the above transcriptomic analysis of finding brain cancer signatures using only the genes that are annotated to encode extracellular proteins . We trained ISSAC on a total of 767 genes that matched this criterion, which led to a new brain cancer marker-panel composed of 41 gene-pair classifiers from 71 unique features . When loOur marker-panel, composed entirely of genes encoding extracellular products, obtained an average classification accuracy of 87% in 10-fold cross-validation , which cAll transcriptomic data used in our analysis are publicly available at the NCBI Gene Expression Omnibus (GEO). We integrated 921 microarray samples of six brain cancers  and normal brain across 16 independent studies into a transcriptome multi-study dataset. Importantly, we obtained the raw data (.CEL files) from each of these studies and preprocessed them uniformly using identical techniques to greatly reduce extraneous sources of technical artifacts (discussed below). All data manipulation and numerical calculations were performed using MATLAB (MathWorks).no samples were excluded because their gene expression profiles were abnormal for their associated phenotypes. We are aware that this may allow mislabeled samples, e.g. samples that were originally misclassified by the histopathologist upon class labeling , we used the following strict criteria and reasoning for brain phenotype selection: 1) Expression profiles must have been conducted on either the Affymetrix Human Genome U133A or U133 Plus 2.0 microarray platform. This allowed maximum microarray sample collection without considerable reduction in number of overlapping classifier features . 2) Transcriptomic datasets  for each phenotype must have been collected from at least two independent sources to help mitigate batch effects. 3) All datasets must have consisted of no fewer than 5 microarray samples. 4) All datasets must have originated from primary brain tumor or tissue biopsies. Expression profiles from cell-lines or laser micro-dissections were not used in our study to better ensure sample consistency. 5) Raw microarray intensity data (.CEL files) must have been available on GEO for consensus preprocessing (described below). 6) Sample preparation protocols must have been fully disclosed on GEO. 7) All microarray samples in a dataset of a given phenotype were used in order to take into consideration all sources of heterogeneity. That is, labeling , to be uhttp://price.systemsbiology.net/downloads) as a community resource for those who wish to conduct their own analyses.Raw microarray intensity data (.CEL files) were obtained online from GEO and preprocessed uniformly. More specifically, common probe-sets were found across all transcriptome samples, and consensus preprocessing was performed on all the raw microarray image data to build a consensus dataset. This step removes one major non-biological source of variance between different studies. These preprocessed samples were used to build a multi-study integrated dataset of human brain cancer and normal brain transcriptomes. Finally, stringent probe-set filtering was used to remove spurious classifier features. Our consensus preprocessing and probe-set filtering methods are explained in further detail below. Our integrated and uniformly pre-processed dataset is available on our group's webpage  that bind to a probe set, or the set of complementary 25-mer oligonucleotide probe sequences. The intensities of all probe sets (raw measurements in the form of .CEL files) are then adjusted for background variability and normalized across all samples to obtain the target genes' final expression values.Raw .CEL data files were downloaded directly from GEO. Probe set information used in this study were based on the latest Affymetrix annotations. Raw intensity measurements of all microarray samples considered in our study were preprocessed collectively (consensus preprocessing) using the MATLAB implementation of the microarray preprocessing GCRMA p-value is greater than 0.06, representing no significant difference between the measurements of the “perfect match” and those of the ‘mismatch’ probes Probe sets of Affymetrix microarrays have “perfect match” probes that are exactly complementary to the target gene's mRNA sequence. They also have “mismatch” probes that contain a mismatched nucleotide halfway along the probe sequence, and are used to estimate the degree of non-specific binding. To ensure that a probe set is reliably detected, the measurements of the “perfect match” probes must be significantly greater than those of the “mismatch” probes. This is usually assessed based on statistical measures. The MAS5 preprocessing software makes expression quality calls based on the nonparametric Wilcoxon signed-rank test. The “absent” call is made when the All GCRMA preprocessing and MAS5 probe set filtering procedures were conducted separately for training and test set samples, i.e., inside each cross-validation or hold-out loop, in order to avoid possible cross-talk between the two datasets. Genes that were excluded based on the MAS5 “absent” calls on the training data were also removed from the corresponding test data.YLet LT. Each node ttLT contains all the phenotypes and each leaf  of T delineates a single phenotype. Overall, this representation is nested, in the sense that the set of phenotypes at every non-terminal node is the disjoint union of the phenotypes of the two child nodes. This tree is built from the training data by agglomerative hierarchical clustering derived from features of the profiles, as discussed below.The main assumption is that there are natural groupings tf for every node ttf is a function of the expression profile x. Put simply, tf is a collective “test” for phenotypes in tL versus all other phenotypes. More formally, the classifier returns two possible outcomes: tf (x) = 1  signals that we accept the hypothesis that YtL, and tf (x) = 0  signals that we reject this hypothesis and conclude that YtL. In particular, tf is not a test for tL versus the phenotypes in the sibling of t, as would be the case with a standard decision tree. Rather, tf looks for traits within a given profile x which characterize all phenotypes in Lt simultaneously, such that a positive result signifies that the classifier assumes the true class of x belongs to tL.There is a binary classifier tf for node t are all the expression profiles of the phenotypes in tL. The negative training data are all the profiles of the phenotypes that are not in tL.Classifier learning begins at the two child nodes of the root, and the classifiers are learned from two types of training data. The positive training data for learning the classifier sensitivity of tf is the probability that tf (x) = 1 given x is from the positive training data, and the specificity of tf is the probability that tf (x) = 0 given x is from the negative training data. Due to the coarse-to-fine, hierarchical manner in which the classifiers are processed, we required the sensitivity of tf to be as close to unity as possible. This can be accomplished at the expense of specificity by adjusting a threshold, as discussed below. The reason for imposing a high sensitivity on each classifier is that if a test profile is rejected from belonging to Lt by the classifier when in fact it does belong to tL, it cannot be recovered. However, the reduced specificity is only local to each node, and the overall specificity increases with testing at subsequent nodes.Being binary, each classifier has two performance metrics: the tf performed if and only if all its ancestor tests are performed and are positive. More specifically, tf is performed if and only if sf = 1 for every node sT between t and the root. As soon as tf = 0 for a non-terminal node t, none of the descendant classifiers in the sub-tree rooted at t are performed. This is because a negative response of tf means that the phenotype is unlikely to belong to tL and the set of phenotypes associated with descendant of t, which are necessarily subsets of tL. This facilitates pruning whole subsets of phenotypes at once.The strategy for processing any given profile x with the diagnostic hierarchy is breadth-first, coarse-to-fine. Starting from the two child nodes of the root, classifiers are executed sequentially and adaptively, with set of detected phenotypes. We denote this set by L(x) tf from root to leaf. Equivalently, L(x) is the set of phenotypes that are not ruled out by any test performed. During the diagnostic process, a profile may traverse only one path all the way to the terminal node. In this case, L(x) consists of a single phenotype d, and the diagnostic process terminates with Y = d as the predicted phenotype. However, a profile may also traverse multiple branches to the terminal nodes of T, in which case L(x) consists of multiple candidate phenotypes (see the discussion on resolving ambiguities below). Moreover, a profile may reach no terminal nodes, in which case L(x) is empty. When no terminal node is reached, the profile is determined to be outside of L, and labeled as ‘Unclassified’.The complete coarse-to-fine screening process for x results in a L(x) consists of multiple phenotypes, it becomes necessary to refine the diagnosis. The ambiguity is resolved by another tree-structured process – an ordinary decision tree based on the same diagnostic hierarchy. For every pair of sibling nodes, e.g., t,sg which tests YtL versus YsL, just as in an ordinary decision-tree (the process of classifier identification is elaborated below). Starting from the root of the tree, execution of the decision-tree classifiers directs a profile to one of two sibling nodes sequentially until it reaches a terminal node. Unlike the process of traversing the hierarchy of node classifiers, a sample that enters the decision tree is directed to one and only one leaf node, and hence uniquely labeled.When G be the set of all genes for which we have microarray expression data, and denote the set of all distinct pairs of genes by f is constructed from a small set of gene-pairs PP, and a constant threshold k. More specifically, f (x) = 1 if k, and f (x) = 0 otherwise. The threshold k takes values between 1 and |P|.Every node classifier is based on expression level comparisons between two genes. Let tT, except for the root. The set of gene-pairs tP = P and threshold tk = k depend on the node t. Hence, for each t, the classifier tf = 1 if at least tk of the gene-pair comparisons in tP are positive tf = 0. The comparisons are chosen such that, for each pair tP, we expect to see gene tL, whereas if the phenotype of x does not belong to tL, we expect to see the reverse. For every node t, every pair of all gene-pair combinations is “scored” by the difference between the probability of the event that YtL and the probability given YtL. These probabilities are estimated from the training data, and the subset of pairs with the highest scores are chosen.There is a classifier of this nature for every node YtL , we can then favor sensitivity over specificity by varying the threshold tk. That is, by choosing a relatively small value for tk relative to the number of comparisons in tP, we can make it highly likely that the classifier responds positively when in fact the sample belongs to the set tL. We show the sets of gene-pairs tP for each of the twelve nodes in our diagnostic hierarchy in t,sg are all based on comparisons of single gene-pairs at all edges of the diagnostic hierarchy.Since each positive  comparison is viewed as evidence for http://price.systemsbiology.net/downloads.While multiple gene pairs were used at each decision point in the node-based tree, only a single gene pair was used at each decision point in the decision-tree. This is due to the difference in the motivation of building the two trees; the node-based tree was constructed to maximize sensitivity and minimize false-positives with as many pairs as necessary, while the decision-tree was designed to resolve multiple diagnoses (i.e. ties) which could be done with only one pair. We show the pair of genes for each of the six decision-tree classifiers in Using Gene Ontology (GO) annotations, we have identified a list of 767 genes  in every transcriptome sample that encode for possible blood-borne proteins. Specifically, we selected only the genes whose products are annotated to be in either the ‘Extracellular Space’ or the ‘Extracellular Region’ cellular locations. We use this gene set as a starting point for targeted blood diagnostics. All computational steps and analyses in regards to molecular signature discovery are identical to those discussed above.Figure S1A ISSAC constructs the framework for brain cancer diagnosis – a tree-structured hierarchy of all brain cancer phenotypes built using an agglomerative hierarchical clustering algorithm on gene expression training data. B Training on gene-expression data from all brain phenotypes, ISSAC identifies disjoint, gene-pair classifiers at all nodes (excluding the root) and edges of the diagnostic hierarchy, and accumulates them into their respective marker panels. The chosen pairs are the ones that best differentiate between the phenotype sets, and are based entirely on the reversal of relative expression. C ISSAC uses the gene-pair classifiers for class prediction. Briefly, given a gene expression profile, ISSAC executes the node classifiers in a hierarchical, top-down fashion within the disease diagnostic hierarchy to identify the phenotype(s) whose class-specific signature(s) is present. In case of multiple class candidates (i.e. node classifiers for multiple leaves are positive), the ambiguity is resolved by aggregating all the decision-tree classifiers into a classification decision-tree, thereby leading any expression signature down one unique path toward a single phenotype.The overall method of ISSAC can be summarized into three main steps. (TIFF)Click here for additional data file.Figure S2Brain phenotypes are grouped into a global diagnostic hierarchy, which allows an intuitive representation of the classification process. The diagnostic hierarchy is built using a data-driven, iterative approach, and is free of manual, ad-hoc construction. In each iteration, two classes, or two groups of classes, with the lowest TSP score Click here for additional data file.Figure S3A Ten-fold cross-validation is conducted ten times to obtain the average accuracy. In every iteration of cross-validation, the order of samples within a particular class are randomly permuted before training/test set allocations. B Our marker panel achieved a 90.4% average of phenotype-specific classification accuracies, showing strong promise against a multi-category, multi-dataset background at the gene expression level.Performance evaluation using ten-fold cross-validation. (TIFF)Click here for additional data file.Figure S4A ‘Immunity and Defense’ was the most enriched biological process for ‘gene-set i’, reflecting the chronic inflammatory conditions inside the GBM tumor. B ‘Neuronal Activities’ was the most enriched biological process for ‘gene-set j’, reflecting decrease in neuronal behavior and possibly other brain cell activity inside the GBM tumor. The genes in ‘gene-set i’ and ‘gene-set j’ were the most enriched in C Chromosome 1 and D Chromosome 10, respectively, reflecting the major chromosome aberrations of the two brain cancers. Ψ delineates the most enriched category. aBiological process abbreviation (name): AAM (Amino acid metabolism), APT (Apoptosis), CM (Carbohydrate metabolism), CA (Cell adhesion), CC (Cell cycle), CPD (Cell proliferation and differentiation), CSM (Cell structure and motility), DP , HMS (Homeostatis), IMD (Immunity and defense), IPRT (Intracellular protein transport), MCN (Muscle contraction), NA , NAM (Nucleic acid metabolism), PMM (Protein metabolism and modification), SM (Sulfur metabolism), ST , and TR (Transport).Statistical enrichment analysis on PANTHER database biological processes and chromosome numbers of the top 500, 1,000, and 1,500 gene-pair classifiers for GBM vs. OLG. (TIFF)Click here for additional data file.Figure S5Gene-pair classifiers based on only the genes that encode extracellular products. Gene pairs are shown at their corresponding nodes in the brain disease diagnostic hierarchy. The corresponding node-based marker panel consists of 41 classifier pairs and 71 unique classifier features.(TIFF)Click here for additional data file.Table S1Phenotype specimen descriptions and main results for all GEO accessions used in this study.(PDF)Click here for additional data file.Table S2GEO microarray sample IDs used in this study.(PDF)Click here for additional data file.Table S3i/Gene j: the gene expressed higher and lower in the gene-pair, respectively, within each corresponding phenotype. Gene name/Chromosome locus: according to Entrez Gene. Affymetrix Probe ID: For both Affymetrix Human Genome U133A and U133Plus2.0 Arrays. k: The minimum number of gene-pair classifiers whose decision rule outcomes for a test sample are required to be ‘true ( = 1)’ for the sample to be classified as the phenotype(s) of the corresponding node.Node marker panel for brain cancer and normal transcriptome classification. Node #: Corresponds to numerical labels shown in the brain phenotype diagnostic hierarchy . Brain p(PDF)Click here for additional data file.Table S4i>Gene j ?), 1 and 0 delineates ‘true’ and ‘false’, respectively, and ‘–’ denotes that the outcome is not used for classification. The vertical binary pattern under each class label corresponds to a phenotype-specific molecular signature.Decision-tree marker panel for brain cancer and normal transcriptome classification. For each classifier decision rule (i.e. Is Gene (PDF)Click here for additional data file.Table S5i/Gene j: the gene expressed higher and lower in the gene-pair, respectively, within each corresponding phenotype. Gene name/Chromosome locus: according to Entrez Gene. Affymetrix Probe ID: For both Affymetrix Human Genome U133A and U133Plus2.0 Arrays. k: The minimum number of gene-pair classifiers whose decision rule outcomes for a test sample are required to be ‘true ( = 1)’ for the sample to be classified as the phenotype(s) of the corresponding node. Ranked expression differences of each gene pair (i.e. Rank_gene_i – Rank_gene_j) were calculated for each sample, and Mean, St. dev., Max., Min., and Median were found across all samples within classes of respective Node #.Summary of expression differences between genes-pair classifiers. Node #: Corresponds to numerical labels shown in the brain phenotype diagnostic hierarchy . Brain p(PDF)Click here for additional data file.Table S6Hold-one-lab-in validation accuracies of glioblastoma signatures.(PDF)Click here for additional data file.Table S7Hold-one-lab-in (H1LI) and leave-one-lab-out (L1LO) validation accuracies of glioblastoma signatures when training data were constrained to 50 total samples. H1LI and L1LO validations were performed ten times for each category of training data. In each validation trial, 50 samples were randomly selected from the single microarray dataset (for H1L1) or from the multi-study, combined dataset (for L1LO).(PDF)Click here for additional data file.Table S8Sample size: Average proportion of total samples that reached unique diagnoses via node marker panel. Accuracy: Reflects average performance in ten-fold cross-validation conducted ten times.Ten-fold cross-validation accuracies when only the node marker panel was required to reach unique diagnoses. (PDF)Click here for additional data file.Table S9ahttp://www.ncbi.nlm.nih.gov/gene.Functional roles of 11 previously identified GBM serum markers that are present in our extracellular-product encoding marker-panel. (PDF)Click here for additional data file.Table S10aAccuracies reflect average performance in ten-fold cross-validation conducted ten times. The main diagonal gives the average classification accuracy of each class (bold), and the off-diagonal elements show the erroneous predictions. bUC (Unclassified samples). When using the node classifiers, expression profiles that did not exert a signature of any phenotype  were rejected from classification. In this case, the Unclassified sample is treated as a misclassification.Ten-fold cross-validation accuracies of gene-pair classifiers composed of genes that encode extracellular products. (PDF)Click here for additional data file.Text S1Step-by-step description of how ISSAC works.(PDF)Click here for additional data file.Text S2Advantages of using relative expression reversals to build classifiers.(PDF)Click here for additional data file.Text S3Twenty-five anatomical regions of the human brain from which normal transcriptome samples were obtained.(PDF)Click here for additional data file.Text S4Global statistical enrichment analysis of gene-pair classifiers.(PDF)Click here for additional data file.Text S5Whether ISSAC can play a role in identifying misdiagnoses.(PDF)Click here for additional data file.Text S6Reasoning for selecting only Affymetrix microarray platforms and for not using probe-specific offsets.(PDF)Click here for additional data file.Text S7Candidates of brain cancer molecular signatures.(PDF)Click here for additional data file.


Gene expression patterns characterizing clinically-relevant molecular subgroups of glioblastoma are difficult to reproduce. We suspect a combination of biological and analytic factors confounds interpretation of glioblastoma expression data. We seek to clarify the nature and relative contributions of these factors, to focus additional investigations, and to improve the accuracy and consistency of translational glioblastoma analyses.We analyzed gene expression and clinical data for 340 glioblastomas in The Cancer Genome Atlas (TCGA). We developed a logic model to analyze potential sources of biological, technical, and analytic variability and used standard linear classifiers and linear dimensional reduction algorithms to investigate the nature and relative contributions of each factor.Commonly-described sources of classification error, including individual sample characteristics, batch effects, and analytic and technical noise make measurable but proportionally minor contributions to inconsistent molecular classification. Our analysis suggests that three, previously underappreciated factors may account for a larger fraction of classification errors: inherent non-linear/non-orthogonal relationships among the genes used in conjunction with classification algorithms that assume linearity; skewed data distributions assumed to be Gaussian; and biologic variability (noise) among tumors, of which we propose three types.Our analysis of the TCGA data demonstrates a contributory role for technical factors in molecular classification inconsistencies in glioblastoma but also suggests that biological variability, abnormal data distribution, and non-linear relationships among genes may be responsible for a proportionally larger component of classification error. These findings may have important implications for both glioblastoma research and for translational application of other large-volume biological databases. Glioblastoma (GBM) is the most common primary malignant brain tumor in adults, and optimal surgical and medical management of this disease result in a mean survival of only 12–14 months The recent development of high-throughput molecular techniques for comprehensive characterization of tumor genomes and transcriptomes has been embraced by the translational neuro-oncology community and has been applied to the challenge of molecular GBM subclassification. Numerous investigators have reported successful identification of gene expression patterns characteristic of distinct tumor genomic profiles associated with unique clinical phenotypes Initial optimism that molecular classification tools represent a major breakthrough in GBM management has, more recently, been tempered by the lack of consistency and reproducibility of genomic signatures with putative associations to survival phenotypes. While multiple groups have reported the ability to predict patient survival accurately based upon specific gene expression signatures One step to address the challenges associated with analyzing and interpreting this data has been the development of a central repository of genomic and epigenetic data for GBMs. The Cancer Genome Atlas (TCGA) project Comprehensive investigation of the potential sources of molecular classification errors in GBM requires questioning the fundamental assumptions regarding both the nature of the expression data itself and the analytic strategies used for its analysis. We hypothesize that a combination of biological and mathematical factors confound interpretation of this data, and we have undertaken a comprehensive investigation into the nature and relative contributions of these factors to inconsistencies in molecular classification of GBMs. To investigate our hypothesis, we constructed a logic model  to organLogic models are narrative or graphical depictions of processes that communicate the underlying assumptions upon which an activity is expected to lead to a specific result. Such a model describes the logical linkages between elements of a process as a linear sequence of inputs, activities , outputs, and outcomes Eleven analyses were conducted to test the logic model outlined in Unsupervised classification using hierarchical clustering (HCL) Unsupervised classification using hierarchical clustering with bootstrapping support (HCS), KMS, and PCA was performed on the dataset containing 340 GBMs normalized to controls to investigate whether unbiased clustering algorithms would separate the population of GBMs into genomic subsets with corresponding, clinically-relevant phenotypes. Eight versions of each analysis were performed, each with samples colorized according to one of the eight clinical phenotypes with known or potential clinical significance recorded in the TCGA dataset. The results of the complete PCA analysis are presented in To study the extent to which the principal components (PC) describe the variability in the GBM population, we examined the eigenvalues resulting from the PCA matrix decomposition of the dataset containing 340 GBMs normalized to controls. Eigenvalues for each PC were computed both as a raw value and as a percent of the cumulative total eignevalue. The eigenvalue plot, as well as the individual eigenvalues and the eigenvalue percents associated with each of the first 20 PCs, are presented in HCS and PCA of the 340 GBMs  demonstrated a cluster of 14 tumor “outliers” that consistently segregated from the remainder of the group in multiple analyses . We hypop = 0.01) or 857 (p = 0.05) that would be differentially regulated by pure chance]. EASE The significance analysis for microarrays (SAM) 2  data. We studied the distribution characteristics of the variables  in the dataset for the complete expression matrix  as well on a per-gene and per-sample basis. We characterized the distribution by computing standard descriptive statistics as well as skew and kurtosis measurements (and descriptive statistics for each of these measurements). The results of this analysis are summarized in One assumption common to most linear classifiers used in microarray analysis is that the values of the variables upon which classification is based conform to a Gaussian distribution 2(ratio) data reveals that the complete expression matrix is centered around a mean and median of 0.03 with a standard deviation of 0.92, suggesting a normal (standard Gaussian) distribution. However, the skew of this distribution is −0.21, an effect that is easily overlooked on visual inspection , SES = 0.001 and the test statistic = −207.2, indicating that the distribution has a statistically-significant degree of skew. Similarly, the kurtosis of the distribution is calculated at 6.45 (excess kurtosis = 3.45). The statistical significance of the magnitude of the kurtosis must also be considered in the context of the number of variables (n) associated with the distribution, and a test statistic reflecting the number of standard errors of kurtosis (SEK) This equation essentially performs a two-tailed test of skew≠0, where a test statistic value of ±2 corresponds to n = 5,838,480, SEK = 0.002 and the test statistic = 1701.6, indicating that the distribution has a highly statistically-significant degree of leptokurtosis , the mean skew is 0.64 (mean test statistic = 4.84) and the mean kurtosis is 3.07 . A similar analysis done on a per-sample basis  demonstrates a mean skew of −0.18 (mean test statistic = −9.64) and a mean kurtosis of 5.82 , suggesting significant deviations from a pure Gaussian distribution in both analyses. Examples of the diversity of skew and kurtosis of gene distributions are given in Unsupervised classification using PCA was performed on the dataset containing 340 GBMs  to investigate whether unbiased clustering algorithms would separate the population of GBMs into subsets corresponding to technical differences related to the samples or to their processing. Eight versions of the analysis were performed, with samples colorized according to one of eight technical parameters that could introduce classification error. The results of this analysis  demonstrMultiple technical variables associated with sample preparation, hybridization efficiency, physical array quality, and image acquisition can contribute to technical noise and subsequently to classification error. These effects have been studied extensively i) based upon the relative signal levels measured for the perfect match (PM) and mismatch (MM) cells for each probeset. A one-sided Wilcoxon's Signed Rank Test is used to calculate a p-value associated with Ri. Comparison of the p-value with user-defined threshold values (α1 and α2) serves as the basis for the algorithm's present/absence call, and the p-value itself can be conceptualized as a measure of confidence that the measured signal is the result of valid hybridization rather than background noise. Qualitative assessment of the extent of the noise in the TCGA dataset was performed by calculating the percentage of genes in the expression matrix for which p>0.05 (41.76%) and p>0.01 (51.50%). This analysis suggests that approximately 40–50% of the individual expression measurements in the matrix are not significantly different from background. We performed similar analysis on smaller sets of previously-published expression data for GBMs (n = 20) 1 = 0.05, α2 = 0.065, τ = 0.015) for the TCGA dataset and for the two comparison datasets and again found comparable values in each category  for each gene across the set of 340 arrays and subsequently filtered out genes for which pm>0.05 from the RMA-normalized dataset used in our other analyses. We then performed PCA on this filtered dataset and found no qualitative improvement in clustering into survival phenotypes (data not shown). The sum of the first three eigenvalue percentages from this analysis was slightly better than that of the unfiltered data (34.4% vs 29.6%). Together, the results of these analyses suggest that technical noise is present within the TCGA dataset but that it does not appear to be the major factor confounding clustering into clinically-relevant phenotypes.Next, we used the Chip quality was also performed to identify areas of chips with inhomogeneous signal characteristics. This can be attributable to physical errors on the chips, fluidics errors, regional binding inconsistencies, or other regional effects. We first examined the virtual chip images created using the RMA Express Two potential sources of statistical noise that may impede classification are genes with either relatively static expression levels throughout the data set or genes for which expression values are absent in a significant portion of the expression matrix. We performed a series of data filtering analyses designed to investigate the effects of these variables, in isolation and in combination, on unsupervised classification into survival groups. We first performed PCA using the baseline data set of 340 GBMs (relative to controls) and then repeated this analysis using progressively stringent variance and present/absent data filters on the normalized dataset. These filters removed the least variable 50% or 80% of genes (variance filter) or limited inclusion of genes to those with measurable signal present in >50%, >80%, or in 100% of samples (percent present filter). Data filtering trials were performed with individual filters or with combinations of the variance and percent present filter in place at various individual thresholds. PCA was performed on each of the filtered datasets, and the ensuing classifications were evaluated qualitatively by examining the plots of the first three principal components  and semi-quantitatively by analyzing the eigenvalue percentages associated with each of the first three principal components in each of the trials . The resThe TCGA GBM data set includes 340 samples derived collected from patients treated at multiple clinical centers and processed at one of several sample preparation laboratories. Combining the data from such samples may introduce batch effects into the concatenated data set. This phenomenon has been well described by array statisticians post hoc estimate of the classification impact of batch effects. We implemented this analysis by first performing baseline PCA analysis  with eigenvalue decomposition on the complete set of 340 GBMs. Next, we applied the ComBAT algorithm Recognizing the limitations of the “bottom-up” analysis, we also performed qualitative and semi-quantitative analysis of batch effects using a “top-down” approach. We assumed the presence of some batch effects associated with clinical center of origin and TCGA batch, attempted to correct empirically for such errors, and compared the PCA plots and the eigenvalue data before and after correction to obtain a Although increasing the number of samples in a microarray experiment may increase classification power in some instances, this presupposes that the signal-to-noise ratio of the samples is higher than those of the base set. If the opposite is true or if the SNRs are approximately equal, then increasing population size may either disproportionately introduce noise that could impede classification or may have little to no added effect, respectively. To study the possibility of this effect in the TCGA GBM data, we used a random number generator to randomly divide the RMA-normalized, 340 tumor dataset into 6 subgroups of approximately equal size . We then progressively combined the subsets, creating five sub-populations of 114, 171, 228, 284, and 340 samples. Qualitative PCA and eigenvalue analysis were performed on each of the six subsets and on the five sub-populations. The results of these analyses demonstrate approximately stable magnitude of the eigenvalue percentages associated with each of the first three principal components in all subsets and concatenated sub-populations, and qualitative analysis of the PCA data does not demonstrate appreciable improvement (or decline) in phenotypic classification accuracy as the population size increases . These fA necessary condition for successful class discrimination by linear classifiers or data reduction algorithms is that the relationships between the variables used for classification must be linear When the underlying data relationships are both linear and orthogonal, two-dimensional plots of successive principal components (PC) should demonstrate that the samples display, on average, a linear relationship to one another with the regression line having a y-intercept and slope that approximate zero. Using the RMA-normalized data set of all 340 GBMs, we performed PCA and then examined the three, two-dimensional plots of the first three principal components  n) and the number of variables measured per sample (p). In traditional datasets, relatively few variables are tracked across a proportionately greater number of samples (n>p). The inverse is true of modern molecular datasets, where the number of variables measured per sample  far exceeds the number of samples in a given investigation (p≫n). This difference calls into question the validity of traditional models of error and noise reduction through increasing sample size, and two types of gross evidence appear to validate these concerns. First, statistical investigations regarding selection of sample size in microarray experiments, the details of which are beyond the scope of this discussion, suggest that large sample sizes may not always be necessary to draw meaningful biological conclusions Large-scale, molecular data sets differ fundamentally from traditional biological and clinical data sets, however, in the relationship between the number of samples being studied  and that error modeling may need to be individualized Another possibility is that cancers, such as GBM, are much more molecularly diverse than we have previously recognized. While there may be some common molecular events contributing to the development and progression of the disease, it may be that the clinical phenotypes are mediated by large numbers of interacting genes and that this diversity would require even larger patient populations to identify the subtle signals that may be present. Alternately, it may be that a more hypothesis-driven, pathway-based approach to identifying phenotypically-relevant subclasses is necessary.Identification of genotypic tumor subgroups with phenotypic significance has been used as the benchmark for “successful” molecular tumor subclassification in this investigation. This strategy has been chosen because it reflects the real-world needs of translational neurobiologists and neuro-oncologists. These investigators, who attempt to identify molecular markers in glioblastoma patients that define discrete diagnostic, prognostic, or response-to-therapy subpopulations, represent a significant proportion of the target audience of the TCGA project and are currently among the most frequent consumers of the gene expression data housed in this database. While phenotypic correlations of genomic data are important in nearly all avenues of research conducted using such databases, we recognize that the benchmark we have chosen may not be ideally suited to the specialized needs of some mathematicians, basic scientists, and systems biologists who may be analyzing the TCGA data with non-clinical objectives. Notwithstanding, even investigators with such alternate aims may benefit from the conclusions drawn from this analysis.Although specific sources of noise and their relative relationships may vary across diverse datasets and for specific clinical questions, there must be some logical framework through which to approach the identification of such confounding data. To address this challenge, we created a logic model we believe is useful for analyzing many of the potential sources of noise that result in inconsistent phenotypic classification of malignancies based upon gene expression data .Our model is predicated upon two, necessary preconditions; that distinct phenotypic groups exist within the dataset and that genomic profiles can be used to predict phenotype. Assuming these conditions are met, we asked five basic questions, the answers to which frame a logic model for investigating the potential sources of noise that impedes reproducible classification into clinically-relevant phenotypes based upon molecular data. Finally, we apply this logic model to the TCGA glioblastoma expression dataset to investigate the potential sources of noise specific to this dataset and to the challenge of molecular classification of GBM into subgroups with distinct, clinically-relevant phenotypes. This exercise serves two purposes. First, it provides a specific analysis of the potential sources of noise and error that have been confounding attempts at reproducible subclassification of GBM. Second, it illustrates the broader concept of design and implementation of a logic model for error analysis that can be applied, with some individual modifications, to understand sources of noise and error in similar, translational investigations.In the following section we describe the five questions used to frame our logic model and the results of the 11 individual analyses performed within the context of the model to investigate the potential sources of noise and error that result in classification inaccuracy or variability in the TCGA GBM data set.Preconditions. In order for the TCGA dataset to be appropriate for the task of molecular subclassification of GBMs into genomic subgroups with phenotypic significance, two preconditions must be met. First, there must be multiple, discrete phenotypes of GBM. This precondition is satisfied by a wealth of clinical experience demonstrating that patients with GBM have variable time to progression and recurrence, survival rates, and response to therapy multiple tumor stratification analysis (#1) demonstrated that the TCGA GBM expression data could be used to successfully separate GBM from other malignancies, validating this specific precondition for this particular dataset.Question 1: Do unbiased class discovery methods accurately identify clinical phenotypes of GBM based upon analysis of the TCGA GBM expression data? The unsupervised classification and clinical parameters analysis (#2) addressed this question by applying several unsupervised classification algorithms to the TCGA GBM expression data. No algorithm was able to accurately stratify the tumors by subclass in any of 8 clinically-relevant, phenotypic categories. These results demonstrate that gene expression-based analysis fails to classify patients into appropriate clinical subclasses, suggesting that some type(s) of noise is present within the data and confound attempts at molecular classification. This observation differs from some of those previously reported independently by several groups, who found that phenotypically-relevant  GBM subgroups could be identified based upon gene expression data Question 2: Do clinical and phenotypic differences exist within the data set? One potential reason for failure of the molecular classification strategies that must be addressed at the outset would be the absence of genomic or phenotypic differences within the TCGA GBM dataset. Phenotypic variability among GBMs has been well described in general (see preconditions) #2 (data not shown), indicating that various clinical phenotypes are also present within the dataset. Genomic variability is also well described among GBMs biologic analysis of the outlier group (#3) exercise demonstrates plausible biological variability among samples in this specific data set. Accordingly, we conclude that genomic and phenotypic differences do exist in the TCGA GBM dataset.Question 3: Why are underlying differences among the expression data not identified by the unsupervised analysis? Several unsupervised, linear classifiers and dimensional reduction strategies failed to classify the TCGA GBM data into phenotypically-significant groups. The eigenvalue data analysis (#4) suggests that these classifiers (for which we use PCA as a prototype) fail to capture a large percentage (>70%) of the variability inherent in the dataset. In general, the types of linear algorithms used commonly in molecular GBM subclassification (and therefore used in this exercise) are unsuccessful when one of two, necessary criteria are not satisfied; either (1) the mean and standard deviation do not accurately and sufficiently represent the dataset, or (2) the relationships between genomic profile and clinically-relevant phenotype are either non-linear or non-orthogonal. Individually, these criteria form the basis of questions four and five.Question 4: Do the mean and standard deviation accurately and sufficiently represent the dataset? In this analysis, the mean and standard deviation may fail to appropriately represent the dataset in one of two circumstances. First, they insufficiently describe the dataset when the values of the variables serving as the basis of classification do not follow a Gaussian distribution expression data distribution analysis (#5) demonstrates that the underlying data distribution in the TCGA GBM data is not purely Gaussian, with statistically-significant skew and kurtosis apparent in both the composite expression matrix and among the expression levels of the individual variables (genes). The ultimate significance of this with regard to classification requires further, dedicated investigation noise is defined as the presence of measured variable values  that do not contribute to the intended molecular classification, and, as discussed earlier, is relative to the clinical question(s) being asked. In molecular classification experiments using gene expression data, both technical and biological noise can impede accurate classification. Technical noise is present when procedural artifacts are reflected in the values of the variables. We investigated potential sources of technical noise, including those associated with sample preparation (#7), the array platform (#7), the inclusion of low-variance or incomplete expression data (#8), batch effects (#9), and sample size (#10). We demonstrated the presence of technical noise associated with all of these domains within the TCGA GBM dataset, although no single domain appeared to be the primary contributor to inaccurate phenotypic subclassification. Investigation of the relative magnitude of the individual contributions to the overall error and the aggregate effects of these sources of technical noise on the overall classification accuracy require detailed modeling strategies that are only beginning to be developed and that are beyond the scope of this analysis. Notwithstanding, our analysis demonstrates the importance of ongoing research into such modeling and highlights technical noise as one, important area for additional investigation.Biological noisea priori knowledge of the underlying systems biology of the tumor that may not be known. These, too, must be areas of active research if the sources of error in large-scale, molecular datasets are to be accurately described.A second circumstance where mean and standard deviation fail to appropriately represent the dataset is when features with the largest variance are not the primary contributors to the clinically-relevant phenotypes. Stated differently, linear classifiers fail when the signal-to-noise ratio of the dataset is low. In this context, the term Question 5: Are the relationships between features (genes) and clinically-relevant phenotype linear and orthogonal? Linear classifiers and dimensional reduction algorithms, by definition, are designed to classify based upon linear combinations of feature values associated with the members of the dataset (#11). These findings argue in favor of application of classifiers that do not assume linearity or orthogonality , strategies which are now being described by array statisticians The investigation that we have performed focuses on the TCGA gene expression data and is therefore subject to two, general limitations inherent to this dataset. First, the TCGA expression data that we have analyzed reflects only genome-level changes in tumor biology. While gene expression is believed to correlate with phenotype, protein-level data may also be a valuable resource for molecular classification of glioblastoma We also recognize that the logic model that we have described and the analysis of the TCGA GBM data that we have performed have some limitations. First, we recognize that our model may not encompass every potential source for noise or error in this particular dataset and that generalization of our model to other datasets and/or to investigations of alternate hypotheses may require inclusion or exclusion of additional confounding factors. Nonetheless, we believe that our model provides a logical framework for analyzing error that is readily expandable should modification of the specific sources of noise or error become necessary or should alternate, potential sources of error be identified. Similarly, while our logic model may be useful for prospectively exploring and limiting potential sources of error when designing novel studies, we would not support the conclusion that such a strategy necessarily assures generation of “high-quality” data and caution that such a conclusion exemplifies the logical burden-of-proof fallacy. Finally, while our model has allowed us to identify several targeted areas for future efforts to improve the quality of the TCGA GBM data, we recognize that research in additional domains may also be necessary to improve the translational potential of this dataset.We have constructed a logic model that facilitates a comprehensive analysis of potential sources of noise and error that may be responsible for some portion of the inconsistencies and lack of reproducibility observed in recent attempts to use constituent data from the TCGA GBM expression dataset to classify GBMs into clinically-relevant phenotypes. We believe that this model is a useful tool to conduct such analyses and can be readily adapted as new information regarding error and noise becomes available, and we believe it may be readily generalized to the systematic study of error in other translational domains.Based upon our application of this logic model to the TCGA GBM expression data, we make seven conclusions regarding the potential sources of inconsistencies and error among molecular classifiers based upon the expression data comprising this dataset.Because there is evidence that supports the existence of genomic and phenotypic variability within the dataset, the failure to discover genomic associations correlated with clinically-relevant phenotypes may reflect a classification failure by the common, linear classification algorithms commonly employed in this and in most other analyses of gene expression data.The mean and standard deviation do not accurately and sufficiently describe either the overall expression matrix or the expression values of the individual genes contained therein.Neither the distribution of the overall expression matrix nor the expression distributions of the genes or samples conform to a pure Gaussian distribution.Technical noise, and likely biological noise as well, limit the signal-to-noise ratio of this dataset with respect to molecular tumor classification into phenotypically-relevant subsets.Technical noise is present within the data, but none of the measured sources appear to individually represent the primary source of classification-limiting noise.Biological noise is assumed to be present based upon the logic model but cannot be accurately measured using current error modeling strategies.There is a suggestion of non-linear and/or non-orthogonal relationships among genes affecting clinically-relevant GBM phenotypes in this dataset.Finally, the results of our analysis suggest that targeted research in four, specific areas may further efforts to improve classification accuracy and reproducibility by reducing noise in the TCGA dataset. These areas include: (1) additional research regarding the underlying distribution of the dataset and the implications of this distribution for classification algorithms, (2) expanded development and application of non-linear and/or non-orthogonal classifiers to this data, (3) efforts to improve modeling of technical noise, (4) and expanded efforts to model biological noise in the context of improved understanding of the systems biology of GBMs.A logic model for directing data analysis was constructed and is depicted in The primary GBM gene expression dataset for this project was compiled from publically-available, TCGA GBM data 2 converted.Three datasets were prepared and were used in all subsequent analyses. The first was composed of all 340 GBM samples, the second comprised all 340 GBM samples plus the four normal brain controls, and the third consisted of all 340 GBM samples plus the 24 non-GBM tumors. Each dataset was normalized using the Robust Multichip Average (RMA) 2  expression value for each of the remaining genes was computed in each of the 340 samples, and this dataset was used in subsequent analyses involving biological interpretation of data. This method of data preparation has been previously described in GBM expression analyses.For the dataset containing normal brain tissue, and average expression value for each gene across the four control samples was computed. The expression variance among each gene was computed for the controls, and the most variable 10%, representing the genes with the least consistent expression in the control dataset, were removed from downstream analyses (as previously described). The logA one-sided Significance Analysis for Microarray (SAM) Access to restricted clinical data was granted through the standard TCGA data access protocol. Clinical data (when available), including survival, time to progression, time to recurrence, Karnofsky performance status (KPS), sex, adjuvant chemotherapy, adjuvant radiotherapy, and presence of necrosis and endothelial proliferation in histologic sections, was merged with expression data for analyses involving these parameters. Similarly, data regarding the technical processing and specimen quality, including prior treatment, relative abundance of tumor cells, tissue block size, hospital of origin, TCGA batch number, extracted RNA concentration, OD260/280, 28S/18S, and RNA integrity number (RIN), was similarly collected and merged for analyses involving these parameters.p<0.05, unless otherwise indicated.Microsoft Excel was used for data preparation. The TM4 software package Initial analyses used multiple classification algorithms, including HCS, KMS, and PCA to verify that all of these linear methods resulted in qualitatively-equivalent results in class discrimination investigations. Subsequently, PCA was used as a prototypical discriminator because of the robust nature of the algorithm and the logical, graphical representation of its class discrimination.Figure S1Logic Model for Analyzing Variability in the TCGA GBM Dataset, Unannotated. Logic model applied for data analysis, without annotations (see (TIF)Click here for additional data file.Table S1Genes Differentially Expressed between the Outlier Cluster and the Remaining GBMs. Relative fold change values computed using 2-sided significance analysis for microarrays (SAM) with false discovery rate (FDR) = 0. Statistical significance of categorical annotations computed using the EASE score (adjusted Fisher Exact Test). 1A: Overexpressed in the outlier group relative to the remainder. 1B: Underexpressed in the outlier group relative to the remainder. 1C: EASE analysis of overrepresented categories in the overexpressed genes. 1D: EASE analysis of overrepresented categories in the overexpressed genes. 1E: Overlap between differentially expressed genes and categories in the outlier group and those identified in a previously-described GBM expression fingerprint with putative relationship to the survival phenotype (XLS)Click here for additional data file.