A key goal of systems biology and translational genomics is to utilize high-throughput measurements of cellular states to develop expression-based classifiers for discriminating among different phenotypes. Recent developments of Next Generation Sequencing (NGS) technologies can facilitate classifier design by providing expression measurements for tens of thousands of genes simultaneously via the abundance of their mRNA transcripts. Because NGS technologies result in a nonlinear transformation of the actual expression distributions, their application can result in data that are less discriminative than would be the actual expression levels themselves, were they directly observable.Using state-of-the-art distributional modeling for the NGS processing pipeline, this paper studies how that pipeline, via the resulting nonlinear transformation, affects classification and feature selection. The effects of different factors are considered and NGS-based classification is compared to SAGE-based classification and classification directly on the raw expression data, which is represented by a very high-dimensional model previously developed for gene expression. As expected, the nonlinear transformation resulting from NGS processing diminishes classification accuracy; however, owing to a larger number of reads, NGS-based classification outperforms SAGE-based classification.Having high numbers of reads can mitigate the degradation in classification performance resulting from the effects of NGS technologies. Hence, when performing a RNA-Seq analysis, using the highest possible coverage of the genome is recommended for the purposes of classification. In recent years, modern high throughput sequencing technologies have become one of the essential tools in measuring the number of transcripts of each gene in a cell population or even in individual cells. Such information could be used to detect differential gene expression due to different treatment or phenotype. In our case we are interested in using gene-expression measurements to classify phenotypes into one of two classes. The accuracy of classification will depend on the manner in which the phenomena are transformed into data by the measurement technology. We consider the effects of Next-Generation Sequencing (NGS) and Serial Analysis of Gene Expression (SAGE) on gene-expression classification using currently accepted measurement modeling. The accuracy of classification problem has previously been addressed for the LC-MS proteomics pipeline, where state-of-the-art modeling is more refined, the purpose being to characterize the effect of various noise sources on classification accuracy .NGS technology provides a discrete counting measurement for gene-expression levels. In particular, RNA-Seq sequences small RNA fragments (mRNA) to measure gene expression. When a gene is expressed, it produces mRNAs. The RNA-Seq experiment randomly shears and converts the RNA fragments to cDNAs, sequences them, and finally outputs the results in the form of short reads ,3. AfterSAGE technology produces short continuous sequences of nucleotides, called tags. After a SAGE experiment is done, one can measure the expression level of a particular region/gene of interest on the genome by counting the number of tags that map to it. SAGE is very similar to RNA-Seq in nature and in terms of statistical modeling. The SAGE data processing pipeline is traditionally modeled as a Poisson random vector ,8. We foOur overall methodology is to generate three different types of synthetic data: (1) actual gene expression concentration, called MVN-GC, from a multivariate normal (Gaussian) model formulated to model various aspects of gene expression concentration ; (2) PoiNext-Generation Sequencing refers to a class of technologies that sequence millions of short DNA fragments in parallel, with a relatively low cost. The length and number of the reads differ based on the technology. Currently, there are three major commercially available platforms/technologies for NGS: (1) Illumina, (2) Roche, and (3) Life Technologies . The Dlv low-variance non-markers are uncorrelated and have identical one-dimensional Gaussian distributions with parameters where  ,27. Figuj for sample point i by Xi,j, where it is assumed that in an NGS experiment each lane has a single biological specimen belonging to either class 0 or class 1. Furthermore, we select a model where the number of reads for each gene is generated from a Poisson distribution with a known parameter. We calculate the expected number of reads (mean of the Poisson distribution) from the generalized linear model  serves as the single parameter of a Poisson distribution that models the NGS/SAGE processes by generating random non-negative integers, as the read counts or tag numbers data, having expected value equal to the right-hand side of equation (2).COV aims to model the unknown errors that can occur during an/a NGS/SAGE experiment, including basecalling, mapping reads to the reference genome, etc. The term i in the experiment, we first randomly generate a number si from a uniform distribution, U , where α > 0 and β > α. Then, a random sample point, λi,j, is drawn from the MVN-GC model and its value is perturbed with θi,j, which is drawn randomly according to its distribution defined in (3). Using (2), the mean of the Poisson distribution is calculated for each sample point i and gene j, and a single random realization of Xi,j is generated. The processes of generating count data for RNA-Seq reads and SAGE tag numbers are very similar, but the total number of reads per NGS run is significantly more than tags for a SAGE experiment. Therefore, we only change α and β to get the desired number of read counts or tags. We also assume that the SAGE experiments always have a fixed range for the total number of tags, whereas RNA-Seq has a variety of ranges for the total read counts. By having different ranges for the read counts, we can compare the performance of classification resulting from NGS-reads and SAGE-tags models under different experimental settings.To generate synthetic datasets for the purposes of our simulation study we proceed as follows: for a sample point FXY, where Y ∈ {0,1} is the unknown class label of X. In the context of genomics, the feature vector is usually the expression levels of many genes and the class labels are different types or stages of disease to which sample points belong to. A classifier rule model is defined as a pair , where Ψ is a classification rule, possibly including feature selection, and Ξ is a training-data error estimation rule. In a typical classification task, a random training set FXY and the goal is to design a classifier X as the input and outputs a label Y. The true classification error of ψn is given by εn = P (ψn (X) ≠ Y). The error estimation rule Ξ provides an error estimate, ψn.The setup for the classification problem is determined by a joint feature-label probability distribution t-test feature selection (as a part of the classification rule) before the classifier design procedure to select d ≤ D features with highest t-scores. The training set with d features is then used to design the classifier. The same d features are also used for finding the true error of the designed classifier.In this study, we consider linear discriminant analysis (LDA), three nearest neighbors (3NN) and radial basis function support vector machine (RBF-SVM) as the classification rules, and report the true error of the classifiers. We implement x to class 1 if and only ifLDA is the plug-in rule for the Bayes classifier when the class-conditional densities are multivariate Gaussian with a common covariance matrix . The samy ∈ {0,1}, and where kNN rule (with k = 3), which is a nonparametric classification rule based on the training data. The kNN classifier assigns a label, 0 or 1, to a point x according to the majority of the labels of the k nearest training data points to it. To avoid tied votes in binary classification problems, an odd number is usually chosen for k.3NN is a special case of A support vector machine finds a maximal margin hyperplane for a given set of training sample points. If it is not possible to linearly separate the data, one can introduce some slack variables in the optimization procedure that allow the mislabeled sample points and solve the dual problem. Alternatively, one can transform the data and project it into a higher-dimensional space, where it becomes linearly separable. The equivalent classifier back in the original feature space will generally be non-linear ,34. If aFigure D = 20,000 as the number of features and n ∈ {60, 120, 180} as the number of sample points available for each synthetic feature-label distribution. Because there is no closed form expression to calculate the true error of the designed classifiers, we generate a large independent test sample of size nt = 3000, with samples divided equally between the two classes. Because the RMS between the true and estimated error when using independent test data is bounded above by To emulate real-world scenarios involving thousands of genes with only a few sample points, we choose α, and β to get the desired number of read counts or tags. We assume that the parameter COV can take on two values, 0.01 and 0.05, representing two different levels of noise and unknown errors in the experiment. In its current state, RNA-Seq technology can provide different numbers of reads per sample, depending on many factors, such as quality of the sample, the desired coverage, sample multiplexing, etc. In this study, we examine a variety of ranges for RNA-Seq experiments. We start with a very low total number of RNA-Seq reads, which may not match the real-world experiments, however, they are necessary for comparing the SAGE-tags and NGS-reads models with similar coverage. Furthermore, demonstrating the classification results for the NGS-reads model with wide ranges introduces an extra internal variability, which makes interpretations of the results rather difficult. Table α = 2.0 and β = 3.75 in the distribution of si, the observed number of tags usually falls within this range. Similarly, the parameters α and β are chosen to meet the range requirements in the NGS-reads model.As explained in the previous section, the datasets generated from the MVN-GC model are transformed into the NGS-reads and SAGE-tags datasets through equation (2) and Poisson processes. We only need to properly set the parameters COV, n from the MVN-GC model, transform them to the corresponding NGS-reads and SAGE-tags samples, and apply the classification schemes to each training sample. By taking such an approach, we aim to compare the classification performance of the three pipelines, in terms of the true error of the deigned classifiers. We also study the effect of NGS-reads and SAGE-tags transformations on the performance of a simple t-test biomarker discovery method, where we report the probability that global markers are recovered when d ≪ D features are selected after the feature-selection step.Our goal is to study the performance of the traditional classification schemes on different sources of random samples; thus, we take a Monte-Carlo approach and generate 10,000 random training sets of size t-test feature selection can be approximated empirically by the percentage of experiments  that detect such a number of markers. This probability depends on the size of the training data sample, quality of features, and underlying joint probability distribution of the features. Here, we only show the results for the MVN-GC model, with d = 10, in Table ρ: {0.05, 0.4}, {0.05, 0.8}, {0.1, 0.4}, {0.1, 0.8}, respectively. In each table, we also report the corresponding results for the SAGE-tags model in a row with the NGS-reads range of [ 50K − 100K]. A successful feature selection should identify as many global markers as possible, however the situation is worsened because with small sample sizes the noisy features sometimes may appear to be good among the 20,000 features. As the number of sample points increases, we expect to get better results for the feature selection and this is exactly what we see in Tables The probability of recovering a certain number of global markers after a The true error of a designed classifier measures the generalization capability of the classifier on a future sample point. Given a set of training sample points and a classification rule, one needs the full feature-label probability distribution to calculate the true error of the classifier designed on the training set. In this study, we find the true error of a classifier with a large independent sample. Because the training sample is random, the true error is a random variable with its own probability distribution. Therefore, to demonstrate the actual performance of the designed classifiers, the estimated probability density function (PDF) of the true error for each classification rule, distribution model and all data generation models  is presented.d global markers are directly used for classification. In the second scheme, t-test feature selection is done before a classifier is designed to select the best d features. Figures ρ = 0.8) and COV = 0.05. Distributions with higher and tighter densities around lower true errors indicate better classifier performance. If the PDFs can be approximated as univariate Gaussians, then a good classification performance amounts to smaller mean and variance. In all cases, for similar sample sizes and similar settings for ρ and COV, MVN-GC outperforms the two other data types. This is not surprising since it is considered as the ground truth. Also, it is evident that a larger sample size gives better classifiers with smaller variance as illustrated by the distribution of the true error.We report the true errors of the designed classifiers for two different feature-selection schemes and classification rules. In the first scheme, no feature selection is performed and the first d features (with or without feature selection) in the model, SAGE-tags and NGS-reads for small ranges of the total reads yield similar results  in terms of the classification performance, and both are inferior when compared to the ground truth. This may lead to a conclusion that one should not expect improvements in classification performance when gene-expression levels are processed through an NGS-reads pipeline. However, our main objective here is to show that as one increases the total number of reads for the NGS-reads model, improvement can be achieved and the error rates decrease. This conclusion confirms the intuition provided by a simple calculation about the increase of the separability of two Poisson random variables with means proportional to the number of reads. However, notice that the modeling of the sequencing pipeline introduces randomness in the means of the respective Poisson parameters describing the individual gene reads. Moreover, our focus is not that much on the separability of the two classes/phenotypes but rather on the classification performance as measured by the classification error and its dependence on ground truth (MVN-GC model) sample size, sequencing depth, and feature vectors. Our goal for modeling NGS-reads with small ranges is to demonstrate the performance of SAGE and RNA-Seq, when their data is similar. This result suggests that having a larger number of reads for the RNA-Seq experiments could compensate for the errors that can occur during the NGS-reads pipeline sample processing. Here we have shown the results only for COV = 0.05, since in our observations, it appears that changing COV has little effect on the distribution of the true error for both SAGE-tags and NGS-reads models, except that it slightly increases the variance of the PDFs. The effect of feature selection (right columns) on the performance is best shown when the sample size is small. In this case, the variance of the true error is so large that drawing any conclusion regarding the performance on a small sample would be futile. Another interesting observation is that, on average, 3NN and RBF-SVM classification rules outperform LDA for both NGS-reads and SAGE-tags data, supporting the idea that using linear classifiers for these types of data is not the best way to proceed in a highly non-Gaussian model – as is the case after the data have gone through the pipeline. The best rates for the expected true error across all settings are achieved by the RBF-SVM classification rule, even for small samples, especially with small variance.Figures In this paper, we model gene-expression levels as a multivariate Gaussian distribution that statistically captures the real mRNA levels within the cells. The newly developed technologies of sequencing DNA/RNA, referred to as NGS, and their ascendant SAGE technology generate discrete measures for gene expressions. The count data from these technologies can be modeled with a Poisson distribution. The multivariate Gaussian gene expressions are transformed through a Poisson filter to model NGS and SAGE technologies. The three categories of data are subjected to feature selection and classification. The objective is to evaluate the performance of the NGS technologies in classification. Our simulations show that when the gene expressions are directly used in classification, the best performance in terms of the classification error is achieved. The NGS-reads model generates considerably higher coverage for genes and can outperform SAGE in classification, when the experiment generates large number of reads. Even though NGS still has a variety of error sources involved in its process, its high volume of reads for a specific gene can lower the chance of misclassification. Thus, it is important to use the highest possible coverage for the entire genome while performing a RNA-Seq analysis if the goal of the study is to distinguish the samples of interest. Nevertheless, one must recognize that, as is typical with nonlinear transformations, the NGS pipeline transforms the original Gaussian data in such a way as to increase classification difficulty. As more refined modeling becomes available for the NGS pipeline, further study needs to performed, as has been done in the case of the LC-MS proteomics pipeline , to deteThe authors declare that they have no competing interests.NG proposed the main idea, planned/structured the study, and worked on the simulations and manuscript. MRY contributed to the main idea, simulation, structure of the study, and the manuscript. CDJ and II helped in conceiving the study and revised the manuscript. ERD contributed to the formulation of main idea and also to the steps of the study, and revised the manuscript. All authors read and approved the final manuscript.


Metastatic prostate cancer (PCa) has no curative treatment options. Some forms of PCa are indolent and slow growing, while others metastasise quickly and may prove fatal within a very short time. The basis of this variable prognosis is poorly understood, despite considerable research. The aim of this study was to identify markers associated with the progression of PCa.Artificial neuronal network analysis combined with data from literature and previous work produced a panel of putative PCa progression markers, which were used in a transcriptomic analysis of 29 radical prostatectomy samples and correlated with clinical outcome.ANPEP, ABL1, PSCA, EFNA1, HSPB1, INMT and TRIP13. Two data transformation methods were utilised with only markers that were significant in both selected for further analysis. ANPEP and EFNA1 were significantly correlated with Gleason score. Models of progression co-utilising markers ANPEP and ABL1 or ANPEP and PSCA had the ability to correctly predict indolent or aggressive disease, based on Gleason score, in 89.7% and 86.2% of cases, respectively. Another model of TRIP13 expression in combination with preoperative PSA level and Gleason score was able to correctly predict recurrence in 85.7% of cases.Statistical analysis yielded seven putative markers of PCa progression, This proof of principle study demonstrates a novel association of carcinogenic and tumourigenic gene expression with PCa stage and prognosis. Prostate cancer (PCa) is the most common male cancer in the United Kingdom and accounts for 12% of male cancer deaths, making it the second most common cause of male cancer death . At presQuantitative real-time PCR (qPCR) has become a useful tool for validation and reduction of data produced from microarray studies , HPRT1 (hypoxanthine phosphoribosyltransferase 1), PBGD (porphobilinogen deaminase) and SDHA , which all showed little variation of expression in prostate and PCa tissues  labelled slides were used to identify regions of the highest grade of PCa and these regions were marked. The HE slide was then used to guide sampling of the tissue of interest. Where possible, duplicate 0.6 mm punches were taken using a manual tissue microarray platform . A total of 41 samples were included at this stage.PBGD (reference gene) was amplified by qPCR  in triplicate for each sample and also once for each RT-negative control. Additionally, several no template controls and a positive control in triplicate were included. These ‘sighting shot’ experiments were performed as detailed by CT (cycle threshold) of <40 was classed as showing good quality cDNA.RNA was extracted and reverse transcription (RT) performed as detailed by The cDNA was prepared and loaded, and the Taqman arrays sealed as described by CT of <35 for both 18S and PBGD. Taqman array data were converted to normalised expression ratios using two methods; the first method was the Applied Biosystems recommended method (2T−ΔΔC) and the second was a method that allows for variation of amplification efficiency (Pfaffl method). Both methods were used to corroborate any findings by two independent analyses. First, the 2T−ΔΔC method   were considered for further analysis. Genes found to be significantly different were then assessed for correlations before logistic regression to build a model of PCa progression. Correlating variables can confound models as it is difficult to ascertain, which variable makes the greater contribution  on samples that showed a T method  was usedin PBGD) . This daribution . LogistiPBGD expression was then used as a measure of cDNA quality. A PBGD CT <40 was taken as an indication of adequate quality cDNA. If no amplification was detectable within 40 cycles, it was assumed that there was insufficient starting material.Quantification (absorbance at 260 nm), purity (260/280 and 260/230 nm) and integrity (sighting shot experiment) for each sample are presented in the CT of <35 is traditionally thought of as optimal, samples that had a CT of >35 were loaded onto the Taqman array at 600 ng μl–1 rather than at 300 ng μl–1 to increase the starting material present. Variation of cDNA starting quantity did not pose a problem as quantification was relative to the amplification of PBGD from the same sample.Following these initial RNA and cDNA analyses, samples were taken forward to be used in the large scale Taqman array study. As a μl–1) of the pooled sample and from this the amplification efficiency and R2 values were calculated.Amplification of cDNA from each sample was performed without replication because of low RNA yield. Therefore, to assess intra- and inter-assay variation, a pooled sample  was amplified in triplicate at the beginning and end of this research trial. The initial amplification graph of the pooled sample was used to set the threshold fluorescence, above which each amplicon is classed as having ‘come up’ . The threshold was set at 0.5 so that it was above background noise but still within the exponential phase of amplification . A standR2 of >0.98 . A further measure of qPCR accuracy and reproducibility is the variation between replicates and between different Taqman arrays. Coefficient of variation (CV) was calculated from replicate values within the replicate plates and mean values between replicate plates  in recurrent disease while TRIP13 expression was significantly increased (to 95% confidence) in recurrent disease. Preoperative PSA and Gleason score were also assessed as predictors of recurrence. Gleason score was subjected to a Mann–Whitney U-test rather than a Gleason as the data are ordinal.A more accurate method of studying progression of disease is to compare gene expression of patients who have suffered recurrent disease, biochemically or with evidence of distant metastasis, with those who have not. Levels of PSA should be undetectable within 4 weeks of radical prostatectomy . The genes with the highest ‘percentage predicted correctly’ were combined to find the best model from the data sets. Receiver operator characteristic curves were drawn for the single and combination markers listed in ANPEP and ABL1 (AUC=0.889) in the 2T−ΔΔC data set and ANPEP and PSCA (AUC=0.869) in the Pfaffl data set, for prediction of Gleason score grouping. A combined model using TRIP13, PSA and Gleason score in the Pfaffl data set was also very good at predicting recurrence (AUC=0.888).An ideal ROC curve should have high sensitivity and specificity (or low 1-specificity), which means a low probability of false negatives and a low probability of false positives . The besT−ΔΔC data set, the best predictive model was a combined gene expression model of ANPEP and ABL1. This model could correctly assign 89.7% of patients to indolent or aggressive disease groups based on Gleason score. Using a cutoff of −0.16, patients with a higher score could be identified as indolent with 82% sensitivity and 94% specificity. Patients with lower expression of ANPEP and ABL1 could be at risk of potentially aggressive disease. Similarly, the best predictive model within the Pfaffl data set was a combined model of ANPEP and PSCA, which could correctly assign 86.2% of patients to indolent and aggressive groupings based on Gleason score. A cutoff of –0.25 could predict indolent or aggressive disease groupings with 82% sensitivity and 94% specificity. Again, a score below this cutoff would be indicative of a more aggressive phenotype. Finally, a good predictive model was also found from the Pfaffl data set, a combined model of TRIP13 with pre-operative PSA and Gleason score of radical was able to correctly assign recurrence in 85.7% of patients. A cutoff of 0.14 could predict recurrence with 87.5% sensitivity and 90% specificity. A score above this cutoff would be indicative of a high chance of recurrence.Within the 2EFNA1, ABL1 and HSPB1 as predictors of Gleason score grouping, and INMT and ANPEP as predictors of recurrence grouping. Both EFNA1 and ABL1 (using the 2T−ΔΔC data set) had the ability to predict 79.3% of patients as having indolent or aggressive disease. EFNA1, with a cut point of −0.40, could predict aggressivity of disease with 82% sensitivity and 83% specificity and ABL1, with a cut point of −0.30, could predict aggressivity of disease with 73% sensitivity and 79% specificity. INMT, identified using the 2T−ΔΔC data set, could predict recurrence of disease in 72.4% of patients and had a sensitivity and specificity of 89% and 65%, respectively, with a cut point of −1.16. Using the Pfaffl data set, HSPB1 could accurately assign patients into aggressive grouping by Gleason score in 82.8% of cases and using a cut point of −0.54 had a sensitivity and specificity of 64% and 94%, respectively. ANPEP could assign patients into recurrence grouping in 79.3% of cases and with a cut point of 0.31 could predict recurrence with 78% sensitivity and 70% specificity. As with the combined gene models, a score lower than the cut point is indicative of aggressive disease.Single gene models identified This study highlights that, because of the complexity of cancer, many genes and proteins are likely to be differentially regulated, and underpins the difficulty of biomarker discovery. Many biomarkers (both diagnostic and prognostic) are identified by researchers but few result in clinically viable tests. Biomarkers that are not significantly better than current testing methods, such as PSA, are unlikely to be embraced by clinicians and, so far, none of the biomarkers discovered have the sensitivity or specificity to replace PSA. Owing to the complexity of PCa , it is perhaps naive to imagine that a single gene or protein marker will indeed be able to fulfil all of the ‘ideal biomarker’ criteria. This is a ‘proof of concept’ study that highlights the utility of an informed approach when searching for disease biomarkers. ANPEP, EFNA1, ABL1, INMT, HSPB1 and PSCA. TRIP13 was also an accurate predictor of recurrence when used in conjunction with preoperative PSA and Gleason score.Taqman arrays customised for potential PCa progression markers were used to study gene expression in a pilot series of 29 patients. By a variety of data analysis methods, 10 genes were identified, 6 of which could be accurately used to predict PCa progression (defined by Gleason or recurrence) in up to 89.7% of cases. The six genes that were best used to model PCa progression included: Aminopeptidase N, or CD13 (ANPEP) is a zinc-dependent matrix metallopeptidase that is membrane bound and thought to have an important role in angiogenesis  is a methylation catalyst involved in the methylation of tryptamine during the production of N N-dimethyltryptamine , decreased expression of INMT was associated with cancerous tissue when compared with normal lung (INMT in the more aggressive PCa cases. The implication of the downregulation of INMT in association with the progression of PCa is unclear.Indolethylamine yptamine . This paTRIP13 has also been implicated as a marker of early disease related mortality in multiple myeloma as part of a 70-gene model. Further multivariate analysis identified a subset of 17 genes that had a similar predictive power, but TRIP13 did not feature in this smaller panel (TRIP13 is a thyroid receptor interacting protein (er panel .The use of multiple biomarkers showed a greater ability to correctly identify aggressive disease than single biomarkers alone and therefore, multivariate analysis may be preferable to univariate analysis. This is unsurprising as cancer is a temporally and spatially complex disease that occurs as a result of several mutations to genes involved in a variety of biochemical pathways. Moreover, carcinogenic mutations may not be identical for each individual. It may be inappropriate to expect a single marker to be able to identify progressive disease with a great deal of accuracy as has been suggested in this study and others (Quantitative real-time PCR has become one of the most widely utilised methods of gene expression analysis in disease onset and progression. Many studies have utilised this technique to validate microarray data. The design of this study may have resulted in potential sample bias as only radical prostatectomy samples were used. Radical prostatectomy tends only to be performed with organ-confined lower grade disease so there is a bias towards low recurrence risk disease. Perhaps a further study of biopsy specimens would overcome any potential bias. An alternative approach might have been to utilise mass spectrometry methods to identify putative protein markers and then design a Taqman array accordingly, but this would be a more biased approach. Additional characterisation could take place at the protein level by immunohistochemical methods and analysis of serum expression via western blotting or ELISA.ANPEP and TRIP13 found to be significantly differentially expressed in terms of Gleason score and biochemical and metastatic recurrence, respectively (ABL1 was not found to be significant but was approaching significance. Several other genes identified from this study were also verified by In conclusion, this ‘proof of concept’ study presents a putative gene panel of PCa progression. In order to verify any signature, a further study on a large ‘test sample set’ is warranted to investigate disease recurrence or survival and would verify and validate any potential prognostic indicators. Interestingly, these markers were assessed in the mRNA expression data produced by ectively . ABL1 wa


The identification of robust lists of molecular biomarkers related to a disease is a fundamental step for early diagnosis and treatment. However, methodologies for the discovery of biomarkers using microarray data often provide results with limited overlap. These differences are imputable to 1) dataset size (few subjects with respect to the number of features); 2) heterogeneity of the disease; 3) heterogeneity of experimental protocols and computational pipelines employed in the analysis. In this paper, we focus on the first two issues and assess, both on simulated (through an in silico regulation network model) and real clinical datasets, the consistency of candidate biomarkers provided by a number of different methods.We extensively simulated the effect of heterogeneity characteristic of complex diseases on different sets of microarray data. Heterogeneity was reproduced by simulating both intrinsic variability of the population and the alteration of regulatory mechanisms. Population variability was simulated by modeling evolution of a pool of subjects; then, a subset of them underwent alterations in regulatory mechanisms so as to mimic the disease state.The simulated data allowed us to outline advantages and drawbacks of different methods across multiple studies and varying number of samples and to evaluate precision of feature selection on a benchmark with known biomarkers. Although comparable classification accuracy was reached by different methods, the use of external cross-validation loops is helpful in finding features with a higher degree of precision and stability. Application to real data confirmed these results. In the last decade, transcriptome analysis performed with high-throughput microarrays has experienced a huge diffusion and has profoundly changed the approach to the study of complex diseases. In an experimental design, the data typically come from different subjects and phenotypes. The analysis of these data has proven extremely useful for the identification of biomarker genes and for the development of new physiologic hypotheses useful for answering diagnostic, prognostic and functional questions. However, for complex diseases such as cancer, the high-throughput analysis carried out in different laboratories and research centers has given different results, with limited overlap or reduced statistical significance Datasets often include small numbers of subjects (some tens) with respect to the number of variables (tens of thousands of genomic probes in human) The most complex pathologies, such as cancer, are heterogeneous and multifactorial, as a result of the alteration of multiple regulatory pathways and of the interplay between different genes and the environment, rather than referable to a single dysfunctional gene like in monogenic diseases Widely used methodologies for the identification of biomarkers using microarray data are based on computing differential gene expression by univariate statistical tests. Such approaches provide information on the effects of specific genes as individual features, whereas it is now widely recognized that the interplay between weakly up/down regulated genes, although not significantly differentially expressed, might be extremely important to characterize a disease status As shown in A slightly different issue, although related with list stability, is the precision of biomarker identification, i.e. the ability to select true biomarkers, defined as features biologically related to the physiological or clinical condition under study as cause or effect of it. A second contribution of our work is the generation of a simulated dataset to assess alternative methods' performance across multiple studies and varying number of samples, and to evaluate precision of feature selection on a benchmark with known biomarkers. We extensively simulate the effect of heterogeneity and variability on different sets of synthetic microarray data consisting of two balanced groups of 50, 20, 15 or 10 subjects. Sample heterogeneity characteristic of complex diseases is reproduced within the same group by simulating both intrinsic variability of the population and the alteration of regulatory mechanisms induced by the disease. Population variability is simulated by modeling evolution of a pool of subjects in terms of pairing, mutation and selection in order to generate individuals characterized by different genotypes. Then, a subset of this population undergoes alterations in regulatory mechanisms so to mimic the disease state; these perturbations are slightly different across the patients in the diseased group, so to reflect the lack of homogeneity among patients that is typically reported in the literature for complex diseases Different methods for binary classification and feature weighting and ranking are applied to both simulated and real data. In particular, the classical Support Vector Machine algorithm (SVM) N = 10000 genes, based on the simulator described in W, with weights ijw different from zero if gene-product j directly affects the expression of gene i. The sign and the magnitude of ijw indicate the sign and the strength of the regulation. Differential equations were used to model the dynamics of transcription and degradation as continuous variables and to describe transcription delay with different time constants for each gene  and a specific phenotype (the system attractors). Given specific initial conditions (i.e. environment condition that we consider fixed for the purpose of this work), the initial population at generation 1 consisted of M individuals with identical connectivity matrix W and with N dimensional vectors of expression values obtained by considering the steady state reached by the system. Gene specific kinetic parameters iα and iβ were sampled from Gaussian distributions with means αμ, βμ and standard deviations ασ, βσ. For each subject, αμ and βμ were set to 20 and 0.2, respectively, whereas ασ and βσ were sampled from a Gaussian distribution with means 0.5 and 0.02 and standard deviations equal to 0.075 and 0.0025, respectively. Parameters values  with probability 0.025/1619. The new value of each mutated ijw was sampled from a Gaussian distribution with mean and standard deviation equal to 0 and 1, respectively. Therefore, at each iteration, each subject mutated with probability 0.025.c. Selection. Assuming, in a naïve simplification of reality, that individuals behaved as haploid organisms and that the initial phenotype was essential for survival, subjects with at least one mutated ijw were allowed to survive only if their phenotype did not change with respect to the original population. In practice, we calculated the Euclidean distance between the expression profile of each mutated subject  and the average expression profile of subjects at generation 1; if Euclidean distance exceeded the value of 0.81 (corresponding to the percentile 99.5 of the observed distances) the subject was eliminated, otherwise he/she survived. At each generation, M individuals were generated, independently of the number of parents survived in the previous generation. Evolution proceeded for a time sufficient to have a final population of M subjects with the same phenotype but different genotype, i.e. 150 generations , as described in j was simulated by setting to 0 its expression and all the elements of row j in matrix W. Consistently, the knock-down of gene j was simulated by halving its value and all the elements of row j in matrix W. Diseased subjects had 4, 5 or 6 genes belonging to W900 that were knocked out or down. The proportion of subjects with 4, 5 or 6 genes affected was set equal to 1/3, 1/3, 1/3, respectively. For each gene, the proportion of subjects affected by knock-out and knock-down was set equal to 1/3 and 2/3, respectively. ij in W900, and the heterogeneity of the disease. Comparison between simulated and Affymetrix data  showed that the datasets have very similar distribution .Once the base population was simulated, two groups, each of 500 subjects, were defined. The pathological condition was simulated by knocking out or down six target hubs, defined as those genes with the highest out-degree and expression value at steady state higher than 0.88, so that their knock-out (down) achieved an effect. The knock-out of gene The putative biomarkers were defined as those genes directly or indirectly regulated by at least one of the six hubs, having expression modified by the knock-out (down). This resulted in 155 biomarkers on a total of 10000 features.To consider the effect of sample size, we partitioned the two groups of 500 healthy and 500 diseased subjects into 4 sets of 10 balanced non-overlapping datasets of size 50, 20, 15 or 10 subjects per group (10 datasets for each case study), for a total of 40 simulated datasets.Publicly available data from three breast cancer microarray studies were collected from Gene Expression Omnibus repository (GEO) with accession numbers: GSE2990 Datasets were all hybridized using Affymetrix U133 Genechips™ (HG-U133A). Samples that have known estrogen-receptor (ER) status were selected so to have balanced groups (ER+ and ER−), homogeneous with respect to characteristics such as age, tumor size and histological grade. We chose to investigate the ER status because it is always assessed in breast biopsies, therefore it is very often present among the clinical/pathological information given with the datasets. Moreover, the assessment of the ER status is important to divide breast cancer into molecular classes and to treat cancer with the hormone blocking therapy c  and, for the Gaussian kernel, of the bandwidth σ.Support Vector Machines σ of the internal kernel.Iterative Relief SRDA algorithm embeds the classical Discriminant Analysis into a regression framework through the use of spectral graph analysis For GSVM, IRSVM and SRDA, parameter tuning was performed through a preliminary 3-fold cross-validation (without feature ranking) run for a set of possible parameter values.The four methods, LSVM, GSVM, IRSVM and SRDA, were used both in single cross-validation and in a Monte Carlo bootstrap resampling schema with B = 100 external training/test splits with 3-fold cross-validation as internal resampling . This strategy has been proved to be an effective countermeasure against unwanted selection bias effects In the four aforementioned methods, the Entropy-based Recursive Feature Elimination (ERFE) procedure was used as the ranking schema The SAM test Algorithm performance was evaluated in terms of the ability to select true biomarkers, to provide stable lists of biomarkers and to accurately classify the subjects.vs. recall (number of true positives divided by the number of true biomarkers) curve was also considered to outline the ability of the different methods to rank the features, a task related with the ability to select the true biomarkers.The ability to select the true biomarkers was evaluated in term of precision (number of true positives divided by the number of selected features) obtained by the different methods according to their choice of the optimal number of features. The area under the precision T1 and T2 of p ranked features, the Canberra distance between them is defined as:1(i)τ and 2(i)τ indicate the rank, i.e. the position, of feature i in the ordered lists T1 and T2, respectively. The stability indicator for a given set of lists was computed as the mean of the Canberra distances between pairs of lists in the set, normalized by its expected value on the whole permutation group on p features: the obtained value ranges then between 0  and 1.4 , with 1 as the case of randomly generated lists. A different extension based on quotients of permutation groups allowed comparing lists T1 and T2 of different length 1l, 2l:p is the total number of analyzed features and Γj  belong to the set Sj of all the lists having the first jl features ordered as in Tj and the remaining (p–jl) elements ordered in all the (p–jl)! possible combinations. This is called the complete version of the partial lists distance: neglecting its component depending only on the discarded features we ended up with a different measure  better tailored to highlight variations on partial short lists To evaluate the ability of the different methods to provide stable lists of biomarkers, the algebraic stability indicator derived by Canberra distance was used MCCThe Matthews correlation coefficient, Statistical significance of the comparison between each method and its bootstrap variant was assessed using Wilcoxon signed ranks test with significance level α equal to 0.05. Differences among the four multivariate feature selection methods in their bootstrap variant were assessed using Friedman test (α = 0.05), followed, if significant, by Wilcoxon signed ranks test to examine between which methods the differences actually occur, with a significance level α equal to 0.05/6 = 0.0083 to correct for multiple testing. Finally, SAM was compared with the other eight methods using Wilcoxon signed ranks test with a significance level α equal to 0.05/8 = 0.00625 to correct for multiple testing.Application of the nine biomarker discovery methods on the forty simulated datasets provides information on precision of feature selection, stability of biomarker lists and classification accuracy.There are no appreciable differences among different bootstrap methods in terms of precision . In the case of 50 subjects per group, SAM detects differentially expressed features with average precision comparable to that obtained by the other methods, but GSVM, IRSVM and IRSVM_B, which perform statistically significantly better than SAM . With 20 subjects per group, SAM is not able to select any gene with FDR lower than 0.05 in six datasets, whereas in the remaining four, it selects in average 50 features with high precision (0.85 in average). In these latter cases, SAM performs statistically significantly better than LSVM  and SRDA , i.e. two methods without the bootstrap approach. Finally, with less than 20 subject per group, SAM is not able to select any gene in any of the dataset with FDR lower than 0.05; thus we could not report any result in these latter two cases.vs. recall curves (AUC) obtained by ranking features . However, when the number of subjects is lower than 20, SAM performance in feature ranking dramatically drops with respect to classification based methods . This behavior is consistent with the inability of SAM to select any feature with 15 and 10 subjects per group.−11 for sample size 20, 15, 10 and Wilcoxon signed ranks test gave p-value lower than 0.001 for every significant pairwise comparison).The ability of the various methods to select the same features across different datasets is depicted in In the case of 50 subjects per group, SAM shows list stability comparable to the one obtained by the other methods. With 20 subjects per group, SAM is as good as IRSVM_B; however, results are limited to the four datasets for which SAM was able to select features below the 0.05 FDR threshold. As for feature selection, with less than 20 subjects per group we do not report any results since SAM was not able to select any gene in any of the dataset.Bootstrap approach also improves classification accuracy : with 50Application of the various methods on breast cancer data provides information on list stability and classification accuracy.Results on dataset GSE3494 are not shown since none of the different methods gave good accuracy . On the other two datasets, results confirmed those obtained by simulated data. In particular, bootstrap resampling schema leads to an improvement in list stability , appreciDifferences between bootstrap and standard approach are statistically significant for every method  with dataset GSE2990 and for LSVM and GSVM with dataset GSE7390.vs. GSE7390) the core Canberra distance is equal to 0.63. SAM results are not shown in In terms of stability, SAM performance is poor: when 20 subjects per group are repeatedly sampled from each dataset the core Canberra distance between lists of biomarkers ranges between 0.04 and 0.37 (average 0.27) for GSE2990 and between 0.13 and 0.31 (average 0.22) for GSE7390; on the other hand, between the complete datasets  the intersection of the lists obtained by the four bootstrap methods on datasets GSE2990 and GSE7390; 2) the intersection of the lists obtained by IRSVM_B on datasets GSE2990 and GSE7390. The two lists of genes and the results of enrichment analysis are available in The identification of an appropriate and robust biomarker signature of a disease is a fundamental step for early diagnosis and treatment. However, for complex diseases such as cancer, high throughput analysis carried out in different research centers may exhibit poor reproducibility, with limited overlap or reduced statistical significance. The results of the MAQC-II study address in a comprehensive analysis this issue on real datasets by comparing methods and procedures between data analysis teams barbara.dicamillo@dei.unipd.it).A first contribution of this work is the comparison of different classification methods applied on real microarray datasets, in terms of consistency of lists of candidate biomarkers and classification accuracy. A second contribution of our work is the generation of a simulated dataset to extensively assess average method performance on a large number of studies and experimental conditions, and to evaluate precision and feature ranking performance on a benchmark with known biomarkers. Heterogeneity of samples in each group is obtained by simulating both intrinsic variability of the population and heterogeneity of the disease. Despite its simplicity with respect to real systems, the simulator provides a versatile test bed to assess a wide spectrum of methodologies. The dataset is available upon request  and stability comparable to the one obtained using IRSVM_B, although this latter outperforms SAM in feature ranking. Finally, with less than 20 subjects per group, SAM performance dramatically drops with respect to classification based methods.With real data, only list stability and classification accuracy can be assessed. In both cases, results of classification methods tightly resemble those obtained with simulated data.In conclusion, our analysis confirms the MAQC-II indication that comparably good classification accuracy can be reached by different methods on the same task, provided that a valid Data Analysis Plan is adopted Figure S1Progression of population mutation with generations. Total number of subjects mutated with respect to the original population with the progress of generations. Only survived subjects are represented for each generation.(TIF)Click here for additional data file.Text S1In Silico model of regulatory networks.(DOC)Click here for additional data file.Text S2Selected genes.(DOC)Click here for additional data file.


Identifying stable gene lists for diagnosis, prognosis prediction, and treatment guidance of tumors remains a major challenge in cancer research. Microarrays measuring differential gene expression are widely used and should be versatile predictors of disease and other phenotypic data. However, gene expression profile studies and predictive biomarkers are often of low power, requiring numerous samples for a sound statistic, or vary between studies. Given the inconsistency of results across similar studies, methods that identify robust biomarkers from microarray data are needed to relay true biological information. Here we present a method to demonstrate that gene list stability and predictive power depends not only on the size of studies, but also on the clinical phenotype.Our method projects genomic tumor expression data to a lower dimensional space representing the main variation in the data. Some information regarding the phenotype resides in this low dimensional space, while some information resides in the residuum. We then introduce an information ratio (IR) as a metric defined by the partition between projected and residual space. Upon grouping phenotypes such as tumor tissue, histological grades, relapse, or aging, we show that higher IR values correlated with phenotypes that yield less robust biomarkers whereas lower IR values showed higher transferability across studies. Our results indicate that the IR is correlated with predictive accuracy. When tested across different published datasets, the IR can identify information-rich data characterizing clinical phenotypes and stable biomarkers.The IR presents a quantitative metric to estimate the information content of gene expression data with respect to particular phenotypes. The challenge to identify stable tumor prognosis and predictive outcome markers remains critical in clinical cancer research. Many studies rely on microarrays to determine which genes are predominantly indicative of clinical cancer phenotypes or prognosis. However, biological and technical variations across samples and studies make it challenging to identify true, predictive clinical biomarkers ,2. Identn by p matrix with n being the number of samples and p the number of measured features or probesets. n tends to be much smaller than p. Dimension reduction techniques are widely used to reduce the dimensionality of the data from p to a smaller value d .-21.12-21d = 4) space. Differential expression and corresponding p-values of differential expression were calculated in projected and residual space for a series of phenotypic variables. Hence, for each gene and combination of phenotype and study there are two p-values.The following section presents the results of the analysis of several publicly available microarray datasets. For each dataset the normalized expression values were projected to a lower dimensional  data from lung and breast tissue showing different clinical phenotypes was performed as shown in Figure Comparing the p-values of the projected expression value and the residual expression value can bring light to where information lies in the measured expression data. The information may reside mainly in the residual space, mainly in the projected space, or somewhere in between. A comparison of logp (blue crosses) onto Sn shows high significance  compared to the residuals lpr (red crosses), almost all significance from the original data (x-axis) is expressed in lpp, as shown by the distribution of p-values. The ratio between the lpr and lpp (information ratio) is low  onto Sn shows similar p-values compared to the residuals lpr (red crosses). The information ratio is almost 0.5   onto Sn shows very low absolute values compared to the residuals lpr (red crosses). The information ratio is almost 1 .   and only one well-defined experimental variation is performed, then all differential expressions should represent only the biological variation in the sample and should be associated with the first principal component of the PCA. This correlates with a Type 1 genome-wide differential expression pattern where the resulting distribution is dependent on the study design. In contrast, clinical studies have a high biological heterogeneity, which is not well characterized In order to quantify the patterns we introduce the information ratio (IR). The IR describes the ratio of differential expression, which is stored in the residual space, compared to the information in both the residual and projected space. However, rather than using fold change values, p-values of differential expression are used. In order to suppress false results from genes with low overall differential expression, the IR is calculated as weighted sum of p-value ratios:pr, i is the p-value of the residual for gene i, and pp, i is the p-value of principle component projections for differential expression of gene i. The weights, wi, for each gene i guarantee that the genes with high sensitivity contribute more to IR than genes with low sensitivity. Here we use an intrinsic weight distribution so that all gene groups with similar sensitivity contribute equally to the IR.where The IR is calculated for different phenotypes and reveals a property specific to the clinical phenotype Table . As seen10 p-values for each gene across the pairs of tumor characteristics. As displayed for grade 1 versus grade 2 on Figure For classification of clinical samples based on microarray data, prediction is usually performed with a gene list, a subset of all available genes. It is important to understand and gauge the stability of gene lists across different studies. To this end we used a dataset consisting of 8 breast cancer studies described in Table Results shown in Figure A detailed analysis of gene list stability and IR for seven factors and all 8 breast cancer studies is displayed in Figure Ein-Dor et al. estimated the stability of ranked gene lists in studies with respect to outcome of tumor therapies in terms of the size of the clinical study . The ove10 (IR) (Pearson's r2 = 0.76).Our analysis, which considers gene list ranking with respect to various physiological phenotypes, shows that the impact of the sample size depends on the type of classification Figure . The sig2 = 0.25. The mean accuracy for classification tasks with IR ≤ 0.25 was 81%, while for tasks with IR > 0.25 the mean accuracy was 70%. The difference in prediction accuracy is significant with p < 0.005 (Welch two sample t-test). See Figure The IR is a suitable indicator for gene list stability, with a high IR being indicative of a stable gene list. Non-stable gene lists are problematic for classifiers . Here we2 = 0.43). The mean loss of prediction accuracy is 18% for cases with IR ≤ 0.25, and 28% for cases with IR > 0.25 (p < 1e-12).If one study is used to derive a gene list, and this gene list is used to build a classifier for another study, a decrease in accuracy can be observed. Figure A body of synthetic expression data was generated with dimensionality between 1 and 100. For this data, IR and prediction accuracy was calculated. Results demonstrate that IR and prediction accuracy depend on dimensionality which is analogous to observations in real gene expression data. However, the dependency of the IR on the specific phenotype was not apparent in the simulated data. For details on methods and results see additional file Gene expression data sets were projected into a four-dimensional subspace and in a residual gene expression space. Depending on the phenotype the information is distributed differently between the subspace and the residual space. We introduced a p-value based information ratio, IR, to quantify this observation. When comparing cancer cells to control tissues, most information resides in the subspace Figure , howeverAn interpretation could be that in Type 1 classifications, where IR values are low, the genome-wide differential expression associated with the shift in the phenotype, can be expressed by a combination of a few independent leading variations in the differential gene expression pattern. These variations may be represented by biological heterogeneity and the disease-related pattern in the sample. Hence, the true dimensionality of the genome-wide differential expression pattern becomes very low, such that variation in sample size within the range of standard clinical studies will have a significant impact on the stability. In contrast, the genome-wide differential expression shift of Type 2 and 3 classifications  cannot be reduced to the leading biological heterogeneities and hence retain high dimensionality. As the impact of sample size variation may depend on the dimensionality of the differential expression pattern, Type 2 and 3 classifications will benefit significantly less from increased sample sizes, which can be seen in typical clinical studies. Moreover, the qualitative heterogeneity of the genome-wide information distribution across different studies for high IR phenotypes indicate that biomarkers which are identified using ranked gene lists, will most likely not be predictive through statistical approaches alone. The information ratio can serve as a method to better understand the stable phenotypic variations within a study, especially since studies performed by various groups are often unable to identify stable gene lists despite the similar disease types or tissues under investigation -25. ExpeIn summary, the IR provides a metric for the capability of gene expression data to support clinical decisions. It has been observed elsewhere  that the2 values.Eight breast cancer, one lung cancer, and one prostate gene expression data sets along with clinical information were downloaded from the EBI ArrayExpress website . See Tab2 ≥ 0.99 one sample was omitted from this analysis. This occurred between the breast cancer studies E-GEOD-4922, E-GEOD-2990, and E-GEOD-3494. Then, to avoid a bias due to erroneous chips, samples with extreme mean expression rates (> 5σ) (one sample from E-GEOD-4922) have been omitted.The sets of samples from different sources did partly overlap. In order to remove redundant measurements, the correlation of all samples with all other samples was calculated and from pairs of samples with RAll probe set identifiers were mapped to Entrez gene symbols. In case several probe sets share the same gene symbol, the probe set with the largest mean expression over all samples was used as representative for that symbol. Across all studies, 6384 symbols were shared and only those were used for further analysis. It should be noted that probe sets representing the ER gene ESR1 and ESR2) were included. The associated clinical information was transformed to a binary value: Grade , tumor size (>25 mm vs. <20 mm), ER status (positive versus negative as reported), and outcome  , as implemented in MATLAB, was used to decompose {Cij} into its eigenvectors and eigenvalues, where the first eigenvectors represent the dominant, coherent variations in the data set. We denote the space, spanned by the first n eigenvectors, as S(n). Each eigenvector k represents a metagene whose expression Xk, l in each tissue l is given by the weighted sum of the contribution of all genes j to the eigenvector:For each data set, the correlation matrix {Cik-values, does not explicitly contain the data from the respective data source. However, since PCA represents the dominant variations within the respective data sets, normally the vectors Gl, quantifying the contribution of all genes to eigenvector l, depend on the composition of the data sample.This representation, using only the GG1,...,Gn depend on the composition of the samples, the subspace Sn, spanned by the set of all the first n vectors, depends significantly less on the sample composition. An appropriate value of n may depend on the variability studies, in the studies analyzed here n = 4 was used leading to sufficient results. Higher n did not lead to more significant differential expressions of the projections pp, i with respect to Type 1 classifications. This indicates that the subspace Sn is related to biological features. Changes in sample composition merely result in a rotation of the "coordinate system" spanning Sn, which can be represented not only by the vectors {G1,..., Gn}, but also by all orthogonal vector systems which can be generated by the rotation of {G1,..., Gn}.Although the individual vectors p, which is part of Sn and a residual component, xr, which is part of Sr, the subspace is then orthogonal to Sn:The expression values of each gene i in each tissue k can be split into two components: a component xi, k onto Sn using the solution ri of the following linear equation system for each gene i:The decomposition is performed by the projection of xThen it holds:i and for both split components xp, i and xr, i with respect to different physiological or clinical phenotypes. For example, (i) we set mamma carcinoma of grade 1 and grade 2 to be class 1 and tumors of grade 3 in class 2. Next, (ii) we use the p-values of a two-sided t-test (or parameter-free Wilcoxon test) to quantify the differential expression of each gene between tissues of class 1 and class 2. Finally, (iii) we get the genomic set lp = log10 p, which are the logarithms of these p-values. The significance values for the projections xp and xr are then denoted as lpp and lpr, respectively.The decomposition splits each expression value for each gene in each sample into 2 components. This apparent doubling of complexity yields additional insights into the information contained in the genomic data. Then we calculate the information content of the original expression values for each gene xw, we observed that the distribution of the genomic log10 p-values with respect to almost all physiological factors satisfy an exponential distribution  distribution as depicted in Figure where λ is chosen such that p and lpr. Since this depends on the choice of n, n was evaluated in a range of 1 to 10. The IR decreases with increasing n and stabilizes at n = 4. This value was selected after visual inspection (see additional file The information ratio was calculated based on lpSeveral metrics for comparing the order of gene lists between studies are available , here weThe correlation between the IR and the potential accuracy of a predictor was evaluated. For this we used SVM as implemented in the libSVM library  and utilA loss in prediction accuracy can be expected when a gene list derived from one study is used for classification in another study. From the first study and factor, the top 5% differentially expressed genes were extracted. This gene list was then used to train an SVM for each study with default parameters. The out of bag prediction accuracy was established. In turn, each study was used to derive a gene list, and this list was evaluated with all the other studies. The derived accuracy for the first study was better than the mean accuracies for other studies. Figure See additional file AS, NSA, and SS performed research and analysis for the paper. SS and AS conceived and designed the study. NSA wrote the paper with contributions from all authors. All authors have read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1755-8794/4/73/prepubSimulated data. Simulated expression data and estimation of IR and predictor accuracy for different dimensionalities of the data.Click here for fileFigure S1. Plot of subspace dimensionality against IR.Click here for fileWorkflow. Three slides with illustrations of the used workflow to calculate the IR and predictor accuracies.Click here for file


DNA microarray analysis is characterized by obtaining a large number of gene variables from a small number of observations. Cluster analysis is widely used to analyze DNA microarray data to make classification and diagnosis of disease. Because there are so many irrelevant and insignificant genes in a dataset, a feature selection approach must be employed in data analysis. The performance of cluster analysis of this high-throughput data depends on whether the feature selection approach chooses the most relevant genes associated with disease classes.Here we proposed a new method using multiple Orthogonal Partial Least Squares-Discriminant Analysis (mOPLS-DA) models and S-plots to select the most relevant genes to conduct three-class disease classification and prediction. We tested our method using Golub’s leukemia microarray data. For three classes with subtypes, we proposed hierarchical orthogonal partial least squares-discriminant analysis (OPLS-DA) models and S-plots to select features for two main classes and their subtypes. For three classes in parallel, we employed three OPLS-DA models and S-plots to choose marker genes for each class. The power of feature selection to classify and predict three-class disease was evaluated using cluster analysis. Further, the general performance of our method was tested using four public datasets and compared with those of four other feature selection methods. The results revealed that our method effectively selected the most relevant features for disease classification and prediction, and its performance was better than that of the other methods. DNA microarray analysis is an important tool in medicine and life sciences, because it measures simultaneously the expression levels of thousands of genes. In the past few years, many multivariate data analysis methods have been developed and applied to extract the full potential from microarray experiments including cluster analysis t-test Microarray data typically consist of a relatively small sample size  and a large number of genes , most of which may be irrelevant, insignificant, or redundant for disease classification and prediction In this study, we describe the development of a novel multiple orthogonal partial least squares discriminant analysis (mOPLS-DA) model and S-plots to conduct three-class gene selection. Partial least squares (PLS), a well-known dimension-reduction method, is a simple and efficient algorithm that conducts two-class and multiclass classification We used Golub’s leukemia microarray expression dataset We chose this dataset, because it is publicly available and has been analyzed by many others. Further, this dataset comprises the two main classes of ALL and AML, and ALL can be classified into ALL-B and ALL-T subtypes. We first treated this dataset as three classes with subtypes and then as three parallel classes.10-transformation. Each preprocessed variable was scaled by centering for principal component analysis (PCA) The microarray dataset was preprocessed following Golub’s preprocessing steps PCA is a multivariate projection method E describes the residual matrix that was not fitted in the PCA model.Where T is the score vector, P is the loading vector, and OPLS-DA The purpose of S-plot is to select putative interesting variables from the X-matrix based on OPLS-DA model. One advantage of the S-plot derived from the predictive component of OPLS-DA is that it combines the contributions and reliability of variables in a scatter plot. That is, the x-axis of the S-plot describes the fitted covariance vector, cov, and the y-axis represents the correlation coefficient vector, corr. The two vectors of the S-plot are calculated as follows t is the score vector in the OPLS-DA model predictive component, ts is the standard deviation of the OPLS predictive score vector, and Xs is the estimated standard deviation vector for each variable. This combination is particularly important, because it facilitates determination of variables with the highest correlation coefficient and the largest contribution to the model separation between two classes.Where The number of variables selected as potential markers is application-dependent, but the rule of thumb is recommended. In the present study, following Golub’s paper 2Y and Q2Y, respectively, according to the previous literature 2Y was calculated as follows:The goodness of fit and prediction of the OPLS-DA model were evaluated using model parameters RWhere PRESS is the prediction error of the sum of squares and SS is the sum squares of the response variables. The statistical significance of the OPLS-DA model was evaluated using the cross validation-analysis of variance (CV-ANOVA) technique http://www.lirmm.fr/~caraux/PermutMatrix/).Cluster analysis was used to test the rationality of the selected gene markers. For cluster analysis, the Euclidean or Pearson distance was chosen to compute the distance of dissimilarity using software PermutMatrixEN t-test as references to test the general performance of our method. OVRSVM http://people.kyb.tuebingen.mpg.de/spider/) in the MATLAB environment were employed to filter genes and fit classification model of OVRSVM.Four publicly available datasets, including 11_tumour We first used these methods to extract features from the training set and utilized them to fit the classification models of SVM and cluster analysis. Observations in the training and independent test sets were classified and predicted using the classification models. The number of misclassified observations was counted.#67), it was difficult to assign this sample to one class. Therefore, we selected two representative samples  by performing PCA on all ALL-T samples from the training and test sets  sample . This OPLS-DA model had good predictive power as indicated by the goodness of the prediction parameter (Q2Y = 0.798) computed using cross-validation (2Y and Q2Y was only 0.18. The CV-ANOVA test showed that the model was significant (P = 3.28488e-009). Score plots of the first OPLS-DA model revealed a clear separation was achieved along the first predictive component (data not shown).Because ALL-B and ALL-T are subtypes of acute lymphoblastic leukemia (ALL), we first regarded Golub’s leukemia data as a three-class disease with subtypes. Therefore, we fitted two OPLS-DA models using the training set . The firlidation . The dif2X = 0.348, R2Y = 0.99, Q2Y = 0.806) (The parameters of the goodness of fit and prediction of the second OPLS-DA model showed that this model was good (R= 0.806) . The P-vModel evaluation parameters suggested that the two OPLS-DA models were reliable. Two S-plots from these OPLS-DA models were employed for feature selection, and the top-ranked 50 genes were selected from 3571 genes as the most informative. We selected 40 genes with the largest correlation and covariance related to model class information using the first OPLS-DA model in which the expression levels of 20 genes increased and 20 others decreased in the AML group . Thus, tTo evaluate the classification performance of the top-ranked 50 genes, we performed PCA on reduced training and test sets of 50 genes selected above. The PCA score plot of the reduced training set showed that AML, ALL-B, and ALL-T were completely separated and localized to three regions . PCA of The results of cluster analysis showed that the classification performance of the test set comprised of top 50 genes was excellent, because only one sample was misclassified (#66) . This saWe next considered AML, ALL-B, and ALL-T as three parallel classes without subtypes to select characteristic genes for classifying disease. Therefore, we selected features for each class through the corresponding OPLS-DA models and S-plots. Three OPLS-DA models were fitted using training set of AML vs. ALL-B and ALL-T, ALL-B vs. AML and ALL-T, and ALL-T vs. AML and ALL-B . The parSeventeen top genes were selected from each OPLS-DA model using the S-plot . The numTo examine the rationality of the top 50 genes, we next performed cluster analysis on the training and test sets, respectively, using the Euclidean distance and complete linkage. The cluster tree for the training set of the new top 50 genes showed that all samples were assigned accurately into ALL-T, ALL-B, and AML, except for one sample (#17), which belongs to the ALL-B group but was incorrectly classified as AML . We fount-test using Golub’s dataset. The OVR-t-test was employed to select significant features for each class. Selection methods were described in To assess the performance of our approach, we compared the classification power of mOPLS-DA with that of t-test methods all performed well and our method outperformed them with no misclassification. When cluster analysis was used as the classifier, the performance of BW, S2N-OVR, KW, and the OVR-t-test became weaker, particularly for the SRBCT dataset, whereas the performance of mOPLS-DA was the same as that of SVM  using only one dataset, we evaluated our method using internal cross-validation  and external validation with an independent data set. Comparative results from Golub’s leukemia data and four public datasets showed that mOPLS-DA and S-plots worked better for selecting the most relevant genes for three-class classification and prediction than the other four feature selection methods. As suggested in t-test, which is a standard and widely used method for feature selection, outperformed the other gene-selection approaches, including BW, KW, and OVR-S2N. This finding is consistent with Haury’s report Golub’s leukemia data was used to evaluate the performances of Bayesian model averaging (BMA) When ALL-B, ALL-T, and AML were considered as three parallel classes, the clustering dendrogram revealed that the gene expression pattern of ALL-B was much closer to that of AML compared with ALL-T , which i10-transformation of data in preprocessing made the ranges of gene expression levels in an acceptable limit, the centering scaling was employed in PCA, OPLS-DA, and cluster analysis. Further, for pareto scaling, the classification accuracy of reduced training and test datasets was lower than that of centering scaling (data not shown).OPLS-DA is a scaling-dependent chemometrics approach. Three scalings are available for OPLS-DA such as centering, unit variance (UV), and pareto. Centering scaling increases the influence of variables with larger amplitudes on the model. UV  is the most commonly used alternative scaling technique that centered each variable and then divided by its standard deviation. The disadvantage of UV is that it often inflates the importance of noise, which may mask the variables of interest. Pareto scaling is a compromise between center and UV scaling. For generating S-plots from OPLS-DA, centering and pareto are available. Because logThe PLS and OPLS models are fitted using a strategy that extracts components from matrix X, which is different from traditional regression modeling that depends on covariance decomposition. The robustness of PLS and OPLS models are not affected by multi-colinearity of variables and not constrained by larger number of variables than that of the observations Figure S1Heatmaps of cluster analysis of top 51 genes selected by mOPLS-DA models using training set (A) and test set (B) of SRBCT dataset. Three classes including RMS, EMS and NB were used in current work. No observation was misclassified by cluster analysis of top51 genes selected by our proposed method. In original work, test sample 1, 8, 14, 16, 23 and 25 were diagnosed as NB; test 2, 6, 12, 19, 20 and 21 was diagnosed as EWS by histological examination; test 4, 10, 17, 22 and 24 belonged to RMS.(TIF)Click here for additional data file.Figure S2Plots of cluster analysis of reduced training set (A) and test set (B) consisting of top 51genes selected from 11_tumour dataset by mOPLS-DA models. No observations were misclassified in training set and wrongly predicted in test set.(TIF)Click here for additional data file.Figure S3Heatmaps from cluster analysis of reduced training set (A) and test set (B) of top 51 genes chose from dataset of Leukemia_2 by mOPLS-DA models. In training set, only one observation (MLL 17) was misclassified; all observations in independent test set were predicted correctly.(TIF)Click here for additional data file.Figure S4Cluster analysis plot from reduced training set (A) and test set (B) contained top 51 genes selected by mOPLS-DA models using 14_tumour dataset. From these two figures, we can see that all training samples were classified correctly and all test observations were predicted into each class without a mistake.(TIF)Click here for additional data file.


Meta-analysis of gene expression microarray datasets presents significant challenges for statistical analysis. We developed and validated a new bioinformatic method for the identification of genes upregulated in subsets of samples of a given tumour type (‘outlier genes’), a hallmark of potential oncogenes.in vivo by immunohistochemical staining data from clinical glioblastoma samples. Immunohistochemical data were available for 65% (19 of 29) of these genes, and 17 of these 19 genes (90%) showed a typical outlier staining pattern. Furthermore, raltitrexed, a specific inhibitor of TYMS used in the therapy of tumour types other than glioblastoma, also effectively blocked cell proliferation in glioblastoma cell lines, thus highlighting this outlier gene candidate as a potential therapeutic target.A new statistical method  was developed by modifying and adapting algorithms originally developed for statistical problems in economics. We compared the potential of the GTI to detect outlier genes in meta-datasets with four previously defined statistical methods, COPA, the OS statistic, the t-test and ORT, using simulated data. We demonstrated that the GTI performed equally well to existing methods in a single study simulation. Next, we evaluated the performance of the GTI in the analysis of combined Affymetrix gene expression data from several published studies covering 392 normal samples of tissue from the central nervous system, 74 astrocytomas, and 353 glioblastomas. According to the results, the GTI was better able than most of the previous methods to identify known oncogenic outlier genes. In addition, the GTI identified 29 novel outlier genes in glioblastomas, including TYMS and CDKN2A. The over-expression of these genes was validated Taken together, these results support the GTI as a novel approach to identify potential oncogene outliers and drug targets. The algorithm is implemented in an R package . Most of these studies have been based on relatively homogeneous datasets consisting of comparably small numbers of samples. However, when results from such individual studies are compared with each other, the overlap of the differentially expressed gene sets is often minimal and disappointing. In order to identify consistently differentially expressed genes based on robust statistics, it is advisable to systematically combine multiple public datasets. The power of this ‘meta-analysis’ strategy has been demonstrated in the case of ArrayExpress Out of the many statistical methods used for the identification of differentially expressed genes In general, outlier analysis offers a unique and powerful approach for the identification of key pathogenetic genes involved in a subset of disease samples. The strength of cancer outlier profile analysis was powerfully demonstrated by the identification of the TMPRSS2-ERG fusion oncogene in prostate cancers A gene showing over-expression in a subgroup of disease samples based on a cut-off threshold is defined as an outlier . Our aimCOPA and OS statistics were derived from the t-statistic by replacing the mean and standard errors used in the t-statistic with the median and median absolute deviations, respectively. ORT has been proposed as a more robust statistic that utilizes the absolute difference of each expression value from the median instead of the squared difference of each expression value from the average.In this study, we adapted an existing method from economics (the poverty index) with a comparable goal, addressing the question of how many people live below the poverty line in any given country, a formula developed for socioeconomic studies by Amartya Sen Let k,jThe formula for the standard unpaired t-statistic is:The two-sample t-test assumes that all disease samples for a particular gene are over-expressed. This assumption is not the case in cancer gene outlier analysis, where the genes are only assumed to be over-expressed in a subset of samples within the disease group that is assumed to be over-expressed th percentile of the disease samples' standardized expression values jmad and the constant 1.4826 is approximately equal to the standard error for normally distributed random variables.The COPA The approach used in the COPA statistic addresses the problem of more accurately identifying genes with an outlier population than the t-statistic.th percentile of the disease samples is The COPA statistic is described asjmed, the sample standard error sj by the median absolute deviation jmad, and the disease sample mean by the rth percentile Compared to the t-statistic, COPA intuitively replaces the normal sample mean by the all-sample median th sample percentile is almost equal to using information from a single sample.It is evident that the COPA statistic may not be very robust, since a fixed rth percentile with a sum over the outlier samples from the disease group above a given cut-off. The OS statistic was designed to lower the false discovery rate (FDR) of COPA, as noted by Wu ij) through dividing the result of (xij – medj) by madj. However, only expression values above a given cut-off are utilized for the final score.j is the set of outlier samples from the disease group defined by the following heuristic criterion:1,n1+1,....n.The outlier sums statistic was introduced as an improvement over the COPA statistic. Here, the OS statistic The outlier robust t-statisticThe median absolute deviation will then be estimated asHere, the average (avg) is replaced with the sample median, and the squared difference is replaced with the absolute difference as a more robust estimate of the variance.1.ORT is then described asIt should be noted that only the normal group samples are used to estimate outliers when calculating an ORT score.The GTI algorithm was originally used to calculate an index for a single treatment group k based on a standard cut-off. In this study, there was no defined standard cut-off per gene j for every normal tissue. Therefore, we defined a cut-off (B) based on the expression of gene j among all samples (n). These samples were obtained from one body part or tissue type such as the breast of normal (k = 1) and cancer-affected (k = 2) individuals. We then asked whether the proportion of samples above the cut-off is larger than it should be. Our choice of B is the standard statistical outlier cut-off (q75+IQR). We propose the following score, which weighs the proportion of outliers by a robust measure of how outlying the outliers are in a single group:We writeExpanding the definition of the GTI and substituting our choice of B, we getWe calculate the actual GTI scores for each group k and gene j multiplied by 100, as this makes them more readable.Finally, the index per gene is a direct association between two groups defined by The pre-processed Affymetrix transcriptome data utilized in this study were derived from the GeneSapiens database Human glioblastoma cell lines A172 and U87-MG were obtained from the ECACC , the LN-405 cell line was obtained from DSMZ , and the U373-MG and astroglia SVG p12 cell lines from the ATCC . The A172, LN-405 and U373-MG cell lines were cultured in DMEM with 4500 mg/L glucose, 10% FBS, 2 mM L-glutamine and penicillin-streptomycin. U373-MG cells were supplemented with 300 ng/mL hygromycin. The U87-MG and SVGp12 cell lines were cultured in EMEM with 2 mM L-glutamine, 1 mM sodium pyruvate, 0.1 mM non-essential amino acids, 1.5 g/L sodium bicarbonate, penicillin-streptomycin and 10% FBS.The antifolate drugs used in EC50 determinations were 5-fluorouracil , gemcitabine  and raltitrexed .The human glioblastoma cell lines A172 and U87-MG were obtained from the ECACC , LN-405 cell line was obtained from DSMZ , and the U373-MG and SVG p12 cell lines were purchased from the ATCC . The A172, LN-405 and U373-MG cell lines were cultured in DMEM with 4500 mg/L glucose, 10% FBS, 2 mM L-glutamine and penicillin-streptomycin. U373-MG cells were supplemented with 300 ng/mL hygromycin. The U87-MG and SVGp12 cell lines were cultured in EMEM with 2 mM L-glutamine, 1 mM sodium pyruvate, 0.1 mM non-essential amino acids, 1.5 g/L sodium bicarbonate, penicillin-streptomycin and 10% FBS. The antifolate drugs used in EC50 determinations were 5-fluorouracil , gemcitabine  and raltitrexed .The suitability of existing outlier methods for the analysis of large-scale multi-study datasets is not only measured by the absolute statistical quality of the results obtained in theoretical settings, but strongly depends on a number of technological and practical issues. In this respect, it is mandatory to meticulously test whether such methods can be used for the analysis of extremely large-scale integrated microarray datasets. Currently, multi-study datasets such as those collected in GeneSapiens, or the clinical data sets provided by large-scale international cancer profiling consortia such as TCGA or ICGC, easily contain hundreds to several thousands of samples. The tendency towards large data sets will further increase with the progress of these integrated approaches, and with the introduction of next-generation genome sequencing technologies in cancer research. Some existing outlier methods may not be suitable to handle matrices with very many sample numbers, in particular if the data points available within these sets vary gene by gene. Even given the suitability of certain statistical approaches for successful outlier identification, the process may be exceedingly slow and may not be suitable for repeated application after every update to a database. For these reasons, we decided to evaluate the suitability of the GTI and the existing methods for identifying outliers in complex multi-study datasets.n(1) = n(2) = 30), in which all expression values were drawn from a standard normal distribution. Next, we generated expression values for a gene assumed to be differentially expressed by adding a constant, m, to the expression values in only the first k cancer samples , where k equals the number of outlier samples, and used this value as the true positive (TP). The true positive and false positive (FP) values were calculated based on 50 simulations.First, to compare the GTI method with previously described outlier identification methods, we conducted simulation studies using ORT, OS, and COPA methods with a fixed statistical outlier cut-off (q75+IQR). In addition, the t-statistic was considered; however, this method did not require any cut-off selection. For the simulation, an artificial dataset was generated representing 1000 genes assuming an equal number of normal and cancer samples  to simulate how the five statistical methods would perform in these different artificial cases. This procedure was repeated for each method as well as each dataset simulated with a varying value of k. The data from the simulations were used to calculate the true positive rate based on a given false positive threshold. These results are summarized in The results presented in The most interesting case of having k = 10 (10 outliers in 30 samples) showed that all methods were equally suited to identify the outliers. To visualize our approach, Importantly, all of the simulations used to derive these results were carried out in a single study setting. However, multi-study data sets are becoming increasingly common and powerful resources, and it would be practical to test the applicability of these methods to multi-study datasets. To achieve this goal, our systematic analysis was extended by using pre-processed data from the GeneSapiens database.The GTI and the ORT, OS, and COPA methods were then tested with publicly available microarray datasets derived from central nervous system (CNS) tissues and tumours. We chose CNS tissue samples because there is an enormous wealth of data on glioblastoma in public repositories such as the Cancer Genome Atlas (TCGA) and GEO. Gliomas make up a group of primary CNS tumours that arise from glial cells. We focused on two subgroups of gliomas, anaplastic astrocytoma (WHO grade III) (74 samples) and glioblastoma multiforme  (353 samples). We used Affymetrix microarray data from healthy CNS tissues (392 samples) as a reference. The overlap of the top 100 genes identified by the GTI, COPA and the OS statistic is presented in a Venn diagram in More specifically, we examined genes previously known to be over-expressed in and/or associated with the development, maintenance, and progression of glioblastoma multiforme and anaplastic astrocytoma and that were identified in the top 100 by any of the methods, as illustrated in www.proteinatlas.org] (HPA) confirmed that GFAP is expressed in all normal brain as well as brain cancer samples, and hence cannot be considered a ‘cancer outlier’ (data not shown). Briefly, and as illustrated by the GFAP mRNA expression profile, although not the only example, a high COPA score did not necessarily imply a significant outlier expression difference between the normal and cancer groups  had been previously linked to glioblastoma in two or more publications . SimilarIn contrast to the 29 unique GTI-identified candidates including TYMS, cyclin-dependent kinase inhibitor 2A, isoform 4/p16 (CDKN2A), was flagged by most of the methods . This should therefore represent a cancer-relevant gene that is over-expressed in subsets of many different cancer types, but is not particularly specific for glioblastoma. For both CDKN2A and TYMS, immunohistochemical staining images were available from the HPA TYMS is a known target of many antifolate drugs, such as 5′-fluorouracil (5FU) and gemcitabine. These drugs have been tested for the treatment of GBM patients, but have offered little general advantage over other treatment options Assuming that the concentration of the target gene would correlate with the sensitivity of cancer cells against the specific drug, we determined the median effective concentration (EC50) values for these three anti-folate drugs in cytotoxicity/cell proliferation assays with four established GBM cell lines and an immortalized foetal astrocyte cell line, SVGp12. Gemcitabine and the TYMS-specific raltitrexed showed EC50 values on the nM scale  against Taken together, the results suggest that the detection of gene outliers with the GTI may provide complementary information on potential oncogenic genes in tumour tissue samples, and that these genes may be biologically relevant and associated with tumour progression.In summary, we have shown how our newly-developed algorithm can be successfully utilized for the robust detection of outlier gene expression in cancers. The GTI performs particularly well with heterogeneous large-scale datasets that contain either variable numbers of samples between the groups being compared or variable numbers of informative samples for different genes. The GTI requires gene expression data from both disease tissues and corresponding reference samples, and approximates the proportion of the outlying population for each gene. Our results demonstrated that the GTI provided valuable complementary information when compared to existing analytical methods. This was demonstrated by the identification of 29 genes with outlier expression profiles not previously linked to glioblastoma. False positives, for example genes with high expression in both normal and cancer tissues, such as GFAP, When the proportion of outliers among the disease samples becomes very small, all existing methods become increasingly insensitive to the presence of small outlier populations. As demonstrated by Wu et al. (2007), COPA and the OS statistic use only cancer group samples to determine the rth percentile that is used as a cut-off to define outliers. Naturally, both of these methods therefore lead to the false-positive identification of general tissue-specificity marker genes such as GFAP, which also show high expression in the non-malignant, normal reference group. The approach of the GTI to utilize the normal control group for the calculation of outlier statistics therefore effectively reduces the number of false positives identified. Moreover, if the number of cancer samples is considerably larger than the number of normal samples , COPA and OS and ORT statistics often show a significant increase in false positives . The ORTWe specifically explored the performance of COPA, OS, ORT, the t-statistic and the GTI in relation to the number of samples that were informative for a given gene. Importantly, the total number of informative samples introduces a strong bias in differential gene expression analysis, in particular among multi-study microarray datasets We validated the genes highlighted by the GTI using data from immunohistochemistry and explored their value as drug targets in glioblastoma cell lines. TYMS is a gene encoding thymidylate synthetase, which affects the production of nucleotide building-blocks used in DNA synthesis and repair. Thus, it is likely that fast growing and genetically unstable cancer cells are more dependent on TYMS function than normal cells. The immunhistochemical staining of the corresponding protein in HPA In order to better analyze heterogeneous large-scale microarray datasets, we developed the gene tissue index (GTI) as a new robust method for detecting cancer gene outliers. The GTI was used to examine over-expression and under-expression in cancer tissues as compared to a reference group. Our simulation results demonstrated that the GTI performs well as an alternative method for identifying outliers using single study datasets. Using a large-scale glioblastoma clinical dataset originating from multiple laboratories, the GTI identified more oncogene outliers than all of the previously described methods. Compared with most existing methods for outlier analysis, the GTI can uncover outliers by determining a cut-off based on all samples  and standardizing the score to achieve better comparability across genes. Most top-ranking genes according to the GTI showed a significant association with cancer.Figure S1Expression profiles of GTI top ranking genes from the simulation study. Plots of expression values in each group, for 12 genes ranked highest by the GTI statistic. Points have been jittered in the vertical direction for clear viewing. The blue colour refers to the normal and red to the cancer group. In the first plot with a GTI score of 6.906, it can be seen that the cancer group has a sub-population with expression above 9.(PDF)Click here for additional data file.Table S1Single study simulations data used to construct the receiver operating characteristic (ROC) curves.(XLS)Click here for additional data file.Table S2Twenty nine genes uniquely identified by GTI among the top 100 that were further followed up in this study.(XLS)Click here for additional data file.Text S1R programming code of the GTI algorithm introduced in this study used to identify cancer outliers.(TXT)Click here for additional data file.