A number of publications have reported the use of microarray technology to identify gene expression signatures to infer mechanisms and pathways associated with systemic lupus erythematosus (SLE) in human peripheral blood mononuclear cells. However, meta-analysis approaches with microarray data have not been well-explored in SLE.In this study, a pathway-based meta-analysis was applied to four independent gene expression oligonucleotide microarray data sets to identify gene expression signatures for SLE, and these data sets were confirmed by a fifth independent data set.Differentially expressed genes (DEGs) were identified in each data set by comparing expression microarray data from control samples and SLE samples. Using Ingenuity Pathway Analysis software, pathways associated with the DEGs were identified in each of the four data sets. Using the leave one data set out pathway-based meta-analysis approach, a 37-gene metasignature was identified. This SLE metasignature clearly distinguished SLE patients from controls as observed by unsupervised learning methods. The final confirmation of the metasignature was achieved by applying the metasignature to a fifth independent data set.The novel pathway-based meta-analysis approach proved to be a useful technique for grouping disparate microarray data sets. This technique allowed for validated conclusions to be drawn across four different data sets and confirmed by an independent fifth data set. The metasignature and pathways identified by using this approach may serve as a source for identifying therapeutic targets for SLE and may possibly be used for diagnostic and monitoring purposes. Moreover, the meta-analysis approach provides a simple, intuitive solution for combining disparate microarray data sets to identify a strong metasignature.http://genomemedicine.com/content/3/5/30Please see Research Highlight:  Microarrays are powerful tools with capability of measuring the transcript abundance of tens of thousands of genes simultaneously in biological samples. Microarray technology has matured over the past 15 years and is now employed for the study of gene expression signatures associated with disease . The cliSome of the challenges associated with identification of gene expression signatures that differentiate the disease state from healthy controls are the availability of samples, sample size, heterogeneous data sets, and reproducibility. Thus, robustness of the gene expression signature derived from one study needs to be validated by other independent studies, preferably with large sample sizes. In practice, however, several studies with relatively small sample sizes are often used to identify gene expression signatures. In these circumstances, it is beneficial to combine the results of several individual studies using meta-analysis. This process enhances statistical power in identifying more robust and reliable gene signatures.et al. [P values of genes across studies to identify gene expression signatures that differentiate cancer tissues from normal tissues and to predict poor or good patient outcomes. Choi et al. [et al. [et al. [Several meta-analysis approaches have been proposed specifically for handling heterogeneous data sets. For example, Rhodes et al.  used thei et al.  used an  [et al. , who per [et al.  suggesteP value and fold change and the results are combined at the pathway level instead of at the gene level  from individual studies are selected using a combination of A number of authors have reported the use of microarray technology to identify gene expression signatures in systemic lupus erythematosus (SLE) -14, mechData sets from five independent microarray studies comparing PBMC samples from SLE patients with those from healthy individuals were obtained from prominent SLE researchers. These data sets are referred to as data sets 1, 2, 3, 4, and 5. Data sets 1, 2, 4, and 5 are associated with peer-reviewed publications (11-14). Data set 3 is composed of unpublished data. Three of the studies  included only pediatric patients, while the remaining two included only adults. All studies employed the Affymetrix GeneChipmicroarray platform  but the versions of the array type varied  for the initial analysis.Quality assessment was done for each data set using the Genedata Expressionist   Figure , step 1.t test was performed on each data set individually. The P value and fold change filters  were used to identify a unique list of DEGs from each data set  in each data set  . ArrayTrPathways common to all of the data sets were identified from the individual lists of pathways enriched in SLE patients compared to healthy controls (one for each data set) Figure , step 4.To validate this technique, a leave one data set out permutation approach was employed Figure , step 7.Gene markers were generated for each of the four scenarios as described in the Meta-analysis section. Gene markers present in at least three of the four scenarios were grouped to comprise a 37-gene metasignature.Confirmation of the final 37-gene metasignature was done using an independent fifth data set, data set 5 Figure , step 8.P value cutoff of 0.01 and a fold change cutoff of 1.5  as discussed in the section above.A separate analysis of the pediatric and adult data sets used to identify DEGs and pathways in the two populations was conducted. Similar gene expression patterns were observed in adult and pediatric populations, although the extent of upregulation of some of the genes was higher in the pediatric data sets (unpublished results).For each scenario, the signature obtained using the three data sets was applied to the fourth data set to observe how effectively the expression of the signature genes could distinguish between the SLE and healthy populations.The PCAs and HCAs obtained for each scenario are presented in Figures The Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) is a validated scoring system that can be used to describe the range of disease activity and comprises a weighted score calculated by the presence or absence of 24 symptoms. The association of SLEDAI scores to expression profiles of SLE patients was evaluated. While the majority of the samples were grouped into their respective classes  were also among the DEGs in SLE patients. Differential expression of many other biomarkers associated with inflammatory and/or immune responses and with cellular proliferation was also observed, as shown in Table A 37-gene signature was generated by the meta-analysis workflow Table . Many IFications ,26,35-37This signature was applied to an independent fifth data set (data set 5) to evaluate its ability to distinguish the SLE samples from the control samples. Figure The novel pathway-based meta-analysis approach proved to be a useful technique for grouping disparate microarray data sets. This technique allowed for validated conclusions to be drawn across four different data sets and confirmed by testing on an independent fifth data set. Since the metasignature was obtained from pathways that were enriched in SLE samples across all of the data sets, it is highly representative of biological pathways related to SLE. The metasignature produced here may serve as a source for identifying potential therapeutic targets for SLE. Also, with further refinement, it might be made clinically more intuitive, which may also prove to be useful for the diagnosis and monitoring of SLE. Moreover, the meta-analysis approach outlined here provides a simple and intuitive solution for combining disparate microarray data sets to identify a robust metasignature.The authors declare that they have no competing interests.DA created the first draft of the manuscript and performed the meta-analysis. PM and SA performed the analysis to confirm the metasignature in an independent data set. SA, PM, and HF helped significantly to draft the manuscript. SA and WT helped coordinate the project and finalize the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1741-7015/9/65/prepub


Recent studies suggest that gene expression profiles are a promising alternative for clinical cancer classification. One major problem in applying DNA microarrays for classification is the dimension of obtained data sets. In this paper we propose a multiclass gene selection method based on Partial Least Squares (PLS) for selecting genes for classification. The new idea is to solve multiclass selection problem with the PLS method and decomposition to a set of two-class sub-problems: one versus rest (OvR) and one versus one (OvO). We use OvR and OvO two-class decomposition for other recently published gene selection method. Ranked gene lists are highly unstable in the sense that a small change of the data set often leads to big changes in the obtained ordered lists. In this paper, we take a look at the assessment of stability of the proposed methods. We use the linear support vector machines (SVM) technique in different variants: one versus one, one versus rest, multiclass SVM (MSVM) and the linear discriminant analysis (LDA) as a classifier. We use balanced bootstrap to estimate the prediction error and to test the variability of the obtained ordered lists.This paper focuses on effective identification of informative genes. As a result, a new strategy to find a small subset of significant genes is designed. Our results on real multiclass cancer data show that our method has a very high accuracy rate for different combinations of classification methods, giving concurrently very stable feature rankings.This paper shows that the proposed strategies can improve the performance of selected gene sets substantially. OvR and OvO techniques applied to existing gene selection methods improve results as well. The presented method allows to obtain a more reliable classifier with less classifier error. In the same time the method generates more stable ordered feature lists in comparison with existing methods.This article was reviewed by Prof Marek Kimmel, Dr Hans Binder (nominated by Dr Tomasz Lipniacki) and Dr Yuriy Gusev Recent studies suggest that gene expression profiles may represent a promising alternative for clinical cancer classification. Molecular-based approaches have opened the possibility of investigating the activity of thousands of genes simultaneously and can be used to find genes involved in neoplasia. A well known problem in applying microarrays in classification problem is dimension of obtained datasets. In work  authors The standard way to use the PLS algorithm is for feature extraction only and not for selecting significant features. Here, we use this method for gene ranking. In  it has bWe propose a new method based on decomposition of a multiclass feature selection problem into a set of two-class problems that are used in one versus rest (OvR) and one versus one (OvO) techniques.An important aspect of feature selection methods is the stability of obtained ordered lists ,12. In [In literature  most of l — number of samples; m — number of genes; g — number of selected genes; L — list of selected genes; K — number of classes; B — number of bootstrap samples; PLS — Partial least squares regression method; s — number of PLS components; w — PLS weight vector; SIMPLS, NIPALS — names of two used PLS algorithms; PLS+MCLASS — PLS based multiclass gene selection method with ‘all classes at once’ approach; PLS+OvO — PLS based multiclass gene selection method with ‘one versus one’ decomposition; PLS+OvR — PLS based multiclass gene selection method with ‘one versus rest’ decomposition; GS — gene selection method proposed in [s1, s2 — stability score indicators; SVM OvO, SVM OvR, MSVM — support vector machines based classification methods; LDA — linear discriminant analysis classification method;posed in ; RFE — RWe use bootstrap technique  which hal, where X= is the input matrix and Y= is the response (class labels). For multiclass problem yi∈{1,2,…,K}, where K is the number of classes. The bootstrap sample is a random sample with replacement of the observations and has the same size as our original dataset. The probes that appear in a bootstrap sample constitute a training dataset. The rest of observations is used as a test dataset. This is done B times to produce B bootstrap samples. To divide our samples into training and test datasets we use the balanced bootstrap method [B times in the collection of B bootstrap samples. This does not mean that all samples should occur in every bootstrap sample, because the first observation can occur for example twice in the first bootstrap sample and not at all in the second. We can do this by constructing a set with B copies of all l observations and then permuting the obtained set. Every l-element successive subset is one bootstrap sample.Let us consider a dataset of size p method ,23. The B. We must find the compromise between analysis time and accuracy of predicted parameters. In our cases we use 500 resampling iterations of all stages of the classifier construction . We did not observed significant changes in the results for all used datasets, after increasing the number of iterations. Of course, the necessary iterations number can change after changing the dataset and depends especially on the number of probes. We generated 500 bootstrap samples only once to reduce the variability of results for all tested methods. The distribution of the misclassification rate obtained during all bootstrap runs was used to estimate the 95% confidence interval. The accuracy of the classifier and the confidence interval were calculated for subsets of first genes on the lists up to 30 genes.The bootstrap resampling is computationally costly. We implemented it on a computer cluster using the MatlabMPI toolbox for parallel computation. The most important parameter for the bootstrap resampling technique is the number of resampling iterations To estimate the prediction error (accuracy) we used the.632+ estimator . The.632w2 from the first PLS component is of the same order as from F-statistics. It is equivalent to the t-test with equal variance and also with the BSS/WSS-statistics, where BSS denotes the between-group sum of squares and WSS the within-group sum of squares. In our comparison we did not scale the data to unit variance, but only centered the data. Boulesteix and Strimmer [In this section we propose a new method for selecting the most significant genes. It is based on partial least squares regression (PLSR) . There aStrimmer  describeStrimmer , which iIn this paper we use the weight vector obtained from the PLS method to select the most important genes.X of size l×m with l probes and m genes we denote the l×1 vector of response value y. The PLS components tii=1,…,sare constructed to maximize the objective criterion based on the sample covariance between y and linear combination of genes (PLS components) t=Xw. We search the weight vector wsequentially, to satisfy the following criterion PLS aims at finding uncorrelated linear transformations of the original input features which have high covariance with the response features. Based on these latent components, PLS predicts response features (the task of regression) and reconstructs an original dataset matrix (the task of data modeling) at the same time. For dataset matrix subject to the orthogonality constraint Xw) Cor2(XwY). To derive components (named “latent variables” or scores), ti, the PLS decomposes Xand y to produce a bilinear representation of the data [This criterion is the mostly used in literature as general description for PLS method. In case of multiclass categorical data this criterion can be simplified as mentioned in  and maxithe data pi are loadings, qi are scalars and Ef are residuals. The idea of PLS is to estimate loadings and scores by a regression. The PLS fits a sequence of bilinear models by least squares. At every step i vector wi is estimated to obtain the PLS component that has maximal sample covariance with the response variable y. Each component tiis uncorrelated with all previously constructed components. There are two main PLS algorithms described in literature: NIPALS algorithm [ticomponents are calculated explicitly as linear combinations of X and second, X is not deflated in each iteration. The SIMPLS algorithm will be assessed in accordance with the criteria eq. (1). In NIPALS the first PLS component t1 is obtained on the basis of the covariance between Xand y, and is qual to the first component of SIMPLS algorithm. Component ti, is computed using the residuals of Xand y from the previous step, which account for the variations left by the previous components. Maximal number of components s is equal to the rank of X.where lgorithm  and SIMPlgorithm . The SIMr is applied to the original X matrix and De Jong [rdirectly from the NIPALS algorithm As we say before the weight vector from SIMPLS algorithm sometimes referred to  De Jong  showed tpi are the loading and wiare the weight vector for i-th component of NIPALS algorithm.where ti for NIPALS and SIMPLS algorithms are the same. In contrast to score vectors, the weight vectors wi and ri for NIPALS and SIMPLS respectively are different for i>1. This phenomenon is a consequence of different method to compute the weights vectors. The wivectors in NIPALS procedure are calculated with deflated data matrices Xi and Yi in each iteration, and the weights riare obtained without the deflation step in SIMPLS algorithm. For this reason in this paper, we use the weight vectors w and rfrom both algorithms to determine the ranked list. In our method the sum of the s PLS components presents the gene importance vector and the “best genes” have the highest values in this vector. First g genes with the highest value in the gene importance vector are selected for the classifier. To test the optimal number of components we use the first squared weight vector and the sum of squared weight vectors from first 5 and 10 components. The standard way to use PLS for a multiclass data, is to search for the best direction for maximization of the covariance between responses with all classes and linear combination of genes. As we mentioned before, we compare our method based on decomposition of a multi-class feature selection problem into a set of two-class problems with a well known ‘all classes at once’ technique. For each two-class selections “best genes” are selected and one ranked gene list is constructed as follows: genes with the highest weight in all two-class selections are located at the top of the list, then genes with the second highest weights, and so on. We must underline, that yfor two-class selections is coded as a vector with value 1 for the first class and −1 for the second class. For the ‘all classes at once’ technique yis a matrix with N rows and the number of columns is equal to the number of classes. In each row a class label has a value of 1 and −1. For our needs we introduce the notation PLS+MCLASS for ‘all classes at once’ technique and similarly PLS+OvO, PLS+OvR for two-class decomposition of the multiclass feature selection problem. On the Figure De Jong proved in  that forg genes. In this framework we consider the list L with first g top-genes obtained from the entire dataset and lists Lb;b=1,2,…,Bobtained from every b of B bootstrap iterations.In ,41 authoIn this paper we assess stability in two ways. The first one is to calculate stability indices. In this case we have used Percentage of Overlapping Genes (POG) criterion  and modiThe second one is to visualize how obtained gene lists are stable by looking at descriptive plots. In the next section we introduce the detailed description of the stability plots used in this article.L against ranks in all b bootstrap iteration lists Lb;b=1,2,…,B. We set the limit to determine which points are extreme to the rank out of the g gene list. Another way to visualize gene lists stability is to plot a so-called Bootstrap Based Feature Ranking (BBFR) plot [s1and s2, is calculated separately for each gene. The BBFR score for the gene number j is defined as To visualize the stability of the ordered gene lists we plot the boxplots of rank for each gene in the list FR) plot . The BBFrbj is the rank of the j-th gene in b-th bootstrap iteration where rbj=g.for the top-scored gene Qjscore is 1. It means that one gene was top-ranked in all B bootstrap iterations. The score Qj takes into account the rank rbjof j-th gene in all B bootstrap iterations.The maximum possible value of the j in the lists Lb;b=1,2,…,BThe modified BBFR score The maximum possible value of the Qjand g genes and 0 for the rest.Both http://www.broadinstitute.org/mpr/lung/. The second is the MLL dataset published by [http://www.broadinstitute.org/cgi-bin/cancer/publications/pub_paper.cgi?mode=view&paper_id=63. The third is the SRBCT dataset published by [http://www.biomedcentral.com/content/supplementary/1471-2105-7-228-s4.tgz[In our study we chose three publicly available multiclass microarray datasets. The first is the LUNG dataset published by . It consished by . It consished by . Each saished by . It consWe chose three multiclass microarray datasets (detailed described in the Datasets section) for our experiments.r vector calculated from SIMPLS method was better than the vector wcalculated from NIPALS algorithm for more than one component. Only for PLS+MCLASS method the accuracy rate is higher when we use more than 1 PLS component. In our study we also applied a method searching for the optimal number of components based on leave one out classification error on training samples and the SVM classifier (results not showed here). In general the results for classification accuracy rate were not significantly better and in some cases even worse. In all tables we bolded the best accuracy rate for tested classification methods and variants of PLS method . In the last columns we show the standard deviations values for the best classifier. The comparison of accuracy rate and stability index s2for all tested datasets proves the advantage of the PLS method  is computed for a list of 30 genes. The BBFR ranking Figure  and the In this paper we proposed a new PLS-based method to select significant genes. Our results have shown that this gene selection method gives very good accuracy rate and stability of obtained gene lists. The principal of PLS is based on the maximization of the covariance criterion, which can lead to good generalization ability and stability. In our opinion this is a reason of the good results obtained with PLS method. Another important result is the fact that it is more effective to solve a multiclass feature selection by splitting it into a set of two-class problems and merging the results in one gene list. The explanation for these result can be the difference between used methods: the idea of MCLASS approach is to look for genes able to distinguish between all classes simultaneously. Such genes are more difficult to find, and they can have smaller discriminatory power. This problem do not exist in the decomposed multiclass problem for OvO and OvR approaches. From the methodological side we suppose, that the MCLASS multiclass feature selection methods are not so good developed, as the 2-class methods, and this fact can be the explanation for our results. The comparison to other feature selection methods shows that the gene lists stability index is the highest for PLS with OvR and OvO techniques. In two cases the stability index is slightly better for PLS+MCLASS method with one PLS component, but the accuracy rate for this method is significantly worse. All other methods indicated much worse stability of obtained gene lists. We can observe that using the GS method with 2-class decomposition technique improves the accuracy rate and with two of the datasets gene list stability increased as well. Another advantage of the 2-class decomposition technique for gene selection methods is easy interpretation of the results by biologists. In all cases the ‘all classes at once’ technique of PLS and GS methods achieves worse classification accuracy than their 2-class versions. The presented method makes it possible to obtain more stable gene selection lists and a more reliable classifier with less classifier error. We show that accuracy rate assessing accompanied with the gene stability analysis gives more reliable evaluation of various gene selection methods. Of course our methods can be applicable also to other high dimensional data where we consider classification problems such as protein microarrays, DNA copy number variation, exome profiling and RNAseq. In all cases, where the dataset has much more features than observations it is recommended to take into consideration the accuracy rate, but also the stability score.In general there are two different approaches to measure the stability of gene lists. The first approach takes into account only the content of gene lists, and ignores the gene order. The second one does not ignore the gene order on compared lists.L1 and L2 of the size g. Let k be the size of the intersection of L1and L2. Then POG is defined as s1=k/g.One of the most frequently used criteria is the Percentage of Overlapping Genes (POG) , belongiL and lists Lb;b=1,2,…,B. Let uj be the placement of the j-th gene in the list L. For the top-scored gene uj=1. Similarly, ubj is the placement of the j-th gene in the list Lb. The POG is calculated as POG criterion may be extended in such a way that it measures the similarity between the list I denotes the indicator function where s2score to estimate the similarity between all lists We introduce the modified relative s1, it does not ignore the rank of the selected genes within the considered subset, hence it belongs to the second mentioned class of stability measures. The value for the gene that is out of Lb is set to g + 1. The value of functions s1 and s2 is scaled to the interval 〈0,1〉 and the higher value indicates better stability of the obtained gene list.In opposite to previous indicator g and the worst is Max where G is the number of all genes. This approach is equal to the number of genes with a non-zero score in the Bootstrap Based Feature Ranking (BBFR) (described in the next section).Another indicator used to estimate the stability of an obtained gene list is given by the number of genes that were selected at least one time in all bootstrap samples. The best value is Y and Yequals predicted value To estimate the prediction error we use the.632+ estimator . First wxiof the whole dataset. This predictor can make overfitted predictions and the estimated error rate will be downward biased. It demonstrates why we obtain error estimator for test data sets in the form where B sets Cb containing samples that do not appear in b-th bootstrap sample and |Cb| is a number of such samples. This estimator will overestimate the true prediction error, and when the test set is small it can have high variance [γto be the error rate of our prediction rule if the inputs and class labels are independent. Let yiequal k and let k, where k is the class label of K class. Then The model trained on a training set will be tested on other samples and not used to fit the model. This provides protection against overfitting. As we have mentioned before, we compute the error rate for variance . To resoThe relative overfitting rate is Now we can define the.632+ estimator by When there is no overfitting problem the.632+ estimator is equal to the.632 estimator Table Table Table s2(bar chart) and accuracy of classification (dot chart) with the 95% confidence interval of the best classifier on the tested feature selection methods for MLL data.Figure s2(bar chart) and accuracy of classification (dot chart) with the 95% confidence interval of the best classifier on the tested feature selection methods for SRBCT data.Figure Figure Figure Figure Figure Figure Figure As the authors state, “gene expression profiles are a promising alternative for clinical cancer classification”. The well-known difficulty is the large dimension of the vector of data, compared to the usually modest number of independent data replicates. The authors propose a new, arguably better combination of known methods to face the classification problem. This is important; however, the most interesting problem tackled in the paper in a novel way is that of stability. Ranked gene lists can be unstable in the sense that a small change of the data set leads to serious changes in the resulting ordered lists. The authors address this issue by comparing how different methods yield different stability of results. Eventually, they find a new strategy to find a small subset of significant genes giving very stable feature rankings compared to currently employed methods. The paper seems interesting and suitable for Biology Direct. On the editorial side, some language usages are uncommon and therefore not clearly understandable, such as for example “invariability” which might mean “invariance” or “absence of variability”. I suggest using Oxford English Dictionary Online or a similar source to rectify these ambiguities (or employing a human text editor fluent in scientific English).Needs some language corrections before being publishedAuthor’s responseWe have edited the text and corrected the paper’s language mistakes.The manuscript Stable feature selection and classification algorithms for multiclass microarray data by Sebastian Student and Krzysztof Fujarewicz presents a new feature selection and multi-classification algorithm based on Partial Least Squares and decomposition into separate two-class problems. The authors clearly show that their method outperforms a series of state-of-the-art methods using appropriate benchmarks. The issue addressed is very important for the analysis of high-dimensional data and interesting for a broader readership as addressed by BD. Referencing and relation to state-of-the art is given appropriately. The method presented is novel, original and sound and obviously improves available solutions. Presentation, however, in general is suboptimal and requires revision. Particularly, I suggest the following points: 1. A large number of abbreviations are used and the reader gets completely lost in this jungle. I suggest to add a glossary which decodes and partly explains all abbreviations used, especially the different variants of methods used.Author’s responseWe have added short subsection in theMethodsMethods section named: Symbols and abbreviations.2. The methodical part mixes basal points (e.g. how works PLS) with more peripheral ones (e.g. different benchmarking criteria such as stability plots etc.). The reader is overloaded with algorithmic details and formulae. The latter points are of course also important but many things become clear always on an intuitive level. I suggest to remove all non-essential details  from the methodical part and to shift them into an appendix or supplementary text. The basal idea for benchmarks can be given in the methodical part very shortly in prosaic form .Author’s responseAccording to the suggestion we shift the part of the benchmark criteria into theAppendixsection.3. In my opinion, the methodical part should focus on the kernel of the new method, i.e. PLS and the decomposition into two-class comparisons and comparison with state of the art. Partly this information is given but mostly hidden in a heap of other things (see point 2.). A schematic figure that explains the essentials and novel aspects of the method and also visualizes the workflow might be very helpful. Possibly this scheme might visualize also differences with respect to other approaches. This point represents a real challenge but possibly the authors can solve it.Author’s responseWe have added a new figure with a scheme that explains the gene selection method based on PLS.4. The authors used 3 data sets for verification and 4 types of presentation which provides 3x4=12 figures at the end. This broad data basis allows proper verification of the methods. However, the results of benchmarking of the three different data sets are mostly similar if not identical with respect to the benchmark criteria applied. Here the reader is overloaded with very similar figures with mostly redundant information content. I suggest removing 2/3 of the figures into a supplementary file and to show only one of each type in the main paper.Author’s responseWe have moved figures for MLL and SRBCT datasets into aAppendixsection.5. In exceptional cases the results for the different data sets slightly differ .Author’s responseWe have moved these data sets description into theDatasetssection.7. Main point: The benchmarking demonstrates that the PLS-variants used outperform the other methods. It would be desirable to understand the underlying principal reason for this difference and to generalize this finding. In the Conclusions section this question is shortly addressed. However, this issue, in my opinion, requires much more attention beyond all the benchmarking details. Obviously the decomposition of the multiclass problem into a series of two-class problems is more favorable than to solve the multiclass-problem at once. What is the deeper reason that causes this benefit. On the other hand: why, for example, simple t-testing performs worst. I strongly encourage the authors to extend the paper in this respect.Author’s responseWe have extended theConclusionssection and have discussed these questions, but we must agree, that it is hard to find the real explanation for our findings, especially, because multiclass problems are more complicated, than the two class problems.8. The authors should provide a computer program of their approach along with the paper that might be used by others.Author’s responseBecause bootstrap technique is computationally expensive, we apply our software on the computer cluster, which make it very difficult to publish. The main problem is that our software is not dedicated for personal computers, and for that reason we decided not to publish this code.Further minor points: 9. Both axes in all figures must be assigned. I.e. the y-axes must be labeled in Figures Author’s responseWe have edited the manuscript and corrected these mistakes.AcceptableGeneral comments: The manuscript addresses one of the important problems in gene expression analysis i.e. feature selection for multiclass classicization of microarray data in cancer. While this problem has been investigated by many over past 10 years or so, the importance of utilization of gene expression data for classification of cancer samples remains high. This is mainly because of several potentially important practical applications in cancer diagnostics and prediction of drug response. Also – practical utilization of gene expression signatures has been questioned by many due to the known problems of reproducibility and validation. This paper addresses some of these issues by detailed analysis of stability of existing most popular classification algorithms as well as new method proposed by the authors. This study might have other important implications as it could be applicable to other types of global molecular profiling that are becoming more popular in recent years such as DNA copy number variation, exome profiling and RNAseq. The paper could benefit from additional discussion of this issue of applicability of the proposed methods for other types of omics data such as RNAseq and CNV.Author’s responseWe have added in theConclusionssection the information about other possible applications of the presented feature selection methods.Strengths and weaknesses: This study has several strengths: a comparison of performance of many existing classification methods both in term of accuracy and stability of feature selection for the multiclass analysis. Also – this work is focused on developing of new methodology of effective identification of the most informative genes with main goal of finding a small subset of most accurate features. The authors for the first time have demonstrated effectiveness of decomposition of mutli-class classification problem into series of sub-problem of two-class selection. The important part of this work was applying these methods for analysis of 3 independent data set for 3 types of cancer. Weaknesses of this study include: throughout the study the authors rely on bootstrap resampling for all estimations of accuracy and stability which is quite common technique. However the validity of such approach for testing of gene expression classifiers has been questioned in the literature. It has been reported that permutation based estimates could be a poor substitute for testing a classier on an independent set of real gene expression data. It would be interesting to see how well the proposed methods perform when tested on such independent gene expression datasets. It would be good to see additional discussion of this issue in the paper.Author’s responseWe have added additional discussion of the mentioned issue in the Bootstrap resampling section.Also – this study is using 500 resampling iterations for all steps of classifier construction however it is not clear if this is sufficient to ensure stability of the results, it would be useful if author could include additional comments on the reason for using 500 interactions.Author’s responseWe have added additional explanation in the Bootstrap resampling section.Overall, this is detailed study of the important issues related to classification of cancer samples based on global gene expression profiling. It is addresses several technical issues of accuracy and stability of classification results. Reviewer recommends considering publishing this paper in more specialized journal which could provide a better targeted readership in the bioinformatics community.Author’s responseBecause of applicability of the proposed methods for other high dimension biological data, this problem is important not only for readership in the bioinformatics. In our opinion the problems of reproducibility and stability of obtained features is especially important for biologists, people who work with biological data.AcceptableThe authors declare that they have no competing interests.SS and KF contributed equally to this work. Both authors read and approved the final manuscript.


Microarrays are widely used for examining differential gene expression, identifying single nucleotide polymorphisms, and detecting methylation loci. Multiple testing methods in microarray data analysis aim at controlling both Type I and Type II error rates; however, real microarray data do not always fit their distribution assumptions. Smyth's ubiquitous parametric method, for example, inadequately accommodates violations of normality assumptions, resulting in inflated Type I error rates. The Significance Analysis of Microarrays, another widely used microarray data analysis method, is based on a permutation test and is robust to non-normally distributed data; however, the Significance Analysis of Microarrays method fold change criteria are problematic, and can critically alter the conclusion of a study, as a result of compositional changes of the control data set in the analysis. We propose a novel approach, combining resampling with empirical Bayes methods: the Resampling-based empirical Bayes Methods. This approach not only reduces false discovery rates for non-normally distributed microarray data, but it is also impervious to fold change threshold since no control data set selection is needed. Through simulation studies, sensitivities, specificities, total rejections, and false discovery rates are compared across the Smyth's parametric method, the Significance Analysis of Microarrays, and the Resampling-based empirical Bayes Methods. Differences in false discovery rates controls between each approach are illustrated through a preterm delivery methylation study. The results show that the Resampling-based empirical Bayes Methods offer significantly higher specificity and lower false discovery rates compared to Smyth's parametric method when data are not normally distributed. The Resampling-based empirical Bayes Methods also offers higher statistical power than the Significance Analysis of Microarrays method when the proportion of significantly differentially expressed genes is large for both normally and non-normally distributed data. Finally, the Resampling-based empirical Bayes Methods are generalizable to next generation sequencing RNA-seq data analysis. Microarray technology is widely used to examine the activity level of thousands of genes simultaneously in human cells to better understand differential gene activation across diseases, such as heart diseases, infectious diseases, mental illness, and health disparities across ethnic groups. For example, DNA microarrays are widely used for DNA methylation studies - which are increasingly recognized as an important biological factor in ethnicity-based health disparities. A recent study shows that significantly different DNA methylation levels at birth, between Caucasians and African Americans, partially explain the incidence rates differential of specific cancers between ethnicities DNA methylation experiments typically use single channel or two-color microarrays for detecting DNA methylation differences between different groups. Smyth's parametric model (PM) p-values based on the p-values for non-normally distributed microarray data. Another widely used microarray data analysis method, Significance Analysis of Microarrays (SAM) Smyth developed the hierarchical model of Lönnstedt and Speed into a practical approach for general microarray experiments with arbitrary number of treatments and RNA samples using a moderated http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE49504).The data used in this paper to argue the false discovery controls of PM and RBMs were collected in accordance with the University of Hawaii IRB CHS #20067 terms of approval for a placental DNA methylation study deemed exempt from federal regulations pertaining to the protection of human research participants. Authority for exemption is documented in Title 45, Code of Federal Regulations, Part 46. The methylation microarray data have been deposited in NCBI's Gene Expression Omnibus m null hypotheses simultaneously in a DNA microarray study. Among the m hypotheses, m null hypotheses, we use m null hypotheses simultaneously.Suppose we are testing The framework of false discovery rate (FDR) was proposed by Soric For a discovery-based microarray study, FDR is generally recognized as an appropriate multiple testing error rate with 5% as the most commonly used cutoff value. When comparing different methods for microarray data analysis, high sensitivity and specificity are often desired properties of a good microarray analysis method. Sensitivity is defined as the probability of rejecting a non-true null hypothesis, while specificity is defined as the probability of failing to reject a true null hypothesis. The sensitivity and specificity of a multiple testing procedure can be calculated as follows:Sensitivity relates to a test's ability to identify positive results (giving the proportion of true positives) and is also a definition of power in multiple testing. A test with a high sensitivity has a low type II error rate and high power.Specificity relates to a test's ability to identify negative results. A test with high specificity has a low type I error rate which is important to control for.Resampling-based multiple testing procedures are widely used in genomic studies to identify differential gene expression and to conduct genome-wide association studies (GWAS), particularly when the distribution of test statistics is non-normally distributed or unknown. Meanwhile, resampling-based multiple testing procedures can also take into account dependence among p-values or test statistics. Commonly used resampling techniques include permutation tests and bootstrap methods.p-values even when the sample size is small. Westfall and Young A permutation test is a type of non-parametric statistical significance test in which the test statistics' distribution under the null hypothesis is constructed by calculating all possible values or a concrete number of test statistics  from permuted observations under the null hypothesis. The theory of permutation tests is based work done by Fisher and Pitman in the 1930s The bootstrap method, first introduced by Efron t tests. In SAM, each gene is assigned a score on the basis of its difference in gene expression relative to the standard deviation of repeated measurements for that gene. A scatter plot of the observed relative difference, versus the expected relative difference estimated by the permutation method, is then used to select statistically significant genes based on a pre-determined threshold.The Significance Analysis of Microarrays (SAM) was first introduced by Tusher et al. The SAM procedure can be summarized as follows.i  isIn general, let The design matrix X for design (c) and design (d) could be as follows:Different from two color microarrays, single color microarrays usually have a single expression value for each gene and each array. The design matrix for single color microarrays can be formed exactly as in classic linear model settings based on biological factors in microarray experiments.gth gene. Let jth diagonal element of g. The ordinary t-statistic under these assumptions isWe assume A prior distribution on The posterior mean of t-statistic, based on a hybrid classical/Bayes approach, is defined by:Then the moderated p-value for testing The To carry out the permutation/bootstrap test based on the moderated t-statistics proposed by Smyth g.Compute the moderated Permute/bootstrap the original data in a way that matches the null hypothesis to get permuted/bootstraped resamples, and construct the reference distribution using the moderated α test based on the upper α percentile of the reference distribution, or obtain the p-value by computing the proportion of permutation/bootstrap test statistics or p-values that are as extreme as, or more extreme than, the observed moderated p-value.Calculate the critical value of a level p-values for the p-value based permutation/bootstrap methods are calculated according to the following formula:The Similarly, the p-values for the test statistics-based permutation/bootstrap methods are calculated from the following formula:p-value of the lth hypothesis.In the above formulas, p-value calculation method , four RBM methods are proposed: RBM test statistic based permutation method (TSBP); RBM test statistic based bootstrap method (TSBB); RBM p-value based permutation method (PBP); RBM p-value based bootstrap method (PBB). Both the PM and the RBMs follow the empirical Bayes approach proposed by Efron et al. Depending on the resampling method (either permutation or bootstrap) and the m) is set to be 500 with the fraction of differentially expressed genes  distributed miroarray data.In terms of sensitivity (power), the PM shows very high sensitivity across all sample sizes and higher sensitivity than all other methods - even when sample size is small, e.g., 4 or 6 in each group . Both thch group .All methods show comparable specificity when sample size is large . Both thThe number of total rejections for all methods shows similar trends as sensitivity . The RBMFor false discovery rates control, the SAM method has the most conservative control rate among all methods compared for all sample sizes . The conIn summary, the PBB method performs best when sample size is large with high sensitivity, specificity, and low false discovery rate, while the PM performs well on sensitivity and specificity across all sample sizes, except for a slightly higher false discovery rate when the proportion of differentially expressed genes is lower than 50%. The SAM method also performs well with high sensitivity, specificity, and low false discovery rate, except for low sensitivity when the proportion of differentially expressed genes is over 50% for all sample sizes.In many cases, microarray data are not normally distributed and fail to be transformed to follow a normal distribution. Therefore, we explored the sensitivity, specificity, total rejection, and estimated false discovery rates of the PM, the SAM, and the RBMs for both log normally and mixed normally distributed data.When data are log normal distributed (right skewed), the PM method shows the highest sensitivity of all methods under comparison across all sample sizes . Both thThe specificities of the PM method are significantly lower than all other methods regardless of sample size and proportion of differentially expressed genes for both log normal distributed data and mixed normal distributed data . Both thThe number of total rejections for all methods also shows similar trends to sensitivity when data are either log normally or mixed normally distributed . In contThe PM method's false discovery rate is the highest of all methods compared, for both log normally or mixed normally distributed data. The estimated false discovery rates for the PM method are significantly higher than all other methods especially for data characterized by a small proportion of differentially expressed genes such as 10% or 25% . Of all In summary, the PBB method performs better than any other methods in terms of sensitivity, specificity and false discovery rate controls, when data are not normally distributed. The SAM method performs well, except for low sensitivity when the proportion of differentially expressed genes is over 50% across all sample sizes.Preterm birth, which is defined as birth occurring before 37 weeks of gestation, can be harmful to the short-term and long-term health of the infant, and creates a large economic burden in the US. The estimated lower boundary of annual societal economic burden associated with preterm birth in the United States was in excess of $26.2 billion in 2005. A recent study on the role of DNA methylation in preterm birth indicates that DNA methylation is an epigenetic risk factor in preterm birth which may influence the risk of preterm birth, or result in changes predisposing a neonate to adult-onset diseases A methylation two-color microarray study (2012) was conducted at the University of Hawaii John A. Burns School of Medicine to identify placental DNA methylation loci associated with preterm delivery. The DNA of 9 women's placental tissue (originating from 4 premature births and 5 term deliveries) was analyzed. The gestational age distribution between the preterm delivery group and the term delivery group had been compared using the permutation test and no significant difference was found between this two groups  are significantly better than those achieved by the PM, but never as good as those achieved by the PBB method and the SAM method when data are not normally distributed. However, for normally distributed data, the performance of all other RBMs is similar to the PM.Overall, the PBB method has the highest sensitivity, specificity, and the best FDR controls of all methods compared in this paper. Furthermore, the PBB method has much higher sensitivity than the SAM method when the proportion of differentially expressed genes is large, and much better FDR controls than the PM - especially when data are not normally distributed. The RBMs methods are computationally more intensive than Smyth's method as a result of the resampling approach; however, the computational efficiency of the RBMs methods could be greatly improved through a Bayesian algorithm that would reallocate more efficiently the number of resamples based on p-values A vexing issue with the Smyth's approach to microarrays analysis is its propensity to generate erroneous findings, especially when normality assumptions are violated. Our results show that the PBB method significantly improves microarray data analysis when normality assumptions are violated, and promotes accurate interpretation of findings from microarray studies. As Larsson pointed out, fold change criteria in the SAM method is problematic and can critically alter the conclusion of a study due to compositional changes of the control data set in the analysis


In psoriasis, only limited overlap between sets of genes identified as differentially expressed  was found using statistical and fold-change cut-offs. To provide a framework for utilizing prior psoriasis data sets we sought to understand the consistency of those sets.Microarray expression profiling and qRT-PCR were used to characterize gene expression in PP and PN skin from psoriasis patients. cDNA (three new data sets) and cRNA hybridization (four existing data sets) data were compared using a common analysis pipeline. Agreement between data sets was assessed using varying qualitative and quantitative cut-offs to generate a DEG list in a source data set and then using other data sets to validate the list. Concordance increased from 67% across all probe sets to over 99% across more than 10,000 probe sets when statistical filters were employed. The fold-change behavior of individual genes tended to be consistent across the multiple data sets. We found that genes with <2-fold change values were quantitatively reproducible between pairs of data-sets. In a subset of transcripts with a role in inflammation changes detected by microarray were confirmed by qRT-PCR with high concordance. For transcripts with both PN and PP levels within the microarray dynamic range, microarray and qRT-PCR were quantitatively reproducible, including minimal fold-changes in IL13, TNFSF11, and TNFRSF11B and genes with >10-fold changes in either direction such as CHRM3, IL12B and IFNG.Gene expression changes in psoriatic lesions were consistent across different studies, despite differences in patient selection, sample handling, and microarray platforms but between-study comparisons showed stronger agreement within than between platforms. We could use cut-offs as low as log10(ratio) = 0.1 (fold-change = 1.26), generating larger gene lists that validate on independent data sets. The reproducibility of PP signatures across data sets suggests that different sample sets can be productively compared. Psoriasis is a common chronic inflammatory skin disease characterized by keratinocyte hyperproliferation, changes in epidermal differentiation, and immune infiltrates in lesions. Histological analysis reveals patterns of abnormal epidermal hyperplasia and differentiation A common approach to comparing PP versus PN differences between data sets is to generate lists of differentially expressed genes (DEG) using thresholds of magnitude of difference and statistical significance to classify genes as differentially expressed. This strategy was used in several studies exploring differences between PP and PN skin in psoriasis Here we expanded the exploration of gene expression in psoriasis, and compared the data sets to each other using a global approach. We also addressed the question of whether relatively small changes between PP and PN skin  are reproducible across different psoriasis sample sets. Of particular interest are cytokines involved in Th1 and Th17 biology , which have been shown to play a role in psoriasis In this study, we used eight microarray sample sets to explore commonalities and differences in the psoriasis lesions . Of thesFor our microarray analyses, we used a sample preparation method for microarrays that generates labeled cDNA instead of cRNA In previous publications, the number of differentially expressed probe sets or transcripts reported for comparisons between PP and PN samples was less than 1000 for sample sizes of up to about 60 PP/PN pairs Several groups have characterized the transcriptome in psoriatic lesional skin For this comparison across data sets we used a single p-value cut-off. The robustness of the conclusions from the comparison to the effects of p-value and fold-change cut-offs can be investigated. We used pair-wise comparisons to ask how many genes that are identified as up-regulated using varying cut-offs in one sample set (the “source” sample set) do appear as up-regulated in a second sample set (the “target” sample set). We can then refine this question: How many of the selected genes from the source sample set change in the same or opposite direction in the target sample set when that direction is well determined (p≤0.05) in the target experiment? The target experiment filter in the latter question eliminates probe sets that have larger variances in the target data set, either because of noise in the target data set population or in the target data set technology. We then tested different restrictions on fold-change or p-value to examine the effects on concordance between data sets.To compare the different data sets, we first normalized the data using the same method for in-house and previously published data sets. Because differential gene expression analysis is commonly performed with data transformed into logarithmic space, the PP/PN fold-changes were expressed as log10(ratio). The log10(ratio) and p-values were estimated using a linear model and t-test for each probe set in each data set separately. We then applied different cut-offs for log10(ratio) and p-value to classify genes as up- or down-regulated. When no cut-off was used, any log10(ratio) >1 was classified as up-regulated and any log10(ratio) <1 was classified as down-regulated. In pair-wise comparisons, each probe set was classified as agreeing if it was up-regulated (or down-regulated) in both sample sets. The number of agreeing and disagreeing probe sets in each pair-wise comparison were counted and plotted using various cut-offs.For all possible pair-wise comparisons between data sets in this analysis we examined results produced by four different p-value and log10(ratio) cutoffs: i) no log10(ratio) or p-value cut-off, ii) p≤0.05 in the source set only, iii) log10(ratio)≥0.1and p≤0.05 in the source set only and iv) log10(ratio)≥0.1and p≤0.05 in the source set as well as p≤0.05 in the target set. The smaller number of probe sets contained on the HG-U133A and HG-U133A_2 arrays compared to the HG-U133_Plus_2 represents a non-random subset that consists of a large number of well-expressed probe sets. This may bias some of the comparisons. After applying the cut-off values to the source data set, the fraction of selected probe sets agreeing in the target set was calculated . The recWhen a p-value ≤0.05 cut-off in the source set was used, the proportion of agreeing probe sets increased to a range of 0.814 to 0.932 for cross-platform comparisons and from 0.838 to 0.986 for comparisons made among data sets run on the same platform . At thesThe good agreement between platforms continued into comparisons not just between PN and PP, but also between pre-dose and post-dose samples from etanercept treatment The agreement between summary results for any pair of data sets depends on the filtering criteria used to produce the summary results. In the case of our data sets (microarray comparisons between two conditions) there are two filtering criteria: a p-value and a fold-change cutoff. Thus, the agreement between any pair as measured by gene-set overlapdepends on four variables. It is challenging to visualize the agreement for multiple pair-wise comparisons as a function of the four filtering variables. We have chosen to plot the number of agreeing and disagreeing probe sets for various pair-wise comparisons of the U133_Plus_2 data sets as a function of the source set cut-offs ; we alsoClearly, there is a higher level of disagreement between than within platforms. In order to explore how additional threshold criteria on the target data set would affect the number of agreeing and disagreeing probe sets. The absolute number of probe sets where the fold-change disagreed in sign between the source and target sets decreased considerably in For another characterization of the trade-offs made by different fold-change cut-offs, we can select a fixed p-value in the source sets in The agreement between probe sets from two data sets from different platforms is shown in The extent of quantitative agreement at low log10(ratio) values can be seen by selecting at set of probe sets within a tight range of log10(ratio) values (±0.005 of a chosen log10(ratio) value) in one data set and looking at the distribution of values for those probe sets in the other data set. While the distribution broadens with increasing log10(ratios), probe sets with specific log10(ratios) in the source data set were distributed in a range around that value in the target data set . This imMicroarrays have a relatively small assay range, with data compression at high expression levels and background signal and potential for cross-hybridization at low expression levels. This can limit reliable detection and quantitation of differential expression Several studies have used whole-genome microarrays or qRT-PCR to explore gene expression changes in psoriatic PP skin. Each study identified biologically relevant sets of genes that were differentially expressed between PP and PN psoriatic skin. Differences between studies can be due to threshold effects, technology choices, or population differences, and do not necessarily reflect pertinent biological differences. We investigated the differences between several microarray data sets and, for a few genes, the differences between qRT-PCR and microarray within one sample set and came to the conclusion that most published data sets are both qualitatively and quantitatively similar.For genes with expression within the dynamic range of microarray, fold-changes qualitatively agreed between qRT-PCR and microarray analyses with different labeling methodologies for most of the tested genes. Highly expressed genes such as KRT16, IL1F9, and S100A8 suffered fold-change compression on microarray compared to qRT-PCR but remained correlated and were easily classified as differentially expressed. While differential expression was obtained with microarray and qRT-PCR, the latter likely provides a more realistic estimate of the magnitude of differential expression. The hybridization methodology - cRNA vs. cDNA - can also have an effect on ratios. Several cytokine genes with overall low expression but high differential effect in psoriasis  displayed higher magnitude changes with cDNA than with cRNA hybridization, whereas the highly expressed gene IL1F9 showed a higher magnitude change with cRNA hybridization. The higher log10(ratio) for low-abundance transcripts with cDNA hybridization applied to several cytokines that are of interest in psoriasis but is not a universal effect of this platform. Other cytokines and receptors with low expression levels and consistently minimal or no changes in psoriasis  showed similar results across all three platforms. While on a population basis there are remarkably consistent quantitative gene expression changes, this does not rule out considerable variation in the fold-changes for particular genes on the individual patient level.For comparisons across sample sets, microarray results are often summarized by creating a list of genes identified as differentially expressed between two conditions. These sets are usually determined by applying a selection criterion such as a p-value or a fold-change, or often both 389-IL2 The most interesting observation was the consistency between data sets in the direction of regulation. This consistency was obtained despite many differences between studies ranging from patient population to microarray hybridization. Biopsies were either snap-frozen The large extent of agreement between any pair of studies exploring gene expression in psoriatic lesions supports the combination of multiple psoriasis data sets to enhance the strength of conclusions that can be drawn from data sets of this type. It also implies that discoveries found in single data sets should reproduce in others. A very large number of genes are affected and these genes are differentially detectable with different methodologies. Even at low fold-change values, there was high consistency between studies. Differences in clinical protocols, patient groups, and experimental platforms cause a small increase in disagreement in the direction of gene expression changes between PP and PN tissue. However, these factors do not appear to be major contributors to the extensive and clear PP versus PN differences detected consistently in our data sets and the previously published ones. Our analyses indicate that previously used thresholds can be reduced while retaining a low false-positive rate. This results in a larger list of DEGs and leaves open the opportunity to discover yet more information relevant to psoriasis biology.Clinical trial NCT00867100 was conducted according to the principles expressed in the Declaration of Helsinki. All patients provided written informed consent for expression analysis of skin biopsies and for their information to be stored in the study database and used for research. Amgen protocol 20060279 was approved by the Institutional Review Boards of the Royal Adelaide Hospital, Adelaide , The Alfred Hospital, Melbourne  and Research Review Board Inc, Waterloo (Canada). The skin biopsy collection through Asterand, Detroit, MI was approved by the review board at Chesapeake Research Review Inc.Paired psoriatic PP and PN frozen skin punch biopsies were procured from Asterand, Detroit, MI (n = 14) and from clinical trial NCT00867100 . The Asterand samples were from male and female patients with mild to moderate plaque psoriasis aged 23 to 71 years. These patients did not report use of medication to treat psoriasis. The patient population of NCT00867100 has been described One 5 mm (Asterand) or 6 mm (NCT00867100) punch biopsy was taken from an active psoriatic lesion and one 5 mm (Asterand) or 6 mm (NCT00867100) punch biopsy was taken from an adjacent, uninvolved area. Skin samples were immediately flash frozen in liquid nitrogen and stored at −80°C. RNA samples from GSE11903 were obtained from Dr. Jim Krueger’s laboratory (pre- and post-treatment). Raw data from previously published studies  and further homogenized with a TissueRuptor  in the presence of lysis/binding buffer or were directly homogenized with either a TissueRuptor or Polytron  in the presence of lysis buffer. Total RNA was isolated from frozen psoriatic PP and PN skin punch biopsies using the mirVana miRNA Isolation Kit  modified to include on-column DNase treatment with RNase-free DNase . The integrity of the RNA samples was assessed using a Bioanalyzer 2100. Total RNA concentration was measured on an ND-1000 Spectrophotometer .−ΔΔCt.Quantitative Reverse Transcription-PCR (qRT-PCR) was performed on a subset of 8 PP/PN biopsy pairs from the Asterand sample set. The house-keeping genes β-actin and UBC were used to normalize expression levels. For β-actin the primers were 5′-CCT GGC ACC CAG CAC A-3′ and 5′-GCC GAT CCA CAC GGA GTA CT-3′, and the probe was 5′-VIC-ATC AAG ATC ATT GCT CCT CCT GAG CG-3′.For UBC the primers were 5′-TGA CAA TGC AGA TCT TCG TGA AG-3′ and 5′-GGT GTC ACT GGG CTC AAC CT-3′ and the probe was 5′-VIC-TG ACT GGT AAG ACC ATC AC-3′. For the other transcripts, pre-designed Taqman® Gene Expression assays from Applied Biosystems, Carlsbad, CA were used (see Table S1). A total of 0.75 µg RNA per sample was reverse transcribed in 50 uL  reactions primed with random hexamers using the High Capacity cDNA Reverse Transcription Kit (Applied Biosystems) according to the manufacturer’s protocol. Resulting cDNA was diluted to 2 ng/µL with 1 ug/µL glycogen . Five micro liters (10 ng) of each cDNA preparation were amplified in duplicate 20 µL reactions using TaqMan Universal PCR Master Mix (Applied Biosystems) according to the manufacturer’s protocol. Real-time PCR was performed in 384-well optical plates on a 7900HT Sequence Detection System (Applied Biosystems) running SDS version 2.3 software with the following conditions: 50°C for 2 min and then 95°C for 10 minutes followed by 40 cycles of 95°C for 15 sec and 60°C for 1 min. Average raw cycle threshold (Ct) data were normalized to ΔCt by using the mean of both invariant genes (β-actin and UBC). Fold difference between target gene levels in PN and PP samples was calculated with the comparative Ct method using the formula 2Fifty nanograms of total RNA were amplified on an ArrayPlex  using the Ovation® RNA Amplification System V2 and WB reagent . Of the amplified cDNA, 4.4 µg was labeled using the FL Ovation™ cDNA Biotin Module V2  according to the manufacturer’s recommendations. The labeled cDNA was hybridized onto Affymetrix human genome HG-U133_Plus_2 arrays  and processed according to Affymetrix technical protocols. The average intensity of each array was scaled to a target intensity of 500.The following data sets were downloaded from GEO: GSE14905 The microarray CEL files from the published data sets as well as the in-house data sets were imported into Rosetta Resolver 7.2 . Data was preprocessed in Rosetta Resolver using their standard processing pipeline. Ratios between PP and PN groups were generated within each data set using a pipeline that consisted of: (1) the Resolver Affymetrix Rosetta Intensity Profile Builder (saves reporters), (2) the Affymetrix Rosetta Intensity Experiment Builder (group dependent), and (3) the Affymetrix Ratio Builder (no error-weighting). This pipeline calculates a weighted-average variance that includes two components, a lower bound variance based on predetermined chip-specific background noise, and a data scatter variance derived from the probe measurements within the experiment. As N increases, the data scatter variance contributes more to the final calculation. To assess the extent of agreement between pairs of data sets, we examined all possible pair-wise comparisons. For each pair, we compared the log10(ratio) for each sequence between the two data sets: if the average log10(ratio) was greater than or equal to zero or less than or equal to zero in both cases, it was classified as agreeing; if the log10(ratio) did not agree in sign, it was classified as disagreeing. Subsets of probe sets based on log10(ratio) values or p-values were created and agreements computed for p-values ranging from 0.01 to 0.2 in steps of 0.01 in the source set.Supplemental Table 1Taqman assays and microarray sequencecodes for Taqman-microarray comparisons.(XLSX)Click here for additional data file.Supplemental Table 2Complete list of average log (ratio) andp-values for each data set.(XLSX)Click here for additional data file.


Glycolysis in presence of oxygen with high glucose consumption is known to be the metabolism of choice in many tumors. In lung cancer this phenomenon is routinely exploited in diagnostic PET imaging of fluorodeoxyglucose uptake, but not much is known about the prognostic capabilities of glycolysis level assessment in resected lung tumor samples.In this retrospective study, we used real time polymerase chain reaction(RQ-PCR) to assess the expression level of the gene for Glyceraldehyde 3-phosphate dehydrogenase(GAPDH), key enzyme for glucose breakdown, in tumor samples from 82 consecutive early stages resected non small cell lung cancer(NSCLC) patients. We then compared our results in six large publicly available NSCLC microarray datasets collecting data from over 1250 total patients.GAPDH gene over expression was found to be an adverse prognostic factor in early stages NSCLC (n = 82 HR = 1.30 p = 0.050). This result was confirmed in 5 of 6 public datasets analyzed: Shedden et al. 2008: n = 442 HR = 1.54 p < 0.0001; Lee et al. 2008: n = 138 HR = 1.31 p = 0.043; Tomida et al. 2009: n = 117 HR = 1.59 p = 0.004; Roepman et al. 2009: n = 172 (TPI1 gene) HR = 1.51 p = 0.009; Okayama et al. 2012: n = 226 HR = 3.19 p < 0.0001; Botling et al. 2013: n = 196 HR = 1.00 p = 0.97). Furthermore, in the large and clinically well annotated Shedden et al. microarray dataset, GAPDH hazard ratio did not change whether calculated for the whole dataset or for the subgroup of adjuvant naive patients only (n = 330 HR = 1.49 p < 0.0001).In our study GAPDH gene over expression in resected tumor samples is an adverse prognostic factor in NSCLC. Our results confirm the prognostic value of glucose metabolism assessment in NSCLC. Cancer cell metabolism characterized by high glycolysis rate in presence of oxygen has been confirmed in many tumors . This phThis field of cancer research is promising. In fact the high glycolysis rate in tumors, as assessed by diagnostic positron emission tomography (PET) imaging of fluorodeoxyglucose (FDG) uptake, is also exploited in clinical practice, in the differential diagnosis of nodules of unknown origin, and, more recently, also in prognostic studies -7. HowevDiagnostic PET imaging is routinely performed in NSCLC, the most frequent histological type of lung cancer (still the leading cause of cancer death in the world ). There New effective prognostic factors could be very useful for NSCLC patients. Presently, pathological stage of the resected tumor is the main prognostic factor used in clinical practice to select NSCLC patients to be referred for additional therapies after surgery , but manGAPDH has an essential role in glucose metabolism, where the corresponding enzyme converts glyceraldehydes-3-phosphate to 1,3-diphosphoglycerate with reduction of nicotinamide adenine dinucleotide (NAD+) to NADH. In fact GAPDH gene is expressed in all tissue, so to be classically used as housekeeping gene, but it is known to be over expressed in many tumors as compared to normal tissues, and also to be correlated with poor prognosis or tumor aggressiveness in ovarian, breast, renal, colorectal, melanoma cancer [GAPDH gene is known to be expressed at high levels as compared to the surrounding non cancerous lung biopsies [GAPDH expression level in resected NSCLC samples is still to be assessed. In this retrospective study, we measured GAPDH gene expression, by RQ-PCR, on tumor samples from a group 82 resected NSCLC patients. After detecting a significant correlation of GAPDH with survival from our patient follow-up data, we decided to further investigate the expression of GAPDH gene in six large publicly available NSCLC microarray datasets, collecting data from over 1250 total NSCLC patients.Among these genes, a cancer . Furthera cancer -18. In la cancer , and GAPbiopsies . HoweverOur study included 82 consecutive patients, stage I-III NSCLC, who had undergone radical surgical resection at National Institute for Cancer Research, Genoa, Italy (IST) between July 2005 and March 2007. All tumors were surgically removed without microscopic residual disease. None of the patients received adjuvant radiotherapy or chemotherapy. Follow up period lasted from July 2005 to December 2010 and survival time was computed from the date of surgery. Informed written consent from the patients and approval of our institute (IST) Bioethics Board were obtained. Patient and tumor characteristics are in Table B2M) and beta-glucuronidase (GUSB) were used as housekeeping genes and a virtual housekeeping gene was calculated using BestKeeper software [RNA was isolated from paraffin-embedded tumor samples using the High Pure FFPE RNA Micro Kit  with minor modifications. RNA were reverse-transcribed with SuperScript™ II RT  according to the manufacturer’s instructions. Resulting cDNA was amplified by the LightCycler 480 Real Time PCR System II (Roche Applied Science). Relative gene expression levels were calculated using the Qgene software  featurinsoftware . PCR priGAPDH RQ-PCR gene expression, with hazard ratio (HR) and confidence interval (CI) calculation, were performed on our patient data by using the Cox regression model. Multivariate Cox regression was performed with adjusting for tumor stage categorized in 3 classes (I – II - III). The Kaplan-Meier curve was plotted by separating patients on the median GAPDH gene expression level. Cumulative survivals were assessed by the Kaplan-Meier model. All calculations and plots were performed by using R 2.14(64bit) software [Overall survival analysis for software . Gene exhttps://array.nci.nih.gov/caarray/project/details.action?project.id=182; GSE8894, Lee et al. [http://research.agendia.com ; GSE31210, Okayama et al. [http://www.ncbi.nlm.nih.gov/geo/. Full patient and tumor characteristics are in cited papers; a summary of dataset characteristics, and our patient data for comparison, is in Table GAPDH HR results. In fact, our patients had not received adjuvant treatments but, in most of the microarray datasets, no information about adjuvant treatments was available at patient level.Six publicly available NSCLC microarrays datasets were used: Shedden et al.  Sh2008)e et al.  (Le2008)e et al.  (To2009)e et al.  (Ro2009)a et al.  (Ok2012)a et al.  (Bo2013)008e et aGAPDH gene level was calculated for each patient sample as the mean level of all its probes mapped to GAPDH. Stage was categorized in 3 classes (I-II-III) or 2 classes (I – II + III) when patient numbers were low in higher stages. For the Le2008 dataset, relapse free survival data was used in regressions due to overall survival data unavailability. For the Ro2009 dataset, a large and clinically well annotated dataset, the probe annotation file had no GAPDH gene reference; we decided to use, at least in part, this dataset in the present work by analyzing the gene expression for the strictly metabolically related triosephosphate isomerase 1 (TPI1), that was found in fact highly correlated with GAPDH gene in all other microarray datasets (Pearson's r > 0.79 for GAPDH-TPI1 expression levels). Also, patient characteristics and clinical data accuracy were diverse among the microarray datasets; so we performed the survival analysis separately for each microarray dataset, and reported the results using a forest plot [Statistical analysis for the datasets was performed similarly as for our patient RQ-PCR data. A single est plot  style coest plot  and Biocest plot .Patient and cancer characteristics are reported in Table GAPDH gene expression, measured by RQ-PCR, was found significantly correlated with patient survival   Figure , where pGAPDH HR value was lower than in univariate model and not significant   . Cumulative survivals and dataset sizes are also reported in the plots.Cox regression analysis for GAPDH HR and 95% CI values found in our patients were in good agreement with the values calculated in the microarrays datasets, with the exception of the Bo2013 dataset (HR1.00 p = 0.97). This latter also featured a five years cumulative survival lower (0.42) than most other datasets (in the range of 0.50-0.84) , after adjusting for tumor stage, HR value was decreased and not significant. In multivariate Cox proportional hazard model together with GAPDH gene expression, tumor stage HR values were found high (as expected) for most microarray datasets . However, for Bo2013 dataset, tumor stage HR values were lower than in the other datasets, and not significant for stage I-II despite the high patient and event numbers .By comparing our regression results before and after adjusting for tumor stage . Finally, in random sampling of subgroups of patients from Sh2008, we observed that GAPDH HR was not specifically affected when calculated in subsets featuring low cumulative survivals  together with GAPDH itself; Sh2008 was used as verification dataset. Authors don't show the prognostic performance of GAPDH alone; however, our results, confirmed on a large number of public datasets including Sh2008, suggest that large part of the prognostic performances shown in Sh2008 have to be attributed to GAPDH alone.Our results for g et al.  in whichTPI1 gene levels instead of the unavailable GAPDH ones. Actually this substitution was based on the strict metabolic relation between the two catalytic proteins – however the high correlation of the two genes was verified in the other datasets, and is confirmed by other authors too [TPI1, very similar to GAPDH results in the other datasets, can further support that the prognostic capabilities of GAPDH in NSCLC reflect the role of the corresponding enzyme in glucose metabolism.In the forest plots we showed the Ro2009 dataset results too, by plotting its hors too . So, Ro2GAPDH correlation with prognosis (HR = 1.0); this dataset was also featuring some characteristics different from all the other ones: i) a low cumulative survival, also at low tumor stages, and ii) a low tumor stage HR and significance, despite the high patient and event numbers. We have no data supporting a correlation of these characteristics with a strong decrease of HR values for GAPDH, so we can only conclude that the Bo2013 dataset is different from the other datasets from more than a single point of view.However one dataset (Bo2013) had a null result for GAPDH HR was not affected when selecting only patients that had not received any adjuvant therapy; we performed this comparison in the Sh2008 dataset. This result was helpful for our data analysis; in fact our patients had not received radiotherapy or chemotherapy, but in most microarray datasets the information, whether adjuvant treatments had been performed or not, was not available at patient level. Actually, adjuvant treatment presence could confound a survival analysis because there is-finally- evidence that it can increase survival also in lower stages patient [GAPDH HR value .From the clinical point of view the GAPDH as housekeeper gene in NSCLC prognostic studies involving RQ-PCR measurements; we also suggest that any past NSCLC prognostic study using GAPDH as housekeeper gene should be considered potentially biased.Finally, not only our results warn researchers from using GAPDH gene expression level in resected tumor, as assessed by RQ-PCR or microarray, is an important prognostic factor in NSCLC, that confirms the importance of investigating metabolism in lung cancer.In conclusion, The authors declare that they have no competing interests.RP, GS conceived the study, contributed to the interpretation of the data and wrote the manuscript. GS and AE performed RQ-PCR analysis and interpretation. RP performed all statistical analyses and public microarray dataset selection, collection and handling. SS and MT performed tumor sample handling, staging, storage and selection. MGDB, GB, ER, CG, CS fulfilled ethical authorizations, collected and stored patient data, clinical data and follow-up. UP, FG contributed to the interpretation of the data and revised manuscript. All authors reviewed and approved the manuscript.GAPDH primer for RQ-PCR and RQ-PCR data. Primer sequence used for GAPDH RQ-PCR and RQ-PCR data for IST patients.Click here for fileGAPDH HR variation in low survival subsets.GAPDH Hazard Ratio (HR) variation in low cumulative survival subsets, investigated by random sampling from Shedden et al. 2008 dataset.Click here for file


Highly parallel analysis of gene expression has recently been used to identify gene sets or ‘signatures’ to improve patient diagnosis and risk stratification. Once a signature is generated, traditional statistical testing is used to evaluate its prognostic performance. However, due to the dimensionality of microarrays, this can lead to false interpretation of these signatures.A method was developed to test batches of a user-specified number of randomly chosen signatures in patient microarray datasets. The percentage of random generated signatures yielding prognostic value was assessed using ROC analysis by calculating the area under the curve (AUC) in six public available cancer patient microarray datasets. We found that a signature consisting of randomly selected genes has an average 10% chance of reaching significance when assessed in a single dataset, but can range from 1% to ∼40% depending on the dataset in question. Increasing the number of validation datasets markedly reduces this number.We have shown that the use of an arbitrary cut-off value for evaluation of signature significance is not suitable for this type of research, but should be defined for each dataset separately. Our method can be used to establish and evaluate signature performance of any derived gene signature in a dataset by comparing its performance to thousands of randomly generated signatures. It will be of most interest for cases where few data are available and testing in multiple datasets is limited. It has provided insight into the biological mechanisms underlying tumour formation and identified new therapy targets et al.A major challenge with DNA microarray technology is to take account of variability across a very large number of parameters Statistical significance of the differences in gene expression patterns for different patient groups or tumour types is often determined with traditional statistical testing procedures, such as the two-sample t-tests or Wilcoxon rank sum tests More importantly multiple testing is often not considered in evaluating the prognostic power of signatures. Once a signature is created, its prognostic power is determined with traditional survival statistics and standard cut-off values for significance. We hypothesise that this can lead to high numbers of false prognostic signatures when the number of evaluated datasets is limited. Therefore we sought to develop a simple method to take into account the high-dimensionality of microarrays in the phase of evaluating signature prognosticity.To quantify the problem of multiple testing we have developed a method to test batches of random signatures in microarray datasets. We show that the average chance that a random signature produces a prognostic result in one dataset is approximately 10% but can range from 1% to ∼40%. Increasing the number of datasets reduces this false positive rate significantly. As a result of this high degree of variability amongst datasets, we developed a method that can be used to determine an appropriate threshold level of significance that must be reached for a given signature. This is done by testing a set of randomly chosen signatures along with the signature of interest within the dataset under investigation.In order to assess the potential for identifying prognostic gene signatures by chance alone in microarray based datasets a method was developed to test the prognostic value of batches of randomly generated signatures. Six different publicly available microarray datasets with follow-up data were used . These sFor each dataset separately 5 batches of 10,000 random signatures were generated and tested. In each batch the number of genes (UnigeneIDs) in a gene set was predefined. The number of genes (UnigeneIDs) in the five batches were 10, 25, 50, 100 and 200 respectively. For example, the first batch included 10,000 random signatures, each consisting of ten genes. For each signature a patient score was derived, defined as the average of the expression of the genes in a signature (equation 1). Each signature score was then tested for prognostic value by ROC analysis and determination of the AUC. et al.To define a reasonable cut-off value for the AUC values, we first searched for AUCs used in published gene signatures. However, the majority of studies do not evaluate gene signatures using the AUC. Most gene signatures are evaluated with Kaplan-Meier survival curves and log-rank tests. Kaplan-Meier survival analyses and ROC analyses are linked; a high AUC corresponds to a good separation in distinct survival groups. To be able to define a cut-off, we calculated the AUCs for the different gene sets as evaluated in the review by Ntzani In Sampling 10,000 gene sets is a small number compared to the total number of possible gene sets. In order to show that the 10,000 random gene sets are sufficient to estimate the AUC distribution, we tested batches of 1,000,000 signatures consisting of 10 genes in the six datasets. The AUC distributions for this permutation study were similar to the distributions for the batches of 10,000 gene sets .From the differences between the six datasets , it coulIt has already been reported in previous studies One of the parameters that could account for differences in the number of prognostic signatures for a given dataset is filtering. To briefly explore the influence of filtering, two simple filtering methods were applied on the Miller dataset. After this filtering, again five batches of 10,000 signatures were tested.The first filtering procedure was to only consider reporters that had no absent calls in the patients. This very stringent filtering resulted in a reduction in number of reporters from ∼45,000 to ∼7,300 . The second filtering method, often used in microarray based studies, consists of simply applying a threshold to the fold change. To show the effect of this step on the number of false positives a twofold threshold was applied. Only genes that show at least a two-fold change across the patients are considered. This reduced the number of reporters from ∼45,000 to ∼23,000.The results for these analyses show that both filtering methods have a different effect . Fold chTo demonstrate that this random signature method can be used with all sorts of signature evaluation methods, two additional evaluation procedures were tested in the Miller dataset. In the previous analyses the signature score was used as continuous variable.Here we selected 10,000 random samples of 10, 25, 50, 100 and 200 genes. In the first setup the signature score was used to median dichotomize the patients. In the second setup these gene subsets were in a K-nearest neighbor classification (KNN) combined with leave-one-out-cross validation (LOOCV). Both procedures results in patient classification into two groups, which were then coupled to outcome and evaluated by the AUC. Similar AUC distributions are obtained with these different signature evaluation procedures, exact distributions characteristics differ slightly . The numTo show that the random signature testing method is a valuable tool in microarray based studies, several published gene signatures were tested. In short, the suggested procedure to test a signature in a dataset is as follows . For theThe Wound signature For the Miller dataset also Kaplan-Meier survival analyses were performed, since the two early hypoxia signatures were previously tested in this dataset. The results of Kaplan-Meier survival analyses and the random signature testing are given in We assessed six patient microarray datasets spanning different cancer types, numbers of patients and arrays to evaluate the effect of false positives on gene signature evaluation. Different-sized batches of 10,000 random signatures were tested in all datasets. With the given threshold, the average chance that a randomly generated signature was considered prognostic was approximately 10%, but ranged from 1% to ∼40%.Testing batches of random signatures in different datasets revealed that the AUC distribution varied widely between datasets. Choosing an arbitrary cut-off value for significance is then clearly not suited for gene signature evaluation. Rather a dataset-based cut-off value should be considered. The random testing method we propose here can be applied to calculate the level of AUC necessary to reach significance beyond random for a given signature size in a given dataset.et al.The random testing procedure can also be used to directly test whether the performance of a certain signature could be due to chance. A schematic overview is given in We have shown that proper validation is absolutely essential in gene signature research. This supports several previous studies, which have argued that signature performance is often overestimated due to improper validation in a large number of studies A method to overcome this multiple testing problem is validation in multiple independent datasets. We have shown that testing random signatures in two datasets decreased the chance that a random signature is called prognostic dramatically . Howeveret al.et al.et al.From the analyses on all six datasets, several parameters could influence the number of false positives. To assess the effect of these variables, several parameters were manipulated in one of the datasets. However, of the tested parameters, only patient number influenced the false positive rate dramatically. The need for large patient groups to obtain reliable results has already been recognised in other studies. Ntzani Testing batches of random generated gene sets in different gene expression microarray datasets showed that the use of an arbitrary cut-off value for evaluation of signature significance is not suitable. Further it is important to use the same signature evaluation procedure for the random gene sets as for the signature of interest, since the AUC distribution can differ when using a different method. Thresholds should be defined for single datasets separately in order to obtain reproducible results. This permutation method can be used to establish and evaluate signature performance of any derived gene set within single or multiple datasets by comparing its performance to the performance distribution of thousands of randomly generated signatures. However it will be of most interest for cases where limited data is available.A method to test the prognostic value of random gene signatures of a predefined size on a microarray dataset was developed in Matlab . Unless indicated otherwise, analyses were performed using this program. The program creates a user-specified number of random gene sets, consisting of a user-specified number of genes. For a given dataset all genes on the respective microarray were used to create the random signatures. This batch of random signatures was then tested on a dataset by means of a signature score calculation.http://www.ncbi.nlm.nih.gov/projects/geo/) and Stanford Microarray Database (SMD: http://genome-www.stanford.edu/microarray) and elsewhere. Accessory clinical and follow-up data were also given or provided by the authors on request. Patient microarray and clinical follow-up data were collated to test the random gene sets. Datasets are publicly available in the microarray databases Gene Expression Omnibus ). When a gene was represented by more than one reporter on an array, the expression of the reporters was averaged before signature calculation. The signature scores for each patient were then coupled to the survival data of the patients.i,m, gene expression of gene i in sample m.Where: Score, signature score; N, number of genes in the signature; expThe signature score was used to median dichotomize the patient cohorts.In a second setup expression of the genes in the signature were used for K-nearest neighbor classification (KNN) combined with leave-one-out-cross validation (LOOCV). With this method one patient is withheld and the class membership of this patient is predicted using the KNN model (knnclassify function in Matlab) built on the remaining patients. The event parameter of the survival data was used as training class. This procedure was repeated for each patient, resulting in a class prediction for the whole cohort.The signature scores, median dichotomized groups or KNN classifications were evaluated with the area under the curve (AUC) of the receiver operator curve (ROC). Definitions for AUC calculations are as follows:True positive: patient in the high score group that died from diseaseFalse positive: patient in the high score group that is aliveTrue negative: patient in the low score group that is aliveFalse negative: patient in the low score group that died from diseaseet al.A signature score was considered prognostic when the AUC is ≤0.4 or ≥0.6. This cut-off value was based on the AUCs of several published gene signatures evaluated in the study of Ntzani Figure S1Effect of number of genes, number of probes per gene, filtering and signature evaluation procedure on AUC distribution. A: Percentages of signatures that pass the criteria of AUC≤0.4 or AUC≥0.6 for the 5 batches of 10,000 runs for the Miller dataset, groups consisting of half of the UnigeneIDs of the Miller dataset. B: Percentages of signatures that pass the criteria of AUC≤0.4 or AUC≥0.6 for the 5 batches of 10,000 runs for the Miller dataset, taking only one or multiple reporters per gene into account. C: Percentages of signatures that pass the criteria of AUC≤0.4 or AUC≥0.6 for the 5 batches of 10,000 runs for the Miller dataset, taking two different filtering methods. D: Percentages of signatures that pass the criteria of AUC≤0.4 or AUC≥0.6 for the 5 batches of 10,000 runs for the Miller dataset, taking three different signature evaluation procedures .(TIF)Click here for additional data file.Figure S2Evaluating random gene sets in multiple datasets. Percentages of signatures that pass the criteria of AUC≤0.4 or AUC≥0.6 for the 5 batches of 10,000 runs for the van de Vijver and Beer datasets separately and combined.(TIF)Click here for additional data file.Figure S3Kaplan-Meier survival analysis of 4 published gene signatures. Kaplan-Meier survival curves for the Miller dataset for 4 published signatures .(TIF)Click here for additional data file.File S1The supplementary material contains a section with supplementary materials and methods and a section supplementary results. The supplementary materials and methods is a more detailed description of the data analyses. The supplementary results describe the analyses to check the influence of several parameters on the random signature AUC distribution that had minimal to no effect. Further additional tables are included.(DOC)Click here for additional data file.