While global breast cancer gene expression data sets have considerable commonality in terms of their data content, the populations that they represent and the data collection methods utilized can be quite disparate. We sought to assess the extent and consequence of these systematic differences with respect to identifying clinically significant prognostic groups.We ascertained how effectively unsupervised clustering employing randomly generated sets of genes could segregate tumors into prognostic groups using four well-characterized breast cancer data sets.Using a common set of 5,000 randomly generated lists (70 genes/list), the percentages of clusters with significant differences in metastasis latencies  was 62%, 15%, 21% and 0% in the NKI2 (Netherlands Cancer Institute), Wang, TRANSBIG and KJX64/KJ125 data sets, respectively. Among ER positive tumors, the percentages were 38%, 11%, 4% and 0%, respectively. Few random lists were predictive among ER negative tumors in any data set. Clustering was associated with ER status and, after globally adjusting for the effects of ER-α gene expression, the percentages were 25%, 33%, 1% and 0%, respectively. The impact of adjusting for ER status depended on the extent of confounding between ER-α gene expression and markers of proliferation.It is highly probable to identify a statistically significant association between a given gene list and prognosis in the NKI2 dataset due to its large sample size and the interrelationship between ER-α expression and markers of proliferation. In most respects, the TRANSBIG data set generated similar outcomes as the NKI2 data set, although its smaller sample size led to fewer statistically significant results. Over the past decade, a large number of global gene expression data sets of human breast cancers have become publicly available -6. TheseThe strong inter-relationships between clinical features, gene expression patterns and prognosis has led to the postulate that, depending upon the underlying relationship between the clinical and prognostic factors in a given data set, prognostic gene expression signatures may simply function as a proxy measure for these established clinical variables. For example, Gruvberger et. al. showed that a gene expression signature derived from the van't Veer data set , for whiWhile breast cancer gene expression data sets have considerable commonality in terms of their data content, the populations that they represent and the data collection and analysis methods can be quite disparate. The advantage of this heterogeneity is that it provides an opportunity to test the robustness of a hypothesis across multiple populations represented in these data sets . HoweverTo gain insight into the extent of potential systematic differences in obtaining statistically significant results across breast cancer gene expression data sets, we ascertained whether there were intrinsic differences in the likelihood of observing an association between the expression of a selected set of genes and metastasis using four well-characterized data sets. To minimize bias in our approach, an unsupervised clustering algorithm was used to segregate tumors into one of two clusters based on the expression levels of randomly selected sets of genes (30–400 genes/set). These clusters were then compared for differences in metastasis latencies. We found that one data set, the Netherlands Cancer Institute (NKI2) set of young women with early stage disease, was considerably more likely to give a significant finding when examining either all tumors or only ER positive tumors, as compared to the other data sets examined. Factors that contributed to an increased likelihood of a random gene list being predictive of metastasis within a given data set were 1) the number of tumors analyzed and 2) the inter-relationships between ER expression, proliferation and metastases. We suggest that these intrinsic differences between the data sets should be considered in the design and analysis of future studies incorporating gene expression data.http://www.rii.com/publications/default.htm) [Global gene expression and clinical data (including estrogen receptor status and metastasis recurrence latencies) were analyzed in four independent, publicly available breast cancer gene expression data sets. The Netherlands Clinical Institute (NKI2) data set contains data on 295 women with early stage breast cancer (downloaded from ult.htm) ,19. The ult.htm) . The TRAult.htm) ,21.All probes from each data set were used in the analyses except in the NKI2 data set where only the probes where there was complete data on more than 291 of the 295 subjects were used . Missing values for genes in the NKI2 set were imputed using the "impute" option in the FastClus procedure. Gene expression data for the NKI2 data set were given as log10 expression ratios, while data from the TRANSBIG and KJX64/KJ125 sets were log2 expression values. Expression values were log2 transformed in the Wang gene expression data set. Each gene probe in each data set was mapped to a unigene cluster ID using the SOURCE database (source.stanford.edu).For each gene expression data set, a new data set containing a single set of expression data for each unique unigene cluster ID was created. In the Wang, Miller and KJX64/KJ125 data sets, the expression values for each probe was set to have a median value of 0 and standard deviation of 1. In instances where there were multiple gene probes with a common unigene cluster ID, the median expression value of all the common probes was used. There were 11,318 genes with unique unigene identifiers that were common to all four data sets.Random lists of genes of various sizes (30 to 400 genes per list) were generated by simple random sampling (Surveyselect procedure). For each randomly-generated gene list, tumors were separated into two groups using an unsupervised hierarchical clustering procedure that was based on a correlation matrix derived from standardized centroid components for cluster assignment (Varclus procedure). The two groups represent the first bifurcation of the clustering hierarchy. Random gene lists were also generated using subsets of genes that were identified as being associated with overall metastases latencies. The subsets of genes utilized in this analysis were those that had a relatively modest proportional hazards p-value of less than 0.1 in a Cox regression analysis.All survival analyses were based on 5-year metastatic recurrence latencies. All subjects not experiencing metastasis within 5 years were censored at that time point. Cox proportional hazards regression models were used to ascertain differences in latencies between groups assigned by hierarchical clustering (PHReg procedure). Univariate proportional hazard ratios and p-values are reported for all analyses and represent the differences in risk between the relatively "poor" prognosis group as compared to a relatively "good" prognosis group. In univariate analyses examining the association between ER status and latencies, hazard ratios represent the change in risk for ER positive tumors, as compared to ER negative tumors. Multivariable models adjusted for ER-α expression used the ESR1 probe in the NKI2 data set and the "205225_at" gene probe in other data sets.For each pair of clusters created by the hierarchical clustering procedure, the percentage of all ER negative and ER positive tumors contained in each cluster was computed. The maximum value of the ratio of the ER negative percentage to the ER positive percentage among the two clusters was then computed. If ER positive and negative tumors were assigned to clusters in equal proportions, this ratio would be approximately 1. Otherwise, this ratio would be greater than 1, indicating that the clusters contained a relatively disproportionate number of ER negative tumors, as compared to ER positive tumors.To eliminate correlations between all genes in the NKI2 data set and either ER-α or proliferation genes, the data were globally adjusted by fitting a least-squares regression line to each probe in the dataset and then computing the residuals (GLM procedure) . Each prThe total variance for a data set was computed by summing up the variances for each gene in the data set. To determine the variance explained by globally adjusting a data set for a gene, the total variance of the adjusted data set was computed and the percent change of this value relative to the unadjusted value was determined. The proportion of the proliferation-associated variance explained by adjusting for ER-α was calculated using the equation:where Var(EP) is the percent change in variance after sequentially adjusting a data set for ER-α and proliferation and Var(E) and Var(P) are the percent changes in variance after individually adjusting for either the ER-α or proliferation genes, respectively.All calculations were performed using SAS version 9.1 . All statistical tests were two-sided, and a p-value less than 0.01 was considered statistically significant.We sought to ascertain whether there may be intrinsic bias among publicly available breast cancer gene expression data sets that would influence the likelihood of observing a significant difference in metastatic tumor recurrence latencies based on gene expression patterns of primary tumors. To address this question, we employed an empirical approach whereby we determined the probability of identifying groups of tumors with statistically significant differences in recurrence latencies using hierarchical clustering that was driven by randomly generated lists of genes. Four previously published data sets were examined. Marked differences in the ability to segregate good and poor prognosis tumors were observed between the data sets using randomly generated gene lists of various sizes figure . The lowWe next assessed whether the differences among the data sets was the result of differences in the content of the genes represented on the microarrays used in each study. We performed clustering analysis using a set of 5,000 random lists (70 genes/list) that were comprised of genes that were common to all four data sets figure . For thiIn light of the known association between estrogen receptor (ER) status and disease outcome  and the Gruvberger et. al. has suggested that prognostic gene lists be tested independently on ER positive and negative tumors in order to control for the effects of ER-α gene expression on tumor assignment . We re-eIn summary, these results indicate that there is a large disparity in the likelihood of observing significant differences in metastasis risk among breast cancer data sets using arbitrary gene expression levels as a classifier. In particular, it was 4–5 times more likely to obtain a significantly predictive gene list in the NKI2 data set than the other data sets, even in stratified analyses examining ER positive tumors.The fact that multivariable adjustment for ER-α decreased the number of predictive gene lists in the NKI2 and TRANSBIG data sets suggests that the hierarchical clustering algorithm is functioning in large part by stratifying tumors into groups based on ER status. To confirm this, we examined the association between ER status and cluster assignment across the data sets. Specifically, we determined if there were disproportionate proportions of ER negative, as compared to ER positive, tumors in the different prognostic groups that were produced by the 5,000 random gene lists (70 genes/list) figure . In the While clustering is associated with ER status, it is not clear whether this association is due to the specific contribution of ER-α correlated genes, as has been suggested , or whetIn summary, these data show that ER-α correlated genes can serve to either increase or decrease the likelihood that a random set of genes will cluster tumors into groups with significant differences in latencies. The differences in the direction of the effect are related to the underlying association between ER status and the risk of metastasis in each data set.It has been previously shown that genes associated with cellular proliferation are strong predictors of metastases ,14,24-26While global adjustment for proliferation eliminated the predictive capacity of the random gene lists in the NKI2, Wang and TRANSBIG data sets, adjustment for ER-α had differential effects across these data sets. To better characterize the relationship between genes that are correlated with ER-α or proliferation and cluster assignment, we characterized the inter-relationships between these gene clusters. We first examined the proportion of the total variance in each data set accounted for by adjustment for either ER-α or proliferation correlated genes figure . In the It is important to note that the variation explained by the ER-α and proliferation-correlated genes is not independent. Thus, in contrast to the above similarities between the data sets, the proportion of the proliferation-associated variance that is explained by ER-α differs substantially figure . In the To gain more insight into possible differences between the NKI2, Wang and TRANSBIG data sets, we next examined the ER positive tumors. Amongst ER positive tumors, a random gene list was substantially more likely to give a significant result in the NKI2 data set, as compared to the other data sets. Among ER positive tumors, global adjustment for proliferation also eliminated the prognostic abilities of all of the random gene lists in the NKI2, Wang and TRANSBIG data sets figure . The conAnother explanation for the disparities in the probability of identifying significantly different classes of tumors with random gene classifiers between the data sets, especially the NKI2 and TRANSBIG data set which share many commonalities, is differences in the sample sizes table . The NKIIn the present study, we show that there are marked differences in the likelihood of observing a positive association between the expression patterns of a set of genes and metastasis across four breast cancer global gene expression data sets. Most striking was the observation that over 60% of randomly selected lists of 70 genes each could segregate tumors into two groups with significant differences in outcome in the NKI2 data set. This finding is particularly salient in light of the fact that this data set has been frequently used to both generate and verify prognostic lists of genes ,27-29. TGruvberger et. al., noting a strong association between ER status and outcome in the van't Veer data set , suggestWhile ER-α expression may be an important contributor to cluster assignment, it is not the primary determinant for obtaining a significant prognostic gene list. When each of the data sets was globally adjusted for proliferation, there were almost no significantly predictive gene lists, even though tumors tended to cluster by ER status. Genes associated with proliferation have been shown to be the essential contributors to an effective prognostic gene list in gene expression data sets of combined ER positive and negative tumors or just ER positive tumors ,14,26. WStratifying our analyses by ER status had differential effects on the data sets. In the NKI2 data set, almost 40% of gene lists were predictive in ER positive tumors, while less than 5% were predictive in the TRANSBIG data set. This result is interesting since both data sets had a similar proportion of events occurring among ER positive and negative tumors and ER status was associated with outcome in both data sets. When we examined the distribution of the hazard ratio estimates, which show the magnitude of the differences in rates of metastases of the tumors in the cluster pairs, the hazard ratio estimates tended to be larger in the TRANSBIG data set. This analysis suggests that clusters generated with the random gene lists tend to have larger differences in the metastasis latencies in the TRANSBIG data. Thus, the most likely explanation for the difference in the proportion of significant findings between these data sets is the larger sample size of the NKI2 data set. Other than sample size variations, these data sets were similar in many regards. The similarities may reflect commonalities in their patient populations, both of which were derived from young women in northern European countries.In contrast to the ER positive tumors, virtually none of the random gene lists was prognostic in ER negative tumors in stratified analyses. This observation may suggest that, among these data sets, there is not a large, dominant cluster of correlated genes associated with prognosis among these tumors. Another possible explanation is that there is more heterogeneity among ER negative tumors, which can be comprised on Her2 amplified tumors, basal as well as familial forms of breast cancer -32. We wet. al. showed in the van't Veer data set that the set of genes selected for a prognostic classifier is highly variable depending upon the training set used to identify prognostic genes [We found that, in general, random gene lists were less predictive of recurrence in the Wang data set. In analyses of all tumors, this effect could be partially attributed to the lack of an association between ER status and outcome in this data set, as described above. The random gene lists were also far less predictive among ER positive tumors in this data set, as compared to the NKI2 data set. This difference cannot be attributed to sample size differences as these data sets are approximately the same size. Like the TRANSBIG data set, the Wang data set is comprised of lymph node negative patients. While this difference could account for disparities between the Wang and NKI2 data sets (which utilized tumors from patients with node positive and negative disease), it would not account for differences between it and the TRANSBIG data set. Thus, the relevance of the difference in tumor composition with respect to this clinical variable is not clear. Another notable difference between the data sets is the older age representation among participants in the Wang study which could contribute additional heterogeneity amongst these tumors. It is also interesting to note that random gene lists containing genes that were independently predictive of metastases were only predictive approximately 55% percent of the time in this data set, versus 90% or more of the time in the other data sets . This might suggest that there are more independent clusters of genes predictive of prognosis in this data set. Consistent with this hypothesis, we have previously found a higher degree of correlation among independently predictive genes in NKI2 data set versus the Wang data set . Michielic genes . Based oThe KJX64/KJ125 data set produced considerably different results than the other data sets. Virtually none of the random gene lists were prognostic in this data set. It is important to note that it is possible to generate a large set of prognostic genes lists in this data set, as we found that 97% of random gene list derived from genes independently associated with metastasis were indeed predictive . In addition, this data set has also been used to generate other validated gene lists [In summary, these analyses demonstrate that there are systematic differences among the four breast cancer gene expression data sets examined herein. Most notable is the fact that the NKI2 data set provides a very high likelihood of obtaining a significant association between expression and metastases for a given set of genes. This disparity persists when examining all tumors in the data set or when looking only at ER positive tumors. The TRANSBIG data set behaved similarly to the NKI2 data set and the lower likelihood of obtaining a positive result in this data set is likely attributable to its smaller sample size. Fewer random gene lists are predictive in the Wang data set due to the lack of an association between ER status and the risk of metastasis. These differences in data sets should be considered when using them for developing new prognostic classifiers or assessing the robustness of classifiers that were generated with other data sets.The authors declare that they have no competing interests.JDM and RAK contributed equally to the design and implementation of this study. All statistical analyses were performed by JDM. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2407/9/214/prepub


In our previous study , we havehttp://bioinformatics.med.yale.edu/proteomics/BioSupp2.html.Our study shows improvement in classification accuracy upon expanding the mass range of the analysis. In order to obtain the best classification accuracies possible, we found that a relatively large training sample size is needed to obviate the sample variations. For the ovarian MS dataset that is the focus of the current study, our results show that approximately 20–40 m/z features are needed to achieve the best classification accuracy from MALDI-MS analysis of sera. Supplementary information can be found at  Proteomics is an integral part of the process of understanding biological systems, pursuing drug discovery, and uncovering disease mechanisms. Because of their importance and their very high level of variability and complexity, the analysis of protein expression and protein:protein interactions is as potentially exciting as it is a challenging task in life science research . Comparano information is used from the testing set to build a classifier from the training set. Internal CV is used in many current MS studies, whereby the selection of m/z features has utilized information from all the samples, which will substantially under-estimate classification error. In our previous study  and ∑i=1, …, g Pi = 1, i.e. it predicts the probability that sample  belongs to class k is Pk.We denote the RF classifier built from a learning set C = 1|y = 1). Specificity is also known as the true negative rate, which is the probability of classifying a sample as normal when it is actually from a normal sample, i.e. Pr = 2|y = 2). The relationship between sensitivity and 1-specificity is well known as a ROC curve in medical research.For our ovarian cancer dataset we only have two classes, cancer (y = 1) and normal (y = 2) samples. For two-class classification problems we can define sensitivity (θ) and specificity (η), which are inherently related to classification errors. Sensitivity is also known as the true positive rate, which is the probability of classifying a sample as cancer when it actually derives from a patient with cancer, i.e. Pr is a hard-decision classifier, we can estimate sensitivity and specificity using sample proportions,If The most commonly used classification error (Err) is estimated as1 and n2 are the sample size for the cancer and normal groups, 1-θ is classification error for the cancer group, and 1-η is the classification error for the normal group. If we have a very un-balanced sample set, i.e. n1>> n2 or n1>> n2, we can see that the previous definition of Err will encourage classifying all samples into the group with the larger sample size, which obviously is not the optimum approach. To avoid this problem we can use a balanced classification error definitionwhere nThis error definition gives equal weights to the cancer versus control groups and makes the subjective assumption that specificity and sensitivity are equally important. If the latter assumption is not optimal for a particular application, the above equation allows for easily changing the relative importance of specificity versus sensitivity depending upon the experimental context.In the case where we have a probability output, we first select a threshold α and then define the hard-decision classifier asWe can then estimate θ, η and Err similarly as beforeα ∈  As mentioned above, the relationship between The purpose of pre-processing, arguably the most important step in MS data analysis, is to reduce the effects of noise and to facilitate interpretation of the MS datasets. Before we submit the dataset to our final classifier, we carry out the following preprocessing steps: mass alignment, normalization, smoothing and identification of m/z data points that occur in peaks. The detailed pre-processing steps used in the current work are discussed in . More ad+ for the C12 isotope of the internal standard Bradykinin, which is 1060.569, as data point zero. A similar approach is then used with the linear data points which are aligned by assigning the m/z data point closest to the expected average MH+ of oxidized insulin chain B, which is 3496.95, as data point 100,000. Spectra are normalized , by determining a linear normalization factor for each spectrum which minimizes the summed difference between all intensities observed within the specified reflectron + linear mass range and the corresponding intensities in the overall, baseline-corrected median spectrum for all samples. Baseline smoothing begins by taking the natural log of all intensities. A sliding window of 1,000 data points is then used to determine the least squares, robust local polynomial fit curve that best represents the baseline. We then subtract the corresponding baseline intensity for the polynomial fit curve from each data point. Finally, we assume that only data points in completely or partially resolved peaks result from peptide or small protein ions and are likely to be meaningful. To be classified as occurring in a peak, at least 3 of 4 successive data point intensities before or after each candidate marker data point must show a progressive increase or decrease in background corrected, normalized peak intensity. Each peak data point is then extended by an additional 4 data points before/after the last data point that passed the peak test. Hence at this point we include a band width around each peak.Briefly, mass alignment is accomplished by numbering reflectron data points consecutively (in both a positive and negative direction) by assigning the observed m/z value that is closest to the expected MHSince we do not have a test set, CV was utilized to provide a nearly unbiased estimate of the classification error. The idea of CV is to randomly partition the original data into two parts: a training set used to build the classifier, and a testing set used to estimate the performance of the classifier. As discussed in , the comN and the number of selected m/z features M. In this study we have fixed the classifier to be RF. We will evaluate the impact of N and M on Err. Our strategy is to model empirically the functional relationship Err for a grid of values of N, M. For MS data the total number of features is usually very large, there are a total p = 130,000 m/z ratios for each of our ovarian cancer Linear + Reflectron spectra and the total number of selected m/z features is usually in the range of 10 ~ 100. In our study we evaluate Err for M ranging from 5 to 100. The total number of samples is usually very small compared to the total number of features. There are a total of n = 170 samples in our current ovarian cancer dataset. We need to carry out an extrapolation to estimate the impacts of N on Err. As discussed in [N, Err(N) = β 0 + β 1N−α, which is approximately true for large sample size data-sets . β0 is the asymptotic classification error and  are positive constants. Our current dataset has a relatively small sample size (n = 170) compared to its high-dimension feature space . Under this situation it is very dangerous to rely on the learning curve model to extrapolate β0, which corresponds to an infinite training sample size N = ∞. But within a limited range we can still rely on this model to extrapolate the classification error to full sample size n = 170. To estimate parameters , we need to obtain at least three observations. In our use of 5-fold CV to estimate classification errors, we first used one of the groups as a testing set, which produces a training set of N = 170/5*4 = 136 samples. We then use two, three and four of the groups as testing sets, which will give N = 102, 68, 34. For each N we estimate classification errors with M = 5, 6, …, 100 m/z features. We can estimate the learning curve based on these errors.It is obvious that Err depends on the underlying classifier, sample size M) for each M. Please see the supplementary website for numerical values of the learning curve parameter estimations.0.25 is the first quartile intensity, q0.5 the median intensity and q0.75 the third quartile intensity. We can clearly see the difference between cancer and normal groups. However, there is no single m/z feature that can completely distinguish cancer from normal groups; rather, there are considerable overlaps between these two groups. For some m/z features the normal group has higher intensities, while the cancer group dominates at other m/z features.One of the major interests in utilizing MS data is to identify important m/z features to build molecular diagnosis and prognosis tools. As discussed in , the RF For the 40 identified m/z features, some of them originate from the same peptide ion. In the following we show four unique and representative m/z features, which  upon which our classifications are based is by pre-fractionation of sera samples and the subsequent analysis of multiple spectra/sample.Another neglected aspect in many current studies is the justifications for numerically identified m/z features as being biologically meaningful. We agree that when classification errors begin to be sufficiently low to allow the corresponding m/z features to be clinically useful, then it would be advisable to identify the protein origin of those markers. Since we did not find any significant m/z feature that was specific to either the cancer or control sample spectra, we imagine that many of these markers may derive from proteins that are not involved in the disease process being studied and that they may prove to result from secondary effects . While these types of m/z features may have little or no value in furthering our understanding of the disease process itself, this would not lessen the value of these m/z features for properly classifying unknown samples. It also is important to point out that we do not currently know if the m/z features whose relative intensities enabled a >80% success rate at classifying ovarian cancer versus control sera samples are indeed specific to


Detection of discriminating patterns in gene expression data can be accomplished by using various methods of statistical learning. It has been proposed that sample pooling in this context would have negative effects; however, pooling cannot always be avoided. We propose a simulation framework to explicitly investigate the parameters of patterns, experimental design, noise, and choice of method in order to find out which effects on classification performance are to be expected. We use a two-group classification task and simulated gene expression data with independent differentially expressed genes as well as bivariate linear patterns and the combination of both. Our results show a clear increase of prediction error with pool size. For pooled training sets powered partial least squares discriminant analysis outperforms discriminance analysis, random forests, and support vector machines with linear or radial kernel for two of three simulated scenarios. The proposed simulation approach can be implemented to systematically investigate a number of additional scenarios of practical interest. Detection of discriminating patterns in gene expression data can be accomplished by using various methods of statistical learning. Such patterns are of interest as candidate biomarkers/biosignatures to classify samples, for example patients eligible for treatment or not. The Biomarkers Definitions Working Group defines a “biological marker” (biomarker) as a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacological responses to a therapeutic intervention . For cliIdeally, biomarkers alone or in combination (as biosignatures) allow consistent classification of an individual to a predefined group. Differential regulation of features, however, does not generally qualify such features as good biomarkers—often validation shows poor classification results for features which were found to be significantly differentially regulated . Given In our study we address an additional problem: Often small amounts of RNA per sample make sample pooling necessary to enable microarray screening experiments. However, sample pooling possibly compromises the analysis results, especially regarding detection of biosignatures: An Affymetrix technical note, for example, points out that once RNA samples are mixed, it is impossible to identify outliers or misclassified samples . As a maIn our study we ask for the actual influence of pooling on the possibility to find multivariate discriminating patterns for the classification task and the dependency on differential expression, technical noise, and choice of method. As this question has not been studied using concrete examples, we set out to develop a simulation framework eligible to consider the following questions: how are multivariate patterns altered by pooling, and which methods are sensitive to pooling designs, which are robust?g in sample i by Xg,i. We assume for each g ∈ {1,…, G} that Xg,1,…, Xg,n are independent and identically distributed random variables with mean μg and variance σ2.We consider a gene expression experiment and denote the expression of gene n denotes the number of individuals and σ2 represents the biological variance between the subjects. We get the following gene-wise model for the measured value of the gene expressionHere ɛi, ɛi ~ N.Xg,i is normally distributed and the pool size is mp, 1 ≤ mp ≤ n, it follows that the gene expression of a pool p  (1 ≤ p ≤ n/mp) has the distribution Xgp ~ N. As a measured value of a pool p we getWe only consider the case of an ideal pool where each individual contributes equally to the pool; therefore, the expression level of a pool is the average of the individuals which form the pool (For details see subsection Pooling design). If ɛp ~ N, as in , as in .nP denotes the number of pools. The variance of σɛ2 and σ2; the variance of n = mpnP.We compare two designs, a single sample design and a pooling design. The variance of ze); see . When weA and B. A pattern, as we understand it, allows a partion of a feature space into regions belonging to the classes of samples.To exemplify multivariate patterns we consider patterns which were built out of two features for two classes, A and B. Our data matrix has 60 rows (corresponding to the subjects) and 1000 columns (corresponding to the features). The 60 rows are partitioned in 30 rows per class. The matrix contains an informative part and a large noninformative part with respect to the classification. In the noninformative part the values are normally distributed random variables with mean eight and variance 0.22.Simulating two designs, a design of a pooled study and a design of a study with single samples, we consider the influence that pooling has on a classification task. Data were generated randomly for two classes A and B. For the second scenario we simulate linear dependent genes (details below for the linear pattern). For the third scenario we combine both independent differentially expressed genes and linear patterns.We investigate three scenarios for the informative data part. In the first scenario we only simulate ten of 1000 genes as independent differentially expressed between the two classes μA of class A is randomly chosen (according to the uniform distribution) from the interval . The 30 gene expression values which belong to the subjects of class A are N-distributed random samples. The gene expression values of the class B individuals are N-distributed, where Δ is randomly chosen according to the uniform distribution on the interval . Technical noise is added with the two variance levels 0.22 and 0.42.For the first scenario we simulate ten independent differentially expressed features. This submatrix forms the first ten columns of the simulated 60 samples × 1000 features data matrix. For each of the ten genes we proceed in the following way. The mean Xg1,iA and Xg1,jB with i, j = 1,…, 30, both with variance σ2 = 0.22 and a mean randomly chosen according to the uniform distribution on the interval  form the first component of the pattern, which is built without loss of generality of the genes g1 and g2. We calculate the values for Xg2,iA in the following way:For the second scenario we simulate a two-dimensional pattern. The normally distributed ɛi we get Yg2,iA = 2Xg1,iA + ɛi for all i. By analogy we calculate the values for class BYg2,jB = 2Xg1,jB + δ + ɛj for all j. We use δ = 0.4. Again we evaluate two noise levels with variances 0.22 and 0.42. Thus we simulate pairwise linear dependent genes which are illustrated in For the third scenario we implement these two scenarios in combination, simulating again ten independent differentially expressed features and ten linear patterns.We simulated 500 training sets as described above. As test sets, for each repetition, we simulated again a 60 × 1000 data matrix with 30 rows per class under the same conditions. Comparing a single sample design and a pooling design for each scenario by accounting for the two noise levels, we report the prediction errors of five statistical learning methods. Here we understand as prediction error the relative frequency of wrongly classified samples of the test set.A and the black circles the subjects of class B . In the left scheme a single sample is measured on an array. The right scheme shows that pooling takes place before the samples are analysed. Therefore we transform the simulated normally distributed data by exponentiating the data with basis two and pool this transformed data. After the pooling step we transform the data back on the log scale again by taking base-two logarithms of the values. We consider three different pool sizes mp = 2,3, 5. In each validation step the individuals of a pool were randomly chosen without replacement. For choice of sample size we account for n = mpnP. Every individual sample only contributes to one pool and all original samples were used for our classification task.In the pooling design the methods are trained with the pooled samples (training set) and tested with independent new single samples (test set).We use five different statistical learning methods.Support vector machines with a linear kernel (SVML). Support vector machine (SVM) classification is based on large-margin separation and kernel functions . Support vector machines with a radial basic kernel (SVMR). The radial kernel is an example for a nonlinear classification approach. For both we use the R-package e1071  with a 1Random forests: random forests (RF) is an ensemble method which is based on decision trees and makes a classification using majority votes over all trees . We chooPowered partial least squares discriminant analysis: powered partial least squares discriminant analysis (PPLS-DA) is an advancement of powered partial least squares (PPLS) for highly collinear predictors with Fisher linear discriminant analysis (FLDA). The influence of not so important predictors can be reduced in this approach by restricting the power parameters like in the powered partial least squares (PPLS) approach . The lint-test and after choosing the ten best candidates with regard to the P-value we use these t-test candidates for a linear discriminant analysis (LDA).Linear discriminant analysis: As a univariate method we use the For the methodological comparison we consider prediction errors of these five methods in a pooled and a single sample design. Confidence intervals are calculated taking a Bootstrap approach  as impleOur codes are available on request.We compare three different scenarios. In the first scenario ten independent genes are simulated with mean value differences between the two classes, in the second scenario ten linear dependent genes discriminating the groups are built, and in the third scenario we combine both.A and B is randomly chosen according to the uniform distribution on the interval .t-test candidates, and for pools of size mp = 1,2, 3,5  by accounting for two noise levels. For the right panel noise with a variance of 0, 42  was added and for the left panel less noise with a variance of 0.22 . The height of the bars represents the mean rate of wrong predicted samples for 500 validation runs, in addition the 95% bootstrap confidence intervals are given. With increasing size of the pools the prediction error increases over all methods. For the single samples (mp = 1), pools of size two and three, LDA after t-test makes the lowest prediction error with around 0.03, 0.09, and 0.19. But the prediction error of LDA almost doubles for each step in size of the pools. Considering PPLS-DA for all sizes of pools the prediction error is under 0.25. The results of the two support vector machine methods are similar with prediction errors between 0.23 and 0.38. For the higher noise level all methods except SVMR show an increase of the prediction errors particularly when comparing the single sample design with a design of pools with size mp = 2. For both noise levels in the scenario of independent differentially expressed genes LDA shows the highest increase of the error rate while stepping up size of pools. In the case of a pooling design with mp = 5 and for a noise level of 0.22 the prediction error of LDA is more than 12 times higher as in the single sample design. For a technical noise of 0.42 the LDA prediction error increases nearly by four times between the single sample design and the pooling design with mp = 5. In the first scenario the difference between the mean values of class 2 and 0.42 for single samples and pools with the different sizes. The construction of the linear pattern causes also a mean value difference between the two classes of 0.4 for the second gene. The linear pattern is persistent to pooling.mp = 5. Especially SVMR in the higher variance cases has only up to three more misclassifications. Regarding only the right plot of mp = 5. Again LDA reacts with the strongest increase viewed over all pool sizes for both noise levels. PPLS-DA performs in average superior to the other methods for this scenario with prediction errors lower than 0.22 and less than 0.34 for the variance levels similar to the biological variance and four times higher as the biological variance. In 2. Viewed over all settings for this scenario PPLS-DA shows the lowest prediction error. For the different noise levels the prediction error we found behaves very similarly for SVML, SVMR, and RF. The result of the third scenario, the combination of independent differentially expressed genes and the linear pattern, is shown in We propose a simulation framework to explicitly investigate parameters of pattern, pool size, noise, and choice of method and their influence on classification performance. We exemplify our simulation approach by using a two-group classification task and simulated gene expression data with independent genes and linear dependent genes. Our results show a clear increase of prediction error with size of pool when keeping the total number of samples constant, for all five methods investigated. PPLS-DA outperforms the other methods for the linear pattern and the combination of independent differentially expressed genes and the linear pattern for all pooled sample designs  will be approximately normally distributed . As ZhanIndependent differentially expressed features were simulated using Δ between 0.1 and 0.5 on the log-scale—which is equivalent to between 1.07- and 1.4-fold changes in gene expression. These values for differential gene expression are comparatively small (compare to example ). Our chWe want to point out that for investigation of pooling effects on classification abilities of different methods, noise regimes and data with independent features with distinction of mean values or with pairwise linear dependent features, it is important to choose a model simple enough to allow investigating the impact of the single effects. It remains an open task to approach possibilities to analyse experimental data in a similar way.γ.Internal cross-validation loops for the training step were necessary where the methods had additional parameters to optimise . For PPLIt is certainly a demanding future work to fit further realistic simulation scenarios to experimental data and draw conclusions for methodological recommendations for data analysis. However, to date it is unclear how to determine to which extent experimental data consist of univariate differentially expressed genes, and how much is the result of regulatory interactions, that is, multivariate patterns. It would be conceivable to “substract” differentially regulated genes from a dataset to result in a scenario with only multivariate discriminating patterns remaining. Our choice of pattern is largely arbitrary, many more alternative possibilities exist, also truly multivariate patterns where the number of interacting features exceeds two. Hence the pattern dependency of pooling effects remains to be investigated. Similarly our choice of methods is an arbitrary subset from what is currently applied for classification in OMICs datasets. Here we look forward to further implementations of our simulation approach using further methodological alternatives, possibly also for investigating why some methods seem to be sensitive to sample pooling while others are not. Feature selection performance would be a further interesting characteristic—in addition to pure predictive error—to compare different methods using our proposed simulation framework: Does pooling alter the possibility to find the correct, responsible discriminating features?Summarising we want to propose our approach to investigate effects of sample pooling on classification performances for different datasets  and classification methods. As for many occasions pooling cannot be avoided, we look forward to methodological recommendations to analyse such designs with respect to detection of biosignatures for prediction, based on systematic investigations using our proposed framework.


The prognostic value of these signatures was tested in five large clinical microarray data sets. More than 1000 patients with breast, renal or lung cancer were included. One of the signatures (110 genes) had significant prognostic value in all data sets. Stratifying patients in groups resulted in a clear difference in survival . Multivariate Cox-regression analyses showed that this signature added substantial value to the clinical factors used for prognosis. Further patient stratification was compared to patient stratification with several well-known published signatures. Contingency tables and Cramer's V statistics indicated that these primarily identify the same patients as the proliferation signature does. The proliferation signature is a strong prognostic factor, with the potential to be converted into a predictive test. Furthermore, evidence is provided that supports the idea that many published signatures track the same biological processes and that proliferation is one of them.Tumour proliferation is one of the main biological phenotypes limiting cure in oncology. Extensive research is being performed to unravel the key players in this process. To exploit the potential of published gene expression data, creation of a signature for proliferation can provide valuable information on tumour status, prognosis and prediction. This will help individualising treatment and should result in better tumour control, and more rapid and cost-effective research and development. From  The abilities to predict outcome and to identify key players in biological mechanisms that lead to poor outcome are two important objectives in cancer research. Recently, efforts to exploit gene expression profiling have been made to identify gene sets, or so-called gene signatures, that can improve diagnosis and risk stratification . Furtherin vitro gene expression profiling is used to identify gene sets that play an important role in a specific biological process. The identified gene set is then applied to gene expression data from patients to evaluate its prognostic value. This approach has a more broad application because the gene sets can be applied in almost every patient group. Furthermore, it can be used not only to investigate whether a certain process is important in a distinct cancer type or patient group but also potentially to select patients in those groups that would be expected to benefit from therapies directed to the biological process of interest  and Stanford Microarray Database (SMD) or elsewhere. Accessory clinical and follow-up data were also given or provided by the authors on request. In Expression data of the genes in the signature was extracted from the data set. The following step was used to calculate a signature score for each patient in the data set. This score was defined as the weighted average expression value of the genes in the signature . A weighN, number of genes in the signature; i, gene; wi, weight of gene i; expi, gene expression of gene i.The signature score then reflects the status of the studied process in a tumour. When a gene was represented by more than one probe on an array, the expression of the probes was averaged before signature calculation. In P-values, also the percentage of P-values from the 1000 clustering runs that were significant was calculated to evaluate the prognostic power of the signature and stability of the clustering.A loop of 1000 clustering repeats with the K-means clustering function in Matlab was applied to split the patients in two groups according to their signature score. Outcome in the two groups was analysed and compared by the Kaplan–Meier method. Differences in outcome were tested for statistical significance by the log-rank test for different common end points. For breast and renal cancer, the common end points are 5- and 10-year survival, and for lung cancer, these are 2- and 5-year survival; all end points were analysed when follow-up was long enough. Results for the log-rank tests are given as the average, standard deviation and the range of the Multivariate Cox-regression analysis with stepwise backward selection procedure was performed in SPSS  to show the clinical relevance of the signature.Matlab was used to integrate all parameters in a model with and without addition of the signature to the clinical parameters. Differences between the models were assessed using receiver–operator curve (ROC) analysis by calculating the AUC. Further details are provided in the A method to test a predefined number of random signatures of a predefined size on all the data sets was developed. To show the strength of a signature, 10 000 random generated gene sets, with sizes equal to the size of the signature of interest, were tested on the data sets. These random gene sets were tested in a similar manner as the other signatures.Group 1: 0–5 mitoses in 10 high power fields.Group 2: 6–10 mitoses in 10 high power fields.Group 3: ⩾11 mitoses in 10 high power fields.Mitotic index was assessed  in the 295 breast tumours of the From published microarray studies two different proliferation signatures were compiled. Another method to derive a proliferation signature with microarrays was used by in vitro data as the wound signature. Therefore, these signatures were also analysed.The wound  was calculated for each clinical parameter and the best proliferation signature. Results of this analyses show that the proliferation signature has a high AUC compared to the clinical parameters in all data sets .P-values paired t-test ≪0.0001).To quantify the gain in prognostic power obtained with this signature, a model of the clinical factors with and without the signature was generated and evaluated with the AUC. Part of the data set was used as training set, to generate the model, and the other part as a test set. Only the data sets with more than 1 clinical parameter and more than 150 patients are included. Different sizes of training and test sets were evaluated; the overall performance did not change significantly (data not shown). The results shown in To show the strength of the proliferation signature, 10 000 random generated signatures were tested on all data sets. Of these 10 000, no signature gave a significant result on all data sets.Log-rank tests and Kaplan–Meier survival curves show that the wound and IGS signature give clear risk stratification in four and five data sets, respectively . FurtherTo investigate whether different signatures identify the same patients, two-way contingency table analyses  were perin vitro data; however, this does not necessarily mean that the signature truly tracks proliferation in vivo. To investigate this, mitotic index (MI) was assessed for the vs the MI. There is a clear correlation between the average proliferation signature score and the three classes of MI (correlation coefficient: 0.968).The proliferation signature is extracted from in vitro microarray studies based only on genes that differ in expression in different parts of the cell cycle  calculation. Overall results of these single-parameter indicators are disappointing (In conclusion, we have shown that the application of phenotype-based signatures such as the proliferation signature can be used in patient risk stratification, in addition to clinical parameters. It has a high prognostic value and unlike other signatures it has the potential to be converted into a predictive test. Furthermore, we provide evidence that supports the idea that many published signatures track the same biological processes and that proliferation is one of them. Whether the proliferation signature can be converted into a predictive test should be evaluated in a large prospective trial in which other measures for proliferation should also be evaluated.


Large repositories of biomedical research data are most useful to translational researchers if their data can be aggregated for efficient queries and analyses. However, inconsistent or non-existent annotations describing important sample details such as name of tissue or cell line, histopathological type, and subject characteristics like demographics, treatment, and survival are seldom present in data repositories, making it difficult to aggregate data.We created a flexible software tool that allows efficient annotation of samples using a controlled vocabulary, and report on its use for the annotation of over 12,500 samples.While the amount of data is very large and seemingly poorly annotated, a lot of information is still within reach. Consistent tool-based re-annotation enables many new possibilities for large scale interpretation and analyses that would otherwise be impossible. Quantitative gene expression experiments provide key information for elucidating biological pathways and understanding diseases. Many methods have been developed over time, from hybridization based, Northern blotting, real-time polymerase chain reaction, high throughput microarrays and serial analysis of gene expression (SAGE), up to modern synthesis based sequencing methods . Microarrays are a popular technology for large-scale measurement of gene expression. Substantial public repositories have been set up to capture the wealth of information on gene expression generated by researchers world-wide: the National Center for Biotechnology Information's Gene Expression Omnibus (GEO) , the Eurraw measurement values is almost inevitable.Due to the public availability of large microarray repositories and reasonable standardization of measurement values, many 'meta-analyses' and meta-analysis systems emerged ,8. TheseHowever, analysing the raw data is challenging, as it requires (1) consistent and high quality probe annotations for all involved platforms, (2) appropriate cross-platform normalization methods, and (3) detailed sample annotations. Only few were able to report success with this approach . While mWe addressed the problem of annotating gene expression samples with a consistent set of variables on a large number of existing studies. While this seems to be a daunting task, there were sufficient similarities between samples to make it possible to annotate many samples at once. Moreover, we hypothesized that it was possible to obtain high quality annotations by non-expert individuals who received proper training.Previous manual or automatic annotation attempts relied or tried to rely on domain experts to do the annotation , which wIn the past, several groups proposed annotation tools for microarray experiments -18. ThesOur database currently contains the 45 most popular platforms from GEO, 2,445 studies, and 58,432 samples. We also imported a total of 1.6 billion raw measurement values that can be used for new analyses with the help of a cross-platform probe annotation and a cross-platform normalization tool.We were able to efficiently annotate more than 12,500 samples. More than half of these samples have been redundantly annotated by at least two different annotators. Within four weeks of work of one full time and one part time annotator and a following five weeks of work with four full time annotators, a total of almost half a million variable assignments were made. On average, every sample received 32 annotations.The most frequently available variables were 'tissue' with 24,602 assignments, followed by 'disease state' with 12,098 annotations, 'sample type' with 9,792 annotations, and 'cell line' with 11,925. Information about genetic modification is also readily available in 11,135 annotations. Other frequently available variables were 'treatment', 'time series', 'gender', 'patient age', 'lymphatic spread', 'estrogen receptor status' and 'tumor type'.virtual studies. Another possible extension is the employment of this or a similar system directly at the public repositories so the uploaded data is annotated by the original submitter.Many possibilities of further processing are within reach with a method at hand that facilitates an efficient and rich annotation of existing gene expression data. One obvious use case is the regrouping of existing samples to perform new The potential benefit of these detailed annotations is clear: While finding a suitable set of samples across studies was virtually impossible using existing unstructured information alone, we are now able to easily find and compare, for example, 4,405 breast cancer samples with 473 normal breast samples, or 2052 ER+ versus 284 ER- samples. Figure We have shown the feasibility of collecting publicly available information about previous microarray experiments and subjecting them to a consistent and efficient annotation. With the current speed and quality of annotation and a total of ten full-time annotators we project the time to annotate all of GEO  with two redundant annotations per sample to be 50 weeks. In the future, we plan to assemble new large datasets and perform new differential expression analyses, avoiding the high cost of sample collection, preparation and hybridization by exploiting existing data.The annotation process started with the identification of possible studies for a certain disease by doing a keyword search. This search yielded a list of studies that had the keyword somewhere in their description or in the description of their samples. Then, annotators proceeded study by study, looking through available information in the local database and following publication links and their respective supplements. After going through all this information, the annotators usually had a good idea of what kind of samples they would subsequently be looking at. Armed with this, they proceeded sample by sample through each study.Most of the annotation was conducted by one graduate and three senior biology students. An initial pilot annotation was done by one senior biology student and a physician. From this pilot annotation we obtained a feasible set of variables for large scale annotation. In a separate report  we proviThe first step towards structured annotation was importing large parts of GEO into a relational database by parsing the SOFT files . While tOur database has a straightforward structure and captures just the essentials, like title, description, and organism for sample and studies. This information is the same for all popular expression databases like ArrayExpress or CIBEX, and the importer can be adapted to pull data from these sources.We created a custom web front-end that showed the most consistently present fields and linked back to the original GEO entries. This web-platform was subsequently extended as a platform for uniform sample annotation. Moreover, we re-annotated probe-to-gene mappings of most of our 45 locally available microarray platforms using AceView  as a uniOne of the major difficulties was finding suitable variables that were frequently available and uniquely iden-tified concepts, while keeping the number of variables low. A concise and complex medical nomenclature had to be clearly separated into distinct terms and predefined choices, enabling non-expert annotators to comprehend existing descriptions and correctly label the described samples.The underlying idea of the annotation system was to freely attach variables to samples, allowing any variable to be used for any sample. New variables could also be created and assigned if necessary. However, after the initial pilot study, a predefined set of variables with a predefined range of values was established for every disease through discussion and evaluation by experts. This encouraged consistent use of variables and values while at the same providing some freedom to capture seemingly important additional information. To further promote consistency between annotators, newly created variables and non-predefined values were visible to all annotators, albeit less prominently displayed than predefined variables in order to avoid cluttering the interface. This helped us keep a consistent naming scheme even for less important variables in case this information was also available. This guided yet flexible system proved extremely useful as we extended our annotation efforts to cover more diseases.To facilitate faster annotation of frequently used values, new values could later be added to the set of predefined values and were then directly visible. Predefined variables were grouped into the categories 'essential', 'patient', 'treatment', and 'sample'. Variables were categorized according to individual diseases they appeared in. So far we created these sets of variables and annotated samples for breast cancer, colon cancer, insulin dependent diabetes mellitus, rheumatoid arthritis and systemic lupus erythematosus.Definition of new variables and possible values is performed using a text format. After review, these are added into the active list of variables by running a script on the text file. This way it is also possible to create new annotation forms that consist of existing variables using the variable definition format shown in Table We built a web-based annotation platform to collaboratively create sample annotations in our expression database. Figure The right side of Figure sample id, variable id, value) inside our relational database that can be used to check for concordant annotations, extract samples with certain properties and their associated gene expression values.The internal format of the annotations was a list of tuples of the form (The authors declare that they have no competing interests.EP, JK, PG and LOM acquired the original data, performed data interpretation and guided the import process. All authors were involved in the system design and conceptualization of the user interface. EP implemented all necessary software systems and prepared the draft of the manuscript. RL, CH, LOM and EP selected an initial set of variables. RL, CH and LOM decided about further inclusion of variables after the initial pilot study. CH participated in the pilot annotation. RL supervised the annotation process and interpreted the quality of the annotation. RL and LOM contributed substantial revisions to the manuscript. PG, JK and CH reviewed the software and provided significant feedback for its further development. All authors read and approved the final manuscript.


The aim of this work was to study the performances of 2 predictive statistical tools on a data set that was given to all participants of the Eadgene-SABRE Post Analyses Working Group, namely the Pig data set of Hazard et al. (2008). The data consisted of 3686 gene expressions measured on 24 animals partitioned in 2 genotypes and 2 treatments. The objective was to find biomarkers that characterized the genotypes and the treatments in the whole set of genes.We first considered the Random Forest approach that enables the selection of predictive variables. We then compared the classical Partial Least Squares regression (PLS) with a novel approach called sparse PLS, a variant of PLS that adapts lasso penalization and allows for the selection of a subset of variables.All methods performed well on this data set. The sparse PLS outperformed the PLS in terms of prediction performance and improved the interpretability of the results.We recommend the use of machine learning methods such as Random Forest and multivariate methods such as sparse PLS for prediction purposes. Both approaches are well adapted to transcriptomic data where the number of features is much greater than the number of individuals. One statistical method that can be considered is discriminant analysis, where the phenotype is modelled as a linear combination of a subset of gene expressions. However, in the case of transcriptomic data, gene expressions are highly correlated, leading to multicollinearity problems in discriminant analysis. Furthermore, the high number of variables limits the use of this method. To circumvent this problem, one can pre-select the variables, or use approaches that can deal with a large number of variables.Often, an important goal of transcriptomic analyses is to identify differentially expressed genes; the expression level of each gene is explained by the phenotype in a linear model setting . Another important goal is to find biomarkers, In this study, the objective was to assess the behaviour of 2 prediction methods on one data set ,2, wherek are constructed as follows:Random forests  is a cla1,..., BN) are drawn from the original data.- N bootstrap samples  is used as a training set to construct an unpruned tree Tk. Let p be the input variables of the tree. For each node of Tk, m variables are randomly selected (m<<p) to determine the decision at the node, where m is constant during the forest growing. Then the best split among these m predictors is chosen to split the node.- Each sample BThe predictions of the N trees are then aggregated to predict new data by majority vote for classification or by average for regression. Random forest avoids the need to perform a separate cross validation test to estimate the prediction error of the forest when performing a classification or a regression. While the forest is constructed, it generates an internal estimation of the generalisation error as follows:k, about one-third of the cases are left out of the bootstrap sample and are not used in its construction. These data are called "Out-of-bag" or OOB data and are used as an "internal" test set for each tree that is grown.- While constructing each tree T- The OOB predictions are then aggregated and the error rate, called "OOB error estimate" is computed for the whole forest and should lead to an accurate and unbiased generalisation error .The Mean Decrease Accuracy measure was used in this study as a feature selection criterion, where the OOB data are used to obtain estimates of variable importance by evaluating their contribution to the prediction accuracy. The values of each variable in the OOB cases are randomly permuted and are run along the tree. The proportion of cases in the correct classes with permuted OOB data is then subtracted from the proportion of cases in the correct classes where OOB data have not been permuted. The Mean Decrease Accuracy averages the difference between these two accuracies over all trees in the forest and normalizes it by the standard error.n × q matrix of phenotypes with the 4 columns indicating each combination  for the 24 animals, and let X be the n × p matrix of the 3686 gene expressions measured on the 24 animals. Partial Least Squares regression (PLS) was introduced by Wold [Let Y be the p- and q- dimensional vectors uh and vh for each PLS dimension h and are respectively associated to the X and Y data sets. The associated latent variables are defined as ξh = Xuh and ωh = Yvh. As in PCA, the loading vectors uh and vh are directly interpretable, as they indicate the importance of the variables from both data sets in relation with each other. The latent variables ξh and ωh, that are n-dimensional vectors contain the information regarding the similarities or dissimilarities between the individuals or samples [n<<p problem. Lê Cao et al. [The loading vectors are the  samples . PLS is o et al.  proposedo et al.  that peno et al. .H and the number of predictive genes to select on each dimension of sPLS: the Root Mean Squared Error Prediction (RMSEP), and the Qh2 that measures the marginal contribution of each latent variable to the predictive power of the PLS model. Briefly, 1-Qh2 is the ratio of the average PRediction Error Sum of Squares to the average of the Residual Sum of Squares, over the variables  does not require fine-tuning of its parameters. In this study, however, random forest classifications with 10000 trees were performed in order to obtain stable results. When applied to the whole data set (3686 genes), RF gave a reasonably high prediction power, with an OOB estimate of error rate equal to 12.5%. After a pre-selection of differentially expressed genes (662 genes with a FDR < 20%), the prediction was perfect (with 0% of the OOB estimate of error rate).Comparing the significant level of genes .Hierarchical clustering is widely used as a statistical tool for microarray data to look for similarities between genes and samples in an unsupervised way. Hierarchical clustering using the Ward method and Euclidian distance  were thuH to be retained was estimated with the Qh2 criterion, for which a value below the threshold 0.0975 indicates a significant contribution for the prediction purpose [2 values calculated for each dimension of the PLS and the sPLS showed that 2 dimensions were enough to capture the whole information for both PLS or sPLS. An equivalent coding for Y is the 2 column matrix of genotype and treatment factors that will be considered in the following.Recall that Y is the phenotype matrix of the 4 indicators  for each animal. The number of dimensions i.e. the lowest RMSEP obtained was 10 genes on each dimension. In this case, the sPLS gave better predictions than PLS (not shown).The number of dimensions being fixed to 2, the optimal number of genes selected on each dimension , was determined with the RMSEP for both sPLS and PLS. The optimal result, Figure Figure Figure The list of significantly expressed genes (t test) did not exactly match with the list of sPLS predictive genes. This shows that the information captured by the 2 approaches may bring complementary as well as relevant results.Due to the clear structuring of the data, it is difficult to compare the performances of the statistical prediction approaches. A thorough biological interpretation of the results is now needed to validate the use of these methods.q > 1 in the Y matrix, few other approaches have been developed for variable selection and integration of two-block data sets based on elastic net procedure [i.e. a symmetric relationship between the data sets, which is not the case in this study. The reader can refer to [In the case where rocedure  or shrinrocedure . Howeverrefer to  (Canonicrefer to  (Co-inerrefer to ) for bioq = 1, as performed with RF when we combined the phenotypes in one class vector, we find ourselves in a typical multiclass problem. Several approaches have been developed for feature selection, among them the reader can refer to Recursive Feature Elimination [In the case where mination , Nearestmination  or Optimmination , that caThe differential analysis in , and theIn microarray data, the statistical criteria are often limited by the small number of samples. Therefore, it is strongly recommended to combine statistical assessments with a sound biological interpretation of the data, as was shown for example in . They shThe authors declare that they have no competing interests.KALC developed the sparse PLS method, and wrote the corresponding R code. CRG and MSC analysed the data set. All authors participated to the redaction work, read and approved the final manuscript.


Patients with primary operable oestrogen receptor (ER) negative (-) breast cancer account for about 30% of all cases and generally have a worse prognosis than ER-positive (+) patients. Nevertheless, a significant proportion of ER- cases have favourable outcomes and could potentially benefit from a less aggressive course of therapy. However, identification of such patients with a good prognosis remains difficult and at present is only possible through examining histopathological factors.Building on a previously identified seven-gene prognostic immune response module for ER- breast cancer, we developed a novel statistical tool based on Mixture Discriminant Analysis in order to build a classifier that could accurately identify ER- patients with a good prognosis.We report the construction of a seven-gene expression classifier that accurately predicts, across a training cohort of 183 ER- tumours and six independent test cohorts , ER- patients of good prognosis  independently of lymph node status and treatment.This seven-gene classifier could be used in a polymerase chain reaction-based clinical assay to identify ER- patients with a good prognosis, who may therefore benefit from less aggressive treatment regimens. Oestrogen receptor (ER) negative (-) breast cancer accounts for about 30% of all breast cancer cases and generally has a worse prognosis compared with ER positive (+)disease ,2. NeverRecently, attempts have been made to explain the observed clinical heterogeneity of ER- disease in terms of gene expression signatures -7. HowevThe important role that immune system-related gene expression signatures play in breast cancer prognosis has been further supported by four recent reports -16. SpecThe discovery and construction of a molecular classifier that can robustly identify ER- patients with a good prognosis is important for two main reasons. First, identification of ER- patients with a good prognosis based on histopathological predictors like LN status or Adjuvant! is far from optimal . Second,X of dimension p × N, where p is the number of dimensions  and N is the number of training samples . We also assume that we have a test set Y of dimension p × n and that we have C phenotype classes among the training set samples.Before discussing Mixture Discriminant Analysis (MDA), it is convenient to briefly review Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) . We assuπ, θ) =  such that the likelihood functionIn the training process of discriminant analysis one attempts to learn parameters that specify the clusters associated with each of the phenotype classes. In the maximum likelihood framework, one learns parameters , which by Bayes' Theorem can be written asHaving estimated the parameters, we can now classify a test sample y to the class which maximises this posterior probability (the maximum probability criterion) minimises the expected misclassification error. Thus,Assigning fk or, if the functional form is prespecified, the parameters θk. The simplest functional approximation one can make is to assume that the clusters are multivariate Gaussians, so thatTo compute the posterior probabilities one needs to estimate the functions μk is the mean and Σk the covariance matrix of the Gaussian. If, furthermore, we assume that the covariance matrices are identical for each cluster , then the classification function becomes a linear function of y, known as LDA. In the more general case where the covariance matrices of each class are allowed to differ, the classification function is a quadratic form of the y and the analysis is known as QDA.where fk as a mixture of multivariate Gaussians, since any general density can be approximated by such a mixture. Therefore, one assumes thatThe assumption that a phenotype class is best modelled by a multivariate Gaussian is often violated. In the context of gene-expression analysis, gene expression profiles often exhibit bi-or multimodality, even when restricted to one phenotype class . Similark is given by Gk. This number may or may not be specified in advance resulting in a variety of different implementations. In ordinary MDA [Gk is known in advance for each class k and that the covariance matrices are all identical . However, these assumptions are not necessary and instead one can use the training data to learn the best mixture model fit for each phenotype class using for example the Bayesian Information Criterion (BIC) [τkj,μkj,Σkj, Gk), from which classification of test samples proceeds as before using the maximum probability criterion. Therefore, MDA is a direct generalisation of LDA and QDA and may reduce to these if the data does not support multiple components per phenotype class [where the number of Gaussians to use for phenotype label nary MDA , one asson (BIC)  or a varon (BIC) . This mope class .C phenotype classes, whereas in certain circumstances we may be only interested in classifying into certain subtypes within the phenotype classes. Therefore, while in MDA one allows for heterogeneity of each phenotype label by estimating the density of each class as a mixture of Gaussians, classification is subsequently performed into each phenotype class. On the other hand, it is possible to classify samples into the Gaussian subcomponents inferred for each phenotype class, a variation of MDA called Heterogeneous Mixture Discriminant Analysis (MDAhet), because this explicitly takes the heterogeneity of each phenotype class into account by attempting to classify the samples into these subcomponents.Using mixtures of Gaussians, the densities of each phenotype class can be estimated more accurately. Thus, provided that the inferred Gaussian components are biologically meaningful, this approach should in general lead to an improved classification performance. However, the implicit assumption in MDA is that we are still interested in classifying samples into the As an example, consider the case of two phenotype classes with MDA predicting two Gaussian components for each class. Thus, training data is used to learn the parameters and weights for four Gaussian clusters and classification of test samples is subsequently performed via the Bayes' classifier (equation 3) on these four subclasses. Note therefore that in MDAhet, the cluster-inference step of MDA is used to define the classes for which classification is then performed. Since these inferred classes make up subtypes of the original phenotype labels, this classification framework explicitly takes the heterogeneity of the phenotypes into account.In the context of cancer gene-expression studies it has been a problem in certain cancers to derive reliable prognostic classifiers as is the case for ER- breast cancer. Typically, in the context of prognosis one would expect discriminative gene-expression profiles to exhibit bimodal distributions with the two modes mapping roughly to the two prognostic groups (good and poor) . HoweverSE(t) and specificity SP(t) values using Kaplan-Meier estimators for the predicted subclasses. In our context, we assume that samples have been classified into two groups, so that the predictor X = 1 predicts poor prognosis, while X = 0 predicts good prognosis  Thus,Following the work by Heagerty and colleagues , we estit) denotes the Kaplan-Meier estimator for the overall survival function, while t|X = c) denotes the Kaplan-Meier survival estimate for the particular subgroup X = c  [where  = 1, 2) . In our  = 1, 2)  we can oXCL2, HLA-F, C1QA, TNFRSF17, SPP1, IGLC2, LY9) [Applying a feature selection method designed to remove false positives  to an inC2, LY9) . FurtherC2, LY9)  into twoC2, LY9) . SpecifiThese results motivated us to investigate the prognostic role of the immune response-module further in four additional ER- data sets for which gene expression and clinical data were available ,18-20. UTo derive a single-sample prognostic classifier we first applied a mixture discriminant classifier to the same training set of 186 ER- patients and across the seven identified genes. The heterogeneity of the good-prognosis phenotype, as shown by the gene expression patterns of the immune response-module Figure . The choHaving estimated the parameters for each of the phenotypes, external samples can then be classified by applying the MDA to the test sample's gene expression profile, yielding probabilities of the sample belonging to each phenotype class, and subsequently using the maximum probability criterion for class assignment. Since each phenotype class is modelled as a mixture of two Gaussians Figure , class ay to one of the four subclasses c  using the maximum probability criterionIn detail, MDAhet assigns a test sample with a seven-gene expression profile G denotes the seven-dimensional multivariate Gaussian and the parameters where et Table .The classification distribution of samples from the six external cohorts into the four subclasses as determined by MDAhet showed that test samples classified most often into the 'poor-down' and 'good-up' classes Table . Since sTo evaluate the performance of the MDAhet classifier in the training and test cohorts we used several different measures and models of prognostic separation, depending on the variable of clinical outcome used. As binary outcome we used absence or presence of a disease-specific death event, or the surrogate-distant metastasis if the former was not available. Since this does not take time dependence of events into account, binary outcome was also used at four years after surgery adapting methods for time-dependent receiver operator curve (ROC) analysis . In addia priori known proportions was observed in all test sets  further confirmed the better prognosis of the predicted 'good-up' group Figure . These sSince the maximum probability criterion assigns test samples to classes without regard to how large the maximal posterior class probabilites are, we tested the robustness of our results by only classifying samples passing a minimum probability threshold. For a probability threshold of 0.3 , 94% of all test samples passed this threshold, indicating that our results are indeed robust. For a threshold of 0.4, we found 68%of samples were classifiable and results were still in line with those reported for the minimum threshold of 0.25 (data not shown).p < 10-6)  Table . Importa6) Table . In termThe remarkably high NPV values in the test cohorts, however, raise some important questions. First, we found that the performance in the test sets was better than in the training set Tables  and 4. WA second important point relates to the nature of the MDAhet classifier. As remarked in a previous study , in the SPP1) is consistently underexpressed in patients with a good prognosis. To conclude, we can therefore hypothesise that the MDAhet classifier and associated immune response-module may be identifying another good prognosis ER- subset of tumours, but with a significantly better prognosis than medullary high-LI breast cancer  in six independent test cohorts and validity in more than 469 patients, and which performs independently of LN status. We propose to develop a reverse transcriptase-polymerase chain reaction-based clinical assay based on these seven genes to identify ER- patients of good prognosis that may benefit from a less aggressive course of chemotherapy.C1QA: complement component 1, q subcomponent, A chain; CI: confidence intervals; CT: chemotherapy; ER: oestrogen receptor; HER2: human epidermal growth factor receptor 2; HLA-F: major histocompatibility complex, class I, F; HR: hazard ratio; IGLC2: immunoglobulin lambda constant 2; LDA: Linear Discriminant Analysis; LI: lymphocytic infiltration; LN: lymph node; LY9: lymphocyte antigen 9; MDA: Mixture Discriminant Analysis; MDAhet: Heterogeneous Mixture Discriminant Analysis; NPV: negative predictive value; PPV: positive predictive value; QDA: Quadratic Discriminant Analysis; ROC: receiver operator curve; SPP1: secreted phosphoprotein 1 (osteopontin); TNFRSF17: tumour necrosis factor receptor superfamily member 17; XCL2: chemokine (C motif) ligand 2.The authors declare that they have no competing interests.AET conceived of the study, performed all statistical analyses and wrote the manuscript. CC contributed to the writing of the manuscript.