Our method achieves good results with classification accuracy , and can also be used to study CUP samples on a gene-by-gene basis. It is not tied to any  Cancer of unknown primary origin (CUP) is a classification given to a malignant neoplasm when a metastasis is discovered but the source of the primary tumor remains hidden. If counted together as a single clinical entity, CUP is one of the most common cancer types diagnosed in the world. Some 3 to 5% of all newly diagnosed cancers are CUPs, which qualifies this disease entity as one of the ten most common cancer types, with an incidence that is greater than that of, for example, leukemia or pancreatic cancers ,2. Even et al. [et al. [et al. [et al. [et al. [Several methods for identifying CUP samples based on their gene expression profiles have been developed. Talantov et al.  and Vara [et al.  presente [et al.  proposed [et al.  presente [et al.  applied a priori defined set of genes whose expression in the test sample is measured.Since the development and adoption of gene expression microarrays, there has been interest in developing a microarray-based cancer classification, including a test to identify the origin of CUP cases. Microarrays provide a robust way to measure the expression of a large number of genes, and recently have been proven to be applicable in the clinical setting as well -12. At lAll the previous methods for identification of CUP tumors thus rely on a fixed set of training samples, sometimes with a narrow representation of histological types and anatomical sites, from which the informative genes have been determined. Thus, these methods cannot take into account the constantly accumulating scientific knowledge on gene expression across all types of cancers. Therefore, a more universal and adaptable method for microarray-based CUP prediction is desirable. If the identification of CUPs is performed algorithmically from genome-wide expression profiles, as opposed to from a defined gene list, the method is scalable, more flexible and open to improvement as reference data increase in both quality and quantity. Importantly, definitions of the histopathological and molecular subgroups of the reference tumors will dramatically influence the classifiers, possibly requiring major changes and improvements to existing disease classifications. For example, it may be important in the future to develop specific predictors for, for example, estrogen receptor-positive and -negative breast cancers, or the five major breast cancer subgroups, or for other very small subgroups, such as anaplastic lymphoma kinase-positive non-small cell lung cancers ,19. In oet al. [Staub et al.  demonstret al.  as a refHere, we set out to create a CUP classifier that could easily be adapted to any reference data set. For this purpose, we analyzed test samples by aligning their microarray profiles against the annotated and normalized GeneSapiens microarray reference database and applied a slightly modified alignment of gene expression profiles (AGEP) method - weighted AGEP (wAGEP) - which we recently developed and described for classification of cell differentiation patterns . The wAGThe aim of this study was to study CUP sample characterization using the previously published AGEP method . The intThis study uses a modified version of the AGEP methodology. Briefly, AGEP calculates a tissue specificity score (ts-score) for each gene in a test sample for each predefined group (such as a tissue or cancer type) in the reference data. The ts-score measures, on a scale of -1 to 1, how well the gene's expression in the test sample classifies the sample as belonging to the group. A score of -1 indicates that, according to this gene, the sample is anything but a member of this group, while a score of 1 means a perfect fit to the group to the exclusion of all other groups. A score of 0 means the gene's expression is indeterminate when considering if the sample should belong to the group or not. A final similarity score between the test sample and each group in the reference data is then calculated taking the mean of all ts-scores for each group. The original AGEP algorithm can be found in .The weight for a gene in a particular cancer type was calculated as follows. First, density estimates for the gene for each cancer type in the reference data were constructed as demonstrated in . We thenGene weights were applied as follows. When calculating the final similarity score between a test sample and a cancer type (mean of the ts-scores for each gene for that cancer type), each gene's ts-score was multiplied by the weight that gene had for the cancer type in question. The resulting ts-scores were then divided by the mean of all gene weights for that cancer type. This was done to normalize the different amounts of specific genes different cancer types possess. Finally, the similarity score between the test sample and the cancer type was calculated by taking a mean of the ts-scores. The workflow is depicted in Additional file Reference data, both expression values and annotation, were fetched from the GeneSapiens database . The canThe test data were from GEO  study GSAll data analysis was done with R .The test samples were arranged according to the highest similarity score they had attained for any cancer, and whether this cancer was a correct classification was also recorded. From this, the fractions with the highest score above a certain threshold were trivial to calculate. A graph showing accuracy as a function of the highest similarity score was calculated using a sliding window. The width of the window was 0.1 (in similarity units) and it was moved in steps of length 0.005 over the ordered test sample population. The percentage of correct classifications within the window at each step was calculated  were used for the hierarchical clustering of both genes and samples.An R library that contains the original AGEP functionality and function for calculating and applying the gene weight (wAGEP portion) can be found at .et al. [AGEP compares the expression value of a gene in a test sample to the distributions of expression levels of the same gene across all reference sample groups , and determines how well the expression level for the gene in the test sample fits with the corresponding distributions in the reference data. This analysis is then repeated for each gene. For a test sample, AGEP thereby provides a tissue match score (tm-score) for each gene for each reference tissue type, which quantifies how well that gene's expression corresponds to the levels in the reference tissue types. The AGEP method also evaluates how uniquely the tm-score categorizes the test sample among the tissues of the reference data. This is the tissue specificity score (ts-score). The output from an AGEP analysis are the tm- and ts-scores for each gene of the test sample in relation to each tissue type in the reference data. For a more in-depth description, please see Kilpinen et al. .Tm- and ts-scores allow for comprehensive interpretation of the molecular nature of the query sample in relation to the entire reference dataset. For example, among healthy tissues, the tissue with the highest average ts-score for a test sample indicates the tissue of origin with high accuracy (93.6%) . The oriThe key advantage of the AGEP method is that it examines each gene of the test sample and each sample group (such as cancer types) in the reference database independently, and then compares the results across tissues to find the genes that best classify the test sample. This attribute is retained with the addition of the weight factor, and the weight only enhances the classifying potential of genes with cancer-specific expression profiles. Additional file KIT in gastrointestinal stromal tumor  and KLK2 and KLK3 in prostate cancer . Also, some new cancer-specific genes are found, such as TMEM204, which has the highest weight for GIST, 0.96; when looking at GeneSapiens [The method used to determine gene weight gives, as expected, a high weight factor to genes already known to be highly expressed in certain cancers. For example, eSapiens  data, thIt is important to note that the classification is still based on all genes; some genes in each cancer type just have a bigger impact than others in determining the tissue specificity.et al. [We used the cancer samples from the GeneSapiens database as the reference data ,24. The et al. .in vivo cancer types when a match to similar cancer types was accepted or 79% if the exact match was required. Average sensitivity with the less strict criteria was 0.748 with a specificity of 0.999. Without the application of gene weights  the total accuracy of training data LOOCV was 78%, substantially less than with wAGEP (88%).To validate the integrity and applicability of the reference database for AGEP analysis, we performed a LOOCV analysis of the entire reference data. Thus, the tissue origins of all 5,577 individual malignant samples were analyzed by reconstructing cancer-specific gene density estimates without the sample in question. AGEP analysis revealed a total accuracy of 88.2% within the search space of 56 different Test data were from GEO  study GSWe originally compared the test samples against both the healthy tissue samples  and the 56 different cancer classes of the GeneSapiens database. The accuracy of prediction was 69% if we considered both appropriate healthy tissues and cancers as correct. Interestingly, we found that only 7% of the test samples had a healthy tissue group, as opposed to a primary cancer group, as their best match. This was the case for both test groups, the dedifferentiated primary tumors and metastases. We therefore conclude that the test samples, which imitate CUP problem solving, resemble cancers significantly more than their differentiated healthy tissue counterparts. As a consequence, subsequent analyses for this study were done by comparing the test samples only against the cancer reference data. Figure Comparing the GSE12630 test set against reference tumors, we achieved 78.1%  total accuracy in identifying the tissue of origin. Classification was counted as accurate when (a) the cancer type with the highest similarity score was exactly the same as the test sample's annotation ('exact'); (b) when the cancer type with the highest similarity score was from the same organ, such as lung adenocarcinoma being identified as lung squamous cell carcinoma ('similar'); or (c) when the cancer type with the highest similarity score was from the anatomical site of the metastasis and the second highest cancer type was of category a or b above ('same site'). These results would all prompt a physician to consider the primary tumor in the correct anatomical site. Of the metastasis test samples, 64.8% were accurate according to definition a, 12.5% additional cases according to definition b and and additional 0.8% according to criteria c, resulting in a total accuracy of 78.1%. The percentages for the primary samples were 71.2%, 6.8% and not applicable (a sample from a primary tumor cannot fulfill this criterion), respectively, resulting in a total accuracy of 78.0%, with an average sensitivity of 72% and specificity of 99% across all samples  and a representative primary cancer of the same tissue ('cancer of target tissue') from the reference data. This was done for all metastasis samples. Of the 128 metastasis samples, 83 could be assigned to both a target tissue and a cancer of target tissue. We then studied whether the similarity of these test samples to either their target tissues or cancer of target tissue was dependent on any of the following: similarity to their original cancer, their cancer type, or the target tissue. In 62 of the 83 cases, the test sample's similarity to the cancer of target tissue was higher than its similarity to the target tissue. In all target tissues except lymph node the vast majority of the test samples resembled the cancer of target tissue more than the target tissue. In the case of the lymph node there was an about even split. In terms of the original cancer type, the results are similar. All other cancer types except thyroid carcinoma resemble their cancer of target tissue more often than the target tissue. For thyroid carcinoma, five out of the six samples resembled the target tissue more than the cancer of target tissue. However, four of these samples were lymph node metastases. The findings are not surprising, as any epithelial tumors metastasizing to lymph nodes will not start resembling lymphatic tissue derived cancers. The numbers for each target tissue and original cancer type can be seen in Tables Figure All these results reaffirmed our decision to analyze the test samples by comparing them to cancer only reference data.An advantage of the wAGEP method is that the results can be analyzed on a per gene basis. Thus, it is possible to identify the genes that would be good classifiers in the reference data  and explore whether those genes are useful in the identification of the metastasis samples.We looked at the samples that were metastases of renal cancer from the test data, and specifically at genes having renal cancer-specific expression levels. There were 58 genes with gene weight >0.25 in renal cancer, and these were selected as the renal cancer-specific genes. Forty of these were present in all test samples. When their tissue specificity scores are plotted, a subset of genes are seen to loose their renal cancer-specific expression in the metastases Figure . The 40 SLC) genes. The genes that retain their renal cancer-specific expression in all samples do not seem to differ strongly from the genes that do not. Of the genes that do not retain their renal cancer-specific expression in all samples a few are worth pointing out. One interesting gene is CNDP2, known to be overexpressed in renal cancer [The vast majority of the renal cancer-specific genes encode membrane bound proteins, such as the numerous solute carrier family , but the level of divergence from the renal cancer-specific expression seems to be stable, independent of the sample's overall similarity to renal cancer.Also, the three angiogenesis-related genes, Similar analyses were done for melanoma Figure  and gastIn the gastric cancer case we see a group of four genes, on the left side of the plot, which display different tm-scores between samples. In most samples the genes retain gastric cancer-specific expression, but in a few samples the genes' expression seems dramatically altered. As with the melanomas discussed above, most of the genes that have melanoma-specific expression seem to have lost that expression.In both the melanoma and gastric cancer sample sets, one or two samples had completely lost their cancer-specific expression for all genes. These could be samples originally incorrectly annotated, or metastases that are dedifferentiated to the extent that they have no resemblance to their original cancer type.Metastasis is an indicator of poor prognosis for any cancer patient, but the issue is even more difficult if the primary tumor is unknown and the diagnosis has to be made solely based on the discovery of metastases. This 'type' of cancer is known as a cancer of unknown primary (CUP) and represents a condition requiring specific clinical attention. The origin of the metastasis needs to be identified as primary treatment regimes for cancer are typically based on the anatomical origin and histological type of the primary tumor. Studies by several groups -7,20 havWe show here that the wAGEP method is capable of identifying the tissue of origin of CUP samples with 89% accuracy when excluding the most uncertain 30% of the samples. If we, like some of the previously published studies have done , categorPancreatic cancer is quite common as a source of metastatic disease (between 25% and 12.5% of post-mortem identified CUP cases ), and itin vivo mouse studies [in vivo metastatic samples are still too few in number and size to allow systematic studies of this subjectAs the wAGEP method makes it possible to identify the tissue similarity as well as the genes behind the similarity, we were able to show which cancer-specific genes lose their cancer of origin-specific expression in metastatic samples Figure . Even th studies . HoweverThe ability to directly interpret expression profiles of CUP tumors using a constantly increasing body of scientific data and knowledge allows for a faster and more economical way of providing more accurate diagnostics for CUP patients. This is essential as having metastatic carcinoma of unknown origin is a difficult situation for cancer patients; the average survival of these patients is only a few months . ApplicaFull genome microarray analyses of CUP patients, like of any cancer patient, will also provide more information than just the tissue of origin. As the poor survival statistics of metastatic cancer patients show, CUP patients would need more than just the identification of the origin of metastasis. The wAGEP approach will provide data on the expression of all genes in the metastatic tumor, including information on potentially important subgroups or the expression of therapeutic targets that could be simultaneously assessed. Tailored medication based on these observations might prove to be a more useful approach than the traditional approach of anatomy- and histology-based treatment regimes.The wAGEP method proved to be good for classifying CUP samples. More than that, however, it showed that it was capable of finding and analyzing differences between the metastasis samples and their primary cancer types, thus providing interesting information that could have clinical significance. It is also not tied to any predefined gene list, or indeed anything predefined. It is fully scalable and able to adapt to new emerging scientific data.AGEP: alignment of gene expression profiles; CUP: cancer of unknown primary origin; GEO: Gene Expression Omnibus; GIST: gastrointestinal stromal tumor; LOOCV: leave-one-out cross-validation; tm-score: tissue match score; ts-score: tissue specificity score; wAGEP: weighted AGEP.KAO and SKK are inventors on a patent application regarding the original AGEP method. SKK and OPK are shareholders in Medisapiens Ltd, which develops microarray data analysis technologies.KAO designed the study, performed the majority of the work and wrote the manuscript. SKK did the LOOCV validation part, provided ideas for the project and participated in the writing of the manuscript. OPK supervised the project. All authors have read and approved the manuscript for publication.An illustration of the method used to calculate similarities between a test sample and the reference data. (a) The weight of a gene for each cancer type in the reference data is calculated by taking the area where the gene's density estimate is higher than that of any other cancer. Since the area under each density estimate is 1, the resulting weights are numbers between 0 and 1. The weights are unique for each cancer, and represent the ability of the gene's expression to distinguish that cancer from all others. (b) A schematic of the process for calculating the similarity between a test sample and the reference data. The AGEP procedure is modified by having the gene weights calculated from the density estimates and applied to the ts-scores of the normal AGEP result. Then, as per normal AGEP procedure, the tissue similarity for the test sample and each cancer in the reference data is calculated by averaging the now weighted ts-scores for that cancer.Click here for fileA summary of the reference data, the name of each cancer type and the number of samples it has.Click here for fileHeatmap of all genes and all cancers used in the analyses. Genes are colored according to their weight.Click here for fileTMEM204 geneGeneSapiens boxplot of the .Click here for fileResults for each individual test sample. Each sample is annotated as accurately as possible, and the five highest similarity scores and their corresponding cancer types are shown.Click here for fileA combination of the two images from Figure vice versa if the line is dashed. The icons are colored based on the tissue where the metastasis was.Click here for file


Sample tracking errors have been and always will be a part of the practical implementation of large experiments. It has recently been proposed that expression quantitative trait loci (eQTLs) and their associated effects could be used to identify sample mix-ups and this approach has been applied to a number of large population genomics studies to illustrate the prevalence of the problem. We had adopted a similar approach, termed ‘BADGER’, in the METABRIC project. METABRIC is a large breast cancer study that may have been the first in which eQTL-based detection of mismatches was used during the study, rather than after the event, to aid quality assurance. We report here on the particular issues associated with large cancer studies performed using historical samples, which complicate the interpretation of such approaches. In particular we identify the complications of using tumour samples, of considering cellularity and RNA quality, of distinct subgroups existing in the study population (including family structures), and of choosing eQTLs to use. We also present some results regarding the design of experiments given consideration of these matters. The eQTL-based approach to identifying sample tracking errors is seen to be of value to these studies, but requiring care in its implementation. This issue has had a high profile of late following an error at a personalized genetic testing service (It is common to use replicated control samples at known points on a plate Known phenotypes with a sole (or strong) genetic component can also be used to check sample validity (or rather to seek to detect plating errors - as they are unlikely to have enough power to confirm that a sample is that which it claims to be). Sex is the obvious phenotype in this regard. With a careful sample layout, as is discussed later, errors on a plate scale would be detected by a sex-check, but individual switches of any pair may not. Clearly for some studies, e.g. in prostate cancer, this will not be an option. Other traits such as blood group could be compared to the appropriate genotypes, but for a trait with a narrow driving locus there is too great a chance that there will be a miscalling of the genotype class simply to exclude samples based upon this metric. Many such traits are therefore needed.Expression Quantitative Trait Loci (eQTLs) that regulate the transcript abundance of particular mRNAs can be identified systematically using high-throughput technologies The ability to predict SNPs from expression data has recently been considered from a data-security context Our approach is conceptually similar to the recently published MixupMapper approach to this problem Both approaches rely on defining a distance between an expression array and a genotyping array. Our measure (the ‘BADGER score’) is the sum (across eQTLs) of the squared difference between the number of ‘B’ alleles called from the genotype array and the number predicted from the expression array. MixupMapper, on the other hand, uses a normalized sum of z-scores for the difference between predicted and observed expression values. As a minor additional detail, MixupMapper considers the magnitude of their score, while BADGER looks at the rank. Naturally, a low score is indicative of a match. The vast majority of arrays  will not be a match, so the score of an array that does match should be outlying and take a rank of one.As the concept of identifying plating errors using eQTLs has been demonstrated, we will not focus on justifying or demonstrating the approach once more . Rather, we shall highlight the challenges of applying such an approach to a population cancer genomics study, and note where the distinctions of our take on the approach lend themselves to such data.While our primary purpose is not to demonstrate once more that an eQTL-based approach to calling and identifying mismatched samples can work, we note that our results here would support the message of Westra and colleagues Westra and colleagues For the 127 samples for which all four arrays  are available, the qualities of the matches from normal and tumour tissues are shown in The concern then is that, in circumstances where we have not processed the normal genotype, the tumour genotype would sometimes not be a good enough match to allow us to assign the match correctly. Anecdotally, this does seem to occur. A possible explanation is that the non-diploid (or diploid but suffering from LOH) nature of the tumour genotype interferes with the calling of diploid SNPs. If the majority of the SNP/expression relationships observed are not directly causally linked, then this would explain the relatively poor match qualities of tumour SNP arrays as compared to normal tissue SNPs.Since the normal genotype array is shown to be a better match than the tumour genotype array, even though our eQTLs were defined mainly from tumour samples, it seems reasonable to suppose that cellularity  will have little effect on the performance of approaches such as BADGER. Normal contamination will increase the chances of calling heterozygous SNPs, even when the tumour has undergone LOH or allele-specific DNA copy-number changes.Although the disrupted genotyping calls due to the copy-number aberrations in tumours can impede approaches such as BADGER, these remain an accurate description of the tumour and we must endeavour to identify the sample and match it to an expression array. As has previously been noted, a poor quality expression array can also disrupt the process www.stats.ox.ac.uk/~davison/software/shellfish/shellfish.php), one can project Affymetrix SNP data onto a triangle where the three corners represent the HapMap populations from which the loadings were derived. For convenience we will term the groups of samples that form in these corners the ‘Europe group’, the ‘Africa group’, and the ‘Asia group’. We have also observed individuals lying between the Africa and Europe groups  and between the Europe and Asia groups .Using principal component loadings published for the shellfish tool (We plot the average BADGER score (NB score not rank) associated with each genotype array against group in the left hand panel of a priori reason to believe that this group will be over-represented in the number of genotyping arrays for which no matched expression array exists. For a relatively high proportion of this group we cannot be sure of the match between genotype and expression, but this is more likely to be a consequence of the higher scores seen in the Africa group than a cause of it.More important than the average score is the minimum score that, for any genotyping array with a matched expression array in the data set, we might expect to be comparable regardless of ethnicity. In the right hand panel of We cannot claim, for any eQTL pair we use, that the genotype we observe is actually driving the expression (see next section). At best it is likely to be a tagging SNP for the causal variant (if one exists), and the performance of the tagging SNP will vary between ethnic groups. It has been observed that only 50% of eQTLs are seen in more than one population, and a very small minority in several et al. To illustrate further aspects we consider an additional data set: the genotype and expression data associated with the HapMap (Phase I) samples MixupMapper identified only one mix-up in this data set, finding that the best expression match for the genotype data supposedly from sample NA18515 was that supposedly from sample NA18517.In the CEU population, there is a problem with one of the expression arrays associated with "NA10856" (labelled GSM232786_NA10856_2_2). However it is clear that the other three expression arrays are good matches, and with some investigation, it is apparent that the values for this array are identical to one of the Yoruban arrays (labelled GSM232802_NA18503_1_1), but that the values differ in GEO, making it unclear where the problem arose. We can speculate that since MixupMapper would have found a good match for the NA10856 genotype array, that it would not have flagged this as being problematic.With BADGER, we also find that the four expression samples associated with NA18515 all offer the genotype array mapped to NA18853 as the best match. It is true that the expression arrays associated with NA18517 are the best matches for the SNP array mapped to NA18515, however this is because NA18517 is a parent of NA18515. The expression arrays associated with NA18516 (the other parent) are the next best match. In the absence of an expression array for the child, we would expect the best match to be one of the parents. In fact, we can easily see from the genotypes of the mother/father/child trio that the error (if there is an error) must be in the expression array.et al. so can make no claims regarding the original study The genotype array that offers the best match to all of the NA18515-associated expression arrays is that associated with NA18853. The match is marginally worse than that for the expression array associated with NA18853, but noticeably better than that associated with NA18854 (the child of NA18853). Comparison of the full expression profile suggests that the NA18515 expression arrays are not simply accidental replicates of NA18853. Thus the mix-up is difficult to resolve from such a distance, and we would recommend removing the expression arrays associated with NA18515 from analyses but would leave the genotype arrays as they are. Note that we are using the expression matrix given by Westra While the match to a close relative tends not to be as good as that to the correct sample, we do see enough overlap in values that if one of the samples were missing then we would be in danger of mistakenly associating the two as being from the same individual. This would presumably be more of a danger if the genotype array were missing as the existence of relatives will be harder to deduce from the expression data.Taking the Han Chinese and Japanese individuals, as these do not contain complicating family groups, we can simulate sample switches and confirm the utility of BADGER. We find that half of the samples have to be switched to require more than one productive iteration , and two-thirds must be switched for there to begin to be unresolved switches. This may be an over-estimation of performance, since our ‘external’ eQTL set was, in fact, defined by Westra et al. from these data, but competing against this are the lack of subtlety in the corrections that in practice is afforded by the human assessment of the potential switches and lab-validation between iterations. The assessment here was simplistic and automated for the simulation study. Even allowing for these points, the fact that 80 of the 90 samples would need to be deranged before an approach such as BADGER is unable to add value is remarkable.Defining the eQTLs from the data set therefore can potentially impact upon any genetically distinct minority group in a study. Westra and colleagues et al., but may provide a robust basis for correcting errors. By exploiting a technical artefact in this way, we would hope to be less sensitive to genuine biological differences between groups of patients than we would be if relying in uniformity of genuine eQTL behaviour across populations.Many eQTL studies separate out such expression probes in order to avoid spurious associations We initially chose a set of 383 eQTLs on the basis of the strength of association seen in our data. The significance of the association is as much a measure of the distribution of genotypes as it is the discriminatory power of the expression-SNP association, but this is a first pass and the set is refined as part of the BADGER approach as detailed in the When the set of expression-SNP associations is reduced to 125 probe-pairs, it is further enriched for SNP-covering probes. Only 26% of the 199 probes that do not cover SNPs survive into the refined set, while this increases to 33% for the 119 probes that cover one SNP, 47% for the 38 probes that cover two SNPs and 59% for the 27 probes that cover more than two SNPs.To reinforce this point, amongst our set, there is a trend that the more SNPs the expression probe covers, the smaller the discrepancies between the predicted and observed values of the SNP-probe in the eQTL association. This is shown for the Europe and Africa groups in By choosing distinct but differing patterns, by sex, for the sample layout on a plate, and ensuring that the patterns do not have rotational symmetry, the plotting of the inferred sexes of samples by plate will provide a clear and definitive diagnostic as to whether the correct plate has been used and whether in the correct orientation. Individual, simple, switches of neighbouring samples (in column or row) may not be picked up, but we can maximize the chances of doing so by choosing patterns under our constraints that minimize the numbers of neighbouring pairs (in columns or rows) of the same sex.If the numbers of the sexes are equal in a study, a chequer-board pattern minimizes the numbers of neighbouring pairs of the same sex, but has rotational symmetry. Small perturbations from the chequer-board pattern will be susceptible to confusion , or to one another (via slightly more switches). It seems likely that a regular pattern will be preferable, for ease of setting up the plate, and if only a small number of plates are required then this may be feasible. Note though that in order to avoid rotational symmetries, and indistinguishable plates, the number of simple switches on a plate that cease to be detectable  increases rapidly shown in .If we had 100 plates on which the layout of the sexes was random, then by simulations we expect the closest pair of plates to differ in 30 wells (95% CI: 26 to 32), which would require a minimum of 15 simple switches to explain. Thus a random arrangement of the sexes will allow for the detection of plating errors. We would expect, with a random layout, to miss approximately 85 (95% CI: 72 to 98) of the potential simple switches on a plate . Thus if the complication of attempting to use one of these regular designs was to come at the cost of doubling the probability of making a simple switch, then we might not expect it to bring benefits over a random layout. In such circumstances, if the natural ordering of samples would provide an ‘effectively random’ ordering of the sexes, then employing it is a highly defensible approach.vice versa). If the samples were neighbouring on both platforms then it may not be possible to identify the source of the error and so not subsequently possible to correct it.If using eQTLs to identify our mismatches, in a scenario where only one sample per person is run on any technology, then whatever our design, we will always be able to detect a simple switch between samples run on neighbouring arrays. Once this is observed we can correct the recorded association between the genotype and expression arrays. If, however, we have additional sample information , the question may still remain as to which clinical sample is associated with each pair of arrays (this may not matter for some studies). If the two samples were neighbouring on the expression platform, but in completely different batches for the genotyping platform, then the confusion can be attributed to the expression platform and rectified  is not acceptable. Rather there is a desire then to have designs for the layout of the two experiments (expression and genotyping) that avoid  neighbouring samples in one experiment being neighbouring samples in the other. This divergence of layout would ideally happen at the earliest possible stage in sample preparation. However, as with sex-based detection, there is a concern that adopting such an approach would enable us to detect plating errors at the expense of increasing their number. One might argue that, if all such errors were simple switches, then the success rate seen for resolving them in our approach and that of Westra There is no approach that will be optimal for all combinations of study and laboratory. Each study group should then seek to balance these tensions as best they can within their own circumstances. What is clear though is that, for the successful resolution of mix-ups, complete layouts of samples at all stages of an experiment are required whether those layouts are ‘designed’ or not.What is apparent from these results is that when dealing with retrospective clinical studies, there may be legitimate reasons why a genotyping array will not appear to be a match for an expression array even if material from the same person were run on both. Therefore, if one discards arrays solely on the basis that they do not appear to match, then there is a risk that one is discarding well-matched arrays. Moreover, these might be the really interesting samples where something unusual and revealing is happening. The importance of a predictor for the performance of BADGER is also clear. We do not wish to expend energy pursuing an apparent mismatch when the results have been driven by array quality.Ideally then, we should discard arrays not because they do not appear to match, but because we can identify  the error that led to the mismatch and yet not a way to correct for it. If we cannot identify the error then there are risks both in retaining the arrays (we may contaminate the study with erroneous data) or in removing them .If there is a distinct genotypic subgroup forming a small minority of samples in the study, then we have seen that these are particularly prone to being called as mismatches in error. Depending on the nature and purpose of the study, it may be that it makes sense not to include this subgroup in a primary analysis. Studies are often designed with inclusion criteria that restrict the heterogeneity of samples , and this could be seen in that context. If this is the view, it may make little practical difference if the arrays are excluded on the basis of the apparent mis-mapping of samples, or if they are excluded because they represent a minority pattern, in which case the policy of removing unresolved samples might be more attractive.Had our purpose been to search for eQTLs, then removing the Africa group from analyses would undoubtedly have increased the number of ‘eQTLs’ that we would have found, but this can only be reasonable if we can recognize the restrictions on the interpretation of our findings that would result. In this example, it is clear what is happening, and it seems likely that we could relate our findings to a valid external population. Matters may not always be so clear, and the removal of samples on the basis of genetic patterns  will always have the potential to bias downstream analyses. A pragmatic approach may be to perform multiple analyses with different inclusion criteria in order to check the robustness of the results.Crucial to this process is the existence of a set of eQTLs. These do not need to be of ‘publishable’ standard, and indeed their significance is not the same as their predictive power. One has the choice of using an independent data set to define these  in the genotyping arrays and limiting the search to a set of unique arrays can be beneficial for interpretation of the BADGER results. If a genotyping array’s supposedly matching expression array was not the array with best BADGER score, it might not be as easy to identify that this is because an accidental duplication of that array has occurred. From an expression array’s point of view, since genotype duplicates can easily be removed, the supposedly matching genotype array should always have a BADGER rank of 1.Additional beneficial factors include the greater consistency of quality between the genotype arrays, an easier to define metric for distance between genotypes than between expression values , and a possibly lower chance (in our experience) of plating errors occurring in the genotyping arrays than in the expression arrays. If, however, we were using externally defined eQTLs, then we may prefer to work from the genotype arrays towards the expression arrays as the quantification of the eQTL association may be dependent on factors, such as ethnicity, that we can infer only from the genotyping arrays.In determining what error may have occurred it is important that we can access all information about a sample’s location at various stages in its preparation. Because not all samples are eligible for inclusion in a study, the ordering/layout of extracted RNA/DNA may not match the ordering/layout of the samples. In turn, perhaps to accommodate control samples or repeats of earlier failed arrays, the processed RNA/DNA may have a different ordering/layout again. Finally, if using a BeadChip with multiple arrays, the layout may change once more (hence the increased risk for the expression arrays mentioned above). If for the RNA and DNA we have access only to the final layout information then we may miss the source of an error or attribute it to the wrong experiment (e.g. RNA rather than DNA), which would be worse.It would be preferable to minimize the number of plating errors, or to increase the chances of identifying and correcting them. We have mentioned some of the aspects of experimental design that may impact on both of these objectives. Utilization of bar codes from the earliest stages of sample preparation, and the automation of processing to as great an extent as possible would seem desirable to minimize the number of plating errors, or maximize the chances of detecting any that do occur.The confirmation of some sample switches via STR genotyping is beneficial, but the level of BADGER evidence for some mix-ups is so compelling that the validation of all sample switches in this way would be a waste of time and resources. Moreover, there will not always be sufficient residual DNA in the sample plates to validate successfully in this manner, nor can all errors (particularly errors in transferring from plate to microarray) be identified with this approach. Given the considerable efforts required to conduct such validation, any additional methods that might provide evidence  should be given consideration.Our exploitation of cis-eQTL artefacts ignores one particular resource. Since the second generation of Illumina BeadArray, there have been large numbers of pairs of probes that differ at a single nucleotide . These pairs of probes most often differ at a known SNP, and so could provide a stronger association between genotype and expression measurements, although there are no guarantees that the particular genes involved will be useful for any particular study.The BADGER approach to identifying errors can undoubtedly be extended and improved. We have already mentioned that more informative priors could be used. Also tissue-specific eQTLs could be used to identify errors where multiple tissues are being contributed from each participant. We have been anticipating that the process will begin with an intended matching of expression and SNP arrays, however if pre-defined eQTLs are used (either known tissue specific eQTLs or cis-eQTL artefacts) then it will be possible to infer the matching of arrays from scratch.A similar approach can be adopted for studies involving other data types, for example methylation and miRNAs, as well as purely expression studies involving multiple tissues where we might want to predict genotypes from each tissue-type to validate the identities of samples. There may be other resources on the arrays that could be exploited to aid mapping. Ultimately though, what matters is not how we find and correct the errors, but that we can and do find and correct them.Across a number of studies we have assembled a large number of genomic measurements of breast cancer samples , and the relatively ‘low-throughput’ nature of this stage of the study, we are confident that no samples were switched in this data set.10(p)>15 and the SNP within 1 megabase of the expression probe) and considered only the expression probes with perfect annotation. This last criterion is possibly excessive, but still returns 383 eQTLs for our data set, despite the relatively small number of arrays from which we are deriving them.We look for cis-acting eQTLs . Using our samples, these associations are quantified and then used to ‘predict’ genotypes corresponding to each expression array.For every combination of expression and genotype array a distance (the BADGER score) is calculated between the predicted genotypes from the expression array and the observations from the genotyping array. For each expression array, the matches to all genotyping arrays are then compared and ranked such that the best match is of rank 1. If the genotyping array that was thought to match the expression array is of rank 1, then we accept this mapping. Otherwise we search to see whether there is a likely plating error. If a plating error is found then  we adjust the annotation of the samples on the arrays accordingly and return to the beginning. At this step we may also refine our set of eQTLs, removing any that show little or no diagnostic value. We judge an eQTL pair of expression and SNP probes based on the squared differences between the observed call from the SNP probe and the predicted SNP from the expression probe, summed over all pairs of arrays that we believe to match well.i), and a set of known well-mapped arrays, we use kernel density estimation techniques to generate three density functions for expression, corresponding to the three genotypes. These we denote We consider only three possible genotypes , despite the possible non-diploid status of the samples we are investigating. Under our diploid assumption we can represent these numerically by taking as our statistic the B-allele count . For a given eQTL pair of an expression probe and genotype  for a matching array. If our eQTL list is long enough, or our value of Consequently, we sort the set g is the largest number of replicate genotype arrays that we might encounter in the data set.is required, where Our choice of prior is somewhat naive; we might for example make better use of known allele frequencies. Alternatively, if desiring a ‘fair’ prior, we might wish to give more weighting to the AB genotype, however our use of the expected B-allele count ensures that as the information from the expression data decreases the predicted genotype already tends to the AB genotype. Thus we do not wish to punish too heavily differences in observed and predicted B-allele counts when the predicted count is 1. Nor do we wish to punish errors that could be genotype miscalling, which will nearly always be between heterozygous and homozygous rather than going from AA to BB.A convenient measure of signal quality for the Illumina expression BeadArray is the 95th percentile of log-intensities (P95) reported by the scanner. This is a general indication of the signal level, and while a useful quality statistic, it has no absolute interpretation as batch effects and scanner settings can cause large variation in the p95 value. This is true even after adjusting for the similarly defined P05 value which can be regarded as a baseline.We consider a second metric for the quality of signal on an array. This statistic we denote For an overview of Illumina expression array quality metrics, see Ritchie et al. In the majority of cases the genotype array that is meant to match the expression array will have a BADGER rank of 1. This genotype and expression array can then be removed from the analysis. The number of potential plating errors will hopefully be small enough that the manual assessment of each case will be feasible. Our experience is that the majority of simple cases can be resolved easily by considering neighbouring arrays as potential matches . Care must be taken that one is identifying the entire error and not just a part of it. For example if the order of four consecutive arrays was reversed, then the middle two arrays might be identified as a simple pair of swapped neighbours and removed from further analysis, hindering the detection of the nature of the error for the outer two arrays.If the design of the experiment allows, indeed if plating details are available for all stages, then we may be able to write code to investigate likely plating errors  systematically, and report if any substantially increase the agreement between expression and genotype arrays. Else we can use graph theoretic techniques to suggest errors.The problem can be represented as a bipartite graph where one set of nodes represents unresolved expression arrays and the other set of nodes represents unresolved genotyping arrays. An edge is then drawn between two nodes if the BADGER rank for that expression array/genotyping array combination is less than some threshold. Swaps will then appear as short cycles in the graph, more complicated rearrangements will appear either as long cycles  or long chains on the graph. These can be detected algorithmically or spotted easily by eye if the graph is visualized.Where BADGER suggested that there existed an inconsistency between the expression and genotype data, and the scale of the inconsistency is small enough that the BADGER prediction alone is not sufficient evidence, validation has proven possible via the residual DNA to be found in the wells of RNA extraction plates. This approach is, of course, dependent on enough DNA being present in those wells.http://badger.r-forge.r-project.org/.Sweave S1Sweave file (ZIP)Click here for additional data file.Sweave S2Sweave file (ZIP)Click here for additional data file.


Phytophthora sojae from training sets including phenotype, genotype, and gene expression data. The challenge test set was divided into three subcategories, one requiring prediction based on only genotype data, another on only gene expression data, and the third on both genotype and gene expression data. Here we present our approach, primarily using regularized regression, which received the best-performer award for subchallenge B2 (gene expression only). We found that despite the availability of 941 genotype markers and 28,395 gene expression features, optimal models determined by cross-validation experiments typically used fewer than ten predictors, underscoring the importance of strong regularization in noisy datasets with far more features than samples. We also present substantial analysis of the training and test setup of the challenge, identifying high variance in performance on the gold standard test sets.A major goal of large-scale genomics projects is to enable the use of data from high-throughput experimental methods to predict complex phenotypes such as disease susceptibility. The DREAM5 Systems Genetics B Challenge solicited algorithms to predict soybean plant resistance to the pathogen  Predicting complex phenotypes from genotype or gene expression data is a key step toward personalized medicine: the use of genomic data to improve the health of individuals, for instance by predicting susceptibility to disease or response to treatment It is difficult to objectively measure progress on algorithmic challenges without standard benchmarks; within this context, the Dialogue for Reverse Engineering Assessments and Methods (DREAM) initiative As a top performer on the second part of the challenge, we were invited to present our results at the DREAM5 conference and contribute to the DREAM5 collection in PLoS ONE; this paper describes our approach. We provide a comparison of several regularized regression models and find comparable performance of elastic net, lasso, and best subset selection. We also carefully analyze the level of noise in the data and consequent variability in performance and offer practical suggestions for similar data analysis and data pre-processing.Phytophthora sojae, were crossed and their offspring were inbred for more than 12 generations to produce a population of recombinant inbred lines (RILs). Individuals within each RIL exhibited almost no genetic variation, whereas distinct RILs displayed much genetic variation owing to their differing mixtures of parental genes. Each RIL was screened for 941 genetic variants and gene-expression profiled for 28,395 genes; gene expression was measured in uninfected plants because the goal of the challenge was to predict disease susceptibility using only information gathered under normal  conditions.The data for this challenge were collected from a systems genetics experiment conducted at the Virginia Bioinformatics Institute P. sojae, the plants were assayed for two continuous phenotypes, each a measurement of the amount of pathogen RNA in the infected tissue sample. The first phenotype measured the fraction of pathogen probe sets that yielded a detectable hybridization signal as determined by the MAS5 presence/absence call in the Affymetrix software used to analyze the data. The second phenotype measured the ratio between the sum of all background-subtracted soybean probe intensities and the sum of all background-subtracted pathogen probe intensities. We abbreviate the two phenotypes as P1 and P2.After infection with http://wiki.c2b2.columbia.edu/dream/index.php/D5c3).The training data, from 200 RILs, thus consisted of a We began our analysis for this challenge by computing correlation coefficients of the genotype and gene expression training features against the two phenotype variables. The magnitudes of these correlations guided our choice of modeling technique; we also later used correlation-sorted rank lists to limit the scope of computationally intense calculations to those features most likely to be relevant.On first glance the highest correlations, above 0.3 for the expression data , appear These observations suggest that most features have little or no predictive power, and hence proper regularization is crucial for modeling this dataset. Additionally, the small difference between training correlations and the random background distribution indicate that the prediction task at hand is difficult; the amount of signal in the data is likely quite small.In light of the above considerations, we sought to keep our modeling simple and chose regularized regression as our general approach. Before fitting the data, however, we needed to ensure that the relation between predictor and response variables was as linear as possible, and so we considered data transformations and basis expansions.Upon plotting the phenotype training data, we discovered that the variance in the distribution of phenotype 1 is dominated by outliers. Among the 200 measurements of phenotype 1, the largest outlier is 5.83 sample standard deviations from the mean. Moreover, the seven most deviant samples account for more than half of the total variance. For phenotype 2, the largest outlier is a substantial 3.77 standard deviations above the mean but overall the distribution does not have unusually long tails compared to a normal distribution. A plot of the fractions of variance explained by increasing subsets of largest outliers in phenotype 1, phenotype 2, and random data illustrates this behavior .Motivated by the Spearman correlation-based scoring scheme used in this challenge, which judges predictions based on ordering rather than absolute accuracy, we applied a rank transformation to phenotype 1 to remove the impact of outliers on regression models. More precisely, we replaced the numerical values of phenotype 1 measurements with their ranks among the 200 sorted samples. Because the approaches we applied minimized squared error , asking our models to predict ranks rather than actual values removed the heavy weight that outlier values would otherwise have received. Absolute predictions could of course be recovered by interpolation if desired.With only binary genotype data available for prediction in subchallenge B1, we hypothesized that the true phenotypic response for a genotyped sample would be far from linear. The simplest possible example of a nonlinear effect is interaction between genotype markers: for instance, if two genes act as substitutes for one another, their function is only suppressed if both are turned off. Similarly, if two genes are critical to different parts of a pathway, turning off either one would impair its function.With these examples in mind, we considered applying logic regression To gauge the efficacy of these combined features, we compared the largest fractions of variance explained by single boolean combination features (using single-variable least-squares regression) to the best fits obtained by two-variable regression on pairs of the original genotype features. Looking at the 20 best-performing regressions from each group , we see An important caveat to keep in mind when interpreting these measurements is that the number of feature combinations considered is very large (nearly 2 million), thus allowing random chance to inflate best performances as in the case of correlations examined above. Nonetheless, we expect that the relative trends are still informative.Upon closer inspection of the best boolean combination markers, we discovered that some were near-trivial due to linkage disequilibrium : for insHaving taken steps to linearize the predictor-response relationship, we applied regularized regression to model the data. Classical linear regression on a predictor matrix Our main approach of choice was elastic net regression For the purpose of comparison, we also tried fitting the data with a simple best subset selection approach, which seeks to minimize squared error using only a limited number of regressors. . We then used simulated annealing to compute subsets of size 1–20 features obtaining approximately optimal linear fits to each training fold. The annealing procedure consisted of 5 runs of initialization with a random feature subset of the required size followed by 5000 iterations of attempted swaps, using a linear cooling schedule. Explictly, the acceptance probability of a swap was We evaluated our regression methods using 7-fold cross-validation on the 200-sample training set, measuring goodness of fit with Spearman correlation to match the DREAM evaluation criterion. We chose to use 7 folds so that our cross-validation test sets during development would have approximately the same size as the 30-sample gold standard validation set, allowing us to also estimate the performance variance to be expected on the validation set. We applied each regression technique–elastic net, lasso, and approximate best subset selection with simulated annealing–to fit phenotype 1 (rank-transformed) and phenotype 2 individually, using sets of regressors corresponding to the three subchallenges of DREAM5 Systems Genetics B: genotype only (B1), gene expression only (B2), and both genotype and expression (B3). Within subchallenge B1, we ran two sets of model fits, one using only raw genotype markers as regressors and the other using the boolean basis expansion described in Methods.Because of the relatively small number of samples and large number of predictors, the random assignment of samples to cross-validation folds caused substantial fluctuation in performance, even when averaging across folds. We overcame this difficulty by running multiple cross-validation tests for each model fit using different fold assignments in each run (20 replicates for elastic net and lasso and 5 replicates for best subset selection), thus obtaining both mean performances and estimates of uncertainty in each mean. We chose regularization parameters for each method in each situation to optimize mean performance; Overall, the three regularized regression techniques perform quite comparably. Note that elastic net regression necessarily always performs at least as well as lasso . Unfortunately, our regression models did not attain a performance increase from B2 to B3 with the inclusion of genotype data along with expression data, nor did boolean basis expansion appear to help with performance on B1.Surprisingly, the rank transformation we applied to phenotype 1 turned out to have the greatest impact of the pre-regression data transformations we attempted. For the purpose of comparison, we performed the same model-fitting as above using raw (untransformed) values of phenotype 1. In all cases the rank transformation increases average Spearman correlations considerably . For subTaking a closer look at the optimal regularization parameters for elastic net, lasso, and approximate best subset selection, we discovered strikingly low model complexity prescribed by cross-validation in each case. As an example, the blue curves of With lasso, we likewise see that performance drops off quickly as model complexity increases; here, the complexity parameter To better understand the strong regularization, we provide heat maps displaying the feature weight distributions chosen by the elastic net to predict phenotype 1 (rank-transformed) and phenotype 2 for a set of cross-validation runs on subchallenge B2 . As expeAs mentioned earlier, our cross-validation analysis also allows us to estimate the accuracy to which algorithm performance can be measured using a 30-sample test set. Unfortunately, we find that this test size is insufficient for accurate evaluation: whereas the greatest-weight features selected by our models are relatively stable from fold to fold , the SpeThe red curves of Notwithstanding the caveat just discussed regarding uncertainty in results on a small test size, we include the final results from the DREAM5 Systems Genetics B challenge for completeness . Our teaP. sojae; on the contrary, we found that given the limitations of small sample size and noise in the training data, the best models we discovered were among the simplest we tried. Regularized least squares regression with careful cross-validation and linearization (using the rank transform we applied to phenotype 1) proved to be as effective an approach as any other we are aware of, and the noise-to-signal in the data was such that the best linear fits needed only a few well-chosen regressors.While the performance achieved by our methods–indeed, by every team's methods–is modest, our work does highlight a few important lessons in statistical learning and in the setup of algorithmic benchmarking challenges such as DREAM. Regarding the first, our analysis did not lead us to a radically new and complex model for the genotype-phenotype relationship in One might hope that the transparency of such simple models can shed light on the underlying biological mechanism at work; while this may be possible, we also should caution against trying to glean more from the models than the data allow. Simplicity may be due to the involvement of only relatively few genes or just to the fact that heavy regularization makes models less prone to overfitting. In light of the noisiness of the dataset, we suspect the latter may be true. As a case in point, while we were disappointed that modeling pairwise interactions through boolean basis expansion did not improve fitting using the genotype data, we still find it quite plausible that such effects are at work and may aid modeling in situations when more data is available. With this dataset, our techniques were likely unable to discern these effects because the limited data size could not support the increased complexity that modeling interactions would entail.Overall, while this contest was perhaps too ambitious for the data available, we feel it succeeded in stimulating research and discussion in the field. The original motivation of developing methodology for combining genotype and gene expression data to improve phenotype prediction remains a worthy goal and interesting open question.


Colon cancer (CC) pathological staging fails to accurately predict recurrence, and to date, no gene expression signature has proven reliable for prognosis stratification in clinical practice, perhaps because CC is a heterogeneous disease. The aim of this study was to establish a comprehensive molecular classification of CC based on mRNA expression profile analyses.BRAF, KRAS, and TP53 mutations, CpG island methylator phenotype, mismatch repair status, and chromosomal instability status, and were screened with whole genome and transcriptome arrays. 566 samples fulfilled RNA quality requirements. Unsupervised consensus hierarchical clustering applied to gene expression data from a discovery subset of 443 CC samples identified six molecular subtypes. These subtypes were associated with distinct clinicopathological characteristics, molecular alterations, specific enrichments of supervised gene expression signatures , and deregulated signaling pathways. Based on their main biological characteristics, we distinguished a deficient mismatch repair subtype, a KRAS mutant subtype, a cancer stem cell subtype, and three chromosomal instability subtypes, including one associated with down-regulated immune pathways, one with up-regulation of the Wnt pathway, and one displaying a normal-like gene expression profile. The classification was validated in the remaining 123 samples plus an independent set of 1,058 CC samples, including eight public datasets. Furthermore, prognosis was analyzed in the subset of stage II–III CC samples. The subtypes C4 and C6, but not the subtypes C1, C2, C3, and C5, were independently associated with shorter relapse-free survival, even after adjusting for age, sex, stage, and the emerging prognostic classifier Oncotype DX Colon Cancer Assay recurrence score . However, a limitation of this study is that information on tumor grade and number of nodes examined was not available.Fresh-frozen primary tumor samples from a large multicenter cohort of 750 patients with stage I to IV CC who underwent surgery between 1987 and 2007 in seven centers were characterized for common DNA alterations, including We describe the first, to our knowledge, robust transcriptome-based classification of CC that improves the current disease stratification based on clinicopathological variables and common DNA markers. The biological relevance of these subtypes is illustrated by significant differences in prognosis. This analysis provides possibilities for improving prognostic models and therapeutic strategies. In conclusion, we report a new classification of CC into six molecular subtypes that arise through distinct biological pathways.Please see later in the article for the Editors' Summary Cancer of the large bowel  is the third most common cancer in men and the second most common cancer in women worldwide. Despite recent advances in the screening, diagnosis, and treatment of colorectal cancer, an estimated 608,000 people die every year from this form of cancer—8% of all cancer deaths. The prognosis and treatment options for colorectal cancer depend on five pathological stages (0–IV), each of which has a different treatment option and five year survival rate, so it is important that the stage is correctly identified. Unfortunately, pathological staging fails to accurately predict recurrence (relapse) in patients undergoing surgery for localized colorectal cancer, which is a concern, as 10%–20% of patients with stage II and 30%–40% of those with stage III colorectal cancer develop recurrence.Previous studies have investigated whether there are any possible gene expression profiles (identified through microarray techniques) that can help predict prognosis of colorectal cancer, but so far, there have been no firm conclusions that can aid clinical practice. In this study, the researchers used genetic information from a French multicenter study to identify a standard, reproducible molecular classification based on gene expression analysis of colorectal cancer. The authors also assessed whether there were any associations between the identified molecular subtypes and clinical and pathological factors, common DNA alterations, and prognosis.The researchers used genetic information from a cohort of 750 patients with stage I to IV colorectal cancer who underwent surgery between 1987 and 2007 in seven centers in France. The researchers identified relevant clinical and pathological staging information for each patient from the medical records and calculated recurrence-free survival (the time from surgery to the first recurrence) for patients with stage II or III disease. In the genetic analysis, 566 tumor samples were suitable—443 were used in a discovery set, to create the classification, and the remainder were used in a validation set, to test the classification. The researchers also used information from eight public datasets to validate their findings.Using these methods, the researchers classified the colon cancer samples into six molecular subtypes (based on gene expression data) and, on further analysis and validation, were able to distinguish the main biological characteristics and deregulated pathways associated with each subtype. Importantly, the researchers found that that these six subtypes were associated with distinct clinical and pathological characteristics, molecular alterations, specific gene expression signatures, and deregulated signaling pathways. In the prognostic analysis based on recurrence-free survival, the researchers found that patients whose tumors were classified in one of two clusters (C4 and C6) had poorer recurrence-free survival than the other patients.These findings suggest that it is possible to classify colorectal cancer into six robust molecular subtypes that might help identify new prognostic subgroups and could provide a basis for developing robust prognostic genetic signatures for stage II and III colorectal cancer and for identifying specific markers for the different subtypes that might be targets for future drug development. However, as this study was retrospective and did not include some known predictors of colorectal cancer prognosis, such as tumor grade and number of nodes examined, the significance and robustness of the prognostic classification requires further confirmation with large prospective patient cohorts.http://dx.doi.org/10.1371/journal.pmed.1001453.Please access these websites via the online version of this summary at colorectal cancer and also about how colorectal cancer is stagedThe American Cancer Society provides information about colon and rectal cancer and colon cancer stagesThe US National Cancer Institute also provides information on  BRAF and KRAS mutations) Despite advances in screening, diagnosis, and treatment, colorectal cancer (CRC) is the third most common cancer and the fourth-leading cause of cancer death worldwide In this study, we exploited a large, multicenter, and extensively characterized series of CC samples to establish a robust molecular classification based on genome-wide mRNA expression analysis. Then we assessed the associations between molecular subtypes and clinicopathological factors, common DNA alterations, and prognosis. To confirm the robustness of the subtypes obtained, we further validated our molecular classification in a large independent set.The use of the tumor collection was approved by the following ethics committees and institutional boards: lle de France II , Marseille , Strasbourg , the Human Research Ethics Committee of Saint-Antoine Hospital , the Toulouse Hospital board , and Nice . The informed consent of the patients was recorded as required by a French law in force until 2007. Since the last inclusion in this study was 2007, the standard hospital blanket consent was considered sufficient.The French national Cartes d'Identité des Tumeurs (CIT) program involves a multicenter cohort of 750 patients with stage I to IV CC who underwent surgery between 1987 and 2007 in seven centers. Fresh-frozen primary tumor tissue samples were retrospectively collected at the Institut Gustave Roussy (Villejuif), the Hôpital Saint Antoine (Paris), the Hôpital Européen Georges Pompidou (Paris), the Hôpital de Hautepierre (Strasbourg), the Hôpital Purpan (Toulouse), and the Institut Paoli-Calmettes (Marseille), and prospectively collected at the Centre Antoine Lacassagne (Nice). Patients who received preoperative chemotherapy and/or radiation therapy and those with primary rectal cancer were excluded from this study. Clinical and pathologic data were extracted from the medical records and centrally reviewed for the purpose of this study. Patients were staged according to the American Joint Committee on Cancer tumor node metastasis (TNM) staging system n = 443) and a validation set (n = 123), well balanced for the main anatomoclinical characteristics (n = 457) and/or patient outcome (n = 449) data available. Within the discovery (n = 443) and the validation  sets, 359 and 416 patients with stage II–III CC and documented relapse-free survival (RFS) were available for survival analysis, respectively  mutation was assessed by allelic discrimination using TaqMan probes and the same protocol as that for KRAS mutations. TP53 mutations (exons 4–9) were assessed as previously described The seven most frequent mutations in codons 12 and 13 of http://www.ncbi.nlm.nih.gov/geo/; accession number GSE39582).The GEP of 566 primary CC samples were determined on Affymetrix U133 Plus 2.0 chips. For 19 patients, adjacent non-tumor tissue  was also available and was tested. The methods used for RNA purification, quality control, fluorescent probe production, hybridization, and raw data processing were as previously described A total of 464 of the 750 primary CC samples from the CIT cohort could be analyzed for array-based comparative genomic hybridization (CGH) on a BAC array containing 4,434 bacterial artificial chromosome clones with a median resolution of 0.6 Mb. DNA labeling, hybridization, and data processing were as previously described n = 1,459). To obtain a robust classification, we used a consensus unsupervised approach Unsupervised classification of the discovery set was performed using hierarchical clustering (Ward linkage and 1 − Pearson correlation coefficient distance used) on the most variant class of probe sets  GEP of NT counterparts from our dataset; (ii) previously published supervised signatures based on intestinal stem cell phenotype The ColoPrint and Oncotype DX prognostic classifiers Survival analysis was intentionally restricted to the subgroup of patients with stage II–III tumors because reliable prognostic biomarkers are most needed for these patients. Indeed, most stage I patients will not derive benefit from adjuvant chemotherapy because of their excellent prognosis after curative surgery, and most stage IV patients, already metastatic, will die from their disease and therefore should be analyzed independently for progression-free survival. RFS was defined as the time from surgery to the first recurrence and was censored at 5 y. Survival was analyzed according to the Kaplan-Meier method, and differences between survival distributions were assessed with the log-rank test. Univariate and multivariate models were computed using Cox proportional-hazards regression  : C1 , C2 , C3 , C4 , C5 , and C6   . The con45, 10%) . Based o45, 10%) , cluster45, 10%) , and C6,TP53-mutant, and distal (p<0.001), without any other molecular or clinicopathological features able to discriminate these three clusters clearly. Tumors classified as C2, C4, and C3 were more frequently CIMP+  and proximal. C2 was enriched for dMMR (68%) and BRAF-mutant tumors (40%). C3 was enriched for KRAS-mutant tumors (87%). No association between clusters and TNM stage was found, except enrichment for metastatic (31%) tumors in C4.Associations with anatomoclinical and DNA alterations data are shown in The analyses of CGH arrays revealed that CIN+ samples shared a typical DNA copy alteration pattern including +7, −8p, +8q, +13q, −17p, −18, +20q. Differences between subtypes mainly reflected their relative content of CIN+ samples. However, some specific alterations were observed for the two CIN subtypes, C5  and C1  .We analyzed cancer-related signaling pathways from the Kyoto Encyclopedia of Genes and Genomes database for specific deregulation in each subtype signature . As expeThese six molecular subtypes were further investigated using GEP data from NT and previously published supervised signatures based on DNA alterations and cellular phenotypes to explore the subtype origins. Based on the growing amount of data suggesting that cancer is closely linked to stem cells, a mouse-derived intestinal stem cell signature As previously described for breast cancer BRAF mutations and the serrated adenoma pathway has been reported BRAF-mutant-like supervised signature has been described by Popovici et al. BRAF wild-type subgroup, 30% of which were KRAS mutants and 13% of which were double wild-type CC. This signature was also applied to our GEP data: subtypes C2, C3, and C4 were enriched in BRAF-mutant-like GEP tumors.Serrated CC, in contrast to conventional CC, may arise through a recently introduced serrated neoplasia pathway ImmuneDown”; C2, “dMMR”; C3, “KRASm” (for “KRAS-mutant”); C4 “CSC” (for “cancer stem cell”); C5, “CINWntUp”; and C6, “CINnormL”.A schematic summary of the subtype characteristics is shown in BRAF-mutant and stage IV tumors. When applied to the Agilent TCGA dataset (n = 152) p<10−16).To validate our six-subtype classification, a 57-gene centroid classifier was built from the discovery set by a 10-fold cross-validation approach  differed, but not significantly so, with patients whose tumors were classified as C4 and C6 having a relatively poorer outcome  (n = 416) (p = 0.0009), with a worse prognosis confirmed for patients with C4 and C6 tumors (p = 0.0003)  (p<0.01) when common DNA alterations  were added to the model  . The pro6 tumors ; The six 0.0003) . To compdataset)  and S6 adataset)  and S6. he model .p = 3.4×10−6; The Oncotype DX recurrence score We also attempted an exploratory analysis of the signature described by Salazar et al. Using a large comprehensively characterized multicenter cohort of CC patients, we identified six robust molecular subtypes of CC individualized by distinct clinicobiological characteristics. Importantly, this six-subtype classification was validated in nine independent datasets. Furthermore, classification into high- and low-risk subtypes was of prognostic value.Although retrospective, our cohort was very representative of the clinicopathological characteristics and common DNA alteration frequencies observed in the population of patients with CC.BRAF was associated with the dMMR subtype, but was also frequent in the C4 CIMP+ poor prognosis subtype. TP53- and KRAS-mutant tumors were found in all the subtypes; nevertheless, the C3 subtype, highly enriched in KRAS-mutant CC, was individualized and validated, suggesting a specific role of this mutation in this particular subgroup of CC. There was no significant association between our classification and pathological stage, suggesting that tumor subtype is established at the initial stages.Our findings clearly demonstrate that anatomoclinical factors and common DNA alterations alone are helpful for highlighting subtype characteristics, but they are not sufficient to define boundaries between subtypes and to describe the molecular heterogeneity of CC. Our classification successfully identified the dMMR tumor subtype, and also individualized five other distinct subtypes among pMMR tumors, including three CIN+ CIMP− subtypes representing slightly more than half of the tumors. As expected, mutation of KRAS-mutant tumors Exploratory analysis of each subtype GEP with previously published supervised signatures and relevant deregulated signaling pathways improved the biological relevance of the classification. Indeed, this analysis suggested that different types of CC may arise from distinct cell origins, and distinguished between the two main pathways, defined as the serrated and the conventional precursor neoplasia pathways. Interestingly, we not only individualized the dMMR subtype among the serrated precursor neoplasia subtypes, but also within the C4 CSC and the C3 KRASm subtypes. This finding is consistent with the serrated polyp classification showing two main groups: the sessile serrated adenomas, commonly associated with dMMR tumors, and the traditional serrated adenomas, commonly associated with secreted frizzled-related protein 2 (SFRP2), described as a key factor in stem cell regulation growth arrest-specific 1 (GAS1)—were included in the poor prognosis cluster signature reported by Oh et al. Another interesting finding is the reproducible association between the stem cell signature and the poor prognosis C4 subtype. Almost half of the top genes deregulated in C4—including normL subtype was more difficult to characterize; it belongs to the CIN+ subgroup but has a GEP and RFS that are distinct from those of the other two CIN subtypes. Several genes up-regulated in C6, in particular carbonic anhydrase II (CA2) and solute carrier family 4, sodium bicarbonate cotransporter, member 4 (SLC4A4), were also included in the prognostic classifier described by Lin et al The C6 CINThe two other CIN subtypes, C1 and C5, were more difficult to distinguish from each other. They show common clinical and DNA alteration characteristics. They share some gene expression patterns, leading to lower co-classification rates than for the other subtypes. Moreover, these two subtypes are combined if the number of clusters is set to five instead of six. As a result, the division of C1 and C5 into two distinct subtypes can be questioned . HoweverBRAF-mutant tumors and may correspond to the poor prognostic cluster reported by Salazar et al. containing the same proportion of BRAF-mutant tumors The biological relevance of our subtypes was highlighted by significant differences in prognosis. In our unsupervised hierarchical clustering, patients whose tumors were classified as C4 or C6 had poorer RFS than the other patients. Thus, our study, like others BRAFwt . Indeed, the markers BRAF-mutant, CIMP+, and dMMR may be useful for classifying a small proportion of cases, but are uninformative for a large number of patients. This was illustrated in the study by Salazar et al. in which BRAF mutation was found in both good and poor outcome clusters, but was rare in the intermediate prognosis cluster used to build the ColoPrint prognostic classifier Prognostic analyses based solely on common DNA alterations can distinguish between risk groups, but are still inadequate, as most CCs are pMMR CIMP− The ColoPrint and Oncotype DX prognostic classifiers were developed recently to improve risk prediction in early-stage CRC Our multivariate analysis has some limitations. In particular, some established predictors of CC prognosis, notably tumor grade and number of nodes examined, were not included because this information was not available for a substantial proportion of cases In conclusion, we report a new classification of CC into six robust molecular subtypes that arise through distinct biological pathways and represent novel prognostic subgroups. Our study clearly demonstrates that these gene signatures reflect the molecular heterogeneity of CC. This classification therefore provides a basis for the rational design of robust prognostic signatures for stage II–III CC and for identifying specific, potentially targetable markers for the different subtypes.Figure S1Discovery and validation sets used in the study. The data used in this study were collected from the CIT program cohort (a French multicenter cohort) and from publicly available datasets. There were 750 CC samples from the CIT program suitable for common DNA alteration characterization, and 566 of these provided tumor RNA samples satisfying stringent quality control criteria. These RNA were hybridized on an Affymetrix chip (asterisk) and used for molecular subtype determinations. The discovery set was composed of 443 tumors from the CIT cohort. The validation set was composed of the remaining CIT cohort CC samples, CC samples from seven Affymetrix publicly available datasets (indicated with their NCBI GEO accession number), and CC samples from the non-Affymetrix TCGA program (performed on an Agilent platform). For survival analyses, only stage II and III patients were considered, stage I and IV patients not being informative as almost all survive or die, respectively; there were thus 359 cases in the CIT discovery set and 416 in the CIT validation set and three public datasets included in this analysis. pbs, probe sets.(PDF)Click here for additional data file.Figure S2Heatmaps of subtype-discriminant probe set expression profiles in the discovery set and in the Affymetrix validation set. (A) Heatmap of the discovery set samples ordered according to gene hierarchical clustering  and by subtypes. (B) Heatmap of Affymetrix validation set samples ordered as in (A). For each subtype, discriminant probe sets were selected from the discovery set using a moderated t-test, comparing the given subtype to the other subtypes, with an adjusted p<10−5 and a |log fold change|>0.5, yielding 1,108 discriminant probe sets.(PDF)Click here for additional data file.Figure S3Associations between molecular subtypes and anatomoclinical characteristics, DNA alterations, and supervised signature annotations in the discovery and validation sets. Associations were assessed in (A) the Affymetrix discovery set, (B) the Affymetrix validation set, and (C) the TCGA, non-Affymetrix, validation set. For each subtype and variable, the proportion of each modality is represented , and the percent of the main feature (dark grey) within each subtype is indicated. The Chi-squared test p-values are indicated in red.(PDF)Click here for additional data file.Figure S4Subtype genomic alteration profiles along the genome. CC molecular subtypes present different copy-number-change profiles. The profiles were established using genome-wide array-based CGH available for 356 samples. (A) Frequencies of gains (frequency>0) and losses (frequency<0) observed at a given location on the genome are shown for all samples  and by subtype . (B) Subtype-specific genomic regions of copy-number change. Bars represent significant p-values , after a logarithmic transformation, for the differences in the proportions of samples with each chromosomal abnormality between the different subtypes. For all samples, regions having an alteration frequency higher than 20% are displayed.(PDF)Click here for additional data file.Figure S5Determination of subtype prediction centroids. (A) Percentage of misclassification of the discovery set as a function of the number of top up- and down-regulated gene pairs used in the centroids. Misclassification is computed for the validation set by a 10-fold cross-validation procedure and is plotted by subtype (top) and averaged (bottom). (B) Heatmap of the 57-gene centroids used to assign a new dataset.(PDF)Click here for additional data file.Figure S6Prognostic value of the recoded CIT classification and of the Oncotype DX–like and Coloprint-like prognostic classifiers in the discovery and validation sets for patients with TNM stage II or III CC. RFS according to the recoded molecular subtype classification (C4/C6 subtypes versus other subtypes) in each TNM stage category , RFS of high- and low-risk patients as predicted by the Oncotype DX–like classifier, and RFS of patients belonging to cluster 1 and cluster 2 of the ColoPrint 17-gene expression signature in the discovery set (top), the validation set (middle), and the both datasets combined (bottom).(PDF)Click here for additional data file.Figure S7Prognostic Oncotype DX–like classifier within each CIT molecular subtype in the combined discovery and validation sets. RFS curves of high- and low-risk patients as predicted by the Oncotype DX–like classifier within each of the six CIT molecular subtypes.(PDF)Click here for additional data file.Figure S8Selection of the number of clusters. (A) Cumulative distribution function plot for each tested number of clusters; (B) cumulative distribution function delta area plot; (C) consensus matrix for different numbers of clusters (k = 5 to 8).(PDF)Click here for additional data file.Table S1Patient and tumor characteristics. CIMP+/−, 3–5 methylated markers/0–2 methylated markers; CIN+/−, CIN>20%/CIN≤20%; F/M, female/male; WT/M, wild-type/mutant.(XLS)Click here for additional data file.Table S2List of the 1,459 most variant probe sets used to perform unsupervised analysis. The GeneCluster column corresponds to the gene cluster letters in p-values of the moderated t-test comparing Cj versus the other subtypes.(XLS)Click here for additional data file.Table S3List of the 1,108 subtype-discriminant probe sets. For each subtype, discriminant probe sets were selected from the discovery set using a moderated t-test, comparing a given subtype to the other subtypes (adjusted p<10−5 and a |log fold change|>0.5), yielding 1,108 discriminant probe sets. The GeneCluster column corresponds to the gene cluster letters in p-values of the moderated t-test comparing Cj versus the other subtypes.(XLS)Click here for additional data file.Table S4Associations of anatomoclinical characteristics, DNA alterations, and supervised signature annotations with the six subtypes based on logistic regression. Associations were assessed by logistic regression using a multinomial logit model .(XLS)Click here for additional data file.Table S5List of the 57 genes used to assign subtypes. The 57 genes selected to build the subtype predictor, given with each subtype's centroid values.(XLS)Click here for additional data file.Table S6Univariate and multivariate Cox analyses including the classification and clinical annotations. Associations of the classification and clinical annotations with RFS were assessed by Cox proportional-hazards regression analyses on (A) the discovery set, (B) the validation set, and (C) both sets. Univariate Cox analyses were performed on each variable independently. The best multivariate model was determined by using a backward–forward selection approach to restrict the multivariate model to the most informative variables for the subset of samples for which all the variables were available. Value indicates the modality of the annotation associated with the HR. H.R., Cox HR.(XLS)Click here for additional data file.Table S7Univariate and multivariate Cox analyses including the classification and other molecular annotations. Associations with RFS of the six-subtype classification—including BRAF, KRAS, and TP53 mutations, MMR status, and CIMP status—were assessed by Cox proportional-hazards regression analyses on the discovery set. Univariate Cox analyses were performed on each variable independently. The best multivariate model was determined by using a backward–forward approach to restrict the multivariate model to the most informative variables for the subset of samples for which all the variables were available. The TP53 mutation variable was excluded from the multivariate analysis, as only 201 samples were characterized and as it was not significantly associated to outcome. Value indicates the modality of the annotation associated with the HR. H.R., Cox HR.(XLS)Click here for additional data file.Text S1Supplementary methods.(PDF)Click here for additional data file.