Formalin fixed, paraffin embedded tissues are most commonly used for routine pathology analysis and for long term tissue preservation in the clinical setting. Many institutions have large archives of Formalin fixed, paraffin embedded tissues that provide a unique opportunity for understanding genomic signatures of disease. However, genome-wide expression profiling of Formalin fixed, paraffin embedded samples have been challenging due to RNA degradation. Because of the significant heterogeneity in tissue quality, normalization and analysis of these data presents particular challenges. The distribution of intensity values from archival tissues are inherently noisy and skewed due to differential sample degradation raising two primary concerns; whether a highly skewed array will unduly influence initial normalization of the data and whether outlier arrays can be reliably identified.Two simple extensions of common regression diagnostic measures are introduced that measure the stress an array undergoes during normalization and how much a given array deviates from the remaining arrays post-normalization. These metrics are applied to a study involving 1618 formalin-fixed, paraffin-embedded HER2-positive breast cancer samples from the N9831 adjuvant trial processed with Illumina’s cDNA-mediated Annealing Selection extension and Ligation assay.Proper assessment of array quality within a research study is crucial for controlling unwanted variability in the data. The metrics proposed in this paper have direct biological interpretations and can be used to identify arrays that should either be removed from analysis all together or down-weighted to reduce their influence in downstream analyses. Many institutions have large archives of formalin-fixed paraffin-embedded (FFPE) tissue. Compared to the general availability, sample collection protocols, and time-sensitive nature of fresh-frozen tissue, these large archives of FFPE tissues are easily assessable and provide a unique opportunity for understanding genomic signatures of disease on a large scale as well as the ability to evaluate long-term prognostic associations ,2. TheseThe prevailing consensus in the literature is that normalization is a necessary step for microarray platforms in order to adjust for the influence of factors extraneous to the primary biological question such as sample preparation, scanner efficiency, and cross hybridization of probes ,12. A fulumi pipeline [Although normalization will equalize the distribution of feature intensities across the arrays, there remains a need to assess the quality of the data. For example, of 7 FFPE experiments submitted to Gene Expression Ominbus  only the latter two studies acknowledged that array quality assessments were even conducted and neither of these two studies reported their findings -17. Recepipeline . AlthougStress) and 2) a measure of how much a single array deviates from the remaining arrays within the experiment post-normalization (dfArray). We compare our findings to currently available metrics for FFPE samples using the DASL assay and show the benefit of removing arrays of questionable quality from an experiment where differential expression is the primary objective.In this work, we introduce two metrics that easily can be used to assess microarray quality regardless of the platform under consideration and have direct clinical interpretations. These two metrics are used 1) to measure how much data from a single microarray needs to be “stretched” during the normalization process in order to make its marginal distribution match with the remaining arrays , which was a Phase III trial where patients were randomized to three arms: (Arm A) doxorubicin and cyclophosphamide followed by weekly paclitaxel, (Arm B) same as Arm A but followed by 1 year of sequential trastuzumab, or (Arm C) same as Arm A but with 1 year concurrent trastuzumab started the same day as weekly paclitaxel . PatientIij of the ith feature  from the jth sample  is expressedWe begin with a description of a basic statistical model for microarray data and will follow the notation as described by McCall and others -23. The Oij represents background intensity present in the data due to scanner inefficiencies and non-specific binding of probes. This background is typically subtracted from the data using vendor-specific methods or user specified packages. We leave it up to the user to specify which correction is to be used and simply move to the commonly used log-linear model form of (1)The term Yij denotes the intensity values after background correction, μij = log2θij represents the “true” relative amount of a feature hybridized to the array and is the primary parameter of interest in microarray experiments, Sij = log2Kij represents systematic biases, and εij = log2θij represents random variation with mean 0 and variance σi with the subscript indicating that the variance is feature specific.Where Sij represents an arbitrary bias function for the ith feature on the jth array and is assumed to be independent of the remaining parameters in equation (2). Examples of biases might be variations in sample dilution that would add a constant value to probes on the array, or other more complicated effects. The bias function is estimated using any number of user-specified normalization routines of which the most popular is quantile normalization and is used throughout this work [The term his work . We denoY′ij. Bolstad’s first metric referred to as the Relative Log Expression (RLE) is defined asBolstad et al.  and McCaRLE is used to determine if a particular array has predominately low- or high-expressed features as indicated by an overall shift. This metric is easily applicable to any microarray platform. However, for normalization routines that leverage probe-specific information such as loess, RLE ≅ 0 by definition so one does not expect to see large shifts. Moreover, the spread in the distribution of RLE is not independent of feature variance σi2. This makes distribution summaries difficult to interpret for the purpose of outlier detection as an outlier for a particular feature can be masked by the other features with large variance.and compares a given array’s feature intensity relative to the median level of intensity for that feature across all j arrays. The array-specific distribution of Normalized Unscaled Standard Error; NUSE) and later modified by McCall , was developed for evaluating the quality of Affymetrix array data. Both measure array quality relative to the standard error of the estimated feature abundance. In the case of Affymetrix, the primary feature of interest is gene level expression and is estimated by taking a robust average of probe sequences (average of 11 probes per gene on the HGU133plus2 array). Whereas RLE is used to look for overall shifts in the distribution of intensity between arrays, NUSE and GNUSE assess the variability of the estimated feature intensity across arrays and is defined asThe next metric, first introduced by Bolstad ) whereas NUSE re-estimates this for each new experiment. Regardless of which form is used, if the median NUSE or GNUSE for a particular array is high, this would be an indication that many of the features are behaving poorly and thus the array should be considered for removal. A value of 1.25 for the median NUSE or GNUSE has been suggested by McCall as a guideline for identifying bad arrays as this suggests that the variation for the array is 25% higher than an average array.The two measures only differ in that the SE (Y′ij). For the Illumina platform, depending on the gene annotation used, between 45 to 55% of genes have only one probe per gene making calculation of SE (Y′ij) at the gene level meaningless for a high proportion of the array. Each probe on the Illumina array has on average 30 to 40 beads per probe that are used to quantify probe specific expression and various authors have leveraged this aspect of the design in their assessment of differential expression [SE (Y′ij) for the calculation of NUSE. There are two fundamental flaws to this approach. First, as shown in Figure One drawback for NUSE and GNUSE is that they are tailored towards the Affymetrix platform as multiple probes per gene are needed in order to estimate pression ,26. Potelumi package developed by Du et al. for DASL arrays [lumi package is the “distance to the average array”, which we will define asRecently, Chow et al. described their quality assessment workflow using the L arrays ,27. ThisZij represents the feature mean centered and scaled pre- or post-normalized expression data for the ith feature from the jth array and Targeti represents a robust estimate of the feature mean across all arrays and is a correspondingly a pre- or post-normalized estimate (Table th array from the Target or one minus the correlation between the jth array and the Target. The lumi package considers an array as an outlier whenever lumiOutlierj > Th  ×  median (lumiOutlierj) , where Th is a user-specified threshold (default specified in the package is Th = 2). It is difficult to attribute a biologically meaningful interpretation of this metric in such a way as to make it easily transparent to the average researcher. Another drawback is that the threshold is defined relative to the current sample of arrays. Thresholds that are sample dependent are problematic in practice as they vary from batch-to-batch and provide no sense of global quality of an array beyond the average array within the current batch. If, for example, the average array is also of poor quality, the researcher is left with an experiment containing many poor arrays jeopardizing the validity of the study.Where te Table . The disRLE, NUSE/GNUSE, and the lumiOutlier, yet are flexible enough to be implemented on a broad spectrum of microarray platforms with direct biological interpretation. Importantly for the analysis of archival tissues, the proposed metrics allow for the identification of poor arrays that have undue influence during the normalization process. Such arrays are fairly obvious to identify when evaluating data from fresh-frozen samples; however, it is less obvious to determine a threshold for determining poor samples with archival samples.To address the shortcomings of the metrics purposed thus far, we propose two metrics that combine the essence of Sij can be viewed as an estimate for the amount that each feature on an array needs to be shifted for normalization; we refer to this as the Stress measure due to its similarity to the same concept in multidimensional scaling. The overall distribution of Stress captures the amount of deformation that was applied to an array during normalization. Since we are not concerned about arrays that differ from other arrays by a constant shift  the array-specific mean of Sij is subtracted, which leads to our first metric called Stress and is defined asFrom Equation (3), log2 is used here to indicate that the index will need to be transformed to the fold-change scale. Also, by taking the absolute value, features that are up or down regulated by “x-fold” are considered equally Stressed. Various distributional summaries and figures can be generated on Stressj, but we found the median to be the most useful. Arrays can be rank ordered according to their Stress values, and the arrays with the highest or more disparate Stress values would be considered as suspect for inclusion in the study. As an example, if the median Stress of an array is 2, this would indicate that half of the features had to be adjusted by 100% or more relative to their initial values. For many studies, a 2-fold change is the biological effect size of interest. Any final result becomes highly suspect when it is of the same order of magnitude as the biases that were removed from the data.and is calculated across all i features on a specific array. The Stress provides a biologically meaningful measure of how much the global distribution of a sample will change during normalization, it does not leverage feature-specific information. From Equation (3), and under the assumption that a preponderance of features on a microarray are not differentially expressed, an estimate of intensity for the ith feature is simply the mean of that feature across the n arrays th feature by excluding the jth array. We define our next metric dfArray (Deviation oF Array) asAlthough dfArray is similar to the RLE metric proposed by Bolstad, but dfArray is scaled by the standard deviation of the remaining arrays so that features are scaled both within and across arrays as the distribution of each feature will have a mean of zero and a standard deviation of one. Recalculation of Equation (4) for each array is computationally tedious especially for large experiments and a more efficient approach is to replace th feature in place of dfArray to be useful for quality assessment purposes, but as a single summary of this metric we useand is analogous to common diagnostic tests in linear regression modeling known as Cooks distance . The numas values that fall above or below dfArray is highly correlated with the dissimilarity metric used in the lumi package. Since the dissimilarity metric is used in clustering procedures, this indicates that arrays with a large dfArray index may be associated to clinical subclasses not accounted for in the normalization process. Our proposed quality assessment strategy for FFPE samples analyzed is outlined in Table http://mayoresearch.mayo.edu/mayo/research/biostat/splusfunctions.cfm.As we show in the results, log2 transformed intensity values showed that the quality of the data varied dramatically between the samples. Specifically, it was apparent that some of the samples failed completely, while there were other samples for which it appeared that some of the probes worked while other probes did not. Figure As described above, the case study used throughout consists of 1618 HumanHT-12 v4 Expression BeadChip DASL assays that were generated as part of an ongoing breast cancer study that analyzed FFPE archival tissues. Boxplots of the GNUSE or NUSE cannot be applied as the majority of the features on the DASL array do not have multiple probes nor is there a comprehensive archive of the HumanHT-12 v4 Expression Beadchip DASL assay to define a reference distribution. Figure RLE metric on the same set of arrays shown in Figure Stress, dfArray, and lumi Outlier in an attempt to identify poor-quality samples. First, we compared Stress against the IQR  as the reference sample. There were 100 arrays that were considered as outliers by all three methods and 140 arrays that were considered as outliers only by dfArray and Stress, but not lumi Outlier. At each stage, when additional samples are added to the reference sample of 1378 arrays, the data were re-normalized using quantile normalization and the corresponding feature mean and variance was recalculated and compared to the estimates from the reference sample. Figure Stress/dfArray) , Figure Lastly, to understand the impact of questionable arrays, we estimated the bias and variance of feature level expression by considering the 1378 arrays that were of good quality for all three metrics , class prediction (prediction of disease state), and class discovery (identification of new subclasses of disease base on gene expression profiles). Recently, interest has moved from the bench to the bed side where treatment decisions based on gene-expression profiles obtained from microarrays are being considered . In factAs the use of microarrays has increased, so to have the concerns about the validity of this technology -33. SomeAnalysis of archival tissues presents a new challenge and is complicated by poor RNA quality and significant variation among FFPE samples that have been preserved over the course of many years and under different conditions. As we have shown here, this variation in sample quality for FFPE samples creates large variation in the expression profiles across arrays that are typically not seen when dealing with fresh-frozen samples. This has spurred many questions regarding the normalization, quality assessment, and analysis of array based studies using FFPE samples .The choice of normalization routine may have an impact on downstream analyses when it comes to FFPE samples. Many of the FFPE samples in the present study exhibit a high prevalence of “dead probes” where little or no signal is generated beyond background. Many of the more popular normalization routines  used in practice were developed on data where the prevalence of dead probes was very small. Therefore, we believe additional studies are required to determine the best normalization strategy for data that is generated from the FFPE samples.Stress and dfArray, are easily applicable to any microarray platform for this purpose. For studies using FFPE samples, removing arrays that are of poor quality from the normalization process reduces the bias in the estimated feature abundance and the noise level in the data and thus increases the ability to detect biologically-meaningful differences. Some have suggested that the information provided by the quality metrics could also be used to weight downstream analyses towards arrays with better quality [Stress metric as being an outlier have the greatest influence on the normalization process and therefore will need to be excluded. However, the Stress metric could be recomputed after removing outliers and either the newly-computed Stress metric or dfArray could be used to down weight arrays during differential-expression analyses.It is important to note that normalization is not the end all step to preprocessing microarray data and certainly not a solution for poorly-designed studies. Assessing the quality of microarray data is essential and the two metrics proposed here,  quality . This isBiologically Oriented Repository Architecture (BORA), which is an informatics warehouse of “-omics” data that is linked to the tissue pathology and clinical characteristics of the patient. These types of initiatives require robust quality metrics to accurately assess high dimensional data across multiple studies especially when the data has been preprocessed and summarized prior to storage.As more high dimensional data become publicly available, there is an increasing interest to pool data across studies, or at the very least, mine these repositories for promising biomarker signatures prior to initiating a research project. At our institution, such an endeavor is being implemented through the creation of the Two robust quality control metrics are presented that provide the end-users with valuable information regarding the quality of the arrays within their study. These metrics are directly applicable to any high-dimensional platform and can be easily implemented into preprocessing pipelines.Package name: Stress.dfArrayhttp://mayoresearch.mayo.edu/mayo/research/biostat/splusfunctions.cfmPackage source: http://www.r-project.org/)Requirements: R-2.14.0 or later (RLE: Relative Log Expression; NUSE: Normalized Unscaled Standard Error; GNUSE: Global Normalized Unscaled Standard Error; dfArray: Deviation of array; DASL: cDNA-mediated Annealing Selection extension and Ligation; Th: Threshold; FFPE: Formalin-Fixed, Parriffin-Embedded; Q1,Q3: First and third quartiles; IQR: Interquartile range; SE: Standard Error; sd: Standard deviation.The authors declare that they have no competing interests.DWM, TMT, SKA and JE-P developed the statistical methodology. JJ, JPAK, MMR and EAP provided guidance on the statistical methodologies. MMR and EAP provided the case study. All authors read and approved the final manuscript.


First-generation molecular profiles for human breast cancers have enabled the identification of features that can predict therapeutic response; however, little is known about how the various data types can best be combined to yield optimal predictors. Collections of breast cancer cell lines mirror many aspects of breast cancer molecular pathobiology, and measurements of their omic and biological therapeutic responses are well-suited for development of strategies to identify the most predictive molecular feature sets.We used least squares-support vector machines and random forest algorithms to identify molecular features associated with responses of a collection of 70 breast cancer cell lines to 90 experimental or approved therapeutic agents. The datasets analyzed included measurements of copy number aberrations, mutations, gene and isoform expression, promoter methylation and protein expression. Transcriptional subtype contributed strongly to response predictors for 25% of compounds, and adding other molecular data types improved prediction for 65%. No single molecular dataset consistently out-performed the others, suggesting that therapeutic response is mediated at multiple levels in the genome. Response predictors were developed and applied to TCGA data, and were found to be present in subsets of those patient samples.These results suggest that matching patients to treatments based on transcriptional subtype will improve response rates, and inclusion of additional features from other profiling data types may provide additional benefit. Further, we suggest a systems biology strategy for guiding clinical trials so that patient cohorts most likely to respond to new therapies may be more efficiently identified. Breast cancer is a clinically and genomically heterogeneous disease. Six subtypes were defined approximately a decade ago based on transcriptional characteristics and were designated luminal A, luminal B, ERBB2-enriched, basal-like, claudin-low and normal-like ,2. New cin vitro cell line systems provide the only experimental data that can be used to identify predictive response signatures, as most of the compounds have not been tested in clinical trials. Our study focuses on breast cancer [In breast cancer, cell lines mirror many of the molecular characteristics of the tumors from which they were derived, and are therefore a useful preclinical model in which to explore strategies for predictive marker development ,9. To tht cancer ,11 and et cancer -14, by i50) was used as the response measure for each compound. Compounds with low variation in response in the cell line panel were eliminated, leaving a response data set of 90 compounds. An overview of the 70 cell lines with subtype information and 90 therapeutic compounds with GI50 values is provided in Additional file We assembled a collection of 84 breast cancer cell lines composed of 35 luminal, 27 basal, 10 claudin-low, 7 normal-like, 2 matched normal cell lines, and 3 of unknown subtype , mRNA expression (Affymetrix U133A and Exon 1.0 ST array - ArrayExpress accessions E-TABM-157 and E-MTAB-181), transcriptome sequence (RNAseq - Gene Expression Omnibus (GEO) accession GSE48216), promoter methylation (Illumina Methylation27 BeadChip - GEO accession GSE42944), protein abundance  ranged from 0.6 to 0.77 at the cell line level, and from 0.58 to 0.71 at the gene level. Promoter methylation and gene expression were, on average, negatively correlated as expected, with correlation ranging from -0.16 to -0.25 at the cell line level and -0.10 to -0.15 at the gene level. Across the genome, copy number and gene expression were positively correlated . When restricted to copy number aberrations, 22 to 39% of genes in the aberrant regions showed a significant concordance between their genomic and transcriptomic profiles from U133A, exon array and RNAseq after multiple testing correction  of these compounds. A comparison across all 90 compounds of the LS-SVM and RF models with highest AUC based on copy number, methylation, transcription and/or proteomic features revealed a high correlation between both classification methods , with the LS-SVM more predictive for 35 compounds and RF for 55 compounds .Thirteen of the 51 compounds 25%) showed a strong transcriptional subtype-specific response (AUC >0.70), with the best omics signature not adding predictive information beyond a simple transcriptional subtype-based prediction (AUC increase below 0.1) , copy number, and promoter methylation for 51 compounds for which predictive power was high . In addition, exon array outperformed U133A, with a P-value of 0.0002.With seven data types available on a single set of samples, we were well-positioned to assess whether particular technologies or molecular data types consistently out-perform others in the prediction of drug sensitivity. To obtain a ranking of the importance of the molecular datasets, we compared prediction performance of classifiers built on individual data sets and their combination for 29 common cell lines . Importantly, no single data type performed well for all compounds, with each data type performing best for some compounds . The majority of these significant pathways, however, were also associated with transcriptional subtype . These were filtered out to capture subtype-independent biology underlying each compound’s mechanism of action. The resulting non-subtype-specific pathways with FDR P-value <0.05 are shown in Additional file ERRFI1[DUSP6/7/8[SPRY1/2/4[GADD45A was associated with resistance to GSK2126458, PF-4691502 and the AKT1/2 inhibitor, which is consistent with the observation that AKT inhibition modulates cell growth via activation of GADD45A[We surveyed the pathways and biological processes represented by genes for the 49 best-performing therapeutic response signatures incorporating copy number, methylation, transcription, and/or proteomic features  with AUC >0.7  and BIBW2992 (1.6e-5) and to EGFR inhibitors AG1478 (2.5e-4) and gefitinib (9.5e-4). In addition to the association of overall mutation status with tamoxifen and CGC-11144 response discussed above, we also found several individual mutations to be relevant for treatment response. The presence of mutations in TP53 was strongly associated with response to the PI3K inhibitor BEZ235, with 13/25 (52%) of the sensitive cell lines harboring TP53 mutations compared to 3/19 (16%) for the resistant cell lines . This may be an indication of synthetic lethality resulting from BEZ235 inhibition of ATR (Ataxia telangiectasia and Rad3-related protein) leading to replicative stress in TP53-deficient cells [et al. [TP53 and PIK3CA was positively associated with response to BEZ235. In our study, mutation status for PIK3CA was associated with response to the PI3K inhibitor GSK1059615B, with 11/27 (41%) sensitive cell lines carrying PIK3CA mutations compared to 2/21 (10%) for resistant cell lines . These findings are consistent with recent clinical observations in patients with breast and gynecologic malignancies where treatment with similar agents resulted in response for 30% of patients with PIK3CA mutations compared to a response rate of 10% in wild-type PIK3CA patients [Genomic aberrations and transcriptomic/proteomic features played prominent roles in some of the candidate response signatures. For copy number aberrations, ERBB2 amplification was strongly associated with response to the ERBB2 targeting compounds lapatinib  although gefitinib and NU6102 were exceptions  , hosted 5-FU: fluorouracil; AUC: area under the receiver operating characteristic curve; EGFR: epidermal growth factor receptor; ER: estrogen receptor; FDR: false discovery rate; GEO: Gene Expression Omnibus; GI50: concentration at which growth is inhibited by 50%; GO: Gene Ontology; KEGG: Kyoto Encyclopedia of Genes and Genomes; LS-SVM: least squares support vector machine; MAPK: mitogen-activated protein kinase; PAM: Prediction Analysis for Microarrays; PI3K: phosphatidylinositol 3-kinase; RF: random forests; RPPA: Reverse Phase Protein Array; SNP: single nucleotide polymorphism; TCGA: The Cancer Genome Atlas; TGI: total growth inhibition.The authors declare that they have no competing interests.AD was partly supported by a BAEF Fellowship of the Belgian American Educational Foundation for postdoctoral research, OLG was supported by a Fellowship from the Canadian Institutes of Health Research, EAC is supported by NCI 5K08CA137153-02.Supplementary Methods, Supplementary Results, Figures S1 to S10, and Tables S4, S6, S8, S9, S10, S12, and S13. Supplementary Methods: detailed description of the therapeutic compound response data, molecular data for the breast cancer cell lines, molecular data for the external breast cancer tumor samples used for validation, classification methods, data integration approach, statistical methods, and pathway overrepresentation analysis. Supplementary Results: assessment of cell line signal in tumor samples, inter-data relationships, prediction comparison of datasets, validation against other cell line datasets, and the patient response prediction toolbox for the R project for statistical computing. Table S4: overview of genes with good correlation  between SNP6 and gene expression; 22 to 39% of genes in copy number aberration regions show a significant concordance between their genomic and transcriptomic profile after multiple testing correction. Table S6: data type ranking of the importance of the molecular datasets by comparison of prediction performance of LS-SVM and RF classifiers built on individual data sets and their combination, and by comparison of the average appearance of data types in the top 100 of ranked features, with and without inclusion of RPPA data. Examples are also provided of compounds for which (most) datasets give similar results or for which one dataset performs better (shown in bold). Table S8: performance for 'splice-specific' response predictors (RF) with an AUC increase >0.05 when comparing all transcript features to gene-level values alone. Table S9: statistical association between clinical variables and predicted response for 306 TCGA patients with expression, methylation and copy number data available. For each compound, the best performing model was utilized . Table S10: resistant/intermediate/sensitive cutoffs for 22 compounds with model AUC >0.7 and at least one patient with probability of response >0.65. Cutoff value 1 separates patients considered resistant from intermediate. Cutoff value 2 separates patients considered intermediate from sensitive. The percentage value for each group indicates the percentage of total patients (n = 306) in each group. Table S12: presence and variance of filtered features from U133A and exon array cell line data in tumor samples. Features from U133A and the exon array that passed the variance and presence filter in the cell lines were present in the majority of breast cancer tumor samples. Table S13: summary of 167 predictors in random forests classifier for lapatinib . Figure S1: data summary in terms of number of features before and after data-type-specific reduction and unsupervised filtering based on variance and signal detection above background. Figure S2: overview of the mutation prevalence in the cell line panel and TCGA data set for the list of seven common coding variants detected by TCGA, with a distinction between luminal, basal and ERBB2-enriched. Cell lines with unknown subtype are displayed in orange. To make the subtypes comparable, luminal A and B were grouped into luminal for the TCGA data set, whilst basal and claudin-low cell lines were grouped into basal. The mutation rate in TCGA and the cell line panel shows a similar distribution across the subtypes. Figure S3: comparison of the best LS-SVM and RF models for the 90 compounds, sorted according to highest AUC obtained with either model. Figure S4: validation of the cell line signature for vorinostat in tumor samples grown in three dimensions: heatmap of the 150-gene signature for vorinostat in the cell line panel and 13 tumor samples treated with valproic acid. Seven out of eight sensitive samples (87.5%) and four out of five resistant samples (80%) are classified correctly with a probability threshold of 0.5 for response dichotomization. Figure S5: predicted probability of response of TCGA tumor samples to compounds lapatinib, sigma AKT1-2 inhibitor, GSK2126458 and docetaxel. The TCGA tumor samples are ordered according to increasing probability of response. Figure S6: correlation-based coherence heatmap for two cell line-derived gene signatures: coherence among 67 genes of the U133A signature for the sigma AKT1-2 inhibitor in the cell lines (left) and TCGA tumor samples (right) ; coherence among 109 genes of the RNAseq signature for everolimus in the cell lines (left) and TCGA tumor samples (right) . Figure S7: comparison of the best model per dataset for the 90 compounds, sorted according to highest AUC obtained with either model (LS-SVM or RF). For RNAseq and exon array, the highest AUC is shown among models built on gene-level data only or all features . Figure S8: distributions of response probabilities for 5-FU determined by mixed model clustering and used for cutoff selection. With a cutoff of 0.74, 23.9% of TCGA tumor samples were predicted to respond to 5-FU  intensity data for 70 (phospho)proteins with fully validated antibodies in 49 cell lines. See Supplementary Methods in Additional file Click here for file50 dichotomization threshold for each compound, defined as the mean GI50 for the 48 core cell lines.GIClick here for file50 with TGI, validation results of the cell line signal in the TCGA tumor samples, and most significant non-subtype related KEGG/BioCarta pathways from Additional file Overview of the best LS-SVM/RF model for all 90 therapeutic compounds with comparison to the LS-SVM AUC based on subtype and ERBB2 status. For the subset of 51 therapeutic compounds with test AUC exceeding 0.7, additional information is provided on clinical trial status, comparison of GIClick here for fileP-value <0.05. Per category/pathway information includes FDR P-value and the number of signature genes, percentage of signature genes and list of signature genes that are part of this category/pathway. Significant pathways associated with both drug response and transcriptional subtype were excluded, to capture biology underlying each compound’s mechanism of action.List of significant non-subtype specific GO categories and KEGG/BioCarta pathways with FDR Click here for fileCompound response signatures for the 22 compounds featured in Figure Click here for fileValidation results for six drugs  in 11 HER2+ lines.Click here for fileRaw drug response data. Raw drug response data used to compute GI50 values used in this study. The columns represent the following: cellline = cell line lineage; compound = compound tested; drug_plate_id = unique identifier for the plate of 3 compounds; T0_plate_id = unique identifier for the time 0 h control plate associated with the drug plate; background_od1, background_od2 = background od values (for correction of background luminesence); od0.1, od0.2, od0.3 = triplicate measures for untreated cells; od1.1, od1.2, od1.3… od9.1, od9.2, od9.3 = triplicate measures of number of cells alive after treatment with lowest to highest drug; T0_background_od1, T0_background_od2 = background od values (for correction of background luminesence); T0_median_od = median od at T0; c1 to c9 = drug concentrations tested; units = units of drug concentration tested.Click here for file


DNA methylation plays an important role in many biological processes by regulating gene expression. It is commonly accepted that turning on the DNA methylation leads to silencing of the expression of the corresponding genes. While methylation is often described as a binary on-off signal, it is typically measured using beta values derived from either microarray or sequencing technologies, which takes continuous values between 0 and 1. If we would like to interpret methylation in a binary fashion, appropriate thresholds are needed to dichotomize the continuous measurements. In this paper, we use data from The Cancer Genome Atlas project. For a total of 992 samples across five cancer types, both methylation and gene expression data are available. A bivariate extension of the StepMiner algorithm is used to identify thresholds for dichotomizing both methylation and expression data. Hypergeometric test is applied to identify CpG sites whose methylation status is significantly associated to silencing of the expression of their corresponding genes. The test is performed on either all five cancer types together or individual cancer types separately. We notice that the appropriate thresholds vary across different CpG sites. In addition, the negative association between methylation and expression is highly tissue specific. DNA methylation plays an important role in cancer through hypermethylation to turn off tumor suppressors and hypomethylation to activate oncogenes ,2. It isWhen the relationship between methylation and gene expression is discussed, both are often described as binary signals  . For exaTo jointly analyze methylation and gene expression, an ideal dataset would be a large collection of samples for which both data types are available. The Cancer Genome Atlas (TCGA) project provides such data for a large number of cancer samples -11. MoreIn this paper, we downloaded DNA methylation and gene expression data in TCGA. Data for a total of 992 samples were available, covering five cancer types. We extended the StepMiner algorithm  to identhttps://tcga-data.nci.nih.gov/tcga/tcgaDownload.jsp) provides three ways for accessing the data. Two of them, ‘data matrix’ and ‘bulk download,’ require investigators to manually select a subset of the data and then automatically collect relevant data files into a compressed.tar file for download. After that, additional effort is needed to parse and assemble the downloaded files into formats useful to programming environments such as Matlab or R. Since TCGA data keep growing and the manual selection can be tedious when multiple data types and disease types are considered, it is difficult to keep track of the manual selections and guarantee reproducibility. Therefore, we chose the third way, ‘open-access http directory,’ which contains links for all individual data files in TCGA (http://tcga-data.nci.nih.gov/tcgafiles/ftp\_auth/distro\_ftpusers/anonymous/tumor/). We created Matlab scripts to programmatically grab methylation and RNA-seq data files for each individual disease type, automatically parse them, and organize them into tab delimited spreadsheets for subsequent analysis. Our scripts for automatically downloading TCGA data are available at http://odin.mdacc.tmc.edu/~pqiu/software/DownloadTCGA/.TCGA data portal (Mi/(Ui+Mi) for each CpG site i. Mi represents the signal intensity of the methylated probe for CpG site i, and Ui is the signal intensity of the unmethylated probe. Therefore, the numerical range of the data is between 0 and 1. Zero (0) indicates unmethylated, whereas 1 indicates completely methylated. The data contain a small fraction of empty entries, because the corresponding probes either overlap with known single-nucleotide polymorphisms or other genomic variations, or their signal intensities are lower than the background.Genome-wide methylation measurements were generated using the Illumina Infinium Human DNA Methylation27 array platform , which interrogates the methylation status of 27,578 CpG sites in proximal promoter regions of 14,475 genes in the human genome. As of 12 February 2013, methylation data for 2,796 samples across 12 cancer types were available. We downloaded the TCGA level 3 preprocessed data, which are the ratio 5. For each gene, we replaced the zero entries with the minimal non-zero value of this gene across all samples and transformed the data to log scale.TCGA uses several platforms to quantify gene expression, among which the Illumina GA II and HiSeq platforms profiled the largest number of samples. As of 12 February 2013, preprocessed RNA-seq data for 4,108 samples across 11 cancer types were available. The preprocessed data are the RPKM values for 20,532 genes in each sample. Roughly, the numerical range of the data is between 0 and 10The total number of overlapping samples between the above methylation and expression data was 992. The overlap covered five different cancer types: breast cancer , colon and rectal cancer , kidney renal clear cell carcinoma , squamous cell lung cancer , and uterine corpus endometrioid carcinoma . Our analysis was performed based on these 992 overlapping samples.X with an unknown probability distribution and n independent observations of the random variable xk,k=1,2,...,n)≤x(2)≤...≤xn)=μ1I(i≤t)+μ2I(i>t), where i=1,2,...,n and I(.) is an indicator function. Denote the mean of all observations as μ, the deviation of the fitted step function to sample mean as signal t<n. For each possible value of t, calculating the ratio is straight forward because μ1 is the mean of the first t observations in the sorted data and μ2 is the mean of the remaining observations. Once the t∗ that maximizes SNR is identified, the threshold for dichotomizing the observation can be defined as t∗) depends on the distribution of X. If X has an extreme bi-modal distribution and its probability density function is a sum of two delta functions, SNR (t∗) approaches positive infinity. If X follows a uniform distribution, SNR (t∗) equals to 3. If X follows a Gaussian distribution, SNR (t∗) is approximately 1.75, regardless of the values of mean and variance of the Gaussian. Two examples using real data are shown in Figure StepMiner was originally developed to extract binary patterns in microarray gene expression data  and studble xk,k=,2,...,n,x-coordinate is the rank of this observation, and y-coordinate is the observed value itself. The collection of all observations forms a non-decreasing curve which is fitted by a step function. In order to extend the algorithm to bivariate, we define a non-decreasing surface and fit it with a bivariate step function. Given n observations of two random variable X and Y, ,k=1,2,...,n) to a point in a three-dimensional (3D) space. The x-coordinate of the point is the rank of xk with respect to all the observations for X; the y-coordinate of the point is the rank of yk with respect to all the observations for Y; and the z-coordinate of the point is xk+yk. The collection of all points forms a surface, which is fitted by a bivariate step function with six parameters,As shown above, StepMiner is a univariate algorithm, which determines a threshold for each feature by its marginal distribution. Since we are interested in the relationship between the expression of a gene and its methylation, one natural idea is to extend the algorithm to a bivariate analysis and jointly consider two variables, which we call StepMiner2D. As shown in Figure i and j both range from 1 to n.where xk is the ith smallest among all observed value for X, and yk is the jth smallest among all observed value for Y, the data point  is mapped to point . The rank transformed data are shown in Figure z=xk+yk is defined at point  to which the kth observation is mapped. For an  point to which no observation is mapped, we define X and Y contribute equally, the observations are normalized to zero-mean-unit-variance before defining z. Finally, denote z and optimizing SNR on a n×n grid can be time consuming when n is large. For computational efficiency, we approximate the surface on a 50×50 grid. In Figure z is shown as a heatmap, where blue indicates small value and red indicates large value. Figure xk,yk,zk) and the best fit bivariate step function. The optimal SNR for this example is 5.17. Similar to the 1D case, the maximum value of SNR depends on the joint distribution of X and Y. If the joint probability density function is a sum of two or three delta functions, the optimal SNR approaches positive infinity. If X and Y are independent, the optimal SNR is 3 for uniform distribution and approximately 1.75 for Gaussian distribution.To illustrate how to compute StepMiner2D, one example is shown in Figure X and Y, rather than the association between the two variables. For example, if X and Y independently follow two bi-modal distributions, although there is no association between the two variables, the optimal SNR can be large. Thus, SNR does not seem to be suitable for evaluating the association between methylation and expression. Here, we are interested in one particular kind of association, whether methylation of a CpG site leads to down-regulation of its corresponding gene expression. After dichotomizing methylation and expression data, the sufficient statistics become counts of points in the four quadrants in Figure N denote the total number of samples; R is the total number of methylated samples (sum of points in the upper-right quadrant and the lower-right quadrant); U is the total number of samples with high gene expression . Condition on N, R, and U, if the methylation and expression are independent, the number of samples in the upper-right quadrant k follows a hypergeometric distribution K, we can compute the probability of observing K or less points under the assumption that methylation and expression are independent p value The optimal SNR value in StepMiner2D measures the multi-modality of the joint distribution of http://odin.mdacc.tmc.edu/~pqiu/projects/MethExpr/.We preprocessed the TCGA data by filtering out CpG sites with small variance or many missing data points and matching methylation and expression data according to genes. The methylation data we downloaded from TCGA were generated by the Methylation27 array platform, which provided the methylation status of 27,578 CpG sites in 14,475 genes across 992 cancer samples. We excluded CpG sites whose annotated genes are not present in the expression data. We also excluded CpG sites with more than 1% missing data and ones whose methylation beta value is smaller than 0.01 for more than 95% of the samples. After applying these filtering criteria, we obtained a total of 11,189 CpG sites annotated to 7,344 unique genes. For approximately half of the genes, only one CpG site is measured for each gene; data for two CpG sites are available for the majority of the other half; for a very small number of genes, measurements of multiple CpG sites are available. In the subsequent subsections, for the methylation data of each of the 11,189 CpG sites, we extracted the expression data of its corresponding gene and focused our bivariate analysis on features paired according to genes. Preprocessed data and the code for our analysis is available at p value threshold of 0.01 and Bonferroni correction, 2,976 pairs showed significant association, and the ESR1 example in Figures For each of the 11,189 methylation-expression pairs, we applied StepMiner2D to examine the data for all 992 samples together. Using such a pan-cancer analysis strategy, we identified thresholds to dichotomize the data. We performed hypergeometric test to examine whether methylation was significantly associated to the down-regulation of expression of its corresponding gene. We filtered out cases where the number of samples in the upper-right quadrant minus that in the lower-left quadrant was more than 10% of the total number of samples, which obviously did not support the concept of methylation turning off the expression. Using a The association between methylation and expression of ESR1 was significant when all 992 samples were examined together. However, when we examined individual cancer types separately, the methylation-expression association for ESR1 became insignificant. In Figure We evaluated the association between methylation and expression using samples in individual cancer types separately. Figure We performed integrative analysis of methylation and gene expression data of five cancer types in TCGA. First, we pooled samples from all five cancer types together and applied StepMiner2D to identify thresholds for dichotomizing the methylation and expression data. In such a pan-cancer analysis strategy, the diversity and variation among samples allow us to observe positive and negative signals in sufficient number of samples and empower the method to identify the appropriate thresholds. Then, we applied hypergeometric test to identify CpG sites whose methylation is significantly associated to silencing of the expression of their corresponding genes, either using all five cancer types together or using individual cancer types separately. When all five cancer types were examined together, 2,976 CpG sites showed significant negative association with gene expression. However, when samples in different cancer types were considered separately, a much smaller number of significant associations were observed in at least one cancer type. We speculate that the associations only significant in pan-cancer analysis are likely to be induced by tissue differences, whereas significant associations observed in individual cancer types are more likely to reflect regulatory relationships between methylation and gene expression. For future work, there are a few possible extensions. The methylation data used here are generated by the Illumina Methylation 27k platform. TCGA also generates methylation data using the Illumina Methylation 450k platform, which measures roughly 20 times more CpG sites. We plan to redo the analysis using the 450k methylation data, which will enable us to identify more associations between methylation and expression. Moreover, the proposed analysis strategy can also be applied to examine associations among measurements made by other modalities, such as microRNA expression, DNA copy number variation, protein expression, etc.The authors declare that they have no competing interests.


High throughput sequencing technology provides us unprecedented opportunities to study transcriptome dynamics. Compared to microarray-based gene expression profiling, RNA-Seq has many advantages, such as high resolution, low background, and ability to identify novel transcripts. Moreover, for genes with multiple isoforms, expression of each isoform may be estimated from RNA-Seq data. Despite these advantages, recent work revealed that base level read counts from RNA-Seq data may not be randomly distributed and can be affected by local nucleotide composition. It was not clear though how the base level read count bias may affect gene level expression estimates.In this paper, by using five published RNA-Seq data sets from different biological sources and with different data preprocessing schemes, we showed that commonly used estimates of gene expression levels from RNA-Seq data, such as reads per kilobase of gene length per million reads (RPKM), are biased in terms of gene length, GC content and dinucleotide frequencies. We directly examined the biases at the gene-level, and proposed a simple generalized-additive-model based approach to correct different sources of biases simultaneously. Compared to previously proposed base level correction methods, our method reduces bias in gene-level expression estimates more effectively.Our method identifies and corrects different sources of biases in gene-level expression measures from RNA-Seq data, and provides more accurate estimates of gene expression levels from RNA-Seq. This method should prove useful in meta-analysis of gene expression levels using different platforms or experimental protocols. Massive parallel sequencing of RNA has provided researchers a powerful tool for transcriptome analysis. The protocols of gene expression analysis using the sequencing approach fall into two major categories. The traditional approach is 3' tag digital gene expression (DGE), in which oligo-dT was used for synthesis of cDNA libraries, resulting in enrichment in the 3' end of polyadenylated mRNAs. These polyadenylated fragments were further digested by certain restriction endonucleases to produce short cDNA tags. The short tags were then sequenced by massive parallel sequencing technologies to give "digital counts" of the mRNA molecules originated from each gene. A variety of DGE approaches differ in their choices of restriction enonuclease, cDNA cloning step, and usage of barcodes ,2. TheseCompared to microarray-based gene expression analysis, DGE and RNA-Seq have essentially no background, and therefore overcome the ratio compression problem  persists in microarray technology; the dynamic range of sequencing based approaches also expands with higher sequencing depth . MoreoveDespite these advantages, a comprehensive understanding of the characteristics and potential artefacts in the data is needed to most appropriately analyze RNA-Seq data. In an important study of RNA-Seq data characteristics, Oshlack et al. found transcript length biases in RNA-Seq data . SpecifiIn addition to length biases, other factors such as read mapping uncertainty and sequence base composition may also confound results from RNA-Seq experiments. Read mapping uncertainty across different isoforms  and diffIn this article, based on multiple publicly available RNA-Seq data sets, we showed that the commonly used measures of gene expression levels, such as raw read counts or RPKM/FPKM, are biased in transcript length, GC content, and dinucleotide frequencies. We also investigated the effects of biological sources, experimental protocols and data processing methods on these bias patterns systematically. We found that the bias patterns are specific to experimental protocols but not specific to biological sources, suggesting that they are technical artefact rather than biologically relevant patterns. It motivated us to develop a generalized additive model based algorithm to jointly correct for these biases. In contrast to  and 15]15] whereMicroArray quality control (MAQC) data sets ,23 contaThe two samples are labelled MAQC2 and MAQC3. MAQC2 contains 7 technical replicates of brain reference RNA samples and 7 technical replicates of UHR RNA samples sequenced on 14 lanes of two flow cells. MAQC3 contains UHR RNA samples from 4 different library preparations sequenced on 14 lanes of two flow cells.http://www.ncbi.nlm.nih.gov/sra/ with accession numbers SRX016366, SRX016368-SRX016372. Taqman quantitative RT-PCR technology first reverse transcribes RNA transcripts into cDNA, and then measures concentration of each cDNA template at each cycle of PCR by detecting the fluorescent signals released from a target-specific fluorogenic hybridization probe that is hydrolyzed by Taq polymerase during the extension phase. It is often used as gold standard for microarray studies. Taqman qRT-PCR data for 1,044 probes, corresponding to 1,001 RefSeq genes, were obtained from GEO database with accession numbers GSM129638-GSM129645, which represent 4 technical replicates of Taqman qRT-PCR for each of the two samples. Reported values are normalized expression values as All experiments used standard Illumina RNA-Seq protocol. The raw sequence files were downloaded from the Sequence Read Archive http://bioinf.wehi.edu.au/resources/  [This dataset was generated in  and usedurces/  .This dataset was described in  and alsoThis dataset was described in  through ftp://ftp.sanger.ac.uk/pub/transseq/.This dataset was generated in  for humaAll the data sources, accession numbers, characteristics of the techniques and related information are listed in Table Three procedures were performed in processing sequence data to obtain estimates of expression levels. For Procedure 1, we used TopHat (v1.1.0)  to aligni Ybe the inferred gene expression level for gene i (from one of the three procedures discussed above), and X be a matrix with the i-th row i X≡  representing the values of 18 potential bias factors for gene i. Specifically, Xi, 1 is the gene length on log scale, and  are bias factors related to nucleotide composition. Our goal is to obtain a less biased estimate of expression level X, denoted as X(-1), and obtain the first K principal components Pi, 1, Pi, 2,..., Pi, K that can explain more than 95% variation in X(-1). Then we used these principal components to fit the following generalized additive model (GAM): log(Yi) = αi + Xi, 1 + s(Pi1) + s(Pi2) +⋯+ s(PiK) + εi. Finally we obtain the bias corrected expression as the sum of the grand mean of Y and the residual from fitted model, i.e. We consider the bias of gene expression levels in terms of gene length, gene GC content, and the frequencies of the 16 possible dinucleotides. Let In mammalian genomes, one gene may contain multiple transcript isoforms, and sequencing reads can fall on a genomic region corresponding to different isoforms or even genes. What is more, there is no consensus on "gene" unit to summarize gene expression. For example, Oshlack et al. defined gene length as the median of all transcript length related to that gene, which is a simple high level summary but may not be accurate when different isoforms have very different expression levels . Hansen http://bioinformatics.med.yale.edu/group/. This GAM-based correction method is computationally efficient. It takes ~4-5 seconds on a 2.67 GHz Xeon CPU to correct ~30,000 gene expression levels for human genome with ENS 59 annotation.The GAM correction method and facilitating functions were implemented in an R package (RNASeqBias) and this package can be obtained as Additional file To detect potential biases in the gene expression levels, we grouped genes into bins according to each of the potential bias factors, e.g. gene length, GC content, and dinucleotide frequencies, and plotted the median expression levels versus median bias factors for the five data sets in Figure + mRNAs transcribed from protein coding genes, investigation using only SINO genes may lose considerable amount of information, and patterns learned from SINO genes may not be applicable to general protein coding genes.Many data processing tools have been developed for RNA-Seq data analysis in the literature. To evaluate whether the bias pattern we observed in RNA-Seq data is due to certain data processing methods involved in the processing of these data, we implemented multiple processing procedures for each data set, and summarized expression levels using different "gene" unit definitions (see Methods section). Procedure 3 differs from Procedure 1 in that reads will be double counted instead of being assigned to different isoforms probabilistically for overlapping transcripts or genes. Even for single isoform non-overlapping (SINO) genes, Procedure 1 may be different from Procedure 3 due to two features of Cufflinks. First, when gene model annotation (.gtf) file is provided, Cufflinks only counts fragments that fall on gene model features for the "total number of fragments mapped". Second, Cufflinks uses each fragment's map quality to down-weight reads that were mapped to multiple genomic locations. By comparing bias patterns from these different procedures, we examined whether different mapping policies, gene model annotations and expression estimation methods have any effect on the observed bias patterns. We found that all three data processing methods and both gene level and transcript level expression estimates yielded similar patterns  tended to have similar patterns although different biological samples or library preparation protocols were used, which indicates experiment-specific biases. In the MAQC data set, both base-level and gene-level patterns indicate strong GC content related bias. However, they do not always agree well, and probably capture different characteristics of the data. For example, there are strong base-level biases for reads starting from CT and GT, but this is not reflected in the gene-level bias patterns.To correct for the observed gene-level biases, we calculated principal components of dinucleotide compositions and gene length, and fitted a generalized additive model of log(FPKM) on the eigenvectors (see Methods section). As shown in the bias plots Figure , 5 and 6We compared the expression estimates from RNA-Seq data with Taqman RT-PCR and QuantiGene platforms, which measure gene expression levels using different technologies. The purpose is two-fold. First, by comparing the patterns of expression levels across genes from different platforms, we could distinguish technical bias from biological patterns. Technical bias should be specific to certain platforms, whereas biological patterns should be consistent across platforms. Second, by comparing the estimated expression levels for each gene from different platforms, we could evaluate whether the GAM correction is able to reduce technical biases. If the corrected expression estimates from RNA-Seq data are less biased, we would expect it to correlate better with other quantitative platforms.We compared the patterns of gene expression levels in terms of gene length, GC content and dinucleotide frequencies for 571 and 606 genes measured by both Taqman RT-PCR and RNA-Seq platforms and expressed above the detection thresholds in brain and UHR samples Figure . SimilarTo examine the relationship between expression measurements from different platforms, we performed a series of pair-wise comparisons through Pearson's correlations. Multiple technical replicates were averaged for each platform, and the units for expression levels were ΔCT for RT-PCR, log for QuantiGene, and log(FPKM) for RNA-Seq. We first considered 182 and 200 genes that were measured by both Taqman RT-PCR and QuantiGene assays and expressed above the detection thresholds in brain and UHR samples respectively. The correlation was 0.810 for brain sample, and 0.815 for UHR sample. Then we compared genes present on both Taqman RT-PCR and RNA-Seq platforms. The correlation coefficients (r) were 0.690 (brain) and 0.748 (UHR) before GAM correction, and increased to 0.754 (brain) and 0.763 (UHR) after GAM correction. Furthermore, when we stratified genes into three quartiles by the absolute value of log fold change between corrected and uncorrected expression levels, the relative improvement in correlation increased as correction magnitude increased  from the fitted generalized additive models for the same 8 examples as above . Similar to the random hexamer bias correction method, the mseq-corrected gene expression levels were very close to uncorrected expression levels, and yielded the same bias patterns across genes  employed parametric or non-parametric regression techniques to relate base-level read counts to local nucleotide composition. While achieving better uniformity in read distribution within genes, these methods only corrected for gene level expression estimation to a limited extent, as shown in Additional files 2 relative enrichment of exon level read counts in each GC bin against the mean GC content for the bin. The binning and curve fitting strategy is similar to our method but it is simply applied to one bias factor and does not involve principal component regression. Moreover, in contrast to their usage of the log enrichment of read counts in multiple lanes as response variables, our method is able to correct for the absolute gene expression levels even when there is no technical replicates on multiple lanes.During the review of our manuscript, a related correction method for RNA-Seq data was published . This meIt is well known that due to the random sampling nature of RNA-Seq data, length normalized gene level expressions exhibit larger sample variance for shorter genes . Since oIn summary, our work identified several potential biases in RNA-Seq data, and proposed a bias correction method that can provide a more accurate gene level expression estimate. We believe this method will help meta-analysis of RNA-Seq data with other quantitative platforms and alternative protocols.The authors declare that they have no competing interests.WZ performed statistical analysis, wrote the RNASeqBias package and drafted the manuscript. LC participated in the data analysis and R package development. HZ conceived of the study, designed and coordinated the study and helped to draft the manuscript. All authors read and approved the final manuscript.Summary of data processing procedures for each data set.Click here for fileR package RNASeqBias developed in this study.Click here for fileBias plots for Marioni, Mamanova, Lee and Nagalakshmi data sets .Click here for fileBias plots for MAQC data using transcript level expression summaries.Click here for fileBias plots for Marioni data using Procedures 2 and 3.Click here for fileBias plots for SINO genes in MAQC data.Click here for fileP-values of testing whether bias patterns are platform-specific or sample-specific.Click here for filePCA plots for 8 data sets. The left panel shows the variance explained by each principal component. Black bars are PCs included in the GAM model (they explain at least 95% variance in the data), and gray bars are PCs not included. The right panel shows the biplot for the first two PCs. The length of each red vector represents the standard deviation of each predictor, and the angle between any two vectors represents the correlation between them. The relative orientation to x and y axis shows the relationship between individual predictors and the first two PCs. The inset table shows the non-parametric F-statistic for each predictor in the fitted GAM model and the corresponding p-value.Click here for fileImprovement of cross-platform correlations with different number of PCs in GAM. GAM models using gene length only or length plus 1-7 PCs were fitted to correct the MAQC2 brain and UHR RNA-Seq data set, and the correlations (r) between uncorrected/corrected seq data and RT-PCR/QuantiGene data were calculated.Click here for fileScatter plots with different GAM terms. Correlations between corrected log(RPKM) using (length + GC content) and corrected log(RPKM) using 1, 2, 3, or minimal number of PCs explaining 95% of the variance. Subfigures A-H are plots for 8 data sets in the same order as in Additional file Click here for filePlots of the component smooth functions that make up the fitted GAM objects with two predictors: gene length and GC content. The left panel shows the 1-D smooth for log gene length, and the right panel shows the 1-D smooth for GC content. The rug plot at the bottom of each panel shows the values of each predictor. The dotted lines are 2-standard errors above and below the estimated smooth (solid lines). Subfigures A-H are plots for 8 data sets in the same order as in Additional file Click here for fileBias plots for MAQC data using Procedure 3 and comparing with random hexamer bias correction method.Click here for fileBias plots for SINO genes in Lee data using Procedure 3 and comparing with random hexamer bias correction method and mseq method.Click here for fileComparison of correlations between RT-PCR and MAQC2 RNA-Seq data. Gene expression levels from RT-PCR are compared with original RNA-Seq data (Procedure 3), RNA-Seq data with random hexamer bias correction, and RNA-Seq data with GAM correction.Click here for file


GOBO supports a wide range of applications including: 1) rapid assessment of gene expression levels in subgroups of breast tumors and cell lines, 2) identification of co-expressed genes for creation of potential metagenes, 3) association with outcome for gene expression levels of single genes, sets of genes, or gene signatures in multiple subgroups of the 1881-sample breast cancer data set. The design and implementation of GOBO facilitate easy incorporation of additional query functions and applications, as well as additional data sets irrespective of tumor type and array platform.Microarray-based gene expression analysis holds promise of improving prognostication and treatment decisions for breast cancer patients. However, the heterogeneity of breast cancer emphasizes the need for validation of prognostic gene signatures in larger sample sets stratified into relevant subgroups. Here, we describe a multifunctional user-friendly online tool, GOBO ( Breast cancer is one of the most common malignancies in the world. It is evident that breast cancer is a heterogeneous disease characterized by different tumor biological and histological subtypes, age of onset, clinical course and responsiveness to treatment. Although currently used prognostic and treatment-predictive markers for breast cancer have markedly improved treatment decisions, additional markers are required to further tailor treatment for individual patients.CCNB1 gene and a Core Serum Response gene signature as two examples we illustrate the usefulness of GOBO for rapid online analysis in a large breast cancer data setMicroarray-based gene expression profiling has been used extensively to characterize the transcriptome of breast cancer, resulting in the identification of new molecular subtypes and markers or signatures of potential therapeutic and prognostic importance The web interface of GOBO allows precompiled data sets to be queried by the three main applications of GOBO: Gene Set Analysis (GSA), Co-expressed Genes (CG), and Sample Prediction (SP) . CurrentPooling of gene expression data sets generated at different institutions and time points may introduce artifacts due to, e.g., different reagent lots, scanner settings, RNA extraction procedures, sample selections and laboratory practices. To investigate whether such data set related artifacts existed in the 1881-sample set we undertook three investigations. First, we performed principal component analysis (PCA) and analyzed associations between principal components and a variety of sample annotations  is computed, prior to dividing the data set into patient cohorts based on gene expression quantiles. Association with outcome is investigated for stratified patient cohorts using distant metastasis-free survival (DMFS), relapse-free survival (RFS) or overall survival (OS) in different breast cancer subgroups. GSA-Tumor allows division of the 1881-sample data set into five major subgroups based on clinical annotations available from GEO CCNB1) and CDK1 CCNB1 is also included in several prognostic gene signatures such as the 21-gene recurrence score CCNB1 gene expression  to CCNB1 in our analysis. However, when performing a CG analysis in the Neve et al. cell line panel we found PLK1, AURKB and CENPE to be correlated (Pearson correlation >0.5) with CCNB1 given a gene expression log2 standard deviation cut-off >0.4. Association with outcome for a derived co-expressed gene set may be further investigated by subsequent GSA-Tumor or SP analysis  and the breast cancer cell line data set. The list of co-expressed genes can be further submitted to iterative correlation analysis, identifying which of the co-expressed genes that show tight co-expression with each other in an iterative process across the selected data set. The result of such an analysis could be a smaller cluster of genes that all are highly correlated with each other, representing a more distinct metagene or gene module . Using the CG application, we searched for genes highly co-expressed with l. CCNB1 . When sul. CCNB1 . Functiol. CCNB1 . CCNB1 manalysis , File S1A multitude of microarray-based gene signatures have been derived stratifying breast cancer into subtypes Cancer invasion and metastasis represent the often-fatal steps in tumor progression, but remain poorly understood. Clearly, modifications in the tumor microenvironment involving not only tumor cells but also surrounding tissue, e.g., tumor-associated fibroblasts, are required for a tumor to become invasive and able to metastasize. Observations of histological similarities between the tumor microenvironment and normal wound healing have led to the hypothesis that tumors represent “wounds that do not heal” CCNB1 GSA-Tumor analyses are consistent with results from meta-analyses of different prognostic gene signatures, including CSR, identifying proliferation as the major constituent of several signatures SP analysis also reveals if a signature is associated with outcome within subgroups of breast cancer stratified using clinical and molecular variables. Association with outcome for the CSR signature was mainly observed for ER-positive, LN-positive, LN-negative, untreated tumors, tamoxifen (TAM) alone treated tumors, or ER-positive/LN-negative tumors . In furtCCNB1 and the CSR gene signature as two examples for rapid online extraction of gene expression patterns, co-expressed genes, and survival analysis in a large breast cancer data set.GOBO is a convenient and user-friendly online tool for preliminary analysis of association with outcome for gene expression levels of single genes, sets of genes or gene signatures in a large public breast cancer microarray data set. Moreover, GOBO offers the possibility of investigation of gene expression levels in breast cancer subgroups and breast cancer cell lines for gene sets, as well as creation of potential metagenes based on iterative correlation analysis to a prototype gene. We have here demonstrated the usefulness of GOBO using GOBO is designed in three modules; a data set module, a web interface module, and a data processing module . The dathttp://co.bmc.lu.se/gobo. The design and implementation of GOBO facilitate easy incorporation of additional query functions and applications, as well as additional data sets irrespective of tumor type and array platform in the form of precompiled R-data sets. Future integration of data sets with additional information such as mutation status, copy number alterations or methylation data may further enhance the usefulness of GOBO.GOBO is freely accessible from the GOBO website Table S1Result of Co-expressed genes analysis for CCNB1. An Excel table listing genes co-expressed with CCNB1 in the combined breast cancer data set.(XLS)Click here for additional data file.Table S2Result of functional analysis of genes co-expressed with CCNB1 using DAVID. An Excel table showing the results from functional analysis using DAVID for CCNB1 and 34 co-expressed genes.(XLS)Click here for additional data file.File S1Supplementary Methods. A Word document describing the preprocessing of the breast tumor and breast cancer cell line microarray data sets, and settings for described GOBO analyses.(DOCX)Click here for additional data file.File S2Investigation of data set bias in the combined 1881-sample data set. A PDF file with figures S1 to S5 displaying result of PCA analysis of the 1881-sample set, examples of gene expression levels for Affymetrix probe sets matching ER and HER2 across clinical and molecular annotations, and clustering of the 1881 cases using the PAM50 gene set.(PDF)Click here for additional data file.


Our goal was to examine how various aspects of a gene signature influence the success of developing multi-gene prediction models. We inserted gene signatures into three real data sets by altering the expression level of existing probe sets. We varied the number of probe sets perturbed (signature size), the fold increase of mean probe set expression in perturbed compared to unperturbed data (signature strength) and the number of samples perturbed. Prediction models were trained to identify which cases had been perturbed. Performance was estimated using Monte-Carlo cross validation.Signature strength had the greatest influence on predictor performance. It was possible to develop almost perfect predictors with as few as 10 features if the fold difference in mean expression values were > 2 even when the spiked samples represented 10% of all samples. We also assessed the gene signature set size and strength for 9 real clinical prediction problems in six different breast cancer data sets.We found sufficiently large and strong predictive signatures only for distinguishing ER-positive from ER-negative cancers, there were no strong signatures for more subtle prediction problems. Current statistical methods efficiently identify highly informative features in gene expression data if such features exist and accurate models can be built with as few as 10 highly informative features. Features can be considered highly informative if at least 2-fold expression difference exists between comparison groups but such features do not appear to be common for many clinically relevant prediction problems in human data sets. Gene expression data are commonly used to develop multi-gene prediction models for various clinical classification problems. Several gene expression-based multivariate prognostic and treatment sensitivity predictors have been developed for breast cancer and numerous other "gene signatures" have been reported to predict specific biological states including pathway activity and mutation status of p53, BRCA, PIK3 and other genes in cancer -9. HowevSupervised classification models are developed through comparison of groups of samples that differ in clinical outcome of interest. The first step typically involves identification of informative probe sets/genes (i.e. features) that are differentially expressed between the groups. Next, these informative features are considered as variables to train a multivariate classification model. Intuitively, the predictive performance of classifiers must depend on the number of informative features, the magnitude of difference in feature expression levels between the groups of interest, and the number of informative cases in each group. These critical parameters are expected to vary from classification problem to classification problem and from data set to data set. However, it is not well understood how each of these components influence the success of the classifier development process and what the minimum requirement to develop successful predictors might be.The goal of this analysis was to take public breast cancer gene expression datasets, spike these with a series of artificial "gene signatures" and assess how well these spiked-in gene signatures could be recovered and used to develop a multi-gene classifier to predict "spiked-in" status of a sample. The artificial gene signatures consisted of real probe sets whose expression values were increased (i.e. spiked) with a constant. The extent of perturbation varied over a broad range for three key parameters: (i) the number of samples perturbed (i.e. informative cases), (ii) the number of probe sets included in the artificial signature (i.e. signature size), and (iii) the fold increase in mean expression value for the spiked probes (i.e. signature strength). To place our findings into context, we also calculated gene signature size and strength for nine different real-life clinical prediction problems in six different data sets.http://www.bioconductor.org.We used 3 publically available human breast cancer gene expression data sets each generated with Affymetrix U133A gene chips. These included the Microarray Quality Control Consortium (MAQC II) breast cancer data  to be perturbed in each data set. In the classification exercise described below, these s perturbed samples represent one class and the remaining samples in the dataset represent the other class. For each s sample set, we randomly selected g probe sets  to represent the informative features (i.e. spiked gene signature). We altered the normalized, log2-transformed expression values of each g probe sets by adding the same c constant . This is equivalent to multiplying the original scale value by 2c. So, c = 0 corresponds to unperturbed data, c = 1.0 corresponds to a 2-fold increase and c = 2 corresponds to a 4-fold increase. By perturbing probe sets only in the s randomly selected samples, we are generating gene signatures between the two classes of increasing size and strength. We also created perturbed data sets where the probe sets were perturbed with randomly picked constants within brackets of c values using uniform distribution including c = (0.0 to 0.5), (0.50 to 1.0), (1.00 to 1.2), (1.2 to 1.5), (1.5 to 2.0), (2.0 to 2.5), (2.5 to 3.0), and (3.0 to 4.0). We repeated the entire perturbation process 20 times for each possible s(sample)-g(probe set)-c(constant) combination in all 3 data sets.We randomly selected n features  to construct multivariate prediction models using Diagonal Linear Discriminant Analysis (DLDA) for each s-g-c combination [We used t-tests to compare the spiked cases with the rest of the cases to identify differentially expressed probe sets among all probe sets represented on the arrays. We ranked all probe sets by p-value and used the top bination ,19,21. Dn features in each iteration of the t-test. As reference point for these observations, we also used the same method to identify informative probe sets for 9 real clinical classification problems. First we compared ER-positive with ER-negative breast cancers, then cancers with pathologic complete response (pCR) to chemotherapy versus lesser response (RD), and subsequently cases with pCR versus RD among ER-negative cancers (GEO data set: GSE 16716). During these analyses we kept the overall sample sizes and the proportion of informative cases identical for each 3 comparisons in order to standardize the power of the analysis. The smallest sample size was the comparison of pCR versus RD among ER-negative cancers that included 91 cases . Therefore, we only used a randomly sampled subset of 91 cases from the total of 233 MAQCII cases for the "ER-positive versus ER-negative" and "pCR versus RD" comparisons and also fixed the proportion of informative cases to 41 versus 50 for each comparisons. To obtain further reference values to interpret our spiked-in simulation results we also generated lists of informative probe sets to distinguish cancers that relapsed with those that did not relapse , inflammatory from non-inflammatory breast cancer (GSE 22597), p53-mutant from p53-normal (GSE 3494) and PIK3CA-mutant from PIK3-normal cancers [fdrtool software package [We tracked how many of the spiked probe sets (i.e. truly informative features) were included in the top ncer GSE 597, p53-s-g-c combination (N = 9(s) × 9(g) × 8(c) × 20 (replicates) = 12, 960 data sets). To estimate predictive performance, we performed stratified 3-fold Monte Carlo Cross-Validation (MCCV) with 100 iterations. During cross validation, we randomly selected 2/3 of the data to train the model and the remaining samples were used as a test set. This selection process was done separately within the spiked and non-spiked groups in order to maintain the same proportion of perturbed and unperturbed cases in the training and testing sets. New feature selection was performed during each iteration.Our goal was to predict which samples were perturbed. We built prediction models with the top 10, 25, 50, 100, 500 features selected by t-test and ranked by p-value within each The classification process was assessed by two metrics: (i) the area above the receiver operating characteristic curve (AAC) which is the complement of the area under the curve (AUC) (AAC = 1-AUC) and (ii) the spiked probe recovery rate. When the total number of spiked probe sets is less than the number of features included in the classifier, we computed the recovery rate by dividing the number of correctly selected spiked probes included in the model by the number of all spiked probes. If the number of spiked probes was greater than the number of features included in the classifier we calculated the recovery rate by dividing the number of correctly selected spiked probe sets by the number of all features included in the classifier. The ability to include the perturbed/spiked probe sets in the classifier is critical for producing a model with good predictive accuracy (i.e. performance). Thus the rate at which spiked probe sets are included in the classifier  is a useful metric for evaluating the classification process.An R-code package has been developed to run the complete probe set spiking, model building and predictor evaluation process and is included in Additional File s = 10 to 100  and the fold change of the spiked probe sets over c = 0.5, 1.0, 1.2, and 1.5. Fold change (i.e. signature strength) had the greatest effect on feature recovery rate. With c = 1.5, the average recovery rate was > 60% (range: 30-80%) even when only 10 samples were spiked. When the spiked sample size reached 40  the average recovery rate increased to > 80% (range: 70-100%). On the other hand, when the c was a modest 0.5, the recovery rate remained low, around 40% even when the spiked-in sample size was 100. Similar trends were seen when 50 or 500 probe sets were spiked (Supplementary Results). These observations were consistent across all 3 data sets and indicate that fold increase in the expression value of informative probe sets has a major influence on feature recovery rate. Probe sets with less than 1.5-fold difference in mean expression values between comparison groups are difficult to identify as informative features.First, we examined the spiked probe set recovery rates as function of signature strength and number of spiked samples. Figure g = 10-500), results were least sensitive to the number of features included in the classification models  when 20-25 samples (9-11% of samples) were spiked , (0.50-1.0), (1.00-1.2) and (1.20-1.5). The individual c constants that were added to the log2 expression values of each of the probe sets selected for spiking were randomly picked from all possible values within a given bracket. This experiment yielded similar results as presented on Figure We also examined the more realistic scenario when we altered the expression of the selected probe sets over a range of fold change that differed from probe set to probe set but remained within brackets of pre-specified maximum , 1.00-1. and  Figure . PredictTo further explore the effect of noise (i.e. uninformative features included in the model) on predictor performance, we kept the number of spiked probes at 10 and set the the predictive model size at 100 features. Figure Genomic classifiers developed to predict clinical outcome in breast cancer usually yield substantially lower AUC values than what we could achieve in our simulation experiments ,19,20. TThese observations confirm that for easier classification problems, such as ER status prediction, a large number of informative probe sets exist and these show large fold differences but for all other prediction problems that we tested the number of informative features was low and feature strength was modest at best.The goal of this project was to assess how gene signature size (i.e. the number of informative probe sets), signature strength (i.e. fold difference in the mean expression of the informative probe sets between the comparison groups) and the number of informative cases in a data set influence the success of developing multi-gene prediction models. We also examined how model performance deteriorates as increasing amount of noise (i.e. un-informative features) are included in a model. To study these variables we altered the true expression values of randomly selected genes in real human gene expression data sets. What motivated this research was to find out what signal elements in the genomic model building process may explain the substantial difficulty to find clinically relevant multi-gene predictors for many cancer classification problems.Our results demonstrate that signature strength had the greatest influence on the success of model building followed by the size of the gene signature and the number of informative cases. Predictors tolerated large amounts of noise included in the model as long as it also contained numerous strong informative features. It was remarkably easy to develop almost perfect predictors with as few as 10 informative probe sets if the fold difference in the expression values of these probe sets was > 2.0, even if the informative samples represented < 10% of the total sample size. These simulated results are better than the reported performance of the majority of empirically developed genomic outcome predictors ,23-27. TThese results indicate that current statistical methods efficiently identify highly informative features in complex gene expression data sets if such features exist. Features can be considered highly informative if at least a 2-fold expression difference exists between the means of the two comparison groups. With as few as 10 such features almost perfect classification models can be built. However, highly informative features do not appear to be common for many clinically relevant prediction problems. There may be several biological explanations for this. First, the dynamic regulation of the expression of the thousands of mRNA species in cancer cells is not well understood. What level of change in mRNA expression leads to functionally important consequences  is unknown and it is likely to be different from gene to gene. Some physiological variables are very tightly regulated, a 10% change in serum sodium levels can lead to life threading consequences whereas variables such as hear rate or blood pressure have broad normal dynamic range. It is likely that similar phenomena also occur in the mRNA world and a 15-50% percent change in the expression level of some mRNA species may result in functional consequences but such genes would not define strong predictive features. Also importantly, protein expression levels correlate with mRNA expression only moderately and protein levels or posttranslational changes in proteins may represent the functional activity of biological pathways more accurately than mRNA levels. Structural variations in genes at the DNA level can profoundly alter the functional activity of proteins such mutations may also not cause large scale changes in mRNA expression.While our spike-in simulation studies yielded much useful information, they are limited in several respects. In real datasets expression values responding to regulatory influences would presumably change in a coordinated rather than random fashion. However, the degree of change and number of genes involved in a particular coordinated change may be relatively small and thus not conducive for use in a predictive model. Overall these observations are consistent with previous sensitivity analysis of prediction models that have highlighted the vulnerability of predictors to feature strength -30.In summary, our findings suggest that in the currently available mRNA gene expression data sets of breast cancer there may not be enough highly informative genes to develop clinically useful genomic classifies to predict certain types of clinical outcomes. We recognize that the sample sizes that are currently available for analysis are modest, and particularly small analyses are performed in disease subsets. Our simulation results suggest that predictor performance in the absence of strong predictive signatures may improve only modestly even if sample size increases. We developed an R software code package that could be used to estimate the number of informative probe sets, based on gene spiking, that is needed for accurate model building in any experimental data set. This could be used as a tool to estimate the adequacy of a given data set to yield empirically derived classifiers.Conception and design KH, LP; Acquisition of data WFS, YQ, LP; Analysis of data CW, KH, YQ, TI, LP; Interpretation of data KH, CW, YQ, TI, WFS, LP; Drafting the manuscript or revising KH, CW, YQ, TI, WFS, LP; Final approval KH, CW, YQ, TI, WFS, LP.The Breast Cancer Research Foundation R Code. R language programming code used to perform the analyses presented in the paper.Click here for fileSupportive Results. Supportive results of analyses on additional datasets.Click here for file