Molecular biomarkers that are based on mRNA transcripts are being developed for the diagnosis and treatment of a number of diseases. DNA microarrays are one of the primary technologies being used to develop classifiers from gene expression data for clinically relevant outcomes. Microarray assays are highly multiplexed measures of comparative gene expression but have a limited dynamic range of measurement and show compression in fold change detection. To increase the clinical utility of microarrays, assay controls are needed that benchmark performance using metrics that are relevant to the analysis of genomic data generated with biological samples.2 ratio deviation plots made for each dataset. Small but statistically significant differences in diagnostic performance were observed between standardized assays available from the array manufacturer and alternative methods for target generation. Assay performance using the reliable range of comparative measurement as a metric was improved by adjusting sample hybridization conditions for one commercial kit.Ratiometric controls were prepared from commercial sources of high quality RNA from human tissues with distinctly different expression profiles and mixed in defined ratios. The samples were processed using six different target labeling protocols and replicate datasets were generated on high density gene expression microarrays. The area under the curve from receiver operating characteristic plots was calculated to measure diagnostic performance. The reliable region of the dynamic range was derived from logProcess improvement in microarray assay performance was demonstrated using samples prepared from commercially available materials and two metrics - diagnostic performance and the reliable range of measurement. These methods have advantages over approaches that use a limited set of external controls or correlations to reference sets, because they provide benchmark values that can be used by clinical laboratories to help optimize protocol conditions and laboratory proficiency with microarray assays. As one of the first 'omic scale technologies in wide-spread use, microarrays provide a test case for addressing issues associated with highly multiplexed assays and high dimensional datasets. During the last decade, community-wide efforts were initiated to address concerns that arose with the increased use of microarray technology in research. These concerns include data comparability, the lack of universal controls for gene expression experiments, the need for data reporting standards, and methods for determining appropriate statistical approaches. The MicroArray Quality Control (MAQC) projects led by the FDA ,2, the EThe use of microarray-based clinical tests, although currently limited, is expected to expand with the increasing incorporation of personalized medicine into medical practice. The availability of clinically directed standards would help promote the development of highly multiplexed gene expression assays in laboratory medicine . The useObjective criteria to measure performance on microarrays can be developed for samples which are designed to have known differences in expression. We developed a system for assessing technical performance with microarray assays that involves two biologically complex samples (mixed tissue ratiometric controls (MTRC)) that are representative of experimental samples . MTRC aret al. identified testis, liver, brain (cerebellum and cerebrum), skeletal muscle, and heart as the tissues with the highest number of organ-specific genes using a set of cDNA microarray data from 19 different organs from 30 individual donors [et al., the largest numbers of specifically expressed gene transcripts were found in testis, pancreas, liver, placenta, thymus, salivary gland, kidney, heart, and cerebellum [Mixed tissue ratiometric controls for monitoring performance with microarrays are two samples that are composed of RNA of high and reproducible quality from 3 or 4 different tissues that are sufficiently different in gene expression. The samples are designated MTRC-3 or MTRC-4 if they are composed of three or four different RNA sources, respectively. The four components of the human MTRC-4 used as the example in this study are UHRR, HBRR, liver RNA, and skeletal muscle RNA. Two components of the MTRC-4 (UHRR and HBRR) were chosen because they have been extensively tested in the MAQC-I project and are commercially available in large lots. The additional components of the MTRC-4 were chosen from the normal human tissues that had high levels of selective gene expression in published reports. Son l donors . Of the rebellum . We survrebellum .2 signal intensity in that tissue and its maximum value in the remaining tissues was calculated to determine a tissue-selective index (TSI) for each dataset. Using a mean TSI cutoff of 3.22 log2 units, a total of 1035 tissue-selective analytes were identified for the MTRC-4 that included 429 HBRR-, 255 liver RNA-, 197 UHRR-, and 154 skeletal muscle RNA-selective probe sets  compared to the true negative (1-to-1) subset and the corresponding AUCs calculated as described previously [2 ratios of 1035 tissue-selective analytes. The mean ρ (± standard deviation) for datasets 1A-4, 1B-4, 3A-4, and 3B-4 was 0.897 ± 0.02, 0.928 ± 0.004, 0.861 ± 0.019, and 0.852 ± 0.003, respectively. No significant difference in 1.5-fold AUC was found between two versions of a manufacturer's kit (datasets 1A-4 and 2A-4) or between variations in the aRNA protocol in the time of labeling (datasets 1A-4 and 1B-4). There was also no significant difference in the AUCs when the amount of ss-cDNA hybridized to arrays was changed within a protocol (dataset 3A-4 vs. 3B-4).For each labeling method, a p-value was calculated for each MTRC-4 analyte using a paired t-test comparison of normalized logeviously . No signeviously  which ca2 ratio deviation (LRD) for MTRC analytes as a function of signal intensity. The LRD is the difference between the observed and expected ratio for each MTRC analyte. In LRD-plots, analytes are rank ordered by intensity and divided into bins containing approximately equal numbers of data points. Next, the mean and standard deviation of the LRD and the mean and range of intensities are calculated for each bin and plotted. Finally, the LRD is used in a chi-square test to evaluate each bin as follows:Comparative expression data from microarrays is constrained at the upper and lower regions of signal intensity by the output range of the scanner and the noise inherent in measuring low intensity signals, respectively . A Ration is the number of analytes per bin. The standard deviation (s) in the log2 ratios of all probe sets is used as the variance estimate and a comparison to the χ2 distribution at α = 0.01 is used to evaluate the goodness of fit within each bin. The set of contiguous bins that pass the χ2 test is identified and used to define the upper and lower limits of a reliable range of measurement using the maximum and minimum intensities from the "passing" bins, respectively  designed to have one component that is 1.5-fold higher in Mix1 than Mix2, one component that is 1.5-fold lower in Mix1 than Mix2, and one component at a 1-to-1 ratio for normalization. To choose which RNA source in the MTRC-4 to exclude in the MTRC-3, we systematically omitted one of the four RNAs in the MTRC-4 and recalculated the number of tissue-selective analytes amongst the remaining three RNAs in the comparison. Excluding skeletal muscle RNA, liver RNA, or HBRR produced 1085, 952, or 773 analytes, respectively. The highest number of tissue-selective analytes (1151) was achieved by leaving out UHRR . Our results indicated that the shorter incubation time could be used without significantly impacting assay performance. For these process improvement applications, we have found that a test for accurate measurement of 1.5-fold changes in comparative expression is a better discriminator of subtle effects on performance than assays for changes that are two-fold or higher. A target ratio of 1.5-fold is included in the MTRC-4 as one of 4 components or in the MTRC-3 as the single true positive change. We recommend using the 1.5-AUC metric, which incorporates a measure of diagnostic accuracy and proficiency in replication of results with the MTRC, as a first tier test for process improvement.Although correlation to reference sets or within replicates is often used to assess performance on microarrays, this metric is of limited value for measuring improvement in laboratory techniques. Datasets generated using methods 3A and 3B had similar mean Pearson's correlation coefficients (data not shown) and Spearman's rank correlation coefficients for ratio reproducibility between technical replicates. However, a clear improvement in the RI-plot Figures  and in tWe demonstrated two options in sample design for mixed tissue ratiometric controls. The same metrics  can be calculated with either design and yielded similar results. The MTRC-3 design is easily tunable to measure a target ratio of interest and has an advantage over the MTRC-4 in requiring only 3 different sources of reliably good quality human tissue RNA. The human MTRC-4 is directly comparable to the previously described rat MTRRM design used to illustrate the utility of a ROC-plot AUC to identify outliers in performance in a large set of proficiency testing data ,15.Previously, we have described methodology for using the MAQC samples A and B, and two dilutions of A and B, for performance assessments based on ROC-plot metrics . HoweverWhile DNA microarrays are widely used in medical research in developing improved assays for prediction or diagnosis of disease or for assessing the efficacy and adverse effects of pharmaceuticals, their use in clinical practice is currently limited. One microarray-based transcriptional profiling assay is currently approved by the FDA for predicting responsiveness to certain therapeutic interventions for breast cancer and several others are under development . The expHuman tissue RNA was obtained from Ambion  or Stratagene . The catalog and lot numbers of the products tested are indicated in Table RNA quantification and assessment of purity was performed on a NanoDrop spectrophotometer . RNA quality was assessed using an Agilent RNA 6000 Nano kit, an Agilent Bioanalyzer , and the manufacturer's software to assign RINs . The RIN2 signal intensities were derived and quantile normalized using the Robust Multichip Analysis (RMA) algorithm in the Affymetrix Expression Console software. A mean TSI for each probe set was calculated from the three replicate datasets and the threshold for tissue selectivity was a mean TSI greater than 3.22 log2 units. For the MTRC-4, 429 HBRR-, 255 liver RNA-, 197 UHRR-, and 154 skeletal muscle RNA-selective analytes were identified  and hybridized to Human Genome U133A 2.0 arrays . Three replicate datasets were created by labeling and hybridizing the four tissue RNA samples on separate dates to introduce technical variation. The commercial lots of RNA used for analyte selection are indicated in Table A and hybFour batches of MTRC-4 were prepared using total RNA from four human tissues from different lots Table . BatchesMTRC-3 were prepared from total RNA from three human tissues using the lots indicated for Batch 5 in Table http://www.ebi.ac.uk/arrayexpress.Labeled target was prepared from total RNA samples using one of the following three reagent kits: (1) 3' IVT Express kit (Affymetrix Part No. 901229), (2) IVT kit (Affymetrix Part No. 900449), or (3) Ovation RNA Amplification System V2 and the FL-Ovation cDNA Biotin module kits . Labeled target was hybridized to Affymetrix Human Genome U133A 2.0 arrays in a GeneChip Hybridization Oven 640. The arrays were washed on an Affymetrix GeneChip Fluidics Station 450 using fluidics protocol FS450-002 and scanned using an Affymetrix GeneChip Scanner 3000 7G. The variations made in protocols for each target labeling method are listed in Table Each dataset that was created using a different target labeling protocol or different batch of MTRC was processed separately with RMA. The selective analytes for the 1-to-1 component (UHRR for the MTRC-4 and HBRR for the MTRC-3) were used to normalize Mix2 with respect to Mix1. For each technical replicate, the difference in the 10% trimmed mean intensity between the Mix1 and Mix2 data for analytes in the 1-to-1 component was calculated and used to correct the Mix2 signal data.2 ratio [For ROC-plot calculations using the MTRC, the true negatives are the tissue selective analytes for tissue RNA that is mixed in a 1-to-1 ratio between Mix1 and Mix2. Separate ROC-plots are generated for each true positive subset in the MTRC. For MTRC-4, the true positives are the 1.5-, 2-, and 4-fold changes and for the MTRC-3, the true positives are the 1.5-fold changes in both directions. For singlicate assays, analytes are ranked by log2 ratio . For rep2 ratio  using th2 ratio .2 signals were averaged across technical replicates of either Mix1 or Mix2. The average log2 signals (ratio (R) and intensity (I) for each analyte as follows:For each dataset, the normalized logs) ranged from 0.30 - 0.34 for MTRC-4 datasets and from 0.16 - 0.21 for MTRC-3 datasets.For MTRC-4 datasets, the analytes (excluding the 1-to-1 component) were distributed into 19 bins of 42, with a single bin of 40 at the lowest intensity. For MTRC-3 datasets, the analytes (excluding the 1-to-1 component) were distributed into 20 bins of 28 analytes, omitting the 5 lowest intensity analytes. The standard deviation , and Tissue Selective Index (TSI). This file is formatted as an Excel spreadsheet.Click here for fileLists of MTRC-3 analytes for Affymetrix HG-U133A 2.0 arrays. Analytes selective for HBRR, human liver RNA, and human skeletal muscle RNA are identified by Affymetrix probe set identifier, gene name (annotation date 3-17-2008), and Tissue Selective Index (TSI). This file is formatted as an Excel spreadsheet.Click here for file


Multivariate assays (MVAs) for assisting clinical decisions are becoming commonly available, but due to complexity, are often considered a high-risk approach. A key concern is that uncertainty on the assay's final results is not well understood. This study focuses on developing a process to characterize error introduced in the MVA's results from the intrinsic error in the laboratory process: sample preparation and measurement of the contributing factors, such as gene expression.Using the PAM50 Breast Cancer Intrinsic Classifier, we show how to characterize error within an MVA, and how these errors may affect results reported to clinicians. First we estimated the error distribution for measured factors within the PAM50 assay by performing repeated measures on four archetypal samples representative of the major breast cancer tumor subtypes. Then, using the error distributions and the original archetypal sample data, we used Monte Carlo simulations to generate a sufficient number of simulated samples. The effect of these errors on the PAM50 tumor subtype classification was estimated by measuring subtype reproducibility after classifying all simulated samples. Subtype reproducibility was measured as the percentage of simulated samples classified identically to the parent sample. The simulation was thereafter repeated on a large, independent data set of samples from the GEICAM 9906 clinical trial. Simulated samples from the GEICAM sample set were used to explore a more realistic scenario where, unlike archetypal samples, many samples are not easily classified.All simulated samples derived from the archetypal samples were classified identically to the parent sample. Subtypes for simulated samples from the GEICAM set were also highly reproducible, but there were a non-negligible number of samples that exhibit significant variability in their classification.We have developed a general methodology to estimate the effects of intrinsic errors within MVAs. We have applied the method to the PAM50 assay, showing that the PAM50 results are resilient to intrinsic errors within the assay, but also finding that in non-archetypal samples, experimental errors can lead to quite different classification of a tumor. Finally we propose a way to provide the uncertainty information in a usable way for clinicians. Oncotype DX®) [®) [Multivariate assays (MVAs) using gene expression as their contributing factors are becoming commonly used in assisting treatment decisions in medicine, especially in oncology. Examples of MVAs available for planning breast cancer treatment include the 55-gene subtype classifier (PAM50 Breast Cancer Intrinsic Classifier) , the 21-ype DX®)  and the DX®) [®) . Though The PAM50 assay measures the expression level of 55 genes (50 classifier genes and 5 housekeepers) creating a "signature" that is compared, using Spearman's Rho as a distance measure, to each of five centroids ,5 represThe complexity of the PAM50 assay demonstrates the challenge of identifying and understanding error sources from the moment a sample is received by the reference laboratory until a clinical report is generated. Error sources to be considered include heterogeneity and sample preparation, as well as technical variability, which may be separated into error from measuring devices as well as intrinsic and explicit classification uncertainties. While the overall purpose of this project is to develop a model for estimating the effect of all error sources on final MVA results, in this paper we focus on how to estimate the effect that intrinsic gene expression measurement errors, including those associated with sample and assay preparation, have on the classification of tumors.A comprehensive experimental study to estimate the effect that the intrinsic gene expression measurement errors have on the classification of tumors and gene-score classifications requires, in principle, repeated testing of a significant number of samples from each subtype and thorough analysis of the misclassifications observed. Such a comprehensive approach is unfeasible in terms of cost and sample availability. Here we have adopted a hybrid approach in which we perform repeated experimental measurements on one sample from each subtype  to determine the experimental variability of the measured gene expression for each of the 50 genes included in the PAM50 assay. Using this experimental information we proceed to generate a Gaussian error distribution that can be used to generate multiple data sets by way of Monte Carlo simulations. These simulations impose random errors, given by the Gaussian distribution, on the set of experimental measured samples. Monte Carlo simulations are well suited for this hybrid analysis because there is extensive literature for using this approach to estimate errors in high-dimensional problems, such as fluids and thermodynamics -10, wherSpecifically, in this study we followed five major steps: (1) collect and prepare four archetypal samples representative of each cancer subtype; (2) characterize the intrinsic error for each gene's expression values in the assay by making twelve measurements for each gene's expression on each archetypal sample and determine the distribution type that best models the experimental errors; (3) using Monte Carlo simulations generate a sufficient number of simulated test samples, based on a defined confidence interval width, by imposing the errors generated using the distribution from (2) onto the archetypal samples; (4) determine the effect of the variability imposed in the simulated samples on their classification; and (5) repeat steps (3) and (4) on an independent set of samples from the GEICAM 9906 clinical trial (GEICAM) .In order to characterize the error in gene expression measurements, four archetypal samples representative of each cancer subtype with sufficient genetic material were constructed - since most single samples do not have enough genetic material to be tested more than twice. Cell lines representative of Basal-like (ME16C) and Luminal B (MCF7) subtypes were grown in the Reagent Lab at ARUP Laboratories. Luminal A and HER2-enriched subtypes were not readily available as a cell lines. As such, 20 patient tumor samples previously identified as archetypal Luminal A (10 samples) and HER2-enriched (10 samples), based on PAM50 gene scores and classification, were collected under IRB approved protocols at the University of Utah to be combined and treated as single tumor samples.http://www.thermofisher.com) followed by dehydration in absolute ethanol. RNA extraction was completed on a Biomek NX Laboratory Automation Workstation  using the AgenCourt FormaPure Kit  according to the manufacturer's instructions and including a DNase I step. RNA quantification was done on a Paradigm Detection Platform  using the Quant-iT RiboGreen Assay Kit . cDNA synthesis was performed on the Biomek FX Laboratory Automation Workstation  using 600 ng of RNA, uracil containing dNTPs , random primers , gene-specific, downstream PCR primers , and SuperScript III Reverse Transcriptase .RNA was extracted from tumor-enriched areas of formalin-fixed, paraffin-embedded (FFPE) tissue blocks, containing more than 70% tumor cells, as determined during review by a board-certified pathologist. Samples were deparaffinized using Citrus Clearing Solvent  and 1.67 ng cDNA were added to the experimental sample wells. Sample cDNA was incubated with LightCycler Uracil-DNA Glycosylase  at 40°C for 10 min and inactivated at 95°C for 10 min prior to performing Real-time, quantitative PCR (RT-qPCR). RT-qPCR was performed on the LightCycler (LC) 480  as follows: 45 cycles at 95°C for 4 sec, 58°C for 6 sec and 72°C for 6 sec. To assure target specificity, RT-qPCR was followed by a melting curve analysis: 95°C for 15 sec, 65°C for 1 min followed by raising the temperature to 99°C while taking 10 fluorescence acquisitions/°C. We then classified the RT-qPCR data from each run. One run from the Luminal A sample failed quality control and was not included in further analysis.Each 5 μL reaction contained 1X LightCycler 480 SYBR Green I Master Mix . The error distribution function type could not be estimated using only the twelve measurements for each gene within a given sample subtype, therefore to determine the error distribution function type for each gene, all four sample subtypes were median-centered by gene and combined, giving forty-seven data points per gene, since one of the archetypal Luminal A samples failed quality control. As depicted in Figure The R Statistical Package (R) [In order to estimate the intrinsic experimental error in gene expression measurements within our laboratory, we performed twelve measurements for each gene on each archetypal sample. Each of the four archetypal samples was separated into, and treated as, twelve individual samples, and measured by RT-qPCR on the Roche LightCycler (LC) 480  .Before performing the Monte Carlo simulations an analysis to justify sample size was performed to ensure sufficient confidence for the analysis when using our target of 100,000 simulated samples. We calculated the 95% confidence interval width around the percentage of correct classifications using 100,000 simulated samples for each archetypal sample. For a dichotomous variable (i.e. misclassified or not), a confidence interval width (W) can be calculated (Eq. 1) given an alpha0.05 = 1.96), expected proportion of misclassifications (θ = 0.02) and the sample size, or in this case, the number of simulations . The calculated confidence interval width is ± 0.00087 for each simulation of 100,000 samples, which is an acceptable value.level (ZMonte Carlo simulations were performed using the mean (μ) value and standard deviation (σ) for the expression of each gene within the archetypal samples, as described above. This procedure created a total of two hundred independent distributions, i.e. fifty Gaussian distributions (one for each gene) for each of the four archetypal samples. For example, the mean expression value (μ) for ACTR3B from the twelve Luminal B values was 1.94 and the corresponding standard deviation (σ) was 0.085. Therefore the ACTR3B expression value for each of the 100,000 simulated Luminal B samples was randomly selected from a Gaussian distribution centered on a mean (μ) of 1.94 and with a standard deviation (σ) of 0.085. Randomly selecting a value from the Gaussian distribution, as described, does not assume gene expression values are independent of one another, rather the method assumes that the measurement error for each gene is independent. Specifically, since each Gaussian distribution is centered on the gene's mean expression value for the given sample, any genes within the sample that are generally upregulated will all be generally upregulated in the simulated samples, while allowing the error to deviate independently. The 100,000 simulated samples for each archetypal sample were classified using the standard PAM50 process. The effect of intrinsic gene expression measurement error on the tumor classification was assessed by determining the percentage of simulated samples that were classified identically to the original sample. This value provides an estimate of the reproducibility of the results for archetypal samples.Testing archetypal samples is valuable for determining how the PAM50 assay will perform under ideal circumstances, but these results may not be informative when the samples are not as well characterized as the archetypal samples. Thus, the method described above for the archetypal samples was adapted and applied to the larger and more diverse set of independent samples from the GEICAM 9906 clinical trial. A total of 911 breast tumors collected by the GEICAM group for the GEICAM 9906 clinical trial were run and classified by the PAM50. Tumor samples were prepared following the same methods described above for the archetypal samples and those with insufficient tumor content to be classified were excluded from further analyses.R function "loess"  and graphed using "panel.smoother" from the R lattice graphics package. A 95% confidence interval for the fitted line was also calculated to test "best-case" , "worst-case"  and "average-case" (the fitted line) scenarios for subtype reproducibility. The "worst-case" scenario is considered such because it uses the highest estimated standard deviations, or error. Once the loess models were developed, we used these models to predict the standard deviation (σ) to generate Gaussian distributions for the Monte Carlo simulation. Specifically, given the test sample's original subtype and the expression value for a given gene, we used the loess model to predict the standard deviation (σ), or error, to be used in the Gaussian distribution for said gene and sample. The expression value was used as the mean of the Gaussian distribution. We repeated this process for all genes within each sample.As depicted in Figure The model described above was used to generate random variants of the 847 GEICAM samples remaining after excluding those that could not be classified. The subtype classification reproducibility for each GEICAM sample was tested by generating 100,000 simulated samples using Monte Carlo simulations for each of the error models considered above, i.e. "best-," "average-" and "worst-case," for a total of 300,000 simulated samples per GEICAM sample. Based on the same sample justification analysis used for the archetypal samples, the calculated confidence interval width is ± 0.00087 for each simulation of 100,000 samples, which is an acceptable value.After simulating 300,000 samples for each GEICAM sample based on the error models described above, subtype reproducibility for the individual samples was summarized for each tumor subtype  using two statistics: (1) the total percentage of simulated samples that did (or did not) change subtypes with respect to their parent sample and (2) the proportion of simulated samples, corresponding to a single original sample, that were classified (or misclassified) identically to the parent sample. These data serve as an estimate of PAM50's misclassification rate based on intrinsic error within the assay when samples are not in proximity to the PAM50 centroids.All simulated samples derived from the archetypal samples were classified the same subtype as the parent sample, indicating that the PAM50 subtype classification for samples with characteristics close to the PAM50 centroids is highly reproducible and resilient to experimental errors in gene expression measurements. Although this positive result is encouraging for PAM50 classifications, it is not generalizable beyond the four archetypal samples.Tables Results for the GEICAM sample simulations suggest that all subtype classes are highly reproducible. When considering the three different error models, the most reproducible subtype among all GEICAM samples was always the Basal-like for which 98.47% (average-case) of the simulated samples did not change classification, followed by Luminal B , Luminal A  and HER2-enriched . The differences in reproducibility between "best-" and "worst-case" ("best-case" percentages minus "worst-case" percentages) are 0.89%, 1.31%, 1.31% and 4.35% for Basal-like, Luminal B, Luminal A and HER2-enriched, respectively. Therefore, the selection of the error model is not a determining factor in assessing the robustness of the classification due to experimental errors in gene expression measurement.Figure We have developed a method based on Monte Carlo simulations and limited experimental measurements to estimate the effect of the intrinsic experimental errors in the measured factors contributing to MVAs. While the specifics of the error distribution functions given in Additional file Using the proposed method based on Monte Carlo simulations and the error model described here, we have presented data that suggests PAM50's subtype classifications are highly reproducible on a large, independent sample set from the GEICAM 9906 clinical trial. We also show that there are a non-negligible number of samples for which a significant number of the Monte Carlo simulated samples classify differently than the parent sample, indicating that the classification of the original sample may not be reliable. To address this problem we suggest a new score card that can inform clinicians on the probability that a particular sample could be classified as a different tumor subtype.Dr. Philip Bernard is an inventor of the PAM50 breast test and is a co-founder of Bioclassifier LLC, which has licensing rights to the PAM50 signature.ME participated in the study design, performed all computational and statistical analyses, and drafted the manuscript. RB performed laboratory experiments for the repeated measures experiment. KB participated in the study design and edited the manuscript. IS performed laboratory experiments for the GEICAM series. MM designed and carried out GEICAM 9906 Clinical Trial. EC designed and carried out GEICAM 9906 Clinical Trial. RC designed and carried out GEICAM 9906 Clinical Trial. PB participated in the study design and coordination. JF participated in the study design and coordination, and edited the manuscript. All authors read and approved the final manuscript.Error Distributions. Prior to performing Monte Carlo simulations to generate simulated samples, we needed to identify the distribution type that best models the repeated measures (error) data for each gene. Since twelve data points for each gene were not sufficient to identify a distribution type, data from each gene within each of the four archetypal samples were median-centered and combined, giving forty-seven data points per gene, since one of the archetypal Luminal A samples failed quality control. As shown in each of the fifty plots, the repeated measures data is most closely related to a Gaussian distribution, given the symmetry of most genes as well as the data clustering around a single mean value.Click here for file


Gossypium species. Using these assemblies as templates, we describe the design and creation and of a publicly available oligonucleotide array for cotton, useful for all four of the cultivated species.Microarrays offer a powerful tool for diverse applications plant biology and crop improvement. Recently, two comprehensive assemblies of cotton ESTs were constructed based on three Arabidopsis. A small portion of the oligonucleotides target unidentified protein coding sequences, thereby providing an element of gene discovery. Because many oligonucleotides were based on ESTs from fiber-specific cDNA libraries, the microarray has direct application for analysis of the fiber transcriptome. To illustrate the utility of the microarray, we hybridized labeled bud and leaf cDNAs from G. hirsutum and demonstrate technical consistency of results.Synthetic oligonucleotide probes were generated from exemplar sequences of a global assembly of 211,397 cotton ESTs derived from >50 different cDNA libraries representing many different tissue types and tissue treatments. A total of 22,787 oligonucleotide probes are included on the arrays, optimized to target the diversity of the transcriptome and previously studied cotton genes, transcription factors, and genes with homology to http://cottonevolution.info.The cotton oligonucleotide microarray provides a reproducible platform for transcription profiling in cotton, and is made publicly available through  Gossypium contains nine different genome groups comprising approximately 50 species whose phylogenetic relationships have been well-studied  have been domesticated for their abundant seed trichomes. These species collectively provide the foundation for the textile industry worldwide, with most cotton today deriving from G. hirsutum, or upland cotton. Relationships among genome groups have been quantified in several studies, and the closest living relatives of the diploid genome donors to allopolyploid cotton have been identified [T genome [where the T subscript indicates the A genome in the tetraploid (AD) nucleus], was a species much like modern G. arboreum or G. herbaceum, whereas the allopolyploid DT genome is derived from a progenitor similar to modern G. raimondii. These well-established relationships provide a phylogenetic framework to investigate the evolution of gene expression both in terms of domesticated fiber production and polyploidy.-studied . The A-,rbaceum A, Gossypim), and relative ease of adding probes. A small EST assembly  was previously used to generate oligonucleotide probes for cotton fiber .Microarray images were captured using an arrayWoRxThree replications of a three treatment loop design  were hybridized on nine microarrays, where bud and bud' simply represent different aliquots of the same aRNA. The signal intensity data were natural log transformed and median normalized, and the 9,654 genes with complete data were examined for expression differences among the three sample types . We considered a standard mixed linear model for the data from any single gene given byyijk = μ + δi + τj + sk + eijk,yijk denotes the normalized log-scale signal intensity (averaged over duplicate spots) for dye i, sample type j, and slide k; μ denotes a an intercept parameter; δi denotes the effect of dye i; τj denotes the effect of sample type j; sk denotes the random effect of slide k; and eijk denotes a random error term that is intended to capture all other sources of variability.  Here i = 1, 2 (Cy3 and Cy5); j = 1, 2, 3 ; and k = 1, ..., 9 (microarray slides 1 – 9). On the basis of this model, t-tests for differential expression between each pair of sample types  were conducted. The 9,654 p-values from each of these comparisons were converted to q-values using the method of Story and Tibshirani [q-values were used to identify the number of differentially expressed genes for a given comparison when controlling the false discovery rate at various levels.where bshirani . These qESTs: Expressed sequence tagsJJW, JJL, and ZJC designed and provided the first set of oligonucleotides. JAU, JMS, JFW, and ZJC designed, analyzed, and provided the second set of oligonucleotides. FC, AWW, CDT, and ZJC designed and provided the final set of oligonucleotides. LF and JAU grew the plants and extracted RNA. RH, LF, and RAR amplified, labeled, and performed the microarray hybridizations. DN provided the statistical model and analytical framework. LF, JAU, and DN analyzed and interpreted the data. ARG customized an internet accessible MIAME-compliant Stanford Microarray Database for these arrays and created a GEO microarray platform for this and future experiment submissions. JAU, RAR, and JFW conceived the experiment. JAU and JFW drafted the manuscript. All authors have read and approved the manuscript.Project name: The evolutionary genomics of cottonhttp://cottonevolution.infoProject home page: Operating system: Platform independentProgramming language: HTML and XMLLicense: no license requiredComposition of the cotton oligonucleotide microarray. 22,789 oligonucleotides were designed from three separate sets of genic sequences from cotton (See Table Arabidopsis hit. The black boxes indicate the number of probes designed from singletons from their respective assemblies. The remaining boxes with dotted squares indicate the number of probes targeting transcription factors.Click here for fileDistribution of the number of matches of oligonucleotide probes to the Cotton Gene Index 8 (CGI8) assembly. All three sets were queried within the sequences of the CGI8 assembly and only a small number  of probes target (>93% percent identity) more than one CGI8 unigene indicating a potential cross-hybridization or an 'over-split' assembly for the targeted gene.Click here for filend set were derived from contigs or singletons representing individual libraries Many of the oligonucleotides from the 2. Library totals reflect the contigs (respective library's ESTs > 90%) and singletons used to design the oligonucleotides. The two large G. raimondii libraries created from heterogeneous seedling and whole flower tissue are not illustrated.Click here for file


We identified genes differentially expressed between normal lymph nodes of disease free dogs and a subset of the diseased dogs diagnosed with B-cell lymphoma using each technology. There is statistically significant overlap between the RNA-Seq and microarray sets of differentially expressed genes. Analysis of overlapping genes in the context of biological systems suggests elevated expression and activity of PI3K signaling in B-cell lymphoma biopsies compared with normal biopsies, consistent with literature describing successful use of drugs targeting this pathway in lymphomas.Comparative oncology is a developing research discipline that is being used to assist our understanding of human neoplastic diseases. Companion canines are a preferred animal oncology model due to spontaneous tumor development and similarity to human disease at the pathophysiological level. We use a paired RNA sequencing (RNA-Seq)/microarray analysis of a set of four normal canine lymph nodes and ten canine lymphoma fine needle aspirates to identify technical biases and variation between the technologies and convergence on biological disease pathways. Surrogate Variable Analysis (SVA) provides a formal multivariate analysis of the combined RNA-Seq/microarray data set. Applying SVA to the data allows us to decompose variation into contributions associated with transcript abundance, differences between the technology, and latent variation within each technology. A substantial and highly statistically significant component of the variation reflects transcript abundance, and RNA-Seq appeared more sensitive for detection of transcripts expressed at low levels. Latent random variation among RNA-Seq samples is also distinct in character from that impacting microarray samples. In particular, we observed variation between RNA-Seq samples that reflects transcript GC content. Platform-independent variable decomposition without  Canis familiaris, form spontaneous tumors and have a strong breed predilection towards specific types of cancer, the dog is an excellent model for identifying the genetic underpinnings associated with cancer  in humans Canine lymphoma most commonly manifests as multicentric lymphadenopathy with or without other organ involvement Next Generation Sequencing (NGS) is the next frontier for scrutinizing molecular pathology. As sequencing cost decreases and access to instrumentation increases, NGS will likely replace microarrays for gene expression analysis due to added benefits such as quantitation of transcripts, improved dynamic range, and additional capabilities for detecting expressed single nucleotide variants (SNVs), translocations, and transcript isoform switches latent variation with very different covariance. Our objective is to decompose variation among the gene expression profiles so that we can directly inspect 1) variation in transcript abundance among samples 2) differences in sensitivities of the two technologies and 3) latent variation due to each technology. Many studies have address the first two issues by gene or tag counts and assessments against PCR data, respectively combined RNA-Seq and microarray data provides for capturing latent variables within each technology. Understanding distinct technological variation is a prerequisite to examining biologically pertinent transcriptional pathways.Our data include gene expression profiles for each of ten canine lymphoma samples and four lymph nodes samples from disease free subjects using both Illumina NGS and Affymetrix microarray technology. This design is a member of an important class: multivariate observations made in batches that exhibit All studies were approved by the Institutional Animal Care and Use Committee (IACUC) at Animal Clinical Investigation (ACI) concomitant with owner consent forms.Fine needle aspirates (FNAs) were collected longitudinally from 30 dogs with lymphosarcoma as part of a study conducted by Pfizer Animal Health for the purpose of finding the maximum tolerated dose of an investigational phosphatidylinositol 3-kinase (PI3-K) inhibitor. Inclusion criteria required that at least one lymph node tumor measure >20 mm in diameter so that FNAs could be collected at 0 hr, 6 hr, and 24 hr after treatment from a single node. Samples were collected at three clinical sites under the coordination of ACI, Washington, DC. These sites were Friendship Animal Hospital, Washington, DC (sample id FS); Red Bank Veterinary Hospital, Tinton Falls, NJ (sample id RB); and New England Veterinary Oncology Group, Waltham, MA (sample id NE). After FNA collection, the 30 samples were shipped to the CLIA  accredited Clinical Reference Laboratory (CRL) for RNA isolation and genomic profiling using the GeneChip Canine Genome V2.0 Array (Affymetrix). Seven days after treatment, the change in lymph node tumor volume was assessed and used to classify responders and non-responders. Five responder and five non-responder 0 hr (pre-treatment) RNA samples previously used for GeneChip analysis were chosen to undergo additional total-RNA transcriptome sequencing using the Illumina HiSeq 2000 platform at the Translational Genomics Research Institute (TGen). In addition, a control set of four non-diseased lymph node FNAs were collected from three dogs in the Pfizer Beagle animal colony for RNA isolation and genomic profiling on both the GeneChip and HiSeq 2000. The ten lymphoma and four non-diseased lymph node samples with both GeneChip and HiSeq data are the focus of this report. Four of the lymphoma dogs had been given previous treatment, as summarized in Genome-wide expression profiles were obtained from fourteen RNA samples using both RNA-Seq and microarray. The set of samples chosen for each analysis depended on the stated goal of the analysis . For comCytology slides were prepared and Immunohistochemistry (IHC) was performed for each FNA through ACI. The cytology report was used to determine the percentage of lymphoblasts in the sample, and IHC staining for CD3, CD79a, and MDR was used to designate a T-cell, B-cell, and Multi-Drug Resistant immunophenotype, respectively .RNA from FNA was extracted and purified with the RNeasy Micro kit (Qiagen) at Clinical Research Laboratory (CRL) and in order to proceed to genome profiling a sample had to exceed the quality control (QC) requirements: RNA yield >20 ng, A260/280≥1.8, and RNA Integrity Number (RIN)≥6.0. RNA that passed QC was amplified using the WT-Ovation™ Pico RNA Amplification System . CRL carried out genomic profiling in 8 batches of 12 samples  using the GeneChip Canine Genome V2.0 Array (Affymetrix). GeneChip data was normalized to a 95% trimmed mean intensity of 500 using the MAS 5.0 algorithm within the Affymetrix Gene Console. Distributional analysis was performed using the R Affy package and image plots were generated with the AffyPLM package. Arrays with greater than 3-fold change from mean or lower than 30% present probes compared to the total number of probes on the array failed QC. In addition, visual inspection of the array images was performed to identify arrays that passed QC but had clear visual anomalies (e.g. air bubbles on the array). Re-processing occurred in instances of failure where sufficient RNA from the sample remained available. The distributional analysis and visual inspection was done for the longitudinal samples of all 30 subjects; none of the pre-treatment samples from the ten subjects used for additional RNA-Seq profiling or samples from the normal lymph nodes failed QC or required re-processing.Microarray files for the 14 samples analyzed in this report are available on GEO, accession GSE41917.NGS was performed on the same RNA that was isolated at CRL and used for microarray analysis for ten of the 30 dogs  on the Illumina HiSeq 2000. The final sample concentrations were 11.0 pM on flow cell A and 12.0 pM on flow cell C. Standard paired-end sequencing using Illumina SBS sequencing Kit reagents  occurred over 9 days with turnaround chemistry at four days post-initiation. Binary data from the sequencer was converted into plain text format using BCL Converter software  software using the default quality thresholds to discard low quality reads. This data was preprocessed and converted to standard FASTQ format containing 100-mer paired-end reads. These FASTQ files were used as inputs for the data analysis. The workflow is presented in For differential expression analysis, sequencing data from each of the 14 lanes was analyzed in parallel, using 8 processing cores per lane and 112 processing cores simultaneously on the Saguaro2 high performance computing cluster resource (jointly provided by TGen and Arizona State University). The data was aligned to the canFam2 reference genome using Bowtie Compressed raw sequencing files are provided in the Short Read Archive, accession SRA059558.Calculations were performed using the R language and environment for statistical computing and graphics Probe intensities were summarized for each probe set in each sample, and a detection call was made, using MAS5. Probe sets, Ensembl genes, and  pairs obtained from Biomart Genes that were not called present in at least three of thirteen samples were removed. The value three was chosen because the smaller, non-disease group has three samples from different dogs. We do not require presence in four samples because we consider that the two samples taken from the same dog artificially reduce the variance and we do not want to discard genes that may be present in only normal but not disease dogs. Log-transformed expression statistics,, were obtained from FPKM usingThe addition of 0.25 eliminates the singularity at FPKM = 0. We chose 0.25 because it is approximately the 5% quantile of the non-zero FPKM and because in practice it does not substantially impact the results.Genes differentially expressed between two sample groups  Surrogate Variable Analysis (SVA) RNA expression statistics section) was converted to a human gene space (Entrez identifiers) via BioMart and the R package Homovert for use with Gene Set Enrichment Analysis (GSEA) and GeneGO pathway analysis. This set is directly input into GSEA (v2.07) and run by setting the ‘collapse to gene symbol’ option to false, using ‘on-the-fly’ phenotypes corresponding to the labels for lymphoma samples versus normal samples, and permuting 1000 times on the gene set rather than the phenotype due to the small sample sizes. This is done separately for the microarray and RNA-Seq data since the intensity values are not equivalently derived. The gene set database versions 2.5 and 3.0 for all curated sets was used to show results in the expanded gene set space and persistent lymphoma hits.The set of canine genes (Ensembl identifiers) called present in at least three BvN samples by RNA-Seq and microarray . Among genes probed by both methods, percent present detection frequencies of 69% and 44% were obtained by RNA-Seq and microarray, respectively. Among genes called present using microarray over 97% were detected using RNA-Seq.Both RNA-Seq and microarray observations provide present detection calls for 15,092 genes in each of the 14 samples. The percent present detection calls provided by the two technologies agreed with high frequency (73%) and were statistically associated  SVA identified two surrogate variables. The variation associated with the surrogate variables is much larger for the RNA-Seq samples than for the GeneChip samples . The greIt is reasonable to expect that the larger dynamic range of RNA-Seq will result in larger variation. SV1 loadings are also associated with expression level and it is possible to identify genes for which RNA-Seq exhibits, with high statistical significance, an enhanced dynamic range. However, we found no clear evidence of an association of SV1 loadings with nonlinear response in the microarrays. Thus, the general impression is that this enhanced dynamic range makes a smaller contribution to the enhanced dispersion.Filtration based on the first surrogate variable allowed us to analyze the RNA-Seq and microarray expression profiles simultaneously. More precisely, by row-centering the expression matrices obtained from RNA-Seq and microarray for each sample and by removing variation associated with the first surrogate variable from RNA-Seq data only, the variation associated with the difference between technologies is substantially less than the variation associated with the differences between samples. For example, both principal component analysis  and hier10−15). Identification of differentially expressed genes based on a false discovery rate threshold (0.2) by the two methods were highly associated . The univariate analyses were repeated after filtering latent variation associated with the first surrogate variable, which impacts RNA-Seq variation. The results (not shown) are qualitatively consistent but the p-values obtained after filtering are in general more significant.RNA-Seq and microarray observations were each used to compare samples that reflect B-cell lymphoma with normal samples (BvN). Genes  detected in at least three samples using both methods were used to evaluate differential expression. Both methods detected differences between normal and B-cell lymphoma samples at the limit of the permutation test (p = 1/35) Pathway analysis was limited to the B-cell lymphoma sample set and requires a lossy conversion of canine Ensembl gene IDs to human Entrez gene IDs that results in a set of 5,733 input genes . This geGeneGo disease sets support the lymphoma diagnosis and identify immune response activation congruent with B-cell function . AdditioThe data model presented here represents both technical and biological sources of variation. Variation shared between RNA-Seq and microarray expression profiles reflects variation present in the RNA samples, while the remaining variation is associated with the technologies. The statistical treatment of the data allowed us to distinguish variation arising from these different sources. Indeed, applying SVA to our data revealed a strong correlation with known technical biases associated with RNA-Seq We used two strategies for evaluating the extent to which RNA-Seq and microarray expression profiles share variation. Perhaps the most common approach to comparing differential expression profiles across platforms is to compare the results of a collection of univariate statistical analyses, one analysis for each gene. While SVA provides the basis for a satisfactory interpretation of the combined data set, it is important to address three limitations of our use of SVA. First, we have not attempted to capture differences in dynamic range between RNA-Seq and microarray. It is reasonable to expect that the larger dynamic range of RNA-Seq will result in larger variation. We found latent variation to be highly associated with GC content but, in contrast, found no clear evidence of an association pointing to nonlinear response in the microarrays. Second, the model underlying SVA does not explicitly capture different latent variables for the different technologies. However, Our statistical procedure produced three latent sample classes. The normal samples are readily identifiable along with two distinct cancer classes. These latter two classes do not correspond to an obvious distinction in breed, age, prior chemotherapy, drug response to the PI3K inhibitor, treatment center, flow cell, lane effects, % tumor cellularity, or B- vs. T- cell lymphoma. An interesting possibility, at least in dogs, is that we have identified a therapeutically relevant lymphoma subtype that is more robust than a B- or T-cell lymphoma diagnosis. The differentially expressed gene list between the latent classes represented a general shift in the importance of cell cycle regulation. Although it is unclear what other characteristics might separate these latent groups, we note that the candidate doxorubicin resistance phenotype is only found within the E-J latent class. A larger sample set would be needed to explore the robustness of these classes and possible distinguishing features.Additionally, the sequencing depth of these genomes is sufficient to detect genomic features such as differential splicing and alternative transcripts. In general, there are two ways to approach this for an organism with a reference genome and genome annotation: either genome-reference based transcript assembly or reference-annotation based transcript assembly  require human information and were designed to operate from microarray platforms. While GeneGo does provide some functionality for canine IDs directly, the topology tool does not and thus we experienced the same lossy conversion between canine and human gene spaces when using GeneGo as with the other tools. However, it is promising that the converted canine data align with many human pathways known to be involved in human lymphoma, discussed briefly below. Although we are limited to seeing transcriptional changes in the current study, the GSEA and GeneGo expression analysis make the most of identifying context for transcriptional expression changes, while the addition of the topology tool helps identify additional areas of activity that likely direct transcriptional changes without necessarily being modified at the transcriptional level.Numerous signaling pathways implicated in human lymphoma were identified in canine lymphoma by GSEA and GeneGo pathway analysis with topologic discovery. We observed PI3K pathway activity, which was particularly relevant since the origin of these samples was a phase I trial of a novel PI3-K inhibitor in dogs with B and T-cell lymphomas. During the originating dose escalation study, the anti-tumor effects of the drug were found to be largely dose dependent, and thus statistical association between drug response and predicted levels of pathway activation at analytical baseline (prior to onset of therapy) could not be assessed. However, there is anecdotal evidence for effect in subject NE42 with B-cell lymphoma and a high score for inferred PI3K pathway activation , who achieved a partial response per RECIST criteria (data not shown). The PI3K/Akt/mTOR pathway is known to be deregulated especially in NHL (a primarily B-cell lymphoma in humans) Heliobactor pyloriThe identification of a viral disease state in the lymphoma samples  could siGSEA analysis of BvN dogs picks up a highly ranked and robust signature for doxorubicin resistance. We were also able to predict this resistance on an individual level for each dog using an internal prediction engine operating a rules-based method for assigning drug resistance phenotypes. In all but one case (RB03), this engine predicts that the dogs are insensitive to doxorubicin based on extreme expression of the ABCC1 multi-drug resistance transporter (data not shown). The engine also predicts 5-fluorouracil insensitivity (data not shown) in all but one case (NE42), though at a lower confidence than for doxorubicin. Although this study could not address the predictive power of each method, distinguishing the value of expression signature versus biomarker methods on either a cohort or individual level is an important long-term goal for the research in our lab. For human patients, identifying these resistant phenotypes offers a way to avoid undesirable side-effects by reducing the number of agents used in combination regimens for lymphoma treatment. Doxorubicin and 5-fluorouracil had been used together in the F-MACHOP regimen until the late 90s, and doxorubicin continues to be used today in standard therapies for B-cell lymphoma (either CHOP or R-CHOP) In summary, we have identified genes differentially expressed between the normal lymph nodes of disease free dogs and dogs diagnosed with B-cell lymphoma using two methods: RNA microarray and RNA-Seq. We report statistical methods for treating the combined data from these technologies as a means for collaboratively supporting biological discovery from the different platforms. Together, the data suggest the elevated expression of genes in the PI3K signaling pathway in B-cell lymphoma, a finding that has also been noted in human lymphoma. Since dogs form tumors spontaneously and have a strong breed predilection towards specific types of cancer, the dog is an excellent translational model for identifying the genetic basis associated with lymphoma.Figure S1Sequencing Workflow. A sequencing workflow based on the Formalwear Suite of Bowtie-Tophat-Cufflinks was employed to examine differential expression from the sequencing data.(PNG)Click here for additional data file.Table S1Summary of canine subjects. Age is given in years. Gender is provided as male (M), male-castrated (M/C), female (F), female-spade (F/S). Relapsed dogs received chemotherapy treatment prior to FNA collection for this study. The immunophenotype includes the multi-drug resistant (MDR) status as positive (pos) or negative (neg). Best RESICT  score is given of: complete response (CR), partial response (PR), stable disease (SD), or progressive disease (PD).(PDF)Click here for additional data file.Table S2Top 20 GSEA gene sets. Supporting gene set enrichments from the top 20 sets by GSEA using the version 2.5 and 3.0 gene set databases (Sets referring to B-cell biology highlighted). The three B-cell maps shown in version 2.5 but not version 3.0 are still represented in the 3.0 enrichment list, but at a lower rank. Namely, BIOCARTA_BCR_PATHWAY is rank 22, ST_B_CELL_ANTIGEN_RECEPTOR is rank 27, and SIG_PIP3_SIGNALING_IN_B_LYMPHOCYTES is rank 33.(PDF)Click here for additional data file.Data File S1Overlapping Gene Set Used for Pathway Analysis.(XLSX)Click here for additional data file.Data File S2GSEA Results Files.(ZIP)Click here for additional data file.Data File S3GeneGo Reports.(ZIP)Click here for additional data file.Data File S4Pivot Table of Original FPKM Values.(ZIP)Click here for additional data file.


Normalization of gene expression data has been studied for many years and various strategies have been formulated to deal with various types of data. Most normalization algorithms rely on the assumption that the number of up-regulated genes and the number of down-regulated genes are roughly the same. However, the well-known Golden Spike experiment presents a unique situation in which differentially regulated genes are biased toward one direction, thereby challenging the conclusions of previous bench mark studies.This study proposes two novel approaches, KDL and KDQ, based on kernel density estimation to improve upon the basic idea of invariant set selection. The key concept is to provide various importance scores to data points on the MA plot according to their proximity to the cluster of the null genes under the assumption that null genes are more densely distributed than those that are differentially regulated. The comparison is demonstrated in the Golden Spike experiment as well as with simulation data using the ROC curves and compression rates. KDL and KDQ in combination with GCRMA provided the best performance among all approaches.This study determined that methods based on invariant sets are better able to resolve the problem of asymmetry. Normalization, either before or after expression summary for probesets, improves performance to a similar degree. The normalization of data is a crucial step in the analysis of microarray data. The main purpose is the removal of systematic variations, while preserving biological variations of interest. Most of the algorithms that have been proposed and utilized over the years are based on reasonable assumptions that are generally true for real data and large scale studies.Quantile normalization  and loesThe aforementioned assumptions could be seriously biased under certain conditions. One well accepted condition is that a moderately sized group of genes is enhanced or suppressed in the same direction, invalidating the common assumption of equal up- and down-regulation ,5, as ocAnother group of normalization methods is based on invariant sets, which are selected as probes that are not differentially expressed across conditions. An invariant set is used to form the standard curve for intensity based normalization ,10. InvaThe computational derivation of invariant sets can be traced back to the work of Li and Wong . This apNi et al. moved away from rank-based methods, identfying an invariant set using a two-step kernel method that is not sensitive to asymmetry . HoweverIn general, normalization methods based on invariant sets are performed in three-steps: selection, curve fitting, and scoring. Usually selection is based on heuristic criteria determining the cutoff for inclusion or exclusion of probes in an invariant set. This step is essential if this method is to outperform regular global normalization methods. In the second step, normalization curves are fit, such that the intensity-based patterns can be described using only the probes included in the selected invariant set. In the third scoring step, the fitting results from the second step are extended to the probes not included in the selected invariant set. The range of probe intensity in the selected invariant set does not necessarily cover the entire range of intensities among all of the probes. Scoring the data lying outside the range of the invariant set is a statistical issue.This study proposes a Kernel Density weighted Loess (KDL) method, adopting a concept similar to the detection of invariant sets through the estimation of density. KDL methods are carefully configured for single color arrays, particularly on the Affymetrix platform. However, there are no specific settings for the format of arrays; therefore, it is well suited to fitting the data of two-color arrays. Although Ni et al. estimated density based only on M signals, our approach employs two-dimensional kernel density estimation for both the M and A, simultaneously. This eliminates the need to bin the data or introduce an iterative procedure. Our proposed approach was first implemented as the soft criterion described in Section 5.2 to improve general loess normalization. The estimation of density provides different weights for each data point in the subsequent loess normalization. The estimation of density can also be used to select invariant genes, as described in Section 5.3, thereby integrating the idea of invariant sets with quantile normalization.Normalization should generate data that reflects true variations in gene expression, to ensure that the subsequent step of detecting differentially expressed genes functions correctly and returns a good estimate of fold change. In Section 2.2, we compare the detection power of popular normalization methods such as loess, quantile, and invariant set based methods, including dChip, GRSN, and the proposed kernel density weighted loess (KDL) and kernel density based quantile (KDQ) normalization methods. Performance was assessed using ROC curves applied to the Golden Spike experiment and simulation data. The compression rate between the expected fold change and the observed fold change is investigated in Section 2.3. The proposed methods provide the largest area under the ROC curves and the lowest compression rate.The Golden Spike experiment  consistsa priori whether a gene has shut down completely. However, the existence of "empty genes" might be the result of either non-functioning genes or bad probes. As far as we know, there is no specific methodology for automatically removing empty probes. A number of studies have suggested removing known empty sets when comparing methodologies using Golden Spike data [The MA plot in Figure ike data ,20, and The heterogeneity between the two groups of null genes appears to be well resolved in Figure The second issue in Figure Figure According to the above observations, the proposed KDL method captures the correct "invariant set" throughout the range of all intensity levels, outperforming the other methods. The major difference is the result of the bipartite pattern of null genes, comprising both 1× spike-in genes and empty genes. These two groups appear entirely different, making it impossible to merge them as one with any normalization methods, except for KDL and KDQ. Because the empty genes are highly artificial, they are unlikely to form a large set in real world data; however, the robustness of KDL and KDQ with respect to special groups of genes can greatly reduce the concern of slightly enhanced influential data points.Because the expected fold changes for spiked-in transcripts are known, the tradeoff between false positive and false negative detection can be compared using ROC curves. This study compares some of the most popular combinations of preprocessing, probe set summarization, and detection methods to understand the advantages of adopting approaches based on invariant sets. GCRMA is used for background correction in most of the normalization methods except for dChip; therefore, we selected median polish for our summarization step. The normalization step can be applied prior to probe set summarization using KDL, KDQ, or global quantile normalization, which is the default of GCRMA. The Li-Wong model works with the invariant set in dChip as a whole. Because one of the methods we compare is GRSN, and it is implemented after probe set summarization, we also applied the proposed KDL after probe set summarization. The final step is to make inference on differentially expressed genes. We demonstrated this using both fold change and a t-test.As shown in Figure When the Golden Spike experiment was first published , post suThe results in Figure ROC curves are used to compare the performance of different normalization methods in terms of both sensitivity and specificity from the perspective of ranking. Accurate estimation of true expression level is also desirable. Figure We have demonstrated that KDL and KDQ are both able to improve the bias caused by asymmetry between the number of up- and down-regulated genes. However, it is important to determine at what cost this gain in performance is achieved, under conditions other than those specific to this scenario. Hence, we conducted a simulation study to discover whether the proposed methods could achieve the same level of performance with symmetrical expression change. We also evaluated the effects of altering parameters through simulation.http://www.ncbi.nlm.nih.gov/geo/ accession number GSE5788 [The simulation was based on the study by Gadbury et al., in whic GSE5788 , for the1. Effect size distribution: We first summarized the 14 arrays with RMA  and then2. We randomly separated the eight control samples into two groups of four samples each, assigning one group to be a simulated control group and the other the simulated treatment group.3. We randomly selected a proportion of probe sets as simulated significant genes. We used 10%, 5%, and 1% of the total number of genes to generate results.4. We randomly selected another set with the same number of probe sets and recorded the t-statistics derived in Step 1, as the simulated effect sizes.i of the ith probe at the log scale across the simulated control samples. We then added the number Sixdi to the log of the ith probe of the simulated treatment samples, where di was the ith simulated effect size selected in Step 4. We took this as the exponent of 2 to set the expression level back to the original scale.5. For each probe set selected in Step 3, we calculated the standard deviation SAll of the methods were then compared with ROC curves using the probe level data simulated above. The results are presented in Additional files Taking the absolute value of the effect sizes selected in Step 4 generates data with asymmetrical effects similar to those in the Golden Spike experiment. We compared all of the methods based on invariant sets with the ROC curves in Figure To better understand the influence of the tuning parameters on either method, we compared the results with the same simulation data using a variety of settings. The data was simulated with either 1% or 10% significant genes, and all treatment effects were positive. We first assessed the power of the kernel density used in the weighting scheme of KDL. Additional files The design of the Golden Spike experiment provided a good opportunity to review existing normalization methods and determine how they handle asymmetric changes in expression levels. Although it is an extreme design, the information provided is exceptional, compared to other spike-in studies.This study demonstrated the superiority of methods based on invariant sets, compared to global normalization methods for data with an asymmetric expression structure. The two proposed normalization strategies based on kernel density, KDL and KDQ, improve the popular normalization methods, loess and quantile normalization, at either the probe-level or post-summary level. Both strategies can be integrated with any procedures containing an independent normalization step.As with most normalization strategies, a number of parameters in the proposed method require a degree of tuning. As with loess normalization, the smoothing parameter was set to 0.2 in this study. Local structure can be captured using smaller values. In addition, we used density estimates to the power of four as weights in the loess normalization for KDL, although this could certainly be set higher or lower to put more or less emphasis on the data in the middle of the major trend. This multiplicity plays an important role when asymmetry is strong.KDL is applied to the entire data set with a weighting scheme while KDQ adopts an approach similar to that of other invariant-set normalization methods. This is accomplished by initially selecting a subset of data to be normalized and extending the normalization process to the other data not included in the selected subset. The selection of subset is based on kernel density estimation as well as knowledge related to the proportion of genes expected to be differentially expressed across arrays. Selecting the proportion of genes in the training subset should be based on a thorough understanding of the data. According to our empirical study and simulation, a conservative proportion would prevent the inclusion of any variable genes, particularly in situation involving a high degree of asymmetry. In other respects, KDQ is quite robust with regard to this parameter.In the above comparison with the Golden Spike experiment and simulation data, we found that KDQ performed slightly better than KDL in probe level normalization, although both of these methods produced similar results. Nevertheless, the soft weighting scheme of KDL automatically included all of the data points in the analysis, while KDQ relied on extrapolation to normalize data points outside the range of the invariant set. Hence, if special patterns are observed in the intensity-based curvature across arrays at the two ends of the data, it is preferable to adopt KDL over KDQ.The steps of preprocessing and expression summary play an important role in determining the accuracy of estimation. The statistics for the detection of differentially expressed genes further improves the retrieval of correct signals from noise. This study does not compare all possible combinations of analysis strategies to provide the best suggestion, as this would be impractical. Rather, we aimed to demonstrate the importance of dealing with asymmetric patterns in the normalization step and how methods based on invariant sets are better able to improve existing methods. If the characteristics of the data are accurately detected and dealt with, statistical testing for inference will proceed far more smoothly. The proposed KDL and KDQ methods are presented as improved substitutes for both loess and quantile normalization in general, and more specifically for integration with GCRMA.The proposed KDL approach is a simple strategy to improve the accuracy of GCRMA in estimation. Based on our results using t-test and fold change, it is clear that detection power is enhanced; therefore, it is highly recommended for the routine practice of microarray data analysis. Both KDL and KDQ are implemented in JMP Genomics Version 5.0.http://www2.ccr.buffalo.edu/halfon/spike/spikedownloads.html. The Golden Spike experiment was the first, and one of the few entirely controlled experiments with known transcript levels for every gene in the array. The experiment consists of three replicates for the S group and three replicates for the C group using DrosGenome1 GeneChip. The S group includes 1331 spiked-in transcripts with higher concentrations than C group, and 2535 spiked-in transcripts of equal concentration in the two groups. The other 10144 probe sets in the arrays are empty sets that do not target any spiked-in sequences.To demonstrate the strength of the proposed methods, we used a dataset with spiked-in transcripts of known concentrations. The Golden Spike experiment  has beenCriticism of this data generally relates to the confounding effects between transcript level and fold change, (assigning larger fold changes to higher intensity probes), an unfair experimental design, asymmetry in gene effects toward up-regulation and the existence of an unusually large empty set ,18,21. Hf  be the joint density function of the bivariate random variable , and let , i = 1, ⋯, n be a sample of size n drawn from this distribution. The kernel density estimate of f  based on this sample isThe kernel method is a non-parametric technique for the estimation of density. Let X h> 0 and Y h> 0 are the bandwidths and ϕ is the standard normal density.where Ni et al. proposed a kernel density based method, fitting kernel density to the M values restricted to points within a particular range of A values, where M stands for the difference between the two compared samples and A stands for the mean intensity of the two samples. The mode of one-dimensional kernel estimation is used to represent the bias of effect size on non-differentially expressed genes within each region of A, and the normalization curve is derived by connecting the modes across various regions of A with smoothing splines. This two-step approach requires a number of empirical parameters with which to select modes.Unlike the method of Ni et al., our approach measures density in two dimensions (M and A) jointly, and loess normalization is used to fit polynomial curves locally with the data. Points proximal to one another along the x axis jointly determine the main trend of the curve. KDL simply assigns different weights to the data points according to the estimated kernel density when fitting the loess curve. This paper applied the estimated kernel density to the power of 4, as the weight of the corresponding probe. Genes deviating from the major group, which consists of null genes, are down-weighted. The weighting strategy of KDL benefits from not having to establish any hard decisions in the selection of the invariant set, while allowing the normalization process to rely heavily on probe intensities that remain consistent across arrays. Because all of the probes are included throughout the normalization process, the common issue of having to extrapolate scores for methods based on invariant sets does not exist in KDL.The only requirement of KDL is that null genes be distributed more closely than others. This is true in most empirical studies, but the Golden Spike experiment violates this assumption by including two distinct null sets, 1× genes and empty genes. These genes do not perform like a single group, and therefore distort most of the normalization strategies. We will discuss the results of both including and excluding empty genes.The SAS procedures, PROC KDE and PROC LOESS , are used for estimating kernel density and loess model fitting. First, the average of all arrays is taken as a common reference. Each array is normalized against the reference by fitting the weighted loess curve to the MA plot. The A dimension is employed as reference data, rather than using the mean between the target array and reference array. This slight difference provides an enormous advantage in resolving heterogeneity in data, as shown in Section 2.1.Kernel density estimation can also be used to select invariant sets, which should include the null genes. We integrated this idea with quantile normalization. The global normalization strategy associated with quantile normalization can be adjusted for asymmetric data. The proposed steps are as follows.We first apply kernel density estimation to the MA plot between each array and the reference array. The reference array can be objectively selected or set to the average of all of the arrays. As with KDL, we assign the common average among all arrays as the A component. The M component represents the difference between each individual array and the reference array. The estimation of density is a score showing the relative importance of each data point in the plot. Probes with greater importance scores are included in the invariant set. Similar to other invariant set selection methods, there are always a number of thresholds that must be determined. KDQ requires prior knowledge with which to establish the proportion of data included in the invariant set. It is recommended that this proportion be set lower than but close to the expected proportion of probes without differential expression across arrays. For the Golden Spike experiment, we set this proportion to 85% and 50% including empty null genes and excluding empty null genes, respectively. This tuning parameter is sensitive and data dependent.The second step involves conducting the general quantile normalization procedure on the invariant set for all of the arrays.n probes in the invariant set are x1, x2, ..., xn. Their order statistics are x(1), x(2), ..., x(n). The corresponding normalized intensities from Step 2 are y(1), y(2), ..., y(n) ordered from smallest to largest. When considering a probe that is not in the invariant set with intensity x falling between x(i) and x(i+1), the interpolated value will be y(i)+(x-x(i))x(y(i+1)-y(i))/(x(i+1)-x(i)). If x is greater than x(n), extrapolation will depend on the highest m values of the invariant set. In this case, m is set to 100. Let xa be the average of x(n-99), x(n-98), ..., x(n) and ya be the average of y(n-99), y(n-98), ..., y(n). The normalized value for x will be ya+(x-xa). This is only adjusted for the average shift between the original and the normalized data of the rightmost 100 observations in the invariant set.Data not included in the invariant set has to be scored relative to the invariant set. For data within the range of the invariant set, linear interpolation is applied within each array. For data outside the range of the invariant set, linear extrapolation is adopted, based on a small set of data at the boundary of the invariant set. To clarify this point, we will illustrate it with an example. Assume that the intensities of Our proposed normalization based on kernel density can be applied to probe level data as well as probe set level data, and these have been summarized for each probe set in Affymetrix GeneChip data. The implementation of post summary normalization is straightforward, as described in Sections 5.2 and 5.3. For probe level data, a small twist is made. We first take the median of each probe set and apply normalization to the medians. The probe level data is then reconstructed by adding back the corresponding difference of each probe relative to the median.http://www.stat.nthu.edu.tw/~wphsieh/KD.htm.Both KDL and KDQ were implemented in SAS 9.2  and integrated into JMP Genomics Version 5.0 . The SAS codes and corresponding SAS dataset are available at: WPH, TMC and RDW conceived the project. TMC implemented KDL and KDQ. WPH and YML did the comparison and simulation. WPH, TMC, YML and RDW drafted the manuscript. All authors read and approved the final manuscript.Figure S1 - MA plots for probe-level normalization using data with the empty genes. (a) Data are normalized with quantile normalization and summarized with the median of each probe set. (b) Data are normalized with loess normalization and summarized with the median of each probe set. (c) Data are first background-corrected with GCRMA and normalized with quantile normalization. The expression summary is then computed using median polish. (d) Data are first background-corrected with GCRMA and normalized with loess normalization. The expression summary is then computed using median polish. The blue points are empty genes, the red points are the 1× genes and the green ones are spiked with higher concentration in the S group than in the C group.Click here for fileFigure S2 - MA plots for invariant set based normalization methods using data with the empty genes. (a) Data are background-corrected with GCRMA and normalized at the probe level with KDL. Median polish is used to summarize the probe set expression level. (b) The same as (a) while replacing the normalization with KDQ. (c) Data are background-corrected with GCRMA and normalized at the probe level with quantile. Median polish is used to summarize the probe set expression level. The data are then normalized again at the post summary level with GRSN. (d) Li and Wong's dChip method implemented in R.Click here for fileFigure S3 - ROC curves with the empty sets. For the probe level data, quantile normalization is substituted with either KDL or KDQ in the workflow of GCRMA. They are presented in red and black respectively. The original GCRMA is in light blue. The upper panel uses T-test and the lower panel uses Fold Change as the criterion to select differentially expressed genes. They both report similar orders of performance. The two post summary normalizations, KDL and GRSN, are presented in pink and green respectively.Click here for fileFigure S4 - Scatter plots for the probe data after GCRMA background correction. The vertical and horizontal lines at left bottom coner of each plot indicate certain truncation applied for the low-end intensities.Click here for fileFigure S5 - ROC curves related to dChip with empty genes. The ROC curve derived from dChip is in blue. GRSN and KDL are then applied on the dChip data as the second normalization and are shown as green and red curves, respectively. The upper panel uses T-test and the lower panel uses Fold Change as the criterion to select differentially expressed genes.Click here for fileFigure S6 - ROC curves on simulation data with symmetric expression change. The data was simulated with 1% of significant genes as described in the context. The treatment effects could be either positive or negative. The curves for GCRMA, GCRMA + post GRSN and GCRMA + post KDL are closely overlapped.Click here for fileFigure S7 - ROC curves on simulation data with symmetric expression change. The data was simulated with 5% of significant genes as described in the context. The treatment effects could be either positive or negative. The curves for GCRMA, GCRMA + post GRSN and GCRMA + post KDL are closely overlapped.Click here for fileFigure S8 - ROC curves on simulation data with symmetric expression change. The data was simulated with 10% of significant genes as described in the context. The treatment effects could be either positive or negative. The curves for GCRMA, GCRMA + post GRSN and GCRMA + post KDL are closely overlapped.Click here for fileFigure S9 - ROC curves on simulation data with asymmetric expression change. The data was simulated with 1% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S10 - ROC curves on simulation data with asymmetric expression change . The data was simulated with 5% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S11 - GCRMA with KDL on asymmetrical data generated with 1% of significant genes. The data was simulated with 1% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S12 - GCRMA with KDL on asymmetrical data generated with 10% of significant genes. The data was simulated with 10% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S13 - GCRMA + post KDL on asymmetrical data generated with 1% of significant genes. The data was simulated with 1% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S14 - GCRMA + post KDL on asymmetrical data generated with 10% of significant genes. The data were simulated with 10% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S15 - GCRMA with KDQ on asymmetrical data generated with 1% of significant genes. The data was simulated with 1% of significant genes as described in the context. All the treatment effects were positive.Click here for fileFigure S16 - GCRMA with KDQ on asymmetrical data generated with 10% of significant genes. The data was simulated with 10% of significant genes as described in the context. All the treatment effects were positive.Click here for file