Multivariate ordination methods are powerful tools for the exploration of complex data structures present in microarray data. These methods have several advantages compared to common gene-by-gene approaches. However, due to their exploratory nature, multivariate ordination methods do not allow direct statistical testing of the stability of genes.In this study, we developed a computationally efficient algorithm for: i) the assessment of the significance of gene contributions and ii) the identification of sample outliers in multivariate analysis of microarray data. The approach is based on the use of resampling methods including bootstrapping and jackknifing. A statistical package of R functions was developed. This package includes tools for both inferring the statistical significance of gene contributions and identifying outliers among samples.The methodology was successfully applied to three published data sets with varying levels of signal intensities. Its relevance was compared with alternative methods. Overall, it proved to be particularly effective for the evaluation of the stability of microarray data. Ordination methods are useful tools for the analysis of gene expression microarrays. Principal component analysis (PCA) and correspondence analysis (CA) have both been used to extract the main sources of variation present in highly multivariate microarray data ,2. The sConstrained ordination methods are very efficient for sample classification and class prediction. They are flexible and can be used easily to identify groups of genes associated with classes of samples. Geometrical interpretations are generally required to investigate the gene-sample relationship. Genes of interest can also be ranked according to their discriminative power. However, considering the exploratory nature of these methods, it is not trivial to assess the significance of a given gene dysregulation in a multivariate setting. These methods rely on solving an eigenvalue problem whose solutions are given by the leading eigenvectors and whose theoretical statistical properties are particularly complex to study. To overcome this issue, resampling techniques have been proposed to estimate the stability of multivariate analyses. These techniques were described in a variety of scientific frameworks including environmetrics ,6, chemoIn the present work, we propose a specific methodology for testing the stability of constrained ordination methods applied to microarray data. Unlike previous studies, our approach is dedicated to supervised multivariate analyses. To our knowledge, very few studies addressed the issue of stability assessment in supervised multivariate analyses. The potential of associating stability analysis in the supervised multidimensional context is multiple. By using the information of sample descriptors, genes can be associated with a given class of samples and the significance of this association can be assessed. A derived significance testing strategy regarding gene contributions is proposed. Further resampling methods based on jackknifing are also proposed to identify influential observations as an aid in outlier detection in microarray data sets. A comprehensive set of R functions illustrating our methodology was developed. The package is freely available on request.The present manuscript is organized as follows. The first section introduces some theoretical aspects of ordination methods (with a particular focus on CA) and constrained ordination methods . The subsequent sections describe the different resampling strategies used in this project, as well as details about the algorithm. Illustrative examples demonstrating the implemented technique are given.Both PCA and CA are commonly used in microarray data analysis. Some authors stressed that CA has several advantages over PCA ,12. LikeLet us define the following:Y: the (n × m) matrix of gene expression data • P = Y/N: the data matrix divided by its grand total• r: the n-dim vector of row sums of P• c: the m-dim vector of column sums of P• Dr = diag(r): the diagonal matrix of row sums• Dc = diag(c): the diagonal matrix of column sums• Y is obtained by carrying out the singular value decomposition of the doubled centered and standardized matrix Z:The correspondence analysis of Λ the k × k (k = rank(Z)) diagonal matrix of singular values associated with Z with λ1 ≥ ... ≥ λk > 0, U an (n × k) matrix whose columns are the left singular vectors of Z and V an (m × k) matrix whose columns are the right singular vectors of Z. The rows of U and V are orthonormal with respect to Dr and Dc respectively:with U and UΛ. The principal axes and column coordinates are respectively given by V and VΛ.The principal components and row coordinates are respectively given by Y containing the gene expression values, additional descriptive variables X controlled in the experimental design generally characterize samples. Constrained ordination methods aim to display the variations in the data which are explained by the descriptive variables. These two-table methods are dissymmetric because the information from X is used to constrain the analysis of Y. Correspondence analysis with respect to instrumental variables (CAIV) [Y given the class descriptor x is simply the correspondence analysis of the table Ŷ corresponding to the table of means per group. BGA is the analysis of the per-class centroids. It provides up to g - 1 discriminating axes (g is the number of classes). The initial samples are thereafter projected as supplementary rows in the BGA sub-space. The BGA procedure provides the best linear combination of variables which maximizes the between-group variance.In microarray experiments, besides the main table s (CAIV) , which is (CAIV) , and bets (CAIV)  are two s (CAIV) , diseases (CAIV) , and rems (CAIV) . BGA is y) onto the vector of the class centroid (x)  table Ŷ corresponding to the per-class mean of Y. Let us define X, the (n × g) table of dummy variables coded from x and Ŷx = X(XTX)-1XTY, the (n × m) matrix of fitted values. Bootstrapped samples are built based on the residuals E = Y - Ŷx. Residuals are sampled with replacement (E*) and new data sets are built as follow: Y* = Ŷx + E*.Introduced by Efron , bootstrz-scores were defined for each gene as the ratio of the bootstrap estimates to its standard deviation. p-values were estimated according to the "bootstrapped eigenvector" procedure [The analysis of 100 to 1000 perturbated data sets is generally required to assess the parameters' distribution of a multivariate model . Out of these empirical distributions several indicators of the stability of gene contributions were calculated. Non-parametric 95% bootstrap confidence intervals were constructed using the 2.5% and 97.5% quantiles of the bootstrapped distribution. rocedure  as the pConvex hulls were used to graphically display the spread of the bootstrapped gene coordinates on the dominant principal axes. The relative inertia of these gene coordinates was measured by the ratio of gene inertia to the total inertia.a posteriori projections of the resampled elements on the original reference subspace provided by the analysis of the initial data set. Total bootstrap performs a new analysis on each of the resampled data sets. Both strategies have been implemented in the current project. It is noteworthy that partial bootstrap does not involve successive steps of singular value decomposition which makes it considerably faster than total bootstrap. Moreover, because total bootstrap requires a complete CA to be carried out for each bootstrapped table, the new sets of row and column coordinates belong to different subspaces which make their comparison more complex. Unlike partial bootstrap, total bootstrap is potentially subjected to axis reflection or inversion [Lebart  proposednversion . Signs onversion ,20,21. TJackknifing is another resampling technique introduced by Quenouille  and laten samples, n consecutive analyses are performed providing n sets of n - 1 sample coordinates. The impact of each individual sample on the other n - 1 samples is measured by the distance from the samples' original positions to their positions after resampling. If a given sample is highly influential, it may importantly impact the position of one or several other samples. A stability plot can be used to visualize the shift in the sample position after jackknifing. A large shift reflects the presence of an influential observation.In a data set including D2) was used. These distances can be evaluated using a χ2 distribution with the appropriate degrees of freedom. Each time a sample removal induced a shift to an extra sample p degrees of freedom, the sample was defined as significantly influential towards the extra sample. Overall, if the median of the n - 1 shifts induced by an observation is greater than a In order to identify observations which significantly influenced the position of other observations, the classical multivariate detection of outliers based on the Mahalanobis squared distance , and a Three publicly available data sets were used to illustrate the different features of our methodology. The first data set consists of a subset of data from the pioneer work of Bhattacharjee and colleagues  using gep-value ≥ 0.05). The proportion of overly unstable genes is refered to as false positive rate.Five hundred bootstrapped samples were built in order to assess the empirical distribution of the gene contributions within each data set. Partial bootstrap was used and the newly calculated gene coordinates were represented with convex hulls on the factorial map defined by the two first principal axes of the BGA . Thus, the false positive rate was 0%.Gene hulls in Bhattacharjee were small (the average relative inertia of the 100 most discriminating genes was 0.01), distant from the center of the plot and the group of genes associated with each factor level did not overlap. This documents the stability and specificity of the markers found in this experiment. This result was further confirmed by the boxplot representation showing the distributions of gene contributions . In the data set of Baty, the degree of hull overlapping was particularly high. The level of false positive rate was moderate in Spira data (8%) and high in Baty data (32%). This level differed from one experimental condition to one another. In Spira, the "Former smoker" group had a higher false positives rate (21%) compared to "Current smoker" (0%) and "Never smoker" (1%). The "Current smoker" category was the one with the most stable gene signals. In Baty, the false positive rate was high. Genes associated with the consumption of water were highly unstable . Overall the false positive rate was measured for the 100 genes with the highest ranking in terms of gene contribution. As to be expected, this rate increased with lower gene ranks.n - 1 positions of each sample were represented with star plots showing the shifts induced by sample removals. In Bhattacharjee data, the relative shifts induced by jackknifing were small  among the genes which mostly participate to the between group discrimination were selected. Hundred and twelve genes with the highest contribution on the first axis in bootstrapped CA were selected. A third list of genes included the 100 most significant genes obtained by ANOVA models. There were 18 genes overlapping between the list from bootstrapped CA and bootstrapped BGA, 27 genes beween bootstrapped BGA and ANOVA and 7 between bootstrapped CA and ANOVA. A functional analysis was carried out on these 3 lists of genes using the web tool DAVID [p-values was slightly more diverse in bootstrapped BGA (n = 70) than in bootstrapped CA or ANOVA (respectively n = 52 and n = 55). On the other hand, the enrichment of biologically informative genes in each GO categories was higher in bootstrapped BGA than in bootstrapped CA and ANOVA. This is in agreement with previously reported findings [The relevance of the results obtained by bootstrapped BGA was compared with those obtained by bootstrapped correspondence analysis  and the ol DAVID . Table 1findings ,11.The advantage of using multivariate versus univariate approaches in highly multivariate data such as microarray data has been already well documented in the literature. The extraction of meaningful gene mechanisms implies that genes are treated as a whole and not separately. This explains why higher enrichment of biologically meaningful GO categories  are generally obtained when using multivariate approaches compared with univariate approaches. The choice of using unsupervised or supervised ordination methods mainly depends on the objectives of the study. When the biologist is interested in finding groups of discriminating genes that explain differences among well-defined patients categories, BGA is a method which should be considered. Because the grouping information is directly incorporated in the BGA model, the dimension reduction of the multivariate data is driven by the phenotypic information. This generally provides noticeable simplification of data interpretation. In contrast, when using unconstrained CA, one first extracts the major compositional variation present in the leading axes then relates this variation to external information. In certain situations, the leading axes are subject to unexpected sources of variation, making their interpretation difficult.When the design of experiment tends to become more complex, , other multivariate approaches can be used to incorporate this external information into a constrained CA model (see e.g. ). Doublea priori knowledge about the strength of their transcriptomics signal. The biologically most well-defined data set was Bhattacharjee with four clearly distinct categories of patients and samples taken from distinct types of lung tumors. The class discrimination was high  and the gene signal very stable and specific. A less stable signal came out of Spira data, where samples were derived from a mixture of airway epithelial cells and the patient groups less clear cut. The identification of stable signals specific to "Former smoker" was particularly difficult although the between-class discrimination was significant . The weakest and highly unstable signals were found in Baty data, where the expected effect was within the physiological range of normal cells. The proportion of explained variance was very low in this example .The bootstrapping technique presented in this study proved to be very useful to evaluate gene stability in microarray data. The three data sets were chosen with an Jackknifing was particularly efficient to detect influential observations or outliers in our setting. This method provided important diagnostic insights in the data as well as the experimental design. A careful exploration of the sample stability can help the experimenter to identify samples with imprecise or wrong group allocation, or group with a heterogeneous behaviour. Furthermore, this method can be used to identify poorly specified sample categories or subcategories of samples. As an example, further investigations might reveal that the "Former smoker" category in Spira might share gene signatures both from "Never smoker" and "Current smoker". With the proposed tools, researchers can identify inconsistent observations/samples or groups and have a strategy at hand to correct for imprecise descriptions in case of sufficient respective evidence.Particular attention was paid to the computational aspects of the resampling calculations. The calculations were considerably accelerated by the use of parallelization. Furthermore, routines used to carry out CA and BGA  have been optimized for the analysis of gene expression data where the number of variables far exceeds the number of samples. Performance testing on data sets of different size showed an improvement of calculation time by a factor 10 to 50. As previously mentioned, the method of partial bootstrap was prefered to total bootstrap for testing the stability of gene contributions since it was simpler to perform and computationally more efficient (approximately twice as fast).a priori sample classification.Dimension reduction methods are powerful tools that help biologists exploring their data and generate new hypotheses. Like other supervised approaches, constrained ordination methods incorporate external information that greatly simplifies the interpretation of microarray data analysis. The principal axes of BGA being defined as the linear combination of genes that maximizes the between-group variance, it is straightforward to extract groups of genes that discriminate between disease categories. By using the resampling methodology described above, it is possible to assess the reliability of solutions in a multivariate analysis of gene expression microarray data. Although both bootstrapping and jackknifing should not be used for formal statistical hypothesis testing, they proved to be useful to identify highly consistent genes, filter out some false positive genes, and to allow detection of influential observations among samples. With regard to this complementary information, the biologist can decide to pay more attention to highly stable discriminating genes, which in turn can be used for subsequent formal statistical hypothesis testing. Based on jackknifing information, the biologist can also decide on croping outlying observations or refining In conclusion, the methodology and the collection of tools proposed in this study are suitable for the assessment of the significance of gene contributions and the detection of outliers in microarray data and this in a multivariate fashion. The set of R functions includes additional functions which test the stability of multivariate analysis results. Overall, the R package we developed constitutes a novel and comprehensive suite of diagnostic tools to evaluate the robustness of multivariate representations of high-throughput gene expression data.FB developed the algorithm, performed the analysis and wrote the paper. DJ and FP contributed to the implementation and the optimization of the algorithm. MMS contributed in the development of the algorithm. MHB supervised the work and was significantly involved in the writing of the manuscript. All co-authors read and approved the manuscript.


BioConductor  is a colRecent studies have shown that alternative splicing is prevalent—approximately 74% of all human multi-exon genes are predicted to be alternatively spliced  arrays, such as the HGU133plus2 chip. Many changes have been made, including the removal of the MM probes, a reduction in the number of PM probes in each probeset from 11 to 4, changes in array design, and changes to the protocols used for RNA preparation. Given the large number of changes, it is important to assess the performance of the arrays. In   the data in order to bring them together. Many techniques exist, most working on the assumption that on average, the majority of data points between samples are unchanged. Thus, a straightforward procedure might simply scale each array to the same mean intensity, perhaps after removing outliers. A more invasive approach might also require each array to have the same standard deviation, or to have the same-shaped distributions. Normalization can be performed on the raw feature/spot levels, or on the probeset data after expression summary. Most recent algorithms perform normalization first, and the same techniques that were applied to 3′IVT arrays are applicable to exon arrays .Expression summary is the process by which the values for each individual probe in a probeset are summarised to generate a single value for that probeset. Again, many techniques exist, but all perform some kind of weighted average, possibly with background correction.Observations about the relative utility of MM probes led to the design of RMA . Importaij = ie + ja + ijɛ, where e is a chip effect and a represents the probe affinity for the thj probe on the thi array.After normalization , the data are fitted to a global model of expression and probe affinities. For each probeset, PMe), the “stickiness” of the probe , and a measurement error (ɛ). RMA works by estimating, for each probeset, values for the probe and chip effects that would result in the pattern of PM values observed in the data. This process of “model fitting” is performed using an algorithm called median polish, a fast numeric technique for estimating model parameters.The model is based on the hypothesis that the intensity measured for a perfect match spot is dependent on three things: the amount of material available to bind to the spot , have at-test) are used to assess how likely changes in differential expression are to have occurred by chance. With a single test, a p-score of, say, 0.05, might be acceptable; there is a 5% chance of chasing shadows. Microarrays present a challenge, because they perform thousands (or in the case of exon arrays ∼1.4 million) tests in parallel. At standard thresholds, many probesets will be identified by chance . Multiple testing correction aims to address this by choosing more stringent thresholds in order to reduce the false positive rate. This must be balanced against the incorrect rejection of real results—i.e., the false negative rate.Statistical tests . FDR is appealing not least because it is easy to interpret and provides a useful measure of the reliability of the dataset in question.Exon arrays present further challenges, because of the size of the datasets and because the non-independence between probesets makes it difficult to accurately compute the FDR . That saA number of strategies have been developed that re-annotate array data by in silico searches against the genome and/or transcriptome. A popular approach is to identify transcripts (or genes) targeted by multiple probesets. These are then combined to produce larger consensus probesets ,20. One Care must, however, be taken when applying these approaches to exon arrays, because the amount of evidence supporting each probeset is varying, and some probesets have been placed on the array with much less confidence than others. Strategies that aim to define new probesets using annotation must deal with these. Another issue is that model-based algorithms such as PLIER and RMA make the assumption that all probes within a probeset target the same thing. When alternate splicing is involved , this assumption may not hold.An alternate strategy is to use the standard probeset definitions provided by the manufacturer, and to apply annotation to the data only after the “interesting” probesets have been selected. This is the approach taken here. Annotation can also, in some circumstances, be used successfully to pre-filter the data before statistical analysis by, for example, removing all probesets except those that match known exons in the Ensembl genome database. Again, this can have a beneficial effect on the FDR.exonmap, which communicates with X:MAP and provides access to its data from within R. The design and implementation of the database is described in more detail in Line 1 simply loads the package.ExpressionSet object raw.data, and relies on the presence of a file  that defines the names of the .cel files to be loaded and their associated experimental parameters. The covdesc file used here is as follows:Line 2 loads the expression data into the groupex1MCF7_r1.CELaex1MCF7_r2.CELaex1MCF7_r3.CELaex2MCF10A_r1.CELbex2MCF10A_r2.CELbex2MCF10A_r3.CELbread.exon loads the data, the additional experimental parameters defined in covdesc are loaded into the ExpressionSet, and can be retrieved by the function pData.When Line 3 takes a little more explanation. The raw array data is stored in a .CEL file, which records, for each of the ∼6 million features on the array, the raw unprocessed spot intensity. These must be grouped into probesets, as specified by a Chip Definition File (CDF). Line 3 tells BioConductor which CDF file defines the array layout by setting the name of the .cdf package (“exon.pmcdf”). CDF files can be downloaded from .Line 4 uses RMA to normalize and generate expression summaries for the probesets. To use PLIER, the following code can be substituted:8x.pli <- justPlierThe concpenalty parameter can be used to adjust how PLIER deals with probesets that have very different values on a small proportion of the arrays in a project . We have found it can be useful to set it to a higher value than the default (0.000001) when processing small numbers of arrays. Otherwise, PLIER can tend to see genuine differences between arrays as noise. This is translated into reduced fold changes for these probesets. PLIER's tuning parameters are described in more detail in .pc to calculate fold-changes and t-test p-values for each probeset on the array. Note that it is using the data loaded in from the covdesc file to specify that the comparison should be between samples labelled “a” and “b” in the column “group”.Line 5 uses p-value less than 10−4, and line 7 fishes the names of these probesets out from the raw.data object. Clearly, more sophisticated approaches  can, and probably should, be substituted here.Line 6 uses the result of the pairwise comparison to find all probesets with an absolute fold-change greater than 1—all data are on a log2 scale, so at least 2-fold differentially expressed—with an unadjusted Irrespective of how the probeset list is identified, these must then be mapped to the genome and to gene annotations. The aim of exonmap is to provide methods to answer questions such as these:1. Which probesets hit exons, introns, transcripts, and/or genes?2. Which probesets hit between known genes—and do they match ESTs or Genscan predictions?3. Which genes/transcripts are differentially expressed or alternatively spliced?X.to.Y that allow lists of identifiers to be mapped between the different levels of annotation. First, however, it is necessary to connect to the database:The basic approach is to provide a series of functions 9> xmapDatabase(“Human”)This relies on a configuration file specifying how to connect to the database. Full details are in the package vignette and installation instructions.10> probeset.to.exon$‵2318663‵[1] “ENSE00000738016”$‵2318675‵[1] “ENSE00000737793”$‵2318682‵[1] “ENSE00001435048” “ENSE00000737789”$‵2318693‵[1] “ENSE00000401699”?probeset.to.gene for more details). Similar functions can be used to perform mappings backward and forward between exons and genes , transcripts, etc. Functions also exist to get detailed information for exons, transcripts and genes—for example:Note that probeset ‘2318682' matches to two different exons (a quick search on X:MAP shows that these are from two different transcripts from the same gene—PER3). Other parameters can force the results to be returned as a vector, to be filtered to remove duplicates transcript_id stable_id seq_region_id nameENST00000377541 226810 ENST00000377541 226034 1seq_region_start seq_region_end seq_region_strandENST00000377541 7766967 7786605 1biotype status description external_db_idENST00000377541 protein_coding KNOWN NA 2000db_display_name display_labelENST00000377541 UniProtKB/TrEMBL Q8TAR6_HUMANdescriptionENST00000377541 PER3 protein (Period homolog 3) (Drosophila).At line 7, an initial probeset list was generated. Many of these probesets match outside Ensembl genes, whilst others match inside genes, but outside exons. Filtering functions are provided to select or exclude these probesets from the probeset list, as follows:12> sigs.exonic <- select.probewiseexclude.probewise allows probesets to be filtered out of a list.Intronic and intergenic probesets can be selected in a similar way, and the analogous function Some probesets contain probes that match the genome at multiple sites. These can be selected or removed as follows:13sigs.mt <- select.probewiseFiltering criteria are based on the number of times each probe within a probeset matches the genome, a gene, or an exon. The aim is to identify those “well-behaving” probesets that consist of probes that match only once to the feature of interest.Exon arrays enable genome-wide, high-throughput searches for alternative splicing events . There aA simple, but effective method, the “splicing index”, was proposed in . Each exThe method depends on the selection of the gene-level expression summary . In exon14> si.sig <- splicing.index)Gene level summaries are calculated, by default, as the median expression of ‘well-behaving' exon probes .p-Values are calculated for all the probeset-treatment pairs; alternative splicing events are selected as those with low p-values:Searches for alternative splicing may also be done for experiments with multiple treatments. In this case, in place of a simple index, there is an ANOVA model, tested for each probeset. The null hypothesis is that there is no alternative splicing for the gene, and the NI level for each probeset should therefore not be significantly different across the samples. 15#in an experiment with more than two replicate groups16> splanv <- splanova17> fval(splanv)exonmap. plot.gene, for example, will plot the structure of the specified gene, coloured according to the expression data  black rectangles, and probesets by larger rectangles coloured by gene expression Click here for additional data file.Text S2(93 KB PDF)Click here for additional data file.gene is a region of the genome that is transcribed into RNA. A transcribed RNA molecule is referred to as a transcript. The RNA produced by genes that encode protein sequences is called messenger RNA or mRNA. In eukaryotes, genes can contain exons and introns. The introns are removed from the initial transcribed RNA (or pre-RNA) by splicing. Splicing can also be used to remove exons. By selectively retaining different sets of exons within the spliced transcript, cells can produce multiple isoforms from a single gene, and, if subsequently translated, multiple proteins. This process is known as alternative splicing. A features (often referred to as “spots”) designed to recognize each molecule of interest. Each feature consists of millions of identical single-stranded 25-mer nucleotide probes, each designed to hybridize to a specific transcript. On a gene-level array, such as the HGU133plus2 chip [Perfect Match (PM) features is accompanied by an adjacent Mis-Match (MM) feature in which the middle residue is changed. Hybridization conditions are designed to maximise binding to the PM features while minimizing binding to the MM ones. Each MM feature can therefore be used to provide a measure of probe specific background for its PM partner. Multiple PM/MM pairs are used for each transcript. On most gene-level arrays, 11 PM/MM pairs are used per transcript, and the complete set of 22 features is referred to as a probeset. Affymetrix expression arrays use a set of us2 chip , each of


Independently derived expression profiles of the same biological condition often have few genes in common. In this study, we created populations of expression profiles from publicly available microarray datasets of cancer  samples linked to clinical information with an iterative machine learning algorithm. ROC curves were used to assess the prediction error of each profile for classification. We compared the prediction error of profiles correlated with molecular phenotype against profiles correlated with relapse-free status. Prediction error of profiles identified with supervised univariate feature selection algorithms were compared to profiles selected randomly from a) all genes on the microarray platform and b) a list of known disease-related genes (a priori selection). We also determined the relevance of expression profiles on test arrays from independent datasets, measured on either the same or different microarray platforms.Highly discriminative expression profiles were produced on both simulated gene expression data and expression data from breast cancer and lymphoma datasets on the basis of ER and BCL-6 expression, respectively. Use of relapse-free status to identify profiles for prognosis prediction resulted in poorly discriminative decision rules. Supervised feature selection resulted in more accurate classifications than random or a priori selection, however, the difference in prediction error decreased as the number of features increased. These results held when decision rules were applied across-datasets to samples profiled on the same microarray platform.Our results show that many gene sets predict molecular phenotypes accurately. Given this, expression profiles identified using different training datasets should be expected to show little agreement. In addition, we demonstrate the difficulty in predicting relapse directly from microarray data using supervised machine learning approaches. These findings are relevant to the use of molecular profiling for the identification of candidate biomarker panels. Clinically validated biomarkers are highly valued in cancer pathology for diagnostic and prognostic purposes . BiomarkDetection of candidate biomarkers from high-dimensional molecular datasets entails separation of signal from noise. As such, techniques adapted from signal processing and machine learning can be applied. The goal of machine learning is to reliably predict the class, or phenotypic state, of a new sample given only a set of measured input variables. The definition of a function that equates input variables to response is called supervised learning. In general, supervised learning consists of three steps: feature selection, decision rule specification and estimation of generalization error . FeatureFeature selection is particularly important in gene expression profiling, in which the number of features (genes) is much larger than the number of observations (microarray data samples). Identification of discriminative features eases the process of data interpretation and communication, decreases computation time for training, and, in biomarker identification, enables the development of reliable clinical assays. Numerous feature selection algorithms can be found in the literature, most of which rank features in a univariate manner, sorting them on the basis of correlation with class distinction ,18-21. IApplication of supervised feature selection methods in microarray analysis identifies a set of genes whose expression patterns, or profiles, are most correlated with response. However, discriminative feature sets identified in multiple microarray studies of the same disease state or biological condition typically share few common genes -27, indiGiven the presence of multiple, generally exclusive expression profiles of disease states such as metastatic breast cancer, it is appropriate to ask whether feature selection identifies expression profiles that classify better than is expected by chance, i.e. better than gene subsets selected randomly. It is also important to determine to what extent technical and biological variability between studies affects the generalization error of classifiers trained on expression profiles. In this study, we analyzed a multitude of publicly available microarray datasets consisting of expression data linked to clinical data for breast cancer, renal cancer, and lymphoma -41. ExprAnalysis of simulated gene expression datasets indicated the effectiveness of the feature selection and classification algorithms used in this study to predict in binary endpoints. Simulated datasets consisting of 100 observations and 1000 features were designed to approximate a binary classification problem . ExpressComputations with breast cancer microarray datasets from four independent cohorts of patients  or a list of known disease related genes (a priori selection)) Figure . Random Our computations indicate that classification error is only weakly dependent on normalization, feature selection, classification and training/testing partition. Breast cancer dataset GSE3494 was used to assess the effects of these classification parameters on predicted error. Figure It has been shown that decision rules based on microarray data are capable of clearly differentiating tumors by outcome when all data is used for feature selection in a leave-one-out cross-validation scheme. Our findings validate previous results in the literature concerning, for example, the prediction of relapse in lymphoma . In theiNext, we tested whether prediction error calculated by within dataset cross-validation holds when decision rules trained on one dataset are applied to arrays from other datasets profiling similar populations. Within dataset cross-validation may be biased according to the degree of non-specific correlation between the training and test data. Non-specific correlation can be described as technical noise that arises in sample preparation, hybridization and scanning and results in data collected from the same lab being more highly correlated than data collected in different labs . To inveBiomarker sets derived from different global gene expression datasets for the purpose of predicting molecular phenotype or relapse in cancer contain very few common genes -27. In aWe also compared the prediction error resulting from supervised feature selection vs. features selected randomly from either the entire set of genes represented on the microarray or an a priori defined subset of disease-relevant genes. Overall, univariate feature selection led to more accurate classification; however, the difference in prediction errors decreased as the number of features increased. Similar results were also observed in the application of decision rules to samples from other gene expression datasets profiled on the same microarray platform. From this, we conclude that the presence of multiple biomarker sets in the prediction of molecular phenotype arises from the large number of genes correlated with response.In contrast, decision rules trained on the basis of relapse-free status classified samples with relatively high prediction errors in breast cancer, DLBCL and CRCC datasets. Specifically, prediction error was approximately 40% in all cases that were studied regardless of the method used for feature selection. Overall, these results indicate the difficulty of developing biomarker sets predictive of cancer relapse using a single microarray dataset. Our results do not apply to meta-analytical approaches, in which cancer relapse predictions are obtained by integrating data from multiple microarray datasets prior to machine learning -55. In aPublicly available gene expression data for a multitude of cancer types  were collected from the online repositories Gene Expression Omnibus (GEO)  and StanEach microarray dataset was analyzed independently to evaluate the error in predicting relapse  using univariate feature selection compared to the resulting error from biomarker sets chosen randomly from either the entire set of genes represented on the microarray (random) or a smaller set of experimentally validated cancer-associated genes (a priori). To this end, we used an iterative supervised, machine-learning approach, described below. For completeness, we tested the dependence of our approach on the use of different pre-processing, feature selection and classification algorithms and cross-validation schemes. The primary focus of our study was on breast cancer, where multiple datasets were available for analysis. Lymphoma and renal carcinoma datasets were used to assess the applicability of our conclusions relevant to breast cancer on other disease states. All work described in this study was carried out using the R statistical environment  and was Microarray datasets were collected in raw format when available . Two pre-processing algorithms, Robust Multi-Array Analysis (RMA)  and MAS Affymetrix probe sets were annotated using the hgu133a package  in R. StA single probe set representing each gene was selected due to the varying redundancy of gene representation on microarray platforms. In datasets compiled in this study, approximately 60% of genes were represented by a single probe set. Genes represented by multiple probe sets were dealt with in the following manner. For Affymetrix datasets, probe set suffixes were used to remove redundant probe sets. For the HG-U133a chip, probe sets are encoded with _at, _s_at and _x_at suffixes that describe the quality of probe design . All _x_Microarray datasets were iteratively divided into learning sets (LS) and test sets (TS) to create a population of classifiers and determine their classification performance in a Monte Carlo cross-validation approach . Two typTwo classification algorithms, diagonal linear discriminant analysis (DLDA) and k- nearest neighbour , were used to create decision rules on the basis of training data. The NN algorithm classified test samples by the class of the three closest samples in the training set using Euclidean distance . DLDA isTo obtain an estimate of generalization error, decision rules were applied to thecorresponding test sets. The confidence (δ) with which each sample was classified was calculated as follows:R and dN are the distances predicted by the decision rule to the poor and good prognosis classes, respectively. Samples are classified as good prognosis if the score is greater than 0.5 and vice-versa.in which dWith this methodology, the classification performance of the decision rulescould be visualized and compared with the use of receiver operating characteristic (ROC) curves . ROC curClassification performance was determined from the area under the curve (AUC). Good classifiers have high detection rates across the range of false alarm rates and therefore have a large AUC. ROC curves were generated for each classifier using the ROC package in R . The scoij was generated with a block structure with σij = 0.2 for | j-i | ≤ 5 and 0 otherwise.Simulated microarray datasets were generated to verify that the machine learning algorithm described above leads to accurate classification. Simulated expression data was created in the manner described by Bura and Pfeiffer . DatasetTo determine the significance of the calculated prediction error metric E for molecular profile prediction in breast cancer and lymphoma, the machine learning algorithm was repeated 1000 times with permuted class labels. An empirical p-value was calculated as the fraction of decision rules based on permuted class labels that performed better than the expected classification performance, E (described above), of decision rules based on true class labels. Permutation processes give an estimate of the likelihood that the true E value could be obtained by chance alone and are frequently used in similar studies for this purpose ,78. ResuIn addition to cross-validated generalization error, we determined the classification accuracy across datasets. To this end, decision rules trained on one dataset were tested in both the corresponding test subset and datasets obtained by other laboratories, resulting in E values for each test case. For across platform comparisons , probe sets were matched by annotation to Entrez Gene identifiers.ROC, Receiver Operating Characteristic; ER, Estrogen Receptor; BCL-6, B-cell CLL/lymphoma 6; cDNA, Complimentary Deoxyribonucleic Acid; mRNA, Messenger Ribonucleic Acid; DLBCL, Diffuse Large B-cell Lymphoma; CRCC, Conventional Renal Cell Carcinoma; RT-PCR, Reverse Transcription Polymerase Chain Reaction; GEO, Gene Expression Omnibus; SMD, Stanford Microarray Database; RMA, Robust Multi-Array Analysis; LS, Learning Set; TS, Test Set; DLDA, Diagonal Linear Discriminant Analysis; NN, Nearest Neighbor; AUC, Area Under the Curve;MG and AT conceived and developed the research plan. MG implemented the integration of algorithms, performed the computations and wrote the manuscript draft. WD, AE and BK also contributed to the conception of the study and implementation of the strategy. All authors read and approved the final manuscript.Prediction error of DLDA classifiers on lymphoma (Broad) and renal carcinoma (Zhao) datasets. Classifiers trained to predict relapse-free status. E is the mean 1-AUC of the corresponding set of ROC curves, calculated as described in the Methods section. Error bars show empirical 95% CIs.Click here for fileSignificance of prediction error  of DLDA classifiers trained to predict molecular phenotype. Bold entries indicate significant P-values < = 0.01.Click here for fileKaplan-Meier plots of survival rates for tumor classes with different classification/cross-validation methods. Classifiers trained on the basis of relapse-free status on diffuse large B-cell lymphoma dataset GSE4475. Column 1: Weighted-voting algorithm. Column 2: DLDA. Row 1: Leave-one out cross-validation. All data used for training and testing. Row 2: Training and test sets selected randomly from the dataset. Training based on leave-one out cross-validation.Click here for file


We have identified a set of genes whose relative mRNA expression levels in various solid tumors can be used to robustly distinguish cancer from matching normal tissue. Our current feature set consists of 113 gene probes for 104 unique genes, originally identified as differentially expressed in solid primary tumors in microarray data on Affymetrix HG-U133A platform in five tissue types: breast, colon, lung, prostate and ovary. For each dataset, we first identified a set of genes significantly differentially expressed in tumor vs. normal tissue at p-value = 0.05 using an experimentally derived error model. Our common cancer gene panel is the intersection of these sets of significantly dysregulated genes and can distinguish tumors from normal tissue on all these five tissue types.Frozen tumor specimens were obtained from two commercial vendors Clinomics  and Asterand . Biotinylated targets were prepared using published methods  and hybridized to Affymetrix U133A GeneChips . Expression values for each gene were calculated using Affymetrix GeneChip analysis software MAS 5.0. We then used a software package called Genes@Work for differential expression discovery, and SVM light linear kernel for building classification models.We validate the predictability of this gene list on several publicly available data sets generated on the same platform. Of note, when analysing the lung cancer data set of Spira et al, using an SVM linear kernel classifier, our gene panel had 94.7% leave-one-out accuracy compared to 87.8% using the gene panel in the original paper. In addition, we performed high-throughput validation on the Dana Farber Cancer Institute GCOD database and several GEO datasets.Our result showed the potential for this panel as a robust classification tool for multiple tumor types on the Affymetrix platform, as well as other whole genome arrays. Apart from possible use in diagnosis of early tumorigenesis, some other potential uses of our methodology and gene panel would be in assisting pathologists in diagnosis of pre-cancerous lesions, determining tumor boundaries, assessing levels of contamination in cell populations in vitro and identifying transformations in cell cultures after multiple passages. Moreover, based on the robustness of this gene panel in identifying normal vs. tumor, mislabelled or misinterpreted samples can be pinpointed with high confidence. Rapid and accurate classification of cancerous tissue samples is an unmet scientific and clinical need. Standard clinical practice in identifying cancer relies on pathological examination of biopsy specimens, radiological images and histology. However, these diagnoses can be incorrect because of atypical morphologies, or poorly extracted biopsies. In cases where the pathologist makes an error in determining whether a surgically resected tumor has sufficient normal cells in its margins, an error could have significant consequences to the patient. A corroboratory analysis may also benefit laboratory experiments on cell lines or tissue samples which might be labelled as cancerous, but might in fact be significantly or wholly contaminated by surrounding or externally derived non-cancerous tissue.http://www.oncomine.com and have attempted to establish a multi-tissue cancer signature and have claimed and demonstrated success in classifying cancer from normal tissue. The main difference between these two approaches is the algorithm used for feature extraction. Xu et al [http://www.research.ibm.com/FunGen/FGDownloads.htm. The experimental model used in Genes@Work determines p-values based on a multi-tissue model derived from replicate measurements on Affymetrix chips to assess stochastic and systematic (handling) errors in microarray data analysis.Several previous studies have attempted to find a common gene signature in multiple neoplasms. One such group at the NIH has also established a gene panel capable of distinguishing benign from malignant tumor in four different tissue types . In termXu et al  used a mXu et al  used a mXu et al , but do Our training set consists of a proprietary sample set for normal and cancerous tissue from breast, colon, lung, prostate and ovary. A detailed description of this data is available in the methods section. Using this high quality multi-tissue data set, we applied an integrated informatics strategy which combined targeted bioinformatics and analytical approaches to identify and validate a panel of genes to distinguish normal from cancer tissue. We also demonstrated that an accurate diagnosis of cancer tissue is possible using modern gene expression arrays.Frozen tumor specimens were obtained from two commercial vendors Clinomics  and Asterand . The data was obtained from five tissue types , , , ,  [Table Affymetrix probes were first mapped to UniGene Ids using their publically available annotation table and then subsequently mapped to Agilent probes using their respective annotation table. Since many of the genes in our panel are represented by multiple probes, the average expression of each probe was measured across all samples and the probe with the highest overall signal for each gene in our panel was selected as the equivalent diagnostic feature.We used a software package called Genes@Work  created http://svmlight.joachims.org/ using a linear kernel option and complete leave-one-out estimations were calculated for each experiment. M-fold cross validation was also performed at m = 5 and 300 samplings with replacement  cross validation as well as m-fold cross validation, we verified that our common "tumor discriminating" gene panel was robust and could separate cancerous tissue from normal tissue with accuracies exceeding 90% when the tissue of origin was known  that takes a gene signature as input and searches all deposited internal and public clinical and non-clinical microarray analyses for similar signatures. When inputting our multi-tissue average signature, the top significant results all refer to normal vs cancer comparisons. Our global signature hit 47% of the cancer vs. normal comparisons which was significant relative to the total distribution that only comprises 5% of the database. We then tested the top publicly available hits to see how accurately we could predict the two classes. Specifically, we tested two datasets containing tissue types outside the scope of our training set.The first dataset analysed was a bladder cancer dataset from Dyrskjot et al  containihttp://www.ncbi.nlm.nih.gov/gds/?term=GSE781 and applied our SVM classifier on this data to distinguish ccRCC from normal kidney. The results are shown in Table To establish how well our panel performed for ccRCC/normal-kidney discrimination we used a previously published dataset which was also used by two other groups ,3 in tesTo further test our accuracy on ccRCC we also analyzed a microarray dataset from an on-going collaboration at the University of North Carolina Medical School consisting of 52 ccRCC samples and 18 patient matched normal kidney samples from two studies -18. The For the public datasets we used, the pathological stage is given in the papers cited above. For the proprietary datasets, Additional File When the gene signature is analysed for its functional classifications, it becomes clear that it is enriched in categories important for cell growth. Among the highly enriched categories include actin cytoskeleton organization, DNA packaging, nucleosome and chromatin assembly, and regulation of blood vessel size.While the Rhodes and Xu gene panels have demonstrated similar success in diagnostic capabilities our gene panel is more robust both in design and performance Table . Our panThe reason for the robustness of the panel is that we use a methodology (Genes@Work) where the error model for expression levels (noise) is derived from actual replicate measurements  rather tOur feature set has several potential uses. A common problem of cancer sample collection is that surrounding normal tissue can severely contaminate the sample. Since the tissue of origin is usually known in biopsies, our panel can be used as a tool for rapid determination of the presence or absence of cancer cells as an aid to pathologists. It can also be used to determine contamination in in-vitro experiments.Expression-based diagnosis and risk assessment is rapidly gaining popularity in the clinic  with diaSVM: Support Vector Machine; GCOD: Gene Chip Oncology Database; GEO: Gene expression Omnibus; qRT-PCR: quantitative real time-polymerase chain reaction; ccRCC: clear cell renal cell carcinoma; LOO: leave-one-out; FFPE: Formalin-Fixed, Paraffin-EmbeddedC. Chris Huang, Yi Zhang, Dmitri Talantov, Sándor Szalma are all employees of pharmaceutical companies of Johnson & Johnson.JI drafted the manuscript and conducted the majority of the bioinformatics analysis described. YZ and DT supplied the data and carried out initial analysis. SS helped conceive the study and was the main driver of this research initiative. GB served as graduate mentor for JI and thus provided the majority of the scientific input. GB, SS, CH all contributed to editing of the manuscript as well as providing direction and guidance to JI. All authors have read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2407/10/319/prepubMelanoma data set (GSE 3189) visualization. Melanoma data set (GSE 3189) gene expression sorted by ratio of gene expression ratio of cancer vs. normal. The middle portion contains nevus samples which are considered benign. Interestingly, they appear to have a mixed signature that is an incomplete transformation from normal to cancer.Click here for fileThe kidney cancer validation sets. Samples on the left are normal, right are cancer. The first image represents the Lenburg et al. [g et al.  data setClick here for fileccRCC data from UNC labs. This image is of the ccRCC data [RCC data ,16 from Click here for fileDistribution of cancer stage in the training data. The following table shows the distribution of stages of the cancer samples in our training set.Click here for file


We developed a quality assurance (QA) tool, namely microarray outlier filter (MOF), and have applied it to our microarray datasets for the identification of problematic arrays. Our approach is based on the comparison of the arrays using the correlation coefficient and the number of outlier spots generated on each array to reveal outlier arrays. For a human universal reference (HUR) dataset, which is used as a technical control in our standard hybridization procedure, 3 outlier arrays were identified out of 35 experiments. For a human blood dataset, 12 outlier arrays were identified from 185 experiments. In general, arrays from human blood samples displayed greater variation in their gene expression profiles than arrays from HUR samples. As a result, MOF identified two distinct patterns in the occurrence of outlier arrays. These results demonstrate that this methodology is a valuable QA practice to identify questionable microarray data prior to downstream analysis. Gene expression profiling can help in identifying potential biomarkers to aid cancer classification in addition to traditional pathology-based methods . For exaMicroarray experiments are complicated processes involving multiple steps. From manufacturing of the arrays to data acquisition by scanning of hybridized arrays, numerous factors at any of these steps can lead to unwanted random or systematic variation in the data. The factors impacting microarray raw data can be technical, instrumental and computational . QA of mIt is important to identify outlier arrays for microarray data analyses. The majority of statistical strategies for differential analysis of gene expression are derived from t-test or ANOVA based analyses . A limitThe PAXgene blood RNA extraction system  was used to collect blood and extract RNA. The system consolidates and integrates the key steps of whole blood collection, nucleic acid stabilization, and RNA purification. It reduces the unpredictability associated with RNA processing and provides enhanced accuracy of intracellular RNA analysis.Blood samples were collected from donors enrolled in the Clinical Breast Care Project following IRB-approved protocols. 2.5 mls of blood were drawn by qualified individuals directly into PAXgene tubes. The PAXgene tubes contain an additive that stabilizes cellular RNA and prepares the samples for RNA purification. The samples were left in the PAXgene tubes overnight at room temperature and then stored at −20°C or −80°C until use.260/A280 ratio must be above 1.7 for use in microarray experiments. RNA quality was further checked using Agilent Bioanalyzer Nano Chip . All extracted RNA must have a 28S/16S ratio greater than 1.5 or a RNA Integrity Number (RIN) above 5 to pass this QA measure.Next, tubes were removed from freezer and allowed to warm to room temperature overnight. Total RNA was extracted using the PAXgene blood RNA kit in accordance with the supplier’s instructions. The procedure started with a centrifugation step to pellet nucleic acids in the PAXgene tubes. The pellet was washed, resuspend, and incubated in optimized buffers containing Proteinase K to digest proteins. Residual cell debris was removed by centrifugation, and the resulting supernatant was transferred to a fresh microcentrifuge tube. Ethanol was added and mixed by vortexing. The lysate was then applied to a PAXgene RNA spin column and a brief centrifugation was used to remove contaminants while RNA was selectively bound to the silica-gel membrane of the column. Remaining contaminants were removed by three wash steps, and pure RNA was eluted using nuclease-free water. To denature the RNA, which was essential for maximum efficiency in downstream applications including cDNA synthesis, the eluate was incubated briefly at 65°C in a heating block. Following incubation, RNA samples were chilled immediately on ice. RNA concentration and purity was determined using a NanoDrop  which provides absorbance measurements at 260nm and 280nm. The Ain vitro transcription was carried out to generate cRNA followed by biotin-labeling. An aliquot of these labeled cRNA was run on Agilent’s bioanalyzer for qualification and quantization. A total of 10 μg of high quality cRNA was fragmented and then hybridized to CodeLink Bioarrays. If a sample did not yield enough cRNA or cRNA failed the quality control, the experiment would restart from cRNA amplification from total RNA. After hybridization, arrays were washed, stained with Cy5 dye then scanned using ScanArray 5000 .We used CodeLink Bioarrays  for microarray experiments. CodeLink utilizes a one-color labeling and detection method based on biotin-labeled cRNA. Human 20K bioarray targets 19881 well-annotated human genes. Each transcript is represented by a 30-mer probe which is designed to conserve exons across the transcripts of targeted genes. Total RNA from each donor was linearly amplified and hybridized to the CodeLink 20K human bioarray following the manufacturer’s recommended protocol. In brief, 1 μg of total RNA together with control bacterial mRNA was used to synthesize the first strand cDNA by reverse-transcription, followed by the second strand cDNA synthesis. After purification of these double-strand cDNA, Scanned images were analyzed using CodeLink Expression software to generate raw data. We manually checked all the grids to be sure that no miss-griding occurred. Results were stored in a data warehouse internally developed for the integration of clinical, genomic and proteomic data. Perl was used to retrieve and preprocess data for the statistical analysis performed by R along with BioConductor. Spotfire  was the visualization tool for the current study.Two groups of microarray datasets were used in this study: 35 arrays of a human universal reference (HUR) RNA sample  and 185 arrays of human blood samples. An HUR sample was included in each microarray experiment as a control for systematic variation from experiment to experiment. These HUR experiments were carried out from September, 2003 to December, 2004. This dataset was used to validate our QA methods and to gain a general assessment of the reproducibility of our experiments. The human blood samples were composed of whole blood from 71 patients with benign breast lesions, whole blood from 82 patients with invasive breast cancer and whole blood from 30 normal controls with no known breast disease. The human blood samples were subjected to microarray experimentation from July, 2003 to August, 2004.In this study, only a subset of the 20470 probes on a standard 20K array was used for QA analysis. Probes with median raw intensity across the array group (e.g. HUR or human blood samples) below 200 or above 10000 were excluded. Based on our experience, we believe that intensity below the 200 was too close to the background level; thus, the signal measurement might not be reliable. On the other hand, intensity above 10000 fell in the saturation range of the scanner and thus did not effectively reflect the difference in the signal strength. Of the 7080 probes which satisfied these criteria, only good data points, as judged by CodeLink spot quality measures, were included in our QA analysis. The selected datasets were then normalized by quantile normalization across the array group. All subsequent QA analyses were performed on the data preprocessed in this way.The CodeLink platform has rules to evaluate the quality of individual spot and to flag problematic data points. CodeLink also provides a QA measure, normalized threshold, to gauge the overall intensity of fluorescent signal level on an array. A “too low” or “too high” signal level usually indicates systematic errors in the experiment, and results in rejection of the whole array.However, these QA measures only focus on a single array without considering its relationship with other arrays. We believe that it can be informative to view individual arrays as a whole against a group of arrays. By monitoring gross changes in a dataset of whole arrays, we expect to identify outlier arrays which are significantly different from other arrays. Our approach is to compare statistics of an array to those of other arrays and look for arrays which are dissimilar to the majority of arrays in the whole group. These “unusual” arrays are candidates for failed arrays which will be investigated further. It should be noted that this rationale is based on two assumptions. First, all samples for comparison are similar enough so that the expression levels of most genes are similar among the samples. When these samples are subjected to the same experimental protocol, it is expected that the gross expression profiles are the same for all the samples. Second, we are confident that most of our array data are of high quality thus consistent with each other. Therefore, a few “unusual” arrays stand out as suspected failed arrays.The HUR sample was obtained commercially. The manufacturer strived to reduce differences between samples from batch to batch. The 185 human blood samples described in this paper were from different subjects. Although there were biological variations among these samples, based on our experience we expected that the variations were not large enough to substantially change the statistical features of the datasets. In addition, all samples were analyzed using the same experimental protocol. Therefore, our samples were suited to the QA approach described above.Two statistical indices were used for array comparison. The Pearson correlation coefficient was computed for each and every pair of arrays in the analysis. A collection of the correlation coefficient values was arranged in a table  to show X̃ and s̃ are the median and MAD (median absolute deviation). As its name implies, a resistant z-score is resistant to outliers’ influences since the median and the MAD are used for calculation. It can tolerate up to 50% outliers without being distorted. A resistant z-score was computed for each data point. A data point with the absolute value of its resistant z-score larger than a preset threshold was designated as an outlier. In this study, we chose to set the threshold to 3 based on our experience with our datasets. We found that this threshold could distinguish outlier arrays from non-outlier arrays. We observed that the more an array differed from other arrays, the more outlier data points were detected on that array. Therefore, the percentage of outlier data points could be used as an indicator for an outlier array. This algorithm using correlation coefficient and percentage of outlier spots was implemented as an R function: microarray outlier filter (MOF), which ranks the arrays by their possibility of failure based on these two statistical indices.where Correlation coefficients were displayed in a correlation table. All visualizations, including the heat map of the correlation table, the heat map showing the percentage of outliers, scatter plots between arrays, and images of spatial distribution of spots on arrays, were developed in Spotfire. Clustering of the correlation table was done by complete linkage hierarchical clustering using Euclidean distance as the similarity measure in Spotfire. To compare an array with a group of similar arrays in a scatter plot, a model array was constructed by taking the median of all intensity values of the same probe across all the arrays in that array group, as the value for that probe on the model array. Thus, the model array could be used to represent the group of arrays. To visualize the locations of probes on an array, a 2-D plot was constructed using local coordinates of the probes on the array as provided by CodeLink.We applied our QA measures to the 35 HUR sample arrays which were expected to be highly consistent. The purpose of analyzing HUR data was to test and validate our QA measures. A correlation coefficient was computed for each pair of the 35 arrays. The average correlation coefficient for an array was used to assess the similarity of the array to other arrays. The percentage of outlier spots on each array was also calculated to serve as an additional indicator of “abnormal” arrays. Arrays with low correlation with the majority of other arrays and/or a higher percentage of outliers were suspected as outlier arrays. There were three arrays  which had substantially lower correlation to other arrays. These three arrays also generated more outlier data points than other arrays . The aveCorrelation coefficients of all array pairs were arranged into a correlation table  which waWe observed that arrays within a cluster typically had high correlation with each other while arrays between the two clusters displayed poor correlation . In detaFiltering out low-quality arrays helps prevent misleading results in the subsequent data analysis. At the same time, QA analysis can benefit the laboratory by providing clues to possible causes to failed arrays. We set off to inspect the three outlier arrays more closely hoping to gain deeper insight into what might have gone wrong with these arrays by using scatter plots of signal intensity and a spatial distribution plot of the probes on the array . In FiguThe dataset of 185 human blood samples was analyzed the same way as the HUR sample arrays. It could be seen that some arrays had overall poor correlation with other arrays . The aveWe performed QA analysis on two groups of microarray datasets. The HUR data was more homogeneous. It was used to test our QA approach employing correlation coefficient and percentage of outlier spots as the QA measures. Having validated the QA approach, we performed the QA analysis in the same way on the dataset of 185 human blood samples and flagged 12 arrays as candidates for failed arrays. We recommended that microarray experiments with these arrays be repeated. Furthermore, using images of spot distribution reconstructed in Spotfire, we confined problematic areas in certain blocks on two failed arrays.Although we demonstrated that our QA measures were effective in identifying low-quality arrays, it should be noted that it is important that the samples for the QA analysis are of similar expression profile in general, and the experiments are conducted using the same protocol. Following these rules ensures that outlier arrays detected by our approach were not due to normal variation in expression profile. For example, we checked correlation between expression arrays done on samples from different mouse tissues and found that correlation in array expression between the tissues was so low that clustering of the correlation table was strictly dictated by tissue type (data not shown). In such a scenario, a few highly similar outlier arrays were, instead of failed experiments, just normal expression profiles which differ from profiles of other samples. Such distinct categorization was not seen among the human blood samples, indicating variation among our samples was not large enough to invalidate this QA approach. We will not be surprised, however, that a solid human tissue may display a different expression profile and thus is flagged by our QA measures as different from human blood samples.Essentially, our QA measures check consistency between experiments. It is not a natural deduction that outlier arrays are always failed arrays. It is possible that the data from outlier arrays are good but other data are consistently wrong. One should always be cautious in rejecting outlier arrays as failed arrays. In our experiments involving the HUR sample, the outlier arrays were not consistent among themselves, making it impossible that all the outlier arrays were good. Therefore, we think that the outlier arrays were truly failed arrays. Based on our experience with the HUR sample, we have confidence in the data quality of most of our human blood samples. Outlier arrays are likely caused by failed experiments although other causes can not be ruled out, e.g. the samples are biologically heterogeneous. It is still an open question whether the outlier arrays may represent some unusual samples, especially in the case of our human blood samples, the difference between the outlier arrays and the other arrays were not as dramatic as in the HUR arrays. More replicates of the outlier arrays should be done to address this issue.Another issue in the application of our QA approach is how to set the threshold to reject an array. We do not think there is a universal standard. Even with the same data, correlation coefficients can change with the number of probes involved in the QA analysis. Importantly, the thresholds for the average correlation coefficient and the percentage of outlier spots should be adjusted when arrays of unknown quality are analyzed. Thresholds should be established empirically. Familiarity with historical data plays an important role in threshold decision-making. Our QA method is designed to provide the likelihood of possible problems on an array. We inspect potential problems starting from the top of the lists of the potentially problematic arrays reported by MOF. Down the list we usually see the severity of problems decreasing. The setting of the threshold depends on the nature of samples, the goal of analysis and even the availability of resources, etc. Of the 35 HUR RNA experiments, 3 arrays significantly differed from the other arrays and were also different among themselves in regard to the average correlation coefficient and percentage of outlier spots. In this case, it was logical to filter out only these three arrays as outlier arrays. However, with the 185 human blood samples, we did not see such a clear cut dividing of the arrays into outlier and normal arrays. Twelve arrays were flagged as problematic arrays not only because these arrays had relatively low correlation with other arrays and a high percentage of outlier spots but also because other techniques such as scatter plots and clustering provided additional evidence. It should be stressed that our QA approach just evaluates the likelihood of failure for each array. Not necessarily every batch of arrays must contain failed experiments.Two statistics are employed in our QA approach. With the HUR arrays, both statistics generated highly consistent results. With the human blood samples, there was overlap and discrepancy between outlier arrays screened separately by correlation coefficient and percentage of outlier spots. Not every array which had poor correlation with the other arrays contained more outlier spots. This was not surprising because human blood samples were more heterogeneous than the HUR samples, and there were more human blood sample arrays than the HUR sample arrays in the current study. Intrinsically, the two statistics do not measure exactly the same properties of a dataset. It is natural they may give different results when the situation becomes complicated. Actually, it is an advantage that these two statistics can confirm and compliment each other. Abnormal arrays detected by either one of the QA measures warrant further investigation.For the correlation coefficient, a minimum of three arrays are needed to identify an outlier array mathematically. However, larger numbers of arrays can give more reliable results. Historical data can be used if insufficient arrays are available. Even though the data may come from different projects, they can be pooled for this QA analysis as long as the two assumptions laid out in Rationale and Algorithm Section are satisfied.Microarray technology has become more and more mature in the last several years. Technical replicates are rarely used, and biological replicates are counted on to reach a biological conclusion . However


The quality of gene expression microarray data has improved dramatically since the first arrays were introduced in the late 1990s. However, the reproducibility of data generated at multiple laboratory sites remains a matter of concern, especially for scientists who are attempting to combine and analyze data from public repositories. We have carried out a study in which a common set of RNA samples was assayed five times in four different laboratories using Affymetrix GeneChip arrays. We observed dramatic differences in the results across laboratories and identified batch effects in array processing as one of the primary causes for these differences. When batch processing of samples is confounded with experimental factors of interest it is not possible to separate their effects, and lists of differentially expressed genes may include many artifacts. This study demonstrates the substantial impact of sample processing on microarray analysis results and underscores the need for randomization in the laboratory as a means to avoid confounding of biological factors with procedural effects. The gene expression microarray has become a ubiquitous tool in modern biology resulting in substantial accumulation of data in public repositories Studies of the reproducibility of microarray data can vary in scope. Most studies use a common set of samples, but the embedded biological signals can be small or large, and they may or may not include truth standards such as ‘spike in’ RNA or mixtures of RNAs from knockout cell lines A/J, C57BL/6J-Chr6A/J and C57BL/6J-Chr15A/J. We will denote these strains as B, A1, A6, and A15 in this paper. Sample were distributed to each of four centers, and one center processed two sets of the 16 samples at different times using different labeling protocols. For simplicity we refer to these as five centers (C1–C5). We selected these strains based on the expectation that differentially expressed genes between the background strain B and each of the CSSs would be enriched for genes on the substituted chromosome. However there are no truth standards so our results reflect the precision but not necessarily the accuracy of the platform. Samples were delivered to each of the sites with the suggestion that they be processed according to standard protocols in a manner typical for that laboratory. Data from each center were provided in the form of CEL files.We collected kidney tissue samples from two male and two female mice from the C57BL/6J strain and from each of three chromosome substitution strains (CSSs) To investigate variability among centers, we applied a typical collection of interpretive analysis tools to data from each laboratory and made quantitative comparisons of the results. These analysis tools address the objective of generating and comparing lists of differentially expressed genes, identifying enriched biological pathways that are in common or differ between experiments, clustering the samples by expression pattern, and classifying new samples The efforts of user groups such as The Microarray and Gene Expression Data (MGED) society We normalized data from each center separately using the Robust Multichip Average method (RMA) Microarray analysis is typically focused on changes in expression level across conditions. Thus differences in intensity between samples are often more important than absolute intensity levels. The median absolute deviation (MAD) across 16 arrays within each center  by Irizarray Strong concordance based on rank order indicates that many of the centers are picking the same genes and suggests that calibration of p-values and q-values is contributing to the difference in list lengths. It does not guarantee that the lists are biological in origin. If we had data from only one center, it would be difficult to detect these problems or to determine how many genes are truly differentially expressed.Gene lists established by statistical criteria may vary across centers, but it is possible that they will still support the same biological functions. To test this idea, we selected differentially expressed genes for overall strain and sex effect at q-value<0.05 and identified significant GO terms Another way to assess the similarity among centers is to identify common features in the major patterns of variation. We applied principal component analysis (PCA) to data from each of the five centers and plot the first vs. second Figure 5et al.In this section we address the problem of combining data across centers for the purpose of classification of samples. We used nonnegative matrix factorization (NMF) method, more specifically the metagene projection procedure as proposed by Tamayo First we computed a hierarchical clustering of samples using the original data for all centers together  we were able to extract a biological signal from the combined data, in practice it may be difficult to know when the projection method is effective.We demonstrated that lists of differentially expressed genes vary across centers. Here we consider a possible explanation for those differences. Variation in the overall intensity distribution and MAD Figure 1Supplementary Center 4 hybridized samples in two days; eight samples from B and A1 on the first day and another eight samples from A6 and A15 on the second day. Center 5 hybridized samples over three days; B and A1 on the first, A6 on the second, and A15 on the third day , C57BL/6J-Chr1Supplementary All centers used the one-cycle cDNA synthesis/in vitro-transcription (IVT) method recommended by Affymetrix to amplify and label cRNA targets for array hybridization . Centers 1, 3, 4, and 5 used the IVT reagent kit supplied by Affymetrix while center 2 used the IVT kit supplied by ENZO. Target fragmentation, array hybridization and post-hybridization array processing were performed as per manufacturer's recommendations. However, specific implementation of labeling, hybridization and array processing protocols were according to routine procedures in operation in each center. Several differences in protocol implementation among centers were noted. These include: 1) amount of hybridization solution applied to each array; 2) rotation per minute (rpm) of hybridization oven; and 3) batch design for labeling, hybridization and post-hybridization processing. Details of the protocols used at each center are provided in Data from each center were provided in the form of CEL files. To investigate variability among centers, we normalized and analyzed data from each center separately.et al.Preprocessing to remove background signal and to normalize across arrays was carried out using the Robust Multichip Average method (RMA) et al.Principal Components Analysis (PCA) is used for unsupervised learning. PCA applies a singular value decomposition to the covariance matrix of gene expression data to identify the combination of conditions that explain the greatest variation in the data. PCA was also used by Waring V using two non-negative matrices W and H, V∼WH where W is a weight matrix that indicates how much each gene contributes to each metagene pattern, and H contains the expression profiles of the metagenes.Accumulated data can be used to classify new samples et al.W obtained from a training set is used to compute the metagene profiles of a target data set . Tamayo et al.The metagene data can then be used to cluster genes or samples. Among many implementation of NMF, we adopted the metagene projection procedure as proposed by Tamayo Figure S1Hierarchical clustering of H matrices from metagene projection procedure using center 1, 2, or 5 as training sets.(1.69 MB TIF)Click here for additional data file.Table S1Deatil sample processing information.(0.03 MB XLS)Click here for additional data file.


With the advent of high-throughput technologies for measuring genome-wide expression profiles, a large number of methods have been proposed for discovering diagnostic markers that can accurately discriminate between different classes of a disease. However, factors such as the small sample size of typical clinical data, the inherent noise in high-throughput measurements, and the heterogeneity across different samples, often make it difficult to find reliable gene markers. To overcome this problem, several studies have proposed the use of pathway-based markers, instead of individual gene markers, for building the classifier. Given a set of known pathways, these methods estimate the activity level of each pathway by summarizing the expression values of its member genes, and use the pathway activities for classification. It has been shown that pathway-based classifiers typically yield more reliable results compared to traditional gene-based classifiers. In this paper, we propose a new classification method based on probabilistic inference of pathway activities. For a given sample, we compute the log-likelihood ratio between different disease phenotypes based on the expression level of each gene. The activity of a given pathway is then inferred by combining the log-likelihood ratios of the constituent genes. We apply the proposed method to the classification of breast cancer metastasis, and show that it achieves higher accuracy and identifies more reproducible pathway markers compared to several existing pathway activity inference methods. The introduction of affordable microarray technologies for measuring genome-wide expression profiles has led to the development of numerous methods for discriminating between different classes of a complex disease, such as cancer, through transcriptome analysis One possible way to address this problem is to interpret the expression data at the level of functional modules, such as signaling pathways and molecular complexes, instead of at the level of individual genes. In fact, one of the weaknesses of many gene-based classification methods is that the marker genes are often selected independently, even though their functional products may interact with each other. Therefore, the selected gene markers may contain redundant information, and they may not synergistically improve the overall classification performance. We can alleviate this problem by jointly analyzing the expression levels of groups of functionally related genes, which can be obtained based on transcriptome analysis To use pathway-based markers in classification, we need a way to infer the activity of a given pathway based on the expression levels of the constituent genes. Recently, a number of pathway activity inference methods have been proposed for this purpose. For example, Guo et al. In this paper, we propose a novel method for probabilistic inference of pathway activities. For a given pathway, the proposed method estimates the log-likelihood ratio between different phenotypes based on the expression level of each member gene. The activity level of the pathway is then inferred by combining the log-likelihood ratios of the genes that belong to the pathway. We apply our method to the classification of breast cancer metastasis, and demonstrate that it can achieve higher accuracy compared to several previous pathway-based approaches. Furthermore, we show that the proposed pathway activity inference method can find more reproducible pathway markers that retain the discriminative power across different datasets.We obtained two independent breast cancer datasets from large-scale gene expression studies by Wang et al. To obtain the set of known biological pathways, we referred to the MSigDB (Molecular Signatures Database) version 2.4  For each pathway, we first identified the genes that were included in the expression profiles in the two breast cancer datasets. The genes that were not included in these datasets were removed from the gene set for the given pathway. Consider a pathway that contains In order to compute the LLR value In order to compare the proposed pathway activity inference scheme with other existing methods, we performed the following experiments. In our first experiment, we selected the top 50 differentially expressed pathways using the method proposed by Tian et al. absoluteIn this work, we compared five different pathway activity inference methods: the mean and the median methods We also evaluated the robustness of each inference method in identifying good pathway markers, by ranking the pathways using one of the two breast cancer datasets, and then assessing the discriminative power of the pathways based on the other dataset. Again, In our second experiment, we computed the In order to evaluate the classification performance of the proposed pathway activity inference method, we performed the following cross-validation experiments.within-dataset experiments, the samples in a dataset were randomly divided into five subsets of equal size, where the samples in four of these subsets were used for training the classifier and the remaining subset was used for assessing the classification performance. This has been repeated by using each subset as the test set to obtain more reliable results. The training set was divided again into three equal-sized subsets. Two thirds were used for ranking the pathway markers and building the classifier , and one third of the training set was used for feature selection (the “feature-selection” dataset). All samples in the training set were used to estimate the PDFs of the gene expression values under different phenotypes. To build the classifier, we evaluated each pathway based on the discriminative power of its activity score to classify samples. The pathways were sorted in increasing order of the For cross-dataset experiments, where one dataset was used for selecting the pathway markers, and the other dataset was used for building the classifier based on the selected markers and evaluating its performance. First, we selected the optimal set of features  based on one dataset, by optimizing the AUC metric. The process for selecting the feature set was similar to the one used in the within-dataset experiments. The samples in the other dataset were divided into five subsets of equal size. Four fifths of samples were used to train the classifier using the selected features, and one fifth of samples were used to evaluate the performance of the constructed classifier. We repeated this experiment by using each of the five subsets as the test set and using the rest for training. The above experiment was repeated for 100 random partitions of the entire dataset, and the average AUC over the 500 experiments was reported as the performance measure. It is important to note that feature selection is performed solely based on the first dataset. During the cross-validation experiments using the second dataset, the training set (that consists of four fifths of samples in the same dataset) is simply used to build the classifier based on the preselected set of features. The overall goal of these cross-dataset experiments is to evaluate the reproducibility of the feature set, selected using the proposed pathway activity inference scheme, across different datasets. In order to evaluate the reproducibility of the pathway markers across different dataset, we performed To compare the proposed method with other existing methods, we performed the described within-dataset experiments and the cross-dataset experiments using other pathway activity inference methods . In addition, we also evaluated the performance of a gene-based classifier that uses individual genes as diagnostic markers, following a similar procedure. In this study, we included the top 50 pathway markers in the initial marker set, which were selected according to the method in Tian et al. In this work, we evaluated the performance of a classifier based on the AUC (Area Under ROC Curve). The AUC metric has been widely used for evaluating classification methods, since it can provide a useful summary statistics of the classification performance over the entire range of specificity and sensitivity values. To compute the AUC, we adopted the method proposed in We evaluated the discriminative power of pathway markers, where the pathway activities were inferred using the proposed method as well as other inference methods. For effective comparison of the proposed inference method with other existing methods, we carried out similar experiments as those performed in To evaluate the reproducibility of pathway markers, we ranked the markers based on one dataset and evaluated their mean absolute aset see , lost moNext, we performed similar experiments for all 639 pathways and all 3,271 genes covered by these pathways, without any prescreening see . The resWe used the proposed pathway activity inference scheme for classification of breast cancer metastasis, to evaluate its usefulness in discriminating different cancer phenotypes. For a fair and effective comparison with other inference schemes, we again adopted a similar experimental set-up that was used in The results of the cross-dataset experiments are shown in the two bar charts on the right of We also repeated the entire experiments using LDA , instead of logistic regression, for building the classifiers. The results are shown in Finally, in order to analyze the overall effect of preselecting the initial marker set, we carried out another set of within-dataset experiments, where the initial markers were reselected in every experiment using only the designated training data. The classification results are shown in Ultimately, we want to construct a robust classifier that yields accurate and consistent classification results on independent gene expression datasets. Given two independent datasets of similar size, where one dataset is used for training the classifier and the other dataset is used for evaluation, a robust classification scheme would show consistent classification performance if the training set were interchanged with the test set. However, the USA breast cancer dataset In this paper, we have proposed a novel probabilistic pathway activity inference scheme that estimates the activation level of a pathway based on the log-likelihood ratios (LLRs) of the member genes. The proposed method can effectively address several shortcomings of the previous pathway activity inference methods, thereby improving the discriminative power of the pathway markers. For example, the methods proposed by Guo et al. The proposed LLR-based method provides an effective solution to these problems. First of all, by using the LLR of a member gene, instead of directly using its expression value, the proposed method can capture the consistent gene expression changes that are related to the phenotypic change. Moreover, since the LLR is computed based on the difference in distribution of the gene expression values under different conditions, the direction and the amount of expression changes do not have large effects on the overall discriminative power of the pathway marker. Furthermore, the proposed method fully utilizes the available discriminative information in all the member genes, not just some of them; and it naturally weights and combines the support from each member gene in a given pathway to increase the discriminative power of the corresponding pathway marker. As we have demonstrated in this paper, the LLR-based pathway activity inference scheme significantly improves the discriminative power of the pathway markers, increases the overall classification accuracy, and finds reliable pathway markers that are more reproducible across different datasets. Therefore, the proposed method may ultimately lead to the construction of more reproducible classifiers. The two-fold cross-validation experiments, where we measured the change in classification error that resulted from interchanging the training and test sets, demonstrated the potential of the proposed scheme for building robust and reproducible classifiers.Currently, one limitation of the pathway-based classifiers is the limited coverage of genes by known biological pathways. We believe that the classification performance of the pathway-based methods will be considerably improved once we have a more complete list of biological pathways. One possible way to overcome this problem is to identify effective pathway (or subnetwork) markers by overlaying a protein-protein interaction (PPI) network with gene expression data and searching for significantly differentially expressed regions in the given network, as proposed in