Microarray data analysts commonly filter out genes based on a number of ad hoc criteria prior to any high-level statistical analysis. Such ad hoc approaches could lead to conflicting conclusions with no clear guidance as to which method is most likely to be reproducible. Furthermore, the number of tests performed with concomitant inflation in type I error also plagues the statistical analysis of microarray data, since the number of tested quantities in a study significantly affects the family-wise error rate. It would, therefore, be very useful to develop and adopt strategies that allow quantification of the quality of each probeset, to filter out or give little credence to low-quality or unexpressed probesets, and to incorporate these strategies into gene selection within a multiple testing framework.We have proposed a unified scheme for filtering and gene selection. For Affymetrix gene expression microarrays, we developed new methods for measuring the reliability of a particular probeset in a single array, and we used these to develop measures for a set of arrays. These measures are then used as weights in standard t-statistic calculations, and are incorporated into the multiple testing procedures. We demonstrated the advantages of our methods using simulated data, publicly available spiked-in data as well as data comparing normal muscle to muscle from patients with Duchenne muscular dystrophy (DMD), in which a set of truly differentially expressed genes is known.Our quality measures provide convenient ways to search for individual genes of high quality. The quality weighting strategies we proposed for testing differential gene expression have demonstrable improvement on the traditional filtering methods, the standard t-statistic and a regularized t-statistic in Affymetrix data analysis. The accuracy and sensitivity of the measurement for any one gene or EST depends on the uniqueness and the binding properties of the probes.Affymetrix GeneChip™ microarrays are used to measure gene expression for thousands of transcripts simultaneously. Each transcript is measured by 11–20 probesets, where a probeset consists of two almost identical sequences of length 25 bp. One member of the pair is the perfect match (PM) probe, where the sequence is the exact complement of a section of the mRNA. The other member of the pair, the mismatch probe (MM), is identical to the PM probe except for the nucleotide in 13It is well known that most probesets perform consistently and reliably, in that similar estimates of expression are obtained from two replicates of an experiment. However, in many cases, the signals from a probeset can be hard to interpret. There may be substantial variability across the probesets in the estimated level of expression, or in the PM-MM differences. PM signals can be smaller than MM signals suggesting high levels of non-specific binding. It is also well known that a large proportion of the genes are expressed in only a few tissues or at a particular developmental stage, and hence many of the genes are not expected to have a measurable transcribed product. Su et al.  have estThe latest release for the Affymetrix GeneChip Expression Analysis Platform, GeneChip Human Genome U133 Plus 2.0 Array , may be In 2001, Affymetrix produced a new algorithm for summarizing the results of a probeset, and the algorithm includes a detection p-value, which represents the probability that the probeset (gene) expression is above zero , and measured reliably and consistently across the probe. In addition, "Present/Marginal/Absent" (P/M/A) calls for each transcript are based on the detection p-value together with thresholds that can be altered by the user . The calAn alternative strategy to filtering is to treat probe-specific variability as a quality index to measure the reliability of each probeset. Although many summaries of Affymetrix GeneChip expression have been proposed  follow an exponential distribution with group-specific mean λqw. Under this distribution, the probability that the detection p-values will be smaller than a threshold ν can be written as Instead of using the Affymetrix present/absent calls, more sensitivity can be gained by basing q) is exponential with mean 1.0), when there is a signal, the p-value distribution is better described by the two-parameter Beta distribution. However, detection p-values are based on rank tests and there is often little variability in the p-values across arrays for highly expressed genes. This leads to difficulties in estimating two parameters. Pounds and Cheng [Although, under the null hypothesis of no signal, the detection p-values can be expected to follow a uniform distribution  Weights based on the detection p-values were incorporated directly in the calculation of group-specific means and standard deviations – we refer to this test statistic as PE) test , which i Weights The performance of our proposed methods was evaluated using simulated probe level data generated from a model incorporating probe level effects, optical noise, and non-specific binding, as well as true signals . We haveThe SROC curve's overall behavior can be measured by the Area Under the Curve (AUC) . Table 2-3; Of these, 12 genes (13 probesets) were confirmed by RT-PCR. This set will be referred to as the "RT-PCR" probesets.Haslett et al.  comparedTables The measure The availability of data from spiked-in experiments  provideFigures In traditional gene selection methods, pre-filtering and selection are two separate procedures, and commonly used pre-filtering methods are based on Affymetrix calls -9. We unOur measure nd Cheng  also fouThe best choice of weighting statistic in the presence of adjustments for multiple testing appears to depend on the size of the treatment effects. The exponential model measure, ν. In order to analyze the effect of different values of parameter ν on our results, we did another simulation study. Table ν when the treatment effect was set to 2.0. The sensitivity decreases and specificity increases as ν gets smaller. The sensitivity associated with ν is the one that comes closest to perfect prediction. For ν = 0.05 and 0.01, the Euclidean distances from the point where sensitivity = 1 and specificity = 1 are 0.153 and 0.157, for the exponential distribution, and 0.219 and 0.221, for the beta distribution, respectively. Therefore, we fixed ν = 0.05 in our analysis; this choice can be thought of as a p-value of 0.05, leading to the interpretation that a representative transcript from a particular experimental group will be called present when the expected (under the assumed distribution) p-value <0.05. In fact, the significance level of 0.05 is widely used in statistical hypothesis testing and is comparable to the thresholds for Present/Marginal/Absent calls used in the Affymetrix software.For quality measures ptg have a uniform distribution under the null hypothesis, and we can not assume that the test statistics have identical distributions. Therefore, we developed weighted multiple testing procedures. Our findings showed that the proposed weighted Benjamini and Hochberg (WBH) adjustment procedure is better than the weighted Bonferroni (WB) and weighted Benjamini and Yekutieli (WBY) adjustment procedures (results not shown). However, we also observed that the proposed t-statistics using WB and WBY had poor sensitivity, regardless of the type of quality measure score (data not shown). We are planning to further extend weighted multiple testing methods in order to redefine Storey and Tibshirani's positive false discovery rate (pFDR) [It is now a common practice to use procedures for controlling multiple testing when identifying differentially expressed genes. Various procedures, such as the Bonferroni correction, the Benjamini and Hochberg false discovery rate , or the e (pFDR) , where tSeveral other modified t-statistics, such as SAM , penalizIt should be noted that the probe-level data analysis  may influence the results of the test procedures we discussed. Our initial analysis of real data and spiked-in data show that the weighted test statistics based on the MAS5 strategy may have slightly better performance than those based on the RMA strategy. In future investigations, we plan to explore how different probe-level data analysis methods, such as MAS5, dCHIP, RMA, PLIER and gcRMA, and different detection p-value calculation methods , may infOur results showed that some true differences are missed with any filtering method, as was noted by others . It is kThe proposed quality measures may also be useful in other applications of microarray experiments. For example, in microarray meta-analysis we could weight based on not just the quality of a study , but theThe quality of a probeset may depend on many quantities such as spatial arrangement on arrays, upper and lower threshold effects, etc. Here we focus on measuring reliability and consistency of a probeset's expression using the probeset's detection p-value. The distribution of expression within a probeset, leading to a detection p-value, is influenced by all stages of the microarray process including scanner brightness, background, RNA quality, chip design, etc. . Therefoi = 1, 2, ..., I arrays in total, each containing g = 1, 2, ..., G genes. Further, assume that there are W treatment groups, each consisting of nw arrays, for w = 1, 2, ..., W. Therefore, for the first aspect (a), we measured the quality of the measure of expression for one transcript based on the detection p-value or the Affymetrix Present/Absent/Marginal call [qgi, the detection p-value, and In this paper, we defined quality measures hierarchically at 3 levels Table : (a) gennal call . Two meaQg = maxw∈{1 ...W}[Approaches for summarizing across a set of arrays take two forms. Firstly, the quality of a gene's measurement across a particular group of arrays can be defined; we call this qgi) are expected to follow a uniform distribution. Equivalently, if a gene's expression cannot be detected, then we can assume a common distribution for this gene for all arrays in group w, such that -log(qgi) ~ Exp(1). That is, the negative log(qgi) follows an exponential distribution with a rate parameter value of 1 [qgi for gene g and array i follows the one-parameter exponential distribution with a group specific mean λgw, w = 1, ..., W, so that -log(qgi) ~ Exp(λgw). The maximum likelihood estimate of λgw is the inverse of the group-specific sample mean. Define ν, a sensitivity parameter, to be a desired threshold for the detection p-values, representing 1 minus the probability that any probeset in a particular treatment group shows a detectable signal. Then P(qgiw ≤ ν) leads to qgi may not follow an exponential distribution exactly, this simple assumption may give adequate results for developing quality measures.To develop a model-based quality measure, we argued as follows: if a gene is not expressed or cannot be measured, then the detection p-values , and testing for differentially-expressed genes with the two-sample Welch t-statistic, not assuming equal variances. For gene g, the test statistic is therefore A and B, respectively, and F statistic could be used instead. The quality measure (Qg) for gene g is then incorporated by tgQg. Therefore, tg when Qg = 1, and In traditional meta-analysis, quality measures are often used when combining results from different studies . WithoutA and B. Of course, it is clear that Qg gets closer to zero. However, the next section describes how these altered p-values were used in modified adjustments for multiple testing.We converted this modified t-test statistic to a p-value by reference to a standard t-distribution with degrees of freedom based on Satterthwaite's approximation , assuminqqi. Here, w = A, B. We obtained pqtg from a t-distribution with degrees of freedom based on Satterthwaite's approximation [This simple approach of weighting the t-test statistics can be contrasted with the approach of weighting the expression intensities within the test statistic. Using standard formula for weighting based on quality, we constructed viations , weighteHi are deemed more important than others. For example, Holm [Several authors have considered the problem of including weights in multiple hypothesis testing situations -33. Oftele, Holm  first int* can also be considered as an implementation of a filtering method. For example, t* is close to zero. Therefore, we propose to adjust the effective number of genes tested to correspond to the number of tested genes of high quality.To give another perspective, our quality-weighted statistics Q1, Q2, ..., QG and significance values p1, p2, ..., pG have been calculated for all G genes. Let Assume that quality measures We applied WBH to the t-statistics modified by the quality measures Affymetrix probe level data were generated based on a unified model proposed by Wu et al. . When trYgij and Wgij are the PM and MM intensities for the probe j in probeset g on array i respectively; O denotes optical noise. N represents non-specific binding (NSB) noise. S is a quantity proportional to RNA expression and the coefficient 0 < Φ < 1 accounts for the fact that for some probe-pairs the MM detects signal. Following Wu et al. [2 and log2. We set e ~ N, so that j of gene g is attached by picking up stray signal, Φgi is generated as Φgi ~ Beta, otherwise, Φgi = 0. The proportion of attached probes among total probes was set to 0.01. Since S follows a power law, we set its base to 2. Therefore, if we denote γg as the baseline log expression level for probeset g, we can select log2(γg) expression levels from 0 to 12, which were generated from γg ~ 12* Beta + 1. δg is the expected differential expression of gene g across different conditions, which is varied in the simulations. bi, which describes the need for normalization, was set to be zero. αgij is the signal detecting ability of probe j in gene g on array i, which is assumed to follow a normal distribution with mean zero and signal detection variance N; number of probes within each probeset: 16; proportion of expressed genes: 0.5; proportion of differentially expressed genes: 0.1; signal detection variance: 0.25; multiplicative error variance: 0.05 and sensitivity parameter Haslett et al.  hybridizThe 'spiked-in' experiment  for AffCG initiated, designed and managed the study. CG also proposed the methods in this manuscript. JB participated in designing and managing the study. PH conducted data analysis and drafted the manuscript. CG and JB revised the manuscript. All authors read and approved the final manuscript.


Assessment of array quality is an essential step in the analysis of data from microarray experiments. Once detected, less reliable arrays are typically excluded or "filtered" from further analysis to avoid misleading results.In this article, a graduated approach to array quality is considered based on empirical reproducibility of the gene expression measures from replicate arrays. Weights are assigned to each microarray by fitting a heteroscedastic linear model with shared array variance terms. A novel gene-by-gene update algorithm is used to efficiently estimate the array variances. The inverse variances are used as weights in the linear model analysis to identify differentially expressed genes. The method successfully assigns lower weights to less reproducible arrays from different experiments. Down-weighting the observations from suspect arrays increases the power to detect differential expression. In smaller experiments, this approach outperforms the usual method of filtering the data. The method is available in the limma software package which is implemented in the R software environment.This method complements existing normalisation and spot quality procedures, and allows poorer quality arrays, which would otherwise be discarded, to be included in an analysis. It is applicable to microarray data from experiments with some level of replication. Assessment of data quality is an important component of the analysis pipeline for gene expression microarray experiments ,2. AlthoQuality assessment procedures can be applied at the probe level or at the array level. Probe quality is influenced by local factors on the array such as printing irregularities or spatial artifacts. For spotted microarrays, spot-specific morphology and signal measurements obtained from image analysis software can be used to assign a quality score to each probe on the array -11. SpotProbe quality assessment is not sufficient because some artifacts only become evident at the array level. Indeed the detection of problems is even more critical at the array level than at the probe level because a single bad array may constitute a sizeable proportion of the data from a microarray experiment. The quality of data from an entire array can be influenced by factors such as sample preparation and day-to-day variability . Sub-staAlmost all the methods cited above classify the data as either "good" or "bad", and exclude "bad" probes or arrays from further analysis. In our experience however the "bad" arrays are usually not entirely bad. Very often the lesser quality arrays do contain good information about gene expression but which is embedded in a greater degree of noise than for "good" arrays. In this article, a graduated, quantitative approach is taken to quality at the array level in which poorer quality arrays are included in the analysis but down-weighted.Quality assessment methods can be divided into those which are "predictive" and those which are "empirical". The operational meaning of quality is that high quality features produce highly reproducible expression values, while low quality features produce values which are more variable and hence less reproducible. Predictive quality assessment methods attempt to predict variability by comparing features such as spot morphology to normative measures. On the other hand, methods which compare duplicate spots within arrays are empirical in that they observe variability.In this article we extend the empirical approach to multi-array experiments for which we measure the discrepancies between replicate arrays. In order to be as general as possible, we do not limit ourselves to simple replicate experiments, but work with a linear model formulation which allows us to handle experiments of arbitrary complexity including those with factorial or loop designs. The degree of replication in such experiments is reflected in the residual degrees of freedom for computing the residual standard errors. Our method is implemented by way of a heteroscedastic variance model. It is common for statistical models of microarray data to allow each probe to have its own individual variance. Our heteroscedastic model allows the variance to depend on the array as well as on the probe. The array variance factors then enter into the subsequent analysis as inverse array quality weights. Importantly, our method not only detects variations in data quality but adjusts for this as part of the analysis.Our approach can be combined with predictive quality assessment methods and is an effective complement to them. Predictive methods can be used to filter spots or to provide quantitative prior spot weights which are incorporated into the linear model analysis. However the causes of poor quality data cannot always be clearly identified. The empirical array weight method described here estimates and accommodates any variation in quality which remains after the spot quality weights have been taken into account, i.e., after prediction has achieved as much as it can. Our approach is particularly effective when arrays vary in quality but the problems cannot be isolated to particular regions or particular probes on the offending arrays.The presence of array-level parameters in our heteroscedastic model means that the statistical analysis can no longer be undertaken in a purely gene-wise manner. A naive approach to fitting the model would be computationally expensive. We propose two computationally efficient algorithms for estimating the model by the well-recognised statistical criterion of residual maximum likelihood (REML). These algorithms view the microarray data as many small data sets, one for each probe, with a small number of shared parameters corresponding to the array variance factors. An innovative gene-by-gene update procedure is proposed for particularly fast approximate REML estimation.The array weight method developed here can be applied to any microarray experiment with array-level replication, including experiments using high-density oligonucleotide arrays, but our experience is mainly with experiments using spotted microarrays. High density arrays allow the additional possibility of measuring reproducibility for multiple probes for each gene rather than relying on gene or probe-set summaries . A full In this paper, the linear model approach to microarray data analysis is reviewed and the heteroscedastic model which includes array weights is introduced. Next, the experimental and simulated data sets used in this study are explained and results for these data are presented. The computational algorithms for fitting the heteroscedastic model are then described, followed by discussion and conclusions. Supplementary materials including data, R scripts and additional plots are available .ygj for genes g = 1, ..., G and arrays j = 1, ..., J. The expression values could be log-ratios from two-colour microarrays or summarised log-intensity values from a single-channel technology such as Affymetrix GeneChips. We assume that the expression values have been appropriately pre-processed, background corrected and normalised. The term gene is used here in a general way to include any ESTs or control probes that might be on the arrays. Assume that the systematic expression effects for each gene can be described by a linear modelLinear models provide a convenient means to measure and test for differential expression in microarray experiments involving many different RNA sources ,22. The E(yg) = β Xg     (1)yg = T is the vector of expression values for gene g, X is a known design matrix with full column rank K, and βg = T is a gene-specific vector of regression coefficients. The design matrix will depend upon the experimental design and choice of parameterisation and the regression coefficients represent log-fold changes between RNA sources in the experiment 34]. Writy1, ..., yG; δ1, ..., δ G, γ) = ℓ for given γ. The profile REML log-likelihood for γ isRather than deal with this large dimensional problem, we eliminate the p = ℓWe consider now the nested iteration for maximising the profile likelihood. WriteAlso letg. The derivative of f with respect to γ is simply Ug,γ evaluated at δg = γ from gene g, conditional on δg = be the REML information matrix for gene δg = evaluated at aaa@330D@.p therefore isThe derivative of the profile REML log-likelihood ℓp isand the information matrix associated with ℓδg = γ can be evaluated by the nested scoring iterationevaluated at γi+1) (= γi) (+ Uγ     (13)γi) .where Aγ·δ and Uγ. For any value of γ, the least squares estimator for βg can be computed using weighted least squares computations (Equation 3) with working weights wgj. The standardised residuals from this regression areIn this section, convenient expressions will be derived for the quantities xj is the jth row of X.where Lethgj = hg,jj for the diagonal elements or leverages of Hg. Finally, let Z be the J × J design matrixbe the projection matrix from the regression and write δg is Using these expressions we can write down computable expressions for quantities from the previous section. The conditional REML estimator of γ isThe score vector for Z2 is the last J - 1 columns of Z and zg is the vector with componentswhere j = 1, ..., J. The information matrix isfor Vg is the J × J matrix with diagonal elements  . As already noted, the REML algorithm is invariant with respect to the gene-wise means and variances, so the gene-specific mean and variance parameters were set to zero in our simulations.While the gene-by-gene update algorithm is fast, it provides only an approximation to the REML estimators Figure This article has presented an empirical method for estimating quantitative array quality weights which is integrated into the linear model analysis of microarray data. Computationally efficient algorithms are developed to compute the array quality weights using the well-recognized REML criterion. As well as full REML estimation, a fast gene-by-gene update method which requires only one pass through the genes is described.t-statistics. In the data example, filtering resulted in no significant genes to follow up, whereas the weighted analysis provided a few hundred sensible candidates.Examples of array quality weights which give less influence to the gene expression measurements from unreliable microarrays and relatively more influence to the measurements from reproducible arrays have been presented. In both simulated and real data examples, it has been demonstrated that array weights improve our ability to detect differential expression using standard statistical methods. The graduated approach to array quality has also been shown to be superior to filtering poor quality arrays both in simulations and for an experimental data set. In the simulations, filtering is shown to perform quite poorly, especially in combination with ordinary The method is restricted for use on data from experiments which include replication with at least two residual degrees of freedom. For simple replicated experiments, a minimum of three arrays are needed and results from simulation studies show that this method is reliable in these situations, even in the presence of non-normally distributed data. Simulations were also used to show that array variance parameters are estimated with greater accuracy when more genes are available for the gene-by-gene update algorithm, and that these computational savings do not seriously compromise the accuracy of the final estimates. As a rule of thumb, we recommend that the full REML array weights be used when there are fewer than 1000 probes and that the gene-by-gene update method be used otherwise. The analysis of the control probes from the QC LMS data set showed that useful array weights can be obtained from the gene-by-gene algorithm with as few as 120 genes. The situation is different when there are no spot weights or missing expression values in the data. In this case the full REML algorithm can be implemented very efficiently and so is recommended for any number of probes.(wgj = 0) presents no problems for the method, although some special numerical treatment not discussed here is needed to ensure the sum to zero constraints are satisfied.The empirical array weights form part of the quality and analysis pipeline and are not intended to replace the usual background correction, normalisation and quality assessment steps. In particular, array weights are not designed to account for spot-specific problems. The array weights method is instead designed to incorporate spot quality weights which might arise from gene filtering or from a predictive quality assessment step. The use of zero weights as prior weights The array weight approach is also not intended to replace diagnostic array quality plots such as MA-plots, and arrays which are catastrophically poor quality should still be discarded. Taking a graduated approach to array quality, does however allow arrays of less than ideal quality, which would otherwise have to be discarded, to be kept in the analysis, but down-weighted.The authors have applied the array weight method to very high quality data sets which featured arrays with low background, well-behaved controls and a good dynamic range of spot intensities. For such data sets, the method assigns approximately equal array weights to each array (data not shown). This indicates that the method does no harm when it is not required.One further topic that deserves some attention is the use of robust linear models to estimate the gene expression coefficients. The array weights method has the same motivation as robust regression methods, but accumulates information on variability across genes on each array, which gene-wise robust regression methods are unable to do. Another consideration is sample size. While robust methods perform well on large sample problems, many microarray data sets such as the METH experiment consist of a small number of arrays and, in these situations, robust methods may not be suitable.MER performed the data analyses, coded the algorithms and drafted the manuscript. GKS suggested the model and algorithms, helped with the coding and analyses and finalised the manuscript. The arrays from the QC data set were manufactured and hybridised by DD, RvL and AH. The METH experiment was planned and conducted by JN and AD.


Like microarray-based investigations, high-throughput proteomics techniques require machine learning algorithms to identify biomarkers that are informative for biological classification problems. Feature selection and classification algorithms need to be robust to noise and outliers in the data.We developed a recursive support vector machine (R-SVM) algorithm to select important genes/biomarkers for the classification of noisy data. We compared its performance to a similar, state-of-the-art method (SVM recursive feature elimination or SVM-RFE), paying special attention to the ability of recovering the true informative genes/biomarkers and the robustness to outliers in the data. Simulation experiments show that a 5 %-~20 % improvement over SVM-RFE can be achieved regard to these properties. The SVM-based methods are also compared with a conventional univariate method and their respective strengths and weaknesses are discussed. R-SVM was applied to two sets of SELDI-TOF-MS proteomics data, one from a human breast cancer study and the other from a study on rat liver cirrhosis. Important biomarkers found by the algorithm were validated by follow-up biological experiments.The proposed R-SVM method is suitable for analyzing noisy high-throughput proteomics and microarray data and it outperforms SVM-RFE in the robustness to noise and in the ability to recover informative features. The multivariate SVM-based method outperforms the univariate method in the classification performance, but univariate methods can reveal more of the differentially expressed features especially when there are correlations between the features. Accurate classification of patients with complex diseases such as cancer is crucial for successful treatment of the diseases. High-throughput proteomics techniques based on mass spectrometry (MS) have made it possible to investigate proteins over a wide range of molecular weights in a style similar to gene expression studies with DNA microarrays. The advancement of these techniques has generated many analytic challenges, among which a central task is to discover 'signature' protein profiles specific to each pathologic state  or differential profiles between experimental conditions (e.g drug responses) from high-dimensional data . The tecLaser Desorption/Ionization Time-of-Flight Mass Spectrometry (SELDI-TOF-MS)  has beenper se. We suspect that failing to do so may be a reason why good results published from one investigation may not be reproduced by other investigations.Like in many other biological studies, a key difficulty in such high-throughput studies is the noisy nature of the data, which can be caused by the intrinsic complexity of the biological problems, as well as experimental and technical imperfections. Another difficulty arises from the high dimensionality of the data. Similar to the situation in microarray studies, typically one proteomics investigation only involves several tens of samples but the measured points on the mass spectrum can be in the thousands or more. Even after pre-processing steps such as peak and/or biomarker detection, the dimensionality is usually still larger than or comparable to the sample size. This makes many standard pattern classification algorithms fail. For those that do work theoretically, there is a high risk of overfitting due to the small sample size. Thus, there is an algorithmic need for feature selection in addition to the biological need of discovering a manageable set of key molecular factors (genes or biomarkers) behind the disease. As observed in  and otheGuyon et. al.  proposedR-SVM and SVM-RFE represent typical machine-learning-based multivariate approaches for the task. Conventional univariate methods are also frequently used for feature selection and classification. We compared the SVM-based methods with the weighted-voting (WV) method  on simulN for class 1 and N for class 2. The rest 700 "uninformative" genes follow independently N for both classes. For each simulation experiment, we generated a training set of 100 samples (50 for each class) and an independent test set of 1000 samples (500 for each class). The two types of simulated data were generated by adding noises to this general model in different ways. The first type (Data-G) mimics the situation where some of the gene expressions in some samples are outliers. For the informative genes, we randomly chose 5% of expression values in all samples as outliers by making them to follow N for the class-1 sample and N for the class-2 sample. The second type of simulated datasets (Data-S) is constructed to contain 5% "outlier samples," which were made by randomly picking 5% of the samples and increasing the standard deviation of every gene in these samples by 10 fold. We did 100 simulations for each type of the data (Data-G and Data-S).We generated three types of simulated data to investigate the characteristics of the feature selection and classification methods. The basic simulation model for the first two types of data is as follows: Each sample contains simulated expression values of 1000 genes. Among all the genes, 300 are "informative" ones, each following independently the Gaussian distribution Data-R in our experiments.The above two simulation models are over-simplified in many aspects. To mimic more realistic situations, we generated the third type of simulated data based on a real human breast cancer microarray dataset obtained with Affymetrix U133 Plus 2.0 microarrays. The dataset originally contains 78 estrogen receptor positive (ER+) cases and 54 estrogen receptor negative (ER-) cases. The data were normalized by the GCRMA algorithm ,11, and We applied the two SVM-based methods to two real SELDI-TOF-MS proteomics datasets. The first dataset is from a rat model used to discover serum biomarkers of liver cirrhosis . It contWe first compared the performance of R-SVM to SVM-RFE on Data-G and Data-S. For each of these datasets, we applied R-SVM and SVM-RFE to perform gene selection, built SVM models on training data with selected genes, and tested the models on the independent test data. Experiments were done 100 times for each data type. The following aspects of performances are inspected at each level of gene selection: number of SVs  used in the SVM model, test error, the percentage of true informative features recovered in the selection, and for Data-S the number of outlier samples used as SVs. Tables It can be seen on both Data-G and Data-S that at most of the selection levels, especially at those lower levels (with fewer selected genes), the number of SVs used by R-SVM is 5 %~8 % fewer than that of SVM-RFE, indicating the better generalization ability of R-SVM. One can also see that at the same selection level, R-SVM recovers significantly more of the informative genes than SVM-RFE, and the improvement is about 3 %~7 % at lower selection levels. These two factors can explain the observation that independent test errors of R-SVM were significantly lower than that of SVM-RFE (by 5–10%) at lower selection levels. On the Data-S with outlier samples, R-SVM also shows a better ability to avoid taking the outlier samples as SVs (R-SVM reduces up to 94% of the outlier SVs than SVM-RFE).As simulation models for Data-G and Data-S are too simplistic, we compared on Data-R and the results are shown in Table We applied R-SVM and SVM-RFE to the two real SELDI-TOF-MS datasets. Table The top 6 biomarkers selected by R-SVM were reported for further biological investigation. They are listed in Table The results of R-SVM and SVM-RFE on the breast cancer dataset are listed in Table The minimal CV error rate 17.6%) is achieved at the 8-feature level by R-SVM. These 8 markers and their t-statistics and ROC statistics are listed in Table .6% is acWe also applied the Random Forest (RF) method  to this Many researchers believe that it is beneficial to use multivariate methods to analyze microarray and proteomics data as genes usually work in collaborative ways rather than as independent factors. However, univariate methods are still useful for identifying differentially expression features and continue to play important roles in many applications. Therefore it is worthwhile to compare the performances of the SVM-based methods with more conventional univariate approaches, of which we take the weighted-voting (WV) method as a representative in this work.Tables all genes that are informative, new strategies will be needed for improving SVM-like methods on this aspect.Table a priori. We expect that in more complicated situations when combinatorial effects play major roles, SVM-based methods will perform even better.We should note that all our simulation designs  are favorable to univariate methods since the true underlying classification information is in the differential expression of individual genes. The models haven't consider any collaborative effects (not even additive effects), thus they bias in favor of the single-variable approach High-throughput genomics and proteomics data open a new route to the classification of complex diseases. Machine learning methods for feature selection and classification have been playing active roles in analyzing such data. We compared two similar methods, SVM-RFE and R-SVM, both adopting recursive procedures to select features according to their importance in SVM classifiers. The major difference between the two methods is the criteria used for evaluating the contribution of genes. Although the two methods did not differ significantly in their cross-validation performances, it appeared that R-SVM is more robust to severe noise and outliers and can recover more informative genes. The successful application of R-SVM on the rat liver cirrhosis SELDI data and human breast cancer SELDI data show that the proposed strategy can help to identify biologically important markers.We compared R-SVM with a representative univariate method, the weighted-voting method on simulated data. Although the comparison is limited in scope , some interesting insights regarding their respective strengths and weaknesses are observed. The SVM-based method performs better in terms of the classification accuracy, but univariate methods can reveal more of the differentially expressed individual features. A more systematic comparative study of the univariate and multivariate methods can be helpful for better understanding the nature of the methods and problems.AR code package of the R-SVM method and a Linux-based executable package are freely available.Project name: R-SVM (R and Linux versions)Project home page: Operating systems: R version: Windows XP, Linux; Linux version: Linux.Programming languages: R, C/C++Other requirements: R for the R version; SVMTorch (provided) for the Linux versionLicense: freeN. We randomly assigned the samples into two classes ("fake-classes"). Since the data set is totally non-informative, the faithful CV error should be around 50% no matter what method is used. But by CV1 scheme we could achieve a CV error as low as 0.025 after recursive feature selection, which shows that the bias caused by the improper cross-validation scheme can be surprisingly large. A more proper approach is to include the feature selection procedure in the cross validation, i.e., to leave the test sample(s) out from the training set before undergoing any feature selection. In this way, not only the classification algorithm, but also the feature selection method is validated. We call this scheme CV2 and use it in all of our investigations throughout. For the above "fake-class" data, the error rate evaluated by CV2 was always around 50% regardless of the specific method used for feature selection and classification.Since an independent test set is not available in many investigations, cross-validation  is often used to assess the accuracy of classifiers. It should be noted that feature selection results may vary with even a single-case difference in the training set when the sample size is small. In some literature, feature selection steps were external to the cross validation procedures, i.e., the feature selection was done with all the samples and the cross-validation was only done for the classification procedure. We call this kind of cross validation CV1, with examples including ,9,18-21.As a powerful and popular multivariate machine-learning method, SVMs have been widely used in biological classification problems. The key idea of the SVM -27 is tox is the gene/protein expression vector of a sample, xi is that of sample i in the training set , yi ∈{+1,-1} is its corresponding class label,  is the vector of weights of the features, and b is a scalar offset. The ai 's and b are estimated from the training set. Only those samples closest to the separating boundary  have non-zero ai 's, and therefore the function f(x) is a linear combination of only the SVs. For a new sample x, the sign of the decision function f(x) predicts the class it belongs to, and the absolute value of f(x) represents how far the sample is from the boundary. Theoretical investigations show that the proportion of SVs in the training set reflects an upper bound of the expected generalization error (error rate of predictions on future samples) /[σ1(g) + σ2(g)], (8)μ1(g), σ1(g)] and  denote the means and standard deviations of the expression levels of the gene for the samples in class 1 and class 2, respectively. The larger the absolute value |cg| is, the more important the gene is for predicting the class memberships. The genes are ranked by their |cg|'s and the top ones are selected. The class predictor is trained on the set of training samples with the selected genes. For predicting the membership of a new sample x with expression xg of gene g, the vote of gene g is vg = cg (xg - bg) where bg = [μ1(g) + μ2(g)]/2. A positive vote means a vote for class 1 and a negative vote is for class 2. The sign of the total vote  is used as the final predictor.where [XZ and WHW initiated this project. XZ invented the basic strategy of R-SVM with the help of WHW and they did the study on the two schemes of cross-validation. XZ developed the initial version of the algorithm. XL added the voting scheme, wrote the codes implementing the current algorithm, and did the simulation experiments and most of the computation on the proteomics data. XL and JSL derived the theoretical relationship between R-SVM and SVM-RFE. JSL and WHW guided XZ and XL on analyzing the experiments, comparing the methods, and improving the algorithms. QS, LNH, JDI and AM produced the breast cancer proteomics data and worked on the biological analysis of the results. XX and HEL produced the rat liver cirrhosis data and did the biological analysis and follow-up validation of the results. XZ, XL and JSL executed the writing with input from all authors.


Oryza. sativa) subspecies japonica and indica, respectively. The non-exonic TARs account for approximately two thirds of the total TARs detected by tiling arrays and represent transcripts likely conserved between japonica and indica. Transcription of 21,018 (83%) japonica non-exonic TARs was verified through expression profiling in 10 tissue types using a re-array in which annotated genes and TARs were each represented by five independent probes. Subsequent analyses indicate that about 80% of the japonica TARs that were not assigned to annotated exons can be assigned to various putatively functional or structural elements of the rice genome, including splice variants, uncharacterized portions of incompletely annotated genes, antisense transcripts, duplicated gene fragments, and potential non-coding RNAs. These results provide a systematic characterization of non-exonic transcripts in rice and thus expand the current view of the complexity and dynamics of the rice transcriptome.Genome tiling microarray studies have consistently documented rich transcriptional activity beyond the annotated genes. However, systematic characterization and transcriptional profiling of the putative novel transcripts on the genome scale are still lacking. We report here the identification of 25,352 and 27,744 transcriptionally active regions (TARs) not encoded by annotated exons in the rice ( Oryza sativa) genome as a model for the Gramineae, the group that includes all major cereal crops, has produced draft sequences of the japonica and indica subspecies japonica genome sequence assembly covers over 95% of the genome ab initio gene prediction, comparative genomics, and a variety of other computational methods Efforts to generate sequence from the rice  in the intergenic regions Because genome tiling arrays provide end-to-end coverage of the target genome and measure transcriptional activity from multiple probes, they are capable of detecting the transcriptome in a comprehensive and relatively unbiased way. Thus, in the context of rice genome annotation, tiling array analysis can be utilized to verify predicted gene models and to identify novel transcription units japonica and indica genome tiling data. We identified 39,018 and 42,470 TARs from japonica and indica, respectively. About two thirds of the TARs, were detected at corresponding locations in the two rice subspecies and do not intersect with annotated exons (referred to as non-exonic TARs). We verified transcription of the japonica non-exonic TARs by using a re-array in which each TAR and gene model was represented by five probes to profile 10 different rice tissue types. The re-array results were consistent with other experimental evidence and validated >83% non-exonic TARs detected in by the tiling array. Through genome-wide bioinformatic characterization, we assigned approximately 80% of the non-exonic TARs to various putative functional or structural elements of the rice genome.In the present study, we performed systematic searches and characterization for TARs in the rice genome, taking advantage of the parallel japonica transcriptome started with custom full-genome tiling microarrays containing 12,254,374 36-mer oligonucleotide probes, with an average space of 10 nucleotides (nt) between adjacent probes. The probes tile both DNA strands of the non-repetitive sequences of the genome and were synthesized in a set of 32 arrays Our characterization of the −10). Comparing the non-exonic TARs with rice organelle genome sequences, which have inserted into and contribute to ∼0.4% of the nuclear genome We next scored genome-wide transcription blind to the annotated gene models and described a transcriptome consisting of 39,018 unique TARs see . Of thesjaponica and indica rice, we re-analyzed the previously reported indica genome tiling data using the same method reported in the present study. Our re-analysis identified 42,470 indica TARs, of which 27,744 (65%) were non-exonic  and indica (8925) and supported by full-length cDNA sequences. These common gene models presumably represent the most conserved portion of the rice transcriptome. Within our data sets there were 3449 japonica and 4230 indica exonic TARs derived from 2808 and 3461 confirmed shared gene models, respectively. We found that 1595 (46%) japonica TARs overlapped with 1599 (38%) indica TARs . To experimentally validate transcription of the non-exonic TARs and to describe their transcriptional regulation, we constructed a new array (designated the re-array) using five independent 36-mer probes to represent each of the 44,385 non-TE gene models and 25,313 TARs in the japonica genome  and 21,018 (83%) non-exonic TARs was detected in all and at least one organ types, respectively. Across organ types, detection rates of TARs ranged from 43% (seedling root) to 79% (flag leaf) . We furtr = 0.92), suggesting that the transcriptional activity represented by non-exonic TARs reflects the activity of the transcriptome and is thus a worthy target for future transcriptome profiling efforts.We determined which genes and non-exonic TARs that were differentially expressed among the 10 assayed organ types. While 4232 (9.5%) genes were differentially expressed between organ types, fewer  non-exonic TARs than genes were differentially expressed across the organ types assayed . Together = 0.97). This prompted us to investigate whether TARs in the first introns, which are the largest and have the most TARs (358) among all introns, could represent alternative first exons that have been reported to contribute substantially to transcript diversity in mammals but only sparsely documented in plants Introns of the rice genome were relatively depleted of transcripts represented in the form of TARs . This isr>0.7) with the corresponding gene model. As many of the rice gene models were only annotated for their coding regions There were 15,472 TARs in the intergenic regions  of whichcis-NATs. Antisense TARs exhibited slightly higher re-array detection rate than TARs on average. For example, 8256 of 8262 anti-exonic TARs were represented in the re-array of which 7061 (85%) were detected  have been broadly implicated in regulating gene expression and chromosome structure detected . These or<−0.4)  . This grr<−0.4) , manifesBecause antisense transcripts can function through small interfering RNA (siRNA) derived from paired sense–antisense transcripts Mutator-like transposable elements carrying fragments of genes, so called Pack-MuLEs, are a characteristic of plant genomes and may be involved in the creation of new genes during evolution We examined the intersection of TARs with elements of the genome other than the annotated protein-coding genes. Chimeric −10). Although it is possible that the detected transcriptional activity of Pack-MuLEs were derived from close homologs elsewhere in the genome, the strong symmetric transcription pattern argues against this possibility. The symmetrical pattern of transcription suggests that Pack-MuLE loci could be a rich source for trans-regulatory RNAs (i.e. double-stranded or trans-antisense) that can potentially affect expression of the original genes We found that 109 TARs intersected 96 ORFs encoded by Pack-MuLEs whilst 112 were located on the antisense strand of 97 ORFs. Among these, 48 ORFs were intersected by both sense and antisense TARs. Such a strong symmetric transcription pattern of Pack-MuLEs consisting of equal number of sense and antisense TARs is in keeping with the observation that most Pack-MuLE derived FL-cDNA were initiated within or near the terminal inverted repeats (TIRs) The intergenic TARs, distal to a gene model and not overlapping with other elements of the genome, numbered ∼8400 in total and do not appear to encode proteins. Two lines of evidence supported conclusion. First, the linear relationship between the GC content of the second (GC2) and the third (GC3) codon positions of the longest deduced ORF, of the six possible translations, from these TARs deviated from that of known genes The non-exonic TARs were compared to putative non-coding RNA (nc-RNA) identified using a high resolution (5-bp) tiling array. This array, which represents a 1Mb region of Chromosome 10, was hybridized to mRNA- or small RNA-derived cDNA samples. Hybridization signals from labeled mRNA-derived cDNA matched known transcripts with high fidelity . When thp<e−15) than that for randomly shuffled TAR alignment blocks, for which only ∼6% are predicted to form structural nc-RNA.Some TARs with corresponding small RNA derivatives were predicted to fold into stable secondary structures in which the nc-RNAs were located near stem-loop motifs . It is ajaponica genome that represent conserved transcripts between japonica and indica rice. We verified transcription for 83% of these TARs through expression profiling and assigned more than 80% TARs to various putative functional or structural elements of the rice genome  using the leaf clip method japonica rice genome, based on the TIGR Rice Pseudomolecules release 1 (September 2003) japonica rice Chromosome 10. A tile path was designed with 36-mer probes at a step of 5 nt.Three different types of microarrays were used in this study: the full-genome tiling arrays, a 1Mb high-resolution tiling array, and a re-array. All arrays were fabricated at a density of 389,000 oligonucleotides per array as previously described th percentile and met most of the above criteria (hence the name ‘re-array’). 10,000 randomly selected probes from the genome tiling arrays and 9000 probes in the introns of PASA models with intensities below the median were also included as controls.We constructed the re-array in which five probes were used to represent each gene model and non-exonic TAR. To ensure that the probes are specific to its target and are free of secondary structures, we used the software OligoArray 2.1 to select probes Xoo-infected leaf, flag leaf, whole flower (floret), carpel, developing seed, and suspension cultured cell.The full-genome tiling arrays and the 1Mb tiling array were hybridized to cDNA mixtures derived from seedling root, seedling shoot, panicle and suspension-cultured cells. Note that the cDNA mixture was used as the template for RT-PCR analysis unless otherwise indicated. Additionally, the 1Mb tiling array was hybridized to targets derived from the non-coding RNA Sample I and II. The re-array was hybridized to three biological replicates of cDNA derived from 10 rice tissue types, namely, light-grown seedling, dark-grown seedling, seedling root, leaf, We were able to map >95% of the 12,254,374 probes to the latest TIGR Rice Pseudomolecules  at the time data analysis was initiated. We used the gene model annotation provided in the form of a .gff file distributed by TIGR We used the sign-test to score annotated gene models th percentile japonica and indica genome, respectively. To match non-exonic TARs to rice Unigenes (sets of transcript sequences derived from the same transcription unit), which were downloaded from NCBI (http://www.ncbi.nlm.nih.gov/UniGene/UGOrg.cgi?TAXID = 4530), a cutoff of E≤e−10 was used.To identify TARs, the tile path was scanned for regions in the genome that are covered by at least five consecutive probes whose intensities lie in the 80http://plantgenomics.biology.yale.edu/nmpp. To correct potential skewed signal distribution within each array due to uneven hybridization or washing, we applied a global distance-weighted spatial smoothing to the raw probe intensities based on MAS 5.0 We developed the NMPP package as a bundle of user-customized tools based on established algorithms and methods to process the re-array data 2, Of the 19,000 control probes included in the re-array, 7669 have a GC content higher than 15, which is the case for all probes interrogating transcription units. These 7669 probes were used as the control for subsequent analysis of the re-array data to minimize the impact of probe GC content on transcription calling. To determine from the control probes the noise distribution, we used a previously reported modeling process http://www.jax.org/staff/churchill/labsite/software/download.html). An F-test was performed to detect differential expression of gene models or non-exonic TARs in the 10 tissue types. Next, FDR-adjusted p values were calculated from permuted samples. A significant level above 99% (p<0.01) was selected for tissue-specific expression calling of gene models or TARs.Determination of differentially expressed transcription units was carried out using standard procedures. Briefly, normalized intensity values were fitted to a mixed effects ANOVA model  and the standard deviation (σG) of all intensity values were calculated and probes with intensity greater than μG+2σG were marked as group I probes. Next, a new mean (μG′) and a new standard deviation (σG′) were calculated excluding the group I probes. The Z scores for the ith probe is given by the equation zi = (xi-μG′)/σG′. Probes with a Z score greater than two were considered signal probes. Sequences of neighboring signal probes were joined to form contigs. Putative non-coding transcripts were identified from the contigs with a minimal length of 55 nt (sample I) or 65 nt (sample II).We converted probe intensity values from the 1 Mb tiling array into Z scores using a two-step process. First, the mean . Clusters of TARs and sequence homologs were aligned using the CLUSTALW algorithm to generate multiple sequence alignment blocks suited for RNAz analysis RNA secondary structures were predicted using RNAfold. To verify the secondary structures, we used 15,000 TARs sequences to BLAST search five plant Unigene databases  downloaded from NCBI. This search identified 708 hits Click here for additional data file.Figure S2Density of different sets of transcription units along the 12 rice chromosomes. The number of transcription units was calculated in 500 Kb-sliding-windows with a 100 Kb step. The black triangle sign in each panel indicates the position of the annotated centromere of the corresponding chromosome.(1.29 MB PDF)Click here for additional data file.Figure S3Comparative analysis of TARs. (a), Comparison of 5464 indica TARs with transcription units in japonica. The indica TARs were identified in a previous genome tiling analysis of the indica genome (Ref.2). Exon, indica TARs matched with exons of japonica genes, Antisense Exon, indica TARs mapped to the antisense strand of japonica genes, TAR, indica TARs matched with japonica novel TARs, Antisense TAR, indica TARs mapped to the antisense strand of japonica novel TARs, Unmatched, indica TARs do not match with any japonica transcription unit, Unmapped, indica TARs not mapped to the japonica genome. (b), Classification of 42,470 indica TARs based on their physical relation with annotated indica gene models. The 42,470 TARs were identified by re-analyzing the previous indica genome tiling data (Ref.2) using the same criteria described in (0.25 MB PDF)Click here for additional data file.Figure S4Re-array data processing. (a) Normalization of the re-array hybridization data. Spatial-effect smoothed probe intensities were normalized following a two-step process. First, quantile normalization was performed on the three replicates within each assayed tissue type. Next, a global scaling was carried out to adjust the quantile-normalized intensities to a common baseline. (b) Modeling the 7669 control probes. To determine from the control probes the noise distribution, we used a previously reported modeling process (Ref.1). Assuming the control probes that have very low intensities (1-9 Log2) were primarily noise, we modeled the noise as a normal distribution by mirroring the low-intensity portion of the overall control probe distribution. (c) Transcription threshold determination for individual tissue types. FDR = 0.05, which represents the 95% confidence in the modeled noise distribution, was used as a cutoff for transcription calling.(1.25 MB PDF)Click here for additional data file.Figure S5Transcriptional verification of non-exonic TARs. (a) Re-array detection of different groups of TARs. Detection rate was calculated as the percentage of TARs detected in at least one of the 10 assayed tissue types by the re-array. All, all non-exonic TARs; +Hit, TARs match with rice Unigenes; -Hit, TARs do not match with Unigenes. (b) RT-PCR analysis of selected non-exonic TARs. Pie charts indicate the percentage of TARs generated a PCR product of the expected size in an RT-dependent manner. PCR for 36 TARs from each group  was performed on reverse transcribed cDNA (+RT) and mRNA (-RT) and resolved on gel side by side. A typical gel from each group is shown. Top, TARs detected by the re-array and have hits against Unigenes; middle, TARs detected by the re-array but have no Unigene hits; bottom, TARs not re-array detected and have no Unigene hits. M, 100 bp molecular weight marker.(0.88 MB PDF)Click here for additional data file.Figure S6Analysis of the coding potential of intergenic TARs. (a) Frequency of the amino acids contained in the deduced protein sequences of rice FL-cDNA, TE and non-TE gene models, and intergenic TARs. (b) Pie chart analysis of the hits against the ProSite database using deduced polypeptide sequences from the intergenic TARs.(0.22 MB PDF)Click here for additional data file.Figure S7http://rice.tigr.org/.High-resolution tiling array analysis of mRNA target. (a) A region from rice Chromosome 10 represented by the tiling array. (b) Tiling array profile showing each probe aligned to the chromosomal coordinate with its fluorescence intensity depicted as a vertical line. (c) TIGR working gene model from the locus LOC_Os10g21910, which encodes a putative acetyl-coenzyme A carboxylase ACC1A. (d) Gene predictions and experimental evidence supporting the gene model. Red bars indicate significant matches with the working gene model. Data downloaded from the TIGR rice genome annotation database at (1.28 MB PDF)Click here for additional data file.


Stochastic dependence between gene expression levels in microarray data is of critical importance for the methods of statistical inference that resort to pooling test-statistics across genes. It is frequently assumed that dependence between genes (or tests) is suffciently weak to justify the proposed methods of testing for differentially expressed genes. A potential impact of between-gene correlations on the performance of such methods has yet to be explored.t-statistics associated with different genes. We report the effects of four different normalization methods using a large set of microarray data on childhood leukemia in addition to several sets of simulated data. Our findings help decipher the correlation structure of microarray data before and after the application of normalization procedures.The paper presents a systematic study of correlation between the t-statistics. By using normalization methods it is possible to significantly reduce correlation between the t-statistics computed for different genes. Normalization procedures affect both the true correlation, stemming from gene interactions, and the spurious correlation induced by random noise. When analyzing real world biological data sets, normalization procedures are unable to completely remove correlation between the test statistics. The long-range correlation structure also persists in normalized data.A long-range correlation in microarray data manifests itself in thousands of genes that are heavily correlated with a given gene in terms of the associated  The empirical Bayes methodology in the nonparametric [p-values) are pooled together and treated as a sample from which to estimate the sampling distribution of this statistic, the false discovery rate (FDR), q-values, etc. The same kind of pooling is typically used in maximum likelihood inference from microarray data [There are two major methodological problems that deal with the issue of stochastic dependence between gene expression signals in microarray data. The first arises naturally when adjustments for multiplicity of tests are made by rametric -3 and parametric ,5, and crametric -8 represray data ,10 and sIn all such approaches, the stochastic dependence between gene expression values or test statistics is a nuisance that hinders their application. The independence assumption is frequently invoked when building a theoretical foundation for a particular method of statistical inference. Some authors  allow fThe stochastic dependence between expression levels and thus between the associated test statistics is really a serious problem. It may cause high variability of statistical estimators and even deteriorate their consistency. To obtain theoretical results it is frequently assumed that weak or almost sure convergence holds for an empirical distribution function constructed from the data pooled across genes . HowStorey  advocate"I hypothesize that the most likely form of dependence between the genes encountered in DNA microarrays is weak dependence, and more specifically, "clumpy dependence"; that is, the measurements on the genes are dependent in small groups, each group being independent of the others. There are two reasons that make clumpy dependence likely. The first is that genes tend to work in pathways, that is, small groups of genes interact to produce some overall process. This can involve just a few to 50 or more genes. This would lead to a clumpy dependence in the pathway-specific noise in the data. The second reason is that there tends to be cross-hybridization in DNA microarrays. In other words, the signals between two genes can cross because of molecular similarity at the sequence level. Cross-hybridization would only occur in small groups, and each group would be independent of the others."This hypothesis does not seem plausible from a biological standpoint because of the pleiotropic character of gene function: one gene participates in multiple molecular pathways. However, the possibility that it may approximately be true for all practical purposes cannot be ruled out. There are two key words in the above quotation: "small groups" and "weak dependence". Whether or not such groups are small and stochastic dependence is suffciently weak can be deciphered only from real world data. To the best of our knowledge, no attempt has been made so far to systematically study dependence structures in microarray data using large data sets. In this connection we would like to continue quoting from : "Many aThe second research area where the dependence between gene expression levels plays a crucial role is the discovery (reverse engineering) of molecular pathways and networks from microarray data . A popult-statistic which is the most popular choice in microarray data analysis. Normalization is intended to mitigate the effect of technological noise that is inherent in microarray data. Normalization procedures tend to reduce the variability of original microarray data  Database on childhood leukemia which falls into this category. Computer simulations provide the necessary, albeit not very realistic, control where the actual model is known and arbitrarily large samples can be generated for testing various methodologies.The present paper is focused on the correlations between test-statistics associated with expression signals produced by each gene and the effects of normalization procedures on these correlations. We limit our consideration to the k et al, ), howevet-statistics (across arrays) for each gene and each pair of subsamples. This computation results in 15 values (corresponding to the 15 pairs of subsamples) of the t-statistic associated with each gene. Then we compute the sample correlation coeffcients between the t-statistics thus obtained for every pair of genes. The resulting coeffcients are summarized in the form of a histogram. We interpret such histograms as pertinent summary characteristics and not as estimators of some population distribution densities. We also look at pre-selected individual genes to determine the range of their correlation with all other genes. This range can be characterized by the number of gene pairs formed by a given gene with the correlation coeffcient exceeding some threshold level. We adopt the value of 0.5 as such a threshold.The design of our study is presented in the Methods section. This design allows us to compute the Using these tools we attempt to answer the following questions:t-statistic in a large population of genes?• What is the (pairwise) correlation structure of the • What is the impact of normalization procedures on this structure?• What is the impact of normalization procedures on the number of highly correlated pairs formed by a given gene?t-statistics estimated from the SJCRH leukemia data for all pairs of genes. It is clear that the distribution is heavily shifted towards high positive correlation between the genes. In particular, more than 36% of pairs have their sample correlation coeffcients higher than 0.75 and only 7.6% have the coeffcients smaller than 0.25. The proportion of gene pairs with correlation coeffcients greater than 0.5 is 76%.Figure GEO, RANK, and QUANT, as defined in the Methods section) are shown in Figures t-statistics were obtained from independent normally distributed data (see the Methods section for explanations) produced by simulations (SIMU1). In this case, the proportion of gene pairs with correlation coeffcients greater than 0.5 is only 1.5%. While the normalization procedure GEO destroys a large proportion of correlation, the procedures RANK and QUANT outperform it as far as the reduction of between-gene dependence is concerned. The effects of the latter two procedures are largely similar. The procedure RANK reduces the proportion of correlation coeffcients greater than 0.5 to 4.3%, while the procedure QUANT reduces this proportion to 7.2%. For comparison, this indicator is equal to 14% for GEO. Thus the procedure RANK has the strongest effect on the correlation structure. Figure The effects of three normalization procedures . This can be seen in Figures possible . The norQUANT is not nearly as effective as in the case of the SIMU2N data. This procedure does not eradicate the overall correlation between genes in the SIMU3N data. In this sense, the effects of normalization seen in the SIMU3N and in real biological data look similar.The effect of the quantile normalization for the SIMU3N, shown in Figure Another way of studying such effects is to look at the number of pairs characterized by a relatively high correlation with a pre-selected gene. Tables RANK and QUANT bring this number down to 700 on average (see Table QUANT procedure. Both procedures indiscriminately reduce the true (intrinsic) correlation and its spurious (nuisance) counterpart. Although less effective, the procedure GEO does a similar job.Consider first the results obtained with simulated data. Each of the twenty initiator genes selected from SIMU2N form exactly 12,558 highly correlated pairs. When applied to the SIMU2N data, the normalization procedures QUANT method the range is from 526 to 2,368 showing that the remaining correlation extends far beyond the specified clumpy structure.The results for the SIMU3N data are different  of highly correlated gene pairs. The rank normalization results in much smaller numbers of highly correlated genes that range from the lowest of 494 to the highest of 2,513. The average is 1,255, which is about twice as much as we get from any normalized SIMU2N data. The variability is also very high, resembling a clumpy effect seen in the SIMU3N set. We do not consider this similarity as evidence for a clumpy structure of microarray data, but the results in Table We then selected 20 initiator genes in the SJCRH data set representing real biological data. The number of highly correlated pairs formed by these genes before normalization ranges from 12,384 to 12,501, which is a very narrow range indeed. As is seen in Table QUANT is also more variable than that of the RANK, which is another dissimilarity of practical importance. Leaving aside the fact that the RANK procedure is applied to gene expressions, while the QUANT works at the probe feature level, the difference between the two normalization methods is that we replace entries in an array by their ranks in the former case and by  in the latter. Recall that  is the average of entries having the same rank over all arrays. Obviously, the QUANT preserves more quantitative information in the data than does the RANK procedure. This explains why the result of the rank normalization is less variable.Another interesting finding in Table QUANT on the distribution of the t-statistics across the genes for the actual and simulated data is shown in Figures t-statistics similar to that in the ideal case shown in Figure t-statistics.The effect of the normalization The results shown in this section are obtained with a single initial random split of the pooled set of arrays into two groups. We have conducted several such splits in this study. All the above-described effects are highly reproducible, and reporting the results for other splits in the paper is not warranted.It follows from our observations that normalization procedures are capable of destroying a significant part of correlations between gene expression signals and associated test-statistics. In doing so, they affect both the spurious correlation induced by the noise and the true correlation that reflects gene interactions. The clumpy structure (involving relatively large clumps of genes) of the SIMU3N data set is more resistant to this effect than the SIMU2N data. This is even more so for real biological data. The weaker effect of normalization seen in the SJCRH data indicates that the actual noise structure may be more complicated than assumed in the simulation studies (multiplicative array-specific random effect model). A clumpy structure of gene expression signals may also play a role in this phenomenon. This observation explains why it is so diffcult to remove correlations from the data.The destructive effect of normalization procedures on pairwise correlations in microarray data is good news for the methods of statistical inference that resort to "pooling across genes". However, it remains unclear whether or not the remaining correlation may still be substantial enough to invalidate such methods by affecting important properties of statistical estimators and tests. The problem invites further investigation. However, we would like to present an experiment specially designed to address the consistency question mentioned in the Background section.To this end, we applied the following algorithm to the SJRCH data:t-statistics across these genes for each pair of subsamples.1. Select randomly 100 genes and compute the arithmetic (sample) mean of the 2. Compute the standard deviation of the sample mean across the 15 pairs of subsamples.3. Select randomly 100 from the remaining genes and compute the arithmetic mean for the 200 genes for each pair of subsamples.4. Compute the standard deviation from the sample means resulted from the previous step.5. Continue until the set of all genes is exhausted.6. Plot the estimated standard deviation of the sample mean as a function of the number of genes involved in each step of the algorithm.k times to generate k trajectories of the standard deviation of the sample mean.7. Repeat the procedure t-statistics associated with different genes. The pattern seen in Figure k = 20 experiments with different random starts. If the standard deviation of an unbiased estimator tends to zero, this estimator is consistent. This is the case for Curve 3 but not quite so for Curve 2. While not a rigorous disproof of consistency of the sample mean in this case, the pattern seen in Curve 2 suggests that the estimator is likely to converge to a random variable (with the variance greater than zero) rather than to the true parameter to be estimated. This is definitely not a good sign for estimation procedures based on pooling across genes such as those built in the empirical Bayes methodology.The results of one such experiment are given in Figure The observed effect of normalization procedures is definitely bad news for the associative network reconstruction from gene expression data. Unless further technological advancements result in a significant reduction of the noise in microarray data, this kind of analysis will continue producing unreliable inferences. To normalize, or not to normalize: that is the question to which no scientifically sound answer is currently known as far as this kind of reverse engineering is concerned. Although limited to cell cultures, the causal inference from gene perturbation (disruption and over-expression) experiments seems to be the only solid alternative. From this standpoint the observations reported in the present paper add to the concerns expressed by several investigators regarding how much confidence to place in the thousands of papers already published using microarray technology .t-statistics associated with different genes. This study leads us to conclude that:The present paper provides quantitative insight into correlation between the t-statistics.• There is a long-range correlation in microarray data manifesting itself in a huge number of genes that are heavily correlated with a given gene in terms of the associated t-statistics computed for different genes.• Using normalization of microarray data it is possible to significantly reduce correlation between the • Normalization procedures affect both the true correlation, stemming from gene interactions, and the spurious correlation induced by random noise.t-statistics independent when applied to the biological data.• It is likely that some noise effects represent non-monotone transformations of the underlying gene expression signals because even the rank normalization does not make the t-statistics associated with different genes in biological data. Furthermore, the long-range correlation structure persists in normalized data. This remaining correlation may be strong enough to deteriorate consistency of statistical estimators built from measurements on the genes.• Even the most effcient normalization procedures are unable to completely remove correlation between the N = 12, 558 genes. Each gene is represented in the data set by the logarithm of its expression level. The data are publicly available on the following website: [normalization = false. Since our focus was on purely methodological problems, we pooled all the available arrays together and shuffed the pooled sample. After randomly choosing and dropping 5 arrays , the pooled sample was randomly split into 30 parts, each containing 11 arrays.There are 335 arrays  in the SJCRH data set, each array representing website: . The SJCt-statistics were computed for each gene, thereby mimicking 15 two-sample comparisons under the null hypothesis of no differential expression. As a result, each gene was associated with 15 values of the t-statistic so that the Pearson correlation coeffcient between the t-statistics thus derived could be computed for any pair of genes. The output of the above-described series of procedures is a 12, 558 × 15 matrix of t-statistics and the associated vector of correlation coeffcients for all pairs of genes. We proceeded through the same sequence of operations when analyzing normalized and simulated data sets.Then 15 pairs of the array samples were arranged and the corresponding 15  non-overlapping pairs of genes to eliminate the spurious correlation between gene pairs due to multiple entries of the same gene in different pairs. Since this procedure begins with a randomly selected pair and proceeds through many steps of successive elimination of the previously selected pairs, it was repeated several times with different (random) starts and each output being analyzed separately. To study the long-range correlation, we picked 20 genes that produce large numbers of gene pairs with correlation coeffcients greater than 0.5. This experiment was designed to see how a given normalization procedure affects the number of such pairs associated with each of the twenty genes. A similar design was used with simulated data.In a separate experiment, we formed We simulated several sets of data to gain a better insight into the effects of normalization. All of them included the same numbers of arrays and genes as in the biological data described in the previous section. Specific characteristics of these data sets are given below.xij, i = 1, ..., 12, 558; j = 1, ..., 335 in SIMU1 represents log-intensity of expression of the ith gene from the jth array. The independent and identically distributed random variables xij are generated from the standard normal distribution. This implies that the original expression signals are modeled as log-normally distributed random variables but we used their logarithms in our computations. This data set was used to illustrate the correlation analysis under independence of gene expression levels.1. SIMU1: Every element xij of the same column is equal to 0.8.2. SIMU2 is a 12, 558 × 335 random matrix that models an exchangeable correlation structure. The entries in this matrix are normal random variables with mean zero and unit variance. The entries from different columns are independent, while the correlation coeffcient between any two elements A. The elements of A are independent and identically distributed. The marginal distri- bution of every element aij of A is uniform over the interval [yij to be xij + aj, where xij is the ij-th entry in the SIMU2 data set. The new matrix Y = {yij} represents the data SIMU2N3. SIMU2N is a data set based on SIMU2. First we generate a 335-dimensional random vector interval . This rak-th group (gene numbers 100·(k-1)+1 through 100·k), for k = 1, 2, ..., 10, then the correlation coeffcient between them equals . Any two genes pertaining to different groups are stochastically independent.4. SIMU3 is a 12,550 × 335 matrix. The 12,550 rows (genes) are divided into ten groups of genes, each containing 1,255 rows. If two genes are both from the 5. SIMU3N is the same as the SIMU3 data set but with an added noise. An array-specific multiplicative and uniformly distributed noise is modeled exactly as in the SIMU2N data.M arrays of length N, and we represent the corresponding log-intensities as an N × M matrix X such that each array is represented by a column in X. In this work, we used the following normalization methods:Suppose there are et al [If the array-specific random noise is multiplicative then a reasonable way to remove it from the expression values is to divide each element of the data matrix by the geometric mean over all gene expression signals on the array to which this element belongs. Szabo et al  discuss et al [Xsort by arranging all gene expression signals for the same array in increasing order. Next we replace every entry in this array by its position (rank) in Xsort counted from the smallest value. The idea behind this method is that ranks are invariant to any monotone transformation, implying a much more general model for the technological noise than the multiplicative array-specific random effect model.This method was proposed by Tsodikov et al  and discet al . In accoM data vectors is the same if the plot is a straight line in the direction of unit vector  but it is not the same otherwise. So we could make a set of data to have the same distribution if we projected the points of an M-dimensional quantile plot onto the diagonal. Much like as with the RANK method, this approach is applied to genes rather than arrays. We refer the reader to [As discussed in ,22, thiseader to ,23 for mThis work represents a truly collaborative endeavor. All members of the research team contributed equally to the design of this study, discussion of its technical issues and formulation of the net results. XQ was responsible for the computational component of the study. AB brought his biological expertise to the project.The effect of normalization with non-overlapping pairs of genes;Click here for fileThe effect of normalization procedures on the correlation structure of simulated data;Click here for filet-statistics across genes.The effect of the quantile normalization on the distribution of the Click here for file


The increasing use of DNA microarrays in biomedical research, toxicogenomics, pharmaceutical development, and diagnostics has focused attention on the reproducibility and reliability of microarray measurements. While the reproducibility of microarray gene expression measurements has been the subject of several recent reports, there is still a need for systematic investigation into what factors most contribute to variability of measured expression levels observed among different laboratories and different experimenters.We report the results of an interlaboratory comparison of gene expression array measurements on the same microarray platform, in which the RNA amplification and labeling, hybridization and wash, and slide scanning were each individually varied. Identical input RNA was used for all experiments. While some sources of variation have measurable influence on individual microarray signals, they showed very low influence on sample-to-reference ratios based on averaged triplicate measurements in the two-color experiments. RNA labeling was the largest contributor to interlaboratory variation.Despite this variation, measurement of one particular breast cancer gene expression signature in three different laboratories was found to be highly robust, showing a high intralaboratory and interlaboratory reproducibility when using strictly controlled standard operating procedures. Gene expression analysis with DNA microarrays has been used to develop molecular taxonomies of various types of cancers -16. SmalSeveral studies have examined the reproducibility of gene expression analysis by DNA microarrays across different laboratories. One study used aliquots of a common mouse liver and pooled RNA, and compared gene expression measurements made in seven laboratories using a total of 12 microarray platforms . They fo®, Agendia BV) [In this study, we examined the interlaboratory reproducibility of a specific 70-gene breast cancer signature ,2, recenndia BV) , using tTo compare DNA microarray data reproducibility within and between laboratories, we employed the experimental scheme shown in Figure In the first phase of this study, we examined the reproducibility of microarray measurements between the California and Amsterdam laboratories. Both laboratories amplified and labeled each RNA sample, and sent aliquots of each labeled product to the other laboratory. Both laboratories then hybridized all the samples labeled in both labs, scanned the slides, and then shipped the slides to the other laboratory for rescanning. The same lot of labeling kits and microarrays were used in both labs. In this manner we could compare the intra- and inter-laboratory variations in each step of the microarray assay, starting with purified total RNA.Each slide contained eight individual microarrays, which could be hybridized separately. The hybridization setup is shown in Table Variability among microarray assays might arise from differences between labeled samples, between the arrays themselves, or between hybridization conditions. A replicate hybridization is defined here as a pair of assays for which the sample labeling and hybridization conditions are held constant; that is, aliquots from a single labeling reaction are hybridized to different arrays at the same location. Comparison of replicate hybridizations allows us to determine the noise attributable to hybridization, washing, and scanning, and to variations among the arrays themselves. Contributions to noise include an additive background, a proportional precision, and a stochastic element. In Figure In order to determine whether differences between microarray scanners introduce significant variability into the results, we scanned the hybridized arrays at each site and then shipped them to the other site for rescanning. Figure 10 ratios for each of the triplicate probes for each of the 70 genes, and then determining the cosine correlation of the average log10 ratios for the 70 genes in the particular tumor sample with the average profile of these genes in tumors from a specific, defined set of patients. This procedure eliminates the effect of random variation in microarray signal strength between probe replicates [As a biologically relevant way of measuring the reproducibility of the microarray results, we computed the 70-gene breast cancer signature correlation value as previously described  for eachplicates . The varFigure 10 ratios of signature probes from the four arrays (two dye-swap pairs) that shared the same labeling and hybridization location, as there is little systematic variation between replicate hybridizations , at a time several months after the initial comparison was finished, using a different lot of microarrays and different lots of labeling reagents.Figure In this study we examined the reproducibility of a 70-gene breast cancer signature in a series of experiments performed in three laboratories, one in Amsterdam, one in California, and one in Paris. In the first part of the study identical RNA samples were labeled and hybridized to identical microarrays using the same platform and protocols, in both the Amsterdam and California laboratories. Reproducibility of signals and ratios was measured for replicate assays in each laboratory. We found that the results were very reproducible between sites. The low noise across the entire platform, as shown by the reproducibility of replicate hybridizations , allowed the averaging of the replicates, with the result that minor differences in the data became more apparent Figure . In the We took care to be sure the same operating protocols were used between all the laboratories, and the operators in all laboratories were well trained. We found that if variations in the wash protocol were introduced between laboratories, significant discrepancies in the results emerged (data not shown). It is clear from our findings and those of others  that micA DNA microarray measurement can be considered as hundreds or thousands of simultaneous analytical measurements of the relative concentrations of mRNAs in a sample. In order to examine the analytical precision, accuracy, and detection limits of these measurements, several laboratories have published cross-platform and other comparisons of microarray measurements -19,23-29It should be noted that the experimental setup used in this study cannot measure every possible source of variation. Since all of the hybridizations involving a common sample were hybridized to arrays on the same slide, and the replicate slides in each laboratory were hybridized on different days, we cannot determine whether any variation observed between the two replicate slides is due to slide-to-slide variability or day-to-day variability, or a combination of the two. However, since the experimental setup compounds both potential sources of variation, we would expect that any such differences would be maximized in this study. Despite this, the 70-gene signature correlation values did not vary significantly by hyb day Figure .Another possible source of variation is inter-individual variability. Since all the labelings and hybridizations done at each site were performed by single individuals, the cross-laboratory variability cannot be de-convoluted from the inter-individual variability. However, we would expect that if two different individuals took care to follow the exact protocols, as in this study, that interlaboratory variation would be greater than inter-individual variation, due to use of a different set of laboratory equipment . Another study reported measuring the 70-gene signature correlation values of two tumor samples repeatedly in the same laboratory, by six different individuals, with very consistent results .10 ratios, which amounts to a 5% difference in the actual expression ratio) but is detectable nonetheless. We used labeling kits from the same lots and purchased at the same time, so all labeling reagents were equivalent. While the labeling site differences were significant for only two of the four tumors when comparing the tumor signature correlation values, the differences extended to all four tumors when examining the log10 ratios of the 70 signature genes on an individual basis. This suggests that the differences seen on an individual gene level are relatively random, and cancel one another out when looking at the signature as a whole, which represents a correlation of the log10 ratios of all 70 genes and averages of measurements from three replicate features for each gene. The variation in individual genes did not correlate with the expression level of the genes, which differs from the findings of Dobbin et al. [We found that the largest discrepancy between the Amsterdam and California sites was in the amplification/labeling step. This discrepancy was relatively small . 200 ng each of Cy3- and Cy5-labeled RNA were hybridized to each array in a 45 ul total volume of hybridization buffer (Agilent Technologies) for 16 hours at 60C, followed by room temperature disassembly in 6× SSC/0.005% Triton X-102, a ten minute room-temperature wash in 1× SSC/0.005% Triton X-102, and a five minute room temperature wash in 0.1× SSC/0.005% Triton X-102. Slides were dried with filtered, compressed nitrogen and scanned immediately in a DNA Microarray Scanner (Agilent Technologies). After slides were scanned in the Amsterdam or California laboratories, they were then shipped overnight to the other laboratory for rescanning. Slides hybridized in Paris were not rescanned.Array images were extracted using Agilent Feature Extraction software, version A.7.5.1, per manufacturer's instructions. After subtraction of feature backgrounds the signals in the test and reference channels were normalized for consistency of the normalization features, as described in the Feature Extraction software documentation.For the 232 genes with three replicate features per array, the signals for the triplicate features on each array were averaged . For eac10 ratios of probes from the four arrays (two dye-swap pairs) that shared the same labeling and hybridization location, we took the probes for the 232 breast cancer-related genes [To assess reproducibility in this study, ANOVA P values were calculated using JMP 5.1 software (SAS). To determine the averaged logRAA participated in the study design, performed all the lab work in California, participated in the data analysis, and drafted the manuscript. AF participated in the study design, supervised and coordinated the lab work in Amsterdam, and helped draft the manuscript. BC participated in the study design, and did much of the data analysis. VL supervised and coordinated the lab work in Paris. AMG participated in the Mammaprint analysis. RP performed all the lab work in Amsterdam. AT participated in the data analysis. HR performed the data processing of the Paris data. FC aided in the protocol design. MSA was a study coordinator. LB supervised the work in California, and helped draft the manuscript. LJVV was the supervisor/project leader, and helped draft the manuscript. All authors read and approved the final manuscript.Person correlations of replicate hybridization pairs. This Microsoft Excel file gives the Pearson correlation values of the sample and reference channels for the 8 pairs of replicate hybridizations for each of the four tumors.Click here for file