Model-based prediction is dependent on many choices ranging from the sample collection and prediction endpoint to the choice of algorithm and its parameters. Here we studied the effects of such choices, exemplified by predicting sensitivity (as IC50) of cancer cell lines towards a variety of compounds. For this, we used three independent sample collections and applied several machine learning algorithms for predicting a variety of endpoints for drug response. We compared all possible models for combinations of sample collections, algorithm, drug, and labeling to an identically generated null model. The predictability of treatment effects varies among compounds, i.e. response could be predicted for some but not for all. The choice of sample collection plays a major role towards lowering the prediction error, as does sample size. However, we found that no algorithm was able to consistently outperform the other and there was no significant difference between regression and two- or three class predictors in this experimental setting. These results indicate that response-modeling projects should direct efforts mainly towards sample collection and data quality, rather than method adjustment. In such laboratory environments, similar variance of response can be observed when cancer cells from different patients are treated with the same drug. Of course, it is expected that differences in drug response are closely related to the type of cancer under treatment. Other factors like ethnic, genetic and histological background of patients even with the same type of cancer play an important role in drug response, making cancer a very individual disease requiring a personalized treatment.It is common clinical practice to treat cancer patients with a variety of therapeutic agents, due to the simple fact that drug efficacy varies among patients. Thus, general recommendations for drug selection and finding the optimal dosage for a certain drug are challenging tasks and often subject to change based on new standards of care. This is one of the main reasons why drug response to cancer therapeutics is being tested in The toxicity of the medication as well as the severity of the disease itself requires a high degree of confidence in the effect of a drug for the specific patient. The field of personalized medicine is dealing with this type of challenge. One research focus is the identification of markers that can be used to individualize treatment recommendations, i.e. to reliably predict the response of cancer to drugs. Markers are selected among high-dimensional genetic data, such as gene expression data from microarrays and machine learning methods are used for the prediction.However, there is a multitude of challenges given a fairly “simple” question (i.e. the prediction of drug response to cancer cell lines). The “no free lunch theorem” In this study, we have examined the factors influencing the feasibility of predicting the outcome of drug response experiments. An accurate prediction of drug response could help economize on the vast resources necessary for laboratory experiments. To this end, we have designed a study using well-known machine learning algorithms, response data of drugs inhibiting cell growth by targeting different mechanisms of the cell, gene expression data from in-house and public efforts. It is expected that the underlying biochemical processes of these cellular mechanisms can be differently well observed through gene expression analysis.Therefore, we have formulated five hypotheses to determine important factors contributing to the above question.Hypothesis relating to the choice of drug:H1: The successful prediction of efficacy depends on the particular drug.Hypotheses relating to the machine learning methods:H2.1: The machine learning algorithms used for the prediction task at hand have a strong influence on the predictability.H2.2: One labeling is superior in predictiveness than the others.Hypotheses relating to the choice of cell line panel:H3.1: There is a connection between the choice of panel and drug predictability.H3.2: The number of samples influences the predictiveness.In this study, we have evaluated drug response predictions for 23 drugs. This prediction task has been solved by seven different machine learning approaches and up to three labeling variants see . Thus, mmodel can be any model according to panel can be any of the utilized cell line panels . Drug denotes any of the 23 drugs considered, to which response was predicted using one of the three labelings . We have evaluated every possible combination of the variables for the original data and for null model data with randomly permutated class labels  and a decision tree for evaluating the prediction results and to identify factors influencing the prediction quality. We could show that the choice of the cell panel and the drug has a greater influence on the prediction quality than expected by chance. A correlation between small sample size and low predictive accurateness and high variation has been found. Thus, we could show that the efficacy of different drugs is not equally good to predict. The machine learning algorithm used for prediction has little to no influence on accuracy.Brown et al. Zhu and Hastie Diaz-Uriarte and Alvarez de Andres Shi et al. Statnikov et al. Riddick et al. Generally, two categories of machine learning algorithms exist: Regression models to predict continuous variables and classification models to predict categorical variables. The choice of the algorithm is then usually based on expert knowledge from the problem domain. A typical classification problem features two classes (binary classification) but some models can also be used for multiclass classification. The process of separating the cell lines according to the IC50 values into two or three classes is referred to as labeling here (e.g. responder vs. non-responder or responder vs. intermediate vs. non-responder). The factor labeling consists of three levels: regression, binary, and class3 (multi classification with 3 classes). All three levels have been applied, since there is no clear direction for a certain classification level suggested by prior research for the problem considered here. Different machine learning algorithms have been proposed A panel of cell lines has different properties that can be expected to influence the prediction task. The cell line panels used in this study have differing numbers of samples (either 22 or 59) from a diverse range of tissues and cancer types. Therefore, it was tested whether the average predictiveness over all drugs is significantly different between the cell line panels. In such a case, it can be further tested whether the number of samples can explain this. Therefore, a model performing well on the larger panel and worse on the smaller panel is reevaluated with only a subset of the samples.The most comprehensive study of compounds in a diverse set of cell lines has been undertaken by Holbeck et al. Further cell lines have been selected to cover a broad set of different indications and according to their response to different standard chemotherapeutics. One aim of this study was to compare the NCI60 panel to our panel of 21 cell lines, including 18 tumor cell lines  and 3 non-tumorigenic cell lines , named hereafter the Bayer Pharma (BPH) panel (see The gene expression microarray data for the NCI60 panel have been downloaded from the European Bioinformatics Institute (EBI) ArrayExpress website The microarray data have been downloaded as CEL-files and have been processed to obtain the gene expression matrix used for modeling, according to the following procedure:simpleaffy  has been determined by analysis of inhibition of cell proliferation after 72h treatment with different concentrations of respective compounds. The concentration ranges from 1E-05 to 2E-10 mol/L. Cells were seeded into 96-well cell culture plates required to ensure approximately 80% confluence in control at end of experiment. After 72 hours, cells growth was analyzed using Cell-Titer Glow™ according to manufactures instruction. One part of the sample was measured after 24 hours to get “zero-growth” value which corresponds to full inhibition. Data were normalized to Dimethylsulfoxid (DMSO) treated cells as “no-inhibition”. The drug response was quantified by the half maximal inhibitory concentration (IC50) for a particular cell line. IC50 determination was done using a statistical approach based on normalized growth curves of different compound-concentrations.Correspondingly, drug response data (IC50) of the NCI60 panelfor 48,129 drugs were downloaded from the NCI website One subset contains all drugs that have been tested on the BPH and NCI60 cell line panels . The second subset is a subset of drugs found by clustering the response values according to the following procedure:Drugs have been removed, if more than five values were missing or less than 29 distinct values existed .Drug response was then standardized for each drug by subtracting the mean value and dividing by the mean absolute deviation.cluster sample = 50 (number of samples to be drawn from the dataset)samplesize = 500 (number of observations in each sample)The medoids of the fourteen largest clusters have been selected as representatives .Different panels of cancer cell lines are commonly available and have been used here to train models for the prediction task. Each panel contains cell lines representing different patients. A matrix of gene expression data X has been constructed for each panel used in this study, as well as a drug response matrix Y that contains individual information of drug efficacy for each cell line.IC50 values were converted to the pIC50 scale (-log10(IC50)). These pIC50 values have been labeled for a binary and a three class classification problem as described in the following.Labeling for a binary classification problem:Find the class threshold by:Removing of all samples that are equal to the minimum or maximum value of the set.Set the class threshold equal to the median of the remaining samples.Label all samples greater than the class threshold with 2 and the remaining samples with 1.Labeling for a three class classification problem:Find the two class threshold by:Removing of all samples that are equal to the minimum or maximum value of the set.Set the upper class threshold equal to the 66-quantile of the remaining samples.Set the lower class threshold equal to the 33-quantile of the remaining samples.Label all samples that are greater than the class threshold with 3, the samples smaller than the lower class threshold with 1 and the remaining samples with 2.The labeling results in three different sets of labels for each drug: The pIC50 values for regression, binary classification and three class classification.1 penalty. The L1 penalty term introduces a variable selection of at most n variables, where n is the number of samples available in the training set. A disadvantage in the usage of a penalty term can be observed for group wise correlated variables. In this case, often only one variable per group is selected, while the others are ignored We have utilized penalized logistic regression which is a linear model to predict categorical variables. The penalty term used here is called LASSO or Lglmnet  constructs a hyper plane in a high or infinite dimensional space to separate classes. By maximizing the distance between the hyper plane and the closest samples, intuitively the generalization error is maximized and the risk of overfitting is reduced e1071  algorithm is an ensemble method randomForest .The implementation utilized by us is from the R package The principal component analysis (PCA) has been used as a dimension reduction method, before applying the random forest or SVM algorithm. For each run of the 10-fold cross validation, we have applied the following steps:Train the PCA model on the training data.Use the first five principal components obtained from the PCA model to training the SVM or RF model.Predict the first five principal components for the samples in the test set.Use the PCA prediction as input for the SVM or RF model to predict the classes or regression values.This way, the instability of the PCA has been included in the cross validation. This would not have been the case, if the PCA would have been calculated on the whole data beforehand. The SVM and RF settings have been chosen as explained above.RF has been used to select the most important genes. These genes have then been used to build the final model. To avoid a feature selection bias, the following steps have been executed for each run of the 10-fold cross validation:The random forest has been trained on the training data.MeanDecreaseGini has been used for ranking and the %IncMSE for regression.The resulting model has been used to rank the genes by the variable importance obtained from the out-of-bag (OOB) data. For classification, the In the next step, the 300 highest ranked genes have been used to retrain the random forest on the training data.Lastly, the so retrained random forest has been used to predict the test set samples.All settings have been chosen the same way as described for random forests above.To establish if a model is significantly better than expected by chance, a suitable null model is needed. One way to generate such a null model is estimating the model performance after random permutation of the class labels or target values. This approach ensures that the data of the null model are based on a similar distribution as the real model. After establishing this null model, an empirical p-value can be calculated to reject or accept the following hypothesis:H0: The classifier can extract information from the data. This means, there is a correlation between the data and the class labels.For our study, we have utilized empirical p-values proposed by Good The metric used to quantify the prediction power of a model plays a key role in finding and tuning the best model. The choice of the metric is important, since metrics put unequal importance to different aspects of the model performance; it is therefore problem and data specific. We have used the following metrics for evaluation:Accuracy: The ratio of all correct (true positive and true negative) predictions to the total number of cases evaluated .The accuracy metric is not suited, however, for unbalanced classification problems as the contribution of each class depends directly on the number of samples Precision: The ratio of the number of correct predictions actually made (true positive) to the total number of incorrect and correct predictions .Recall: The ratio of the number of correct predictions actually made (true positive) to the total number of all correct predictions .F-Score: Uses a weighted combination of Precision and Recall (see above). Further, it is not sensitive to imbalanced classes. The F-Score with β = 1 is called the F1-Score and denotes the harmonic mean of Precision and Recall, used here as defined as follows:Several classifier assign a class probability to each prediction. Precision and recall can be influenced by the choice of a threshold for the class probability. In such a setting, the threshold can be trimmed for varying misclassification costs between the classes. This optimal choice depends on the cost of wrong classification (Precision) versus the cost of missing a sample from a class . In cases where this tradeoff is not clear, the metric incorporating all possible choices best evaluates a classifier.Using this receiver-operator-curve (ROC) analysis, the power of the classifier can be defined independently from the decision threshold by calculating the area under the curve (AUC). The AUC is in general better suited to compare classifiers than the accuracy metric Regression problems deal with continuous values and are therefore differently evaluated than classification problems.1, …, pn] and A = :For this, we have used the Root Mean Square Error (RMSE), measuring the distance between two sets of data points P = [pThe fact, that the divergence between predicted and observed value is quadratic penalized, makes this metric, however, very sensitive to outliers.Different metrics have been compared. When assessed as empirical p-values, it can be shown that these metrics are, in general, correlated (data not shown). However, here we report p-values only based upon RMSE (for regression problems) and AUC (two class) or multiclass AUC (tree class problem). SVMs do not provide class probabilities directly, hence there the accuracy metric (two class) resp. F1-score (three class) was used. See Independent of the metric used to evaluate the classification or regression model, we have also employed a method to estimate the generalization performance. By this, we mean the expected prediction capability of a model on independent test data. The generalization performance can easily be estimated if a large set of samples is available. In this case, the data are partitioned in a test and a training set. Each set is a representative sample of the overall distribution. The error on the test can therefore be expected to be reliable. One possible split for the two partitions can be such that 70% of the data lies in the training set and 30% in the test set.In cases of small sample size, however, k-fold cross validation uses the samples more efficiently and provides a more accurate estimate. The data are first partitioned in k folds. Then, each fold is used as test set once, while the model is trained on the remaining data.Let If the model includes a tuning parameter or any kind of feature selection, a second cross validation has to be incorporated in a nested manner (i.e. the training data is partitioned repetitively in inner-training and inner-testing sets).When dealing with microarray data, often very small sample sizes are available (<100). This results in unstable cross validation error estimates that can depend heavily on the partition used for the cross validation. To stabilize the error estimates, the 10-fold cross validation was repeated ten times with different partitions. The results of all ten runs were then averaged. This method, called the 10×10-fold cross validation, is one of the most accurate estimators for direct comparison between models The following procedure has been used to examine the influence of sample size on the predictive power of the model. The drugs NSC180973 and NSC700861 have been selected due to their relatively low empirical p-values (see Results). The RF  and PCA+RF algorithms described above have been selected for the evaluation due to their fast execution time. For each drug-algorithm combination, the following steps have been computed:Estimate the RMSE using 10-fold cross validation.Remove five samples and perform the estimation again.Repeat (2) until less than 20 samples remain.The 510 p-values see  and S2 hIn addition, a random forest model has been trained on the same data as the decision tree . The variable importance based on OOB predictions has been used to rank the results as can be seen in All empirical p-values for the NCI60 (representative) panel and the synthetic data are summarized in Boxplots for the p-values of the panels BPH and NCI60 are shown side by side in Boxplots for all p-values from one panel can be seen in The two factors model and labeling show no clear connection to the p-values. The choice of the model appears as a splitting variable in Gemcitabin is here used to illustrate the difficulties with class assignments: Differences in panels are not only manifested in sample size, but they may, for some drugs, show differences in IC50 value distribution: Density plots for the overlap of drugs between the BPH and the NCI60 panel can be seen in Two drugs  have been evaluated for various sample sizes with the random forest algorithm and PCA+RF (see Methods). Those drugs were selected because they are highly predictable . It can be expected, that even with decreased sample size the data contains sufficient information to train a model. The RMSE and variation of the results decrease with the number of samples used to train the model. This observation is consistent for all four experiments shown in In our present study, we have evaluated drug response predictions for 23 drugs and positive and negative controls. The prediction task has been solved by 17 different machine learning/labeling approaches. The resulting predictions have been analyzed to find factors that influence the prediction quality. It has been found that the choice of the panel and the drug plays a major role for the prediction quality. The two most predictable compounds were used to demonstrate a connection between small sample size and low predictive accuracy. While this is intuitively expected, it is still possible that other algorithms are more robust towards decreasing sample sizes.The results of this study suggest that the choice of cell line panel and drug have a strong impact on predictiveness, while the selection of machine learning algorithm plays only a minor role in the cases of low predictivity.We will discuss this in the following with regard to the hypotheses postulated in the Introduction. For evaluation, a decision tree was used to determine the factors leading to low p-values (for the hypothesis that a model predicts drug response) and the irrelevant factors. In addition, a random forest was used to determine the importance of factors towards differentiating between low and high p-values . To our knowledge, random forests and decision trees have not been used to evaluate results in this way before.drug and, therefore, the information which drug is being predicted by the model. The random forest variable importance supports that the drug selection has a great influence on predictability  while the quality of prediction is comparable.It has been shown that quality of drug response prediction depends on the drug under investigation. There is a need to learn more about what causes these differences. Especially the question about biological implications of our findings deserves further attention. The causes for these differences could also be explained by factors that cannot be observed by gene expression data, e.g. structural differences in proteins with a compound specific effect. The results can also be influenced by a lack of diversity in the response or by cell lines that are very sensitive or resistant. In addition, the reliability of the IC50 measurements can be a source of inaccuracy.We have shown here that the predictability of one and the same drug can greatly differ if different cell line panels are used to obtain the drug response profile and gene expression data. The origin of these differences and whether sample size or other important factors influence these findings is currently under investigation.Table S1Empirical p-Values for every compound, panel, algorithm, and labeling combination. A comma separated data table: The metric column contains an abbreviation of the used metric, the ‘metric.median’ column has the numerical value in this metric, while ‘p_value’ contains the empirical p-Values estimated from the null-model. Columns expSet, labeling, model, and compound contain the panel, the labeling , the algorithm, and the compound (drug) respectively.(CSV)Click here for additional data file.Table S2cv1’ to ‘cv10’) is reported, otherwise columns are labeled as in Raw prediction quality values for every compound, panel, algorithm, and labeling combination. A comma separated table. Here, the metric used, and the result in each CV run (‘(CSV)Click here for additional data file.


The standard approach for preprocessing spotted microarray data is to subtract the local background intensity from the spot foreground intensity, to perform a log2 transformation and to normalize the data with a global median or a lowess normalization. Although well motivated, standard approaches for background correction and for transformation have been widely criticized because they produce high variance at low intensities. Whereas various alternatives to the standard background correction methods and to log2 transformation were proposed, impacts of both successive preprocessing steps were not compared in an objective way.In this study, we assessed the impact of eight preprocessing methods combining four background correction methods and two transformations (the log2 and the glog), by using data from the MAQC study. The current results indicate that most preprocessing methods produce fold-change compression at low intensities. Fold-change compression was minimized using the Standard and the Edwards background correction methods coupled with a log2 transformation. The drawback of both methods is a high variance at low intensities which consequently produced poor estimations of the p-values. On the other hand, effective stabilization of the variance as well as better estimations of the p-values were observed after the glog transformation.As both fold-change magnitudes and p-values are important in the context of microarray class comparison studies, we therefore recommend to combine the Edwards correction with a hybrid transformation method that uses the log2 transformation to estimate fold-change magnitudes and the glog transformation to estimate p-values. Gene expression microarray is a widely used technology in functional genomics that allows to measure efficiently the expression level of thousands of genes in a single experiment. Among the wide spectrum of available array technologies and suppliers, two common technologies are the in-situ oligonucleotide synthesised GeneChips developed by Affymetrix  and the et al. showed that such interferences can be minimized by optimizing the numerous steps of the microarray experiment, and more particularly the hybridization and the washing steps [et al. discussed also the importance of pre-analytical steps for transcriptome analysis [Whatever its application, the first analytical step in a spotted microarray experiment is the acquisition of an image file with an optical scanner. Then, the image analysis software segments the acquired image into spotted and unspotted regions and returns average and median of the pixels intensities for both the foreground and the surrounding area  of each spot. It is well known that the foreground intensity of a spot does not perfectly reflect the RNA abundance of its corresponding gene due to interferences of non-specific hybridization on the probe . These ing steps . The autanalysis .Raw data returned by the scanner have to be preprocessed in three successive steps . The firα and λ are two positive parameters. The glog transformation is sometimes referred as the generalized arcsinh transformation because of the relationship between the arcsinh and the log functions. The standard background correction method assumes that foreground intensities are affected additively by the background noise. Although well motivated, this standard method was widely criticized for several reasons. The best known drawback is that local background subtraction induces problems when foreground intensities are lower than local background intensities. Correction leads to negative corrected intensities and consequently to missing values after log2 transformation. Another cited drawback is the extreme variability of the log2 fold-changes obtained at low corrected intensities. To circumvent these drawbacks, alternatives to the standard background were proposed. -13. Alteormation ,17. Unliet al. [et al. [et al. [Eight distinct background correction methods were assessed for differential expression using data from two-color spotted cDNA microarrays by Ritchie et al. . In this [et al.  was cons [et al. . The gloIn this paper, we address the problem of the background correction and transformation of spotted microarray data and the subsequent impact on fold-change compression and on the variance of processed intensities. The first objective of this study was to compare various background correction methods and transformations commonly used in the literature. We propose to consider these two steps together because alternatives to the standard background correction methods as well as alternatives to the log2 transformation were initially proposed to circumvent the same problems: the high variability of low corrected intensities in the log2 scale and the missing values obtained after a log2 transformation of negative corrected intensities. These two successive preprocessing steps were assessed on datasets generated with two spotted microarray platforms  as well as with a quantitative PCR platform (Taqman) from the MicroArray Quality Control (MAQC) project . Data geThe first objective of this paper was to compare the effect of eight preprocessing methods combining four background correction methods and two transformations, on the processed intensities, on the log2 fold-changes and on p-values. Background correction methods are implemented in the backgroundCorrect function of the limma package which is a part of the Bioconductor project  developeWe refer to this method when background intensities are subtracted from foreground intensities.We refer to this method when the background intensities are not subtracted. The corrected intensities are thus equal to the foreground intensities. This method was recommended by other authors ,22.In this method, the background intensities are subtracted if the difference between foreground and background is bigger than a pre-specified small threshold value. When the difference is smaller than this threshold value, subtraction is replaced by a smooth monotonic function .α parameter).The Normexp method is based on the normal plus exponential convolution model . The NorThe log2 transformation is the most commonly used transformation for microarray data. This transformation stabilizes the data variance of high intensities but increases the variance at low intensities.et al. [et al. [α and λ parameters are implemented in the transeS and the tranest functions of the LMGene Bioconductor package [The glog transformation was individually developed by Durbin et al.  and Hube [et al.  to stabi package . To alloThe second objective of the paper was to confirm the additive property of the background on the foreground. This assumption which motivates the Standard background correction method was successfully tested on the Eppendorf data because this type of array contains three technical replicate spots used to measure gene expression levels. For each gene, three foreground intensities and three local background intensities are therefore available on a given array. The specific hybridization on each replicate spot for the same gene should be roughly constant. So, the observed differences between the three foreground intensities are mainly caused by the background noise. For each gene and for each Eppendorf array, the foreground and the background intensities of the three replicate spots were used to build a linear regression model using the foreground intensity as the response variable and the local background intensity as the predictor variable. If the assumption of additivity is true, an increase in the local background should produce the same increase in the foreground. As the Eppendorf array measures 294 genes, a total of 5 880 slopes (294 genes * 5 replicates * 2 samples * 2 sites) were obtained. The values of these slopes should consequently be close to 1 if the assumption of additivity holds.Data from the MAQC project were used in this study. As our study focuses on the preprocessing of spotted microarray data, we decided to perform analyses on a high density spotted oligonucleotide microarray platform  as well as on a low density spotted cDNA microarray platform . As data from Site 2 suffer from annotation problems for both platforms, we only used data from sites 1 and 3 in this study. Among the samples measured in the MAQC study, sample A corresponds to the Universal Human Reference RNA (UHRR) from Stratagene while sample B corresponds to the Human Brain Reference RNA (HBRR) from Ambion. Samples C and D are two mixtures of the original samples. In order to maximize the range of the resulting fold-changes, we decided to use samples A and B (rather than C or D). Table codelink package and were preprocessed using the eight different methods. Data were then normalized between samples A and B using a global median normalization. Finally, as the number of biological replicates is relatively low [eBayes algorithm [limma Bioconducor package was used to compute the log2 fold-changes and p-values of the 54 359 genes between samples A and B.Data acquired with the GE Healthcare platform from sites 1 and 3 were downloaded from the Gene Expression Omnibus (GEO) repository (GEO accession: GSE5350) . Raw datvely low , the eBalgorithm  of the llimma Bioconducor package were used to compute fold-changes and p-values for the 294 genes between samples A and B.Data acquired with the Eppendorf platform from sites 1 and 3 were downloaded from the GEO repository. Raw data were imported in R and preprocessed using the eight different methods. For this platform, data acquired at low, medium and high photomultiplier tube (PMT) voltage were available. In this study, we only considered data acquired at low PMT Voltage in order to avoid saturation problems. Processed data corresponding to samples A and B were normalized using internal standards and housekeeping genes, as recommended by the manufacturer. As Eppendorf platforms contain three replicate spots to measure the level of expression of a single gene, the average of these replicate spots was computed for each gene. Linear models of the limma package were used to compute fold-changes and p-values for the 1 004 genes between samples A and B. In this study, these values are referred as gold-standard fold-changes and gold-standard p-values.Normalized data from the Taqman quantitative PCR were downloaded from the GEO repository and linear models of the Product-moment correlation (r) and intraclass correlation (ICC) coefficients were computed between the log2 gold-standard fold-changes and those obtained from both platforms , from both sites (Sites 1 and 3) after each preprocessing method  standard deviations for each preprocessing method. These standard deviations were used to construct a lowess curve representing the intensity standard deviation as a function of the intensity average  for each preprocessing method. 1 176 average processed intensities were computed with the Standard background correction method followed by the log2 transformation. The lowess curves constructed from the variances obtained after each preprocessing method are shown in Figure As variance stabilization is a step in the preprocessing of microarray data that can greatly benefit the performance of subsequent statistical modeling and inference , we asseResults presented in previous sections showed that the combination of the Edwards method with log2 transformation produced low fold-change compression but led to poorer p-values estimations. Conversely, the combination of Edwards method with the glog transformation produced high fold-change compression at low processed intensities but led to better p-values estimations. When microarrays are used in a class comparison application, both fold-change magnitudes and p-values are considered. We propose therefore to combine the Edwards correction with a hybrid transformation method that uses the log2 transformation to estimate fold-change magnitudes and the glog transformation to estimate p-values.-6. This p-value threshold was fixed by taking into account multiple testing. Three volcano plots were generated from GEH microarray data  and its 95% confidence interval  were obtained by trimming 5 % of the 5 880 slopes. On average, the foreground intensity of a spot with a fixed specific hybridization increases by 1.22 unit when its local background intensity increases by 1 unit. As we showed previously, local background intensities depend on the spatial localization and are independent of their corresponding foreground intensities. It can be inferred therefore that the background noise has an additive effect on foreground intensities.In this study, we addressed the problem of background correction and transformation in spotted microarray data. We compared features of eight preprocessing methods which combine four background correction and two transformation methods.We first compared the correlations between gold-standard fold-changes obtained from quantitative PCR and fold-changes obtained from microarray data. The best correlations were obtained with the Edwards and the Standard background corrections coupled with the log2 transformation. The lowest correlations were obtained with the No background correction method and with all preprocessing using the glog transformation. These results were explained by plotting lowess curves of the fold-change compression as a function of the average processed intensity. While all preprocessing methods produced low fold-change compression at high processed intensity, the different preprocessing methods differed markedly in terms of fold-change compression at low processed intensities. Accordingly, the fold-change compression was minimized using either the Standard or the Edwards background correction methods with the log2 transformation. Using a glog transformation conducted to high fold-change compression whatever the background correction method. It is of note that product-moment correlation coefficients are affected by the fold-change compression because this effect is highly dependent on the average processed intensity. A constant fold-change compression across the whole range of processed intensities would indeed have an impact on intraclass correlation coefficients but no impact on the product-moment correlation coefficients.et al. focuses on the protocol technical aspects that can improve the signal-to-noise ratio and decrease the fold-change compression, our study rather focuses on the best choice of the data preprocessing method. While average biases were only estimated on 9 LUS control probes in the Ritchie's study, 856 and 132 probes were used in our study to compute fold-change compression on the GE Healthcare and on the Eppendorf platforms, respectively. Moreover, in Ritchie's study, the compression factors were only available for 2 of the 9 available LUS control probes for which most background correction methods produced a fold-change compression but the VSN method  surprisingly produced a fold-change expansion. In the current study, all preprocessing methods produced fold-change compression. Furthermore the compression affected mainly low intensity data, an effect that can be minimized by using a combination of the Standard or Edwards background correction with a log2 transformation. The observed differences between these current results and those of Ritchie et al. could be explained by inherent differences between both datasets and by real observed technical differences between one- and two-color microarray readings [These results provide information that are complementary to those published in previous studies which reported that microarray data exhibit fold-change compression ,18,29. Wreadings . In one-et al. which showed that the best performing methods are those stabilizing the variance for the purpose of detecting differential expression. These results also agree with those of Cui et al. [In microarray class comparison studies, effect sizes and p-values are computed by dividing the log2 fold-changes by an estimate of variability. The combination of Edwards (or Standard) method with the log2 transformation produced low fold-change compression but extremely high variance at low processed intensities. At the opposite, the combination of Edwards (or Standard) method with the glog transformation produced high fold-change compression but good variance stabilization at low processed intensities. The impact of the fold-change compression and variance stabilization on the p-values estimation was assessed by computing the correlations between the cumulative Gaussian quantiles of the gold standard p-values obtained from quantitative PCR and the cumulative Gaussian quantiles of p-values obtained from microarray data. Compared to the log2 transformation, the glog transformation which effectively stabilizes the variance across the whole range of processed intensities, produced generally higher intraclass correlation and comparable product-moment correlation. These results are in line with those obtained by Ritchie i et al.  which sti et al. ,22 becauet al. [When microarrays are used in a class comparison application, both fold-change magnitudes and p-values are considered. Historically, the first method to identify differentially expressed genes was based on the fold-change ,29. A chet al.  have indJA performed the microarray data analysis presented in this the paper and participated in compilation of the publication. AR and BG supervised the statistical analysis component of the work and assisted with review of the manuscript. JLG, BM and BB supervised the work and assisted in finalization of the completed manuscript. All authors read and approved the final manuscript.


The data from DNA microarrays are increasingly being used in order to understand effects of different conditions, exposures or diseases on the modulation of the expression of various genes in a biological system. This knowledge is then further used in order to generate molecular mechanistic hypotheses for an organism when it is exposed to different conditions. Several different methods have been proposed to analyze these data under different distributional assumptions on gene expression. However, the empirical validation of these assumptions is lacking.Best fit hypotheses tests, moment-ratio diagrams and relationships between the different moments of the distribution of the gene expression was used to characterize the observed distributions. The data are obtained from the publicly available gene expression database, Gene Expression Omnibus (GEO) to characterize the empirical distributions of gene expressions obtained under varying experimental situations each of which providing relatively large number of samples for hypothesis testing. All data were obtained from either of two microarray platforms - the commercial Affymetrix mouse 430.2 platform and a non-commercial Rosetta/Merck one. The data from each platform were preprocessed in the same manner.a priori assumption of any of these distributions across all probe sets is not valid. The pattern of null hypotheses rejection was different for the data from Rosetta/Merck platform with only around 20% of the probe sets failing the logistic distribution goodness-of-fit test. We find that there are statistically significant (at 95% confidence level based on the F-test for the fitted linear model) relationships between the mean and the logarithm of the coefficient of variation of the distributions of the logarithm of gene expressions. An additional novel statistically significant quadratic relationship between the skewness and kurtosis is identified. Data from both microarray platforms fail to identify with any one of the chosen theoretical probability distributions from an analysis of the l-moment ratio diagram.The null hypotheses for goodness of fit for all considered univariate theoretical probability distributions  are rejected for more than 50% of probe sets on the Affymetrix microarray platform at a 95% confidence level, suggesting that under the tested conditions  The current biological literature makes extensive use of gene mRNA expression data from experimental systems called gene chips/gene micro-arrays. These data are used to infer genomic level conclusions. For example, to infer the response of an organism or cell culture under treatment or perturbation. Microarrays as an experimental system are very valuable in that they provide a genome-wide picture  genes). Unfortunately, because of costs of collecting microarray data, the number of samples per treatment is quite small (~2-10).The data from microarrays are noisy. There are a number of reasons to expect variability in the measurements of the expressions of the genes in mammalian organisms. These include biological causes, or the noise associated with the steps involved in the measurement of the gene expression. Depending on the question that the researcher is trying to answer, he/she would have to control for many of these sources of variation of gene expression. This paper is interested in understanding the variation in the expression data after the known/reported sources of variation have been controlled for.The biological variability could be due to genetic or non-genetic factors  studies Among the non-genetic factors explaining the variation in gene expression include the gender of the organism -   have been termed intrinsic, extrinsic and pathway-specific or global noises  being normalized and preprocessed to get estimates of gene expressions. RNA are isolated from the cells obtained from the tissue sample has been drawn from the organism. The RNA are then subjected to the process of reverse transcription (RT) to obtain cDNA that are then subjected to the vitro transcription (IVT) process to obtain cRNA using polymerases. The cRNA are then hybridized to probes on the microarray platform . The facMost of the current journals require the microarray data to be deposited on a database  if thesOne of the main areas where microarray data has had its application is in the identification of differentially expressed genes across varying treatment conditions. The approaches used could be classified based on whether they use parametric assumptions about the underlying distribution the gene expression or not. Kerr et al  used a AThe above paragraph describes a snapshot of the analysis done using gene expression data, some of the analyses make use of distributional assumptions and some do not. Since distributional assumptions are made frequently, it appears prudent to validate this assumption. As mentioned above Newton et al  and KerrSince 2002 a significant amount of data from sources like the Gene Expression Omnibus GEO  and ArraThis paper focuses on identifying and validating empirical distribution fits of genome-wide gene expressions as measured by microarrays. In addition to the normal distribution we empirically tested the empirical fit for a number of well established probability distributions.We analyzed four microarray data sets from the GEO database . They weThe microarray samples used in the analyses in this manuscript were based in part on six separate data sets  were analyzed in a number of papers -54 whereFor each of the four Affymetrix data sets used, raw CEL format data from GEO was normalized using the R Bioconductor  implemenFor the data set of 6219 microarray samples RAM memory limitations prohibit normalizing all samples together. To normalize these, the following steps were followed:1. The samples were partitioned into sets of 75.2. The gene expression data for each of these 75 samples were obtained after running the GCRMA routine.Unpublished manuscript 2001.).3. Using the data from step 2, the gene expression across the whole data set of 6219 samples were normalized with respect to each other using the quantile normalization method as described in with these active genes were also chosen. This combined set of genes is what is used in the analysis in this paper. In the interest of having a relatively large number of genes for analysis, further criteria used by Yang et al [Yang et al  that useng et al  like preFor each of the data sets "Craniofacial", "Liver" and "Brain", "Housekeeping", "Male" and "Female", Kolmogorov-Smirnov (KS) and Anderson-Darling (AD) hypothesis tests were used to test distributional assumptions. Both test the null hypothesis that a set of data comes from a given distribution, with distributional parameters possibly unknown. The AD test is more sensitive to differences in the tails of the data than the KS distribution. In testing for Normality, the AD test is known to be more powerful than the KS test . Normal,F, each of identified  probe sets were tested in the following manner:For each of the data sets and each distribution F1. Use maximum likelihood estimation (MLE) to estimate 2. Use the KS and AD tests at the 90% and 95% to test whether the probe data comes from For each of these distributions, MATLAB version R2009B Statistics Toolbox MLE functions were used. Critical values can be found in .In addition, for housekeeping genes, KS and AD tests were used to test for gamma and Pareto distributions across a set of 6219 samples. For these distributions, tables of critical values do not exist, but a method for generating p-values for KS tests with unknown distribution parameters from  was usedThe l-moment ratio diagram ,64 of l-The "Male" and "Female" data were fitted to mixture of normal distributions using the "mixdist" package  in the RThe Kruskal Wallis test was used on the logarithm of expression of the probe sets in each of the "Craniofacial", "Liver" and "Brain" data sets in order to identify those that were most likely unaffected by any of the conditions involved in the generation of these data sets. The results in Table The results of the goodness of fit Anderson-Darling distribution tests for the "Male" and "Female" data sets showed different characteristics from those of the other data sets. Only around 43-46% of the probe sets rejected the normal hypothesis (as compared with 72-82% for the previous data sets). The logistic distribution was rejected less often than the previous data sets (21-24% as compared with 69-79%). The fit of the extreme value distributions were equally bad for both sets of data. This difference in characteristics of the goodness of fit test results between two different microarray platforms indicates the contribution of technology and/or of the normalization methodology to the distribution characteristics of microarray data.2 values for different polynomial fits . So the mean values of skewness and kurtosis are plotted by the red lines in Figure The dependence of the mean on the higher order product moments are shown in Figures K) and the skewness (denoted by S) have to satisfy the following inequality ,The data in Table Further validation of the lack of fit of gene expression to any of the standard theoretical univariate probability distributions can be seen in the L-moment ratio diagram in Figure The samples from the "Male" and "Female" datasets could be considered more or less homogenous with respect to sex, tissue, diet and experimental conditions. One reason we could be seeing poor fits to standard distributions could be that there are different modes to the distribution of the expression of a given gene reflecting the genetic variation in the F2 cross animals or a stochastic network influence  of its expression . Hence iIn the past several years, there has been an explosion in amount of quantitative biological data either in terms of transcriptomics, sequencing data, genetic structure variation, proteomics or metabolomics. DNA microarrays have been important and valuable resource for understanding perturbations to biological systems in terms of identifying affected gene expressions. The standard statistical methods are being either directly used or modified to work with gene expression data. Unfortunately, only a small number of replicate samples per treatment (2-10) are used for analysis owing to the cost of the experimentsal system. This point plus the fact that the probability distributions of gene expressions as measured by these arrays were not characterized leads one of logically question the use various statistical methods that are based on distributional assumptions. Heuristics are being proposed that attempt to relax the reliance on this distributional assumption. One example of this would be the method of jointly using the p-value from a two sample t-test along with the gene expression fold-change to identify differentially expressed genes. Alternately, there is an increased use of non-parametric methods or permutation-based methods ,38.The essential question that we address in this manuscript is whether the distribution of the logarithm of gene expression as measured by DNA microarrays can be approximated by any of the standard theoretical univariate probability distributions. The results in Table The observed distributional characteristics of gene expression data in this manuscript suggest either the need for the use of non-parametric statistical methods or a need to develop newer statistical/mathematical approaches that are capable of and are optimal for working with these kinds of distributions.Because of the nature of the data used in this paper, we are unable to separate the contribution to the variation of the data due to biological reasons from those induced by the microarray technology. The noise we observe is probably the result of the convolution of these two factors. In light of the increasing use of newer technologies like those based on Next Generation Sequencing  and despThe analyses of the empirical probability distribution of gene expressions from five publicly available data sources with relatively large number of samples have been described in this manuscript. The failure of the distributions to follow any of the known theoretical univariate probability distributions has been demonstrated though the data suggests consistent relationships between the different moments of the distributions. These moment relationships should motivate the development of Bayesian methods with appropriately chosen priors.RT and SM designed the study. LdT carried out the all goodness of fits analyses and contributed to the write-up. XC contributed in checking the annotations of the entire microarray data samples used and also in the interpretation of the results. RT performed the l-moment analysis and drafted the manuscript. SM also contributed to the drafting of the manuscript. All authors read and approved the final manuscript.Table S8: Detailed description of the data in Table Click here for fileTable S3: List of housekeeping genes [ng genes  analyzedClick here for fileTable S1: GEO [ S1: GEO  microarrClick here for fileTable S2: GEO [ S2: GEO  microarrClick here for fileTable S6: Results of goodness of fit tests for the Anderson-Darling tests for the six analyzed probability distributions obtained by the varying the cutoff for the Kruskal-Wallis test.Click here for fileTable S4: Best fit distribution Kolmogorov-Smirnov test results for "Craniofacial", "Liver", "Brain", "Male" and "Female" data setsClick here for fileTable S5: Probe set level best fit distribution results for the Anderson Darling (AD) and the Kolmogorov-Smirnov test (KS) tests at 90 and 95 percent confidence levelsClick here for fileFigure S1: Diagnostic plot for the linear model between the logarithm of the coefficient of variation (CV) and the mean of the distribution of the logarithm of gene expression for the "Craniofacial" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the mean and logarithm of the CV respectively.Click here for fileFigure S2: Diagnostic plot for the linear model between the logarithm of the coefficient of variation (CV) and the mean of the distribution of the logarithm of gene expression for the "Liver" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the mean and logarithm of the CV respectively.Click here for fileFigure S3: Diagnostic plot for the linear model between the logarithm of the coefficient of variation (CV) and the mean of the distribution of the logarithm of gene expression for the "Brain" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the mean and logarithm of the CV respectively.Click here for fileFigure S4: Diagnostic plot for the quadratic model between the kurtosis and the skewness of the distribution of the logarithm of gene expression for the "Craniofacial" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the skewness and kurtosis respectively.Click here for fileFigure S5: Diagnostic plot for the quadratic model between the kurtosis and the skewness of the distribution of the logarithm of gene expression for the "Liver" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the skewness and kurtosis respectively.Click here for fileFigure S6: Diagnostic plot for the quadratic model between the kurtosis and the skewness of the distribution of the logarithm of gene expression for the "Brain" data set. Note the residual plots (subplots (b) and (c)) also provide the pearson correlation (denoted by ρ) between the absolute value of the residuals and the skewness and kurtosis respectively.Click here for fileFigure S7: l-moment ratio diagram for the four data sets as in Figure 6 but including all probe sets including those with mean expression less than 6.Click here for fileFigure S8: l-moment ratio diagram for the four data sets as in Figure 7 but including all probe sets including those with absolute value of mean of log expression ratio less than 0.05.Click here for fileTable S7: Spearman rank correlation between the mean and standard deviation of the measured data for "Craniofacial", "Liver", "Brain", "Male" and "Female" data sets.Click here for file


We have made recent improvements to caCORRECT, including the development of a model-based data-replacement strategy and integration with typical microarray workflows via caCORRECT's web portal and caBIG grid services. In this report, we demonstrate that caCORRECT improves the reproducibility and reliability of experimental results across several common Affymetrix microarray platforms. caCORRECT represents an advance over state-of-art quality control methods such as Harshlighting, and acts to improve gene expression calculation techniques such as PLIER, RMA and MAS5.0, because it incorporates spatial information into outlier detection as well as outlier information into probe normalization. The ability of caCORRECT to recover accurate gene expressions from low quality probe intensity data is assessed using a combination of real and synthetic artifacts with PCR follow-up confirmation and the affycomp spike in data. The caCORRECT tool can be accessed at the website: We demonstrate that (1) caCORRECT's artifact-aware normalization avoids the undesirable global data warping that happens when any damaged chips are processed without caCORRECT; (2) When used upstream of RMA, PLIER, or MAS5.0, the data imputation of caCORRECT generally improves the accuracy of microarray gene expression in the presence of artifacts more than using Harshlighting or not using any quality control; (3) Biomarkers selected from artifactual microarray data which have undergone the quality control procedures of caCORRECT are more likely to be reliable, as shown by both spike in and PCR validation experiments. Finally, we present a case study of the use of caCORRECT to reliably identify biomarkers for renal cell carcinoma, yielding two diagnostic biomarkers with potential clinical utility, PRKAB1 and NNMT.caCORRECT is shown to improve the accuracy of gene expression, and the reproducibility of experimental results in clinical application. This study suggests that caCORRECT will be useful to clean up possible artifacts in new as well as archived microarray data. The reproducibility and reliability of microarray data is a major issue that must be addressed before microarrays can reach their full potential as a clinical molecular profiling tool for personalized and predictive medicine . The FDAQuality Assurance (QA) tools, such as SmudgeMiner  and arrazj and both methods use neighboring information. caCORRECT, however, allows for iterative calculation of this score, and thus allows for the same probe location to be corrected on more than one chip in a batch, whereas the methods of Arteaga-Salas et al. do not . The critical assumption behind quantile normalization is that for large genome-wide studies such as microarrays, the number of genes that are invariant to the experimental variables far outnumbers the number of biomarkers-- genes that respond to or predict experimental variables. Quantile normalization is generally good for the microarray problem, where the distributions are poorly defined and parametric methods such as median centering or Z-score normalization have their underlying assumptions violated. The power of quantile normalization comes with a major caveat. If the probe intensities of the chips are not distributed similarly, the algorithm will indiscriminately warp all the distributions to be the same, including any that may have been correct initially. Fortunately, it is a reasonable assumption that high-quality microarray data from a single source on a single platform follow the same distribution. Unfortunately, this high quality assumption is not valid for much real-world data, where chip artifacts can significantly alter the distribution of intensities on a chip. One bad chip can warp the others when quantile normalization is performed, thus compromising the reproducibility of the entire data set. A way to alleviate this problem is to identify artifacts before quantile normalization, and set them aside temporarily. In theory, perfect knowledge of artifacts would allow for perfect correction. This process is called "artifact-aware quantile normalization." caCORRECT uses four iterations of artifact-aware normalization and artifact identification in order to achieve a near steady-state normalization result with a relatively small amount of computation time.To illustrate the invasive effect that artifacts can have on a dataset when quantile normalization is performed, synthetic microarray data were generated in the following manner: Six high-quality chips from the Schuetz et al. dataset  were choclean dataset is generated with the artifactual data appropriately replaced. The current version of caCORRECT uses a data imputation that is mathematically similar to the model used by RMA and others [Once artifacts have been identified, a d others ,9,10. NoObserved microarray intensity values after global normalization are modeled as a multiplicative combination of target RNA abundance (gene expression) and probe-specific effect (probe affinity). The model is given as:xb,p,j is the observed intensity for the bth probe in the pth probe set on the jth chip, θp,j is the gene expression term, ab,p is the probe affinity term, and εb,p,j is the additive error term.where pth probe set using an additive model of error can be represented in the following matrix form, given a set of N chips and Bp probes in the pth probe set.The set of equations for the εp as defined below.We define the solution to this matrix equation as that which minimizes the Frobenius norm of the above error matrix p. Here, the SVD of Xp is given in the form of Xp = USVT, such that,S, s1, is arranged as the first diagonal element of S, then θp is s1 times the first column of U and V.The solution which satisfies the above condition can be derived from the singular value decomposition (SVD) of Xab,p equals one. The number one is arbitrary here, but it allows the convenient interpretation that the values of gene expression, θ p,j, are on the same scale as the probe intensities, xb,p,j. To satisfy this constraint, the earlier solution for ap and θp can simply be scaled by respective multiplication and division.We chose to introduce the additional constraint that the geometric mean of the lumped probe affinity terms θp and ap learned from SVD, the probe intensities which are expected from the model can be generated via the multiplication θpap Incorporation of artifacts into this model is done in the following 2-step Expectation-Maximization algorithm until θp and ap converge.With values of θp and ap from the SVD of the observed data Xp.1) Estimate model parameters Xp with information from the corresponding elements of θp ap.2) Replace known artifact values in Xp with values from θp ap has the effect of reducing corresponding values of εp to zero, and thus has the property of never increasing the Frobenius norm of εp. Since step 1 is a global minimization of the Frobenius norm of εp given Xp, and step 2 alters Xp in a way that can only further decrease this error, the entire procedure is guaranteed to converge due to the non-increasing nature of the error function, which is naturally bounded by zero. Troyanskaya et al. have previously used a similar "SVDImpute" procedure for missing gene expression data [This procedure of replacing values in ion data . Please In order to quantify the ability of caCORRECT to improve data quality, we altered public microarray datasets with a variety of randomized synthetic artifacts, and then processed the altered data with caCORRECT or Harshlighting. Datasets were generated from a variety of clinical cancer studies using different microarray platforms. To date, the caCORRECT website has been tested with data from 18 different Affymetrix platforms, but the results of our synthetic artifact analysis which are presented here are limited to our in-depth study of two key data sets. Three separate experiments were performed involving application of synthetic artifacts.First, we obtained a large data set that was originally generated by Hess et al.  in the shttp://arraywiki.bme.gatech.edu/index.php/Hall_of_Fame for some examples [A common criticism of our synthetic artifact work is that the size and severity of tested artifacts (for example those provided by Jones' team) are rarely observed in practice. While we have encountered many chips plagued by large, severe artifacts, (see examples ), we setFinally, a realistic mixture of less-severe artifacts were applied to the Schuetz et al.  dataset The dataset which was used to showcase real-world artifact removal and replacement was a set of 49 Hu-6800 Affymetrix microarrays from the study by West et al. This study investigated Estrogen Receptor and lymph node metastatic status . This dahttp://omniBiomarker.bme.gatech.edu). All PCR analysis was performed on independent patient tissue samples with respect to those used for the microarray analysis.To determine the effect that caCORRECT had on the ability to correctly identify biomarkers of disease from microarray data, a panel of 96 genes of interest for RCC was assembled for PCR study in two phases. These genes were identified from a combination of genes previously identified in the literature as well as a set of genes whose biomarker status was disagreed upon between the caCORRECT and non-caCORRECT versions of the Schuetz et al. data sets using omniBiomarker . Up to 3 μg of RNA was used for first strand cDNA synthesis with Superscript III (Invitrogen). PCR was performed with a custom-designed Taqman Low Density Array  in a 96-well microfluidic card format, using the ABI PRISM 7900HT Sequence Detection System . Gene expression data were normalized relative to the geometric mean of two housekeeping genes . LDA runs were analyzed by using Relative Quantification (RQ) Manager (Applied Biosystems) software.t (threshold cycle) value difference of 1.Relative normalized PCR gene expression was compared in renal tumor subtypes. Genes were declared as being "validated by PCR" if they had an average fold change between classes of magnitude greater than 2, corresponding to an average CThe authors declare that they have no competing interests.RAM helped design caCORRECT, including artifact-aware normalization, variance scoring, and missing data imputation methods. RAM designed and performed all microarray-based validation experiments, helped perform PCR validation, and drafted the manuscript. QYG performed biological specimen preparation and PCR validation. THS helped design caCORRECT, including quality scoring, created the caCORRECT web portal and its companion ArrayWiki, and helped caCORRECT to receive silver-level compatibility certification from NIH/NCI caBIG. RMP helped design and prototype caCORRECT's data imputation. JHT participated in PCR experimentation and the collection of source microarray data. JHP implemented caCORRECT's normalization and performed all gene ranking done with SVM. ANY provided all biological specimens, contributed to manuscript preparation, and directed the PCR validation. MDW initiated the microarray quality control investigation, acquired funding to sponsor this multi-year effort, and directed the caCORRECT project, software system certification, and publication. All authors read and approved the final manuscript.Supplemental Materials. Additional descriptions of methodology for: calculation of variance score, implementation of gene expression model imputation, biomarker selection, and PCR experimentationClick here for file


Microarray technology is widely utilized for monitoring the expression changes of thousands of genes simultaneously. However, the requirement of relatively large amount of RNA for labeling and hybridization makes it difficult to perform microarray experiments with limited biological materials, thus leads to the development of many methods for preparing and amplifying mRNA. It is addressed that amplification methods usually bring bias, which may strongly hamper the following interpretation of the results. A big challenge is how to correct for the bias before further analysis.In this article, we observed the bias in rice gene expression microarray data generated with the Affymetrix one-cycle, two-cycle RNA labeling protocols, followed by validation with Real Time PCR. Based on these data, we proposed a statistical framework to model the processes of mRNA two-cycle linear amplification, and established a linear model for probe level correction. Maximum Likelihood Estimation (MLE) was applied to perform robust estimation of the Retaining Rate for each probe. After bias correction, some known pre-processing methods, such as PDNN, could be combined to finish preprocessing. Then, we evaluated our model and the results suggest that our model can effectively increase the quality of the microarray raw data: (i) Decrease the Coefficient of Variation for PM intensities of probe sets; (ii) Distinguish the microarray samples of five stages for rice stamen development more clearly; (iii) Improve the correlation coefficients among stamen microarray samples. We also discussed the necessity of model adjustment by comparing with another simple adjustment method.We conclude that the adjustment model is necessary and could effectively increase the quality of estimation for gene expression from the microarray raw data. Gene expression microarrays are widely utilized for transcriptome analysis of biological samples from different treatments or different phenotypic groups. However, due to the difficulty of extracting sufficient amount of starting mRNA or total RNA from biological materials, such as rice stamen, it is usually impossible to detect the amounts and sequences of mRNA directly. Therefore, amplification of mRNA sample is necessary before performing microarray experiment to detect the fluorescence signals of mRNA -4. RNA lHowever, two-cycle amplification produces larger bias associated with higher amplification efficiency ,8. In orin vitro transcription (IVT) stage during the amplification process. Another reason might be the usage of random primers in the start of the second round amplification, which concludes uncompleted reverse transcription from aRNA to cDNA. The third reason may be the saturation of probe spot on microarray chip. Therefore, it is necessary to revise the microarray data generated with two-cycle RNA amplification before using it to perform further analysis, such as detecting differentially expressed genes, constructing co-expression gene network and so forth.It is also notable that, for mRNA transcripts, the decreasing trend of the two-cycle labeling is heavier than that of the one-cycle labeling, which implies that we are probably misled. Three possible reasons may account for this phenomenon. The first one is the degradation of transcript due to RNA’s instability, especially in 16 hour’s A few works before have also mentioned the problem of RNA degradation during two-cycle labeling -12. HoweThis paper proceeds as follows. In the second section, we described three types of microarray experiments and a Real Time PCR experiment to validate the degradation effect. After that, we proposed a probabilistic model for probe level adjustment, and the parameters are estimated by MLE. In the third section, we displayed the bias existed in microarray experiment and validated the bias by Real Time PCR. Then we discussed the effects that might bring bias, especially the degradation effect in the RNA two-cycle labeling protocol. By applying our model to two stamen microarray data sets, we showed that our model adjustment obtained a significant improvement for the quality of microarray raw data. Finally, we discussed the necessity of our model adjustment and a possible application to RNA-seq data in the last section.In this chapter, we first briefly introduced three kinds of microarray data of rice, and then described the designed Real Time PCR experiment to revalidate transcriptional degradation. For brevity, we refer to these three data sets as Data Set 1, Data Set 2 and Data Set 3, respectively. Plant materials as well as sample collection and total RNA isolation method are the same in three data sets. Data Set 1 was designed to show the phenomena of RNA degradation and to estimate parameters in the model. Data Set 2 was designed to validate the reproducibility of the bias from RNA degradation and to determine whether the bias is sample specific. Data Set 3 was applied to validate the efficiency of our adjusting model. After that we established a probabilistic model to estimate the extent of bias caused by RNA degradation during two-cycle linear amplification and attempted to correct for this bias. As mentioned above, the reason why bias is introduced is mainly due to the using of random primers and RNA degradation, on which our model is based. The saturation effect of micoarray chip is beyond our consideration.C for about one week. And then seedlings were transplanted in plastic pots and cultured at 30 ±2°C under 11 hours day/13 hours night cycle.Rice (O. sativa L. ssp. japonica cv. Zhonghua 15) seeds were soaked in water at 30°The rice embryos were obtained from the seeds soaked in water for 2-3 days. The third mature expanded leaves were collected when they had just fully expended. Rice seedlings were harvested when they were 2-3 cm in height. The panicles were collected when they were 0.6-4 cm in length. Total RNA was isolated separately from rice embryos, the third mature leaves, the fifth mature leaves, seedlings and panicles with TRIzol Reagent from Invitrogen Life Technologies.To obtain microarray data covering almost all genes in rice, total RNA from different samples described above was mixed into a big pool called Pre-amplified mRNA sample (PAM) by almost equal magnitude. Eight subgroups were separated from the purified PAM. The half of them were amplified and labeled by the One-Cycle Eukaryotic Target Labeling Assay from Affymetrix . The resulting samples were called One-Cycle cRNA samples (OCS). The other half were amplified and labeled by the Two-Cycle Eukaryotic Target Labeling Assay from Affymetrix . The resulting samples were called Two-Cycle cRNA samples (TCS). All experimental procedures strictly followed instructions specified in the Affymetrix GeneChip Expression Analysis Technical Manual.This data set contained microarray data from two different samples, including the third leaf primordium and the third mature leaves of rice. Total RNA was extracted from the two samples. And then each of them was separately amplified by two methods, the One-Cycle Labeling Assay and the Two-Cycle Labeling Assay . Two replicates were applied for each Assay. Totally there are 8 hybridization samples for the following microarray experiments.This data set included 21 slides of Affymetrix microarray. There are 7 different stages from rice, including the third leaf primordium, the third mature leaves, and stamen samples at stage 2, 3, 4, 5 and 6 (Stages are partitioned by the development of Stamen) ,14. Biolhttp://www.math.pku.edu.cn/teachers/dengmh/RiceTCAC/index.html.All amplified RNA samples were hybridized with the Affymetrix GeneChip Rice Genome oligonucleotide arrays, which is widely used for rice gene expression analysis. The.CEL files are available at To validate the degradation trend mentioned above, we developed a strategy by applying Real Time PCR to several transcripts. Real Time PCR were carried out using the same RNA samples as that of Data Set 1, including Pre-amplified mRNA samples (PAM), One-Cycle cRNA samples (OCS), and Two-Cycle cRNA samples (TCS).http://www.genscript.com/cgi-bin/tools/primerâˆ–_genscript.cgi?op=standard). Each primer was further confirmed by dissociation curve analysis after the PCR reactions. First strand cDNA was synthesized by reverse transcription using 1 μg of total RNA in 20 μl of reaction volume using SuperScriptTM III Reverse Transcriptase from Invitrogen Corporation. Diluted cDNA samples were used for Real Time PCR analysis with 100 nM of each primer mixed with SYBR Green PCR master following manufacturer’s instructions. The reactions were carried out in Optical 96-Well Fast Plate on the 7500 Fast System .In brief, primers were designed for transcripts using Real Time PCR Primer Design tool (Rij (see Equation 1). We designed several pairs of primers locating from 5’end to 3’end of a transcript. The target of each pair of primers was called amplicon. Then we used the pairs of primers to carry out Real-time PCR with PAM, OCS and TCS separately. The Schematic diagram of Real Time PCR experiments is shown in Additional file MAX (Rij)) located in the region with minimal impact by degradation and amplification. And MAX(Rij) was closest to the actual expression level of the transcript. So we calculated the DPij of each amplicon relative to the MAX (Rij) to make clear the impact of degradation and amplification on different regions of transcript: Where CT (cycle threshold) is the number of cycles required for fluorescent signal to reach the threshold. The larger CT is, the less the amount of RNA is. For a fixed j, among all the amplicons of a transcript, amplicon with the maximum relative expression value  be the positions of 3’ end and 5’ end of a transcript before amplification, where L is the length of the transcript. Let Ai and Bi be the position of the new 3’ and 5’ end after the ith shorten . It’s obvious that A0 = 0, B0 = L. (i) After the first cycle amplification, based on the uniform assumption of RNA degradation, we further assume A1 ~ U and B1 ~ U, where a,b are the degradation limits of 3’ end and 5’ end separately. (ii) In the second cycle of amplification, 3’ and 5’ ends varies more complicated. As random primer is used in the step of the first strand cDNA synthesis from antisense RNA produced in the first cycle of amplification, which may result in incomplete synthesis. 5’ end is decreased while 3’ end remains the same. Therefore, the position of new 3’ end A2 is the same as A1, while the position of new 5’ end B2 ~ U, as random primer can bind to any position of the cRNA. (iii) As the same as in the first cycle, degradation takes place at both 3’ end and 5’ end in the IVT steps of the second cycle amplification. So it’s similarly concluded that A3 ~ U and B3 ~ U, where a,b are the same as in the first cycle.To better understand the model, we introduce the notation. To simplify calculation, we first assume that RNA is degrading at an even speed, regardless the sequence effect, e.g. GC content. Let 0 and Ai,Bi),i = 1,2,3.As most RNA transcripts are more than 200 bp, we utilized integral to substitute sum for calculation. Then we could obtain the joint distribution of (A1 and B1 is:  i) The joint distribution of A2 and B2 is:  ii) The joint distribution of where A3 and B3 is:  iii).The joint distribution of where F3 can be rewritten out piecewise , i = 1,2, which states a probability indicates how probable the the nucleotide on the position of z (bp) retains after the incomplete synthesis in the ith cycle amplification. Here, z, in , represents the distance of a nucleotide on the transcript away from its 3’ end. After the first cycle amplification, the Retaining Rate function p1(z), can be easily calculated from the distribution of 3’ and 5’ end: Thus p2(z) is complicated piecewise, associated with F3. We calculated it by integrating with R software.But for the second cycle amplification, the Retaining Rate function z is p2(z) (Note that z refers to distance of the middle nucleotide of probe from 3’ end of the transcript). For the kth poly(A) RNA, the ideal intensity of ith probe after jth cycle, j = 1,2, is IPSjk; and the observed intensity (PM intensity) for ith probe at position zik is IPijk. Then we can get IPijk as a function of IPSjk: For each transcript, there is a probe set containing proximately 11 probes (99.58% of 57381 probe sets for rice microarray) on microarray to detect its the expression signal. We first define a Retaining Rate for a position on transcript, which is a probability indicates how probable the position of the probe on transcript remains after two-cycle amplification. Before the first step of two-cycle amplification, the Retaining Rate for each probe is 1, when amplification is over, the retaining rate for probe at j, k and all i, For determined IPSjk is: Thus, the maximum likelihood estimation for i = 1,2,…nj, nj is the number of probes in the jth probe set.where Although the above model only uses PM of probe for training the parameters, it can also be used to adjust MM intensity, as Retaining Rate represents the probability of the nucleotide of each position on transcript to be remained after two-cycle amplification. As our model only correct for the bias caused by degradation during amplification at probe level, the corrected intensity didn’t perform the background correction or normalization at probe set level. We think it necessary to perform a further pre-processing to adjust for bias introduced by cross-hybridization or other factors. After adjusting for PM and MM, we could apply many existing popular normalization methods, such as PDNN , dchip  and RMA In the beginning of the first cycle amplification, a transcript is complete poly(A) RNA, whose cap in 5’ ends and tail in 3’ ends to some extent protect the RNA sequence. Thus, degradation of the transcript is relatively slow in this process. However, when the second cycle begins, as the RNA cap in 5’ end and tail in 3’ end have degraded or almost degraded, the degrading effect becomes heavier and thus leads the RNA sequence to be shorter . BesidesRegardless the decaying speed of a mRNA varying by different ribonucleic acids and its secondary structures, we assume that a ribonucleic acid on mRNA degrades linearly relied on the distance to an end, either 3’ end or 5’end. Also, we assume that there exists a degrading limit, behind which degradation seldom takes place. We are clear that mRNA degrades from both 5’ and 3’ ends. In one pathway, mRNA shortening is followed by removal of the 5’ cap structure. Decapping gives a 5’-to-3’ exonuclease access to degrade the remainder of the mRNA . In the DPij was to 0, the less the region of ith amplicon was affected by degradation and amplification in the jth sample, and vice versa. Because of degradation and amplification, we could expect that the DP of amplicons near the 5’ end to be higher. And the further away amplicon was from 5’ end, the lower the DP was. But when amplicons were very close to 3’ end of transcript, the DP would increase again because of 3’ degradation. According to the annotation from Affymetrix company, LOC_Os06g07140 (Gene Nomenclature from MSU Rice Genome Annotation Project) was represented by probe set Os.4216.1.S1_a_at on the GeneChip Rice Genome Array. And its intensity decreased dramatically in TCS comparing with OCS  of each amplicon. The details about DP were described in method section. Briefly, the closer In conclusion, our Real Time PCR results indicated the correlation between position and intensity of probes in amplification processes: the probe close to the 5’ end would be underestimated because of degradation and random primer effect, while the probe very close to the 3’ end would be also underestimated because of 3’ degradation; the probe located but still had a distance to 3’ end might have the most exact expression measurement of probe set.As probes of a same probe set measures the intensity of the same transcript, we expect the PM intensity of these probes to be more alike, i.e. have smaller Coefficient of Variation (CV). Using the microarray data generated with two cycle amplification in Data set 1, we calculated the CV for present probe sets before and after model adjustment, as shown in Figure To further evaluate the biological meaning of our adjusting model, we created 21 rice microarray samples  using the Affymetrix two-cycle RNA labeling.We first adjusted the PM intensity for 15 stamen microarray samples using our adjustment model, and then performed PDNN [Several works have considered the effect of mRNA degradation in microarray during the process of two-cycle linear amplification as well as the bias it caused , howeverIn this study, focusing on the process of two-cycle linear amplification and the features of gene expression microarray data, we found two key factors that could bring biases: (i) Reverse transcription by random primers; (ii) RNA degradation in the first and second cycle amplification. We modeled these processes with uniform distributions and established a model to correct the bias. As we performed the adjustment for PM at probe level, one could further apply any kinds of microarray preprocessing methods, such RMA, dchip or PDNN to perform normalization. Note that we could not only apply the model to PM, but also to MM, as the Retaining Rate measures how probable each position of transcript remains after amplification, which has no relationship to binding mechanism. Other preprocessing methods considering MM could also be combined to do further reliable normalization.To evaluate our model, we applied the adjustment method to microarray data of Data Set 1 and 3. From three aspects: (i) Coefficient of Variation for probes within a probe set decreases significantly, (ii)The clustering diagram shows more reasonable classification for rice stamen samples and (iii) the relationship among them become be much more closer, we could see that our model had obtained relatively more biologically reasonable results.To demonstrate the necessity of our model for adjusting, we applied another adjusting method that assigning different weight to each position of transcript according to mean PM expression intensity. The main steps are designed as follows: (i) Plot the mean PM intensity for probes at 12 ∼588bp (probes in 98.68% of present probests) of transcripts. (ii) Apply lowess  to fit the data. Then compute loess smoothed values for all points along the curve. Normalize all loess smoothed values to make their mean to be 1. Take the reciprocal of the normalized value at each position as the weight for probes at this position . (iii) Adjust PM at each position by multiplying the PM intensity by the weight. (iv) Combine with known preprocessing methods . We call this process of adjusting Curve Adjustment (CA for short). To compare CA with our method, we applied both of them to Data Set 1 and 3. We could see from Figure S6 that, the CV didn’t decrease and the clustering of 15 samples were almost the same as that of none adjustment . Besides, the sample correlation coefficients didn’t raised much after CA . Thus, these results indicate that direct curve adjustment for microarray data is not suitable and our model adjustment is necessary. See Additional file Although we could correct for the bias to some extent, there are still some challenges, such as the assumption of the distribution for random primer to be union, which may be different from actual condition. Besides, we applied a simple linear model to simulate RNA degradation and didn’t consider the different degradation rate caused by different nucleotides, such as GC-content, which may play an important role in two-cycle amplification. A new challenge will be the combination of both position of probe and sequence preference to make more accurate correction of bias.With the development of Next Generation Sequencing, RNA-seq is widely utilized to measure gene expression at transcriptional level at unprecedented precision and throughput -24. Our Although the approaching of Next Generation Sequencing leads to more accurate results of gene expression, it is now still relatively too expensive. Microarray will still be a feasible way to measure gene expression in the near future. Our method could be applied to reuse the existing microarry data generated with two cycle amplification protocols, and the more biological promising results obtained by the adjusting model will surely benefit the following analysis, e.g. detecting differentially expressed genes or gene set analysis.The authors declare that they have no competing interests.FW and DJ were responsible for the construction of the statistical model, FW and RC produced the manuscript, while RC and SB designed the microarray and Real Time PCR experiments and analyzed the Real Time PCR data. MD, SB and MQ provided essential suggestions for this work and helped to prepare the manuscript. All authors read and approved the final manuscript.Supplemental Figures.Figure S1. Position of probes on their transcripts (bp) (away from 3’ end) of Affymetrix GeneChip Rice Genome oligonucleotide arrays.Figure S2. Correlation between position and intensity of probes for present probe sets (By MAS5.0) in Leaf and Leaf Primordium microarray data.Figure S3. Schematic diagram of Real Time PCR experiments.Figure S4. The Real Time PCR results for other transcripts show similar trends as in Figure 2.Figure S5. Estimation of weight for curve adjustment.Figure S6. Distribution of the Coefficient of Variation (CV) for PM intensities of present probe sets after 3 preprocessing methods.Figure S7. Hierarchical clustering of 15 microarray samples after 3 preprocessing methods.Figure S8. Histogram of correlation coefficients between 15 microarray samples after 3 preprocessing methods.Supplemental Formula. Formula F1: The joint distribution for positions of the new 3 end and 5 end after the 3th shorten A3 and B3: F3.Supplemental Results and Discussion. Comparison with Curve Adjustment to demonstrate the necessity of our model for adjusting bias. A simple adjusting method that assigns different weight to probes at different position of transcript according to expression intensity was applied, but the result indicates that direct curve adjustment for microarray data is not suitable and Model adjustment is necessary.Click here for file