High-throughput gene expression profiling technologies generating a wealth of data, are increasingly used for characterization of tumor biopsies for clinical trials. By applying machine learning algorithms to such clinically documented data sets, one hopes to improve tumor diagnosis, prognosis, as well as prediction of treatment response. However, the limited number of patients enrolled in a single trial study limits the power of machine learning approaches due to over-fitting. One could partially overcome this limitation by merging data from different studies. Nevertheless, such data sets differ from each other with regard to technical biases, patient selection criteria and follow-up treatment. It is therefore not clear at all whether the advantage of increased sample size outweighs the disadvantage of higher heterogeneity of merged data sets. Here, we present a systematic study to answer this question specifically for breast cancer data sets. We use survival prediction based on Cox regression as an assay to measure the added value of merged data sets.Using time-dependent Receiver Operating Characteristic-Area Under the Curve (ROC-AUC) and hazard ratio as performance measures, we see in overall no significant improvement or deterioration of survival prediction with merged data sets as compared to individual data sets. This apparently was due to the fact that a few genes with strong prognostic power were not available on all microarray platforms and thus were not retained in the merged data sets. Surprisingly, we found that the overall best performance was achieved with a single-gene predictor consisting of CYB5D1.Merging did not deteriorate performance on average despite (a) The diversity of microarray platforms used. (b) The heterogeneity of patients cohorts. (c) The heterogeneity of breast cancer disease. (d) Substantial variation of time to death or relapse. (e) The reduced number of genes in the merged data sets. Predictors derived from the merged data sets were more robust, consistent and reproducible across microarray platforms. Moreover, merging data sets from different studies helps to better understand the biases of individual studies and can lead to the identification of strong survival factors like CYB5D1 expression. Microarray gene expression data have been integrated to increase statistical power. Increasing sample size is a bottleneck in DNA microarray-based gene expression studies as microarray experiments are time consuming, expensive, noisy and limited to the number of biological samples . To circumvent this problem, microarray gene expression data sets addressing the same or similar biological questions have been analyzed jointly either by so-called meta analysis Some studies combined data sets generated with the same chip A data integration method serves to project expression values for the same gene onto comparable scales. Perhaps the simplest way to approximately achieve this goal is Z-score normalization Another necessary processing step in data merging consists of mapping microarray features to a catalogue of standard gene names. This in turn will result in the definition of the subset of common genes to be retained in the merged data set. Here, the term microarray feature refers to a single hybridization probe, or a set of probes, for which the platform returns a single expression value. Commercially available microarrays often contain multiple features for the same gene. What makes the merging of data sets non-trivial is that different platforms refer to the same genes by different names. Note further that for the reasons outlined above, merging of data sets usually leads to a substantial reduction in the number of genes considered for downstream analysis. Important genes included in only a part of the input data sets may be lost.http://source.stanford.edu) Some studies While some research projects merged the gene expression values in their original continuous representation In previous studies, merging data sets was applied to derive a robust gene signature prognostic of survival time  In previous tests, the potential advantage of data merging was evaluated by means of a quantitative rate of correct classification or validation of previous results. Note however that different studies used different performance measures , percentage of concordance in classification, etc) to this end.Gene signatures are also evaluated in terms of “robustness” and “reproducibility”. Robustness is related to the sample size of the training set from which a gene signature is built and the size of the testing set on which it is validated. A predictor generated from a small training set could have a high prediction accuracy on the training set but may lose generalization power when it is validated on an independent testing set. Moreover, performance estimates obtained with a small testing set have high statistical error, i.e. they come with a large confidence interval. On the other hand, reproducibility In this work, we analyzed the potential benefits of merging data sets for prognostic application in breast cancer diagnosis. Contrary to related work, we did not discretize the clinical follow-up information into good and poor outcome classes, a practice which results in loss of information. Instead, we directly used censored survival data to derive a gene signature that allows for the computation of a risk score from a patients expression profile. The risk score was based on the Cox proportional hazard model, and expected to be inversely related to the time to death or relapse.The basic design of our study is as follows . We usedData sets were merged in their original (continuous) numerical representation using two different data integration methods: (i) ComBat In total, we analyzed 1324 breast cancer samples from public data sets generated with three microarray technologies . To the best of our knowledge, this study is the largest one evaluating the potential benefits of data merging in a quantitative OS/RFS patients risk prediction framework.To assess the removal of microarray bias effect across data sets, Principal Component Analysis (PCA) and hierarchical clustering were applied to the data sets after the application of data integration methods.The results of these tests are shown in For the verification of data integration by PCA, the merged data sets were projected on the planes defined by the first two principal components (PCs). The purpose was to demonstrate intermixing of samples from different sources. As can be seen, ComBat was successful in removing dataset-specific biases  since thThe PCA results were confirmed by clustering. The major goal of our study was to assess the benefits of data merging with regard to disease outcome prediction. To this end, we derived gene signatures based on Cox regression from the individual and merged datasets as described in the We compared two data integration methods: (i) ComBat and (ii) Z-score normalization as described under One possible explanation for this apparent contradiction would be that the first two principle components and the clustering trees obtained after data integration reflect biologically irrelevant technical variance not related to data source. We tested this hypothesis by fitting the first two principal components obtained after the application of ComBat and Z-score normalization to the merged survival data. We obtained the hazard ratios of 1.40 and 0.79 for PC1 and PC2, respectively, after Z-score normalization , and 0.97 and 1.54 for the ComBat merged data sets . While these values do not precisely confirm our hypothesis, they indicate that the first two principal components capture the variance related to survival about equally well in both cases.We observed similar trends when looking at the distribution of Estrogen Receptor positive (ER+) and Estrogen Receptor negative (ER-) samples in the clustering diagrams. There we saw both after the application of ComBat and Z-score normalization many small clusters of ER- samples (the minority class) spread over the entire range of the tree (data not shown). While these findings conciliate the results from data integration verification with those from gene signature evaluation, they also reveal the limited usefulness of the data intermingling test, which in this case provides a misleading picture of the variance retained after data integration.Noting that the gene signatures built from subsets of GSE4335 or Vijver showed higher prediction accuracies in cross-validation than the gene signature built from the merged data set, we investigated how the performance could possibly be improved by selective data integration. With regard to both OS and RFS results, we made the following general observation: (i) The prediction accuracy obtained with a signature from a merged data set lies between the accuracies of the signatures derived from the component data sets. (ii) There are marked differences in performance of signatures from individual data sets, possibly reflecting the quality of the data or the diversity of the analyzed patient cohorts. Overall, these observations are in agreement with results from similar tests where performance was measured by a binary classification assay Taking into account this observation, the data sets with the highest prediction accuracies were merged in different combinations in the hope to improve the overall prognostic power of a signature based on a merged data set. Firstly, the data sets GSE4335 and Vijver were merged and adjusted by a data integration method , resulting in a predictor with an average AUC of 0.73+/−0.01, HR = 4.8 (CI = 4.19–5.50). Adding the data set GSE1992 reduced the prognostic power by 0.01 only . Combining the data sets GSE4335 and GSE1992 resulted in a mean AUC of 0.70 (+/−0.02) and an HR of 3.35 (CI = 2.55–4.30). The AUC obtained from the combination of GSE4335 and GSE1992 is comparable to the result obtained by combining GSE4335 and Vijver but the HR decreased by 1 unit. Adding the Vijver data set to GSE4335 and GSE1992 resulted in a similar prediction accuracy of mean AUC 0.72+/−0.01 but a higher HR = 4.53, CI = 4.03–5.09. These results show that the performance of the merged data set composed of two data sets was as good as the merged data set composed of five data sets. Interestingly, these pairs of data sets (GSE4335 and Vijver or GSE4335 and GSE1992) were generated by different gene expression profiling technologies, indicating that platform-specific biases were not a major obstacle to data merging. Importantly, these results suggest that the potential added-value gained by including a specific data set in a merged data set is predictable from its cross-validated performance measured in isolation.Taken all these together, however, the improvement seen with selectively merged data sets is relatively modest as compared to the prognostic power generated from the merged data sets composed of all OS single data sets. Nevertheless, this improvement is remarkable if one takes into account that the selectively merged data set only contains about one-third (237) or half (410) as many samples as the complete merged data set (849).In the case of RFS , the merComparing the performance results of the merged data sets, we observe that none of the merged data sets was more successful in prediction performance than the best individual data sets. Again ComBat and Z-score integration produced similar results, with a slight advantage of the latter method in this case.To determine the significance and stability of the selected gene signatures, we were then interested to investigate if the top 100-ranked genes would still be selected if a P-value cut-off of 0.05 after Bonferroni correction Regarding the two performance measures used, it is noteworthy that HR was not always directly related to the prediction accuracy expressed as AUC To assess the reproducibility of the gene signatures' performance derived from the merged data sets, the prediction accuracy was evaluated in a leave-one-data set-out manner (section Bias Estimation). In each step, one complete source data set was set aside as testing set while the predictor was built from the merged remaining sets. In parallel, we carried out pair-wise tests, using one source data set as training and another one as testing set. In this test, merging data sets improved overall survival prediction and risk association. This interpretation is based on the fact that for a given testing set, the predictor built from the merged training set outperformed on average 3 out of 4 predictors built from the individual sets. With respect to OS, the survival prediction based on Z-score normalization was higher than the prognosis accuracy obtained with ComBat. A partial improvement of the association of the 100-gene signatures with OS (measured by HR) by merging data sets can also be observed in The partial improvement of survival prediction and risk association that were observed for OS endpoint were also obtained with respect to RFS. Between the merged data sets, Z-score normalization provided higher results than ComBat even though the difference is not significant  was selected in 95% of the cases. The Cox coefficients obtained for this gene were consistently negative, indicating a beneficial effect on survival. With respect to RFS endpoint, this gene was selected in 78% of the cases and the prediction power of the one-gene signature was found to be 0.73.We were not the first to find that a 1-gene signature had an equally good or stronger prognostic power than a gene signatures with a higher number of genes. Haibe-Kains and colleagues The excellent performance of one gene in survival prediction immediately raises the question whether this gene was included in all studies. If this were not the case, then the gene would not be present in the merged data set, which could explain the rather disappointing prediction accuracy figures obtained after merging. Indeed, this gene was absent in a number of data sets. In one other data set where it was present, its performance as a single-gene signature in risk prediction was consistently high . The influence of this gene could also explain the good performance obtained after merging selectively the sets GSE4335 and GSE1992 (see above). A possible reason that adding the Vijver data set did not improve the prediction accuracy might be due to the fact that this gene was absent in the original Vijver study. As expected, the Cox coefficient of this gene was negative for all data sets in which it was present, meaning low risk associated with high expression levels. The case of CYB5D1 suggests that the absence of key risk or survival genes from some microarray platforms may in general be a major limiting factor in data merging, demonstrated here on the specific example of survival prediction for breast cancer patients.The full name of CYB5D1 is “cytochrome b5 domain containing 1”. It is noteworthy that the gene is absent from the well known prognostic gene signatures published by van de Vijver et al. Jenssen and colleagues The expression of Hs.27475 was also found to be positively correlated (Pearson correlation of 0.58) with the expression of Estrogen Receptor 1 (ESR1) Hs.27475 was noticed to be about two-fold downregulated in two other types of cancer, primary colon and primary rectal carcinoma It may also be worthwhile to mention that CYB5D1 belongs to the same protein family as Progesterone receptor membrane component 1 (PGRMC1), a well known player in the breast cancer field In summary, the findings summarized above indicate that CYB5D1 is an important but currently not well known survival gene in breast cancer with potential diagnostic and therapeutic value.Previous breast cancer studies attempted to improve survival prediction by integrating clinical variables with microarray gene expression data Five patients characteristics and clinicopathological parameters such as age, tumor grade, tumor size, estrogen receptor (ER) status and lymph node status that were available in five data sets were used here . We alsoAs Statistical analysis was performed using R The data sets used in this study were pre-normalized in various ways by the authors of the original studies . The genTime to OS is defined as the time between surgery and death from breast cancer or the last date of follow-up. Time to RFS is defined as the time between surgery and the first recurrence of local, regional or distant-metastatic breast tumor or the last date of follow-up. If OS or RFS time refers to death or recurrence of disease, the corresponding samples have a censoring status of 1 (event happened) or 0 otherwise.We limited the analysis to 10 years of follow-up as the majority of breast cancer patients had a follow-up of maximum 10 years. All patients having an overall survival or relapse free survival greater than 10 years were censored and their respective clinical endpoint was set to 10 years. All patients in GSE4335 deceased from any other cause than breast cancer were also censored. Fibroadenoma or normal breast samples were discarded from the study . Replicate samples in GSE1992 were eliminated from the study, too. Note that throughout this document, GSE1456 refers to the merged data set of GSE1456A and GSE1456B, GSE4922 to the merged data set of GSE4922A and GSE4922B, respectively. GSE4335 and Vijver are data sets from clinical trials.The data sets were pre-normalized in the following ways. Global mean normalization was used for GSE1456A&B and GSE4922A&B. The probe set values were natural log-transformed followed by an adjustment of the mean intensity to a target signal value of log 500. The pre-normalization of Vijver data set was performed on an array-by-array basis. Raw intensities from each channel (red or green) were divided by the mean intensity  of the corresponding channel. The other data sets were pre-normalized as described in the legend to Grade, size, ER and lymph node status are discrete variables . Grade wSamples containing missing values were discarded in the analyses using clinical variables. Note that the five variables, grade, size, ER, lymph node status and age were only available in three data sets  with overall survival endpoint  and in five data sets  with relapse free survival outcome .K Nearest Neighbor (KNN) imputation Z-score normalization  and ComBat Genes were selected based on univariate Cox P-value ranking using the coxph function in survival R package. In this feature selection method, the genes were ranked based on their likelihood ratio P-value and the 100 genes with the smallest P-values were retained as the gene signature if not specified otherwise.The genes were selected either in Cross Validation (CV) mode or in a training set independent from the testing set. In case of CV, the expression values of the genes were fitted individually to survival data, and ranked by P-values in each fold. Then, the top 100-ranked genes were selected for the gene signature. In case of independent training/testing sets, the genes were selected from the training set.Principal Component Analysis (PCA) was applied using the prcomp function in R stats package with default parameters except that the center parameter was set to FALSE.The gene expression data were clustered using complete-linkage agglomerative hierarchical clustering based on the Euclidean distance (heatmap.2 function in R gplots package).Patients risk score was calculated as the linear combination of the Cox coefficients estimated from the training set and the corresponding gene expression values (Equation 1).The survival predictors were assessed by the following methods:10-fold cross validation (10fCV) nested in 10 iterations.Leave one data set out: all data sets except one were merged together to form the training set and the left-out set was used as the testing set. This process was iterated until all data sets were used in the training and testing sets.In 10fCV, the data integration (ComBat or Z-score) was applied separately to the training and testing sets in each cross-validation fold. Note that the training and testing sets were generated by merging and subsequent random splitting. Thus both contained the samples from different sources. Since ComBat needs the source- (batch)-identifiers as input, those identifiers had to be carried through the cross-validation protocol. In case of leave-one-data set-out, Z-score normalization was applied to each data set separately prior to merging.Time-dependent ROC curves The association of the gene signatures to survival (OS or RFS) was also measured by a hazard ratio (HR). To this end, the patients of the testing set had to be stratified into predicted high- and low-risk groups. In both 10fCV and independent validation, a testing sample score was considered as high risk, if it was higher than the median score of the training sample and low risk otherwise.In case of 10fCV, HR was averaged by geometric mean (Equation 2). The 95% CI of the HR over 10 iterations of 10-fold cross validation corresponds to the CI of the geometric mean.The gene-signature-based risk scores were also evaluated in the context of clinical parameters, using the same type of cross-validation protocol. The binary risk scores computed for each testing set sample (low/high-risk group as described above) together with the five clinical parameters listed in The survival prediction accuracy and prognosis of clinical risk were neither increased nor decreased significantly by merging data sets. This is explained at least in part by the fact that important risk-associated genes were not present in all data sets. Consequently, the heterogeneity of the data sets generated from different laboratories and with different microarray technologies, was not the only, perhaps even not the major limiting factor for improving prediction accuracy by increasing sample size. Substantial variation of time to death or relapse among breast cancer patients and the heterogeneity of breast cancer disease are other constraining factors that nevertheless need to be considered. Moreover, the heterogeneity of patients cohorts in terms of age, lymph node status, tumor grade, tumor size and ER status might negatively affect the accuracy of survival prediction after merging. It is known, for example, that the ER+ patients have good prognosis  and ER- negative patients have poor prognosis  in the first five years after the diagnosis or surgery.Despite the caveats mentioned above, the results show that selectively merging those data sets which give rise to accurate predictors if used alone, can improve the performance. Moreover, our results confirm that the predictors based on large merged data sets are more robust, i.e. their worst performance observed in multiple iterations of cross-validation tends to be substantially better compared to the worst performance of the gene signatures based on the single data sets. This may be viewed as an advantage by itself. In general, the prediction accuracy of the gene signatures derived from the merged data sets remained consistent and reproducible across independent studies. Prediction accuracies measured in cross-validation were extensible to independent testing sets.The systematic evaluation of predictors built from the single and merged gene expression data sets also led us to the surprising observation that a single-gene signature consisting of CYB5D1 had the highest prediction accuracy and strongest patients risk association in breast cancer, surpassing all gene signatures with different gene size evaluated in this study. CYB5D1 was already mentioned in breast and other types of cancer studies see  section.Our findings about CYB5D1 call into question the current paradigm that composite gene signatures perform better than one-gene signatures in cancer disease outcome prognosis.Table S1Cross-data set performance of breast cancer predictors trained on the individual and combined data sets (adjusted by ComBat) with respect to OS. Significant HR (p<0.05) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Table S2Cross-data set performance of breast cancer predictors trained on the individual and combined data sets  with respect to OS. Significant HR (p<0.05) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. Merged-zscore refers to the merged data set combined from the individual data sets, each normalized separately by Z-score normalization. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Table S3Cross-data set performance of breast cancer predictors trained on the individual and combined data sets (adjusted by ComBat) with respect to RFS. Significant AUC (>0.60) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Table S4Cross-data set performance of breast cancer predictors trained on the individual and combined data sets  with respect to RFS. Significant AUC (>0.60) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. Merged-zscore refers to the data set merged from the individual data set, each normalized separately by Z-score normalization. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Table S5HR of breast cancer predictors trained on the individual and combined data sets (adjusted by ComBat) with respect to RFS. Significant HR (p<0.05) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Table S6HR of breast cancer predictors trained on the individual and combined data sets  with respect to RFS. Significant HR (p<0.05) are shown in bold. The training sets are listed in the column header and the testing sets are indicated in the row header of the table. Merged-zscore refers to the data set merged from the individual data set, each normalized separately by Z-score normalization. * indicates that the predictor was trained from all data sets except the testing set. NA stands for Not Available.(0.04 MB PDF)Click here for additional data file.Figure S1Prediction performance of the breast cancer gene signatures as a function of the number of genes. Gene nb refers to the number of genes.(1.12 MB TIF)Click here for additional data file.


We investigate two types of normalising beads: broad spectrum andspectrum matched. Lastly, we propose two alternative normalisation procedures that are usable in theabsence of normalising beads.A next step to interpret the findings generated by genome-wide association studies is to associatemolecular quantitative traits with disease-associated alleles. To this end, researchers are linking diseaserisk alleles with gene expression quantitative trait loci (eQTL). However, gene expression at themRNA level is only an intermediate trait and flow cytometry analysis can provide more downstreamand biologically valuable protein level information in multiple cell subsets simultaneously using freshlyobtained samples. Because the throughput of flow cytometry is currently limited, experiments mayneed to span over several weeks or months to obtain a sufficient sample size to demonstrate geneticassociation. Therefore, normalisation methods are needed to control for technical variability and compareflow cytometry data over an extended period of time. We show how the use of normalisingfluorospheres improves the repeatability of a cell surface CD25-APC mean fluorescence intensity phenotypeon CD4 Genome-wide association (GWA) studies have revolutionised the mapping of common genetic variants, mostly single nucleotide polymorphisms (SNPs), with susceptibility to a wide range of common, multifactorial disorders , in partMulticolour flow cytometry analysis can provide rich protein level data simultaneously on different subsets of cells; this is of particular importance for post-GWA investigations as genetic heterogeneity identified in disease-associated regions can differentially affect various cell subsets. However, the throughput of current flow cytometry approaches, including data analysis and sample collection, is limited to a small number of samples per day or week, especially when fresh blood is required. As the identification of subtle molecular effects directed by common genetic variants may require the analysis of a relatively large number of samples, flow cytometry experiments may need to span over several months. Owing to the complexity of flow cytometry technology, various technical artifacts, including variability in reagents or measuring instruments, can create time-related biases. Consequently, normalisation procedures are necessary to enable the comparison of samples analysed at different dates.Similar issues have been identified in the context of gene expression microarray analysis. For these analyses researchers typically take advantage of the large number of independent measurements (one per gene or probe), implicitly using the rank of a gene of interest as a summary statistic. Such techniques are not available for flow cytometry data, and therefore specific approaches are required.α-chain (IL-2RA/CD25) gene region /var(X) where (Xi1)i = 1n and (Xi2)i = 1n designate the first and second sets of replicates (n = 15 in this study).The flow cytometry data were analyzed using FlowJo . The remaining data processing/statistical analysis was performed using the R programing language. Gates were automatically extracted from the FlowJo output using an in-house XML parsing script based on the R XML 2.3.0 library. These gates were applied to the raw FCS files using the R flowCore 1.8.3 library. Repeatability + T cell subsets described in the main manuscript.The supplementary figure shows a representative example of the flow cytometric gating usedto define the CD4Click here for additional data file.


In biometric practice, researchers often apply a large number of different methods in a "trial-and-error" strategy to get as much as possible out of their data and, due to publication pressure or pressure from the consulting customer, present only the most favorable results. This strategy may induce a substantial optimistic bias in prediction error estimation, which is quantitatively assessed in the present manuscript. The focus of our work is on class prediction based on high-dimensional data (e.g. microarray data), since such analyses are particularly exposed to this kind of bias.In our study we consider a total of 124 variants of classifiers (possibly including variable selection or tuning steps) within a cross-validation evaluation scheme. The classifiers are applied to original and modified real microarray data sets, some of which are obtained by randomly permuting the class labels to mimic non-informative predictors while preserving their correlation structure.We assess the minimal misclassification rate over the different variants of classifiers in order to quantify the bias arising when the optimal classifier is selected a posteriori in a data-driven manner. The bias resulting from the parameter tuning  and the bias resulting from the choice of the classification method are examined both separately and jointly.permuted uninformative predictors from studies on colon cancer and prostate cancer, respectively. We conclude that the strategy to present only the optimal result is not acceptable because it yields a substantial bias in error rate estimation, and suggest alternative approaches for properly reporting classification accuracy.The median minimal error rate over the investigated classifiers was as low as 31% and 41% based on  It is well-known that almost all published studies present positive research results, as outlined by Kyzas et al  for the Journal of Negative Research Results in Medicine). Authors are thus virtually urged to "find something significant" in their data, which encourages the publication of wrong research findings due to a variety of technical and statistical pitfalls. Microarray studies are especially subject to such mechanisms and known to yield "noise discovery" scovery" .n large p" dimensionality problem . In univariate analyses for identifying differentially expressed genes, the multiple testing problem resulting from high dimensionality can be addressed, e.g. by means of approaches based on the false discovery rate [Technical challenges that particularly affect microarray studies include, e.g. technical errors in the lab, problems with image analysis and normalization. Statistical pitfalls and biases of studies on microarray-based prediction are equally diverse. A problem well covered in the literature is the "small ery rate ,5. In thery rate -10: if tery rate . Hence, L2 penalized logistic regression). This again is an issue that, in principle, can be encountered in all types of medical studies, but affects microarray studies more drastically. Whereas standard statistical approaches - for instance logistic regression for class prediction problems - have become the methodological "gold standard" in conventional medical statistics and allow a comparatively fair evaluation of research results, the field of microarray data analysis is characterized by the lack of benchmark standard procedures and a huge and heterogeneous amount of methods - ranging from adaptations of standard statistical procedures to computer intensive approaches adopted from machine learning - whose respective merits and pitfalls remain partly unexplored. This is particularly true for studies involving class prediction problems, i.e. when the goal is to derive a classification rule for predicting the class membership  of patients based on their microarray transcriptome data.The reported classification error rate can also be lowered artificially by selecting the values of tuning parameters a posteriori, i.e. on the basis of the computed CV error rates. Doing this, one selects the "best" version of a classifier and evaluates it using the same data, which of course leads to an underestimation of the error rate (named "bias source I" in our present paper). A quantitative study on this topic can be found in . Note thIn this context, if the sample is not large enough to put aside a validation data set, it is common practice to evaluate the performance of classifiers based on techniques like cross-validation (CV) including leave-one-out (LOO) CV as a special case, repeated splitting into learning and test data sets, or bootstrap sampling. See the methods section for more details on the cross-validation technique used here and  for an eIn this article, we empirically investigate the consequences of such an optimization. We report the results of an experiment that allows us to quantify the optimistic bias induced by optimal tuning parameter selection (bias source I) and optimal selection of the classification method (bias source II) in a realistic setting based on original microarray data. After we have illustrated the drastic effect of optimal classifier selection, we discuss alternative ways to report results of class prediction studies when no validation set is available, and give suggestions for good scientific practice in this context.L2-penalized regression) based on cross-validation. Some of these 10 classification algorithms are combined with preliminary variable selection or/and used with different plausible tuning parameter values successively. The aim is to investigate the different sources of biases resulting from optimal selection  and their relative importance. All the considered procedures are classical approaches, most of which have already been used in published medical studies. For the sake of reproducibility, all our analyses are based on the freely available Bioconductor package CMA version 0.8.5 [In our experiment we compute the misclassification rate of a total of 10 usual classification methods  non-modified real data sets, ii) data sets with permuted class labels, to mimic non-informative microarray data with a realistic correlation structure, and iii) subsamples of non-modified real data sets including 60, 65, 70, 75, 80, 85, 90 or 95% of the original sample, to be able to evaluate potential effects of the sample size. To average out the random fluctuations due to the permutation process, we consider Two real-life data sets are considered successively: the well-known benchmark colon data set , and theThis study includes only well-known classifiers that are widely used in the context of class prediction in medical studies and yield acceptable accuracies in "neutral comparison studies"  and the method used to construct the prediction rule, potentially including different tuning parameter settings. More specifically, we use the ten classification methods outlined below:KNN (k-nearest-neighbors): the standard nearest-neighbor approach [k = 1, 3, 5, which are all usual choices when the method is used on small sample data.• approach . We consLDA : standard linear discriminant analysis with normality assumption and common within-group covariance matrix, as summarized in [• rized in ,20.FDA : standard Fisher's discriminant analysis [• analysis .DLDA : the same as LDA, except that the common within-group covariance matrix is assumed to be diagonal. The feature makes it applicable to data with n ≪ p.• PLSLDA: Partial Least Squares (PLS) dimension reduction : the random forest method [ntree = 1000, whereas the number mtry of candidate predictors considered at each split is set either to mtry = • t method  aggregate values .SVM (Support Vector Machines): the well-known kernel method by Vapnik [e1071 package), which are known to perform well. The cost is considered as a tuning parameter and optimized via internal 3-fold-cross-validation.• y Vapnik . Tuning PAM : the nearest shrunken centroids method [• s method . The shrL2: the L2-penalized logistic regression approach . In contrast to Lasso, this approach does not yield sparse models. The penalty parameter is optimized via internal 3-fold-cross-validation.• The methods used to construct prediction rules are summarized in Table One of the two sources of bias investigated in this paper is the optimal selection of tuning parameters including as a special case the number of selected genes and the gene selection method - if preliminary variable selection is performed.Regarding variable selection, classification methods can be divided into several categories:k-nearest-neighbors. Such methods almost always yield poor prediction accuracies when applied to noisy data. For instance, in the nearest neighbors approach, array-to-array distances on which prediction is based are computed using all genes irrespectively of their discrimination power. Such methods should be combined with variable selection.1. Methods that do not take each variable's prediction strength into account, such as n ≪ p, like linear discriminant analysis. For such methods to be applicable, previous variable selection is necessary and the number of selected variables should be smaller than the number of observations.2. Methods that take the variables' prediction strength into account, but can technically not be applied to data with n ≪ p, but usually perform better on a reduced subset of relevant genes, like diagonal linear discriminant analysis (DLDA). By definition, DLDA gives more weight to genes with a high signal-to-noise ratio. However, genes with poor or no discriminating power behave as noise and usually decrease classification accuracy substantially.3. Methods that take the variables' prediction strength into account and can technically be applied to data with n ≪ p and whose prediction accuracy is usually not improved by preliminary variable selection. Such methods include, e.g. nearest shrunken centroids or the Lasso, that perform variable selection intrinsically. Note, however, that preliminary variable selection may be necessary in some cases in practice for computational reasons, especially when the number of genes reaches several tens of thousands as common in modern data sets.4. Methods that take the variables' prediction strength into account, can technically be applied to data with p < 10, 000 genes, which makes preliminary variable selection for the retained methods of type 4 unnecessary from a computational point of view. For methods of types 1 to 3, previous variable selection is applied with different (round) numbers of genes p*: 10 and 20 genes when the method requires n > p* and 20, 50, 100, 200, or 500 genes when the method can cope with n <p*, see Table In the present study, we consider only smaller data sets with L2 penalized regression, and the cost parameter in linear SVM. In our study, these method parameters are tuned by performing an internal 3-fold-cross-validation using the learning set only, because i) it is a commonly recommended strategy [Apart from preliminary variable selection, some of the considered classification methods involve tuning parameters, for instance the shrinkage parameter of the PAM method, the penalty in strategy , and ii)k of neighbors as parameter), random forests (with the number mtry of candidate splitting variables considered at each split as parameter), and PLSLDA (with the number ncomp of PLS components as parameter). For these three methods, we consider a few standard values of the tuning parameter successively and investigate the bias resulting from optimal selection, see Table We also consider classification methods involving tuning parameters for which default values are expected to work reasonably well with most data sets. In this case, parameter tuning through inner cross-validation is not as essential and well-established as for methods like penalized regression. In our study, these methods include KNN , and different tuning parameter settings. The number 124 is obtained as the sum for the 10 considered methods ofOn the whole, we obtain 124 variants of classifiers from the combination of 10 classification methods, some in combination with different numbers G is the number of values of p* , three is the number of considered gene selection methods , and T is the number of considered values for the fixed tuning parameters . This formula, yielding a total of 124, is given explicitly in Table where k-fold cross-validation. In k-fold cross-validation (for instance k = 5), the available data set is split in k approximately equally sized subsets. At each of the k iterations, one of these subsets is considered as a test data set, while the union of the k - 1 other subsets forms the learning set. The cross-validated error rate is then computed by averaging the error rates obtained in all k iterations. The reader is referred to [k of iterations is set to the standard value k = 5.It is well-known that the prediction accuracy of a classifier should not be evaluated based on the data that were used for its construction. Instead, if no independent validation set is available, the classifier should be evaluated through a CV-like technique considering several pairs of learning and test sets successively. In the present experiment, we use one of the most widely used evaluation schemes: erred to ,32 for mhttp://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/errorratebias.In this study, we use the Bioconductor package 'CMA'  describep = 2000 genes for 22 normal and 40 tumor tissues (n = 62) that is available from the Bioconductor package 'colonCA'. This data set was analyzed in numerous classification-based articles including comparison studies [We first analyze the well-known colon cancer microarray data set  includin studies . The obt> 35%, see Table L2) or with variable selection based on the t-statistic . The results with the Wilcoxon statistic and the limma statistic are similar. As can be seen from Table We apply the 124 classifiers to this data set and obtain error rates ranging from 11% to The second data set investigated in the present study is introduced by Singh et al  and inclNote that the relative performance of the different methods is substantially different in the two data sets. This is consistent with the widely acknowledged fact that there is no unique "best" classification method yielding top-performance for all data sets. Hence, it would be unwise to try only one classification method: the multiplicity issue investigated in this study is relevant.In this part of the analysis, we simulate non-informative gene expression data sets by permuting the class labels of the two data sets described above, thus mimicking non-informative microarray data with a realistic correlation structure. To average out variations due to the random permutation, 20 balanced permutations are considered successively. The two sources of bias  are assessed separately in the two next subsections.p*, the gene selection procedure and/or the tuning parameters based on the CV-results. For each of the 10 classification methods and each of the 20 permuted data sets, we consider the six following error rates:The aim of this part of the analysis is to assess the bias induced by optimally selecting the number of genes Approach A: Minimal error rate over the different tuning parameter values .Approach C: Minimal error rate over the different tuning parameter values .Approach D: Minimal error rate over the different tuning parameter values .k = 1, 3, 5), all numbers of genes  and the three gene selection methods  is 0.43. It equals 0.45 if one optimizes over the results obtained with k = 5 only (Approach B) and if one optimizes over the results obtained with p* = 100 only (Approach C). Optimization over the results obtained with the t-statistic as fixed gene selection method (Approach D) yields a median error rate of 0.47, while the optimization over the three tuning parameter values yields the higher value 0.50. Most importantly, if all 10 classification methods are considered simultaneously, the bias due to the optimization of the classification method dominates all the other sources of bias, as investigated in more details in the next section.These minimal error rates are computed for all 20 permutation runs and for each of the 10 classification methods. For each classification method and each approach, the median over the 20 runs is given in Table k = 5 and p* = 100 genes selected based on the t-statistic, LDA with p* = 20 genes selected based on the t-statistic, FDA with p* = 20 genes selected based on the t-statistic, DLDA with p* = 100 genes selected based on the t-statistic, PLSLDA with ncomp = 2 and p* = 100 genes selected based on the t-statistic, NNET with p* = 100 genes selected based on the t-statistic, RF with mtry = L2 with cross-validated penalty. Note that p* = 20 genes were used for the methods of type 2 that require n > p.In this subsection, we consider the second source of bias, namely the optimal choice of the classification method. Here the tuning parameters are either optimized through a correct inner cross-validation procedure or fixed to a single plausible value in order to handle all 10 classifiers equally. The minimal error rate is derived over the resulting 10 error rates. The 10 considered combinations are as follows: KNN with H0) in For each of the 20 permutation runs, the minimal error rate p* = 100). The rationale behind this is that it makes sense to consider the gene selection procedure as a component of the classification method rather than as a tuning parameter. This minimal error rate is denoted as Moreover, we also derive the minimal error rate over the 6 × 3 + 4 = 22 classifiers yielded as combinations of a classification method and a gene selection method , and iii) the error rate min/N which would be obtained by assigning all the observations to the majoritary class (dashed line).Additionally, Figure As expected, Figure Note that the bias compared to the baseline 50% is higher for the colon data set than for the prostate data set, which can be at least partly explained by the difference of the sample sizes and the ratio of class frequencies . More precisely, under the null-hypothesis of non-informative gene expression data, the median minimal error rate is 31% for the colon data set, and 41% for the prostate data set.n. For each proportion of the original sample size, the procedure is repeated 20 times to average out variability due to random subsampling. The minimal error rate over the 124 classifiers is derived for each subsample.As an outlook, the bias over the 124 classifiers is also assessed based on subsamples drawn randomly from the original data set. We successively consider subsamples corresponding to 60, 65, 70, 75, 80, 85, 90 or 95% of the sample size Figure cer data  and the cer data . It can Errmin124, because this strategy generates a considerable optimistic bias. In practice, the bias due to the optimal selection of the tuning parameters ("bias source I") is often addressed by nested cross-validation [From the results presented in the previous section, it is clear that one should definitely not report only the best result lidation .n ≥ 150, a condition which is not fulfilled by all microarray studies. When a validation data set is available, the error rate estimates of the cross-validation procedure on the training set are of secondary importance. Cross-validation is only used for selecting the best classifier, i.e. for comparison purposes but not for estimating the error rate. The estimate of the error rate obtained from the validation data set is then unbiased, since this data set was not opened during the training phase. Note that the more classifiers are tried during the training phase, the more the validation accuracy will decrease compared to the best training accuracy, due to multiple comparisons effects.One should test the derived classifier on untouched validation data whenever possible. This approach consists of splitting the available data into i) a training set which is used to construct classifiers and select the "best" one and ii) a validation data, which is not opened until a final classifier has been selected based on the training data, in the vein of the validation strategy described by Daumer et al . In pracNature Precedings is to be encouraged [If the study does not include any validation step with untouched data, one should not only report the results obtained with the best performing method/parameter combination/number of genes, as demonstrated in our study on original and modified real microarray data. In this situation, handling and reporting the results in a fair way is a complex task. Ideally, the considered classification methods should be documented in form of an analysis plan clinical studies) before starting the data analysis. In this context, the publication of analysis plans on open access platform like couraged . Howevercouraged  and 'CMAcouraged ), the stErrmin with the median minimal error rate obtained from data sets with permuted class labels H0) to 11%." This reporting scheme has two strong advantages: i) it automatically adjusts for the number of candidate classifiers and ii) it accounts for the fact that the error rate also depends on the class proportions.Alternatively, we suggest an approach inspired from our permutation-based simulation design. Since it does not make much sense to focus on bad performing classifiers, we suggest to consider the minimal error rate over all tried classifiers as the main outcome of the study. However, to avoid hasty optimistic conclusions, we propose to compare this minimal error rate from the original data In this article, we have quantitatively assessed the bias induced by reporting only the error estimate of an optimally selected classifier. The focus here was on the classifier construction step, which is one of the most complex from a statistical point of view. However, the optimal choice of a method might take place at other levels of microarray preprocessing and analysis, too. Firstly, the normalization technique  can also be optimized, since there exist a number of normalization approaches which can be tested easily, for instance using standardized R packages from the Bioconductor platform. The choice of the normalization procedure may greatly affect the classification accuracy of prediction rules, perhaps even more than the choice of the classifier. However, each team probably has its preferred procedure and applies it systematically without switching to another one, even if the results are disappointing. Hence, the choice of the normalization technique is probably not an important source of optimal selection bias in the sense considered here.Secondly, one could optimize the criterion used to filter out genes that are, e.g. not enough regulated. Several variants of selection criteria can be used, such as the fold change or p-value criteria with various thresholds. For instance, if the results are disappointing using the genes with fold change > 2 in at least five arrays, the statistician might decide to be more stringent and select only genes with fold change > 3. This problem may yield an additional bias. On the whole, the bias observed in this study can on one hand be considered as overestimated , but on the other hand as underestimated .Note that an unbiased error estimate for the optimal classifier can be obtained based on validation data, if such a data set is available. Prediction tools should always be thoroughly evaluated based on fresh external validation data before their application in clinical settings ,38. ValiAs a conclusion, let us mention that the bias outlined in the present article does not only affect biomedical articles, but also potentially methodological articles, where tuning parameters or method features are often optimized based on the data sets that are subsequently used for evaluation and comparison .Developing a new prediction method and evaluating it by means of comparing its performance to that of competing methods using the same data set can lead to over-optimistic conclusions in the sense that the new method's characteristics overfit the considered example data sets , followiThe authors declare that they have no competing interests.ALB identified the problem, performed the analyses and drafted the manuscript. CS contributed to the concept and to the manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/9/85/prepub


The relative abundance of miRNAs is linked to function We compared the performance characteristics of four commercial miRNA array technologies and found that all platforms performed well in separate measures of performance.The Ambion and Agilent platforms were more accurate, whereas the Illumina and Exiqon platforms were more specific. Furthermore, the data analysis approach had a large impact on the performance, predominantly by improving precision. MicroRNAs (miRNAs) are endogenous, non-coding transcripts that regulate a diverse range of functions, including development, differentiation, growth, apoptosis and metabolism. These 17-24 nucleotide RNA molecules confer specific recognition of target mRNAs and modulate gene expression by acting in conjunction with a set of effector proteins of the RNA interference pathway , or by aThe relative abundance of miRNAs in cells is thought to be important for miRNAs to exert their regulatory function. For example, titrated expression of both genomic copies of mouse miR-1 is required for normal heart formation and function during embryogenesis . Aberran® Universal-16 BeadChip arrays were used for this study instead of the Sentrix® Array Matrix, which is the manufacturers supported platform for miRNA analysis of the version 1 bead pool.Microarray technology permits simultaneous expression measurements for hundreds of miRNAs. This technology is already widely used and promises to become a standard tool in the near future. However, a careful assessment of the technology has not yet been performed. This motivated us to evaluate performance attributes of four commercial array platforms for miRNA expression profiling. The miRNA platforms evaluated were Ambion , Agilent , Exiqon , and Illumina . In all cases the sample processing was performed by experienced operators working under standard operating procedures. Samples for three of the four platforms were processed by companies that provide research services on the platform. The study was administered by BIOO Scientific Corporation  to ensure that the sample identities and purpose of the experiment was blinded. With the exception of the Illumina platform, the laboratory personnel did not know the experiment was part of a performance evaluation. Illumina's SentrixSeven synthetic miRNAs [Additional file 2 scale) as expression values. We found that no-background correction and quantile normalization clearly outperformed other approaches, so we used these methods to compare platform performance. For Agilent we used the default dataset according to the vendor's recommendations. Figures using the default dataset for all platforms are included as Additional files Each company provided processed data as part of the standard service using statistical methods produced in house. We refer to these as the default data sets. They are available for download through the NCBI Gene Expression Omnibus (GEO) repository under the following accessions: GSE19248. The Exiqon default data reported the value "NA"  for 51% of the measurements associated with the spiked-in miRNAs, and 59.1% of the 330. We were, therefore, unable to analyze the Exiqon default data by the methods described, and it was not included in this report. In gene expression microarrays various academic groups have demonstrated that the use of alternative statistical methodology can substantially improve accuracy and precision of expression measurements, relative to ad-hoc procedures developed by the manufacturers of the technology . We therWe assessed specificity and sensitivity in a way that can be easily related to practical performance. The use of the same placental total RNA as background material in each hybridization permitted us to assess specificity. Spike-in experiments have been used extensively to assess gene expression technologies as they provide a sensible way of measuring sensitivity ,14. Howesignal detection slope [2) scale, the slope from the regression of expression on nominal concentration can be interpreted as the expected observed difference when the true difference is a fold change of 2. Thus, an optimal result is a slope of one and values higher and lower than one are associated with over and under estimation, respectively. Figure We adapted statistical assessments that have been successfully implemented for gene expression arrays . We staron slope . MicroarNull set. The standard deviation (SD) of these log-ratios serves as a basic assessment of precision and has a useful interpretation: it is the expected range of observed log-ratios for genes that are not differentially expressed. In gene expression arrays, specificity performance has been shown to vary with nominal concentrations [Specificity is another important feature of array-based platform performance. Because the majority of microarray studies rely on relative measures (e.g. fold change) as opposed to absolute ones, we focused on the precision of the basic unit of relative expression: log-ratios. We adapted the precision assessment of Cope et al.  that foctrations . We thertrations  demonstrth percentile of the null distribution as a second summary assessment of specificity. Note that for this analysis 3.3 is the expected 1% value for the 330 human mature miRNAs common to all platforms. However, the number of array features will certainly increase in the near future: the number of false positives (in the top 1%) will increase proportionally.Note that in that in Figure 2 fold change, Δ = 1, will appear in a list of the top 1% (highest log-ratios). This summary statistic, as well as the accuracy and precision summaries described above are shown in Table Precision and accuracy assessments, considered independently, have limited practical use. However, the summary statistics described above can be easily combined to answer many practical questions when posed in a statistical context. As an example, we computed the chance that, when comparing two samples, a gene with true logWe have described an assessment procedure for microRNA microarray data based on a carefully designed spike-in experiments. Strengths and weaknesses were revealed for each platform. Ambion and Agilent were more accurate, while, Illumina and Exiqon were more specific. Strikingly, the data processing methods had a more profound impact on the performance than were observed for differences between platforms. The introduction of background correction adjustment to the raw data was detrimental to specificity, inferring that background correction was the likely cause of lower performance for the three default data sets. The practical implication is that false positive fold changes are most likely to be detected at lower expression signals from default data, and may be reduced by eliminating the background correction from the raw data.We considered quantile normalization to be the best approach among multiple options for this study design because the distribution of the background RNA is identical across the project. For projects where the miRNA fraction of total RNA may be variable across different samples in the project, another normalization method may be more appropriate.The experimental design did not include measurements of day-to-day or site-to-site variability to evaluate platform robustness, so we were not able to draw direct conclusions as to whether these platforms might have performed differently under different circumstances. Reproducibility testing of the Agilent, Ambion and Illumina platforms beyond the scope of this study suggested that the performance reported here is within the expected day-to-day variability (not shown).Both Ambion and Agilent demonstrated good accuracy across the range tested but with less precision than the other two platforms. Agilent performed the best when only the default data set was evaluated for each platform. Considering that we adhered to Agilent's guidance to use the default data, further analysis is required to determine whether excluding the background adjustment or including a global normalization method can improve the performance of the Agilent array.The authors SS and DE are employees of Asuragen. Asuragen provides a commercial service on the DiscovArray™ and Agilent platforms and funded the experiments described in this study. The authors MM, MW and RI declare that they have no competing financial interests.MW and RI designed the research. SS and DE carried out the experiment MM carried out the data analysis. MM, MW, and RI wrote the manuscript. All authors read and approved the final manuscript.Supplementary Table S1. Spike-in sequenceClick here for fileSupplementary Figure S1 - Empirical densities: These plots depict the empirical density of the average (across arrays) expression values for the background RNA, including quantile normalized raw data (A) and default data (B). The tick marks on the x-axis show the average expression at each nominal spike concentration.Click here for fileSupplementary Figure S2. As Figure Click here for fileSupplementary Figure S3. As Figure Click here for file


Gene expression profiling has the potential to unravel molecular mechanisms behind gene regulation and identify gene targets for therapeutic interventions. As microarray technology matures, the number of microarray studies has increased, resulting in many different datasets available for any given disease. The increase in sensitivity and reliability of measurements of gene expression changes can be improved through a systematic integration of different microarray datasets that address the same or similar biological questions.Traditional effect size models can not be used to integrate array data that directly compare treatment to control samples expressed as log ratios of gene expressions. Here we extend the traditional effect size model to integrate as many array datasets as possible. The extended effect size model (MAID) can integrate any array datatype generated with either single or two channel arrays using either direct or indirect designs across different laboratories and platforms. The model uses two standardized indices, the standard effect size score for experiments with two groups of data, and a new standardized index that measures the difference in gene expression between treatment and control groups for one sample data with replicate arrays. The statistical significance of treatment effect across studies for each gene is determined by appropriate permutation methods depending on the type of data integrated. We apply our method to three different expression datasets from two different laboratories generated using three different array platforms and two different experimental designs. Our results indicate that the proposed integration model produces an increase in statistical power for identifying differentially expressed genes when integrating data across experiments and when compared to other integration models. We also show that genes found to be significant using our data integration method are of direct biological relevance to the three experiments integrated.High-throughput genomics data provide a rich and complex source of information that could play a key role in deciphering intricate molecular networks behind disease. Here we propose an extension of the traditional effect size model to allow the integration of as many array experiments as possible with the aim of increasing the statistical power for identifying differentially expressed genes. Microarray technology is becoming an important tool for biological research and clinical diagnostics , but it A two channel microarray technology continues to be used as one of the most common platforms for gene expression profiling ,19. One The model proposed in our study is more general than the model proposed in , and allIn order to assess the usefulness of our model to integrate real data we applied our method to three different expression datasets generated from two different laboratories using three different 2-channel array platforms and two different experimental designs. All three datasets compared normal liver tissue to liver tissue chronically infected with hepatitis C virus (HCV).Before data integration was carried out, an exploratory data analysis as proposed in  was condBy adopting a similar approach to Kim et al.  we preseIn order to test for homogeneity between datasets, we used the Cochran Q statistics given in eq.10 (see the Methods section). The results of the test are shown in Figure duced in . Figure In order to further assess the advantage of data integration, we decided to examine whether genes found in our analysis had direct biological relevance. Genes that are determined to have statistically significant expression level changes may still have low fold increases (or decreases) that might not be biologically relevant. Although there is no consensus in fold increase/decrease associated with 'biological relevance', we chose a fold change of at least 1.5 (increased or decreased) between HCV and normal in at least one of the three integrated studies based on the estimate of the median standard deviation (median sd = +/- 0.23) of fold gene expression measurements in the three experiments (a 1.5 fold cuto3 on gene expression levels is 2 standard deviations away from the mean of genes with no expression (i.e fold = 1), and thus is less likely to be confounded with non-expressed genes). We found a total of 206 genes to satisfy those criteria. Of the 206 genes, 79 genes were integration-driven discovery (IDD) genes as defined in . We have<0.05 for significant overrepresentation we found a number of enriched GO biological processes to be associated with HCV infection including: immune response, defense response and response to virus. Many of the genes in each of the enriched GO categories , in this way the comparison between results obtained with MAID and results obtained with other models is ensured to be done at the same statistical significance level (see Tables E(F P) of 10  = 10, see also Figure E(F P) ≤ 21. The number of significant genes selected by each model is given in Table In Figure E(F P) = 10) enriches relevant biological categories we compared the enriched GO biological process (BP) categories obtained from gene lists selected by each individual model. We also imposed a threshold on selected genes' fold changes by requiring genes to be up (or down) regulated by |fold| ≥ 1.5 (for the reasons noted earlier) in HCV samples when compared to Normals  categories obtained with gene sets selected by the model proposed by Rhodes et al.  and MAIDAs shown in Figure In this study we introduce a new effect size based model for microarray data integration. We demonstrate that our model, together with appropriate data pre-processing methods, can be used to integrate expression data across different laboratories, array platforms and experimental designs that results in an increase in statistical power for identifying differentially expressed genes when integrating data across experiments. Moreover, we show that genes selected as significant by our model enrich relevant biological pathways and processes.In order to obtain the best possible results with our model, a number of important problems relating to each individual data set had to be addressed. First, it is only reasonable to integrate experiments that aim to address the same or similar biological questions. In order to address the problem of matching of samples and experiments, we integrated only experiments that compared samples of same biological type. Second, because most of the disagreement between individual array experiments was found to be due to platform-dependent probe effects , we deciAfter addressing the problem of matching of probes, samples and experimental conditions we used exploratory analysis methods proposed in  to deterThe overlap in results of the top genes selected by each method  indicates that models based on integrating p-values  and effeModels based on summary statistics that integrate p values  or expreτ2 will be biased and would influence overall results [Though we agree in principle with the approach proposed in Choi et al. , we note results ,28.The approach proposed in our study differs from that of  and the t-statistic (see eq.6 in Methods section) which follows a Student's t-distribution times n - 1) degrees of freedom (where n is the number of microarray replicates).In MAID the type of analysis applied depends on the type of data analyzed. Thus for microarray experiments with two groups of data we use the standard effect size model proposed in . For micIn addition to eq.6 (see Methods section) we also propose new estimators for both the pooled standard deviation  and the estimated variance .Although we adapt the same general hierarchical model framework as described in Choi et al. , a majorThe same changes occur in eqs.9–15 with new estimators replacing those described in Choi et al [Traditional effect size models  are limiFinally we have shown that most of the genes found to improve in significance after data integration with our model are of direct biological relevance to the three experiments. High-throughput proteomics and genomics data provide a rich and complex source of information which may help to decipher the complex molecular networks behind disease. Beyond the analysis of the gene expression data presented in this study, our model provides a way of integrating multiple microarray datasets across a broad range of cross-platform studies, and allows a more general and flexible framework for microarray data integration.Three microarray expression datasets from two laboratories were collected. Two datasets were obtained from the University of Washington. These datasets were collected using two different versions of Agilent array technology. One dataset was generated using two-channel Agilent Human 1 cDNA array platform containing 12,814 probes. This study used a direct design and included 13 chronic HCV samples co-hybridized with 13 normal samples. The second dataset was generated using two-channel Agilent Human 1A (V2) 60-mer oligonucleotide array platform containing 20,173 probes. This study used direct design and included 5 chronic HCV samples co-hybridized with 5 normal samples. The third dataset was obtained from the University of Toronto UHN Microarray Liver Disease Project ,31. ThisIn order to assure the best possible match for features across different microarray platforms, we used mappings that matched each feature to genomic information available from the NCBI Entrez gene database. We used the R package AnnBuilder  to matchWe used the R package MergeMaid  to integS as shown in equation belowThe integration model proposed by Rhodes et al.  was implpj is the gene specific p-value for the jth study. The summary S statistics is then compared to an empirical distribution, obtained by computing summary statistics S from 100000 random permutations of p-values from each study. The meta-analysis p-value are computed as the proportion of random S statistics larger than the actual S statistics. To estimate the false discovery rate we used the R package qvalue [λ parameter set to zero that produces an estimate of FDR according to the methodology proposed by Benjamini and Hochberg [where e qvalue ,33 with Hochberg .i) reverse-transcribed into cDNA (ii) labeled with different fluorescent dyes , (iii) mixed in equal proportions and hybridized to DNA probes on the glass slide. In the case of indirect design a common reference is used on all the arrays. The basis for the model used in our analysis was first proposed by Choi et al. [Microarray experiments are done using either two-channel  or single channel arrays (e.g Affymetrix). For two-channel arrays the experimenter can chose between either direct or indirect designs. There are several distinct steps involved in producing two-color microarrays. In a direct design for two color cDNA microarrays, treatment and control target mRNA samples are  for the unbiased effect size is given in [The estimated variance given in  asFor studies that measure the mean difference between treatment and control groups using a direct design approach we introduce a new standardized index given in eq.6tpaired is the the expression for the paired t-statistic given in eq.7where n is the sample size and tpaired is the Student t quantile with (n - 1) degrees of freedom.where tpaired follows a t distribution with (n - 1) degrees of freedom, d" is distributed as t-variate with (n - 1) degrees of freedom. Thus the mean and the variance of the standardized index d" can directly be obtained from the mean and the variance of the t-distribution as shown in eq.8Because ν designate the number of degrees of freedom (ν = n - 1). The null hypothesis H0 tested by tpaired is thus that of no differences between treatments and control for one sample data (i.e H0:μ = 0). We note that for studies with direct design nt and nc denote the number of co-hybridized treatment and control samples for each one of the Cy5 and Cy3 channels with nt = nc = n, where n designate the total number of array replicates.where t-statistic is calculated using either a two sample Welch t-statistic for the two class label case or a paired t-statistic for a single class label case. In both cases the t-statistics is calculated using the mt.teststat.num.denum function from the R package multtest [In our implementation the correct specification of the class labels depends on the type of data analyzed. Thus on the basis of class labels specified, our algorithm identifies the two types of data automatically. Experiments using two channel arrays with direct design correspond to the one-class label case while experiments with two groups of data correspond to the two class label case. Depending on the data type given, a multtest .The hierarchical effect size model proposed by Choi et al.  is givenyj is the observed effect size in study j, θj is the mean gene expression in study j, μ is the average measure of differential expression for each gene across datasets, τ2 is the estimated between study variability and sj is the estimated within-study variance.where j for the two group data case and j for the one group data case. In our implementation yj from eq.9, designate either sj is calculated using either the expression given in eq.5 for the two group data case or the expression given in eq.8 for the one group data case. For the rest of this section, depending on data type to be integrated, yj and sj will designate either the observed effect size given eq.4 and its variance given in eq.5 or the observed standardized score given in eq.6 and its variance given in eq.8.Let τ2 = 0 in eq.9, then a fixed effect model (FEM) is used, otherwise a random effect model (REM) is used. To asses whether FEM or REM is most appropriate the hypothesis H0 : τ2 = 0 is tested using the Cochran Q statistics given in eq.10Following Choi et al.  if τ2 = wj = where τ2 = 0, the statistics Q follows a l designates the total number of experiments.Under the null hypothesis that τ2 = 0 is rejected, then the between study variability is estimated using eq.12If the null hypothesis of In the case of random effect model (REM) where the z score statistic under REM can then be calculated as shown in eq.15t-statistic while for the two class label case the Welch t-statistic is used.In order to estimate the statistical significance of integrated results a permutation-based method developed by Tusher et al.  was usedj = 1, 2,...l containing k = 1, 2,..., p genes, for each permutation b = 1, 2,..B, randomized data zk scores were then ordered (z1 ≤ ... ≤ zp) together with random Within each dataset zth is the threshold on the z score statistic [where tatistic   BP categories obtained with the MAID model (see also Table Click here for file


Gene expression microarray technologies are widely used across most areas of biological and medical research. Comparing and integrating microarray data from different experiments would be very useful, but is currently very challenging due to the experimental and hybridization conditions, as well as data preprocessing and normalization methods. Furthermore, even in the case of the widely-used, industry-standard Affymetrix oligonucleotide microarrays, the various array generations have different probe sets representing different genes, hindering the data integration.In this study our objective is to find systematic approaches to normalize the data emerging from different Affymetrix array generations and from different laboratories. We compare and assess the accuracy of five normalization methods for Affymetrix gene expression data using 6,926 Affymetrix experiments from five array generations. The methods that we compare include 1) standardization, 2) housekeeping gene based normalization, 3) equalized quantile normalization, 4) Weibull distribution based normalization and 5) array generation based gene centering. Our results indicate that the best results are achieved when the data is normalized first within a sample and then between-samples with Array Generation based gene Centering (AGC) normalization.We conclude that with the AGC method integrating different Affymetrix datasets results in values that are significantly more comparable across the array generations than in the cases where no array generation based normalization is used. The AGC method was found to be the best method for normalizing the data from several different array generations, and achieve comparable gene values across thousands of samples. Microarray experiments have become an indispensable part of modern biological and biomedical research. As the number of studies using microarrays is growing all the time, it becomes increasingly important to compare and integrate data from multiple experiments and thereby improve the ability to make meaningful biological conclusions. Collections of microarray data from thousands of samples are emerging, but proper normalization methods are to a large extent lacking. To make optimal use of these datasets, improved methods for normalizing data from different studies in different laboratories are urgently required.There are studies where gene expression data from different studies are systematically combined together. For example, computational models for defining modules in the transcriptional data -3 have bHere, our main objective was to test several known normalization methods for integrating gene expression values across thousands of experiments to be able to select a suitable method when combining datasets across Affymetrix array generations and experiment series. Even though the methods presented in this study are shown to work with the Affymetrix gene expression microarrays, they should be applicable also for integration experiments of other microarray platforms.We compared and assessed the accuracy of five normalization methods for Affymetrix gene expression data using 6,926 Affymetrix experiments from five array generations. The methods that we compared include 1) standardization (Z), 2) housekeeping gene based normalization (HK), 3) equalized quantile normalization (Q), 4) Weibull distribution based normalization (WBL) and 5) array generation based gene centering (AGC). These were tested in the following ten combinations: Pure preprocessed data (MAS) without any further normalization, Z-, HK-, Q-, WBL-normalizations, and all of these normalization methods combined with the AGC method: MASAGC, ZAGC, HKAGC, QAGC and WBLAGC. The MAS, Z, HK, Q and WBL methods normalize the data within the samples, while the AGC method normalizes the data gene-wise between the samples.Goodness of normalization can be measured in many ways. Here, we applied five different ways to estimate the degree of comparability between data from different array generations, including: 1) correlation between technical replicates, 2) correlation between randomly selected genes, 3) classification of the samples based on the anatomical classes, 4) comparison of correlations between the samples computed based on the anatomical classes and array generations, 5) stability of the house-keeping genes.The data collection used in this study contained samples from Affymetrix array generations Hu6800 (HuGeneFL), HG-U95A, HG-U95Av2, HG-U133A, HG-U133 Plus 2. These array generations were selected as there were more than 500 samples hybridized on each of them in the database by the time of the comparison. At least half of the genes were in common between the array generations.α = 0.01. The WBLAGC gave the best results from the AGC methods, but the difference with the other AGC normalization methods was not significant ) = 0 for genes k1, k2 where k1 ≠ k2. The different array generations are known to induce some biases to the gene values that may further cause systematic errors in the data. These kinds of systematic array-wide variations may increase the correlation between randomly selected genes.Another method for comparing the goodness of the normalization methods is the comparison between the correlations of randomly selected genes . Since iα = 0.01 the ZAGC, HKAGC, QAGC and WBLAGC had smaller mean values than the other methods. AGC correction was again found more robust than the other normalization methods, as these AGC-correction methods did not significantly differ from one another and were closer to zero  Correlations from healthy samples from the same array generation but from different anatomical class, and 2) Correlations of healthy samples from the same anatomical class done with different array generations.When AGC was not used, the array generation was superior to biological origin in defining the identity of the sample. In such cases the correlation between samples from the same array generations was significantly higher than the correlations between samples from the same anatomical class. When the data were AGC normalized, the correlations from the same anatomical origin were significantly higher than the correlations from different anatomical classes within the same array generation. The significance was tested with one-way ANOVA and multiple comparisons performed with Tukey's HSD with α < 0.01. As evidenced by the significance analysis, the AGC-normalization method reduced noise due to different array generations Figure .If the data are normalized properly, the housekeeping gene values should be stable between experiments. This is based on the assumption that the housekeeping genes are expressed similarly in all samples across the array generations and tissues. However, it is known that the array generation can impact also on values for housekeeping genes, and the expression values of housekeeping genes in our material also seemed to differ based on the array generation.We investigated the effects of different normalization methods on similarity of distributions of housekeeping genes from different array generations. The housekeeping genes under consideration were the same ones than used in the HK-normalization. The similarities were quantified with the Kullback-Leibler measure. We assumed that the values for each housekeeping gene from one array generation should be distributed similarly with the distribution of the gene across all the array generations. We divided the range of the gene value into 50 bins so that within each bin there are 2% of the gene values of the gene:f is the empirical cumulative density function of the gene. We define D to be a set of all expression values measuring the expression of gene k and array generations to A1,..., Ap. Now, for every array generation i, Di ⊆ D is a set of all expression values of the housekeeping gene that are measured with the array generation Ai. The probability density function of distribution Q for each array generation i is constant:where D) is the number of values in the data set D. We assume that the data values within each percentile group are distributed along this constant distribution. We compute the discrete distribution of the gene values from all array generations within each of these percentiles of the data:where #  can be mWe have employed five different criteria to measure goodness of the normalizations. The results showed that the AGC method improved the results systematically and that the AGC normalized data became comparable across the array generations, as suggested by the classification accuracy of different anatomical samples, and the improved correlation of the data from the same samples analyzed on two different array generations. The AGC method combined with the Q normalization is used in for almost 10,000 samples in GeneSapiens database ,17.The gene expression data from 6,926 samples were analyzed together in order to find computationally effective and well-performing method to normalize a large number of the data samples to be directly comparable with each other. All the samples were measured with Affymetrix microarrays, but the various array generations hinder the comparability. Ten different combinations of five normalization method were utilized. The array generation based centering of gene values was found to perform the best, especially if utilized together with the equalized quantile normalization or WBL-normalization.The data set of 6,926 Affymetrix arrays includes several different array generations with different probe sets. Further, the probe set values need to be converted to gene values. We took median of the normalized values from different probe sets that linked to the same ENSEMBL gene identifier  in orderAs different preprocessing methods often complicate the data integration, we used data from which the raw data (CEL-files) were available. For all these experiments we used MAS5 preprocessing method with default parameters .The selection of preprocessing method for Affymetrix gene expression data is a controversial topic, and although different opinions exist for optimal preprocessing method  in recenGene value standardization is widely used method for normalizing the gene expression values. In standardization the logarithmic signal values of genes are normalized to have zero as mean or median and one as standard deviation:x consists of the logarithmic values of a sample, μ is the mean or median and σ the standard deviation of the sample. The standardized values are often called as Z-scores or median Z-scores. Here, Z refers to median Z-scores.where the vector The housekeeping gene centering (HK-centering) scales the data using a scaling factor that is defined based on the housekeeping genes common with most popular array generations. The assumption behind the HK-centering is that the set of housekeeping genes is expressed identically across the samples. This assumption is found to be unrealistic in several settings . Howeverxj, HK are the values of housekeeping genes in sample j. The gene values of each sample can be calculated withwhere x are the expression values of the sample.where In several cases it is desirable to scale the samples so that the minimum and the maximum values are the same order of magnitude. Further, often down-stream analysis methods assume that at least standard deviations or means of the values are equal. Therefore, we have utilized equalized quantile normalization (Q) algorithm to normalize the data .In the basic quantile normalization all samples are normalized to have the same distribution -28. ThisWe utilized the equalized quantile normalization (Q) that constructs a data set having the desired distribution that has been determined prior to transformation. In the Q-normalization the assumptions are the same as in quantile normalization; the sorted order of the data values should not be changed by the normalization method and the logarithmic signal values can be presented with a predefined distribution. This distribution is the same for every sample through all array generations.In this study, we evaluated the distribution of the logarithmic MAS5 probe set values for every Affymetrix array generation and found that we can approximate the signals by equalizing the signal-log-values to normal distribution. The distribution of logarithmic values from all samples  was very near to the normal distribution with mean value 8 and standard deviation 2 , a way to normalize and correct Affymetrix microarray probe set data . In the βi and the scale parameter ηi for each array generation i. The scale parameter varied from 8.19 to 8.57, while the shape parameter varied from 3.11 to 3.85. The ML-estimates were calculated for each sample and probe set values were adjusted in order to have same parameter estimates for each of the samples within an array generation. For the sample j this normalization can be done with the formula:For each array generation we collected the array-generation data that are a group of samples analyzed using the MAS5 with default parameters ,29. Thesηi and βi are the ML-estimates for scale and shape parameters in the array generation i, and ηj and βj are the ML-estimates for the scale and shape parameters in the sample j. Finally, the WBL-normalized gene values were set to be the median values of the probe sets linked to each gene.where In the AGC method we assume that the mean of expression values of one gene in each array generation should be the same. If the mean value of some of the array generations differs substantially from the others, the shift is assumed to be caused by the array generation based variation. The AGC method aims to correct this variation.k to represent the total distribution of all potential expression values across all tissues in that array generation i. Therefore, the AGC normalization method normalizes the data to have the mean values μi, k = μk for all array generations i, where μk is the mean of all logarithmic values of the gene k. We assume that the minimum and the maximum estimates for the gene value are reached and the range of the gene k should approximately be , where ak is the lowest 2% value and bk is the largest 2% value of gene k. After array-generation based centering none of the values should go over this range. However, if the new centered value exceeds the range, the difference is diminished towards the range limits with coefficient c, 0 ≤ c ≤ 1. Here, the coefficient is set to c = 1/5 in order to diminish the greatest and smallest values. The centered values can now be obtained withThe AGC method requires the collection of samples to be relatively large so that one can assume the distribution of logarithmic values of each gene xk, j is the value of gene k in sample j measured with the array generation i, μk, iis the mean of the logarithmic values of gene k across array generation i and μk is the mean of the logarithmic values of gene k across all array generations. Further, the resulted AGC values are adjusted based on the equationwhere Finally, the values are converted back to the original scale byThe AGC method can be used together with any of the within slide methods presented above. After the AGC normalization the mean values of distributions of array generations are centered to have the same mean Figure .We have utilized equalized quantile normalization Q combined with the AGC method in the GeneSapiens database  with almA patent application of the AGC normalization protocol has been filed.RA, SK and MS contributed to the development and testing of the normalization. SH contributed to the development of normalization and supervised the comparison and validation of the normalization methods. JA and OK supervised the project. RA wrote the manuscript. All authors read and approved the final manuscript.