The expression microarray is a frequently used approach to study gene expression on a genome-wide scale. However, the data produced by the thousands of microarray studies published annually are confounded by “batch effects,” the systematic error introduced when samples are processed in multiple batches. Although batch effects can be reduced by careful experimental design, they cannot be eliminated unless the whole study is done in a single batch. A number of programs are now available to adjust microarray data for batch effects prior to analysis. We systematically evaluated six of these programs using multiple measures of precision, accuracy and overall performance. ComBat, an Empirical Bayes method, outperformed the other five programs by most metrics. We also showed that it is essential to standardize expression data at the probe level when testing for correlation of expression profiles, due to a sizeable probe effect in microarray data that can inflate the correlation among replicates and unrelated samples. Gene expression microarray technology Gene expression microarray results can be affected by minuscule differences in any number of non-biological variablesBatch effects are almost inevitable; largely because most of the available microarray platforms can assay fewer than 24 samples at a time (the latest technology may process 96 samples in each batch). Since hundreds or thousands of samples may be needed for population studies, samples for high-throughput microarray studies must often be processed at different times and/or sites. However, of the thousands of DNA microarray papers published every year, few address the problem. Of the 219 papers using microarray data published from January 1 to July 1, 2010, less than ten percent addressed this issue A number of approaches have been used or developed for identifying and removing batch effects from microarray dataa priori, and experimental data were used for verification.Ideally, all these batch adjustment methods would produce comparable results; however, since they are based on different statistical models, their accuracy, precision and overall effectiveness vary. In this paper, we sought to identify the algorithm that removes batch effects most effectively, including striking the optimal balance between precision and accuracy. Simulated data were used for each initial assessment, so the true values would be known We created two simulation data sets. The first was the Variation Assessment Simulated (VAS) data set, comprising 100 samples, 65 of which were assigned to Profile 1 and 35 of which were assigned to Profile 2, where profile was a generic random variable. Expression values for 991genes were simulated, all of them differentially expressed between the two profiles. This data set was generated twice, with different levels of batch effects incorporated each time, creating technical duplicates for comparison. The second simulated set, the Accuracy Assessment Simulated (AAS) data, consisted of 100 cases and 100 controls, with 1,200 out of 10,000 genes being differentially expressed with 12 different fold change values ranging from −3 to 3. The data incorporated a range of batch sizes and variable amounts of batch effects.http://www.affymetrix.com/support/technical/sample_data/datasets.affx), comprised three technical replicates of 14 separate hybridizations of 42 spiked transcripts at concentrations ranging from 512 pM down to 0.125 pM. Like the simulation data, true positives and true negatives were known for this data set.Both experimental brain expression data sets were produced using the Affymetrix GeneChip Human Genome U133A Array, a short oligonucleotide cDNA microarray. The Stanley Medical Research Institute (SMRI) data set included three technical replicates of 62 individuals, with each replicate processed in one of three laboratories The VAS and SMRI data were used for variation and precision assessment; the AAS and spike-in data were used for accuracy and overall performance evaluation. All the batch adjustment methods were applied after experimental data were pre-processed by robust multiarray analysis (RMA)We first measured how much each program reduces the variation caused by batch effects. In the VAS data set, the variation came only from batch and Profile effects. To verify our simulation result, we used SMRI data, in which nine factors are considered possible sources of variation and batch effects were divided into site and date effects separately. In both the simulated data and experimental data, variation attributable to batch effects before and after batch adjustment were identified using principal variation component analysis (PVCA)To test the programs' precision, we assessed whether the expression values of the technical replicates correlated better before or after batch adjustment, first between the two replicates from the VAS data using Pearson's correlation coefficient, then among the three SMRI replicates as assessed by intraclass correlation (ICC).signal detection slopeAs a measure of accuracy, the programs' abilities to accurately quantify fold change in expression were assessed using the correlation between nominal fold changes and observed fold changes. The To assess the overall detection ability of each program, we used a receiver operator characteristic (ROC) curve. ROC curves plot the true positive rate  against the false positive rate . The actual test statistic is the area under the curve (AUC); the program with the optimal combination of sensitivity and specificity will have the largest area of AUC. The ROC-AUC of the AAS data was calculated, and then the Affymetrix spike-in data was used to validate the simulation result. We also tested each program's ROC-AUC result with variously sized batches in the AAS data.Our ultimate goal was to identify the batch adjustment method that best prepares data from multiple batches for analysis or meta-analysis to be integrated, as measured by batch effects reduction, accuracy, precision and overall performance.We used principal variation component analysis (PVCA)The PVCA revealed that batch effects explained 30.4% of the overall variation in the VAS RMA data . All bat. The next largest effect came from PMI, which accounted for only 9% of the variation.We obtained similar results from the experimental SMRI data set . This tiAfter applying ComBat_p or ComBat_n, only 1% of the total variation was still attributable to batch effects: this represents a 98% reduction in batch effects and a 48% reduction in total variation. PAMR was almost as effective, reducing batch effects by 98%. SVA, DWD and Ratio_G were less so, at 94%, 73% and 57%, respectively.Precision refers to the closeness of the set of values obtained from multiple testing of identical samples. The precision of expression measures can be assessed by testing correlation among replicates, which we did between the VAS duplicates using Pearson's correlation coefficient and among the 62 SMRI triplicates using intraclass correlation coefficient (ICC) Figure S1).First, however, due to the sizeable probe effect in microarray data, which can inflate the correlations between sample pairs regardless of whether the sample pairs are replicatesFigure S2). All batch removal methods increased the replicates' correlation: the improvements of the probes' correlation distribution were all significant with p<0.0001. ComBat and DWD showed the largest median differences .After this standardization, we calculated Pearson's correlation coefficient of the replicates in VAS data, before and after batch adjustment. To compare the difference of the correlation coefficients, we transformed the correlation r to approximately normally distributed z-scores .In the SMRI experimental data, ICC was calculated for each probe set among the three replicate groups. The median z-score of the raw data was 0.188; after batch correction, the median z-scores were 0.305 for ComBat_p, 0.304 for ComBat_n, 0.298 for PAMR, 0.319 for DWD and 0.268 for Ratio_G. The increases in median ICC z-scores were all significant . The differences among ComBat_n, ComBat_p, PAMR and DWD were not significant, but they were all significantly better than Ratio_G (p<0.0001). Only SVA failed to increase the median z-score significantly  (2 = 0.95), all programs increased the correlation between nominal log 2 fold change value and observed fold change value (p<0.0001), ComBat_p (r2 = 0.98), ComBat_n (r2 = 0.98), PAMR (r2 = 0.97), DWD (r2 = 0.97), Ratio_G (r2 = 0.96), compared to the unadjusted RMA result (r2 = 0.95)  Figure 4signal detection slope).We also calculated each program's .To verify these results, we measured each program's correlation and slope in the Affymetrix experimental spike-in data set. SVA had the worst correlation again , and decreased the slope from 0.68 to 0.51 (p<0.0001) ). The results were similar in the Affymetrix spike-in data, except that SVA actually decreased the AUC value, from 0.93 to 0.76 (p<0.0001) .We used ROC curves to determine which program best optimized both sensitivity and specificity, i.e., maximized true positives (TP) while minimizing false positives (FP). To create an ROC curve, TP rate is plotted against FP rate; the actual test statistic is the area under the curve (AUC) Figure S5).To compare the methods' performances over a range of batch sizes, we re-set batch sizes at 20, 40 and 100 samples, re-generated the AAS data and recalculated the AUCs. The difference in AUC between ComBat and the other methods increased as the batch size decreased, suggesting that this Empirical Bayes approach is particularly appropriate for studies with fewer samples per batch .Note, however, that even the most effective batch effect removal program cannot correct for poor experimental design. If cases and controls are run in separate batches, genuine biological variation can be entirely confounded by batch effects. Indeed, this is what we found when we re-configured our simulation data to run cases and controls separately, and re-analyzed it. No method was able to reduce the batch effects sufficiently without also removing the variation caused by case-control differences  method performing the best. DWD and SVD/SVA were not evaluated, because Luo and colleagues wanted to develop their predictive model based on the existing data, referred to as the training data, which would not be affected when the model was applied to future samples. They used MCC because it is informative with very different class sizes, straightforward to calculate, and applicable for all 30,000+ of their models.) the results mirrored our accuracy and ROC-AUC results, the precision results elucidated differences among the programs not revealed by MCC, our accuracy measures or the ROC curves. For example, by testing multiple aspects of performance, rather than relying on a single measure, we demonstrated that while SVA substantially reduced the proportion of variation attributable to batch effects, as well performing with precision, it actually decreased accuracy relative to the unadjusted data. We also showed that PAMR performed very accurately, but slightly less precisely. We performed each test on both simulation data and experimental data; the simulation data let us control the variables and know the true values a priori, and then we were able to validate the simulation results in real data. Because we were not creating a prediction model, which is a combination analysis with batch correction, feature selection, classification and model selection, we were able to assess data before and after batch adjustment directly: this eliminates the potential for variation being introduced by those additional procedures. Finally, we took only date and site effects into consideration, since platform-, channel- or tissue-dependent variations are avoidable with careful experimental design.We, on the other hand, assessed how batch adjustment methods improve the integration of data processed at different sites or times when used as a standard quality control step post-RMA and pre-integration. This meant not only that we had a different aim than Luo and colleagues, but that we had fewer constraints. For example, we were able to assess widely-used algorithms, such as DWD and SVD/SVA, which they did not since would alter the training data. Also, although when we calculated MCC for our data , which is consistent with a previous studyFigure S1).We detected a strong probe level effect after applying RMA to the SMRI data and finding that the probe set expression values among replicates correlated no better than among non-replicates. We observed that the array level variation  data, we generated 100 samples for which 1000 probes were measured. Samples were assigned to one of two undefined profiles, which could represent disease status, drug treatment or similar random variable, 65 to Profile 1 and 35 to Profile 2. The simulation was run twice: the first replicate group was generated as if run in a single batch, while the second replicate group generated as if run in two batches. To address batch effects and treatment effect, we assumed the data followed an L/S model,X is a design matrix for profile conditions, and X. The additive and multiplicative batch effects of batch i for gene g are represented by , respectively. Measurement error is represented by, and is normally distributed, with an expected value of zero and varianceWhere Parameters X is a design matrix for sample conditions. Parameters The second simulated data set was the accuracy assessment simulated (AAS) data, which included probe set expression values for 10,000 probe sets in 100 controls and 100 patients. We again followed Equation (1), where http://www.niehs.nih.gov/research/resources/software/pvca/index.cfm) was used to estimate sources of variability and compare batch effects before and after adjustment in the VAS and SMRI microarray data sets. We first selected the top PCs, enough to explain a proportion of overall variation larger than a pre-defined threshold , and retained the corresponding eigenvalues. Each factor was treated as random in a mixed linear model and matched to a PC, then weighted by that PC's corresponding eigenvector. After we standardized the variation attributable to each factor, we calculated the proportion of total variance each factor explained. Including residuals, four factors' variations were estimated in the VAS data and ten factors' variations were estimated in SMRI data.The PVCA approach Click here for additional data file.Figure S2Distribution of z scores in VAS data. Box plots of the distribution of z scores transformed from Pearson correlation coefficients between simulated replicates. The methods are listed along the X axis. The Y axis is the distributions of all probes' z scores. The top of the box represents top of the third quartile, the bottom of the box represents the bottom of the first quartile, the middle bar is the median value, box whiskers extend to 1.5 times the interquartile range from the box and circles are possible outliers. The differences of correlation distribution are all significant with p value less than 0.0001; differences of distribution's median between RAW data and data have been processed with batch-adjustment methods are listed below the box plots.(TIF)Click here for additional data file.Figure S3Slope test in Affymetrix spike-in data. Observed versus nominal values in Affymetrix Latin square design spike in data, for RMA data and post batch adjustment methods. Expression values are plotted against the log (base 2) of the reported nominal concentration. The regression slope obtained utilizing all the data and the regression slopes obtain within each low, medium and high average log expression (ALE) value strata are shown. The slope of each line is reported in the legend. The vertical lines divide the ALE strata.(TIFF)Click here for additional data file.Figure S4ROC curves in Affymetrix spike-in data. ROC curves are graphical representations of both specificity and sensitivity that take into account both differentially and non-differentially expressed genes. Concentration pairs with fold-changes of 2 in spike-in genes were used to determine TP. Concentration pairs without any fold-changes were used to determine TN. We selected top 1000 log-ratio pairs to report.(TIFF)Click here for additional data file.Figure S5Compare of the AUCs in AAS data. The differences of the AUC between ComBat and the next best batch adjusted method are different when the sample size of each batch varies. From the left, the differences are 0.03, 0.02 and 0.01, as the batch sizes increase from 20, 40 to 100.(TIFF)Click here for additional data file.Figure S6Array level and probe set level variation. Boxplot of the RMA normalized SMRI data for (A) all the 186 arrays and (B) all 7643 probe sets. Y axis is the log2 intensity value and x axis is the (A) arrays or (B) probe sets. After RMA, gene intensity distributions are similar between arrays but not probe sets, leading to artificially large correlations between non-replicate arrays.(TIFF)Click here for additional data file.Table S1Detailed computational description of six programs. Detailed computational features of each program are provided, including software implement, file format, relative execution time, computational burden, batch size, URL of the available software and some program-specific notes.(DOC)Click here for additional data file.Table S2Batch effect completely confounded with outcome variation. We simulated the data with all cases in one batch and all controls in another. There are 1200 true positive genes in this dataset. The binary classification table shows overlapped positives genes among the true condition and with or without batch adjustment methods: before batch correction, there were 6518 significant results in raw data, with 62.4% (5495 out of 8800) false positive rate (FPR) and 14.8% (177 out of 1200) false negative rate (FNR); After batch correction, ComBat_p got 7 positive results with 0% FPR and 99.0% FNR; ComBat_n got 12 positive results with 0.1% FPR and 99.8% FNR; DWD got 402 positive results with 3.2% FPR and 89.9% FNR. None of the true positives were caught by PAMR and SVA with 0% FPR and 100% FNR. No control samples in the first batch so Ratio_G can't be applied for this adjustment.(DOC)Click here for additional data file.Table S3Assessment summary statistics table. The second column indicates the figure to which the summary statistic relates. Columns 3 through 9 show values for RMA, ComBat_p, ComBat_n, PAMR, DWD, SVA and Ratio_G, separately. The statistics are described in the text and best result is shown in bold. Abbreviations: PVCA, principal variation component analysis; ICC, intraclass correlation; ACC, accuracy; MCC, Matthew Correlation Coefficient; AUC, area under the curve.(DOC)Click here for additional data file.


Furthermore, as expected, the ensemble method did not improve the performance of these classifiers significantly. Representative experimental results are presented and discussed in this work.There has been considerable interest recently in the application of bagging in the classification of both gene-expression data and protein-abundance mass spectrometry data. The approach is often justified by the improvement it produces on the performance of unstable, overfitting classification rules under small-sample situations. However, the question of real practical interest is whether the ensemble scheme will improve performance of those classifiers sufficiently to beat the performance of single stable, nonoverfitting classifiers, in the case of small-sample genomic and proteomic data sets. To investigate that question, we conducted a detailed empirical study, using publicly-available data sets from published genomic and proteomic studies. We observed that, under  The c7bagging 5.There has been considerable interest recently in the application of bagging in the classification of both gene-expression data –12 and p4sufficiently to beat the performance of single stable, nonoverfitting classifiers, particularly in small-sample settings. Therefore, there is a pressing need to examine rigorously the suitability and validity of the ensemble approach in the classification of small-sample genomic and proteomic data. In this paper, we present results from a comprehensive empirical study concerning the effect of bagging on the performance of several classification rules, including diagonal and plain linear discriminant analysis, 3-nearest neighbors, CART decision trees, and neural networks, using real data from published microarray and mass spectrometry studies. Here we are concerned exclusively with the performance in terms of the true classification error, and therefore we employ filter-based feature selection and holdout estimation based on large samples in order to allow accurate classification error estimation. Similar studies recently published [t-test to assess whether the expected true classification error is significantly smaller for the bagged classifier as opposed to the original base classifier, under different number of samples, dimensionality, and number of classifiers in the ensemble. Clearly, the heuristic is beneficial for the particular classification rule if and only there is a significant decrease in expected classification error, otherwise the procedure is to be avoided; however the magnitude of improvement is also a factor—a small improvement in performance may not be worth the extra computation required  19–23, buublished 12 rely oublished . In our feature vectorlabelclassifierfeature-label distributionclassification rule is a mapping designed classifierclassification error is the probability that classification is erroneous given the sample data, that is, expected classification errorClassification involves a with replacement. This corresponds to using the empirical distribution of the data bootstrap sampleRandomization approaches based on resampling can be seen as drawing i.i.d. samples ensemble classification ruleIn aggregation by majority voting, a classifier is obtained based on majority voting among individual classifiers designed on the randomized samples for where the classifiers pment in , particuThe issue of how large In this section, we report the results obtained from a large simulation study based on publicly-available patient data from genomic and proteomic studies, which measured the performance of the bagging heuristic through the expected classification error, for varying number of component classifiers, sample size, and dimensionality.balanced bootstrapping, where all samples are made to appear exactly the same number of times in the computation [We considered in our experiment several classification rules, listed here in order of complexity: diagonal linear discriminant analysis (DLDA), linear discriminant analysis (LDA), 3-nearest neighbors (3NN), decision trees (CART), and neural networks (NNET) 27. DLDA putation .t-test [t-test is employed to assess whether the ensemble classifier has an expected error that is significantly smaller than that of the corresponding individual classifier.We selected data sets with large number e t-test  and the e t-test —in the lWe utilized the following publicly-available data sets from published studies in order to study the performance of bagging in the context of genomics and proteomics applications.These data come from the breast cancer classification study in , which athors in . Classifthors in .We employed here the data set "A" from the study in  on nonsmGiven the recent keen interest on deriving serum-based proteomic biomarkers for the diagnosis of cancer , we alsohttp://www.ece.tamu.edu/~ulisses/bagging/index.html. The case t-test, are indicated by bold-face type. This allows one to immediately observe that bagging is able to improve the performance of the unstable overfitting CART and NNET classifiers; in most cases, a small ensemble is required, and the improvement in performance is substantial. In contrast, bagging does not improve the performance of the stable, nonoverfitting DLDA, LDA, and 3NN classifiers, except via a large ensemble; and even so the improvement in magnitude is quite small, and certainly does not justify the extra computational cost . This is in agreement with what is known about the ensemble approach .t-test. One observes that as ensemble size increases, classification error decreases and tends to converge to a fixed value , but wet-test feature selection and a bagged CART in RELIEF feature selection. The case of the prostate cancer proteomic data is peculiar in that it presents the only case where the best option was not a DLDA, LDA, or 3NN classifier, but in fact a single CART classifier, namely, the case t-test feature selection, on the other hand, are very similar to the ones obtained for the lung cancer data set). Note that, in this case, the best performance is achieved by a single CART classifier, rather than the ensemble CART scheme. We also point out that the classification errors obtained with t-test feature selection are smaller than the ones obtained with RELIEF feature selection, indicating that RELIEF is not a good option in this case due to the very small-sample size , in thet-test feature selection are consistent across all data sets. When using RELIEF feature selection, there is a degree of contrast between the results for the prostate cancer protein-abundance data set and the ones for the gene-expression data sets, which may be attributed to the differences in technology as well as the fact that we do not employ baseline subtraction for the proteomics data in order to achieve better classification rates.The results obtained with We remark that results are not expected to change much if ensemble sizes are increased further . We observed that this never was the case for any of the data sets and experimental conditions considered here, except in the case of the proteomics data set with RELIEF feature selection in acute small-sample cases, when nevertheless the performance of a single unstable overfitting classifier  was better or comparable to the corresponding ensemble classifier. We observed that in most cases bagging does a good  job of improving the performance of unstable overfitting classifiers, but that improvement was not enough to beat the performance of single stable nonoverfitting classifiers.In this paper we conducted a detailed empirical study of the ensemble approach to classification of small-sample genomic and proteomic data. The main performance issue is not whether the ensemble scheme improves the classification error of an unstable overfitting classifier , or whether its classification error converges to a fixed limit; but rather whether the ensemble scheme will improve performance of the unstable overfitting classifier The main message to be gleaned from this study by practitioners is that the use of bagging in classification of small-sample genomics and proteomics data increases computational cost, but is not likely to improve overall classification accuracy over other, more simple, approaches. The solution we recommend is to use simple classification rules and avoid bagging in these scenarios. It is important to stress that we do not give a definitive recommendation on the use of the random forest method for small-sample genomics and proteomics data; however, we do think that this study does provide a step in that direction, since the random forest method depends partly, if not significantly, for its success on the effectiveness of bagging. Further research is needed to investigate this question.


High throughput ’omics’ experiments are usually designed to compare changes observed between different conditions (or interventions) and to identify biomarkers capable of characterizing each condition. We consider the complex structure of repeated measurements from different assays where different conditions are applied on the same subjects.We propose a two-step analysis combining a multilevel approach and a multivariate approach to reveal separately the effects of conditions within subjects from the biological variation between subjects. The approach is extended to two-factor designs and to the integration of two matched data sets. It allows internal variable selection to highlight genes able to discriminate the net condition effect within subjects. A simulation study was performed to demonstrate the good performance of the multilevel multivariate approach compared to a classical multivariate method. The multilevel multivariate approach outperformed the classical multivariate approach with respect to the classification error rate and the selection of relevant genes. The approach was applied to an HIV-vaccine trial evaluating the response with gene expression and cytokine secretion. The discriminant multilevel analysis selected a relevant subset of genes while the integrative multilevel analysis highlighted clusters of genes and cytokines that were highly correlated across the samples.R package mixOmics (http://cran.r-project.org/) with associated tutorials to perform the analysisa.Our combined multilevel multivariate approach may help in finding signatures of vaccine effect and allows for a better understanding of immunological mechanisms activated by the intervention. The integrative analysis revealed clusters of genes, that were associated with cytokine secretion. These clusters can be seen as gene signatures to predict future cytokine response. The approach is implemented in the  Recent advances in high throughput ‘omics’ technologies enable quantitative measurements of expression or abundance of biological molecules in a whole biological system. Various popular omics platforms in systems biology include transcriptomics, proteomics, cytomics and metabolomics. These experiments are usually designed to compare changes observed between different conditions or groups and are often used to identify biomarkers capable of characterising pathological states or response to treatment.The decreasing costs of these high-throughput platforms now enable repeated measures experiments on the same individuals or biological samples. Such experiments allow a substantial gain in information. For instance, longitudinal designs are more powerful as they reduce the noise due to inter individual variability, as long as the correlation between repeated observations is taken into account. There exists an abundant literature on the analysis of repeated measurements of omics data ,2. In thThe mixed model approach has been used for the analysis of one single data type (e.g. gene expression). However, a growing number of high-throughput data are generated in standard clinical trials. For example, the evaluation of HIV vaccine in phase I/II trials incorporates measurements of counts of numerous types of cell, of the production of intra and extracellular cytokines and of gene expression . The intIn recent years, several multivariate approaches have been proposed to combine two omics data, often in an unsupervised framework. In contrast to univariate repeated measures analysis, these linear multivariate approaches take into account the dependency between genes, are able to handle large and noisy data sets and do not face computational issues in the high dimensional case as matrix inversions are avoided. Most importantly in the context of this study, they enable the integration of data coming from different platforms and provide interpretable visualisation tools. These approaches aim at selecting correlated biological entities from two -11 or moThe flexibility and versatility of PLS also enable a supervised framework through PLS-Discriminant Analysis (PLS-DA ). A variIn this paper, we consider a two-step approach to model the correlation between repeated measurements while taking advantage of the multivariate approaches. We first propose to extract the within-sample variation -20 beforin vitro conditions on Peripheral Blood Mononuclear Cells: ‘NS’ (no stimulation); HIV Gag peptides ‘GAG+’ (peptides included in the vaccine), HIV Gag peptides ‘GAG-’ (peptides not included in the vaccine) and ‘LIPO5’ .Starting from the classical mixed-model, we present the principle of a multilevel analysis to extract the within-sample deviation of the data and we extend the approach to a two-factor analysis. The within data set is then analysed with either sPLS-DA to select discriminative genes between the groups of subjects on a single data set, or with sPLS to select subsets of correlated variables from two data sets. A simulation study is performed which demonstrates the good performance of multilevel sPLS-DA compared to a classical sPLS-DA. The approach is then illustrated on an HIV vaccination study, where the effect of a lipopeptide based vaccine was explored by measuring before and after vaccination various components of the immune response, including gene expression and cytokine secretion. These repeated measurement were made in several X(N × p) and Z(N × q) represent two data matrices (e.g. gene expression and cytokine secretion). We denote by N the total number of samples (or rows) in the data, and by n the number of experimental units (or unique subjects), p (resp. q) is the total number of genes (resp. cytokines), also called variables or predictors. The dummy matrix Y(N × G) indicates the group/treatment of the samples, with G the total number of groups.Let We first present the mixed-effect model as a pedagogical tool and then introduce the concept of the multilevel approach based on the “split-up” variation. Despite the fact that some similarities exist between the mixed-effect model and the multilevel “split-up” variation approach, we emphasize that the latter is performed completely independently from the estimation of the mixed-effect model. Moreover, the mixed model relies on certain assumptions (such as Gaussian distribution of random effects) that the split-up variation approach does not require.k for subject s with stimulation j. In this context, the mixed model is defined by: Let k, j, which can be further decomposed into j. The s, the residuals Gs) may differ. However, for each subject s we observe no more than one observation for each stimulation j, thus the subject effect interactions with the stimulation factor are confounded with the residuals. We also assume that where for a given gene s and represents the between-subject deviation. In the same spirit, the multilevel approach based on the split-up variation focusses on separating the different sources of variation: the within-subject deviation (“variation”) and the between-subject deviation.This model is known also as the one-way unbalanced random-effects ANOVA. A simple approach for identifying differentially expressed (DE) genes in this model is to test the stimulation effect for each gene and apply a multiple testing correction  gene expression matrix on s=1,…,n subjects with Gs stimulations  matrix containing ones and Xb is the between-subject matrix of size (N× p) defined by concatenating Xb with Xw =X−Xs· is the within-subject matrix of size (N× p), with Xs· the matrix defined by concatenating the matrices Xs·, with The matrix Similarly to the Analysis of Variance, it is easy to show that the sum of squares can be separated into three parts: X||2=trace(XTX). Equation (2) can be used to evaluate the magnitude of the different sources of variation.where ||Xw which includes the stimulation effect, in the same spirit as in [Xw.The mixed-model described earlier can provide an analysis for repeated measurements data in an unbalanced design. It can be viewed as an extension of a paired t-test to test the differences between paired observations. However, to tackle some of the previously mentioned limitations of the approach, we propose to combine a multilevel approach and a multivariate approach as an interesting alternative. Indeed, the multilevel step splits the different parts of the variation while taking into account the repeated measurements on each subject. Since the stimulation effect from each subject can be separated from the between subject deviation (variation), it is possible to examine the differences in stimulation effect within the subjects in a much easier way than without the separation of the difference sources of variation . Westerhit as in -20. Thisk for subject s with stimulation j at time t=1,2. In this context, the mixed model is defined as: We propose to extend this approach for data with two factors: the time (‘before’ and ‘after’ vaccination), in addition to the stimulation factor. Let k, j; t; j and the time t; s and the stimulation j; s and the time t; the residuals  where for a given gene According to the mixed model, we have: where the within-subject deviation can be further decomposed as: The matrix representation gives: Similar to the one-factor decomposition, the multivariate approach will be applied on the within matrix Xw or Once the multilevel approach has been applied to split up the variation in the data, a variant of the multivariate approach PLS Discriminant Analysis  is applied on the within matrix X(N × p) be the within predictor matrix  and Y(N × G) the response dummy matrix indicating the group of each sample. In PLS-DA, X is column standardized. The PLS-DA objective function to solve can be written as [Linear Discriminant Analysis (LDA) and Partial Least Squares Discriminant Analysis  are expitten as : ξ=Xuthe discriminant direction vector, which is a linear combination of the original variables. The vector u is the associated loading vector indicating the weights of each variable in the linear combination ξ. Once step (3) has been performed and the first weight vector u1 has been extracted, both matrices Y and X are deflated such that the following loading vector u2 is orthogonal to the previous one. PLS-DA therefore outputs a set of loading weight vectors u1u2,…,uH and associated discriminant direction vectors ξ1ξ2,…,ξH, where H is the number of PLS-DA dimensions (or deflations).where we denote by L1 constraint on u in order to ensure that some uj will be estimated as exactly zero . Thus, these corresponding variables will not contribute to the discriminant direction. sPLS-DA therefore allows variable selection for choosing the variables that best discriminate/separate the sample groups.The sparse version proposed by Lê Cao et al.  uses theH and the number of variables to select on each dimension (PLS component). Lê Cao et al. [H=G−1. The number of variables to select is, however, a challenging issue as tuning criteria are often limited by the very small number of samples. In this study, we considered two criteria to guide the choice of this parameter, both of them are applied sequentially, dimension per dimension.Two parameters need to be tuned in sPLS-DA: the number of discriminant vectors o et al.  showed tTuning criterion 1. One option is to use cross-validation to choose the optimal number of selected variables in order to avoid selection bias [k fold cross-validation. In the case of a very small sample size (<15 subjects), the leave-one-out cross-validation (denoted “loo”) can be used instead. In the specific context of repeated measures and in order to respect the data structure, the training set is composed of the measurements on all experimental units except the measurements on one subject s which defines the test set denoted β is the regression coefficient matrix from sPLS-DA var(Xu) for each deflated matrix and with respect to the number of selected variables. This is similar to that proposed by Waaijenborg et al. [g et al.  and Parkg et al. . The numXw  and Z(N × q) are two continuous within matrices of two different types of predictors (e.g. gene expression and cytokine secretion). In PLS, both X and Z are column standardized. To improve readability, the subscript w is removed from both these matrices. PLS relates X and Z by a linear multivariate model, while also modelling the structure of X and Z. PLS is particularly useful for analysing noisy, collinear, even incomplete, high dimensional data, see [Xand Z into new variables (component scores) denoted by  for the X-scores and  for the Z-scores. These scores should be few in number , orthogonal to each other within each data set, and estimated as linear combinations of the original variables from X and Z with their weights coefficients indicated in the associated loading vectors uh and vh respectively. In matrix representation, we have X=ΞCT + E, Z=ΩDT + F, where E and F are the residual matrices, and the column matrices in C and D are the coefficients from the local regressions of the score vectors ξh(ωh) onto the current deflated matrices defined as Partial Least Square regression  is in fata, see  for a reξh,ωh). The PLS objective function is: PLS relates both matrices by maximising the covariance between each pair of scores , which is solved with a Lagrangian form (see [X and Z indicated in the loading vectors (uhvh) for each PLS dimension h, and a set of score vectors (ξhωh) that are useful for graphical representations.This PLS form is often referred to as “PLS2 mode A” in the literature  where, sorm , which is the criterion maximized in sPLS2 mode A for each PLS dimension h (see equation (4)). For an optimal number of selected variables from both datasets, one would expect this criterion to achieve also a maximum.As an extension to the tuning criterion 2 from the previous section, and similar to what was proposed by Waaijenborg et al.  and ParkWe first present the results of a short simulation study to show the importance of using a multilevel approach in comparison to a standard sparse partial least square analysis on the original data. We then apply the proposed multilevel approach on an HIV-vaccination study.A simulation study based on the following mixed effects model was performed: k=1,…,1000). For any given pair of genes k1 and k2 in the same cluster, a pairwise correlation for Σπ is a (100× 100) matrix with Σε is a (100×100) matrix with  with To mimic the application, clusters of genes discriminating 4 conditions were generated  , where the mean effect of each stimulation is specified by μk=Tand μk=T.• 2 gene clusters discriminate  versus  with μk=Tand μk=T.• 2 gene clusters discriminate LIPO5 versus GAG+, while GAG+ and NS have the same effect: μk=cTand μk=cT.• 2 gene clusters discriminate GAG- versus NS, while LIPO5 and GAG+ have the same effect: μk=cTand μk=T.• the 4 remaining clusters represent noisy signal (no stimulation effect): ρ=0.7 or 0.8. Different values for The intra cluster correlation was either set to From the simulated data, the within matrix was computed and applied to multilevel sPLS-DA. Figure Firstly, in order to highlight the benefit of the multilevel approach in comparison to the multivariate approach without the split-up variation step, a prespecified number of genes was selected on each dimension in order to assess the ability of each approach to select the true relevant genes. As expected, 3 components (linear combinations of 200 genes) were sufficient to discriminate the effect of the 4 stimulations. Multilevel sPLS-DA (applied on the within matrix) selected 92% of the true simulated discriminative genes as compared to 75% of the true discriminative genes for classical sPLS-DA , see Table Secondly, leave-one-out cross-validation was performed on each simulation run to evaluate the error rate of classification of classical sPLS-DA or multilevel sPLS-DA Table . The clain vitro stimulated Peripheral Blood Mononuclear Cells (PBMC). Cytokine secretion was analysed by cytokine multiplex (millipore) in the supernatant of PBMC after 11-day-culture, and the data set consists of 10 cytokines measurements . Gene expression was analysed using the Illumina HumanHT-12 v4 Expression BeadChip on PBMC before (W0) and 14 weeks after vaccination (W14), 6 hours after in vitro stimulation by either (1) all the peptides included in the vaccine (LIPO-5), or (2) the Gag peptides included in the vaccine (GAG+) or (3) the Gag peptides not included in the vaccine (GAG-) or (4) without any stimulation (NS).The data come from a trial evaluating a vaccine based on HIV-1 lipopeptides in HIV-negative volunteers . The vac2 transformation and quantile normalisation were applied on the gene expression data using the Rlimma package. Probes were further prefiltered for each time point (before and after vaccination) using a P-value detection . The preprocessed data set contained the expression of 25,109 probes for 12 subjects for 4 types of stimulation before vaccination (W0) and the expression of 24,687 probes after vaccination (W14). Some samples were not available due to DNA quality issues, resulting in 44 samples at W0 and 42 samples at W14. For the multilevel approach with two factors, the analysis was performed on the common prefiltered probes before and after vaccination .Background correction, logThe statistical analysis was performed on the probe expression, but the results were biologically interpreted at the gene level.First we present results obtained using a mixed model and discuss some potential limitations of this method in the context of small sample size. Then we present the results obtained using multilevel sPLS-DA for one and two-factor analyses. To shorten the length of the paper, some results have been moved in Additional file mle function from the R package nlme with the maximum likelihood method for the estimation of the different models. A global test (likelihood ratio test) followed by an FDR multiple correction (5%) identified 2308 DE genes in at least one of the stimulation. Pairwise comparisons based on Wald test (FDR = 5%) were then performed to compare LIPO5 vs. NS (2108 DE genes), GAG+ vs. NS (1087 DE genes) and GAG-vs. NS (209 DE genes). The summary of the results is available in Section 1 of the Additional file The one-level mixed model was applied to the W14 transcriptomics data. We used the The univariate mixed model approach is commonly used to analyse data with repeated measurement with an unbalanced design. However, several reasons favor the use of a multilevel approach in this high dimensional setting. Apart from the already mentioned problem of numerous independent tests and the requirement to apply multiple correction , anotherH=3. Respectively 30, 137 and 123 genes were selected with the approach on each dimension according to the tuning criterion 1 for the most parcimonious model. Although k fold cross-validation would have been preferable to use, loo was used in this study given the small number of subjects. The following ‘loo’ classification error rates  were obtained on the first three sPLS-DA dimensions compared to  when applying sPLS-DA on the original matrices , indicating that there was a change in expression level after vaccination  across the samples. While the paired structure of the data is still taken into account in the analysis via the decomposition of the within matrices Multilevel sPLS was applied on the within matrices of the gene and cytokine data sets after vaccination. Given the very small number of cytokines, all cytokines were selected in the model, and the tuning of the number of variables to select was only performed on the gene expression data set. Respectively, a selection of 50, 1 and 60 genes was performed each of the sPLS dimension, corresponding to a correlation of . A drop of the subsequent correlations for the other dimensions guided the choice of 3 components in the model.Although unexpected and indicated by the tuned correlation value of 0.62, the selection of one single gene on the second dimension was not surprising given the sample representation that was obtained (see Additional file In this paper, we have proposed a two-step analysis combining a multilevel approach and a multivariate approach to analyze repeated measures of gene expression. The multilevel approach first extracts the within-sample variation while the multivariate approach applied on the within matrix takes into account the dependency between the variables. The multilevel approach was extended for one and two factors analyses.Two multilevel variants were proposed with either sPLS-DA or sPLS. The multilevel sPLS-DA approach selects genes separating the groups of subjects on a single data set. The simulation study comparing multilevel sPLS-DA and the sPLS-DA applied on the original data demonstrated the good performance of the model. The multilevel sPLS approach integrates two experiments made on different platforms but on the same subjects, and selects subsets of correlated variables from both sets.The application of both types of approaches on the HIV-1 vaccine trial showed their ability to highlight the stimulation groups and to select biologically relevant genes related to immune response. Hence, our combined multilevel approach may help in finding signatures of vaccine effect and allows for a better understanding of immunological mechanisms activated by the intervention. Future work will include a thorough analysis on the gene/probe annotations to fully understand the mechanistic link between gene differential expression, cytokine secretion according to the various stimulations.ahttp://www.math.univ-toulouse.fr/~biostat/mixOmics.The authors declare that they have no competing interests.BL and KL developed the methodology, the R code, performed the simulation and the analysis on the dataset as well as wrote the manuscript. RT developed the methodology, interpreted the dataset as well wrote the Manuscript. HH collected the data and wrote the Application section. All authors read and approved the final manuscript.Supplementaries results regarding the VAC18 study experiments from two assays.Click here for fileR code used for the analysis of the VAC18 study.Click here for file


Microarrays are widely used to assess gene expressions. Most microarray studies focus primarily on identifying differential gene expressions between conditions , for discovering the major factors that cause diseases. Because previous studies have not identified the correlations of differential gene expression between conditions, crucial but abnormal regulations that cause diseases might have been disregarded. This paper proposes an approach for discovering the condition-specific correlations of gene expressions within biological pathways. Because analyzing gene expression correlations is time consuming, an Apache Hadoop cloud computing platform was implemented. Three microarray data sets of breast cancer were collected from the Gene Expression Omnibus, and pathway information from the Kyoto Encyclopedia of Genes and Genomes was applied for discovering meaningful biological correlations. The results showed that adopting the Hadoop platform considerably decreased the computation time. Several correlations of differential gene expressions were discovered between the relapse and nonrelapse breast cancer samples, and most of them were involved in cancer regulation and cancer-related pathways. The results showed that breast cancer recurrence might be highly associated with the abnormal regulations of these gene pairs, rather than with their individual expression levels. The proposed method was computationally efficient and reliable, and stable results were obtained when different data sets were used. The proposed method is effective in identifying meaningful biological regulation patterns between conditions. Using microarray technology combined with computational analysis is one of the most efficient and cost-effective methods for studying cancer. Using this method has enabled scientists to investigate and understand a vast array of cancer information –5, and iIn microarray experiments, the gene expression levels of the samples can be detected using the intensity of probes . HeretofCloud computing and parallel processing are considered valuable techniques because using them can greatly reduce the computation time of a program by efficiently combining multiple computers and processors in parallel. Therefore, we implemented cloud computing techniques to complete the program in considerably little time. Apache Hadoop is a distributed parallel data-processing framework that supports MapReduce-type computations, enabling users to perform distributed computations effectively in increasingly brittle environments . We propPreviously, several methods have been developed to identify differential gene expressions in biological pathways. Most of these methods focus on diagramming gene expression levels and calculating the correlation of clustered genes. Kanehisa et al. proposed an approach, Pathway Miner, which extracts gene-association networks from molecular pathways for predicting the biological significance of gene expression microarray data . When paThe KEGG consists of a suite of databases that record an extensive amount of information on genes, enzymes, and regulation pathways , facilitApache Hadoop is an open source implementation of the Google MapReduce technology that simplifies programming tasks by automatically performing duties such as job scheduling, distributed aggregation, and fault tolerance , 13. TheSeveral experimental studies have examined the genetic profiles of breast cancer samples, and most of them have focused on identifying the differential gene expressions between relapse and nonrelapse breast cancer samples –24. PrevThe system flow of the proposed approach is depicted in affyQCReport [gcrma function of the R package affy was applied to normalize the CEL files by using the Robust Multiarray Averaging (RMA) method [The microarray data were extracted from the GEO and segregated into a nonrelapse condition and relapse condition according to the sample descriptions. Gene profiling was performed using Affymetrix U133A arrays, and microarray quality control was performed using an R package QCReport . The gcr) method .The master node divides the computation into multiple smaller subproblems and distributes them to worker nodes. The master node collects all of the answers submitted by the worker nodes and combines these answers into the output result. Two MapReduce programs were implemented to compute the linear regression validation and correlations for each pair of genes based on Hadoop Version 1.1.1. We used our programs to conduct a performance evaluation on a Hadoop cluster of 10 Xen virtual machines, where nine are the data nodes and one is the name node. Based on our settings, every two virtual machines were deployed on a physical machine. A physical machine was equipped with two CPUs with Intel Xeon E5504 4C 2.0 GHz and 16 GB of RAM; each virtual machine for the data nodes was equipped with a virtual CPU (VCPU) with two cores and 1 GB of RAM; the virtual machine for the name node was equipped with a VCPU with four cores and 4 GB of RAM. To evaluate the scalability of our MapReduce implementations, we also implemented two corresponding sequential Java programs as the basis for performance comparisons. In the performance evaluation experiments, we partitioned the data set into several pieces, uploaded them to the Hadoop file system, and submitted several MapReduce jobs to analyze the pieces of data.The KEGG  Java APIx-axis and y-axis, respectively, for drawing a node. The number of nodes represented the number of samples. Subsequently, linear regression analysis was applied to discard outliers based on the least square value of each node. If the least square value of a node is greater than the average least square value-added threefold standard deviation, then the node would be discarded when calculating the correlation of gene expression. Eighty percent of the data was reserved, at least for the setting of the threshold, to make the data more representative.For example, the correlation of the gene pair A-B was calculated under different conditions. First, we used the expression intensity of genes A and B as the values of the r, ranges between −1 and 1, and the greater the value of r, the stronger the coexpression between the members of the gene pair. In this study, coexpressed genes were defined as genes with a correlation greater than 0.45, and reverse-expressed genes were defined as genes with a correlation less than −0.45. Unrelated genes were defined as genes with a correlation between −0.45 and 0.45. Genes with differential correlations between the nonrelapse and relapse conditions were collected when the difference of the correlation of the genes in the two conditions was greater than AVG + 3∗SD or less than AVG − 3∗SD, where AVG and SD are, respectively, the average and standard deviation of the correlations of differential gene expressions between the two conditions.After discarding the outliers, Pearson's correlation coefficients were calculated for each gene pair. Pearson's correlation, GSE2109 was used for evaluating the performance of the cloud computing platform. The size of the data set for analysis was 54,675 (probes) × 2,158 (samples) of floating numbers. Our MapReduce implementation for linear regression validation and correlation computation is a map-only program. It iteratively issues a MapReduce job by setting the number of Reducer to be zero, and the number of iterations depends on the input data size. That is, the outputs of the map tasks are written directly to the files system. At each iteration, our MapReduce implementation generates different number of map tasks for different input size. Based on our Hadoop MapReduce setting, it generates 2, 3, 6, 12, and 17 map tasks for 5,000, 10,000, 20,000, 40,000, and 54,675 probes, respectively.http://dx.doi.org/10.1155/2014/763237), and The number of differential gene expression correlations between the relapse and nonrelapse breast cancer samples from GSE2034, GSE1456, and GSE4922 is shown in Table S1  family are involved in the breakdown of the extracellular matrix in normal physiological processes, such as tissue remodeling, embryonic development, and disease processes, such as arthritis and metastasis. Core binding factor (CBF) is a heterodimeric transcription factor binding to the core element of many enhancers and promoters. Chromosomal translocations involving CBF are well documented and have been discovered to be associated with several types of leukemia. The protein encoded by FIGF is a member of the platelet-derived growth factor/vascular endothelial growth factor (PDGF/VEGF) family and is active in angiogenesis, lymphangiogenesis, and endothelial cell growth.As mentioned, most of the discovered genes were related to cancer pathways and cancer regulations, including immunity, cell growth, tumorigenesis, and apoptosis. Although the gene expressions between relapse and nonrelapse samples were not substantially different, their correlations were substantially different. Thus, we believe these regulations of genes may be essential for regulating breast cancer recurrence.This study proposes an approach for discovering condition-specific correlations of gene expressions. An Apache Hadoop cloud computing platform was implemented to reduce the computation time. By using microarray data from the GEO, we discovered numerous differential gene expression correlations between the nonrelapse and relapse conditions of breast cancer. The results show that breast cancer recurrence is highly associated with the abnormal regulation of these gene pairs, rather than their individual expression levels. The pathways in cancer specifically show that NKFB2-PTGS2, JUN-FIGF, and RUNX1-CEBPA possess higher correlations of gene expression in nonrelapse samples and that JUN-MMP1 possesses higher correlations of gene expression in relapse samples. In addition, using the cloud computing technology successfully reduces the time required for conducting gene expression correlation analysis of microarray data, and it can be further applied for analyzing the correlation between different transcript isoforms using RNA sequencing data, which is helpful for deciphering the regulatory mechanisms of genes. The results show that our method is effective and can be extended to areas of biological analysis beyond that of breast cancer nonrelapse and relapse. We believe that the proposed method is effective for identifying meaningful biological regulation patterns between conditions and can be applied for developing coexpression networks and protein-protein interactions in the future.Figure S1. The differential correlation of gene expression between relapse and nonrelapse samples in pathways in cancer of the KEGG.Table S1: Correlations of gene expressions between nonrelapse and relapse samples in three data sets.Table S2: Gene expression correlations between relapse and nonrelapse samples in Pathways in Cancer.Click here for additional data file.


The agreement between PAM50 centroid-based single sample prediction (SSP) and PLS-regression was excellent (weighted Kappa: 0.988) within the training samples, but deteriorated substantially in independent samples, which could attribute to much more unclassified samples by PLS-regression. If these unclassified samples were removed, the agreement between PAM50 SSP and PLS-regression improved enormously . Our study ascertained the feasibility of PLS-regression in multi-class prediction, and distinct clinical presentations and prognostic discrepancies were observed across breast cancer molecular subtypes.Multiclass prediction remains an obstacle for high-throughput data analysis such as microarray gene expression profiles. Despite recent advancements in machine learning and bioinformatics, most classification tools were limited to the applications of binary responses. Our aim was to apply partial least square (PLS) regression for breast cancer intrinsic taxonomy, of which five distinct molecular subtypes were identified. The PAM50 signature genes were used as predictive variables in PLS analysis, and the latent gene component scores were used in binary logistic regression for each molecular subtype. The 139 prototypical arrays for PAM50 development were used as training dataset, and three independent microarray studies with Han Chinese origin were used for independent validation ( Multi-class prediction remains a challenge for high-throughput bioinformatics such as analysis of microarray gene expression data. Numerous machine learning algorithms are readily available for high-throughput data analysis, most of which, however, are limited to scenarios of the classification or prediction with only two classes. This difficulty arises not only from the vast data amount produced by high-throughput microarray or sequencing experiments but from the highly-correlated and nonstochastic nature of genetic/gene expression data. For real-world applications, dichotomous classifications between cancer/normal, alive/dead, and responsive/resistant status are mostly encountered, and many machine learning algorithms and bioinformatics tools perform quite well with sufficient discriminative power –3.n  < p (genomic/gene expression features) problem inherited in high-throughput microarray or sequencing techniques is to reduce the high-dimensional data using gene component analysis [p′) is no longer larger than sample numbers (n), facilitating the usage of classical statistical tools.One way to tackle the analysis –7. Gene In previous work we demonstrated that gene component analysis could discriminate estrogen receptor (ER) positive and negative breast cancers and gene component classifiers could be projected into independent samples with high predictive accuracy, as well as an integrated step of automatic gene selection . We alsoThe aim of the study was to apply PLS-regression for breast cancer intrinsic taxonomy, of which five distinct molecular subtypes were identified from microarray experiments. Here we extended the applications of PLS-regression from two-class  into multiclass prediction of the full spectrum of breast cancer intrinsic taxonomy . We hypoIn the past decade, microarray experiments have redefined breast cancers as heterogeneous diseases in terms of molecular aberrations, and a number of taxonomic classifications based on gene expression profiles that have been reported have shown some prognostic significance. One such molecular taxonomy is the “intrinsic subtype” proposed by the Stanford/UNC group. Perou identified 476 intrinsic genes from 65 patients with breast cancers and normal individuals; four subclasses: basal-like, Erb-B2+, normal breast-like, and luminal epithelial/ER+ were revealed by class discovery through clustering analysis , 11. Thehttps://genome.unc.edu/). Centroids were the mean expression values of intrinsic genes corresponding to each molecular subtype. The prototypes included 12 normal breast-like, 57 basal-like, 35 HER2-enriched, 23 luminal-A, and 12 luminal-B tumors.The PAM50 intrinsic signature genes that defined 5 molecular subtypes  were retrieved. The expression values of training samples deriving intrinsic signatures were downloaded from UNC Microarray Database (n (sample size) < p (gene expression predictors) problem became tractable since a much smaller p′ (gene component) was used instead of original microarray gene expression features. At the same time model over-fitting and collinearity of original p genes was avoided due to the limited number of gene components (p′) used in classification algorithm and the uncorrelated nature between successive latent factors.Following identification of intrinsic genes, PLS was used for dimension reduction and latent X-factors (gene components) construction. The troublesome X0 and Y0 be centered and scaled matrix of predictive and responsive variables; one dummy variable Y indicating clinical phenotype was needed for binary classifications in PLS. PLS maximized X0′Y0 for latent factor construction. PLS predicted X0 (and Y0) with the following formula:All gene component regressions were essentially the linear transformations of original gene expression values and could be viewed as the projection of high dimensional predictor space into a few orthogonal latent factor axes. PLS maximized the covariance between the predictor and response variables. In matrix algebra, let x-scores (t = X0w) and y-scores (u = Y0q) were derived to meet the criteria of maximal covariance of t′u where w and q were associated weighted vectors. The vectors p and c were x- and y-loadings, respectively. It should be noticed that all latent factor extractions were under orthogonal constraints; successive latent factors (gene components) were linearly independent to each other, and usually the corresponding eigen-vectors were normalized to unity .The The number of latent factors used for PLS-regression was determined by cross-validation. We used split-sample cross validation to determine the number of latent factors that delivered the minimal predicted residual sum of squares (PRESS) followed by van der Voet's test; the fewest number of gene components that was insignificantly different from the factor number corresponding to the minimal PRESS should be used in regression . In shork was the number of gene component used, and xk was the kth gene component score. To evaluate classifier performance, leave-one-out cross validation was used to prevent model over-fitting. The threshold of a positive prediction was defined to have a more than 0.5 of cross-validated predicted probability. The process of PLS scores construction and LR prediction was repeated for each of the 5 molecular subtypes. Bonferroni corrections with a reduced α level of 0.01 were applied for all PLS-regression classifiers for multiple comparisons. All samples were categorized into one of the 5 molecular subtypes with the highest predicted probability, assuming that probability exceeding the threshold of 0.5. An ambiguous classification was claimed when more than one predicted probability was higher than 0.5 among all subtypes. A sample was designated as unclassifiable if none of the predicted probabilities of 5 molecular subtypes exceeded the threshold of 0.5.After the number of gene component was determined and each gene component score was calculated for all samples, binary logistic regression (LR) was applied for classification/prediction. For binary LR, the predicted probability was estimated byIn each classifier, a binary PLS-regression was fit, with the most relevant genes associated with the subtype enrolled as predictive variables. Each classifier compromised 10 out of the 50 PAM50 signature genes, and this class-specific gene selection avoided using all 50 genes into the PLS regression at the same time.Our microarray experiments and two publicly available microarray studies fulfilled the purpose of external validation –20. Our For relevant pathological features, ER positivity was defined as the presence of at least 10% of nuclei with positive results by immunohistochemical (IHC) analysis, and breast samples displaying low ER positivity (<10% of nuclei with positive stains) were not assayed in the current study. For HER2 status, the ASCO and CAP guidelines were followed: IHC3+ and IHC2+ with fluorescence in situ (FISH) hybridization amplification were considered to indicate HER2 overexpression.Total RNA from cancerous breast tissues was extracted by TRIzol reagent  and RNA was purified using RNeasy mini kits . RNA integration was tested by gel electrophoresis. Affymetrix  GeneChip Human Genome U133 plus 2.0 was used for the microarray experiment. Hybridization and scanning were performed according to the Affymetrix standard protocol. Images were scanned using GeneChip Scanner 3000, and the scanned images were processed with GeneChip Operating Software (GCOS). Robust multi-array average (RMA) algorithm was used to normalize 83 array chips .Two publicly available breast cancer microarray depositories, one from Lu et al. and another from Kao et al., were merged with our microarrays to form the validation dataset –20. BothAll intrinsic genes were mapped to the Affymetrix gene annotation file, and only the most variable probeset measured by inter-quartile range (IQR) across all arrays was used when multiple probesets per gene were encountered. The 535 breast cancer specimens of the Han Chinese patients were assigned to 1 of the 5 molecular subtypes with the nearest centroid method . Spearman's rank correlation coefficients were used, and samples were designated as unclassified if correlation coefficients to all 5 centroids were less than 0.1. To enhance the comparability between the original studies deriving intrinsic genes and independent samples in current study, mean-centering of genes was applied to the expression data of Han Chinese breast cancers, as suggested by the investigators of the Stanford group . All arrn = 4), luminal-B, and normal breast-like subtype since these cases were positively predicted by two classifiers.PLS-regression classifiers based on latent gene component scores were built for each molecular subtype from training dataset of 139 prototypical arrays. PLS-regression was performed for independent Han Chinese breast cancers including our series and two publicly available microarray depositories. To derive the “gold standard” for intrinsic subtype, centroid-based method (SSP) was used to designate each individual of the three studies into 1 of the 5 molecular subtypes.n = 125) of tested samples were categorized as unclassified by PLS-regression.Since no missing value was found in Affymetrix microarrays used for validation, there was no need of missing value imputations. Clinical and follow up data were available for 208 of Han Chinese breast cancers and we compared ER and HER2 phenotypes between distinct intrinsic subtypes designated by PAM50 SSP and PLS-regression . DespiteFigures In the current study, PLS-regression was used for microarray multiclass predictions. Latent gene component scores were used in binary LR, each time with one molecular subtype tested. For breast cancer intrinsic taxonomy, PLS-regression classifiers were built for five mutually exclusive molecular subtypes. Bonferroni corrections were applied for multiple comparisons (5 times of classifications per each case). If the cross-validated predicted probability was higher than 0.5, a positive prediction was recognized. For most instances, there was only one classifier reported a positive prediction and the sample was categorized into the corresponding subtype. If two classifiers reported a higher than 0.5 predicted probability, the case was classified into the subtype with the highest probability but an ambiguous prediction was identified. If all classifiers failed to deliver a prediction higher than the threshold of 0.5 cross-validated probability, an unclassified sample was claimed.Applications of gene component methodology for microarray studies had been reported in literature. West et al. demonstrated the “metagene” model, which used principle component (PC) scores from the top 100 genes showing the highest absolute correlations with clinical ER status of breast cancers and used these PC scores as predictive variables in binary regression . Followiy-scores was reported by Pérez-Enciso and Tenenhaus [Since PLS automatically produced (predicted) response variable (tumor class label), one-step PLS regression, which predicted tumor class directly from latent enenhaus . HoweverMulti-class prediction of PLS-regression was the extension of the regression for binary responses. The strategy of latent score construction remained the same. It was quite intuitive that polychotomous  LR could fill the task of prediction with multiple responsive levels. However, neither ordinal (with one baseline class) nor nominal LR resulted in a converged model in the training or validation dataset. Multi-class prediction remained a challenge for high-throughput gene expression data analysis with classical statistical tools.To overcome aforementioned difficulties, our strategy started with the development of PLS-regression for each of the molecular subtypes individually. In each classifier a binary PLS-regression was fit, with the most relevant genes associated with the subtype enrolled as predictive variables. In 139 prototypical arrays, the agreement between PAM50 SSP and PLS-regression was excellent (weighted Kappa: 0.988), indicating the robustness and feasibility of PLS-regression as an alternative classification method to PAM50 SSP. In validation dataset of 535 Han Chinese breast cancer microarrays, the agreement between PAM50 SSP and PLS-regression deteriorated substantially. If we took a close look at An apparent benefit of PLS-regression rather than centroid-based SSP proposed by PAM50 investigators was that the predictive probability was reported. In our study we used the 0.5 of  predicted probability as the threshold of a positive prediction. If the threshold was relaxed to a lower level, the number of unclassified cases decreased but was with the expense of increased ambiguous classifications (two or more than two classifiers reported a positive prediction). Although we could designate samples into the class with the highest predictive probability, there remained a doubt about the validity of molecular taxonomy when more than one classifier passed the predefined threshold and reported a positive prediction. With current threshold of 0.5, there were fewer than 10% of cases with ambiguous classifications.In centroid-based SSP, since samples were categorized into the subtype with the highest correlation coefficient, and the unclassified threshold was set to a much lower level , the higher proportion of unclassified cases of PLS-regression in independent dataset was not a drawback of purposed algorithm but indicated a more precise and sophisticated statistical rationale. In our opinion, the threshold of 0.1 correlation coefficient in PAM50 SSP was too loose as the proportion of unclassified cases was erroneously reduced with the expense of compromised reproducibility and robustness. The threshold of positive predicted probability could be viewed as a tuning parameter of PLS-regression, as suggested by high area under the curve (AUC) values of most classifiers Tables . In the The clinical ER and HER2 status for 208 Han Chinese breast cancers were presented in R-square as well as compromised sensitivity. Notably, normal breast-like centroid in PAM50 was derived from 29 normal breast samples; in 2009 Parker et al. clained that normal breast-like category of PAM50 should be treated as an internal quality control rather than a breast cancer intrinsic subtype such as normal breast-like subtype in Hu 306 and Sørlie 500 intrinsic signatures [n = 8) was less than the number designated as normal breast-like by PAM50 SSP (n = 56), indicating a more precious and valid prediction of the purposed gene component algorithm. The dubious clinical meaning and doubtful existence of normal breast-like subtype, also reflected in its heterogeneous clinical presentations of ER and HER2 phenotypes, remained unsolved and demanded further evaluations. Perhaps directly assaying true normal breast tissues might shed light on this issue.Tables gnatures . In thisOur study extended the applications of PLS-regression for gene expression data to multi-class taxonomy such as PAM50 intrinsic subtypes purposed by the Stanford/UNC group. With gene component classifiers and class-specific genes for each molecular subtype, the purposed algorithm was validated in original cohort deriving the PAM50 signature as well as in independent Han Chinese breast cancers with modest sample size. PLS-regression was evidenced to be a feasible and efficient alternative to centroid-based SSP when more than two classes were discerned. The increased proportion of unclassifiable cases in independent samples deserved meticulous evaluation. Whether inconsistency in classification threshold or unrecognized patterns in full spectrum of intrinsic taxonomy resulted in these undetermined cases was speculated; further gene expression studies might be directed to answer these questions in an effort to derive a more sophisticated signature for human breasts cancer.


The advent of personalized medicine requires robust, reproducible biomarkers that indicate which treatment will maximize therapeutic benefit while minimizing side effects and costs. Numerous molecular signatures have been developed over the past decade to fill this need, but their validation and up-take into clinical settings has been poor. Here, we investigate the technical reasons underlying reported failures in biomarker validation for non-small cell lung cancer (NSCLC).We evaluated two published prognostic multi-gene biomarkers for NSCLC in an independent 442-patient dataset. We then systematically assessed how technical factors influenced validation success.P = 0.001; biomarker #2: HR 1.42, 95% CI 1.03 to 1.96, P = 0.030). Further, despite being underpowered for stage-specific analyses, both biomarkers successfully stratified stage II patients and biomarker #1 also stratified stage IB patients. We then systematically evaluated reasons for reported validation failures and find they can be directly attributed to technical challenges in data analysis. By examining 24 separate pre-processing techniques we show that minor alterations in pre-processing can change a successful prognostic biomarker  into one indistinguishable from random chance . Finally, we develop a new method, based on ensembles of analysis methodologies, to exploit this technical variability to improve biomarker robustness and to provide an independent confidence metric.Both biomarkers validated successfully (biomarker #1: hazard ratio (HR) 1.63, 95% confidence interval (CI) 1.21 to 2.19, Biomarkers comprise a fundamental component of personalized medicine. We first validated two NSCLC prognostic biomarkers in an independent patient cohort. Power analyses demonstrate that even this large, 442-patient cohort is under-powered for stage-specific analyses. We then use these results to discover an unexpected sensitivity of validation to subtle data analysis decisions. Finally, we develop a novel algorithmic approach to exploit this sensitivity to improve biomarker robustness. The perfect medical treatment would start with instantaneous diagnosis, proceed rapidly to a treatment that offered complete cure with no side effects, and of course would incur minimal costs to the healthcare system. This dream scenario, while distant, has been made more plausible by a rapid reduction in the cost of molecular assays. Molecular profiling of clinical specimens offers hope in two ways. First, new candidate therapeutic targets are being identified ,2. TheseUnfortunately, the field of biomarker development has not reached its translational potential. Despite numerous reports of molecularly derived biomarkers to diagnose disease , predictThe management of resectable non-small cell lung cancer (NSCLC) would particularly benefit from the development of new prognostic tools. Despite improvements in staging, surgical methodologies, chemotherapy regimens and the addition of adjuvant therapies, 30 to 50% of patients with resectable NSCLC suffer relapse and die within 5 years -17. In fGiven the clinical need for a robust prognostic biomarker for NSCLC and the technical challenges that confounded prior studies, a multi-institute effort was undertaken. The Director's Challenge NSCLC study was an attempt to provide a large, sufficiently powered dataset to discover reproducible multi-gene biomarkers . This coThis study begins with an attempt to replicate the results of Subramanian and Simon . Surpriset al.  the original classifier was developed using RT-qPCR' . Our resHR: hazard ratio; CI: confidence interval; MBEI: model-based expression indices; NE: number of events; NSCLC: non-small cell lung cancer; RMA: robust multi-array averaging; GCRMA: GeneChip robust multi-array averaging; MAS5: microarray analysis suite 5.0.The authors declare that they have no competing interests.All authors were involved in the study design. Data were collected and analyzed by MHWS, MP and PCB. Statistical analysis was performed by MHWS, MP and PCB. Data interpretation was performed by MHWS, MP, MST and PCB. The manuscript was initially drafted by MHWS and PCB. All authors contributed to drafting of the manuscript. All authors have read and approved the manuscript for publication.Supplementary tables. Table S1: ProbeSet annotation used in analyses. Supplementary Table S2: overview of pre-processing algorithms. Supplementary Table S3: results of binary prediction performance of the three-gene and six-gene classifier in the Director's Challenge dataset. Supplementary Table S4: results (stage-adjusted) Cox proportional hazard ratio modeling three-gene classifier for all patients and stage IB patients in the 24 different pre-processed Director's Challenge datasets. Significant results (P < 0.05) are given in bold. Supplementary Table S5: results of binary prediction performance of the three-gene classifier in the 24 different pre-processed Director's Challenge datasets. Supplementary Table S6: results (stage-adjusted) Cox proportional hazard ratio modeling six-gene classifier for all patients and stage II patients in the 24 different pre-processed Director's Challenge datasets. Significant results (P < 0.05) are given in bold. Supplementary Table S7: results of binary prediction performance of the six-gene classifier in the 24 different pre-processed Director's Challenge datasets. Supplementary Table S8: results (stage-adjusted) Cox proportional hazard ratio modeling three-gene classifier for all patients in the 24 different pre-processed Bild datasets. Significant results (P < 0.05) are given in bold. Supplementary Table S9: results of binary prediction performance of the three-gene classifier in the 24 different pre-processed Bild datasets. Supplementary Table S10: results of binary prediction performance of the two classifiers in the Director's Challenge and Bild datasets for unanimous and ambiguous classified patients.Click here for fileClinical information and good/poor classification for the three-gene classifier in each of the pre-processing methods in the Director's Challenge datasets.Click here for fileClinical information and good/poor classification for the six-gene classifier in each of the pre-processing methods in the Director's Challenge datasets.Click here for fileClinical information and good/poor classification for the three-gene classifier in each of the pre-processing methods in the Bild datasets.Click here for fileSupplementary Figure S1. (a-d) Performance of the six-gene biomarker was evaluated in a sub-stage analysis (stage IA (a), stage IB (b), stage II (c), and stage III (d) patients), which were visualized with Kaplan-Meier curves. Each patient was classified into good (blue curves) and poor (red curves) prognosis groups using the six-gene biomarker. Hazard ratios and P-values are from Cox proportional hazard ratio modeling followed by the Wald test.Click here for fileSupplementary Figure S2. (a-d) Kaplan-Meier curves for the three-gene  and six-gene  classifiers in the Director's Challenge data [nge data , where dnge data .Click here for fileSupplementary Figure S3. Schematic overview of the methodology used to test sensitivity to differences in pre-processing in multi-gene biomarker performance.Click here for fileSupplementary Figure S4. Results for all Cox proportional hazard ratio modeling analysis for the 24 different pre-processing schemes in the Director's Challenge dataset [ dataset  for the Click here for fileSupplementary Figure S5. (a) Univariate analysis for each ProbeSet in the Director's Challenge dataset [P-value Wald test ≤ 0.05) in Cox proportional hazard ratio modeling analysis was highly variable.  Heatmaps of the hazard ratios (HRs) in each pre-processing schedule (columns) for ProbeSets (rows) with P-value Wald test ≤ 0.05 in at least one pre-processing schedule with default (b) or alternative annotation (c) also display this variance. dataset  revealedClick here for fileSupplementary Figure S6.  Marker performance improved when differentiating patients with identical classifications across all pre-processing schemes from the patients with ambiguous classifications for the three-gene biomarker (a versus b) in the Bild dataset [P-values are from stage-adjusted Cox proportional hazard ratio modeling followed by the Wald test. dataset . Good prClick here for file