Normalization of microarrays is a standard practice to account for and minimize effects which are not due to the controlled factors in an experiment. There is an overwhelming number of different methods that can be applied, none of which is ideally suited for all experimental designs. Thus, it is important to identify a normalization method appropriate for the experimental setup under consideration that is neither too negligent nor too stringent. Major aim is to derive optimal results from the underlying experiment. Comparisons of different normalization methods have already been conducted, none of which, to our knowledge, comparing more than a handful of methods.In the present study, 25 different ways of pre-processing Illumina Sentrix BeadChip array data are compared. Among others, methods provided by the BeadStudio software are taken into account. Looking at different statistical measures, we point out the ideal versus the actual observations. Additionally, we compare qRT-PCR measurements of transcripts from different ranges of expression intensities to the respective normalized values of the microarray data. Taking together all different kinds of measures, the ideal method for our dataset is identified.Pre-processing of microarray gene expression experiments has been shown to influence further downstream analysis to a great extent and thus has to be carefully chosen based on the design of the experiment. This study provides a recommendation for deciding which normalization method is best suited for a particular experimental setup. Analysing gene expression using microarrays is a well established method . Many diSeveral studies comparing different normalization methods have already been conducted, many of them focusing on Affymetrix chips -7, otheret al., Phenocopy - a strategy to qualify chemical compounds during Hit-to-Lead and/or Lead Optimization, submitted 2010). Within this project, the aim is to predict mode of action as well as off-target effects of compounds based on gene expression data. To do so, HaCaT cells were treated with TGF-β as well as seven different compounds in seven dosages and with six siRNAs. Gene expression was measured after 2, 4, and 12 hours using Illumina's Human HT-12 v3 Expression BeadChips [® technology to measure the quantitative abundance of eight genes at three time points that are known to be deregulated [Here we present a strategy for an in depth evaluation of normalization methods aiming at identifying the most appropriate one for a given data set. Our study compares established normalization methods available in the R environment to those offered by BeadStudio software. It focuses on the HumanHT-12 v3 Expression BeadChip, yet the underlying principles are directly transferable to other technologies measuring gene expression. Analyses described here provide the basis for the Phenocopy project  or no background modification (noBg_*) has been applied. In a next step, the data was transformed using either log2-transformation (log) or variance-stabilizing transformation (vst) [rma [forcePos [2-transformation. In a last step, the data was normalized using quantile, loess, or rsn [vsn [average, rankInvariant, cubicSpline) are used for normalization.Expression data was pre-processed in 25 different ways Figure . We focuon (vst) . Since Bst) [rma  or forceforcePos  to be ab, or rsn  normalizrsn [vsn  or the nDifferent pre-processing methods were evaluated by analysing the variance of the resulting gene expression intensities via various statistical measures. Some of these have already been used in other studies . In addiPre-processing methods were scored from -2 to 2 based on how well they match the required criteria for the different analyses described in this section. As it is difficult to clearly categorize the methods based on the examined measures, the final decision of which score to assign to some extent stays subjective. However, it is unambiguously possible to separate better pre-processing methods from worse. A complete overview of the scores assigned and the final ranking is given in Figure One basic assumption of gene expression pre-processing methods is that the majority of genes do not change their expression under different conditions. Additionally, expression intensities of replicates should be very similar compared to the expression of transcripts between differently treated sample groups. Based on these principles, we looked at different statistical measures to identify the method best suited for our dataset with respect to variance.noBg_average, noBg_rankInvariant, bg_rankInvariant, bg_average) show cumulative distribution functions that are clearly above those obtained based on all other pre-processing methods. Applying neither background correction nor any normalization method (noBg_noNorm) results in a data set producing less adjusted p-values < 0.02 than other pre-processing methods. With decreasing significance of the adjusted p-values more pre-processing methods produce fewer p-values of higher significance than noBg_noNorm. Based on the data set used, we expect only a small subset of the transcripts to be significantly deregulated. Since for bg_noNorm compared to other pre-processing methods the fewest genes would be detected as being differentially expressed, i.e. showing a relatively high variation between compared to within group variability, this method seems to provide the best results. The remaining pre-processing methods perform relatively similar and equally well.A good normalization method should minimize the variation within a treatment group. Furthermore, the variation within a treatment group should be smaller than the variation between groups. The F-statistic is a typical measurement to compare the variation between replicates to the variation between conditions or treatment groups ,20. Resu10 should be. When plotting these parameters, an appropriate normalization method should result in smoothly increasing values with not too much scattering around the fitted curve. Figure 10 against the respective variance between the control groups at time points 2 h, 4 h, and 12 h for three of the pre-processing methods, an overview over all results is given in Additional file noBg_cubicSpline, noBg_log_rsn, noBg_vst_rsn, and noBg_vsn. All of the normalizations performed on rma background corrected data as well as bg_vsn display a relatively high -log10 for a relatively high proportion of low between group variability values leading to a high scattering of observations in these regions. Using, for example, the rank invariant normalization of BeadStudio (noBg_rankInvariant) the p-values for the low between group variability tend to be relatively small. This could lead to an overestimation of differentially expressed genes when filtering solely based on p-values.Assuming a stable variance over the within group measurements, the bigger the variance between the groups, the bigger the respective -logbetween) and within (MSQwithin) group variances and their relation to each other. If genes are not differentially expressed, MSQbetween should be comparable to MSQwithin. For genes that are differentially expressed, MSQbetween is supposed to be higher than MSQwithin. Figure between (red) and MSQwithin (blue) values. Since we expect some genes to be differentially deregulated across the different time points under consideration, quantiles of MSQwithin values should lie below the corresponding quantiles of the MSQbetween values. For the differentially expressed genes, within group variance should be smaller than between group variance, whereas for the genes not differentially expressed, the respective MSQbetween and MSQwithin values should show no great difference. Small interquartile ranges (IQRs) of MSQwithin are indicative for a comparable variability between genes.Further indications for good normalization are the distributions of between  which resulted in an MSQbetween of 1.33, indicated by a dashed grey line in Figure 2 ratio of 1 when group 3 is compared to group 1 or group 2, reflecting a relevant difference between those groups. A good normalization method should result in similar expression values for replicates and thus in small MSQwithin values hardly crossing this artificial MSQbetween. Additionally, since we limited the whole data set to expressions measured for untreated HaCaT cells across time, we expect only few genes to be differentially expressed. Thus, only a few genes are assumed to result in an MSQbetween above the artificial MSQbetween.To judge the values for MSQ, an MSQwithin of background normalized data (bg_*) result in outliers crossing the artificial MSQbetween, only those transformed using vst do stay below. Compared to other pre-processing methods, noBg_vst_loess, noBg_log_loess, and bg_vst_loess show a relatively wide IQR for both, MSQbetween and MSQwithin. Methods that meet the described behaviour by showing a low within group variability for which the quantiles generally exhibit lower values than the quantiles of the between group variabilities are bg_vst_quantile, bg_vst_rsn, noBg_log_quantile, noBg_log_rsn, noBg_noNorm, noBg_vsn, noBg_vst_quantile, and noBg_vst_rsn.Almost all boxplots representing MSQbetween and MSQwithin should exhibit clear differences. This fact renders density functions of MSQbetween and MSQwithin as an additional option for investigating these values. Within group variability should be smaller than between group variability and most of the genes should show a between group variability similar to the within group variability, i.e. are not differentially expressed. Thus, the mode of MSQwithin should be smaller than the mode of MSQbetween and the peak of the function for MSQwithin is supposed to be higher than the peak for MSQbetween. Lean MSQwithin functions, on the one hand, reflect a comparable within group variability for many genes. On the other hand, broader MSQbetween functions indicate that at least some of the genes, i.e. the differentially expressed ones, show a higher between than within group variability. Ideal characteristics of density functions as described here are very similar to the characteristics of ideal boxplots mentioned in the previous section. In contrast to density functions, boxplots give a very rough idea about the distribution of the values, also depicting outliers. Density functions deliver a more detailed view of how the values are distributed across different ranges.Density functions of MSQbetween (red) and MSQwithin (blue) for three of the pre-processing methods, a complete overview is given in Additional file noBg_log_quantile, noBg_log_rsn, and noBg_vsn best exhibit the desired behaviour. Unexpectedly the density functions of MSQwithin generated by bg_vst_loess, noBG_log_loess, and noBg_vst_loess are bimodal. One reason for bimodal density functions could be a group of transcripts exhibiting higher variability compared to other transcripts. In general, it is expected that the data shows a consistent variability. Having the opportunity to choose between normalization methods resulting in unimodal or bimodal density functions for MSQwithin, normalization methods leading to a unimodal distribution should be favoured.Figure noBg_average normalization  versus the respective log2 ratios. Due to the tendency of larger log2 ratios being connected to more significant -log10 a volcano like shape is generated. Pairwise comparisons  using a moderated t-statistic [2 ratios and p-values. Our aim is to detect normalization procedures yielding as correct estimates of log2 ratios as possible combined with as informative p-values as possible. As mentioned above higher log2 ratios should tend to have higher -log10. The loess fits of the log2 ratios and -log10 pairs (dark blue curves) of the volcano plots shown in Figure 2 ratios should not be too large.Volcano plots constitute a standard visualisation of microarray results. They are generated by plotting -logtatistic  were per2 ratios the -log10 are relatively low. Additionally, the -log10 for similar log2 ratios tend to scatter extremely. Some volcanos, e.g. bg_average, bg_noNorm, bg_rankInvariant, and noBg_rankInvariant, show an unsymmetrical relation between p-values for negative and positive log2 ratios. Especially noBg_rankInvariant exhibits a bias towards small negative log2 ratios for which the respective -log10 seem to be relatively high. In this region the fitted curve shows a very steep, linear course. Volcano plots generated for all other methods are similar to what would be expected. Still they differ in the variance of the p-values and in that some of the fitted curves show a flatter shape than others. This reflects the fact that some normalization methods generate a smaller variance than others, resulting in lower fold changes but more significant p-values. Ultimately, a method with a reasonable trade off between fold change and variance has to be chosen and cut-off parameters for interesting genes have to be defined accordingly. Volcano plots best reflecting the desired properties in the context of our experiment were generated by noBg_log_quantile, noBg_log_rsn, and noBg_vsn. They show the least scattering of values around the fitted curves, but they probably underestimate fold changes. Plots produced bye bg_forcePos_log_loss, bg_forcePos_log_quantile, bg_forcePos_log_rsn, bg_vst_quantile, bg_vst_rsn, noBg_cubicSpline, noBg_log_loess, noBg_vst_loess, noBg_vst_quantile, and noBg_vst_rsn also fulfil the above mentioned criteria, but show more scattering.All volcano plots based on rma background corrected data do not look very promising. The fitted curves are rather flat, i.e. even for high absolute logIn an optimally normalized experiment the residual standard deviation of fitted gene expression intensities should be low and independent of the expression levels, i.e. the variance over the different expression levels should be stable. This is prerequisite for many statistical methods, like for example linear model fitting and moderate t-statistics , that arnoBg_*) show a moderate or low variance in regions of no or hardly to measure expression. In contrast, nearly all of the background corrected methods (bg_*) result in high and, compared to the other methods, instable variance in the range of low intensity values. Extreme examples especially are rma background corrections and bg_vsn normalization procedures. An exception constitute methods using background normalization in conjunction with variance-stabilizing transformation (bg_vst_*) which in contrast to other procedures using background corrections perform especially well.As displayed in Figure bg_vst_loess, bg_vst_quantile, bg_vst_rsn, noBg_vst_loess, noBg_vst_quantile, and noBg_vst_rsn.Methods which perform best with respect to variance stabilization across all expression levels are bg_rma_log_loess, bg_rma_log_quantile, bg_rma_log_rsn, and bg_vsn, show high variance especially in the range of lower expression. Plots generated based on these procedures exhibit high variability between replicates. Some of the methods like for example bg_noNorm, bg_vst_loess, and noBg_average lead to asymmetric scatterplots indicating a bias in the expression values and a higher variability between replicates. Methods that perform well in stabilizing the variance across different expression levels, for example bg_vst_quantile, bg_vst_rsn, noBg_vst_loess, noBg_vst_quantile, and noBg_vst_rsn, could also be confirmed by the scatterplots. Additionally to those, noBg_cubicSpline and noBg_rankInvariant exhibit symmetric scatterplots with a very low degree of variance between replicates.Scatterplots are an easy and straightforward visualisation tool for judging the comparability of replicates. They clearly show whether high variances are to be expected and, if this is the case, in which range of the expression data. Figure bg_rankInvariant performs best with an AUC of 0.9102, whilst bg_vst_loess performs worst with an AUC of 0.8403.In order to compensate for missing spike-in and dilution data a pseudo-ROC approach  mimickinqRT-PCR has been performed for mRNAs from eight genes that are known to be deregulated by TGF-β signalling to a varying degree . By this means, it is possible to compare the results of the normalization methods to values that reflect the real abundance of the respective mRNA in the cells. Thus, we are able to evaluate the accuracy of the different pre-processing methods with respect to their bias. To guarantee that the comparisons of the normalization methods are not biased towards certain intensities, the mRNAs used in qRT-PCR experiments were chosen such that the respective signals on the chips cover a broad range of expression intensities  show a lower correlation to the TaqMan results than expression intensities that are background corrected (bg_*). An exception in this regard are methods that are based on vst transformation (bg_vst_*). These three methods are amongst the six methods resulting in the lowest correlation coefficient values. Correlation coefficients exhibiting high values are delivered by methods introducing BeadStudio's background correction combined with either rma background correction and log2-transformation (bg_rma_log_*), cubic spline normalization (bg_cubicSpline), or variance stabilizing normalization (bg_vsn).Based on the different normalization procedures for the gene expression experiment and based on the qRT-PCR measurements  best fit the qRT-PCR results  tend to better reflect the real fold changes, i.e. show less bias. On the other hand, pre-processing without background normalization (noBg_*) leads to a more homogeneous variance. Accurately defined, constant experimental conditions across all experiments as well as their parallel conduction probably have lead to a relatively consistent background level across all samples. Since background correction could introduce additional variation, these could be the reasons why, for our data set, data that was not background normalized (noBg_*) in general provides better stabilization of variance than background normalized data (bg_*). Methods combining background normalization with vst (bg_vst_*) constitute an exception. Here, vst leads to a better stabilization of variance while introducing more bias. As vst estimates an offset for the background based on the data [noBg_vst_* and bg_vst_* pre-processing methods could lead to similar results.It is important to select appropriate pre-processing methods for a given data set based on the experimental setup used. On the one hand, if sample sizes of the different groups are relatively small, it is crucial to achieve a homogeneous variance for the groups. On the other hand, if sample sizes are large, variances can be estimated separately and one should focus on unbiased fold changes. Since the sample sizes for the current data set are rather small (three to four replicates per group), a stable variance is more important than an exact representation of the fold change. In general, the data should be normalized without too much reducing real variations. Figure the data , noBg_vsnoBg_log_quantile and noBg_log_rsn. Although log2-transformation in combination with quantile normalization has been approved as performing relatively well by Du et al. [et al. [rsn). In addition to our measures it was selected because rsn is aiming at combining the positive effects of quantile normalization, i.e. preservation of the rank order, and spline interpolation, i.e. continuous mapping of the values, but at the same time circumventing their drawbacks, i.e. discontinuous mapping of intensity values and no rank preservation, respectively [vst as recommended by Dunning et al. [et al.[vst with rsn as successfully used by Du et al. [vst compared to three to four replicates in our setup) or the use of a newer Illumina chip technology, namely HumanHT-12 v3 chips, in our experiment. vst has been validated based on a pre-released version of the HumanRef-8 v1 Expression BeadChip that contained 19 (25% quantile) to 30 (75% quantile) beads per probe. On the HumanHT-12 v3 chips an average of only 15 beads per probe is available. Since vst makes use of those technical replicates, this could lead to a slightly worse performance on the new chip generation. In general, vst still performs well in stabilizing the variance but is outperformed by noBg_log_quantile, noBg_log_rsn, and noBg_vsn in reflecting the results measured by qRT-PCR. When utilising BeadStudio normalizations, in accordance with Dunning et al. [noBg_cubicSpline). As displayed in Figure noBg_cubicSpline outperforms all other BeadStudio normalization methods.One has to keep in mind that, based on the individual analyses, there are several methods resulting in nearly equal quality. Therefore, it is not possible to give a well-defined rationale for using only one specific method. After excluding the methods that clearly violate the imposed criteria, the decision is still subjective. It, for example, depends on whether one would like to account for a good estimate of fold changes or a small and homogeneous variance. Finally the decision remains based on experience; yet, with the analyses and criteria described here, we provide a recommendation on how to pre-select appropriate methods. Since, for our data set, we intended to achieve a low and homogeneous variance, we provided more and to a certain degree overlapping statistics investigating variance. In case the focus is on a good estimate of the fold change, the researcher should higher account for statistics investigating this measure. Correlation to qRT-PCR or slope and intercept of the regression between qRT-PCR and gene expression fold changes are examples of analyses that could be of higher interest in this context. Focusing on variance, best suited for the data set analysed here are u et al.  and Dunn [et al. ,10, we dectively ,23. Surp. [et al.,17,23 anu et al.  used forSpike-in or dilution data is frequently used for evaluating different normalization methods ,7-10. If2 ratios obtained from normalized data. In conjunction with the measures for the variability of the data finally the basis for weighing well measured changes versus low and homogeneous variance is delivered and by this means selecting an appropriate normalization method is possible.In summary, we provide statistical measures based on which researchers can decide on the best suited pre-processing scenario for their own experimental design. If no spike-in data is available, we recommend conducting qRT-PCR for selected, representative transcripts. Thereby, it is possible to estimate the bias of logHaCaT cells were cultured under standard conditions (REF). Cells were seeded in 96-well (ELISA) or in 24-well (RNA expression profiling) plates and grown overnight to a confluence of approximately 70%. Cells were starved for 3 hours in DMEM containing no FCS and subsequently stimulated with 5 ng/ml of TGF-β1 (R&D Systems) or left unstimulated as controls for 2, 4, and 12 hours.RNA isolation was carried out using a MagMAX™ Express-96 Magnetic Particle Processor and the MagMAX™-96 Total RNA Isolation Kit according to the manufacturer's protocol. Total RNA concentration was quantified by fluorescence measurement using SYBR Green II (Invitrogen) and a Synergy HT reader (BioTek) as previously described . The RNAIllumina TotalPrep RNA Amplification Kit (Ambion) was used to transcribe 200 ng toRNA according to the manufacture's recommendation. A total of 700 ng of cRNA was hybridized at 58°C for 16 hours to the Illumina HumanHT-12 v3 Expression BeadChips (Illumina). BeadChips were scanned using an Illumina BeadArray Reader and the Bead Scan Software (Illumina). Data is publicly available in ArrayExpress   was conducted for eight genes  known to be deregulated at at least one time point by TGF-β stimulation.mRNA expression levels of the eight genes were determined by qRT-PCR analysis using a 7900HT Fast Real-Time PCR System (Applied Biosystems) and the Universal ProbeLibrary System (Roche). Gene specific forward and reverse primer sequences were designed using the Universal Probe Library Assay Design Center (Roche). Total RNA was transcribed into cDNA using the High Capacity cDNA Reverse Transcription Kit (Applied Biosystems) according to the manufacture's instructions. qRT-PCR is carried out in a final volume of 12 μl in three replicates for each cDNA sample. Levels of RNA polymerase II were used for normalization of the data. ΔΔCT method was used to relatively quantify mRNA levels of treated samples compared to untreated controls  2.7.0 ,28 in cobg_*). Cubic Spline, Rank Invariant, and Average methods were used for normalization . Expres*_noNorm data has been log2-transformed using the lumiT function. Thus, for background corrected data forcePos has automatically been conducted.forcePos) [rma background adjustment (bgAdjust.affy) [lumiB function of the lumi package were used to transform negative values which can result from BeadStudio background normalization to positive scale to be able to log2-transform the expression values. Background correction referred to as noBg implies that the background normalization has not been applied.forcePositive (orcePos)  or rma bst.affy)  availabl2-tranformation (log) or variance-stabilizing transformation (vst) [For transforming the data, a simple logon (vst)  was usedquantile) [rsn) [loess) [vsn) [vst as well as vsn can handle negative values in the data. Thus, neither forcePos nor rma was applied as pre-processing for any of those two methods to not unnecessarily modify the values in artificial ways.Data was normalized using quantile normalization (uantile) , robust e) [rsn) , local r [loess) , or varis) [vsn) . vst as All methods used are implemented in the R packages affy , vsn 1818, or lusmoothScatter function as implemented within the Bioconductor package geneplotter [In the following, the statistical measures used are briefly summarized. Unless otherwise noted, all statistical calculations were performed using R. A small R-package to conduct and reproduce the described analyses is available from the authors upon request. For the visualizations displayed in Figures eplotter  has beenOne aim of normalization is to minimize, for each gene, the within group variability while maximizing the between group variability also referred to as mean sum of square withinand mean sum of square betweenrespectively.i the size of group i, ij the jth value in group i. The aim is to maximize 1 = n2 = n3 = 4, and Here, k represents, for a given gene, the number of groups, nOne of the main uses of expression arrays is the identification of genes that are differentially expressed under various experimental conditions. A typical identification rule filters genes with p-values and/or fold change exceeding a given threshold. Given a set of known true positives (TP) and false positives (FP), Receiver Operator Characteristic (ROC) curves offer a graphical representation of both specificity and sensitivity for such a detection rule. ROC curves are created by plotting the true positive rate (sensitivity) against false positive rate (1-specificity) obtained at each possible threshold value. Since we only know about TP (20 genes known to be deregulated by TGF-β), we made use of so-called pseudo-ROC curves . The TNs2 ratios, residual standard deviation, and p-values were calculated using linear models in combination with the moderated t-statistic as supplied by limma [The logby limma .princomp function as available in the basic R environment.To get an overall impression of how good of a fit of the fold change levels detected using the different normalization methods to the qRT-PCR results are, an orthogonal regression for the observations was performed using the The following naming conventions are used to refer to different normalization methods:<background correction>_<transformation>_<normalization>, where:   <background correction>={bg_forcePos, bg_rma, noBg},•    <transformation>={log, vst},•    <normalization>={loess, quantile, rsn}, and•    <transformation>_<normalization>={vsn}.• <background normalization>_<normalization>, where   <background normalization>={bg, noBg} and•    <normalization>={cubicSpline, rankInvariant, average}.• PB did the laboratory work, KF-C and RS conducted the statistical and bioinformatics analyses, CI, WH, BB and RE supported the statistical analyses, CI supervised the statistical analyses, DM and KQ supervised the project. RS and PB wrote the manuscript and all authors proofread the manuscript. All authors read and approved the final manuscript.10 against MSQbetween where MSQbetween ≤ 5-log. MSQs were calculated based on the gene expression measured for the three sample groups analyzed, namely untreated HaCaT cells after 2, 4, and 12 hours. Results obtained for the different pre-processing methods used are displayed. The blue line represents a loess-curve fitted to the values.Click here for filewithin (blue) and MSQbetween (red)Density plots of MSQ. MSQs were calculated based on the gene expression measured for the three sample groups analyzed, namely untreated HaCaT cells after 2, 4, and 12 hours. Results obtained for the different pre-processing methods used are displayed. The grey dashed line indicates the expected value for the MSQbetween of 1.33 based on 6, 6, and 7 as measurements for the group means of four replicates for three time points.Click here for fileVolcano plots. Log2 ratios and p-values for the comparison of untreated HaCaT cells at 4 hours compared to 2 hours, 12 hours compared to 2 hours, and 12 hours compared to 4 hours were calculated based on the gene expression measured. Displayed are the -log10 against log2 ratio for the respective comparisons and the different normalization methods used. The blue line represents a loess-curve fitted to the values.Click here for fileResidual standard deviation against minimum expression intensity. For each pre-processing method, standard deviation of the residuals observed for the regression fitted to the expression intensities are plotted against minimum expression intensity of each probe. The blue line represents a loess-curve fitted to the values.Click here for fileResidual standard deviation against mean expression intensity. For each pre-processing method, standard deviation of the residuals observed for the regression fitted to the expression intensities are plotted against mean expression intensity of each probe. The blue line represents a loess-curve fitted to the values.Click here for fileScatterplots between replicates. After application of different normalization methods, expression values for the replicates are plotted against each other. The orange line indicates the main diagonal.Click here for fileRanking of AUC values. AUC values as calculated for the pseudo-ROC analysis displayed in Figure Click here for fileResults of qRT-PCR. 2-ΔΔCt [. 2-ΔΔCt  values rClick here for file2 ratiosOrthogonal regression between qRT-PCR and normalization based log. Regression of log2 ratios based on different normalization methods (y-axis) against qRT-PCR log2 ratios (x-axis). Equations and the respective regression lines are displayed in red. The grey dashed line indicates the main diagonal.Click here for file


In addition, our approach works even on much smaller training data sets and is independent of the sample size of the test data, making it feasible to be applied on clinical studies.One important application of microarray in clinical settings is for constructing a diagnosis or prognosis model. Batch effects are a well-known obstacle in this type of applications. Recently, a prominent study was published on how batch effects removal techniques could potentially improve microarray prediction performance. However, the results were not very encouraging, as prediction performance did not always improve. In fact, in up to 20% of the cases, prediction accuracy was reduced. Furthermore, it was stated in the paper that the techniques studied require sufficiently large sample sizes in both batches (train and test) to be effective, which is not a realistic situation especially in clinical settings. In this paper, we propose a different approach, which is able to overcome limitations faced by conventional methods. Our approach uses ranking value of microarray data and a bagging ensemble classifier with sequential hypothesis testing to dynamically determine the number of classifiers required in the ensemble. Using similar datasets to those in the original study, we showed that in only one case ( Noise has a negative connotation in the classical view of biology. Therefore, one often attempts to remove "noise" from data using various statistical methods before any downstream analysis. However, there are two different types of noise in biological data, experimental noise and inherent cell variation. Distinguishing experimental noise from natural fluctuation due to inherent cell variation is a daunting task, and attempts to de-noise data often remove meaningful cell variation as well. Therefore, in this work, we take a different approach of embracing noise instead.Inherent cell variations could arise from intrinsic and extrinsic sources . IntrinsExperimental noise in gene expression measurement data mainly contains two forms of experimental errors: measurement errors and batch effects. Measurements errors in gene expression microarrays are studied by the MicroArray Quality Control (MAQC) project, a large-scale study led by FDA scientists involving 137 participants from 51 organizations, where they showed that the median coefficient of variation of replicates is between 5% and 15% .An important application of microarrays in clinical settings is to construct a predictive model for diagnosis or prognosis purposes. To do so, we need to overcome the various types of noises mentioned above, especially batch effects . RecentlMost batch effects removal algorithms try to accurately estimate the batch effects before removing them, which is why large sample sizes are required for each batch and a balanced class ratio is often desired. In this paper, we attack the problem from a different angle. Specifically, we propose a computational approach that increases cross-batch microarray prediction accuracy that mitigates batch effects without explicitly estimating and removing them. Our proposed approach uses the following two main ideas. Firstly, it is well known that while batch effects affect the absolute values of the gene expression measured, they often do not affect the relative ranking of the gene ordered by their expression values . Thus, iFour data sets from the MAQC project are used in this paper Table . Three aThe Hamner Institutes for Health Sciences  provided data set A. The objective of the study was to apply gene expression data from the lungs of mice exposed to a 13-week treatment of chemicals to predict increased lung tumor incidence in the two-year rodent cancer bioassays of the National Toxicology Program. Results of this study may be used to create a more efficient and economical approach for evaluating the carcinogenic activity of chemicals. A total of 70 mice were analyzed in the first phase and used as the training set. An additional 88 mice were later collected and analyzed, and subsequently used as the validation set.The University of Texas M. D. Anderson Cancer Center  generated data set D. 230 stages I-III breast cancers gene expression samples were collected from newly diagnosed breast cancers before any therapy. Specimens were collected sequentially between 2000 and 2008 during a prospective pharmacogenomics marker discovery study. Patients received 6 months of preoperative chemotherapy followed by surgical resection of the cancer. Response to preoperative chemotherapy was categorized either as a pathological complete response (pCR), which indicates no residual invasive cancer in the breast or lymph nodes, or residual invasive cancer (RD). Gene expression profiling was performed in multiple batches using Affymetrix U133A microarrays. The first 130 collected samples were assigned as the training set, whereas the next 100 samples were used as the validation set.The Myeloma Institute for Research and Therapy at the University of Arkansas for Medical Sciences  contributed data sets F and I. Highly purified bone marrow plasma cells were collected from patients with newly diagnosed multiple myeloma followed by gene expression profiling of these cells. The training set consisted of 340 cases enrolled on total therapy 2 (TT2) and the validation set comprised 214 patients enrolled in total therapy 3 (TT3). Dichotomized overall survival (OS) and event-free survival (EFS) were determined based on a two-year milestone cutoff.As all the data sets above from the MAQC project are cancer-related, we have therefore gathered an additional non-cancer-related data set from a different source  to show As previously mentioned, we are proposing an entirely different approach towards overcoming batch effects. This computational approach is inspired by two articles in the field of biology, and is further enhanced with idea from our previous work on sequential hypothesis testing .First, we propose using rank values instead of absolute values of gene expression microarray data. This is inspired by the FDA-led Microarray Quality Control (MAQC) Consortium project , where oIn another article , a team S of m samples is given. Suppose x of the samples are "bad"  and y = (m - x) of the samples are "good" . Let q = x/m and p = (1 - q) = (m - x)/m. Let B be a bag of m samples randomly drawn with repetitions from S. The probability of B having k "bad" samples is given by the binomial distribution PB(k) = (mCk)(m-kp)(kq), where mCk means "m choose k". Then the probability of B having fewer "bad" samples than S is given by B having more "bad" samples than S is given by p >q, and thus 1 > 2q, the skew is positive. This means that, in general, the bulk of the distribution falls to the left of the mean x = mq. Thus, PB(<x) > PB(>x), as shown in Figure Suppose a set Therefore, we have shown that training clones produced using bootstrapping technique are likely to be enriched with more clean samples than the original training data.B1, B2, ...,n Bbe n bags of m samples randomly drawn with repetitions from S. Let H be the collection of bags among B1, B2, ...,n Bcontaining more "good" samples than S. Let C(B1) and C(B2) be the classifiers trained on B1 and B2 respectively. It is reasonable to postulate (*) that C(B1) would have a better accuracy than C(B2) if B1 has more "good" samples than B2. Let H' be the collection of bags among B1,B2, ...,n Bcontaining more "bad" samples than S. Let H" be the collection of bags among B1,B2, ...,n Bcontaining the same number of "good" and "bad" samples as S. Based on the postulate (*), h = |H| bags give rise to better-performing classifiers than C(S), while h' = |H' | bags give rise to poorer-performing classifiers, and the remaining h" = |H" | bags give rise to equal-performing classifiers. We know that as n tends to infinity, h/n tends to PB(<x), h'/n tends to PB(>x), and h"/n tends to PB(x). It follows that h >h', when p >q.We further show that an ensemble classifier built using these training clones has better performance. Let S, which is a reasonable assumption for any decent training datasets.This shows that an ensemble classifier built from this collection of bags--which is called a bagging classifier--improves prediction accuracy by embracing  noisy samples, as long as there are many more "good" samples than "bad" samples in the original training set n needs to be sufficiently large to ensure h ≥ h', n is typically determined a priori and arbitrarily [n required for each test instance. In our previous work [Pθ≥{ψ} where θ represents the threshold probability and ψ represents the property. For instance, P≥0.8{X1>5} checks whether the given stochastic model would have variable X1 >5 in ≥ 80% of the cases. Essentially, this procedure draws samples until it can assert or reject a probabilistic formula with statistical guarantees on the error rates.However, in the original flavor of bagging, while itrarily . Here, witrarily , which tous work , we devei Tbe the ith test instance and P(C(nB), iT) be the Boolean prediction on i Tof the classifier trained using nB, which will return True if i Tis predicted to be of the positive class label and False if i Tis predicted to be of the negative class label. We can then formulate the probabilistic formula as P≥0.5{P(C(nB), iT) = True}. That is, given a test instance i Tand a training set S, and a large number of classifiers trained from bags of m samples randomly drawn with repetitions from S. Would more than 50% of these classifiers predict i Tto be of the positive class label? By stating the problem in this format, n is dynamically determined by the sequential hypothesis testing procedure and is minimum for each test instance with statistical guarantees on the error rates. This improves computational efficiency and maintains prediction accuracy over standard bagging. Furthermore, this removes the need to a priori and arbitrarily fixing n. For the purpose of comparison with standard bagging (with 10 and 100 bootstrap replicates), we set the parameters for the OSM sequential hypothesis algorithm as follows. The maximum value of n is set to 100 and the guaranteed false positive and false negative error rates are both set to 10-4.Let In summary, we propose using ranking value of microarray data and bagging with sequential hypothesis testing to dynamically determine the number of classifiers required. Finally, the average of these classifiers scores is taken as the final prediction score for a particular test instance.In this work, our main objective is to improve cross-batch prediction accuracy. Therefore, we will be using it as our performance measurement. The primary performance metric used will be area under the ROC curve (AUC) as it has the advantage of evaluating performance across the full range of sensitivity and specificity. A prediction model will be built using the training set and evaluated using the validation set (forward prediction) and vice versa (backward prediction).t-test and Wilcoxon Rank Sum test as they represent parametric and nonparametric approaches respectively. As for classification methods, we have chosen support vector machine (SMO with buildLogisticsModel set to True), K nearest neighbors (K = 5) and the popular tree algorithm, C4.5 . All clUsing the above-mentioned data sets, feature selection algorithms and classification methods, we measure the difference in AUC before and after our proposed algorithms in each of the possible permutations. There are a total of 9 different data sets, 3 from each data set A, D and F. Data set I is not used to measure performance improvement since it is a negative control; it is used instead to ensure arbitrary improvement is not seen. Together with two different prediction directions (forward and backward), two different feature selection algorithm (t-test and Wilcoxon Rank Sum test) and three different classification methods , there are a total of 108 (9x2x2x3) different possible scenarios.>0.05 in >60% of the cases with only one case (<2%) having reduced AUC of exceeding -0.05. Combining the observations of Figure The main objective of this work is to improve cross-batch prediction performance. In Figure ne case <% having ne case <% having Another important consideration in building prediction models for clinical usage is the required sample size of training and test sets to properly deploy it. As the MAQC project is a large-scale study, its data sets are larger than usual. We did random subset sampling to reduce the number of samples available to us to as low as 25% of the original data, during the training phase, to mimic the low sample size in clinical settings. Despite the reduction in training samples, our algorithm still maintained its improvements with median AUC improvements well above 0.05 (Figure As the PCA plots of Figure Finally, one critical issue highlighted by the MAQC project  is regarIn addition to cancer-related data sets from MAQC projects, we have also obtained a DMD data set from a different source  to demonOvercoming batch effect is an important step before the deployment of diagnostic or prognostic model based on gene expression data in clinical settings. Numerous algorithms have been proposed in an attempt to solve this widespread and critical problem in high-throughput experiments -21. HoweIn this work, we approached the batch effects problem from a different angle. We proposed a computational algorithm that attempts to embrace noise instead of estimating and removing it. By simply employing the ranking of values instead of using the absolute values of data, we were already able to show noticeable improvements. Combining this with bagging and a sequential hypothesis-testing algorithm; we were able to achieve a significant increase in cross-batch prediction performance over a wide range of training data sample size and severity of batch effects. It is important to note that our approach does not face the same limitations as conventional batch effects removal methods; thus making it appealing for use in practical applications.Feature selection algorithms considered in this work use only generic statistical tests that look at one gene at a time. However, more recent feature selection algorithms for gene expression data are increasingly focused on using prior biological information to group genes and perform statistical tests on these group of genes instead of individual genes ,13. The The authors declare that they have no competing interests.Both authors contributed equally to conceiving the proposed method. C.H.K carried out the programming, prepared and ran the experiments, and drafted the manuscript. L.W designed the experiments and played a supervision role for the study. Both authors read and approved the final manuscript.


Illumina whole-genome expression BeadArrays are a popular choice in gene profiling studies. Aside from the vendor-provided software tools for analyzing BeadArray expression data (GenomeStudio/BeadStudio), there exists a comprehensive set of open-source analysis tools in the Bioconductor project, many of which have been tailored to exploit the unique properties of this platform. In this article, we explore a number of these software packages and demonstrate how to perform a complete analysis of BeadArray data in various formats. The key steps of importing data, performing quality assessments, preprocessing, and annotation in the common setting of assessing differential expression in designed experiments will be covered. Microarrays are a standard laboratory technique for high-throughput gene expression profiling in genomics research. The BeadArray microarray platform from Illumina Inc.  consists of an array of randomly packed beads, each bead bearing many copies of a particular 50-mer oligonucleotide sequence (the “probe”). Each BeadArray contains a collection of probes designed to interrogate the majority of protein-coding transcripts in a given organism  along with a large set of both positive and negative control probes. Due to the random sampling of beads during the manufacturing process, the number and arrangement of replicate beads varies from array to array.Multiple BeadArrays are grouped together to form a BeadChip, with gene expression products configured to have six (WG-6), eight (Ref-8), or 12 (HT-12) samples per chip. This format allows samples to be processed in parallel with benefits for experimental design, a key factor in the experimental workflow The experimental process for measuring transcript levels in a sample of interest involves labelling RNA and hybridizing this material to the probes on a BeadArray. The scanned intensities from these probes provide a snapshot of transcript abundance in a particular sample. Comparing the intensities obtained from different RNA species can provide researchers with insight into the molecular pathways regulating the system under investigation. There is a rich literature on the analysis of gene expression microarrays  have been made available, then it should be clear that starting from the TIFF images gives the greatest amount of control over the steps being performed at each stage. In most situations, the default processing methods employed by Illumina to extract intensities from the TIFFs and summarize these values within each sample produce good intensity estimates. Whether these processing steps are carried out in R (see vignette), or using vendor-provided software, is obviously up to the user; however, the ability to perform the entire analysis in a platform-independent, reproducible, and flexible manner in R will be appealing to many computational biologists. In addition, image registration issues Irrespective of whether raw, bead-level, or summarized data are being analyzed, the first opportunity to assess the quality of an experiment occurs as the arrays are being scanned, and without the need for specialized software. The scanner produces a text file that contains various signal-based array quality measures. As an example, beadarray package BeadDataPackR package To obtain raw or bead-level data, modifications to the default scanning settings in BeadScan or iScan are required. Currently, the beadarray can extract background, foreground, and total intensities to the user's specifications. In particular, using a more robust measure of the local background intensity (median) has been shown to be beneficial The import of raw data is handled using the readIllumina function. The availability of TIFF images depends on scanner settings (jpegs are provided by default), and where present beadarray employ this transformation by default. The first within-sample quality plots that one can produce are overall image plots of the array surface . The standard summary statistics to use in beadarray are the mean and standard error of the log-intensity (Illumina's standard statistics to report are the mean and standard error of the raw intensity). Note that the standard error is important for Illumina BeadArrays, as the random design means that we will have differing levels of confidence from one measure of intensity to the next. Besides variations of the standard Illumina outlier removal method offered by beadarray, other robust summary options are possible as described in Kohl and Deigner (2010) RobLoxBioC package.After image processing, a key step is the reduction of raw data  to summary data  in order that we might apply the methods detailed in the next section. beadarraylumilimmaThe most common entry point for the computational biologist is to begin with summarized data obtained from the gene expression module of the BeadStudio/GenomeStudio software. These PC-based programs provide a convenient graphical user interface to import and process BeadArray data from the proprietary format idat files output by Illumina's scanning software. Data are exported from this application in tab-delimited files  with each row giving the summary information for a particular probe, and different columns for each sample. We recommend exporting raw summary values  at the probe level (“probe profiles”) rather than at the gene level (“gene profiles”) for both regular and control probes to avoid combining probes targeting different transcripts of the same gene in an undesirable manner. Such files can be imported and processed in the R software environment using a range of tools that include GEOqueryArrayExpressAnother potential source of summarized data are public repositories such as Gene Expression Omnibus (GEO) arrayQualityMetricsbeadarray and identify potential outlier arrays. Boxplots are commonly used to assess the dynamic range from each sample and look for unusual signal distributions , between-array normalization, and transformation. Two popular methods that implement these steps for Illumina data are neqc and vst from the For background correction, the GenomeStudio option of subtracting the average of the negative controls on an array has been shown on several occasions to be flawed Negative controls are also useful for estimating the proportion of probes that are expressed in a given sample lumi) have emerged and are increasing in popularity. Strip-level processing, which separates probes depending on physical location and normalizes strips containing the same probes between samples, can also be beneficial for older BeadChip versions For normalization, between-array quantile is the method most frequently applied to Illumina data both in the literature limma  Next, relationships between a collection of samples can be assessed via multidimensional scaling MDS,  or princOnce data are preprocessed into a normalized “expression matrix” format used throughout Bioconductor, a wide variety of analyses can take place such as clustering, assessing differential expression, classification, and pathway analysis.limma package for assessing differential expression limma package also allows for observations in a linear model to be weighted according to the confidence in which we hold them. For Illumina BeadArrays, we might naturally want to weight observations by the inverse of the squared standard error (so that observations about which we are more certain are given greater weight), as the standard error should be a function both of the array quality and the number of copies of that type of bead. However, obtaining an accurate measure of the standard error can be problematic. Even if we start out with one, steps such as outlier removal, trimming, background correction at the summary level, and normalization will transform the mean and leave the standard error lacking validity unless it is sympathetically transformed. Thus, it is tempting to assume that our transformation  has removed any mean-variance relationship in the data, in which case the number of beads can be used as a weight to account for technical variation that may arise in an experiment. This ignores biological variation between different arrays, and so we should really use a weight consisting of the number of beads contributing to the observation adjusted by an array multiplier that gives a measure of the reliability of the array from which the observation comes. Array-specific weights have been shown to improve power to detect differential expression Throughout the vignette t-statistics and F-statistics (when multiple contrasts are present), and the resulting p-values are generally used to rank probes in terms of their evidence for differential expression after adjusting for multiple testing.Having fitted our weighted linear model, we then set up contrasts between RNA conditions and proceed to estimate between-sample differences of biological interest. Empirical Bayes shrinkage of the probe-wise variances is then applied to ensure that inference is reliable and stable, even when the number of replicate samples is small By following the steps in the previous section, the researcher may be presented with a list of hundreds if not thousands of differentially expressed probes that are named according to their manufacturer-assigned IDs. At the very least, these must be translated into gene symbols that the researcher can recognize, or into functional pathways that can provide insight into the biological question being investigated.lumiHumanAll.db) or per-chip  basis. The organism-specific packages use the nuIDs from Du et al. (2007) The Bioconductor project provides infrastructure for mapping between microarray probes and functional genomic annotation to be used in downstream analyses. For Illumina chips, these packages are maintained on a per-organism  topGO or GOstats and their associated annotation packages (GO.db and KEGG.db), as can gene set enrichment analysis using the GSEAlm package. In limma, both self-contained gene set testing (using the roast function There are many other analysis tools available from R/Bioconductor that can be used for downstream analysis of Illumina microarray data. For example, gene ontology/pathway enrichment analysis can be performed with BeadArrayUseCases package. These Bioconductor tools expand the set of analysis options offered in the vendor-provided GenomeStudio/BeadStudio software, and are continually being developed to accommodate new applications of BeadArray technology, such as methlyation assays.We have highlighted a number of specially tailored tools and modelling approaches that are available in Bioconductor for the analysis of Illumina gene expression data sets in various formats. A summary of the methods that we currently recommend for Illumina expression analysis are listed in The open-source Bioconductor platform also presents researchers with a choice of operating system for their analysis and a means to write analysis scripts and generate reports based on them using Sweave


Adaptation to stress signals in the tumor microenvironment is a crucial step towards carcinogenic phenotype. The adaptive alterations attained by cells to withstand different types of insults are collectively referred to as the stress phenotypes of cancers. In this manuscript we explore the interrelation of different stress phenotypes in multiple cancer types and ask if these phenotypes could be used to explain prognostic differences among tumor samples.a priori phenotypic information about samples, SLEA calculates an enrichment score per sample per gene set using z-test. This score is used to determine the relative importance of the corresponding pathway or module in different patient groups.We propose a new approach based on enrichment analysis at the level of samples  in expression profiling datasets. Without using Our analysis shows that tumors significantly upregulating genes related to chromosome instability strongly correlate with worse prognosis in breast cancer. Moreover, in multiple tumor types, these tumors upregulate a senescence-bypass transcriptional program and exhibit similar stress phenotypes.Using SLEA we are able to find relationships between stress phenotype pathways across multiple cancer types. Moreover we show that SLEA enables the identification of gene sets in correlation with clinical characteristics such as survival, as well as the identification of biological pathways/processes that underlie the pathology of different cancer subgroups. In breast cancer, enrichment with the CIN signature is strongly related to bad prognosis even among samples with wild-type p53, indicating that indeed the predictive power of this signature is independent of p53 mutation  . As seen sample) ), the CINext we performed EA with all Gene Ontology biological process terms in order to identify the biological properties characterizing CIN-positive tumors. These tumors significantly downregulate genes related to processes such as 'cell communication' and 'wound healing'   [de novo synthesis of sterols, isoprenoids, and fatty acids [HIF1A [HIF1A along with its target vascular endothelial growth factor /Akt pathway and its downstream mammalian target of rapamycin (mTOR) signaling: 'genes deregulated in PI3K-hyper-activated, hormone resistant cells'  (modulesSC1-ko') . Figure ty acids . We usedty acids , to checty acids ). mTORC1s [HIF1A . In agreIn order to determine if we can see similar patterns in other types of cancers, we performed the same EAs in tumor datasets comprising different types of cancer : brain, lung, ovary, bladder and colon. In all the datasets the enrichment of the CIN signature divided the samples into two . There EA is an effective way to analyze the statistically significant gene sets obtained using high-throughput functional genomics data. In this work, we propose an alternative approach for the analysis of tumor genomics data to detect clinically relevant patient subgroups. Instead of finding genes differentially expressed between two groups, we identify differentially enriched modules by performing sample-level EA (SLEA). Our method does not require information related to phenotypic classification of samples and can directly take gene sets as input. Moreover, by comparing enrichment results with available clinical information, SLEA enables the understanding of pathways/processes that underlie the clinical phenotypes such as survival. We applied our methodology to test the prognostic power of a gene signature related to chromosomal instability and to study the prevalence of stress phenotypes in different patient subgroups defined by the expression of this gene signature. The tumors significantly upregulating this signature were strongly correlated with worse prognosis in the three breast cancer datasets studied, but not in other tumor types. In all cancer types, however, the tumors with positive enrichment for this gene signature displayed a transcriptional program pointing to evasion of the senescence barrier and particular stress phenotypes, indicating strong interdependencies between these different pathways and therapeutic vulnerabilities for the tumor.CIN: chromosomal instability; GEO: Gene Expression Omnibus; mTOR: mammalian target of rapamycin; PI3K: Phosphoinositide 3-kinase; SLEA: sample-level enrichment analysis.The authors declare that they have no competing interests.GG and NL designed the study, interpreted the data and drafted the manuscript. GG performed all statistical and other data analysis. All authors have read and approved the manuscript for publication.Supplementary figures. Supplementary figure 1: result of overlap analysis of the modules used. Heat map of the Jaccard indices for overlap analysis among modules used in this study. Pink cells indicate high overlap (Jaccard index ≥ 0.6) while light blue shows no overlap. Supplementary figure 2: comparison of z-score mean and z-score median. Scatter plot of z-scores obtained using mean and median as the test statistic in EA of the Ivshina et al. [et al. [et al. [a et al.  dataset  [et al. ), TCGA-O [et al. ) and GSE [et al. ). Each pClick here for file


Because of technological limitations, the primer and amplification biases in targeted sequencing of 16S rRNA genes have veiled the true microbial diversity underlying environmental samples. However, the protocol of metagenomic shotgun sequencing provides 16S rRNA gene fragment data with natural immunity against the biases raised during priming and thus the potential of uncovering the true structure of microbial community by giving more accurate predictions of operational taxonomic units (OTUs). Nonetheless, the lack of statistically rigorous comparison between 16S rRNA gene fragments and other data types makes it difficult to interpret previously reported results using 16S rRNA gene fragments. Therefore, in the present work, we established a standard analysis pipeline that would help confirm if the differences in the data are true or are just due to potential technical bias. This pipeline is built by using simulated data to find optimal mapping and OTU prediction methods. The comparison between simulated datasets revealed a relationship between 16S rRNA gene fragments and full-length 16S rRNA sequences that a 16S rRNA gene fragment having a length >150 bp provides the same accuracy as a full-length 16S rRNA sequence using our proposed pipeline, which could serve as a good starting point for experimental design and making the comparison between 16S rRNA gene fragment-based and targeted 16S rRNA sequencing-based surveys possible. Metagenomics is a new area of study which has undergone rapid developement as our ability to sequence microorganisms in given environmental samples without culturing them got improved. Consequently, research in this area has provided important insights into a variety of topics, ranging from the relationship between bacteria and human health On the other hand, metagenomic shotgun sequencing data is naturally immune to primer bias. Global Ocean Survey (GOS) is one of the first and still on-going large-scale research projects using this type of data Therefore, in this article, we focus on establishing such an analysis pipeline by conducting data analysis through 16S rRNA gene databases using simulated data. Since individual variable/hyper-variable regions of 16S rRNA gene have inconsistent and more complicated features We first generated simulated 16S rRNA gene fragments datasets with various read length and read numbers from a set of known full-length 16S rRNA gene sequences, thus the underlying taxonomic profile (the “ground truth”) of the generated datasets are known. Then we mapped these simulated 16S rRNA fragments to annotated full-length 16S rRNA sequence databases using BLAST, and used the BLAST mapping results to generate predicted taxonomic profile for each dataset. By tuning above mapping procedure and testing different type of methods of generating OTU predictions using the mapping results, we decided that nearest-neighbor approach (see Methods) is the optimal approach in terms of three performance metrics: sensitivity, specificity and Bray-Curtis dissimilarity.L are able to generate taxonomic profile prediction consistent with that of Y almost full-length 16S rRNA sequences when there is no bias in the data?” To answer this question, we first generated a set of simulated full-length 16S rRNA sequence datasets with various read numbers by sub-sampling from the set of full-length sequences we used to generate simulated 16S rRNA gene fragments datasets above. Then taxonomic profile predictions were generated for both simulated 16S rRNA gene fragments datasets and simulated full-length datasets using the previously-determined optimal approach. By comparing the sensitivity, specificity and Bray-Curtis dissimilarity achieved by these datasets using Mann-Whitney U-test (see Methods), we are able to draw conclusion that 16S rRNA gene fragments, with an average read length of at least 150 bp, could provide the same level of resolution as full-length 16S rRNA gene sequences in terms of α-diversity, Bray-Curtis dissimilarity, sensitivity and specificity.The second step is to establish a relationship between 16S rRNA gene fragments and full-length 16S rRNA sequences. To be more specific, we want to answer the question “How many 16S rRNA gene fragments with average length Our established analysis pipeline that can test for “true difference” between a 16S rRNA gene fragment dataset and a targeted-sequencing dataset sharing the same underlying microbial community uses above conclusion reversely: First, we check if the 16S rRNA gene fragments are >150 bp in length. Then we perform necessary sub-sampling of the two datasets to generate similar-sized sub-datasets. Finally we process these sub-datasets using aforementioned optimal approach and make comparison between them using a likelihood ratio test (see Methods). These sub-datasets, according to our analysis, should generate OTU abundance estimations that are statistically similar to each other. Thus if statistically significant difference are detected, it can be concluded that there are bias in the data used.We used this pipeline to analyze a set of gut data taken from the Human Microbiome Project https://code.google.com/p/shotgun-metagenomics-analysis-framework/.The C++ program for the OTU analysis using metagenomic shotgun sequencing data proposed in this study is available at Two data types were used in the study: simulated data and real data.Two groups of simulated data, as shown in The real data consisted of nearly full-length 16S rRNA sequences, V2 amplicon of 16S rRNA gene, V6 amplicon of 16S rRNA gene, and 454 whole genome shotgun sequencing data introduced by Turnbaugh et al. uclustTwo databases were used to map metagenomic shotgun sequencing data and extract taxonomic information. The first dataset, referred to as “raw database”, consisted of 1428381 high-quality 16S rRNA gene sequences annotated in RDP. The second dataset, referred to as “redundancy-reduced database”, was built by clustering the 1428381 sequences at 3% distance threshold using a program called BLASTN was used to map the data to the databases. We used very stringent E-value and similarity cut-offs, as shown in By mapping metagenomic shotgun sequencing data to a large 16S gene database, we are able to extract out reads which are potential 16S rRNA gene fragments. However, a significant portion of the 16S rRNA gene fragments will get multiple BLAST hits in closely related taxonomic groups, even under very stringent criteria, i.e., similarity cut-off >99% and mapping length >95%, based on the presence of highly conserved regions in 16S rRNA genes, which poses difficulty for accurate OTU abundance estimation. Methods have been proposed to handle multiple hits. One popular approach, which is used in MEGAN Each taxonomic group consists of multiple 16S rRNA gene sequences to which a read may have multiple hits. The nearest neighbor approach assigns each read to the taxonomic group with the highest BLAST score, or, if two or more groups have the same score, to all of those with equal weights summing up to 1. The abundance of a taxonomic group is the number of the weights of its reads.This straightforward method could have many different variations by giving different definitions for “nearest neighbor”. For example, we may consider a read belongs to a taxonomic group if its BLAST hit score in this group is some margin better than hit scores in any other groups. However, preliminary results showed that using above mentioned method, on average >90% of the reads can be assigned correctly, and there is no way to correctly assign the rest 10% by simply looking at their sequence content since they are 100% identical to many taxonomic groups at even phylum-level. Thus we used above approach as a representative of the nearest neighbor approach family.Since probabilistic approach gives chance for a read to be assigned to any taxonomic group it gets a BLAST hit in, these approaches are expected to generate results with higher sensitivity. Thus we tested an MLE method for OTU prediction. The setting we used is a basic EM algorithm for inferring component abundance of a mixture model.i, 1≤ i ≤ n, has k BLAST-hits in taxonomic group j, 1≤ j ≤ m. For the m-th hit, let ml be the length of the local alignment reported by the BLAST, and let ms be the percentage of identities in the local alignment. Then the likelihood that read i is generated from the taxonomic group j isl is the length of the reference full-length 16S rRNA gene sequence.The likelihood model is described as follows: Assume that read jπ be the abundance of a taxonomic group j in a sample, and let i = jZ be the latent variable indicating that read i is assigned to a specific taxonomic group j. Thus for all the N reads and M taxonomic groups, the full likelihood function is then:Let A log transformation gives:jπ}, we impute {i = j)I as “sets”, n selected groups, it carries 1/n weight in each group. Thus the abundance of a taxonomic group is estimated as the total weights of its reads.To estimate the abundance of the selected taxonomic groups, we weight each read by the following rule: if a read hits This approach is similar to the aforementioned greedy algorithm, except that we apply an EM algorithm to estimate the abundance of the selected taxonomic groups to see if this approach could take advantage of both MLE and MCS to generate results with both high sensitivity and specificity.Based on their taxonomy in RDP, we consider the abundance estimation of the original 9773 nearly full-length 16S rRNA gene sequences to be the ground truth. Several metrics, including Bray-Curtis dissimilarity, sensitivity and specificity, are used to compare the ground truth with OTU prediction results of each simulated dataset. Here, sensitivity is defined as the percentage of the taxonomic groups in the ground truth found by analyzing the simulated data. Specificity is defined as the percentage of the taxonomic groups estimated from the simulated data which are also in the ground truth.k1 are needed to generate taxonomic profile prediction consistent with that of Y almost full-length 16S rRNA sequences when there is no bias in the data?” To address this question, we use the approach described below.We want to answer the question “How many 16S rRNA gene fragments with average length k1 and read number n1 as dataset  and 2) the simulated full-length 16S rRNA gene sequence dataset with read number n2 as dataset  below. Since each such defined dataset has 10 replicates, we can use the Mann-Whitney U-test k1, n1) and dataset  to be equivalent if (1) there is no significant difference between their performance measures, in terms of sensitivity, specificity, or the Bray-Curtis dissimilarity, or (2) dataset  gives better performance than dataset , but worse performance than dataset  where n3>n2.First, we denote 1) the simulated 16S rRNA gene fragment dataset with read length 0) or some other biases (H1). If we assume that each dataset is a sample from a multinomial distribution Mult, the maximum likelihood estimates p′1, p′2, …, kp′ of the cell probabilities p1, p2, …, kp are then the corresponding genus-level abundance estimation. Under the null hypothesis, this abundance estimation is determined by estimating the abundance of different genera after pooling two datasets together. The test statistic is the log ratio of the likelihood under the null and the alternative hypotheses:N1 and N2 are the numbers of the 16S rRNA gene fragments and the full-length 16S rRNA sequences for this sample, respectively. k1 and k2 are the numbers of detected genera, p′i1 and p′i2 the estimated abundances of the i-th detected genera, and ni1 and ni2 the numbers of sequences assigned to the i-th genus, respectively. ip′ is the pooled estimate of the abundance of the i-th genus. k is the number of genera detected by at least one sample.Since for real data, we usually have only one replicate for each sample, thus we are unable to compare them using Mann-Whitney U-test. Also, current real 16S rRNA gene fragment data are usually of limited size, which raise another concern that the observed difference between the genus-level abundance estimation of 16S rRNA gene fragments and that of full-length 16S rRNA gene sequences maybe simply by chance under low coverage. We use a Likelihood Ratio Test (LRT) to address above concern and determine if the observed difference between the genus-level abundance estimation of 16S rRNA gene fragments and that of full-length 16S rRNA gene sequences is from low coverage .This test statistic follows a chi-square distribution with the degree of freedom equal to p-value is then corrected using the Benjamini and Hochberg False Discovery Rate (FDR) Since the gut microbiome data we used have 18 samples to be tested simultaneously, the The Wilcoxon signed-rank test is used to find the genera whose abundance estimation is significantly different between two different data types. The abundance estimation of a certain genus for a data type forms an 18-dimensional vector representing the estimated abundance levels from the 18 samples. The Wilcoxon signed-rank test is then performed on each pair of such vectors with a significance level of 0.05.Mothur https://code.google.com/p/shotgun-metagenomics-analysis-framework/.All statistical tests, performance metrics calculation and analysis procedures are implemented in C++, which is available at t-test, p-value = 0.834, 0.188, 0.066 for three metrics, respectively).We compared the mapping results using the raw database with those using the redundancy-reduced database (see Methods). As shown in On average, 75.2% simulated reads get BLAST hits with highest scores in at least 2 different taxonomic groups, suggesting necessity of exploring different approaches to integrate this information.Overall, results suggest that the nearest neighbor (NN) method produces the most accurate results. The maximum likelihood estimate (MLE) method consistently has the lowest specificity, while the greedy algorithm (MSC) consistently has the lowest sensitivity. The MLE of MCS method has better sensitivity than the MSC method, but with lower specificity. The NN method achieves the best combination of sensitivity and specificity  of the difference between Bray-Curtis dissimilarity, sensitivity and specificity of full-length and shotgun samples (see Methods). From these results, we concluded that n 16S rRNA gene fragments would achieve the same level of accuracy as n full-length 16S rRNA sequences when read length is ≥150 bp. In other words, it is highly likely that a read from a 16S rRNA gene ≥150 bp in length would contain enough information to characterize the whole 16S rRNA gene. Under these conditions, either 454 or pair-end Illumina may be the best sequencing platforms since either can achieve a read length >150 bp and thus, according to our conclusion, n 16S rRNA fragments data from these two platforms would achieve the same level of accuracy as n full length 16S rRNA sequences.By comparing the results of the simulated 16S rRNA gene fragments and full-length samples, we are able to establish a relationship between them . This reBy using our proposed analysis pipeline, we studied the human gut microbiome data introduced by Turnbaugh et al. t-test with p-value <10−5. This could result from potential technical bias in 16S rRNA gene sequencing. Therefore, we used an LRT (see Methods) to confirm whether the observed difference between the genus-level abundance estimation of the 16S rRNA gene fragments and full-length 16S rRNA sequences is significant or not. After correcting for multiple testing, 15/18 results showed that the difference were significant . For samples TS1, TS8 and TS9, the test gives FDR values of 0.06, 0.05 and 0.16, respectively, which are not significant at the level of 0.05. However, the overall results have showed there were potential biases during data generation that resulted in the difference between the abundance estimation of the 16S rRNA gene fragments and full-length 16S rRNA gene sequences for this gut microbiome data. After comparing the sequencing protocols of the two data, we tend to believe that the most likely source of bias occurred during priming. Thus, the observed difference between the two data may reflect the effect of primer bias in targeted sequencing on abundance estimation. These results are also consistent with the original study where the significant difference among the full-length, V2, V6 and 16S rRNA gene fragment data was reported According to the result from simulated data shown in p-value <0.05) of the difference between the abundance estimation of each genus in the 16S rRNA gene fragments, full-length, V2 and V6 data. The results suggested that the abundance estimation of the 13 genera supported by the 16S rRNA gene fragment data was significantly different from the other three data types. Among them, three genera belonged to the phylum Bacteroidetes. Nine genera belonged to the phylum Firmicutes. We also found one genus that belonged to the phylum Actinobacteria.It was also claimed in the original survey that the full-length and V6 data tended to show a depletion of Bacteroidetes, while the V2 and 16S rRNA gene fragment data tended to show a relative depletion of Firmicutes when compared with the full-length and V6 data. In order to prove this more rigorously, we used the Wilcoxon signed-rank test to test the significance . However, the recent data explosion has finally made 16S rRNA gene fragments a topic worth investigating. The protocol of shotgun sequencing provides 16S rRNA gene fragment data natural immunity against the bias raised during priming and thus the potential of uncovering the true structure of microbial community. Our established analysis framework could serve as a good starting point for experimental design and making the comparison between 16S rRNA gene fragment-based and targeted 16S rRNA sequencing-based surveys possible. With further development of computational methods An important direction for future studies is benchmarking by repeating some previous studies using 16S rRNA gene fragment data for potential difference in conclusions. Although we have shown that some high-level conclusions from targeted sequencing-based surveys may hold true, there is no guarantee such conclusions would never be affected by bias in abundance estimation.