In the quest for deciphering disease-associated biomarkers, high-performing tools for multiplexed protein expression profiling of crude clinical samples will be crucial. Affinity proteomics, mainly represented by antibody-based microarrays, have during recent years been established as a proteomic tool providing unique opportunities for parallelized protein expression profiling. But despite the progress, several main technical features and assay procedures remains to be (fully) resolved. Among these issues, the handling of protein microarray data, i.e. the biostatistics parts, is one of the key features to solve. In this study, we have therefore further optimized, validated, and standardized our in-house designed recombinant antibody microarray technology platform. To this end, we addressed the main remaining technical issues  and most importantly key biostatistics subjects (e.g. array data pre-processing and biomarker panel condensation). This represents one of the first antibody array studies in which these key biostatistics subjects have been studied in detail. Here, we thus present the next generation of the recombinant antibody microarray technology platform designed for clinical immunoproteomics. High-performing tools for multiplexed protein expression profiling of minimal amounts of crude clinical samples will be essential in the quest for deciphering disease-associated biomarkers for e.g. diagnosis and prognosis –3. A useDuring the last several years, affinity proteomics, represented mainly by antibody microarrays –7, have The process of designing, developing and applying high-performing antibody microarrays for clinical proteomics is a complex process and requires a truly cross-disciplinary approach to be adopted . To thisStandard operating procedure protocols (SOPs) were generated for each step, ranging from sample handling to microarray data analysis, resulting in a standardized SOP for running the recombinant scFv antibody microarray technology platform for clinical immunoproteomics.We used three cohorts of de-identified crude serum samples , denoted cohort 1 to 3, collected at Skåne University Hospital . No clinical information or patient identifiers were retained for samples (since this information was neither needed nor used in this study). The work was approved by the regional ethics review board in Lund, Sweden . Written consent was taken from participants. The samples were aliquoted and stored at -20°C until use. In serum sample cohort 1, 50 samples were mixed to create a reference serum sample, while the others were handled as individual samples, marked as either diseased (n = 151) or healthy (n = 57). Serum sample cohort 2 was composed of 341 samples, marked as either diseased (n = 171) or healthy (n = 170). Serum sample cohort 3 was composed of 1331 samples, marked as either diseased (n = 443) or healthy (n = 888).ref, QClabel and QCnorm, were introduced. QCref is based on pooled human serum samples , which is biotinylated in one batch (see below) and stored as aliquots, and then processed in one subarray on each microarray slide. QCref is intended to be used for evaluation of systematic technical variations and, if so required, for evaluation of the performance of the selected microarray data normalization procedure. QClabel is based on the same pooled human serum samples , but in contrast to QCref, it is biotinylated alongside the target samples of the study at hand. QClabel is intended to be used as a control for the biotinylation step. QCnorm is prepared based on combining aliquots of equal volume of serum samples from each study group  from the study at hand. QCnorm is intended to be used as an alternative route for microarray data normalization if so required.Three types of standardized quality control (QC) serum samples, denoted QCThe crude serum samples were labeled with EZ-link Sulfo-NHS-LC-Biotin  using a recently optimized labeling protocol for serum proteomes , 7. Brieet al, manuscript submitted)  two monoclonal antibodies against CA-19-9 , and 375 human recombinant single-chain variable fragment (scFv) antibodies, including 356 antibodies targeting mainly immunoregulatory proteins and 19 scFv antibodies targeting short amino acid motifs (4 to 6 amino acids long)  were selbmitted) . The spebmitted)  [4, 7, 1E. coli and purified from either i) expression supernatants using affinity chromatography on Ni2+-NTA agarose , or ii) periplasmic space with the use of MagneHisTM protein purification system  and a KingFisher Flex robot . The elution buffer (250 mM Imidazole) was exchanged for PBS through either extensive dialysis (supernatants) or using Zeba 96-well desalt spin plates (Thermo Fisher Scientific). The protein concentration was determined by measuring the absorbance at 280nm using NanoDrop-1000 . The degree of purity and integrity of the scFv antibodies were evaluated by 10% SDS-PAGE . All antibodies were stored at 4°C until use.All scFv antibodies were produced in Thirty μL scFv antibody was added per well to the 384-well printing source plate , containing 3 μL 1 μg/ml Alexa Fluor 555-Cadaverine . Fluorophore-labeled cadaverine was used as a spotting control and, if so required, for assisting in the spot finding step during array quantification. The antibodies were printed on black polymer MaxiSorp microarray slides (NUNC), by spotting one drop (~330 pl) at each position, using a non-contact printer . Four different microarray layouts were used, denoted array layout A to D, including a 94-plex antibody array (25x25 spots) (column x row), a 184-plex antibody array (32x22 spots), 195-plex antibody array (25x28 spots), and 351-plex antibody array (36x34 spots).In layout A, 25x25 subarrays were printed, composed of 94 scFv antibodies, one negative control (PBS), and one positive control/reference marker (biotinylated BSA (b-BSA)). Ten 25x25 subarrays per slide were printed. Within each individual subarray, each scFv antibody was printed in 5 subsequent replicates, while PBS was spotted in 5 subsequent replicates at six locations spread across the subarray . Further, b-BSA was printed as five full rows (5x25 spots), and spread across the subarray, and used to control for any surface defects. In total, nine slides were produced in three individual spot runs (three slides/day). Based on inadequate printing quality, only 83 scFv antibodies were used in the subsequent data analysis steps.In layout B, 32x22 subarrays were printed, composed of 184 scFv antibodies, one negative control, and one positive reference marker. Each subarray was divided in three identical segments where a row of b-BSA consisting of 32 replicate spots was printed at the beginning and the end of each segment, i.e. four lines. Each scFv antibody was dispensed in triplicates, one in each segment, to assure adequate reproducibility. The negative control was printed in eight replicates per segment. Thirteen subarrays per slide were printed, and in total six slides were generated. Due to printing issues, only the first five rows in each segment, corresponding to 121 antibodies, were used in the subsequent data analysis.In layout C, 25x28 subarrays were printed, composed of 195 scFv antibodies, one negative control, and one positive reference marker. The subarrays were divided and printed in three identical segments as in layout B, but with the following changes; the negative control was printed in five replicates per segment and b-BSA was printed in 25 replicates. Fourteen subarrays per slide were printed. In total, sixteen slides were produced in two individual spot runs (eight slides/day).In layout D, 36x34 subarrays were printed, composed of 351 antibodies, one negative control, and one positive reference marker. The subarrays were divided and printed in three identical segments as in layout B, but with the following change; the negative control was printed in nine replicates per segment, and b-BSA was printed in 36 replicates. Fourteen subarrays were printed per slide.Briefly, the printed microarray slides were allowed to dry for 2h (cohort I) or 1 week (cohorts II and III) at RT and were then mounted into a multi-well incubation chambers (NEXTERION® IC-16) . Next, the slides were blocked with 1% (v/v) Tween-20 (Merck Millipore) and 1% (w/v) fat-free milk powder  in PBS (MT-PBS solution) for 2h (cohort I) or 1h (cohorts II and III) at RT. Of note, the composition of the original blocking solution (5% (v/v) fat-free milk in PBS)  was re-oThe ScanArray Express software v4.0  was used to quantify spot signal intensities, using the fixed circle method. Signal intensities with local background subtraction were used for data analysis. In the case of antibodies, each data point represents the mean value of all three replicate spots, unless any replicate CV exceeded 15%, in which case the worst performing replicate was eliminated and the average value of the two remaining replicates was used instead. Log2 values of signal intensities were used for subsequent analysis. The mean value was determined for the positive and negative control spots, using all data points.PBS. Antibodies that were found to have a mean signal intensity below LOD in > 70% of all analyzed samples were removed from the raw data set.In order to only include antibodies with a detectable signal, a limit of detection (LOD) cut-off was implemented. The cut-off was defined as mean PBS intensity plus 2xSDTo identify technical outlier samples in the raw data, a two-step procedure was implemented. First, using the positive control (see above), the sample was excluded if the CV-value was ≥20% for at least 3 of 4 rows . A visual inspection was conducted if 2 of 4 rows displayed inadequate CV-values, potentially leading to the use of only 1 or 2 segments of the subarray . Second, a principle component analysis (PCA) was performed using the Qlucore Omics Explorer 3.1 software  for visual identification of potential outliers.i), thus zero centering the data  of analysis were normalized using the “subtract by group mean” approach . In thisescribed , 24. In norm was not explored as an alternative route for microarray data normalization.In total, nine different normalization strategies, including our currently adopted normalization method, “subtract by group mean” strategy combined with “semi-global normalization” , 24, werThe different strategies included variance stabilization and normalization  , LOESS . LOESS aIn addition, supervised classification of samples was used when comparing different days of analysis. To this end, the Random Forest (RF) function implemented in the randomForest R package was also used to create a RF model with 1000 decision trees (ntree = 1000) , 31.Four different methods for defining a condensed biomarker signature providing the best classification of two sample groups were evaluated, including i) selection based on p-values, ii) backward elimination using support vector machine (SVM), iii) modified backward elimination using SVM consensus approach (SVMc), and iv) RF .First, the biomarkers were ranked based on their Wilcoxon p-values, and the markers with the lowest p-values were selected.C = 1, was used as the classifier in a backward elimination scheme. Given a panel of all biomarkers available, a leave-one-out (LOO) cross-validation estimate of the AUC was calculated. Next, a ranking of the included biomarkers was established using all of the SVMs trained in the LOO cross-validation procedure [In the case of backward elimination using an SVM (BE-SVM), an SVM with a linear kernel and soft margin parameter rocedure . The bioK-fold cross-validation loop was added in which one of the K:th parts was removed from the data cohort before the initiation of the backward elimination scheme. The outermost was iterated, leaving out K parts of the data, hence resulting in K final ranking lists, possibly NxK lists if the outermost loop was randomly repeated N times. A consensus approach [Next, we evaluated a modified version of backward elimination using an SVMc approach. A potential problem with the above approach might be overtraining with respect to a given data cohort, especially when sample sizes are small. Randomly correlated biomarkers might obtain a high rank in the above procedure. To reduce the effect of such a potential overtraining, an additional approach  was thenBoth the above approaches are based on linear classifiers and might for some diagnostic problems lack the necessary complexity to reach high accuracy. SVM classification models could use non-linear kernels to allow for more complex classifiers, but with the cost of tuning more parameters. Random Forest (RF) models  are thusAll condensed biomarker signatures, obtained by the different rankings presented above, were evaluated using a linear SVM. In those cases where the biomarker list was based on RF models, an evaluation using a RF was also added. The reason for this was that if non-linear effects was discovered by the RF ranking method, the use of a linear SVM during evaluation was not optimal. Thus, it was natural to also evaluate a RF ranking list by RF models.In this study, we have further optimized, validated, and standardized our in-house designed recombinant antibody microarray technology platform for clinical immunoproteomics. This was accomplished by resolving the main technical features and assay procedures, and most importantly the protein microarray biostatistics step as specified in We used recombinant scFv antibodies, selected from a large phage display library based on a molecular design adapted for microarray applications, as content and evaluated the probe quality in terms of spotting concentration, on-chip functionality and specificity.First, we optimized and standardized the spotting concentration for each individual antibody. Representative results for three antibody clones, two targeting a crude biotinylated serum sample  and one Stringent phage-display selection and screening protocols were used to promote antibody functionality and specificity see . All theIn order to further support the specificity claims, based on the selection criteria, orthogonal methods and on-chip confirmations  while varying i) the batch of biotin, ii) the user, and/or iii) day of analysis . The repref, QClabel and QCnorm, were introduced in order to further standardize the set-up (see The performance of the microarray assay was improved by further optimizing i) the choice of blocking buffer (1% (v/v) Tween-20 and 1% (w/v) fat-free milk in PBS vs. 5% (v/v) fat-free milk in PBS) (reducing non-specific background binding), and ii) the sample incubation time (2h vs. 1h)  see . In addit-up see .With these new settings in place, we evaluated the source and effect of systemic technical variations on our microarray data. To this end, a 94-plex antibody microarray (83 spotted antibodies passed QC) was used (layout A) to profile the same pooled human serum sample on 90 subarrays, processed on 9 slides, and analyzed over 3 days (3 slides/day) . PrincipIn an attempt to reduce this variation, we employed our current standard approach to neutralize day-to-day variations, namely “subtract by group mean”. PCA analysis of the normalized data, showed that the day-to-day dependence had been minimized . In agreWe standardized and/or further evaluated the data pre-processing procedure. Prior to the key step, array data normalization, the data was subjected to three actions. First, any background signals were subtracted from the measured signals in order to remove any non-biological contributions. Second, the mean value of the replicate spots for each antibody was calculated, after removing any outlier spots. Third, any antibodies below limit of detection, defined as mean blank signal across all samples plus 2 standard deviations in >70% of the samples, were identified and excluded from the data set, as illustrated in ref serum. In total, 16 slides, with 14 subarrays each, were run during 2 days (8 slides/day). In total, nine different normalization strategies were tested and compared to i) our currently adopted normalization method, “subtract by group mean” strategy combined with “semi-global normalization” as well as ii) un-normalized log2 transformed raw data  to profile 224 serum samples (151 diseased and 57 controls) and 16 QCraw data .First, we compared the effect of Quantile, VSN , and LOESS  normalization on the data set to raw un-normalized log2 transformed data by the means of intensity distributions in density plots, boxplots, and meanSdPlots –S3 Figs.In order to address this additional normalization need, we proceeded and tested three additional methods, including (i) global VSN Combat normalization, (ii) global LOESS ComBat normalization, and (iii) ComBat semi-global normalization, and compared the output with that of our current adopted normalization method, i.e. subtract by group mean combined with semi-global normalization. Again, un-normalized log2 transformed raw data was used as reference. To start with, the methods were evaluated by the means of intensity distributions in Q-Q plots, density plots, boxplots, and meanSdPlots . When coNext, we evaluated the normalization approaches with respect to sample PCA plots . A clearAlong the same line, we then assessed the relative impact of the normalization approaches on variables (scFv) using variable PCA plots . This chref) or ii) identical samples , using t-test (ref), showed a refined view. The number of statistically significant (q < 0.01) scFv decreased in the order of global LOESS Combat (n = 26) < Global VSN Combat < ComBat semi-global < Subtract by group mean plus semi-global < un-normalized (n = 158) , based on either i) different samples  . Hence, total = 47) was higher in both ComBat semi-global and Subtract by group mean plus semi-global than in global VSN Combat and LOESS ComBat normalized data sets. Again, the same behavior was observed when comparing two subgroups with the diseased samples only . A FC > 1 or FC > 1.1, with and without a cut-off value of q <0.05 was applied . The resles only . The datref). Using Subtract by group mean plus semi-global normalization as showcase, the data clearly showed that the day-to-day variation had been reduced, as seen for both QCref day 1 vs. day 2 (−9–10−14 to 10−7–10−13) and their individual order changed. Furthermore, 18 of 50 clones were replaced, but these new clones were included irrespective of the two normalization methods adopted. Taken together, our data showed that ComBat semi-global and Subtract by group mean plus semi-global normalization were the two preferred normalization methods.Finally, focusing on the two best normalization methods, Subtract by group mean plus semi-global normalization and ComBat semi-global, we used PCA to investigate the variability between all samples  selection based on Wilcoxon p-values, ii) backward elimination using SVM (BE-SVM), iii) modified backward elimination using SVM (consensus approach) (SVMc), and iv) RF. Two independent samples cohorts, cohort 2 and 3, randomly divided into a training set and a test set, were used as sample sets. The samples were analyzed on 351-plex antibody arrays (layout D).First, condensed biomarker signatures of a fixed length (N = 25) of the top-ranked biomarkers were defined using each of the methods . The claSecond, we re-ran the condensation methods, and defined the best performing condensed biomarker signatures, but without fixing the signature length . The sigThe number of high-performing antibody microarray set-ups at hand is still very low, which is explained by the fact that systematic cross-disciplinary efforts, addressing all of the key methodological areas involved in an array set-up , must beper se have been extensively addressed in recent work by us [et al, manuscript in prep.). But to incorporate a built-in specificity control step in the microarray assay, we have included several antibody clones directed against the same protein antigen, but targeting different epitopes. This is essential, as some antibodies might lose their reactivity caused by i) epitope masking, e.g. caused by the biotinylation of the sample [Among the technical array attributes, the specificity of the dispensed antibodies is a vital feature . The sperk by us , 16–21  and ComBat semi-global, to perform the best. This means that these normalization procedures were capable of handling both day-to-day and array-to-array variations, but with the smallest transformation of the data set and  maintaining biological differences at large in the data set. In future work, any of these methods could be applied, and the precise choice would have to be determined data-set by data-set.Deciphering condensed biomarker panels, going from several hundreds to about twenty markers or less, providing the best discriminatory power for the question at hand, e.g. diagnosis, will be essential in the development of novel tests . In the Finding a condensed panel of biomarkers that performs optimally for a given diagnostic problem can be viewed as a feature selection task in machine learning. Here, the diagnostic problem is transformed into a classification problem using all available biomarkers as features. The task is to find a reduced set of features that results in optimal  classification performance. Here we used the ROC AUC value as a performance measure. The p-value ranking was found to generate biomarker panels displaying the worst AUC values, which could be explained that the markers were selected based on p-values and not whether they provided orthogonal information. This means that many of the selected markers might have provided similar information. Biomarkers selected based on p-values are therefore likely better to reflect the disease and disease state rather than reflecting the best classifier.Further, care has to be taken to avoid overtraining, here meaning the problem of determining a condensed biomarker panel too specialized on one cohort, thereby lacking the necessary generalization to other cohorts of the same diagnostic problem. The overtraining problem is typically present in situations with small sample sizes and a large number of features (biomarkers). The classification methods used in the feature selection process are selected to have as few tunable parameters as possible, to avoid overtraining on method parameters. In this study, we did not see any significant signs of overtraining which might be due to e.g. large sample sizes, but in future data sets this might become essential. It should be noted that any further refinement of a condensed signature should be validated using a novel independent sample cohort(s). Taken together, the results showed that we have defined two excellent ways of defining condensed biomarker signatures, namely SVM and SVMc, the latter method having a built-in function to avoid overtraining. Depending on the nature of the data set, RF might also be a viable option, while the p-value ranking methods is less recommended.et al, unpublished observations). Other issues to resolve could include, but are not limited to, orientated antibody immobilization for improved functionality [Despite the current advances, some features could be subjected to even further optimizations. As for example, the range of specificities included on the array is critical for defining the resolution at which each sample can be profiled. Here, we used up to 351-plex arrays, but have in recent applications used 395-plex antibody arrays , and we ionality , assay aTaken together, we have continued our interdisciplinary efforts, and presented the next generation of our recombinant antibody microarray technology platform for clinical immunoproteomics. This platform could pave the way for the next wave of clinical applications, with great potential for biomarker discovery and clinical endeavors, such as diagnosis, prognosis, and classification.S1 FigDensity plots were used as qualitative measures in order to compare the log intensity distribution of samples output with respect to log2 transformed data, Local LOESS, Local VSN, Global LOESS, Global VSN and Quantile normalizations.(EPS)Click here for additional data file.S2 FigBoxplots were used as qualitative measures in order to compare the log intensity distribution of samples output with respect to log2 transformed data, Local LOESS, Local VSN, Global LOESS, Global VSN and Quantile normalizations.(EPS)Click here for additional data file.S3 FigMeanSdPlots were used as qualitative measures in order to compare the log intensity distribution of samples output with respect to log2 transformed data, Local LOESS, Local VSN, Global LOESS, Global VSN and Quantile normalizations.(EPS)Click here for additional data file.S4 Figref samples as well as diseased and healthy samples. i) QCref samples analyzed on two days. ii) Diseased vs. healthy samples analyzed on different days (mapped for day). iii) Diseased vs. healthy samples analyzed on different days .The sample cohort included QC(EPS)Click here for additional data file.S5 FigThe data set, composed of two groups–healthy (H) and disease (D)—was randomly divided in two three subsets. Condensed biomarker signatures of a fixed length (N = 25) of the top-ranked biomarkers were defined using each of the feature selection methods . The classification performance of the signatures was then tested and described in terms of ROC AUC values. The signatures were defined based on the training set, and evaluated on the independent test set, using a linear SVM or RF. This entire procedure was repeated two times using a 3-fold cross validation scheme to obtain more reliable test performances(EPS)Click here for additional data file.S1 TableThe specificity, affinity , and on-chip functionality of all of these phage display derived scFv antibodies were ensured by using i) stringent phage-display selection and screening protocols  (16), ii) multiple clones (1 to 9) per protein, and iii) a molecular design, adapted for microarray applications 14). In addition, the specificity of several selected antibodies (marked with an *) have been further validated using pure proteins, mixtures of pure proteins, as well as well-characterized, standardized serum samples  and/or specific protein(s) depleted), and/or orthogonal methods, such as mass spectrometry (affinity pull-down experiments), ELISA, MesoScaleDiscovery assay, and cytometric bead assay, as well as using blocking experiments . In addi, 17–21.(DOCX)Click here for additional data file.S2 TableTo this end, microarray data for diseased group 3 vs. group 1 samples was used and compared with respect to No. of down-regulated scFvs antibodies and No. of complete matches per target molecule, using a fold change (FC) filter of FC > 1.(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.


Motivation: High-throughput RNA sequencing (RNA-seq) is now the standard method to determine differential gene expression. Identifying differentially expressed genes crucially depends on estimates of read-count variability. These estimates are typically based on statistical models such as the negative binomial distribution, which is employed by the tools edgeR, DESeq and cuffdiff. Until now, the validity of these models has usually been tested on either low-replicate RNA-seq data or simulations.Results: A 48-replicate RNA-seq experiment in yeast was performed and data tested against theoretical models. The observed gene read counts were consistent with both log-normal and negative binomial distributions, while the mean-variance relation followed the line of constant dispersion parameter of ∼0.01. The high-replicate data also allowed for strict quality control and screening of ‘bad’ replicates, which can drastically affect the gene read-count distribution.Availability and implementation: RNA-seq data have been submitted to ENA archive with project ID PRJEB5348.Contact:g.j.barton@dundee.ac.uk High-throughput sequencing of RNA  and a snf2 knock-out mutant cell line (Δsnf2). Briefly, the extracted total RNA from each of the 96 replicates was enriched for polyadenylated RNA, quality checked, and had an appropriate amount of artificial External RNA Controls Consortium (ERCC) spike-in transcripts added (S. cerevisiae genome (modified to include the ERCC transcript sequences) with TopHat2  the length of the gene. Normalizing the data to compensate for these factors is a critical step in analysing RNA-seq expression data and it is essential that the normalization chosen is appropriate to the biological question being addressed. The simplest normalization scales the raw counts to the total number of sequenced reads in the replicate , however, this is unlikely to be appropriate for a DGE experiment where one might reasonably anticipate that a small number of highly expressed genes are likely to change their expression level dramatically while the expression of the majority remains unchanged. Applying the total count normalization may result in many genes being identified as differentially expressed between conditions when they are, in fact, unchanged. These normalization approaches all result in counts that are no longer integer and this can lead to unpredictable results when testing them against discrete probability distributions . Accordingly, an alternative approach to normalization was applied, that while not suitable for DGE, is more appropriate for testing the underlying distribution of RNA-seq read counts for a gene. When examining the distribution of gene read counts across the biological replicates within a condition, the replicate with the smallest total read count was selected and then the other replicates were randomly sub-sampled to the same total read count. The same approach was applied to read counts across flow cell lanes within each biological replicate. The equal-count data are then mapped to the reference genome obtaining normalized, but discrete, mapped read counts.The obvious disadvantage of this method is the loss of reads  and sensitivity. This approach is essentially a total count normalization that preserves the discrete nature of the data and so is unlikely to be any more appropriate for DGE analysis than standard total count normalization. However, under the null hypothesis, the down-sampled replicates within a condition are realizations of independent and identically distributed random variables, so it allows goodness-of-fits tests to be performed on raw gene counts with no need for any additional normalization. All the tests in this work are done within one biological condition  and no DGE analysis is undertaken. The loss of reads is not a concern for this work, which aims at studying distribution over a wide range of read counts. The mean read count per gene extends from Occasionally errors occur in an experimental protocol and a sample generates improper or ‘bad’ data. In this experiment, the availability of 48 replicates allowed three different criteria to be applied individually and in combination to identify potential ‘bad’ replicates in addition to standard quality checks by FastQC.The differences between each pair of replicates, eplicate . The repnj≠i rij a.Fig. 1The poor correlations shown by some replicates are the result of a small proportion of genes with atypical read counts. These outliers can be identified by comparing each gene’s expression in an individual replicate with the trimmed mean across all replicates. Specifically, the The atypical total read counts of outlier genes are, in our case, the result of an atypical, strongly non-uniform, read depth gene profile. An example of the difference of the read depth profiles between clean and ‘bad’ replicates is shown in ondition c.Fig. 3snf2 replicates, are identified as ‘bad’ . They were subsequently removed from the downstream analysis unless otherwise stated. In the following discussion, the remaining replicates are referred to as ‘clean replicates’. This is an arbitrary and rather conservative selection. For example, one doesn’t expect Δsnf2 replicate 25 to make a huge impact. However, the most anomalous replicates in this selection affect the measured read-count distribution dramatically (Section 3).Interestingly, while there is similarity between the replicates identified as ‘bad’ from the gene expression-based quality measures  replicates for each biological replicate. This allowed the inter-lane variance in the sequencing protocol to be examined. The RNA fragments from each sample were expected to be distributed uniformly and randomly across the flow cell lanes, which would result in a Poisson distribution of sequenced read counts across lanes  are realizations of independent random variables that follow a Poisson law with the same mean (i.e. ion test  is χ2 disnf2) is shown in Forty-eight biological replicates in each of two conditions also allow the quantification of reproducibility in expression data across the replicates within a condition. Because the samples are independent, a Poisson distribution arises naturally from counting the number of reads mapping to each gene across replicates. Any additional variability, either from variability in the experimental protocol or biological variability, will result in broader distribution. The variance-mean relationship across all clean biological replicates (42 in WT and 44 in ΔDESeq and edgeR) or log-normal  distribution. Here, the gene read counts from clean data are tested against log-normal, negative binomial and normal (NM) distributions.Many differential expression tools address the over-dispersion inherent to short read RNA-seq data by assuming the form of the probability distribution underlying gene expression. Popular choices for this distribution include, but are not limited to, a negative binomial . The expression data for all genes, including those with zeroes, was tested for consistency with a negative binomial distribution using the goodness-of-fit test proposed by P-values. Hence, it is more computationally intensive than the normality test. 107 bootstraps were carried out and resulted in P-values that are limited to be The test of normality described by P-value of 0.05. The data are largely consistent with both the log-normal and negative binomial distributions; with snf2 dataset, respectively. The negative binomial test rejects this distribution in only a few genes . In fact, a large fraction of data is also consistent with a normal distribution, perhaps unsurprisingly given that both log-normal and negative binomial distributions approximate the normal distribution for high counts. At very low counts . Each cell culture represents a distinct biological replicate of the given condition. However, each of the samples underwent a series of processing steps: RNA extraction, polyA selection, fragmentation, priming, cDNA synthesis, adapter ligation, PCR amplification and sequencing which can contribute to the overall variability. Accordingly, the observed read-count variability is a combination of true biological variability between individual samples, and the protocol variability introduced during sample preparation and sequencing.The data presented here give us an unprecedented view of the variability inherent to RNA-seq experiments, albeit limited to an organism with gene expression largely unaffected by splicing and other complex mechanisms observed in higher eukaryotes. Distinguishing between variability introduced by experimental procedure and intrinsic biological variation is often tricky in RNA-seq experiments, and is complicated by the opaque use of the terms ‘technical replicate’ and ‘biological replicate’ in much of the literature. In the experiment presented here, 48 edgeR, DESeq, cuffdiff, (baySeq, (A key finding of this work is the demonstration that the read-count distribution of the majority of genes is consistent with the negative binomial model. Reassuringly, many of the most widely used RNA-seq DGE tools  do assuedgeR (DESeq (In this work, the observed variance is tightly related to the mean and follows a line of constant dispersion parameter. Modelling the mean-variance relationship is essential in many DGE tools [e.g. in R (DESeq ], when dThe design of the experiment presented here also allows us to probe the inter-lane sequencing variability in detail. This reflects how reads from one biological sample are distributed between different lanes of a flow cell and arises from a limited number of steps in the RNA-seq protocol, namely loading the samples into individual lanes, cluster amplification and sequencing by synthesis. An even loading and spatially uniform amplification and sequencing should give rise to a Poisson distribution of counts from individual genes. This is technical variability at a very low level. A As shown in


Diagnostics of the human ageing process may help predict future healthcare needs or guide preventative measures for tackling diseases of older age. We take a transcriptomics approach to build the first reproducible multi-tissue RNA expression signature by gene-chip profiling tissue from sedentary normal subjects who reached 65 years of age in good health.One hundred and fifty probe-sets form an accurate classifier of young versus older muscle tissue and this healthy ageing RNA classifier performed consistently in independent cohorts of human muscle, skin and brain tissue  and thus represents a biomarker for biological age. Using the Uppsala Longitudinal Study of Adult Men birth-cohort (n = 108) we demonstrate that the RNA classifier is insensitive to confounding lifestyle biomarkers, while greater gene score at age 70 years is independently associated with better renal function at age 82 years and longevity. The gene score is ‘up-regulated’ in healthy human hippocampus with age, and when applied to blood RNA profiles from two large independent age-matched dementia case–control data sets (n = 717) the healthy controls have significantly greater gene scores than those with cognitive impairment. Alone, or when combined with our previously described prototype Alzheimer disease (AD) RNA ‘disease signature’, the healthy ageing RNA classifier is diagnostic for AD.We identify a novel and statistically robust multi-tissue RNA signature of human healthy ageing that can act as a diagnostic of future health, using only a peripheral blood sample. This RNA signature has great potential to assist research aimed at finding treatments for and/or management of AD and other ageing-related conditions.The online version of this article (doi:10.1186/s13059-015-0750-x) contains supplementary material, which is available to authorized users. For the Kraus data set total RNA was extracted from frozen muscle biopsy samples  using TRIzol reagent and in vitro transcription was performed using the Bioarray high yield RNA transcript labeling kit  as previously described [We implemented fully independent ne-chips . When cone-chips ; [GEO:GSne-chips . For eacne-chips  [GEO:GSE Hoffman  [GEO:GSEescribed . For allTo assess if human brain and skin also demonstrated the same 150 age-related gene expression signature as healthy older muscle, we used young and old samples from the brain-bank array source  and the MuTHER cohort skin data set . The skin data were produced using the Illumina Human HT-12 V3 Bead chip (Array-express: E-TABM-1140) and log-2 transformed signals were normalized using quantile normalization. The 150 Affymetrix probe-sets were mapped to the Illumina platform (giving 129 probes). Due to differences in gene-chip technology, a LOOCV approach was used to classify the age of each skin sample, using only the 150 probes selected at the start of the project. For skin, individuals aged 45 years or less were defined as young, and those aged 70 years or older as old to ensure balanced numbers of young and old samples existed to fairly assess the classifier performance. The three sets of technical replicates were analyzed separately and confirmed the intra-study technical reproducibility of the classifier using repeated RNA profiles of a single clinical sample (data not shown).in vitro transcription reaction were removed using an RNeasy column (QIAGEN Inc.). Hybridization, washing, staining, and scanning of the arrays were performed according to the manufacturer’s instructions .We used a set of tissue samples from a birth cohort of men, such that the same chronological age (~70 years) could be contrasted with the variation in healthy ageing gene score. The ULSAM cohort comprises men born in 1920–1924 and living in Uppsala, Sweden and was used to compare a constant chronological age (and similar environment) with the healthy muscle age gene score across individuals . Dual-enOne hundred and eight samples passed gene-chip quality control procedures (see above). A cumulative gene ranking-based score was calculated using each of the 150 gene expression values for each of the 108 male subjects and the final score was compared in a linear fashion with a number of clinical parameters. For an RNA down-regulated in the original training classification data set  the ULSAM subject with the highest expression was assigned a score of 1 and the subject with the lowest expression 108. For genes up-regulated in the original age classification model, the opposite strategy was used. Thus, both feature selection (genes) and direction of regulation were taken from the original model. The median sum of these rank scores (reflecting the 150 probe-sets) was calculated and that represented the healthy ageing gene score for each individual in the ULSAM cohort. Median rank ensured each gene provided equal weighting and regression analysis was used to study the variation in gene score in these men, all of who had approximately the same chronological age.p > 0.1 as stop criterion . Clinical variables, previously reported [p value [The relationship between the gene score at age 70 years and a number of clinical features was carried out using multi-factor models. Model selection was executed using a forward selection approach, with reported , were ad[p value   cohort in London. Patient selection, design, and clinical data have been reported previously , 97. AD We used two independently produced gene-chip data sets from the AddNeuroMed/DCR consortia, one data-set produced in a UK gene-chip facility and another produced in the USA. Gene expression data was produced using Illumina Human HT-12 v.3 Expression BeadChips for the first case–control study  and Illumina Human HT-12 v4 Expression BeadChips for the second case–control study . cDNA was synthesized from 200 ng total RNA using the TotalPrep™ RNA Amplification Kit (Ambion), which was followed by amplification and biotinylation of cRNA and hybridization. The expression data were first transformed using variance stabilization and then quantile normalized using the LUMI package in R. The 150 probe-sets were mapped from the Affymetrix platform to the Illumina platform. For our primary analysis, control subjects were matched in a manner that created the largest possible group with the same chronological age and gender balance as the AD or MCI groups. Thus, our analysis was carried out on a subset of subjects deposited at the GEO, with each case–control group having a similar median chronological age as the ULSAM cohort. In total 297 samples were utilized . Retrospective inclusion of the entire cohort (n = 717) did not alter the outcome of our analysis.We also utilized two additional large gene-chip clinical studies; one comparing blood RNA in type II diabetes with control  and the p values by randomly sampling 10,000 lists of 150 probe-sets from the hgu133plus2 chip and testing each list for the GO term molecular function using the GOstats package in R. The entire population of probe-sets on the hgu133plus2 microarray was used as the background population for these tests. The resulting p values for each tested probe-set list were corrected using the method of Benjamini and Hochberg. The 150 healthy ageing genes were then tested for GO term molecular function and the p values Benjamini and Hochberg corrected. Positional gene enrichment analysis was used to identify whether the classification genes (or the classifier network genes) were significantly enriched within given chromosomal regions [The bioinformatics tool Ingenuity Pathway Analysis (IPA)  was used regions  as previ regions .


This comparative evaluation is performed on both synthetic and real data, the latter consisting of two manually curated microarray compendia comprising several E. coli and Yeast studies, respectively. Furthermore, the reconstruction of the regulatory network of the transcription factor Ikaros in human Peripheral Blood Mononuclear Cells (PBMCs) is presented as a case-study.We address the problem of integratively analyzing multiple gene expression, microarray datasets in order to reconstruct gene-gene interaction networks. Integrating multiple datasets is generally believed to provide increased statistical power and to lead to a better characterization of the system under study. However, the presence of systematic variation across different studies makes network reverse-engineering tasks particularly challenging. We contrast two approaches that have been frequently used in the literature for addressing systematic biases: p-values for correctly ranking candidate interactions. The results from the PBMC case-study indicate that the findings of the present study generalize to different types of network reconstruction algorithms.The meta-analysis and data-merging methods included in our experimentations provided comparable performances on both synthetic and real data. Furthermore, both approaches outperformed (a) the naïve solution of merging data together ignoring possible biases, and (b) the results that are expected when only one dataset out of the available ones is analyzed in isolation. Using correlation statistics proved to be more effective than using Ignoring the systematic variations that differentiate heterogeneous studies can produce results that are statistically indistinguishable from random guessing. Meta-analysis and data merging methods have proved equally effective in addressing this issue, and thus researchers may safely select the approach that best suit their specific application.The online version of this article (doi:10.1186/s12859-016-1038-1) contains supplementary material, which is available to authorized users. Reverse engineering of gene regulatory network is a vibrant research area , whose sUnfortunately, this assumption is not valid when data from different studies are integratively analyzed. Systematic biases across studies can originate spurious correlations that do not actually reflect any interactions among genes. On the other side, they can hide associations that are actually present among the measured quantities . These sMeta-Analysis  algorithms , while each edge  is equipped with a randomly generated coefficient βij ∈  ∪  representing the strength of the interactions between i and j.These artificial networks were equipped with a parameterization suitable for the simulation of gene expression data and batch effects. Each gene Batch-effects across studies are assumed to be composed of an additive and a multiplicative component, following an approach already used in . The firysjk for sample s, gene j and study k is generated as follows:The expression value ysjk is a linear combination of its baseline value αj and the expression values of its regulating genes ysik, i ∈ IN(j). The quantity ϵj is random noise distributed as N  that represents unmodeled regulatory mechanisms that concur in determining the expression of the gene. The two factors γjk and δjkϵsj' respectively represent the addictive and multiplicative component of the systematic bias in study k, and are both randomly sampled from the distribution N. The random variable ϵj' is again distributed as N.According to this formula, each expression value barabasi.game function from the R package igraph , thus obtaining 27 different scenarios for each network and 135 in total.During our experimentations, five independent synthetic networks with four-thousand nodes each were created using the e igraph . For eace igraph , 10, 50,e igraph , 50, andT of transcription factors includes all genes directly connected to at least twenty other genes. This leads to an average of 18 transcription factors for each network, each one connected on average with 40 genes. We consider only direct interactions in order to ensure that the corresponding associations are strong enough to be effectively retrieved from the data.Finally, for each network the list TFt can be easily reconstructed by computing the vector Ct containing n − 1 associations between TFt and all other probesets i. These association measures are eventually coupled with measures of statistical significance Pt, and the genes Qt belonging to the reconstructed network can be selected by imposing an appropriate decision threshold θ to either the association or the significance values. When multiple datasets are available, the same procedure can be followed with the vectors Ct,X and Pt,X computed through the meta-analysis or data-merging method X , 0.5 corresponds to random ordering and zero to perfectly inverted predictions. Interestingly, AUC values can be interpreted as the probability of correctly ranking two randomly selected interactions according to their status .The Receiver Operator Characteristic (ROC) Area Under the Curve  is a meAUPRC, , i.e., R predictions are known interactions.We also devised a new metric that is specific for assessing partial performances, namely the Area Under the False Discovery Rate  instead of p-values. Φ− 1 is the inverse of the standard normal cumulative distribution function, and the statistic Pt,i,Stouffer. Also in this case Ct,i,Stouffer = Zt,ir method  is conceFixed-Effects approach [Ĉt,i, whose estimation is biased by a study-specific error factor, i.e., Ct,ij = Ĉt,i + ϵj, j = 1 … m. On the basis of these assumptions, Ct,i,Fixed can be computed through a weighted meanapproach  assumes We selected from the literature five MA methods whose operation matches the above definition and that are based on different assumptions and theoretical backgrounds.wj are inversely proportional to the correlations variances. Pt,i,Fixed is computed by comparing Ct,i,Fixed Fisher z-transformation against its theoretical normal distribution [Random-Effects models do not assume that each study estimates the same correlation Ĉt,i; the datasets are assumed to be enough ‘similar’ to be jointly analyzed, but at the same time the ground truth correlation Ĉt,i and unknown variance Ct,ij is an estimation of its corresponding ϵj, i.e., Ct,i,Random is estimated with the Fixed-Effects weighted average, with the weights wj computed as inversely proportional to the sum of the study-specific and between-study variance, i.e., The summary correlation The Rank-Product method differs from the previous approaches since it combines correlation ranks instead of correlations or p-values [Ctj containing the correlations between the transcription factor t and all other probesets in study j can be easily converted in a vector of ranks Rtj, where higher correlations rank first. The Rank-Product method combines ranks Rt,i1, …, Rt,im from different studies by multiplying them: Rt of combined ranks.p-values . The vecThe Rank-Product is actually a special case of a larger family of rank-based methods , differip-values for the Rank-Product results.A common drawback of these methods is that statistical significance must be assessed through permutation-based procedures, which usually are quite computationally demanding. However, in this study we adopt a recently introduced formula  for compwhere the weights ribution .Random-ER and included in the analyses. Moreover, we included one further method, namely the FR-Effects model, based on a combination of Fixed and Random-Effects models. In short, the FR-Effect model first estimates p-value < 0.1) the Random-Effects model is used, otherwise the Fixed-Effects is used.These five approaches were implemented in s Q test  p-value Combat is a method specifically devised for removing batch effects in gene-expression data [ion data . This meIn contrast with meta-analysis, the data-merging approach pools all data together and then estimates statistics on the resulting dataset. Expression profiles measured in different studies, or even in the same study but in different batches, present systematic variations in their distribution , and theysjk is the expression of gene j in sample s in batch k, aj is the overall gene expression of j, X and βj are respectively the design matrix and the gene-specific coefficients vector, while the remaining terms are the additive and multiplicative batch effects, respectively. These effects are estimated through an approach that uses hyper-priors and pool information across all available probesets. We used the Combat implementation of the R package sva in all analyses.RMA  is an aRMA-Combat. We also include the hybrid solution RMA-Combat, consisting in a pipeline that first applies the RMA method and then Combat.Surrogate Variable Analysis (SVA). The SVA approach introduced by Leek and Storey [ysj is the expression of gene j in sample s, aj is the overall gene expression of j, xs is a binary variable indicating whether sample s is a case or a control, βj represents the average difference in expression between the two conditions in gene j, and ϵsj is a random error. The term ysj of K unknown confounding factors gks, multiplied by their gene-specific coefficients γjk. SVA attempts to estimate confounding factors’ global effect by deriving a set of surrogate variables h1, h2, …, hK whose span covers the same linear space spanned by the vectors gk. These surrogate variables can then be used as covariates in all subsequent analysis in order to rule out the effect of the unknown confounding factors.d Storey  attemptsi has a significant effect only on a restricted subset of genes, all major systematic variations involving a large portion of transcripts should be due to experimental factors, batch-effects or confounding factors. Given this assumption, for the data collections used in this study the SVA model becomes: Ct,SVA and Pt,SVA. This means that Ct,i,SVA is a partial correlations [TFt and gene i given the information embedded within the surrogate variables.To the best of our knowledge, no previous study applied SVA on gene-network reconstruction, and a detailed discussion about how to adapt SVA for this task is reported in the Additional file elations , quantifScaling the expression values of each dataset so that all genes have the same mean and standard deviation is a further suitable approach. In particular, we scale the expression of each probeset in each dataset to zero mean and unitary standard deviation.No-correction. The naïve solution of pooling all data together without removing systematic variations is included in the analysis as well, in order to contrast the effectiveness of the other methods.where Single-Dataset approach, consisting in separately analyzing each dataset and then averaging the performance within each data collection. More in detail, let πΠ1 … πΠm be the performances obtained on datasets D1, …, Dm in collection M by using the metric Π. The Single-Dataset approach calculates a weighted performance A relevant question is whether employing complicate statistical techniques in order to co-analyze several datasets actually provides any advantage with respect to analyze a single dataset in isolation. DM and MA methods heavily process the data, following assumptions that are not always satisfied. Consequently, these methods may induce biases rather than remove batch-effects. To answer this question we adopted a Random-Guessing approach consisting in randomly sampling Ct,i from a uniform distribution. Theoretically, we expect this method to achieve the lowest performances among all other algorithms.Finally, we also include a Generalizing the results of this work to any network learning algorithm is out of the scope of this paper. However, we perform a proof-of-concept application in order to provide initial evidence that the results obtained in the context of relevance networks, arguably the simplest type of reverse engineering networks, are also valid when more complicated algorithms are used.To this purpose, we analyze a set of Peripheral Blood Mononuclear Cells (PBMC) gene expression datasets extracted from GEO. We attempt to reconstruct the regulatory network of the Ikaros transcription factor by applying the SES  algorithm . The preThe following sections describe in detail the used data and the analysis pipeline.We assembled a compendium of seven public microarray gene expression datasets of human PBMC. PBMC are the populations of blood cells having a round nucleus that constitute a pivotal part of the peripheral immune system. These include lymphocytes , monocytes, macrophages, dendritic cells. Their abundance and the simplicity of their extraction (an intravenous injection is sufficient for collecting a sample) render them interesting candidate for scientific studies. Note that the selection of human microarray datasets serves for further testing the validity of our results in the spectrum of human subject studies.For assembling this compendium, only studies comprising randomly-selected healthy-control subjects were taken in consideration. In particular, for each study only the control group was retained for our analysis. The idea is that control groups formed by randomly chosen healthy individuals can be considered as independent sampling from the same population, and are thus suitable for being analyzed through MA and DM methods. In total, the collection counts 181 expression profiles all measured with the Affymetrix Human Genome U133 Plus 2.0 Array (41245 probesets). The expression of Ikaros is measured by nine of these probesets. We used in turn each of these probesets and we merged together their respective networks.IIkaros of Ikaros regulatory relationships was built from literature information and computational analyses. Particularly, we built a list IIkaros containing 2658 unique interactions by merging together 2497 Ikaros targets identified through Chip-seq and microarray analysis [Finally, a list analysis  along wianalysis , followianalysis . Brieflyanalysis , and DHSanalysis . Transcranalysis  and the analysis . Genes wThe presence of different cell-types in the PBMC samples implies that expression values are averaged over a mixture of different distributions. Subjects included in each study may have significantly different cell proportions, and this in turn may generate correlations among probesets that do not reflect any underlying gene-gene interaction . In ordeThe SES algorithm  as impleSES belongs to the class of constraint-based, Bayesian Network reconstruction algorithms . While rSES requires the user to set a priori two hyper-parameters, a threshold for assessing p-values significance and the size of the maximum conditioning set. In our analyses these hyper-parameters were set to 0.01 and 5, respectively. The signatures found on single probesets were merged together, as well as the results retrieved on the nine different probesets measuring Ikaros.Based on our previous findings, we picked the Combat and Fixed-Effects methods as representatives for the DM and MA approaches, respectively. We also used the No-Correction and Single-Dataset approaches in order to characterize the scenarios where batch-effects are ignored or a randomly chosen dataset of the PBMC collection is analyzed in isolation. For the Combat and No-Correction approaches the deconvolution and outlier deletion steps were performed on their respective merged datasets, while for the Fixed-Effects and Single-Datasets methods the two pre-processing steps were performed independently for each study of the PBMC collection.QIkaros,X be the list of Ikaros interactions retrieved using SES couple with the MA or DM method X, and ¬ IIkaros the list of genes that are not part of Ikaros regulatory network (|IIkaros ∪ ¬ IIkaros| = n). Precision is defined as Network reconstruction performances were measured in terms of precision, recall and odds ratio. Let n genes as belonging to the Ikaros regulatory program. An odds ratio of one indicates performances that are indistinguishable from random guessing, and we used a hypergeometric test [H0 : OddsRatioX = 1.The odds ratio quantifies the likelihood that a given proportion of regulatory relationships is retrieved by chance, and is computed as ric test  in orderp-value < 0.05, one-tailed paired t-test).Figure All four panels present a similar picture, with several DM and MA methods clustering together and achieving comparable performances, while the Random-Guess, Single-Dataset and No-Correction approaches usually providing significantly worst results. Best performing methods usually present a variability that is smaller than the one of the outperformed methods.All in all, the results show that systematic biases across studies must be taken into account for retrieving gene-gene interactions, and that both MA and DM approaches are effective in dealing with such systematic variations.Retrieving gene-gene interactions in the Yeast dataset collection have proven to be harder than in E. coli. Performances were generally poorer, with AUC and pAUC values up to 5 point inferior than the corresponding performances in the E. coli compendium, and both AUPRC and AUFDR ranging far below 0.05.p-values < 0.05 are indicated with filled markers. The coefficient of variability for each method determines the color of the corresponding marker, with lighter color corresponding to higher CV.Results are further summarized in Fig. The SVA, Combat, RMA-Combat, Fixed-Effect and Scaling methods are confirmed as the best performing methods, occupying the first position in the Rank-Product analysis. SVA shows a relatively high variance, indicating that sometimes it fails in reaching the top positions in terms of performances. The Random-Guess approach is stable in last position, followed by the Single-Dataset, Stouffer and No-Correction methods. The two bottom panels in Fig. Similar figures restricting the Rank-Product analysis to Global and Local performances only, as well as Pearson and Spearman correlations and E. coli versus Yeast are available in the Additional file p-values Pt,X instead of the correlation values Ct,X. In E. coli there is a dramatic worsening in performances for most of the methods. A decrease in performances can also be observed for the Yeast compendium, although to a lesser extent. A possible explanation for these patterns is the presence of several high-significant correlations, whose corresponding p-values are exactly zero or too low to be distinguished at machine precision. These zero p-values create ties that severely affect the ranking of the candidate interactions and consequently the evaluation of the performances.Figure Ct,X and Pt,X -based results have a large percentage of p-values that are exactly zero. SVA, Rank-Product and Fixed-Effects methods do not produce zero p-values, and they do not suffer any performance loss. However, Random-Effects and FR-Effects do not produce zero p-values as well, and they still achieve worse performances when Pt,X is used instead of Ct,X. The answer to this issue lays in the fact that there is not a bijective correspondence between Ct,X and Pt,X for the Random-Effects methods, and consequently neither for the FR-Effects one. In other words, if Ct,i > Ct,j holds, then Pt,i < Pt,j holds as well if the correlations are computed with the Fixed-Effects model, but not if they are computed with the Random-Effects method. The statistical significance of correlation in the Random-Effects approach depends on the estimation of the between-study variance A close inspection of the results seems to confirm this hypothesis. Table AUC ≈ 0.5 for the Random-Guess approach in all cases. The remaining MA and DM methods achieve comparable performances, both in terms of average performance and respective variance. SVA seems to be an exception, thought, achieving quite lower performances. Quite surprisingly, SVA performances drop significantly with the maximum total sample size, i.e., when 50 studies with 50 samples each are analyzed . Concomitantly, the number of surrogate variables estimated in these setting is ~60, versus ~5-10 when the total sample size is lower. We argue that such an elevated number of surrogate variables negatively affects the computation of conditional correlations, leading to a worsening in performances.The results on simulated data for the AUC metric are reported in Figs. Pt,X show a decrease in performance  to random guessing . We correlated the odd ratios and p-values achieved on each dataset with the sample size, and interestingly no association was detected .Table IIkaros.Figure In the present work we have compared two different approaches, Data-Merging and Meta-Analysis, on the reconstruction of relevance networks in collection of microarray, gene-expression data. The comparison has been performed on two compendia of studies retrieved from the literature, on Escherichia coli and Yeast, respectively. Further analyses on simulated data have been used for strengthening and deepening the conclusion of the comparison. Finally, a contrived case-study on human PBMC data have been presented for showing how the results of this study might transfer on more sophisticated network reconstruction approaches.Batch-effects must be carefully taken into consideration for retrieving gene-gene interactions from microarray data. The naïve solution of ignoring systematic biases (No-Correction approach) was outperformed by the other methods in all experimentations. This result supports our claim that batch-effects can hide actual dependencies between the measured quantities or create spurious associations between elements that are not functionally related.DM and MA methods are equally effective in contrasting batch-effects. According to the results it is not possible to state that one approach is universally better than the other one. However, within their respective approaches, and acknowledging that the results vary across the performed experimentations, the SVA/Combat/RMA-Combat and the Fixed-Effects methods have usually achieved the best performances. In contrast, the Single-Dataset method usually provides poorer results, supporting the hypothesis that integratively analyzing multiple datasets leads to improved and more robust findings.Correlation statistics should be preferred to p-values in ranking associations. Performances have proven to drastically change depending on whether they are computed on correlations or p-values. We have observed that this effect is mainly due to ties generated by zero or close to zero p-values.The results on both simulated and real data provide coherent conclusions, which can be summarized in the following points:p-values for ranking gene-gene interactions. A possible practice is to filter the candidate interactions by using the p-values and then raking the most significant gene-pairs according to their correlation values.This study presents a number of limitations that should be carefully considered when implementing the recommendations above. First, within-study batch-effects were only partially addressed, by pre-processing each single dataset with RMA. While the Quantile Normalization step included in the RMA algorithm should have removed at least part of the within-study biases, it is known that this approach is not optimal . This isp-values quite inaccurate. Further studies are needed in order to better investigate this phenomenon.The SVA method merits a separate note. To the best of our knowledge, this is the first study employing this methodology in the context of retrieving gene-gene interactions. Adapting SVA for this task has required a dedicated sub-study, reported and commented in the Additional file Future work will also focus on the generalization of the present results towards more sophisticated network reconstruction algorithms, particularly Bayesian and Causal Networks . We alrep-values for assessing and comparing the strength of associations, especially for large sample sizes.Batch-effects should be carefully taken into account when retrieving gene-gene interactions, and researchers can adopt either a DM or MA approach depending on the specific application at hand. Correlation statistics should be preferred over The data sets use in this article are available from their respective repositories. See Tables S1 to S3 in the Additional file http://www.mensxmachina.org/.Code for replicating the analysis is available at Additional file 1:The Supplementary Material provides additional data and results supporting the conclusions of the study, including detailed descriptions of the E. coli and Yeast data compendia as well as all results produced on these compendia. (DOCX 4691 kb)Additional file 2:Using Surrogate Variable Analysis for Network Reconstruction. This additional file presents a sub-study investigating modifications of the SVA model that allow to use the SVA method on network reconstruction tasks. (DOCX 137 kb)Additional file 3:Simulations Results based on correlations. The Simulation Results table presents the results obtained on the synthetic data by using correlations as measure of association. (XLSX 1787 kb)Additional file 4:Simulations Results based on p-values. The Simulation Results table presents the results obtained on the synthetic data by using the correlation p-values as measure of associations. (XLSX 1885 kb)


Gene expression profiling (GEP) via microarray analysis is a widely used tool for assessing risk and other patient diagnostics in clinical settings. However, non-biological factors such as systematic changes in sample preparation, differences in scanners, and other potential batch effects are often unavoidable in long-term studies and meta-analysis. In order to reduce the impact of batch effects on microarray data, Johnson, Rabinovic, and Li developed ComBat for use when combining batches of gene expression microarray data.We propose a modification to ComBat that centers data to the location and scale of a pre-determined, ‘gold-standard’ batch. This modified ComBat (M-Combat) is designed specifically in the context of meta-analysis and batch effect adjustment for use with predictive models that are validated and fixed on historical data from a ‘gold-standard’ batch.We combined data from MIRT across two batches (‘Old’ and ‘New’ Kit sample preparation) as well as external data sets from the HOVON-65/GMMG-HD4 and MRC-IX trials into a combined set, first without transformation and then with both ComBat and M-ComBat transformations. Fixed and validated gene risk signatures developed at MIRT on the Old Kit standard  were compared across these combined data sets.% of all untransformed probes were significantly different by ANOVA with 0.01 q-value threshold reduced to zero significant probes with ComBat and M-ComBat). The agreement in mean and distribution of risk scores, as well as the proportion of high-risk subjects identified, coincided with the ‘gold-standard’ batch more with M-ComBat than with ComBat. The performance of risk scores improved overall using either ComBat or M-Combat; however, using M-ComBat and the original, optimal risk cutoffs allowed for greater ability in our study to identify smaller cohorts of high-risk subjects.Both ComBat and M-ComBat eliminated all of the differences among probes caused by systematic batch effects (over 98M-ComBat is a practical modification to an accepted method that offers greater power to control the location and scale of batch-effect adjusted data. M-ComBat allows for historical models to function as intended on future samples despite known, often unavoidable systematic changes to gene expression data. Multiple myeloma (MM) is a systemic hematopoietic malignancy of plasma cells that expands within the bone marrow. As plasma cells become cancerous and multiply, levels of monoclonal immunoglobulins in blood increase and osteolytic bone lesions form in the majority of patients. Myeloma is also characterized by genetic heterogeneity at onset as well as variability in patient response to treatment –survivalGene expression profiling (GEP) via microarray analysis measures the expression levels of tens of thousands of genes simultaneously, and has been widely used in clinical practice for cancer classification, risk stratification, and treatment selection. This technique involves analyzing RNA extracted from purified plasma cells, pulled from the bone marrow, on high-density microarray gene chips ,3. Many Ambient conditions during sample preparation and handling such as room temperature, humidity, and ozone levelsSites/laboratories: different laboratories may have different operating proceduresStorage/shipment conditionsSample preparation : different reagent lots may perform differentlyScanner: type, settings, and calibration drift ,15.Batch effect corrections aim to eliminate systematic, non-biological differences that may be introduced from a variety of sources. Some of the main contributors to batch effects include:Despite knowing the many potential sources of batch effects, batch effect associated information is not necessarily recorded for all samples. Oftentimes, data analysts are left with surrogates such as processing date and preparation group to use when correcting for batch effects . Here, wCombining batches of samples can be performed both within a single study or on samples from different studies in a meta-analysis framework. The remainder of this paper will focus on the latter scenario of combining larger cohorts of patients across different studies with a modified alternative of ComBat. The method introduced below, M-ComBat, would work in smaller sample, single cohort situations; however, ComBat or other methods may be more applicable in the single cohort context depending on the focus of the study.n=928) and ‘New’ Kit samples (n=143). Additional external data sets will be used to validate both the M-ComBat method and the GEP5, GEP70, and GEP80 risk signatures. The publicly available data set of previously untreated patients  will be used as well as baseline data from the MRC-IX trial (n=273) [All MIRT samples are baseline purified plasma cells from the bone marrow of patients enrolled on UARK Total Therapies and processed on Affymetrix U133Plus 2.0 microarrays  between 2004 and 2014. In keeping with institutional, federal, and Helsinki Declaration guidelines, all identifiable patients gave written informed consent for undergoing bone marrow sampling for gene expression profiling and the institutional review board of the University of Arkansas for Medical Sciences approved the research studies. Samples analyzed at the Myeloma Institute were prepared with either the One-Cycle and Two-Cycle Target Labeling and Control Reagents (‘Old’ Kit) or, beginning in 2009, the 3’ IVT Express Kit (‘New’ Kit) . All bas (n=273) . Raw intHere we will treat the MIRT: Old Kit data as our ‘gold-standard’, reference batch because it was used to train and validate the GEP5, GEP70, and GEP80. When using M-ComBat, the designation of the reference batch should relate to focus of investigation when combining data across batches. Here we are investigating three risk signatures  which are based on a particular standard (MIRT: Old Kit). Alternatively if we were investigating the EMC-92 risk signature, we would choose the HOVON-65/GMMG-HD4 data as the reference batch as it is the training set of the EMC-92. In a meta-analysis context where we are performing new analyses or searching for new signatures on combined data, there are no set rules on choosing a reference batch; however, it may be best to choose the largest cohort, most relevant, or most familiar as the reference batch. However, the only benefit of M-ComBat over Combat in that context is that a newly discovered signature would be fixed to a known standard rather than the arbitrary location of ComBat transformed data.ComBat  is a higg, sample j, and batch i. Yijg refers to the raw expression data, and γig and ComBat combines expression data by first standardizing the data, given byComBat centers data to the overall, grand mean of all samples which results in an adjusted data matrix that is shifted to an arbitrary location that no longer coincides with the location of any original batch. When using validated and fixed gene signatures on ComBat transformed data, these gene signatures on ‘gold-standard’ data will also be shifted. Many of these ‘gold-standard’ samples were likely used to train and build the gene signature; therefore, altering these samples when combining data sets will also alter the performance and proportion of high-risk individuals identified for a fixed risk signature on the original, reference training data.In this study, we propose a modified version of ComBat (M-ComBat) that shifts samples to the mean and variance of the ‘gold-standard’, reference batch, rather than the grand mean and pooled variance. This is achieved by changing the standardizing mean and variance estimates, i=r.Furthermore, the mean and variance estimates used in the final batch effect adjusted data are calculated using the gene-wise mean and variance estimates of the ‘gold-standard’, reference batch, The M-ComBat adjusted data is then given byR using a modified version of the ComBat script from the sva package. The altered script  is available online for public use at http://github.com/SteinCK/M-ComBat.M-ComBat can be applied in We will illustrate both ComBat and M-ComBat as well as the GEP5, GEP70, and GEP80 risk signatures by transforming baseline purified plasma cell GEP samples from UARK Total Therapies across two kinds of sample preparation (Old and New kits) as well as two external data sets (HOVON-65 and MRC-IX). ComBat will be performed assuming a parametric model with no covariates. The four distinct batches will be shifted by M-ComBat to the MIRT: Old Kit ’gold-standard’ as this was the standard of data used to develop and train the GEP5, GEP70, and GEP80 signatures.% of all probes were significantly differently expressed across at least one batch  was performed on the 5,000 most variable probes for the untransformed, ComBat, and M-ComBat transformed data sets. PCA creates convex linear combinations of a set of observations that are orthogonal and defined in such a way that the components are ordered by variance where the first principal component has the largest variance, the second the next largest variance, etc. Scatterplots of the top two principal components show how both ComBat and M-ComBat remove the differences in probe expression between the batches while shifting the data to different locations Figure . ComBat Differences in the distribution of risk scores across batches are eliminated for GEP5, GEP70, and GEP80 using either ComBat or M-ComBat Figures  and 3. M%, while all other non-transformed data sets have over 43% of patients defined as high-risk  . A similar trend occurs for the other scores, and in general M-ComBat allows for original optimal cutpoints to continue to function well on external data sets for binary, high-low risk scores. Alternatively, ComBat adjusts all data sets, so the proportion of high-risk individuals may change–even in our reference, ‘gold-standard’ batch.The GEP80 is more susceptible to the impact of batch effects, especially in the proportion of patients designated with high risk. The proportion of high-risk GEP80 patients on the untransformed MIRT: Old Kit is 7.5% for MRC-IX and 32.0% for HOVON-65). M-ComBat allows for improved performance of risk scores on external data sets and allows for risk scores to function as intended.In general, hazard ratios from Cox regression of overall survival increase and Wald p-values decrease following ComBat and M-ComBat transformation Figure . The GEPM-ComBat eliminated the overwhelming statistical differences among probes caused by systematic batch effects. In addition to eliminating differences in probes, the agreement in mean and distribution of risk scores coincided with that of the ‘gold-standard’ batch with M-ComBat. The proportion of high-risk patients identified following M-ComBat for external data sets more closely matched that of the reference group as well. The performance of risk scores improved overall using either ComBat or M-Combat; however, using M-ComBat and the original, optimal risk cutoffs allowed for greater ability in our study to identify a smaller cohort of high-risk subjects.By eliminating the differences in means between all probes, all mean risk scores examined coincided with one another across batches following M-ComBat. This allows for potentially increased power in the risk score on external data sets following M-ComBat transformation. M-ComBat provides the structure to compare risk scores more fairly and realistically across different data sets lending itself directly to the further validation and comparison of yet discovered gene signatures scores.Modified ComBat eliminates batch effect related differences while adjusting all samples to a pre-defined standard. The modification transforms samples to the mean and variance of the ‘gold-standard’ batch without altering samples from the original batch. This allows for validated and fixed predictive models to function properly on external data sets and offers a less biased framework for comparing gene signatures. Rather than redefining cutpoints of GEP-based scores for new batches, M-ComBat allows for the same cutpoints and logic used to build these GEP-based scores to function properly on external data sets.Differences in scanner, laboratory, and sample preparation contributed significantly toward systematic batch effects seen when combining gene expression samples across these four batches. The non-biological variation introduced by differences in sample preparation kit, scanner, and laboratory had dramatic impacts on overall GEP expression (seen in PCA plot) as well as the smaller subsets of probes that define gene signatures. These variations are often unavoidable in long-term studies, and both ComBat and M-ComBat are equally as effective in eliminating these confounding batch effect related differences. However, M-ComBat offers an intelligent framework to overcome these changes and the power to control the location and scale of the transformed data.M-ComBat allows for gene expression meta-analyses to combine data from different studies to a known standard. The benefit of a fixed standard in meta-analysis is important for gene signature comparison, differential expression analysis, and the search for new gene signatures on a larger subset of combined samples on a known standard.M-ComBat is a practical modification to an accepted method that offers greater power to control the appearance of batch-effect adjusted data. M-ComBat allows for historical models to function as intended on future samples despite known, often unavoidable systematic changes to gene expression data.