Systematic processing noise, which includes batch effects, is very common in microarray experiments but is often ignored despite its potential to confound or compromise experimental results. Compromised results are most likely when re-analysing or integrating datasets from public repositories due to the different conditions under which each dataset is generated. To better understand the relative noise-contributions of various factors in experimental-design, we assessed several Illumina and Affymetrix datasets for technical variation between replicate hybridisations of Universal Human Reference (UHRR) and individual or pooled breast-tumour RNA.A varying degree of systematic noise was observed in each of the datasets, however in all cases the relative amount of variation between standard control RNA replicates was found to be greatest at earlier points in the sample-preparation workflow. For example, 40.6% of the total variation in reported expressions were attributed to replicate extractions, compared to 13.9% due to amplification/labelling and 10.8% between replicate hybridisations. Deliberate probe-wise batch-correction methods were effective in reducing the magnitude of this variation, although the level of improvement was dependent on the sources of noise included in the model. Systematic noise introduced at the chip, run, and experiment levels of a combined Illumina dataset were found to be highly dependant upon the experimental design. Both UHRR and pools of RNA, which were derived from the samples of interest, modelled technical variation well although the pools were significantly better correlated (4% average improvement) and better emulated the effects of systematic noise, over all probes, than the UHRRs. The effect of this noise was not uniform over all probes, with low GC-content probes found to be more vulnerable to batch variation than probes with a higher GC-content.The magnitude of systematic processing noise in a microarray experiment is variable across probes and experiments, however it is generally the case that procedures earlier in the sample-preparation workflow are liable to introduce the most noise. Careful experimental design is important to protect against noise, detailed meta-data should always be provided, and diagnostic procedures should be routinely performed prior to downstream analyses for the detection of bias in microarray studies. Several recent articles have provided substantial evidence that systematic noise introduced as a result of batch-processing have a detrimental effect on data derived from microarrays -5. Complet al.  revealedet al.  none of et al. . Severalet al. ,10. The et al. , as wellet al. ,13. Theset al. ,15. It het al. , distancet al. , or empiet al. .Awareness of and protection from batch effects is of particular importance in meta- or re-analyses of existing data, collated from multiple sources, where the intention is to exploit increased statistical power to detect subtle differences in expression between well-defined phenotypes. Direct comparison and integration of gene expression data through re-analyses is a highly attractive option due to data repositories such as the NCBI Gene Expression Omnibus (GEO)  and ArraGiven previously reported variation in various array technologies ,15,20-23ComBat [In order to establish the relative impact of different sources of systematic noise we compared the expression profiles and numbers of consistently differentially expressed genes when compensating for variation introduced at a single or at multiple steps in the Affymetrix GeneChip sample-processing workflow. The dataset, previously reported in , consistComBat ) is perfVariance analysis of this dataset provided a more detailed breakdown of the estimated biological and technical error introduced at the various stages of sample-preparation Figure . The majComBat. The high inter-chip variation, compared to inter-run variation, appears to be driven by the new samples run on the HT-12 chips as this was not observed in the standalone analyses of the Ref-8 data alone. As with the Affymetrix data we explored the effect of background correction on these variances  dataset . ExpressThe Pearson-correlation heatmaps in Figure ComBat and correction at two levels was assessed: experiment-wise and run-wise correction. Following ComBat correction by experiment, all pairwise correlations increased to approximately the same level observed in the quantile-normalised Ref-8 data obtained from the first experiment . We also performed an exhaustive comparison of the fold-change in expression due to the different runs between the replicate pools of pre-treatment tumour RNA and each of the fold-changes between duplicate tumour samples . Several descriptive statistics were selected to serve as measures of probe- and mapping-specific properties that could conceivably influence probe expression -28. Thes2 p-value < 2.2 * 10-16). Taking this further, we correlated probe GC-fraction with error estimates from the Ref-8/HT-12 UHRR replicates  and inter-run  levels, but not at the inter-chip level  and at the inter-chip level . As with the Ref-8/HT-12 data, there was no significant enrichment at the lowest level between replicate arrays. To determine whether this low-GC-high-SD association was resolvable between biological as well as technical replicates, we chose a subset of pre-and post-treatment samples from our Illumina Ref-8 dataset . Despite a slightly greater number of probes with low-GC-high-SD in both the pre- and post-treatment sample-sets, the trend was not significant (data not shown).We also compared probe GC content with our variance estimates from the MAQC Illumina and Affymetrix datasets, both of which have a similar overall distribution of probe-GC content, skewed in favour of higher GC fraction (data not shown). A significant enrichment for low-GC-high-SD probes was again observed, both at the inter-laboratory level  we found that the correlation between replicate UHRR hybridisations were consistently poorer than correlations previously reported using the Ref-8 data alone . InteresComBat [ComBat corrections.As in our previous investigation , quantilComBat  once agaVariance estimates using the MAQC Illumina dataset were smaller in magnitude to the variances obtained from our Ref-8 data, revealing a surprisingly high level of reliability between the three laboratories that performed these experiments. In contrast, the MAQC Affymetrix dataset was found to be far more variable than their Illumina data but similar to the magnitudes of variance observed in both our combined Ref-8/HT-12 dataset and our Affymetrix dataset. The reason for the low variation in the MAQC Illumina data is unknown, especially since their study design deliberately split the sample replicates before cRNA synthesis; a much earlier stage in the sample-prep workflow than our replicates (which were split after amplification and labelling). It is possible that the small number of laboratories  performing the MAQC Illumina hybridisations produced highly concordant data completely by chance, while the larger number of laboratories  performing the Affymetrix experiments provided a more realistic reflection of the technical variation in these data.Several studies have found the use of replicate control samples such as UHRR to be a useful standard in microarray experiments, suitable for monitoring expression consistency within and across a variety of genome-wide expression platforms -32. HoweComBat correction than the UHRR controls. If a similar pooled calibrator was used in our previous study it seems reasonable to speculate that the consistency between the gene-lists reported as significantly differentially expressed would have been noticeably higher than the 74.1% achieved using UHRR calibrators.Unfortunately the relatively small degree of legitimate biological differential expression between the pre-and post-treatment tumour biopsies provided little opportunity to assess the relative performance of UHRR and pooled batch calibrators on the consistency of reported differentially expressed probes. However, compared to the UHRR, the pooled tumour RNA controls were shown to more faithfully emulate the individual shift in expression between tumour technical-duplicates as a result of variation introduced between runs and between chips. Had it been possible to identify more legitimately differentially expressed probes between pre- and post-treatment samples, the pooled RNA would almost certainly have made for a better batch-calibrator during We took the opportunity to assess compositional properties of the probes as a potential explanation/surrogate for the technical effects observed in the Ref-8/HT-12 data. A highly significant trend in favour of low-GC content was identified in the core set of probes consistently affected by inter-run and inter-chip variation between sample duplicates. A similar, but less significant, enrichment for low-GC-high-SD probes was also observed in the MAQC Illumina dataset. This suggests that the magnitude of error introduced due to low probe GC-content is sufficiently great that it is resolvable between the replicate cRNA preparations assessed in the MAQC study. A similar observation regarding probe GC-content and expression consistency was recently reported in a comparison of RNA preservation protocols, using matched samples, in terms of the effect on results of downstream expression analyses . A furthThe proximity, with respect to the target transcript, of probes has been reported to strongly influence the correlation of expression measurements between technologies . The anaThe key source of systematic error in any given microarray experiment is unpredictable, but can generally be attributed to RNA extraction and, to some extent, labelling and amplification protocols. The results presented here, including those derived from external data sources, suggest that it is not recommended to analyse individual test samples (e.g. to then try and classify), but instead to run several at the same time to get a better estimate of the experimental variation in order to be better equipped to compensate for it.Sound experimental design is of critical importance to avoid confounding systematic variation. Randomisation of samples over arrays on each BeadChip and careful blocking of samples from each group of interest within and across runs is necessary to protect against confounding systematic error. In situations where this is not possible, detailed meta-data should be preserved for each array that includes, at the very least, the date and time of each hybridisation and scan. In either case, diagnostic procedures such as PCA or SVA should be routinely performed prior to downstream array analyses.RNA was extracted using the RNeasy Mini Kit, including RNase-Free DNase treatment  from paired breast tumour tissue, alternatively a Universal Human Reference RNA  was used. RNA was amplified and biotinylated using Illumina TotalPrep RNA Amplification Kit (Ambion) and quantified on an Agilent 2100 Bioanalyser. 750 ng cRNA per sample was hybridised to Illumina Human Ref-8 v2 or Human HT-12 v3 expression BeadChips  using the Whole-Genome Expression Direct Hybridisation kit (Illumina) and scanned with a BeadStation 500GX (Illumina). Duplication of the clinical samples was performed after labelling and labelled samples were stored as per the manufacturer's recommendations.In addition to previously described samples ,37, fourhttps://catissuesuite.ecmc.ed.ac.uk/caarray/ and on request to the corresponding author.Methods for generating the MCF7 and MCF10A triplicate Affymetrix U133A data can be found in the original publication . MethodsQuality control analyses revealed no systematic or individual problems with the Affymetrix or Illumina data at the array level as assessed by the array-level distributions of signal intensities  using Bowtie   using the detection confidence reported by All Illumina data were detection-filtered and normalised together  before assessment by pairwise Pearson correlation. Results of pairwise Spearman rank correlations following separate filtering and normalisation of the Ref-8/HT-12 chips were very similar (data not shown). The globally filtered/normalised data were used in all subsequent UHRR analyses for consistency with our previous results and for a more reliable interpretation of nested variance analyses.R using functions contained within the affy package. Expression values were produced from probe in intensities using the expresso function with, unless stated otherwise, RMA background correction, quantile normalisation, and median-polish summarisation over probesets. Following this preprocessing, probes were filtered such that those reported 'absent' in more than 20% of the samples were removed.Affymetrix data were processed in nlme package in R [We applied a linear additive model to the Affymetrix and Illumina expression data on the log-scale to estimate the specified variance contributions. These contributions are assumed to be independent and randomly drawn from log-normal distributions. As all factors meet in unique combinations a nested, or hierarchical, variance model is individually applied for each gene. Models of this kind are formally defined in  and haveage in R -44. In aThe main difference between standard array normalisation and batch-correction is that the latter does not make the assumption that all probes are affected equally by batch-effects and, as such, performs individual adjustments on each probe across all samples. The ComBat approach, as discussed in detail in , is a soR (v.2.12.1) [ComBat [limma (v.3.6.9) [SAM (v.1.28) [limma, empirical Bayes variance shrinkage was employed and genes were defined as being differentially expressed after satisfying a minimum fold-change of ± 1.5 and a maximum, Benjamini-Hochberg adjusted, p-value of 0.01. For the SAM analysis, the differentially expressed genes were selected, following 100 permutations, at a maximum predicted false discovery rate of 5% and the same minimum fold-change of ± 1.5.Gene expression changes were compared before and after treatments and between responders and non-responders using Bioconductor  algorith.2.12.1) . Illumin [ComBat . Genes dv.3.6.9)  and SAM (v.1.28) . For thehttps://catissuesuite.ecmc.ed.ac.uk/caarray/R scripts and data archives can be downloaded from. RRK and AHS conceived the study, performed the analyses, and drafted the manuscript; JMD supervised the tumour biopsies; VSS performed the sample-prep prior to the array analyses; and AAS, JMSB, and AHS supervised and funded the work. All authors read and approved the final manuscript.Supplementary material S1. Comparison of estimated variance components in Affymetrix and Illumina data with and without background correction as part of the array pre-processing.Click here for fileSupplementary material S2. Illustration of our Illumina Ref-8 (experiment 1) and HT-12 (experiments 2 & 3) BeadChips, processed in eight batches  corresponding to the different days on which the samples were hybridised and scanned. UHRR samples are labelled as C1-25. Replicate breast tumour clinical samples are identified with a suffix of 'a' through 'd'. The pre- and post-treatment biopsy samples are identified by a triangle to the left and right of the sample IDs, respectively.Click here for fileSupplementary material S3. Correlation of expression change as a result of inter-run and inter-chip technical variation between UHRR and pooled controls with tumour duplicates. Tumour duplicates  are arranged on the x-axis to be close to others processed on the same BeadChip. UHRR and both types of pooled-control  are correlated more strongly with individual tumour duplicates in which one 'half' of the duplicate was processed on the outlying chip 22. Pooled-controls also consistently score slightly higher correlation than UHRR.Click here for fileSupplementary material S4. Scatter plots of fold-changes between tumour duplicates and replicate pools. These plots each show the magnitude of the change in expression between technical replicates introduced by the runs and show how well correlated such changes are between the Pool/tumour-duplicates. In the figure, the pre-treatment tumour duplicate are coloured blue and post-treatment are green; probes that are differentially regulated, up or down, at least two-fold due to the different runs are highlighted in each plot (red points). Note that most of the samples with a duplicate on chip 22 are subject to a greater magnitude of variation than samples with a duplicate on chip 21.Click here for fileSupplementary material S5. Plot of GC-content vs. .Click here for fileSupplementary material S6. Distribution of MAQC Illumina probes as a fraction of target gene length. Light blue points are used for probes that mapped to the antisense DNA strand and dark blue for the sense strand. Left plot represents the fraction of target gene length in terms of 3' and 5' coordinates on each strand while the right plot is 'normalised' for the anti-sense strand and plots the fraction in terms of absolute position along the DNA molecule.Click here for fileSupplementary material S7. Probe position against probe standard deviation: Plots of probe position against probe standard deviation estimated at the inter-laboratory (A), inter-chip (B), and inter-array (C) levels in the MAQC Illumina dataset. Light and dark blue points again identify probes that mapped to the antisense and sense strands, respectively.Click here for fileSupplementary material S8. Quality control metrics over all of our Affymetrix and Illumina arrays including array-level intensity distributions, Illumina bead-standard errors, and bead-representation distributions between the two Illumina array-versions.Click here for file


An increasing number of genomic studies interrogating more than one molecular level is published. Bioinformatics follows biological practice, and recent years have seen a surge in methodology for the integrative analysis of genomic data. Often such analyses require knowledge of which elements of one platform link to those of another. Although important, many integrative analyses do not or insufficiently detail the matching of the platforms.We describe, illustrate and discuss six matching procedures. They are implemented in the R-package sigaR (available from Bioconductor). The principles underlying the presented matching procedures are generic, and can be combined to form new matching approaches or be applied to the matching of other platforms. Illustration of the matching procedures on a variety of data sets reveals how the procedures differ in the use of the available data, and may even lead to different results for individual genes.Matching of data from multiple genomics platforms is an important preprocessing step for many integrative bioinformatic analysis, for which we present six generic procedures, both old and new. They have been implemented in the R-package sigaR, available from Bioconductor. DNA copy number aberrations abound in the cancer cell. The location, size and direction of these aberrations vary between cancers of different tissues, between cancers of the same tissue, and may even exhibit heterogeneity among cells originating from the same tumor . The DNAThe elucidation of the relationship between DNA copy number aberrations and mRNA (and microRNA) transcript levels is key to enhance our understanding of the regulatory mechanism of the cancer cell. To this end, oncogenomic studies profile both the genome and transcriptome of a large number of tumors of the same tissue, of which ,3 are thcis- (and trans-) effect of DNA copy number changes on expression, we need, for each gene measured, its corresponding copy number information. If both profiles are generated on the same platform this is immediate. Pollack et al. . A gene’s association between gene dosage and expression levels is declared significant if its corrected p-value (Benjamini-Hochberg multiple testing correction) is smaller than 0.05. The associated workflow is portrayed in Figure Finally, we illustrate the effect of matching on downstream analysis. We assess the Matching of the features from different high-throughput platforms is a important preprocessing step for bioinformatic analyses of integrative genomics studies. We have described, reviewed and implemented (sigaR-package) the most widely used matching procedures found in literature. Application of the matching procedures to five data sets generated on different platforms revealed that 1) the number of features matched varies considerably between the matching procedures, and 2) the choice of matching procedure may even affect  the DNA copy number signature  assigned to a gene. These observations, which have their consequences on any downstream integrative analysis, facilitate an informed decision on the matching procedure of choice.The matching procedures have shown little difference in the number of features matched and have very little impact on downstream analysis results, in the several examples shown. These results rely on correct pre-processing, of which copy number data segmentation is an important aspect. It should be kept in mind that, although overall results may be robust to matching procedure selection, this may not be true for all genes, as Figure We recommend to start the matching with the overlapAny procedure. This may be conservative in some cases, but certainly has the clearest and most undisputed physical interpretation for matching. If this yields satisfactory results, the task is done. Else, remaining unmatched features may be handled either by the distanceAny (with not too big a window around the gene) or overlapPlus procedure.• Project name: sigaR• Project home page: http://www.bioconductor.org• Operating system: Platform independent• Programming language: R• Other requirements: R • License: General Public Licence (>= 2)• Any restrictions to use by non-academics: none declared.Data set 1: ChinSample type: Breast cancer.• Molecular levels: DNA copy number & gene expression.• Reference: Chin et al.[• in et al..DNA copy number platform: BAC, fabricated at UC San Francisco.• Gene expression platform: Affymetrix U133A.• Number of samples: 89.• Availability: CaBig repository.• Preprocessing: Pre-processing of both DNA copy number and gene expression data used here was as described in [• ribed in , with thribed in , using dData set 2: TCGA ISample type: Glioblastoma.• Molecular levels: DNA copy number & gene expression.• Reference: Verhaak et al. [• k et al. .DNA copy number platform: 244 K Agilent MSKCC.• Gene expression platform: Affymetrix 133A.• Number of samples: 55.• Availability: The Cancer Genome Atlas (TCGA): http://cancergenome.nih.gov/• Preprocessing: All samples from batch 1 to 3, for which both copy number data (244 K Agilent MSKCC) and level-e normalized expression data (Affy 133A) is available, were used. Replicated samples were not taken into account. Probe features were matched to genomic locations using Ensembl and biomaRt . The Affymetrix gene expression array contains 22277 probe features, of which 18607 could be matched to a genomic location. After removal of probes mapping to the Y chromosome, 18528 probes for the gene expression data were left. The Agilent copy number platform consists of 235834 probe features of which all features could be matched to a genomic location. 234416 were available after preprocessing in CGHcall [• ta Affy 1A is avaiNote: The DNA copy number data of this data set is identical to that of the TCGA II data set.• Data set 3: TCGA IISample type: Glioblastoma.• Molecular levels: DNA copy number & gene expression.• Reference: Verhaak et al.[• ak et al..DNA copy number platform: 244 K Agilent MSKCC.• Gene expression platform: Agilent custom 244 K.• Number of samples: 55.• Availability: The Cancer Genome Atlas (TCGA): http://cancergenome.nih.gov/• Preprocessing: All samples from batch 1 to 3, for which both copy number data (244K Agilent MSKCC) and level-2 normalized expression data (Agilent custom 244K) is available, were used. Replicated samples were not taken into account. Probe features were matched to genomic locations using Ensembl and biomaRt . The Agilent gene expression array is a custom design on the 244K platform with 90K unique probes. Only 36196 of the 90797 unique probes could be matched to genomic locations using Ensembl. After removal of probes mapping to the Y chromosome, 35582 probes for the gene expression data were left. The Agilent copy number platform consists of 235834 probe features of which all features could be matched to a genomic location. 234416 were available after preprocessing in CGHcall [·  CGHcall . SegmentNote: The DNA copy number data of this data set is identical to that of the TCGA I data set.• Data set 4: Taylor ISample type: Prostate cancer.• Molecular levels: DNA copy number & microRNA expression.• Reference: Taylor et al.[• or et al..DNA copy number platform: Agilent 244 K array CGH.• MicroRNA expression platform: Agilent microRNA V2.• Number of samples: 49.• Availability: GEO database .• Preprocessing: Data are limited to 49 prostate cancer samples of which DNA copy number, exon expression and microRNA expression (see data set Taylor I) was available. The Agilent 244 k array CGH raw data were provided as tab-delimitedtext files containing the normalised log2-ratios and the values of control measures such as background-to-foreground ratios, spot saturation, spot uniformity and others. The files were imported into R and the data filtered for the aforementioned quality criteria and subsequently merged with the recent annotation data provided by Agilent eArray. The data was segmented and called using the CGHcall-package [log2 expression values were normalized using RMA [• -package  using de-package . The logsing RMA  and the Note: The DNA copy number data of this data set is identical to that of the Taylor II data set.• Data set 5: Taylor IISample type: Prostate cancer.• Molecular levels: DNA copy number & exon expression.• Reference: Taylor et al.[• or et al..DNA copy number platform: Agilent 244 K array CGH.• Exon expression platform: Affymetrix Human Exon 1.0 ST array.• Number of samples: 49.• Availability: GEO database .• Preprocessing: Data are limited to 49 prostate cancer samples of which DNA copy number, exon expression and microRNA expression (see data set Taylor I) was available. The Agilent 244 k array CGH raw data were provided as tab-delimitedtext files containing the normalised log2-ratios and the values of control measures such as background-to-foreground ratios, spot saturation, spot uniformity and others. The files were imported into R and the data filtered for the aforementioned quality criteria and subsequently merged with the recent annotation data provided by Agilent eArray. The data was segmented and called using the CGHcall-package [log2-ratios of 15478 genes (autosomes only) from 49 samples.• -package  using de-package . Briefly-package . After qNote: The DNA copy number data of this data set is identical to that of the Taylor I data set.· For the R-code below it is assumed that the sigaR-package plus its dependencies have been activated, and that the cghCall and ExpressionSet objects  have been loaded Tables  and 18.The authors declare that they have no competing interest.WNvW conceived and carried out the project, and wrote the paper. KU, OK and RXdM provided the data. KU, GGRL, OK, RXdM, BY and MAvdW contributed to the development of at least one of the matching procedures, and made comments and suggestions. All authors read and approved of the manuscript.


N top ranked genes, it would be more appropriate to define and keep a number of gene cluster exemplars.A feature selection method in microarray gene expression data should be independent of platform, disease and dataset size. Our hypothesis is that among the statistically significant ranked genes in a gene list, there should be clusters of genes that share similar biological functions related to the investigated disease. Thus, instead of keeping AP-KL), which combines multiple hypothesis testing and affinity propagation (AP)-clustering algorithm along with the Krzanowski & Lai cluster quality index, to select a small yet informative subset of genes. We applied mAP-KL on real microarray data, as well as on simulated data, and compared its performance against 13 other feature selection approaches. Across a variety of diseases and number of samples, mAP-KL presents competitive classification results, particularly in neuromuscular diseases, where its overall AUC score was 0.91. Furthermore, mAP-KL generates concise yet biologically relevant and informative N-gene expression signatures, which can serve as a valuable tool for diagnostic and prognostic purposes, as well as a source of potential disease biomarkers in a broad range of diseases.We propose a hybrid FS method (mAP-KL is a data-driven and classifier-independent hybrid feature selection method, which applies to any disease classification problem based on microarray data, regardless of the available samples. Combining multiple hypothesis testing and AP leads to subsets of genes, which classify unknown samples from both, small and large patient cohorts with high accuracy.m Microarray data analysis is widely used for the identification of ‘informative’ genes. However, due to the ‘curse’ of dimensionality, where the number of gene probes represented on microarrays far exceeds the available number of cases (samples) as well as the inherent noise in microarray data, feature selection (FS) approaches strive to achieve this goal. Typically, informative genes are selected according to a two-sample statistical test combined with multiple testing procedures to guard against Type 1 errors .A wide variety of FS algorithms has been proposed -5 and deComplementary to this categorization, hybrid approaches have drawn researchers’ interest. Specifically the benefits of usually two different techniques are combined towards the identification of an improved gene subset selection, for example, a univariate filter with a wrapper or an embedded method -20. AparAn FS algorithm should perform efficiently and independently of the sample size and yield its subset within a reasonable period, to enable numerous experiments. Moreover, the subset’s length should be small, for instance, less than 50 genes, and the selected genes should present biological relevance to the inspected disease so as to facilitate further analysis. Despite the plethora of available FS methods, none of them has managed to successfully deal with all the aforementioned issues playing the role of a milestone. For instance, some methods are effective with small cohorts while others with large ones . Aside fAP-KL), which combines multiple hypothesis testing [AP-KL on real data from small and large cohorts, as well as on simulated data, and compared its performance against 13 other FS approaches. According to the results, mAP-KL achieves competitive classification results, particularly in the neuromuscular disease data, as well as in breast and colon cancers data, with subsets of less than 20 genes in most of the cases.We propose a data-driven and classifier-independent hybrid FS method  for further exploitation. We based our decision on keeping only the top 200 genes on the findings of a previous study [The proposed methodology combines ranking-filtering and cluster analysis to select a small set of non-redundant but still highly discriminative genes. In relation to the filtering step, we first employ the maxT function (see Feature selection methods) from the ‘multtest’ package to rank the genes of the training set and then we reserve the top us study , where wIn turn, prior to clustering analysis with AP we have to define the number of clusters, which in essence will be the number of representative genes that finally will compose our subset. We apply the index of Krzanowski and Lai as included in the ‘ClusterSim’ package  to deterk denotes the within-cluster sum of squared errors.when choosing the number of clusters (k) to maximize the quantity n  clusters among the top N genes, according to the pre-defined number, and provides us with a list of the most representative genes of each cluster, the so called exemplars. These n exemplars are expected to form a classifier that shall discriminate between the normal and disease classes in a test set. Finally, we formulate the test set by keeping only those n genes, and proceed with the classification.The final step of our methodology involves the cluster analysis. For this task, we engage the AP clustering method, which detects AP-KL is developed under the R environment [The mironment , in whicironment  packagesAffinity propagation identifies a set of centers (exemplars) from actual data points. Contrary to k-centers technique, which begins with an initial set of randomly selected exemplars and iteratively refines this set to decrease the sum of squared errors, AP considers each data point as a node in a network, and recursively transmits real-valued messages along edges of the network until a good set of exemplars and corresponding clusters emerges. At any point in time, the magnitude of each message reflects the current affinity that one data point has for choosing another data point as its exemplar.Messages exchanged between data points can be of two kinds: ‘responsibility’ r, and ‘availability’ a. ‘Responsibility’ reflects the accumulated evidence for how well-suited point k is to serve as the exemplar for point i, taking into account other potential exemplars for point i. On the other hand, ‘availability’ reflects the accumulated evidence for how appropriate it would be for point i to choose point k as its exemplar, taking into account the support from other points that point k should be an exemplar. Initially, the availabilities are set to zero. AP can be applied to problems where the similarities are neither symmetric nor satisfy the triangle inequality .Regarding the classification phase we employed SVM  with linTo evaluate the classification results, we employed various standard performance measures, which provide different insights. Accuracy (ACC) is one of the most popular performance measures in machine learning classification, though it does not take into account the nature of the incorrect predictions, which can be crucial in clinical medicine and totally misleading about the actual classification performance of a given classifier. Therefore we engaged the area under the receiver operating characteristics (ROC) curve or in short AUC, which has been introduced as a better measure for evaluating the predictive ability of machine learners than accuracy . The ROC, which represents the ratio of correctly classified negatives to the actual number of negatives and controls type I errors, as well as true positive rate (TPR) or sensitivity, which is defined to be the ratio of positives correctly classified to the actual number of positives and controls type II errors. Both, specificity and sensitivity are mutually independent [Additionally, we employed true negative rate (TNR) or specificityependent . The comAP-KL’s performance. Neuromuscular and cancer diseases data comprise the real microarray data and are available in comma-delimited format in the Supplementary section. Neuromuscular diseases are rare among the general population, thus the available tissue samples and whole transcriptome data are very limited. This characteristic is crucial since we intended to develop a FS method that produces robust models even in studies with limited number of samples. We therefore included data from Bakay et al. [In this study, we utilized real and synthetic data to assess my et al.  related y et al.  and Sanoy et al. . The genRegarding the cancers datasets, we utilized microarray data from breast cancer, colon cancer, leukemia, and prostate cancer, all of which are considered benchmark datasets and have been widely used in gene expression-classification studies. Van’t Veer  exploredDatasets from acute lymphoblastic leukemia (ALL) and acute myeloid leukemia (AML) , two disAP-KL’s performance on two synthetic datasets. We intentionally utilized two different simulation setups to examine two different hypotheses. In the first hypothesis, we wanted to verify that mAP-KL provides us with a small subset of representative features, at least one gene per cluster, adequate for accurate classification. Therefore, we considered a binary classification problem simulating a normal-disease case with six different scenarios , one multivariate filter algorithm (cat), three dimension reduction approaches , one embedded method (Random Forest), one hybrid method (HykGene) and one Monte-Carlo like (Rnd) technique.We employed 13 feature selection/elimination approaches on the same real microarray datasets and compared its performance with that from mBetween Group Analysis (BGA) is a multiple discriminant approach that can be used with any combinations of numbers of genes and samples. BGA uses a conventional ordination technique such as Correspondence Analysis (COA) or principal component analysis (PCA) to carry out ordination of groups of samples. For N groups we find N − 1 eigenvectors or axes that arrange the groups so as to maximise the between group variances. The individual samples are then plotted along them. Each eigenvector can be used as a discriminator to separate one of the groups from the rest. New samples are then placed on the same axes and can be classified on an axis-by-axis basis or by proximity to the group centroids. It is especially effective when combined with COA because it allows us to examine the correspondences between the grouped samples and those genes which most facilitate the discrimination of these groupings [roupings .The Hybrid system for marker Gene selection (HykGene) is a hybrid approach that combines sequentially gene ranking and clustering analysis. Firstly, a set of top-ranked informative genes is selected with the aid of filtering algorithms , and secondly a hierarchical clustering is applied on these genes to generate a dendrogram. Finally, a sweep-line algorithm is used to analyze the dendrogram and marker genes are selected by collapsing dense clusters. The best number of clusters is determined systematically by applying the leave-one-out cross-validation (LOOCV) on the training data, trying all different options for extracting clusters from the dendrogram [ndrogram .Principal component analysis (PCA) is a classic and one of the oldest dimension reduction approaches. It searches for linear combinations of the original measurements called principal components (PCs) that can effectively represent effects of the original measurements. PCs are orthogonal to each other and may have dimensionality much lower than that of the original measurements. Because of its computational simplicity and satisfactory statistical properties, PCA has been extensively used in bioinformatics studies, particularly gene expression studies, to reduce the dimensionality of high-throughput measurements and shown to have satisfactory performance [formance . We implThe optimal discovery procedure (ODP) is a high dimensional approach that uses all of the relevant information across tests when assessing the significance of each one. It allows us to test multiple hypotheses simultaneously in such a way that the total number of expected true positive results is maximized for each fixed number of expected false positive results. This procedure can be viewed as a multiple-test extension of the Neyman–Pearson (NP) procedure for testing a single hypothesis. This method is available through the EDGE software program [ program .maxT: It is a function that computes permutation adjusted p-values for step-down maxT multiple testing procedures as described in Westfall & Young [AP-KL. & Young , which p & Young . It dete & Young . This isGenePattern is a software package, which provides a comprehensive environment that can support (i) a broad community of users at all levels of computational experience and sophistication, (ii) access to a repository of analytic and visualization tools and easy creation of complex analytic methods from them and (iii) the rapid development and dissemination of new methods [ methods . The Com methods .t-test: This is the standardized mean difference between the two classes. It is the difference between the mean expression of class 1 and class 2 divided by the variability of expression, which is the square root of the sum of the standard deviation for each class divided by the number of samples in each class.SNR: The signal-to-noise ratio is computed by dividing the difference of class means by the sum of their standard deviations.Partial Least Squares (PLS) is a highly efficient statistical regression technique that is well suited for the analysis of high-dimensional genomic data. The underlying idea of PLS is to find uncorrelated linear transformations of the original predictor variables which have high covariance with the response variables. These linear transformations can then be used as predictors in classical linear regression models to predict the response variables. Since the p original variables are summarized into a small number of relevant new components, linear regression can be performed even if the number of original variables p is much larger than the number of available observations [rvations .Random Forests (RF) are a combination of tree-structured predictors where each of the trees grows using a random process. Given a training set with N samples and M features, the N instances are sampled at random (with replacement), so as to generate a random vector Θ for each tree. For the kth tree, there is a random vector Θk which is independent of the previous random vectors, Θ1, … , Θk−1, but with the same distribution for all trees in the forest. Hence, every tree is grown using the training set and its random vector, resulting in a classifier, which votes for the most popular class.j, values of xj are permuted in the OOB sample, and the class membership of the OOB samples are predicted again from the tree. The number of correctly classified samples after permutation is subtracted from the original count of correctly classified samples and divided by the number of OOB samples for that tree, thus giving the decrease in classification accuracy as a proportion of samples. This permutation procedure is repeated for each tree in the forest, and the mean decrease in accuracy (MDA) is defined as the average of these values over all trees in the forest (multiplied by 100 and presented as a mean percentage decrease in accuracy). In this experiment, a random forest classifier with 1,000 trees is applied [When RF draws the training set for the current tree by sampling with replacement, about one-third of the cases are left out of the sample, and called out-of-bag data (OOB). This OOB data is used to get estimates of variable importance. To measure the importance of variable x applied .Significance Analysis of Microarrays (SAM) identifies genes with significant changes in gene expression by conducting a set of gene-specific t-tests and then assigning a score to each gene relative to the standard deviation of those tests. Genes are characterized as significant if their score is greater than an adjustable threshold (delta). SAM employs permutations of the repeated measurements in order to estimate the false discovery rate (FDR) i.e. the percentage of genes identified by chance. Through the threshold adjustment, we may conclude to smaller or larger sets of genes [of genes .The empirical Bayes moderated t-statistic (eBayes) ranks genes by testing whether all pairwise contrasts between different outcome-classes are zero. It is applied to extract information across genes thus making the final analyses more stable even for experiments with limited number of arrays. Moderated t-statistics lead to p-values with increased degrees of freedom for the individual variances hence, reflecting the greater reliability associated with the smoothed standard errors [d errors . Linear d errors .Correlation-adjusted t’-scores (cat) is the product of the square root of the inverse correlation matrix with a vector of t scores and represents the individual contribution of each single feature (gene) to separate two groups after removing the effect of all other features. This method takes account of correlations among genes before adjusting the t-statistics. In the absence of correlation the cat score reduces to the standard t-score. The cat score offers a simple approach to feature selection, both of individual genes and sets of genes [of genes .random (Rnd) scores refer to these mean values per disease.Apart from these standard methods, we wanted to explore whether the use of a feature selection method over the top 200 list will benefit the prediction or not. Therefore, we decided to select randomly gene probes from the ranked list and then assess its classification performance. In order to control the randomness and finally conclude to a stable outcome we created randomly 10 subsets of 20 gene probes, run the classification process and finally summarized the results. Thus, the AP-KL we designed and executed an elaborate set of analytical experiments with 5-CV on the training set and hold-out validation on a separate set to assess its performance across whole genome expression datasets from both small and large patient cohorts. In relation to small cohorts, we employed data from 6 neuromuscular diseases, while for large cohorts we utilized data from 4 different types of cancer. On those microarray datasets, we also applied 13 other feature selection/elimination approaches and compared the classification results  except for the HykGene in ALS and the PCA in DMD with AUC scores of 0.64 and 0.61 respectively. Similarly, in 5-CV test, only the BGA-COA and the PCA with AUC scores of 0.98 and 0.48 respectively, deviated from the rule. In JDM although all of the methods achieved the highest AUC score (1.00) during hold-out validation, the respective TNR score was 0.88 for the BGA-COA, eBayes, ODP, SNR and cat methods. In 5-CV the PCA was the only method that failed to distinguish correctly all samples (AUC = 0.90).AP-KL and maxT (200) achieved the highest TNR and TPR, too. The TNR score for PLS-CV was 0.91, for RF-MDA, ODP and SNR was 0.73, while for HykGene was 0.45 and for eBayes 0.36. It is worth noticing that the TNR score of the maxT with the 20 genes subset, was considerably lower to that of maxT (200). During the 5-CV evaluation, the classification results are almost ideal, since only PCA had an AUC score less than 1.00.In relation to the LGMD2A, ten methods, including the maxT (200), achieved the highest AUC value, though only BGA-COA, mAP-KL) and no method had TNR greater than 0.80. On the other hand, the 5-CV classification results were very promising since all methods but PCA achieved the highest score i.e. 1.00 in all three metrics.Contrary to the previous datasets, in LGMD2B validation, only three of the methods  and PLS-CV) achieved the highest AUC (1.00) but their TNR score was 0.73, 0.64 and 0.55 respectively. Although many methods distinguish all disease samples correctly i.e. TPR = 1.00, all of them failed to discern all normal samples i.e. TNR < 1.00. Approximately half of the methods had a TNR below 0.50 , RF-MDA, and SNR achieved the optimum score in TNR and TPR metrics. The PLS-CV and BGA-COA had the same TNR score (0.87) but different TPR (0.99 and 0.91) and AUC (0.97 and 0.96). The PCA method achieved the lowest AUC score (0.82) with TNR and TPR scores equal to 0.77 and 0.73. Finally, the eBayes method failed to produce a list of significant genes.Likewise in NM validation, all of the methods faced considerable difficulties in distinguishing disease and normal samples. Only the SNR, the t-test and the HykGene methods managed to reach an AUC score close to 0.90. The TPR results for NM as opposed to those for LGMD2B were discouraging, since only four methods  classify all the disease samples correctly, and the TPR score for the rest of the methods range from 0.20 to 0.60, with the exception of t-test and HykGene (TPR = 0.80). In contrast, in 5-CV ten methods achieved AUC score of 1.00, though only mAs far as the large patient cohorts is concerned, we utilized microarray data from four different types of cancer , with train sets length ranging from 30 to 102 samples and test sets from 19 to 34 (Table AP-KL attained the optimum score (1.00) in TNR metric and the best AUC score (0.87). Two methods, PLS-CV and RF-MDA, achieved competitive TNR and AUC scores of 0.86 and 0.82 respectively. However, all methods faced difficulties to distinguish the non-responsive samples, and except the maxT (200) with a TPR score of 0.83, followed by the Rnd technique (0.78) and the RF-MDA, the HykGene and the SAM methods (0.75). During the 5-CV validation, PLS-CV, RF-MDA, HykGene and cat had an AUC score of 0.91, which was also the highest score attained. The rest of the methods achieved AUC scores between 0.73 and 0.77, but only SAM had a balance performance between TNR and TPR metrics. It is worth noticing that the TPR results for all methods were below the TNR results. The eBayes method failed to fulfil the analysis task.In breast cancer HO validation, mAP-KL presented similar classification behaviour to breast cancer, with an AUC score of 0.89 and a more balanced behaviour between TNR and TPR (0.71 and 0.84). Only the BGA-COA method achieved a competitive AUC score of 0.87. The AUC score for the rest of the methods lay between 0.79 and 0.84. Contrary to breast cancer, the TPR scores were higher than the TNR scores and range from 0.80 to 0.84 for all methods. Regarding the TNR metric, all methods but Rnd (0.73) and PCA (0.43) and cat (0.57) achieved the same score of 0.71. The classification results in 5-CV are very promising with AUC values from 0.98 to 1.00 for all of the methods except PCA, which attained an AUC score of 0.79.In relation to colon cancer, mAP-KL, although achieving high classification scores in 5-CV, failed to predict correctly all AML samples (TPR = 0.43), and as a results its overall performance was 0.71 during the hold-out validation. The PCA, SNR and t-test methods failed to predict any of the 14 AML samples, although they identified all or almost all of the ALL samples. The ODP algorithm failed to analyse the colon dataset.Concerning the leukemia datasets, 10 of the methods  and maxT) performed similarly in both validation tests. Their AUC were close to 1.00 in both cases, and the TNR results were better than the TPR scores. The mAP-KL appear to have an opposite behaviour in relation to TNR and TPR metrics, but the normal: disease ratio tips the scales in favour of PLS-CV (AUC = 0.87). Two algorithms, ODP and cat, could not deal with the prostate data.Finally, in prostate cancer, no method succeeded in discriminating the samples in both types of validation, alike to NM in neuromuscular diseases section. Even more importantly, during the hold-out validation, many of the methods , maxT, PCA, SNR and t-test) failed to identify even a single sample from the normal class. Only the HykGene excelled in this metric with a TNR score equal to 0.89. However, because of the normal: disease ratio , the AUC values of eBayes (0.86) and SAM (0.92) are a little deceptive. On the other hand, PLS-CV and mAP-KL’s classification results of the specific cancer datasets, against those published in previous classification studies of the same data. For the purposes of this comparison, we have cited the author’s name, the classification type, the number of the features used, and finally the achieved accuracy (ACC). Since we utilized three different classifiers to build and test mAP-KL’s models, in this comparison we present all three results achieved.At a different level of assessment, we compared the mAP-KL is the lowest score, we did manage to identify correctly all responsive samples. However, we should consider why we discern only half of the non-responsive samples (type II error).In relation to the van ’t Veer et al.  breast cFourteen methods employed the Alon et al.  colon caAP-KL to Golub classification results, we notice that in CV we identify one more sample, whereas in hold-out we misclassify two samples from Golub, though we did that with only 5 genes. There are many methods that distinguish correctly all samples in CV although only Hewett and Kijsanayothin [AP-KL with the aid of SVM-linear classifier, misclassified one sample in hold-out validation just like Liu et al. [The ALL/AML discrimination in the leukemia datasets, Table nayothin  achievednayothin  reached nayothin  predict nayothin  first emu et al. . HoweverThe power of the proposed FS approach is evident not only from its performance in the statistical metrics, but also from the biological relevance of the selected genes either to a broad range of different molecular pathways and biological processes or more importantly to the respective pathological phenotypes. Representative examples include the genes COL3A1, SPARC and PTTG1IP, which are related to extracellular matrix formation and fibroblast growth, biological processes consistent with the increased fibrosis that is observed in skeletal muscles affected by DMD . In ALS These findings jointly, demonstrate that despite their small size, the discriminatory ‘lists of selected genes’  the identified clusters are less than the actual. Similarly, when the number of DEGs far exceeds the number of N top ranked genes the identified clusters are fewer, for instance 500 DEGs with 200 top ranked genes parameter. Nonetheless, during the real gene expression data experiment, we employed a moderate value for the parameter N = 200 top ranked genes.With respect to the effect of the number of top ranked ‘genes’ kept for clustering, it is evident that the closer to the real number of DEGs, the better the identification and selection of representative genes. Specifically, in cases where the number of DEGs is considerably lower than the number of As a final point, we formed the respective train-test sets for all methods and evaluated their performance with the aid of three classifiers . All methods performed accurately (ACC = 100%) for all three classifiers, see Additional file AP-KL’s subset in relation to the known DEGs included in it. Therefore, we applied on the ‘choedata’ the mAP-KL, engaging a non-parametric and a parametric statistical methods Table In this setup, we were interested in exploring, the length of the mWe then formed classification models with the assistance of three classifiers  and assessed their performance. Despite this remarkable difference in the number of DEGs included in the two subsets, the classification results were accurate in both cases. Nonetheless, including more DEGs in a classifier is of benefit to the biological analysis if not to the classification process itself.AP-KL), demonstrates how effective the combination of a multiple hypothesis testing approach with a clustering algorithm can be to select small yet informative subsets of genes in binary classification problems. Across a variety of diseases and number of samples, mAP-KL presents competitive classification results , which achieved outstanding classification results with AUC = 0.97 in neuromuscular diseases and AUC = 0.89 in cancers, but such a lengthy subset may contain a number of irrelevant genes that will act as “noise” when performing further biological analysis. On the other hand, keeping a subset of top N genes, where N = 5,10,…n, needs several rounds of “trial and error” attempts before concluding to the best N value. Otherwise setting the N parameter arbitrarily does not guarantee robust and efficient classification results, as shown in the case of the 20 genes subset of the maxT. Additionally, forming subsets by selecting genes randomly from an already ranked list may lead to satisfactory classification results. The Rnd technique achieved comparable classification results to maxT either with 200 or 20 genes subset. However, the subsets are not reproducible and no biological evidence can be inferred for them. Taking into account all the aforementioned issues, we claim that the novelty and strength of mAP-KL is the efficient sampling of the ranked gene list, selecting those genes that are necessary for improved classification, rather than keeping just a predefined number of top N ranked genes.Finally, we would like to highlight some points of interest in relation to ranked gene lists, which retrospectively confirm our initial motivation towards the mThe authors declare that they have no competing interests.AS, DS and GS conceived this project. AS carried out all the experiments and the analysis as principal investigator. DS carried out the biological relevance analysis and wrote the relevant section. AS wrote the rest of the paper. GS supervised this project. All authors read and approved the final manuscript.This function implements in R-code, the mAP-KL’s functionality.Click here for fileIn this file, we present the 5-CV classification results for all real microarray data, when using three different classifiers .Click here for fileIn this file, we present the Hold-out validation results for all real microarray data, when using three different classifiers .Click here for fileIn this file, we have cited the subsets of genes according to the mAP-KL method.Click here for fileContains the microarray data used in this experiment. For each disease, we provide the ‘class_labels.csv’, ‘train.csv’ and ‘test.csv’ files, which represent the analogy of samples as described in table Click here for fileIn this file, we have cited the clustering setup parameters, the DEGs position per simulation dataset, as well as the DEGs identified per method.Click here for fileIn this file, we present the classification results of  in the first simulation setup, where the clustering identification was under investigation. We employed three classifiers .Click here for fileIn this file, we present the classification results in the ‘choedata’ when using two different mAP-KL’s subsets, stemming from two different ranking approaches. We used the SVM-linear, KNN, and RF classifiers to assess their performance.Click here for fileThis file contains the relevant scripts and functions for generating the simulated data. The ‘clusterSim’ r-package is required.Click here for file


Unfortunately, changes in experimental planning and execution, such as the unavailability of previously anticipated samples or a shift in research focus, may render significant numbers of pre-purchased GeneChip® microarrays unprocessed before their manufacturer’s expiration dates. Researchers and microarray core facilities wonder whether expired microarrays are still useful for gene expression analysis. In addition, it was not clear whether the two human reference RNA samples established by the MAQC project in 2005 still maintained their transcriptome integrity over a period of four years. Experiments were conducted to answer these questions.The Affymetrix GeneChip® data generated by the MAQC project in 2005.Microarray data were generated in 2009 in three replicates for each of the two MAQC samples with either expired Affymetrix U133A or unexpired U133Plus2 microarrays.  These results were compared with data obtained in 2005 on the U133Plus2 microarray.  The percentage of overlap between the lists of differentially expressed genes (DEGs) from U133Plus2 microarray data generated in 2009 and in 2005 was 97.44%. While there was some degree of fold change compression in the expired U133A microarrays, the percentage of overlap between the lists of DEGs from the expired and unexpired microarrays was as high as 96.99%. Moreover, the microarray data generated using the expired U133A microarrays in 2009 were highly concordant with microarray and TaqManOur results demonstrated that microarray data generated using U133A microarrays, which were more than four years past the manufacturer’s expiration date, were highly specific and consistent with those from unexpired microarrays in identifying DEGs despite some appreciable fold change compression and decrease in sensitivity. Our data also suggested that the MAQC reference RNA samples, stored at -80°C, were stable over a time frame of at least four years. We noticed that unforeseeable changes in experimental planning and execution, e.g., unavailability of previously anticipated samples or a shift in research focus could render pre-purchased GeneChip® microarrays unprocessed before their manufacturer’s expiration dates. Therefore, it is important to know whether the expired but expensive GeneChip® microarrays are still useful and reliable in gene expression studies.As a powerful tool in genomic research, microarray technology has been widely used for simultaneously monitoring expression levels of tens of thousands of genes -3. The Ap-value cutoff for DEG selection and the percentage of overlapping genes (POG) for evaluation of the reproducibility of DEGs between different experiments. Excellent inter-laboratory and inter-platform reproducibility in terms of DEG lists was observed in the MAQC project [A critical goal in the application of microarray technology is to identify differentially expressed genes (DEGs) between sample groups under different conditions. However, several studies showed poor overlaps of DEGs across different platforms or different laboratories using the same sets of RNA samples -8, raisi project ,16.In the current study, we assessed the consistency between gene expression data generated with expired and unexpired Affymetrix microarrays using the same batches of MAQC samples and a similar data analysis approach as proposed by the MAQC project. Our results showed good reproducibility in terms of DEG lists between the expired and unexpired microarrays. At the same time, we also observed a high level of reproducibility between DEG lists from data generated in 2005 and those newly generated in 2009 with the same type of unexpired U133Plus2 microarrays, suggesting a good degree of stability of the two MAQC reference RNA samples stored at -80°C over a period of four years.We designed and conducted a comparative study to answer the question whether gene expression data generated with Affymetrix U133A microarrays, four years past their expiration date, are useful and reliable. Figure ® assays by the MAQC project in 2005 were used as references to assess the stability of the MAQC samples stored at -80°C for four years by comparing new microarray data with those obtained four years ago. The MAQC reference data also allowed for further evaluation of the usefulness of the data generated with expired U133A microarrays. The new gene expression data were generated with 12 microarrays (2 types of microarrays × 2 samples × 3 replicates)  and U133Plus2 (unexpired). However, the probe design (probe length and sequence identity) for these two types of GeneChip® microarrays is identical and the consistency of data between the two types of microarrays was demonstrated by the manufacturer . Therefore, the observed difference between expired U133A and unexpired U133Plus2 microarrays can be attributed mainly to the expiration of the former.It should be pointed out that in the current study the observed differences between the expired and unexpired Affymetrix microarrays were confounded with the use of two different types of GeneChipThe two MAQC human reference RNA samples, from the same batches as used in the MAQC project in 2005 but stored at -80°C for over four years, were labeled and hybridized in 2009 on expired U133A and unexpired U133Plus2 microarrays. Before comparing data from the expired and unexpired microarrays, it is necessary to verify the stability of the MAQC samples through assessing the consistency between gene expression data generated in 2005 and in 2009 using the same type of unexpired Affymetrix U133Plus2 microarrays.2 fold changes observed in 2009 and in 2005 with unexpired U133Plus2 microarrays. The correlations are shown in Figures p < 0.05 in either of the two data sets (2009 and 2005), and the 5,880 genes with a p < 0.05 in both data sets, respectively. It can be seen from Figure 2 fold changes. Under the three gene selection scenarios  with increasing stringency, the overlap of DEGs between 2009 and 2005 was 91.37%, 97.44%, and 99.78%, respectively, suggesting a reasonably high degree of stability of the two MAQC samples over a period of four years. Thirteen genes showed opposite regulation directionalities between the data sets generated in 2005 and in 2009 . However, there was a slight fold change compression [To address the issue of using expired microarrays, the logd Figure . The logpression  for the p-value cutoff was applied to either experiment  in both experiments, the overlap increased to 99.98%. That is, only one gene showed opposite regulation directionalities between expired U133A microarrays and unexpired U133Plus2 microarrays. These results suggested that the DEGs detected by the expired U133A microarrays were highly consistent with those by the unexpired U133Plus2 microarrays. The percentage of overlapping genes with the same directional change in expression was 89.94%  when no t Figure . The oves Figure , a fair p < 0.05 cutoff to eliminate genes separately in each experiment and then rank-ordered the remaining genes by log2 FCs. In one experiment, genes ranked higher in either the up- or down-regulation direction are selected as DEGs first. Figure e.g., between expired U133A microarrays and unexpired U133Plus2 microarrays. Microarray data generated in 2005 with the same MAQC samples from five platforms, Applied Biosystems (ABI), Affymetrix U133Plus2 (AFX), Agilent (AG1), Illumina (ILM), and GE Healthcare (GEH) by the MAQC project  were useThe concordance between DEG lists selected from expired U133A microarrays and unexpired U133Plus2 microarrays (2009 or 2005) were about 70% when the number of selected DEGs was as few as 10. These POG numbers increased to near 90% when more genes were selected as differentially expressed. For the same number of DEGs, the POGs between expired U133A (2009) and unexpired U133Plus2 (2009) microarrays, and between expired U133A (2009) and the reference U133Plus2 (AFX in 2005) microarrays were on the upper confidence limit of the reference line. This means that compared to data from the reference AFX microarrays (2005), the DEGs generated with expired U133A microarrays (2009) are more consistent than those from other unexpired, but different microarray platforms  in 2005. In addition, the POG between the unexpired U133Plus2 (2009) and the reference AFX microarrays (2005) was the highest, again, suggesting that the MAQC samples were relatively stable. ® assays have been widely used in gene expression profiling and are generally considered as a good reference for evaluating other gene expression techniques. In this study, the TaqMan® gene expression data generated by the MAQC project in 2005 using the same MAQC samples [® assays. The DEG lists obtained from expired U133A microarrays (2009), unexpired U133Plus2 microarrays (2009), and microarrays from ABI, AFX, AG1, ILM, and GEH (2005) were separately compared with those from the TaqMan® assays performed in 2005 [® assays and the microarray platforms  was shown as a reference line  was close to the reference line when the number of DEGs was as low as 40 but was about 10% lower than the reference line when all genes meeting the p < 0.05 criterion were selected (a drop from ~77% to ~70%). In addition, the POG between unexpired U133Plus2 microarrays (2009) and TaqMan® assays (black squares) and the POG between the AFX (2005) and TaqMan® assays (sky blue triangles) were higher than, or at the same level as, the reference line. That is, compared to unexpired microarrays, expired U133A microarrays exhibited a slightly decreased ability to detect differential gene expression, mainly due to the fold change compression that made the rankings of genes with smaller fold changes more variable. TaqMan samples ,19 were The lower sensitivity of the expired U133A microarrays in detecting differential gene expression could be explained by a decrease in microarray signal or an increase in the scaling factor  are also higher than those for expired U133A microarrays (1.93 for sample A and 1.95 for sample B), indicating a decrease in the ability of expired microarrays to distinguish differences of expression levels among genes in the same sample. In addition, as shown in Figure Figure A Figure  and 7.67A Figure , the expA Figure  and 6.29A Figure . The sta2 fold changes, and the POG of DEG lists, using data newly generated in 2009 with the same pair of reference RNA samples used in 2005 by the MAQC project. In addition, the gene expression data generated using different platforms  in 2005 by the MAQC project were used as references to evaluate the reliability of the expired U133A microarrays and also to assess the stability of the MAQC samples.Many users of microarrays face time pressures to complete their studies while staying within the expiration dates on the microarrays. Moreover, it is important to understand the stability of RNA samples after years of storage. Our current study investigated the stability of two MAQC human reference RNA samples and examined the reliability of expired Affymetrix U133A microarrays in gene expression analyses. We compared the gene expression data of expired and unexpired microarrays from three aspects: absolute expression intensities, log2 fold changes, microarray data generated in 2009 were highly consistent (97.44%) with those generated by the MAQC project in 2005 using exactly the same type of unexpired Affymetrix U133Plus2 microarrays and with the same pair of RNA samples that had been stored at -80°C for over four years  from unexpired U133Plus2 microarrays showed similar results  the stability of these reagents and b) the reliability of data generated on Affymetrix’s U133A microarrays expired four years ago. By comparing the detected log® protocol. One µg of total RNA from each sample, along with poly A spikes (labeling control), were converted to double-stranded cDNA with GeneChip® One-Cycle cDNA Synthesis Kit . After second-strand synthesis, the cDNA was purified with the GeneChip® Sample Cleanup Module (Affymetrix). The resulting double-stranded DNA was then used to generate multiple copies of biotinylated cRNA by in vitro transcription with the GeneChip® 3’-Amplification Reagent Kit for IVT Labeling (Affymetrix). The A260/A280 ratio and yield of each of the cRNAs were recorded. For each sample, 10 µg of biotinylated cRNA spiked with bioB, bioC, bioD and cre (Hybridization Control) was hybridized to a GeneChip® microarray for 16 hours at 45°C. Following hybridization, all microarrays were washed and stained in an Affymetrix GeneChip® Fluidics Station. Stained microarrays were scanned with an Affymetrix GeneChip® Scanner 3000. Quality checks and data analyses were carried out using Affymetrix GeneChip® Operating Software (GCOS) and Quality Reporter. Each of the two samples was processed in triplicate, starting from independent labeling reactions with a new aliquot of the reference sample from the same tube, a) on Affymetrix Human Genome U133A microarrays that were four years ago past the manufacturer’s expiration dates and b) on unexpired Human Genome U133Plus2 microarrays. The expired U133A microarrays were stored in a refrigerator at 4°C, as recommended by the manufacturer. In total, gene expression data were collected from 12 (2 types of microarrays × 2 samples × 3 replicates) microarrays. The comparisons among the microarrays were based on 8,550 common genes, which represented the intersection of the 12,091 genes commonly probed by the microarray platforms used in the MAQC project [® product pipeline. Consequently, comparing gene expression data from the same sample in 2005 and 2009 at the intensity level is not meaningful, because it has been demonstrated that intensity data are highly dependent on the labeling protocols among many other factors.The two reference RNA samples A  and B (Ambion’s Human Brain Reference RNA), which were established and extensively profiled in the MAQC project in 2005 , were pu project  and the ® (TAQ) assays were obtained from the MAQC project (GSE5350) and used in this study as references for comparison. For fair comparisons, the first three replicates reported by the MAQC project from each platform’s first test site were used in this study. For cross-platform comparison, the MAQC consortium mapped all the probe sequences of the five high-density microarray platforms to the RefSeq human mRNA database [® assays  [® and were used for comparing microarrays with TaqMan® assays.The gene expression data generated from the two samples by the MAQC project with five commercial microarray platforms, Applied Biosystems (ABI), Affymetrix (AFX), Agilent (AG1), Illumina (ILM), and GE Healthcare (GEH), and with the TaqMandatabase  and sele-S5.txt) . The prohttp://www.fda.gov/ScienceResearch/BioinformaticsTools/Arraytrack/). Unless stated otherwise, the probe level data were summarized into probeset level data using RMA [The microarray data generated in 2009 and the data from the MAQC project generated in 2005 were imported to ArrayTrack™ 3.5.0 , a softwsing RMA  before s2 intensities, log2 fold changes, and the POGs were conducted within MATLAB  and the corresponding homemade source codes for these analyses are available upon request.The scaling factor of each microarray was calculated with the Affy package in Bioconductor 2.4 within R 2.9. The comparisons of log by using a scaling factor [When probe-level raw expression data are summarized into probeset expression values with MAS5, a trimmed mean of the normalized data is scaled to a default value g factor . The valt-test is one of the common statistical methods for gene selection. The p-value calculated by Welch’s t-test is usually used to measure the statistical significance of a gene differentially expressed in the two groups of samples being compared. In our study, the p-value is directly used for gene filtering without multiple-testing correction. A gene was considered as significantly differentially expressed if its p-value was less than 0.05. Welch’s p ≥ 0.05 criterion, the remaining genes were ranked by their fold changes (reference sample B/reference sample A). Then, at each given cutoff a list of the number of genes considered to be differentially expressed genes was generated for subsequent comparisons. The fold change (FC) of gene expression intensities represents to what extent a gene is differentially expressed between two groups of samples. After filtering genes with a The genes contained in the two compared DEG lists with the same direction of regulation  were considered to be overlapped genes. The percentage of the overlapping genes (POG) ,15,23 amDD and UU are, respectively, the number of commonly down- and up-regulated genes. L is the number of genes with down- or up-regulation. Similar to previous studies [L from one (corresponding to the largest FC) to the lower number of genes in the two regulation directions  with a step of one.  See Figures where  studies ,15,23 weAll microarray data specifically generated for this study are available through the National Center for Biotechnology Information’s Gene Expression Omnibus (series accession number: GSE23906). The authors declare that they have no competing interests.LS conceived the study design. CW, QS, and YH generated the microarray data. ZW, QS, YH, ZS and LS carried out data analysis. ZW, CW, QS, HH, WT and LS prepared the manuscript. All authors read and approved the final manuscript.


Gene expression measurements have successfully been used for building prognostic signatures, i.e for identifying a short list of important genes that can predict patient outcome. Mostly microarray measurements have been considered, and there is little advice available for building multivariable risk prediction models from RNA-Seq data. We specifically consider penalized regression techniques, such as the lasso and componentwise boosting, which can simultaneously consider all measurements and provide both, multivariable regression models for prediction and automated variable selection. However, they might be affected by the typical skewness, mean-variance-dependency or extreme values of RNA-Seq covariates and therefore could benefit from transformations of the latter. In an analytical part, we highlight preferential selection of covariates with large variances, which is problematic due to the mean-variance dependency of RNA-Seq data. In a simulation study, we compare different transformations of RNA-Seq data for potentially improving detection of important genes. Specifically, we consider standardization, the log transformation, a variance-stabilizing transformation, the Box-Cox transformation, and rank-based transformations. In addition, the prediction performance for real data from patients with kidney cancer and acute myeloid leukemia is considered. We show that signature size, identification performance, and prediction performance critically depend on the choice of a suitable transformation. Rank-based transformations perform well in all scenarios and can even outperform complex variance-stabilizing approaches. Generally, the results illustrate that the distribution and potential transformations of RNA-Seq data need to be considered as a critical step when building risk prediction models by penalized regression techniques. RNA-Seq is a relatively new approach for measuring gene expression by making use of next generation sequencing technology. It produces count data having low background noise and hence allows to detect transcripts even at low expression levels and provides a large dynamic range in terms of fold-changes Gene expression measurements from microarrays have often been used for building prognostic gene signatures, i.e. a small set of genes that can predict the clinical outcome of patients. Correspondingly, it is attractive to also use RNA-Seq data for such a task, but the highly skewed nature of the latter might pose difficulties. In the following, we focus on regularized regression techniques for building signatures from RNA-Seq data, as these simultaneously consider all RNA-Seq measurements, can provide automated selection of important genes, and have generally been a popular class of multivariable approaches for microarray gene expression data. For a more general overview of such approaches, see e.g. Binder et al. The aim of this work is to investigate which specific properties of RNA-Seq data, such as skewness, mean-variance dependency and extreme values, influence model building with these approaches. In particular, we systematically investigate transformations of the RNA-Seq measurements to increase the performance of the models, with respect to identification of important genes and prediction performance.While there is hardly any advice for multivariable regression modeling with RNA-Seq data, a multitude of univariate testing techniques have been developed Besides transformations, such as using the VST or ranks, the variances of covariates are a critical issue for regularized regression techniques. Often, standardizing gene expressions is implemented as a default in software packages. For other kinds of molecular measurements, such as single nucleotide polymorphism data, standardization has not always be found to be advantageous In this work we propose a set of different data transformations of RNA-Seq data that can be applied before building a prognostic gene signature for binary endpoints and time-to-event endpoints. Transforming the data can be used to account for mean-variance dependencies and extreme values, both typical for RNA-Seq data. We compare the resulting gene signatures in terms of sensitivity, specificity and prediction performance using a simulation study in which we focus on a binary endpoint and componentwise likelihood-based boosting. As we are not only interested in binary endpoints and boosting, we will apply all transformations on two different real RNA-Seq data sets of patients in which we focus on time-to-event endpoints and boosting as well as the lasso. One data set is from patients with kidney renal clear cell carcinoma and the other from patients with acute myeloid leukemia. In the latter settings, we will use the gene signature to predict the survival times of the patients adjusting for known clinical covariates. Within these application examples we will have a close look on the individual differences of the gene signatures emerging from the different transformations including signature size, variance of selected genes and prediction performance given by the added value compared to a prognostic model only including the clinical covariates.The rest of this work is organized as follows. We will first introduce the application examples with kidney renal clear cell carcinoma (KIRC) data and acute myeloid leukemia (AML) data to further motivate this work and highlight some important properties of real RNA-Seq data. In the subsequent section we will show analytically how the variance of covariates affects the gene selection process with regularized regression techniques. We then provide a simulation study to investigate the effect of transformations and standardization on identification of genes. The simulation study considers a binary response, e.g. reflecting a two-group setting with a logistic regression model, and is based on the covariate structure of the KIRC application to simulate realistic conditions, including different expression strengths with different variances, skewed data, extreme values and correlations. Following the results of this simulation study we will give detailed results on both real data application to the KIRC and AML data, where time-to-event endpoints with a Cox regression model are considered.https://tcga-data.nci.nih.gov/tcga/). RNA-Seq data is available for 470 patients. We excluded five patients because their RNA-Seq data were available twice with different expression values, and one further patient due to an unknown survival time. Genes having a maximum number of 10 counts were excluded as they showed almost no expression (625 genes). After this preprocessing, we normalized the raw counts between the patients using the DESeq normalization proposed by Anders and Huber As a first application example, we consider RNA-Seq data from patients with kidney renal clear cell carcinoma (KIRC), available from The Cancer Genome Atlas (TCGA) project , also available from the TCGA website. There are 200 patients with clinical data available and 182 of them have RNA-Seq measurements. We excluded 13 additional patients due to unknown survival times. For the RNA-Seq expression data we followed the same preprocessing, normalization and truncation steps as for the KIRC data and end up with 169 patients and 18,714 genes. The median overall survival time for these patients is 1.3 years, 1-year and 3-year overall survival rates are estimated to be In this section we analytically highlight how covariates with different variances influence the model building process in regularized regression approaches. In the following we briefly describe the prominent types of regression models where regularized regression techniques are used, namely generalized linear models and the Cox proportional hazards model.In generalized linear models, there is a response In a time-to-event setting, observations often are given by an observed time In the following, we investigate the effect of differences in variance between covariates for componentwise likelihood-based boosting and penalized likelihood-approaches.Likelihood-based boosting transfers the idea of stagewise regression Componentwise likelihood-based boosting starts with an estimated parameter vector In a continuous response setting, i.e. a generalized linear model with identity link, componentwise likelihood-based boosting is equivalent to stagewise regression While componentwise likelihood-based boosting or the lasso provide sparse solutions, ridge regression For simplicity let us now consider a continuous response and orthogonal, centered covariates with variance Correspondingly, the parameter estimates are independent of rescaling covariates.For obtaining estimates in the case For the unstandardized covariates In the special case that the variances for the unstandardized covariates As seen in the KIRC data, data coming from RNA-Seq have three problematic properties, namely a skewed distribution, unequal variances for the individual genes and the presence of extreme values. In the following we propose to transform RNA-Seq data before applying a regularized regression approach to potentially deal with all these properties. We compare the use of the untransformed data with several different transformations. The transformations can be separated into three classes: transformations not including standardization of covariates, transformations including standardization of covariates and non-parametric transformations. Each of the transformations tries to handle either one, two or all of the three problematic properties. The naive way of analyzing RNA-Seq data in regularized approaches is to use the normalized counts without further transformation. We will call the normalized, but untransformed counts In ordinary regression analysis, the log transformation is often used for covariates with skewed distribution, and so might also be useful for RNA-Seq data. The log-transformed data are expected to be more or less normally distributed, depending on the degree of skewness before transformation. As the normalized counts For RNA-Seq data this does not lead to perfectly shaped normal distributions see , but theAnders and Huber The variances for the transformed data are approximately independent of the mean value, but they are still unequal for all genes. The variance stabilized counts have a less skewed distribution but may include extreme values.The Box-Cox transformation is a class of power transformations which has been developed to transform data in such a way that they satisfy the normality assumption Standardizing the covariates is the default implementation in many regularized regression techniques. A gene-wise standardization of the expression values contained in a covariate The standardized log transformation is a combination of log transformation and standardization. We standardize the log-transformed values by their estimated mean Again this does not result in perfect normal distributions, but the transformed data typically are less skewed, have less extreme values and have exactly mean zero and variance one for all genes. The standardized logs therefore potentially address all three problematic properties of RNA-Seq data.The standardized variance-stabilizing transformation is a combination of the variance-stabilizing transformation and standardization. We standardize the variance-stabilized values by their estimated mean This leads to empirical mean zero and variance equal to one for all genes.The standardized Box-Cox transformation is a combination of Box-Cox transformation and standardization. We standardize the Box-Cox transformed values by their estimated mean The standardized Box-Cox transformed data are less skewed, have less extreme values and have exactly mean zero and variance one for all genes.Working with ranks is a simple and popular method used in non-parametric statistics. Correspondingly, we consider The ranks are uniformly distributed from zero to the sample size Recently, the Blom transformation has been used in genetic association studies In order to compare the behavior of the transformations under realistic assumptions, we decided to perform simulations with covariate structure based on a real RNA-Seq dataset, specifically the KIRC data described above. So we do not need to specify an underlying distribution for the RNA-Seq data, which may in reality fit neither poisson nor negative binomial. Based on the real RNA-Seq data we simulate a binary patient outcome, which will be described in more detail in the following.We included the real RNA-Seq expression measurements To simulate a binary outcome In a first run of simulations we assumed that the DESeq-normalized counts To end up with a binary patient outcome we used a bernoulli-distributed random variable with probability For each of the 50 data sets, a logistic regression model is fitted by componentwise likelihood-based boosting. As covariates for this regression model, we consider the DESeq-normalized counts in the model without transforming them further , and all other transformations indicated above. For the ranks and the Blom-transformation we added a small In a second run of simulations, we assumed a logarithmic effect of the gene expressions The influence of outliers is decreased in this logarithmic setting, which may seem to be reasonable.For all 50 simulation runs we calculated a ROC-like curve in which we have the number of false positive genes on the x-axis, i.e. genes with true parameter equal to zero that have nevertheless been assigned a non-zero estimate by the boosting approach, and the number of true positive genes on the y-axis. The area under this curve from zero up to ten false positive genes can be interpreted as the mean number of true positives within ten or less false positive genes. In the scenarios with a linear effect, the four transformations that do not include standardization  have overall worse performance than the transformations including standardization and the non-parametric ones. The variance stabilizing transformation which tries to capture and make use of the distribution of the underlying data, performs worse than the naive analysis, which may be explained by the fact that it tries to estimate a gene's variance using information of similar genes. This is helpful in settings with very low sample sizes but in the setting of prognostic gene signatures, in which one normally has plenty of samples, this will lead to more biased results than estimating the variance just using the information of the particular gene. Standardizing means and variances increases the mean true positive rate, regardless of the specific transformation by which the standardization of variances has been achieved. Standardization via the variance stabilizing transformation is not performing better than standardization of the original scale or the log-scale and the non-parametric transformations achieve similar performance compared to the approaches using standardization. Increasing the number of informative genes from 10 to 20, while not changing the signal-to-noise ratio, does not affect the overall performance or the ranking of the transformations.The bottom panels of Generally, the results of the simulation study show that standardizing the expression data has a large effect on performance with respect to genes that truly have an effect, although different underlying assumptions of the effects result in other orderings of the performances of the transformations. The performance of the standardized variance-stabilizing approach, which is not better than using standardization of the log-scale, may furthermore imply that RNA-Seq data are much more complex than Poisson- or negative binomial distributions or that the complex re-parametrization is not fully capable of handling the extreme values. Furthermore, we find that the robustness of the rank-based transformations cannot only compensate the lack of all distributional assumptions, but can even outperform all other transformations in some scenarios.In the following we compare the transformations on the real datasets of patients with kidney renal clear cell carcinoma and acute myeloid leukemia introduced above. One main question is whether the RNA-Seq data include additional information concerning the survival times of the patients beyond the clinical covariates. We analyzed the clinical covariates using forward and backward Cox regression. For the KIRC data, four clinical covariates showed an impact on the overall survival time, which were age at diagnosis (continuous), laterality (the right or left kidney), tumor stage (I vs. II vs. III vs. IV) and platelet count . For the AML data, two of them showed an impact on the overall survival time, which were age at diagnosis (continuous) and sex . These clinical covariates were included as mandatory and unpenalized covariates in a Cox model fitted either by the lasso For more detailed evaluation, we use a resampling procedure, i.e. we randomly split the data in training and test set for a number of 50 times, drawing To quantify the added value of a model combining RNA-Seq and clinical data compared to a clinical model, we consider integrated prediction error curves (IPEC), i.e. the area under the prediction error curve. Thus, added value in the This measure of added value is zero if there is no improvement in the prediction error, and it is negative if the prediction performance of the combined model is worse than that of the clinical model. For combined models which improve the prediction performance the added value is in To have a closer look on the characterization of selected genes, we build up a model up to 200 steps for the KIRC data. The models with 200 boosting steps will certainly be overfitted, i.e. will include too many genes, but this will allow more stable characterization of the selected genes, e.g. when considering their median variance. We used the DESeq-normalization before transforming the data.The 0.632+ prediction error curve estimates, calculated over all 50 resampling data sets, is shown for each of the proposed transformations in For high-dimensional data arising from RNA-Seq many univariable testing procedures have been developed and implemented within the last five years As regularized regression techniques depend on the covariates variances and may more generally be critically affected by covariate distributions, we have compared different transformations of RNA–Seq data in terms of signature size, identification of important genes, and prediction performance. For gene expression measured by microarrays, it has been argued that no standardization is needed, as the measurements already are on the same scale First, we used a two-group simulation study with covariate structure based on real RNA-Seq data. We included scenarios with a few genes having large signals and scenarios with a larger number of genes having smaller signals. Transformations that result in equal variances for all individual genes were seen to perform better than those not standardizing variances. The performance of the rank-based transformations is consistently competitive in many different scenarios, which, e.g., was not the case for the default of standardization. Also, complex variance-stabilizing approaches did not outperform the rank-based approaches. This is in line with the comparison of univariable testing procedures for RNA-Seq data of Soneson et al. Second, we could show in an application to two different real datasets consisting of patients with kidney renal clear cell carcinoma and acute myeloid leukemia that the transformations behave similar in real data with time-to-event outcomes. Standardization of covariates leads to better prediction performance independent of the underlying transformation used  and independent of the underlying regression model (componentwise likelihood-based boosting and the lasso). The exact type of transformation used has a smaller effect on prediction performance than standardization and its effect seems to depend on the real dataset. The number of selected genes and their variances depend highly on the transformation used. There is a large overlap of selected genes for the transformations including standardization and a small overlap for all other transformations.The results of this study suggest that transforming the data to a distribution with equal variances for all genes is an important step if RNA-Seq data are going to be analyzed in regularized regression. The choice of a suitable transformation is essential and has a large influence on the genes being selected as differentially expressed, on the number of true positives and on the prediction performance of the model. While these results are supported by a simulation study with different numbers of important genes, i.e. sparse and non sparse scenarios, and two real data sets, this naturally does not guarantee generalizability to other data sets. However, the results at least point out that transformation and standardization are important issues that need to be carefully considered as a part of modeling, as they can have a strong detrimental effect on performance for different kinds of multivariable regression approaches. We expect that these results are not specific for While covariate variance and more generally covariate distribution, also might affect performance for other molecular platforms, the pattern might not necessarily be the same as for RNA-Seq data. For single nucleotide polymorphism data an investigation found even somewhat better performance for covariates without standardization