One of the approaches for conducting genomics research in organisms without extant microarray platforms is to profile their expression patterns by using Cross-Species Hybridization (CSH). Several different studies using spotted microarray and CSH produced contradicting conclusions in the ability of CSH to reflect biological processes described by species-specific hybridization (SSH).We used a tomato-spotted cDNA microarray to examine the ability of CSH to reflect SSH data. Potato RNA was hybridized to spotted cDNA tomato and potato microarrays to generate CSH and SSH data, respectively. Difficulties arose in obtaining transcriptomic data from CSH that reflected those obtained from SSH. Nevertheless, once the data was filtered for those corresponding to matching probe sets, by restricting proper cutoffs of probe homology, the CSH transcriptome data showed improved reflection of those of the SSH.This study evaluated the relative performance of CSH compared to SSH, and proposes methods to ensure that CSH closely reflects the biological process analyzed by SSH. DNA microarrays enable researchers to conduct large-scale quantitative experiments on gene expression, which can elucidate the mechanisms and the prediction of biological processes, the assignment of functions to previously un-annotated genes, the grouping of genes into functional pathways, and the prediction of the activities of new compounds [reviewed in ].One approach to microarray-based genomic research is to use cross-species hybridization (CSH) by hybridizing RNA of the studied organism to a microarray chip which contains transcripts of genes of a closely related species. CSH may be used in such cases when a microarray of the studied organism is unavailable e.g. -6). This. This6])Different microarray platform types have been used for CSH. Affymetrix oligo-micorarrays were used for CSH in several studies such as -18. SomeThe use of CSH presupposes the ability to analyze the gene expression profiles, so that biologically meaningful knowledge can be obtained. This paradigm was examined in several studies. Renn et al.  quantifiTo the best of our knowledge, our present study uses for the first time RNA samples from the same species, for both cross-species (heterologous) and species-specific (homologous) hybridizations. Thus, a direct comparison between results generated by CSH and SSH was feasible, and determination of the quality of CSH compared to SSH was determined.To examine whether CSH can produce data that reflects SSH data, we used the same RNA samples for both CSH and SSH microarray experiments. RNA samples were extracted from nematode-infected and non-infected (control) potato plants, sampled at two time-points (5 and 10 days – both represent early time-points of nematode infection). These RNA samples (i.e. test samples) were co-hybridized with a reference sample by CSH to a tomato microarray and by SSH to a potato microarray. The reference sample was pooled from all 8 samples . The CSHs were designated as 'PT' for potato samples on tomato microarrays; the SSHs were designated as 'PP' for potato samples on potato microarrays. The data discussed in this publication including detailed description of the potato and the tomato microarray platforms and all of the PP and PT expression data, have been deposited in NCBI Gene Expression Omnibus  and areTo assess whether the tomato microarray may be used for representation of a biological process that occurs in potato, we determined the extent to which the tomato microarray represents known potato genes. On the one hand, ESTs, which are the only sequences available for the tomato and the potato microarray printed clones, may not support determination of gene representation; their short size may inhibit accurate identification of transcripts (i.e. lack of overlapping between them may not necessarily indicate different genes). On the other hand, the complete sequences of the microarray printed clones – although may be the ultimate sequence class for comparisons – are not available for the tomato and potato arrays. This might also be the case for other microarrays of spotted clones.Therefore, we had to evaluate the complete clone sequences by using their assigned contigs (i.e. unigenes), and use unigenes for determination of tomato and potato microarrays representation of potato known genes. For this purpose, sequences of unigenes that were represented by the microarray platforms were used to blast against sequences of a potato unigene database . Sequences that matched with an E-value less than e-05 were considered homologous. Using this level of homology matching, the tomato microarray represents 29%  of the potato known genes. In comparison, the potato microarray represents 51.5%  of the potato known genes. On the tomato microarray, 83%  of the tomato genes were homologous to potato known genes, compared to 100%  of the potato microarray genes that were found homologous to potato known genes .It should be noted that until a genomic database is complete, these representation levels may change. Species representation level may be affected by the dynamics of a unigene database built. That is because unigenes may be split and re-merge until a unigene final form is attained; they may reflect splice variants ,28 and sNotably, cross-species representation levels may not necessarily be a proof of representation quality. Approximately 16%  of the potato unigenes registered more than one hit during blast between the tomato microarray unigenes and the potato unigene database. This may have been a result of sequence mismatches due to cross-species sequence divergence. Alternatively, multiple hits may result from genes being members of gene families . Multiple hits might lead to a reduction in the accuracy of the reflection of the biological process, since 2 or more different transcripts might hybridize to the same spot on the array.In conclusion, taking into consideration that the tomato microarray is a cross-species platform to potato, and contains less spotted clones than the potato microarray , the 6,147 potato unigenes represented by the tomato microarray  suggest that the tomato microarray may be used in the present study for potato gene expression profiling.In addition, Renn et al.  performeTo determine the extent to which CSH may reflect SSH transcriptomic results, we have performed various analyses for each of PT or PP whole chip expression data, and compared their results. We first, determined the average correlation coefficient of the reference sample signal between all pairs of hybridizations within PT or PP experiments. For this purpose we used present and marginal calls separately for PT or PP . Reference sample signal correlation coefficient of 0.88  or 0.89  was determined for PP or PT experiments, respectively. The similarity between the correlation coefficients suggests hybridization quality consistency for the same RNA sample for CSH, compared to SSH. This similarity implies repeatability of the CSH between the RNA and the microarray probes, despite low homology matches of a transcript to a spot, or where more than one transcript hybridizes to one spot. Although these 2 cases may also be present in SSH (e.g. different members of a gene family may hybridize to one spot in SSH), it is likely that they are more abundant in CSH.Second, Principal Component Analysis (PCA) results were obtained as another parameter for the resemblance between CSH and SSH data. PCA was determined based on test-sample:reference-sample ratios for each of the PT or PP datasets. Two components  were detected by PCA for the PP transcriptomic data Fig . Only onThird, the gene expression range was determined from the lower 0.1 to the upper 99.9 percentiles for all 8 test-sample:reference-sample ratios, and PP compared with PT. A clear reduction by a factor of 0.76  in the ratio was observed for PT compared to PP. This is in agreement with other CSH studies. Renn et al.  have fouConsequently, when gene expression profile was plotted between 5 and 10 d, for the (infected-) test-sample:reference-sample, for both PP and PT Fig , the PP Lastly, we wanted to examine whether the observed inconsistent reduction may be overcome once 2 hybridizations, which are a pair of a paired-observations, were compared. A pair of a paired-observation in our system is nematode-infected hybridization and the corresponding non-infected hybridization (e.g. '5a+' and '5a-'). The fold change for each pair was determined and averaged over biological replicates (i.e. over '5a' and '5b' and over '10a' and '10b'). This value served for identification of differentially regulated (i.e. >2 fold or <0.5 fold) genes for each time-point. For PP whole chip expression, data resulted in 591 and 790 differentially regulated genes at 5 and 10 d, respectively; only 80 and 52 differentially regulated genes were identified at 5 and 10 d in the PT whole chip data. Notably, the relatively low numbers of PT-differentially regulated genes compared to those of PP may have occurred, despite taking a liberal cutoff (2 fold). Therefore, the inconsistent reduction in signal intensity may not be overcome in our experiments by pair-wise fold determination.In conclusion, the reduction in the CSH hybridization results (the ratio between sample and reference), along with its inconsistency, compared to the SSH hybridization result, may bias gene expression profiling of a biological process. This was manifested in the inability to identify PCA components, in aberrant gene expression profiling on a transcriptomic scale and in a decrease in the number of differentially regulated genes. Thus, our results suggest that the overall CSH data does not support the use of a whole chip  analysis to provide a SSH comparative data profiles.Inconsistency between hybridization results obtained from different platforms may be merely due to the fact that different platforms were used , whereasMicroarray representation of species genes). Note that since overlapping of sequences may be essential once cross-platform comparison is considered [Mecham et al.  suggestensidered , a possinsidered , this miTo create a unigene-based matched tomato-potato microarray probe sets, sequences of 7,637 tomato SGN unigenes  served as queries to blast against 10,860 potato TIGR unigenes . BLAST resulted in 13,052 tomato SGN unigene – potato TIGR unigene homologous  pairs. This liberal match cut-off was used to gain a large range of E-values to be examined at later stages. These unigene homologous pairs were found to be representatives of 17,325 clone homologous pairs (i.e. tomato clone – potato clone pairs); each clone generated more than one pair, therefore, the number of homologous pairs is greater than that of the microarray spotted clones. The clone pairs served as probes for a System Expression Matrix (SEM) that was created by merging PP and PT probe corresponding data. SEM was utilized for further CSH and SSH analyses (see below).A measure of reflection of the SSH data by the CSH data may be having all the biological replicates at 5 and 10 d  clustered for the PT and PP data. Therefore, according to this measure, SEM data was filtered according to tomato microarray – potato microarray probe homology  to create subsets of SEM. Clustering results of these subsets were examined. E-value is a function of bit score, but weighs parameters such as unigene dataset size ; therefoHierarchical clustering of biological replicates of the unfiltered SEM data (infected:non-infected ratios) for each of the 17,325 homologous pairs resulted in no matches Fig . PCA of Lastly, filtering out the data of pairs that contained a tomato chimeric clone , support identification of differentially (>2 fold or <0.5 fold) regulated genes, identified in both the CSH and the SSH data. For this purpose, differentially regulated genes were identified based on the infected:non-infected ratio, averaged over 2 biological replicates, for each time-point and for each tendency of regulation . Mutual Of the PT data, 57% were false positives ; whereas 84% were false negatives . A large number of false negatives may result from a reduction in the number of differentially regulated genes in CSH, compared to SSH. This was also indicated by other CSH studies (e.g. ). HoweveIn the present study, we have directly compared results generated by CSH with those generated by SSH to determine the quality of the CSH results and the steps that should be taken in order to extract valid results from CSH. The results show difficulties in inferring transcriptome data from CSH with that obtained from SSH. However, once the information has been filtered for those data corresponding to matching probe sets by restricting proper cut-offs of probe homology, the CSH data reflected more closely that of the SSH to an extent that was quantified by identification of differentially regulated genes.Notably, all of the results that were generated in our PP-PT system may be specific to these particular platforms and to this particular experiment. A deduction from this system to other systems may not be straightforward. It is required that additional experiments will compare CSH to SSH using the same RNA samples (as in our system) in order to deduce general conclusions regarding derivation of SSH-like results from CSH data. Nevertheless, we suggest that this study may provide the reader with an approach to CSH. This may include guidelines for evaluating a CSH system, analyzing CSH data and assessing the quality of CSH results.1. Although evaluation of the phylogenetic distance between the 2 CSH organisms (the one of the microarray and the one in study) may indicate on the CSH ability to reflect a biological process , once ge2. Our data indicated that CSH whole chip results may not reflect SSH transcriptomic scale gene expression profiles. Thus, we suggest that the CSH transcriptomic data may not be trusted to reflect biological processes.3. To facilitate extraction of SSH-like knowledge out of CSH, we have generated cross-platform (and hence cross-species) matched probe sets. We suggest that the creation of corresponding probe sets be performed for each and any CSH system, for which genomic data is available. Once considering the level of homology for cross-species comparison, the bit-scores  should be optimized to the studied system. This may be easily performed in cases, where a species-specific platform exists, by maintaining a few RNA samples of the CSH experiment and hybridize them with the species-specific platform. This SSH supplemental experiment will allow a threshold determination for the specific CSH experiment. As for the rest of CSH systems, where a matched species-specific platform is not available, we suggest that the estimates presented here as a reference to start with, although these may be too strict; this particular experiment includes dissimilarities between CSH and SSH that may have been due to technical differences between the CSH and SSH systems. Finally, cross-species matched probe sets may be generated only for organisms with sufficient genomic data. As for other organisms, methods for extraction of biologically relevant knowledge from CSH need to be developed.4. The generation of the cross-platform matched probe sets was based on unigene sequences. Since several factors such as genomic database information of the studied species may affect the degree of clone representation by unigenes, an ideal candidate for sequence comparison may be the spotted-clone sequences . These sTwo cDNA microarrays, a tomato microarry and a potato microarray were used. The tomato cDNA microarray was developed and printed by the Center of Gene Expression Profiling ; it contains 9,140 sequenced clones, selected at random from a number of different cDNA libraries derived from a range of tissues including leaf, root, fruit and flower . The potTo determine the representation level of potato genes in the tomato microarray, its unigene sequences served as queries for a local BLAST search against a potato unigene database. IDs of the tomato microarray represented unigene were retrieved from the manufacturer . SequencAs a reference to species-specific genes representation, we evaluated the potato genes representation level by the potato microarray. This was done by dividing the number of the potato microarray represented potato TCs by the total number of potato TCs.Solanum tubersum cv. Desiree explants (stem including an axillary bud) were sectioned under sterile conditions from in vitro growing seedlings and transferred to Magenta boxes containing Gamborg's media. Four weeks post transferring to Gamborg's media, these seedlings were planted, each in a pot containing autoclaved quartz sand and grown under 16 h light per diem at 60% humidity. During seedling growing to plants, eggs of Meloidogyne javanica root-knot nematode were extracted from greenhouse cultures, and second-stage juveniles were hatched [M. javanica juveniles, where 6 plants were mock-inoculated with tap, sterile water. Roots from 6 infected and 6 non-infected plants were collected 5 or 10 days post nematode inoculation. Roots were observed under the microscope, and nematode feeding sites were selectively dissected from young lateral roots. To control the effect of tissue sectioning on gene expression, young lateral roots of the non-infected plants were dissected similarly to the infected roots, and collected. Dissected roots were snap-frozen in liquid nitrogen and immediately stored in a -80°C freezer. Consequently, biological material of 8 sources (of 2 time-points × 2 replicates × 2 treatments) was subject to RNA extraction.RNA was extracted from nematode-infected and non-infected (control) potato plants. Two time-points of nematode infection were examined, 5 and 10 d. For each time point, two paired observations were applied. All paired observations were taken at the same greenhouse, but at different times. Each paired observation consisted of 2 groups of 6 plants, grown on adjacent plates. Six plants were nematode-inoculated and the other 6 were mock-inoculated. A detailed description of plant growth, inoculation and dissection follows. These were applied independently for each paired observation. In order to get potato seedlings, 12  hatched . Three wTotal RNA was extracted with the RNeasy Kit  from infected and non-infected roots to yield 8 test RNA samples. In addition, equal amounts of 20 ug were pooled from all test RNA samples to form a reference sample, in which every test sample was equally represented . The tesEach test RNA sample was co-hybridized with the reference RNA sample to both a potato microarray and a tomato microarray, according to a microarray experiment reference design . This meFor the PT experiment, RNA was labeled so that all the test samples were dyed with Cy5 and the reference sample was dyed with Cy3. A detailed description of the methods used for RNA labeling, microarray hybridization and data acquisition, as in .For the PP experiment, RNA was labeled such that all the test samples were dyed with Cy3 and the reference sample was dyed with Cy5. A detailed description of the methods used for RNA labeling, microarray hybridization and data acquisition can be found in .Results of both experiments are available at GEO ; accessiSimilar data normalization was performed for both PT and PP data. The output files of the PT and PP experiments were normalized according to GeneSpring . First, all PP hybridization results were inverted to gain a test-sample:reference-sample ratios, similar to those of the PT experiment. Then, for each microarray experiment (PP or PT), Lowess normalization  was applWhole chip data analysis was performed separately for PP and for PT in a similar manner. For each experiment the following analysis procedures were performed. To determine the reproducibility of hybridization quality, the reference sample data were exported from GeneSpring to a text file. Excel (Microsoft) was used to determine correlation coefficients between pairs of reference sample data of eight hybridizations. This resulted in 28 comparisons, whose correlation coefficients – their averages and the standard-deviation values – were calculated. Next, GeneSpring was used to perform a Principal Component Analysis  on 8 teFor signal reproducibility between the experiments, the span from the lower 0.1 to the upper 99.9 percentiles was determined for each test-samples:reference-sample ratio. These values were compared between PP and PT hybridization data, for each of the 8 RNA sample hybridizations (e.g. between PT '5a+' and PP '5a+' data) by dividing the PT value by that of PP. The average and standard deviation was calculated for these 8 resulted ratios. Next, the (infected-) test-sample:reference-sample ratios were averaged over biological replicates and plotted as transcriptional profiles between 5 d and 10 d. Lastly, differentially regulated genes, with expression ratio of >2 fold or <0.5 fold between infected and non-infected samples  were identified by using GeneSpring.In order to generate tomato-microarray – potato-microarray clone match probe sets, unigene match probe sets were initially generated. For this purpose, the tomato microarray unigene sequences served as queries for a local BLAST search against the potato microarray unigene sequences. Unigene sequences of the tomato and the potato microarrays were retrieved from the manufacturers . A database index file was created for the potato microarray unigenes by using the NCBI FORMATDB utility. The tomato microarray unigene sequences served as queries for NCBI local BLASTN against the potato microarray unigene database. The search was limited to E-values <e-01. The BLASTN search gave a list of homologous unigene pairs along with their match information, including E-values and bit-scores.For each potato-tomato pair of unigenes the associated potato and tomato microarray clones were reconstructed to generate clone matched probe sets. For each potato-tomato unigene pair, each of the tomato unigene associated clones was paired with each of the potato unigene associated clones. This resulted in a list of clone pairs, which formed potato-microarray – tomato-microarray clone matched probe sets. These clone match probe sets were denoted as the unigene-based matched probe sets, and served for the construction of SEM (see below).The unigene-based matched probe sets were used to merge the corresponding data of PP and PT expression matrices (exported from GeneSpring) into one system expression matrix (SEM) using a Matlab  script. The merge included for each pair of matched clones the hybridizations data, pair-wise match information , and a flag that indicated the number of unigenes associated with a tomato clone (i.e. 1 or 2 for non-chimeric or chimeric clones).The system expression matrix was uploaded to GeneSpring such that, for each biological replicate, the infected:non-infected data was applied. Additional normalization was done by setting all values below 0.01 to 0.01 and by dividing all values by the median of the corresponding biological replicate values at each time-point examined . The following are analytical steps applied for SEM data. First, Centered Pearson correlation was used for hierarchical clustering of biological replicates for the whole SEM data and for SEM subset data. The latter corresponded to a range of bit-scores , in order to find the bit-score  threshold for a perfect match between PP and PT . Then, PCA was performed for both SEM whole data, and for sub SEM data that corresponded to the bit-score  threshold (disclosed before). Lists of genes, that were differentially regulated (i.e. had >2 fold or <0.5 fold change in expression) between nematode-infected and non-infected plants were extracted for each time-point by filtering on fold change on the averaged biological replicate . Corresponding differentially regulated gene lists were intersected (e.g. PP 5 d upregulated gene list was intersected with PT 5d upregulated gene list).CGEP: Center for Gene Expression ProfilingCSH: Cross-Species HybridizationE-value: Expectation valueGEO: Gene Expression OmnibusMYA: Million Years AgoNCBI: National Center for Biotechnology InformationPCA: Principal Component AnalysisSEM: System Expression MatrixSGN: Solanaceae Genomics NetworkSSH: Species-Specific hybridizationTC: TIGR ConsensusTIGR: The Institute for Genomic ResearchCB conceived the study, participated in its design, in the extraction and amplification of plant RNA, performed the hybridizations, scanning and quantification of the PT microarray experiment, executed the study analysis and participated in drafting the manuscript and writing the final draft of the manuscript; MB carried out plant growth and inoculation; TG participated in extraction and amplification of plant RNA; YK participated in the design of the study and revision of the manuscript; HC participated in the design of the study and revision of the manuscript; HK supervised the study, participated in its design and led its coordination and the writing of the manuscript. All authors read and approved the final manuscript.Tomato – Potato homologous unigene pairs. A list of unigene pairs, of tomato unigenes that are represented by the tomato microarray, each paired with its homologous potato unigene. Each pair record contains a tomato SGN unigene ID, a potato TIGR consensus (TC) ID, and the E-value of their match; an Excel file.Click here for fileSystem Expression Matrix (SEM). A list of clone pairs of the potato microarray spotted clones, each paired with a tomato microarray spotted clone (be noted that this clone pairing for SEM based on matching the clones associated unigenes). Each pair record contains: a tomato SGN clone ID, a tomato SGN unigene ID, a potato TIGR clone ID, a potato TIGR Concensus (TC) ID, E-value and bit-score of) of the two unigenes homology blast match, the tomato clone-unigene association type , and the infected:non-infected ratios of each of the biological replicate for the PP and the PT experiments; an Excel file.Click here for fileDifferentially regulated gene list. (A) Lists of differentially regulated genes of each experiment  for each time point (5 or 10 d) and for each tendency of regulation (up or down).(B) Lists of differentially regulated genes that were found mutual between PP and PT., for each time point (5 or 10 d) and for each tendency of regulation (up or down). Differentially regulated genes were detected for a subset of the SEM data, which corresponded to bit score ≥ 129 . Each gene record includes tomato SGN unigene ID, potato TIGR Consensus (TC) ID, fold change of gene expression ; an Excel file.Click here for fileDifferentially regulated gene list. Lists of differentially regulated genes that were found mutual between PP and PT., for each time point (5 or 10 d) and for each tendency of regulation (up or down). Differentially regulated genes were detected for a subset of the SEM data, which corresponded to bit score ≥ 129 . Each gene record includes tomato SGN unigene ID, potato TIGR Consensus (TC) ID, fold change of gene expression ; an Excel file.Click here for file


Nature Biotechnology provides a unique opportunity to probe into the true level of technical noise in such data.Microarray gene expression data are commonly perceived as being extremely noisy because of many imperfections inherent in the current technology. A recent study conducted by the MicroArray Quality Control (MAQC) Consortium and published in In the present report, the MAQC study is reanalyzed in order to quantitatively assess measurement errors inherent in high-density oligonucleotide array technology (Affymetrix platform). The level of noise is directly estimated from technical replicates of gene expression measurements in the absence of biological variability. For each probe set, the magnitude of random fluctuations across technical replicates is characterized by the standard deviation of the corresponding log-expression signal. The resultant standard deviations appear to be uniformly small and symmetrically distributed across probe sets. The observed noise level does not cause any tangible bias in estimated pair-wise correlation coefficients, the latter being particularly prone to its presence in microarray data.The reported analysis strongly suggests that, contrary to popular belief, the random fluctuations of gene expression signals caused by technical noise are quite low and the effect of such fluctuations on the results of statistical inference from Affymetrix GeneChip microarray data is negligibly small.The paper was reviewed by A. Mushegian, K. Jordan, and E. Koonin. Nature Biotechnology . A numerical maximization resulted in Δmax = 0.155, a very small number in view of the roughness of our estimate. Consider the following example. An average non-corrected correlation coefficient of 0.94 was observed when correlations were estimated in all gene pairs from a subset of 1000 genes selected from the HYPERDIP data. Using Δmax = 0.155 we can correct this value to obtain ρ > 0.7 and σu, σv ∈ , the result will be Δmax = 0.043, thereby allowing us to state that Setting The above calculations indicate that the presence of technical noise poses no serious problem for the statistical inference of correlation measures from microarray data, let alone two-sample comparisons. This is also an indirect indication that the effects of cross-hybridization are not of serious concern in modern Affymetrix data. Our rationale for the above statement is as follows. Since the competition of different oligonucleotide probes for the same transcript is random in nature, this process is expected to ultimately manifest itself in the observed technical variability, the latter having proven to be low. However, the proposed rationale is purely heuristic and cannot be independently verified as no technical vehicle is currently available for this purpose. The problem will probably become approachable after all "bad" probes are physically removed from the Affymetrix platform.Pairwise correlations between gene expression levels are known to be overwhelmingly positive -11. In sNature Genetics paper [Nature Reviews paper [When gene expression microarray technology emerged some 10 years ago, there was a great deal of enthusiasm. A 1999 cs paper  was entiws paper  was entiws paper . At the xij is random. This latter statement should not be interpreted as a "chaotic behavior" of the biological signal, of course. The random nature of xij has to do with biological variability; the latter is not a nuisance but a source of useful information. For example, the magnitude of inter-subject variability of gene expression is likely to reflect the tightness of control of a particular gene function by genomic regulatory mechanisms. Small sample sizes and inadequate statistical methodologies represent much more plausible explanations for many disturbing situations documented in the literature [The above analysis of the MAQC data demonstrates that the magnitude of technical noise in microarray data has been gravely exaggerated, which point of view is likely to originate from the deterministic way of thinking that attributes all the variability in the data to measurement errors, thereby entirely ignoring the fact that the biological signal terature ,15. We aThere are many potential sources of technical variation in microarray data such as preparation of samples, labeling, hybridization, and other steps of microarray experiment. Such sources have been extensively discussed in the literature ,17. Our Solexa's digital technology for gene expression profiling  has become widely available. We hope that the results of our analysis will positively affect the perception of usefulness of microarray technology. We also maintain the opinion [The authors of the present report strongly believe that the potential of microarray technology is enormous. This resourceful technology yields abundant multivariate information on general quantitative regularities in the functioning of the whole genome machinery at the level of transcription. Such regularities have yet to be deciphered. Furthermore, the structure of other types of future high throughput biological data is envisioned to be quite similar (i.e. array-like) to that of gene expression microarrays. In particular, the basic statistical problems will remain the same even after  opinion  that theThe statistical methods employed in this paper are closely intertwined with the reported results and cannot be presented in a separate section. All the necessary technical details of data analysis and sources of data are given Section 2.The author(s) declare that they have no competing interests.Both authors contributed equally to this work.This is a most useful study that starts to set the record straight on the "problem" of technical noise in Affymetrix gene expression analysis platform. It is astonishing that such a study has not been performed, or at least not published, by the manufacturer or by anyone in the scientific community over more than a decade of utilizing GeneChips – until now.My main question and concern is about the domain of applicability of the authors' conclusions. Here, some attention to careful wording could help. The main linguistic issue that remains to be sorted out is that the official name of the Affymetrix platform is "high-density oligonucleotide array", which in the past was sometimes contrasted with "microarrays", i.e., versions of the printed array technology. More recently, " microarray" has been casually broadened to signify any oligo-on-a-chip platform. I suppose that the authors need to declare what they are talking about, and what they are not, early and more often. For example, they say several times that their analysis concerns Affymetrix platform, and suggest at the very end that massive sequencing technologies, such as Solexa's, may have substantially similar statistical issues, but it is never stated whether or not anything in the current analysis is applicable to the ever-popular two-color arrays. Moreover, the statement in lines 3–5 of the Background (pg 2) is probably incorrect one way or the other, because low reproducibility "across platforms" may indeed be due, at least in part, due to high technical noise in some of said platforms. By the way, Lander's "Array of Hope" paper cited in the Conclusions deals mostly with printed two-color arrays, so citing it in this context is a bit misleading.I suggest to change "all" to "many" in the last line of pg 7: the random noise in Affymetrix data may be high for some of such practical purposes as, for example, expression QTL analysis.A disclaimer: I am not a statistician and cannot evaluate the equation (4).The authors report a statistical re-analysis of the Affymetrix microarray data produced by the Microarray Quality Control (MAQC) Consortium in order to evaluate the level of technical noise in high-throughput gene expression data sets. They report that, one outlier data set notwithstanding, levels of technical replicate variation in gene expression levels measured by the Affymetrix platform are 'quite low'. This low technical variation (i.e. noise) does not lead to any 'serious problem' when measuring correlations between gene expression profiles. Accordingly, the authors insist that rumors of fatal flaws in microarray technologically have been greatly exaggerated.In providing a review of this work, I would first like to issue a disclaimer: I am not a statistician and thus unqualified to evaluate the appropriateness (or lack thereof) of the statistical methods used in the paper. Having said that, the statistical models of technical noise in microarray data used in this paper are mercifully simple. Furthermore, the methods and results are presented clearly and succinctly. This combination of brevity and clarity is a real strength of the paper; the authors have evidently taken pains to ensure that the work is accessible to biologists for whom the implications are obviously most important. However, the lack of methodological and analytical detail is also a liability in some places. Fortunately, this can easily be remedied. Below, I detail some of the places where I believe that a greater depth of analysis and exposition would benefit the manuscript.1. Expression level measure used. The authors re-analyze the MAQC data and use the original CEL files as a starting point. There are a number of different methods that can be used to extract 'expression levels' from the raw data in the CEL files, and it is not clear which method was used here. The Bioconductor RMA background correction procedure is mentioned in passing later in the manuscript, but it is not clear exactly how or whether this was employed to get the expression levels analyzed. More explicit detail should be provided here. In particular, it is important to know whether the authors used both perfect match and mismatch probes or perfect match only.2. Gene versus probe (set) expression level. A more biologically pressing issue is the conflation of gene versus probe (set) in the manuscript. The authors state that they measure "correlation coefficients in gene pairs". However, there are many cases where multiple probe sets map to single genes. How were these handled? Are the correlation values discussed and shown only for probe set pairs?4. Pairwise correlation analysis Figure . The con5. Sources of variation. The sources of technical variation are not considered in this manuscript. Biologists will be very interested to know if there are any systematic and readily identifiable sources of technical noise in the Affymetrix probe sets and associated analytical tools used to derive expression levels. While the suggestions below regarding sources of variation are discretionary, a bit more work on this subject would strengthen the manuscript and make it even more useful to biologists.5A. Sources of variation – expression level extraction. There are a number of analytical tools that are used to derive probe set expression levels from the raw data in CEL files. Among the most commonly used are MAS5, RMA and gc-RMA. As mentioned above, it is not entirely clear which method was used. In addition, it would be very interesting to know how the technical noise levels change when the different methods are used – e.g. which method gives the least noise.5B. Sources of variation – probe GC content. The GC content of probes is can affect hybridization intensity (methods like gc-RMA correct for this). Is there any relationship between probe sequence context and the technical noise in the data analyzed here?5C. Sources of variation – probe set count. Probe sets consist of different numbers (11–20 for newer arrays) of short oligonucleotides. Does the number of probes per set have any effect on the technical noise observed?6. It is not clear how the low level of technical variation indicates that the effects of cross- hybridization are negligible (pg. 5). More evidence, or more explicit argument, needs to be offered to support this point. For instance, does the presence of closely related duplicate genes have any effect on technical noise levels?7. A final minor point concerns the authors' contention that their conclusion of low technical noise in microarray data and the associated robustness of statistical analyses of these data represent a "radical change in the perception of the usefulness of microarray technology." This reads as an unnecessary overselling of their results. While many reservations about the reliability of microarray data have indeed been raised, there can also be little doubt that there is a broad consensus among biologists that these data are still very useful. The sheer number of microarray experiments, databases and publications supports this notion. They might consider toning this statement down a bit to emphasize that their results confirm the utility and reliability of this widely used technology.This paper addresses an obviously important problem and reports a potentially extremely important observation, namely, that the measurement error  in Affymetrix microarray data is not particularly high and does not substantially affect statistical inferences made from microarray data, in particular, correlation coefficient estimates that are particularly sensitive to noise. This work is a rare case of a statistical study that come up with an optimistic message, namely, that the common perception of the extreme noisiness of expression array data making them useless for many purposes is a serious exaggeration.I quite agree with the authors that this is a valuable message that needs to be brought to the attention of the large community of researchers using microarray results for their studies into diverse facets of biology. I am somewhat surprised that this simple analysis and straightforward conclusions were not part of the MAQC paper itself but, of course, such omissions are common enough. At the same time, I would like to note several issues that, in my opinion, require a somewhat more nuanced or more comprehensive approach.First, the point of Klebanov's and Yakovlev's paper is not that the noise is low in some absolute sense but rather that it is low compared to a typical, biologically relevant signal. Put another way, when we detect a large difference between two expression profiles, it is unlikely to be due to technical noise and instead should be interpreted in biological terms. I believe this generally to be true; by the way, this case has been made in the literature, with comparisons to the differences between replicates used as an obvious standard – at the distinct risk of self-promotion, I will mention one of my own recent papers: Tsaparas et al. Global similarity and local divergence in human and mouse gene co-expression networks. BMC Evol Biol. 2006 Sep 12;6:70. However, it also has to be acknowledged that, for many situations, where subtle differences are involved, the level of technical noise will be a real problem. I cannot agree with the authors' statement (in the Conclusions) that the noise level is low for all practical purposes – there must be quite a few purposes, very practical ones, for which even this amount of noise will not be tolerable. In that respect, it does not help that the authors use only one real-life microarray study to showcase the lack of dramatic effect of technical noise. I suppose this will work as proof of principle but, perhaps, some more caution is needed in the conclusions.Second, I suggest that the low level of technical noise reported by Klebanov and Yakovlev is the best case scenario. The replicates they analyze come from a specially launched, well-controlled study on microarray data quality. Indeed, even in that study, there was a serious outlier, test-site #4. I am afraid that, in real life, the noise level is, very often, greater; at least, this issues should be addressed.xij is random in the sense that, due to the inevitable presence of intrinsic noise, this signal contains a random component and should be treated as a random variable. However, a reader could also take this to mean that the signal varies completely stochastically which is, of course, far from the truth. Furthermore, intrinsic noise in gene expression has been analyzed and discussed repeatedly, and some citations seem to be necessary here . In addition, I think that this deterministic worldview is not the only and, possibly, not even the principal source of the common perception of high technical noise in microarray data. Another and, possibly, more common reason is likely to be the genuinely high noise in early microarray experiments; I suspect that, in this case, technology progresses faster than many researchers realize.Third, the authors suggest, in the Conclusions, that the popular notion of the prohibitive noisiness of microarray data stems from the deterministic view of biological data, in other words, disregard of intrinsic, biological noise . I think this is a perceptive remark but it has to be made somewhat more clearly. I interpret Klebanov's and Yakovlev's statement that "the biological signal Fourth, I think that the Background section is somewhat too succinct. Non-specialist readers would benefit from a few extra sentences explaining, in particular, in more quantitative terms, what were the previous estimates of the noise level.In summary, I believe that this paper reports an important observation but could be improved by a more comprehensive discussion of the present and previous results, and perhaps, by slightly more cautious conclusions.


Statistical methods to tentatively identify differentially expressed genes in microarray studies typically assume larger sample sizes than are practical or even possible in some settings.The performance of several probe-level and probeset models was assessed graphically and numerically using three spike-in datasets. Based on the Affymetrix GeneChip, a novel nested factorial model was developed and found to perform competitively on small-sample spike-in experiments.http://www.stat.usu.edu/~jrstevens.Statistical methods with test statistics related to the estimated log fold change tend to be more consistent in their performance on small-sample gene expression data. For such small-sample experiments, the nested factorial model can be a useful statistical tool. This method is implemented in freely-available R code (affyNFM), available with a tutorial document at  Over the past decade, gene expression microarray technology  has becoAn ongoing collaboration with the J. R. Simplot Company involves the cloning of cattle using nuclear transfer (NT). Published results include an evaluation of the differences in NT success rates using cow vs. heifer oocytes  and an eBecause of the high cost in time and money for NT and gene expression studies, the high level of work necessary to perform NT, and the difficulty in achieving successful bovine NT pregnancies, the sample sizes in these microarray experiments tend to be quite low. While many biomedical applications currently involve dozens of samples, the microarray work done thus far with bovine NT studies has been limited to six to eight samples for each comparison. Much of the statistical methodology that has been developed for microarray data analysis over the past decade  has beenThis study evaluates alternative statistical models in an effort to identify appropriate approaches for microarray experiments with small sample sizes. The Results and Discussion section consists of a summary of these methods, including a novel presentation of our probe-level nested factorial model, and a discussion of the relative performance of these methods on spike-in data.The relative performance of competing statistical methods can be assessed using spike-in data, and the most promising method was applied to the bovine NT data. For this study, all data considered are of a simple treatment vs. control design.The motivating bovine NT dataset, mentioned in the Introduction subsection, was taken from an experiment in the cloning of cattle (NT). Tissue samples were taken and prepared from cotyledon (part of the uterine side of the placenta) in an impregnated cow. Four of these pregnancies involved NT and three were control, making 7 total arrays. For these data, we are interested in identifying genes that behave differently in the NT samples than in the non-NT samples. The Affymetrix bovine array was used, with 24,128 probesets.In spike-in microarray experiments, the "truth" is known a priori. In particular, genes whose mRNA has been artificially increased in a sample to be hybridized onto the array are said to be "spiked-in." The "expression levels" of these genes can be controlled by modifying their spike-in concentrations. Thus their "expression levels" can be made to change across "treatments." That is, they are known to be differentially expressed. Hence it can be verified how well statistical models identify differentially expressed genes in general by how well they find spiked-in genes in these types of experiments.Data from three spike-in experiments were used to compare the different statistical models. The Affymetrix HGU95A spike-in data  consist The Affymetrix HGU133A spike-in data  consist The third and final spike-in dataset used was the "Golden Spike" dataset . There hThe Affymetrix GeneChip is among the most commonly used microarray platforms, and typically represents each gene on the array as a set of 11-20 probes, called a probeset. Preprocessing methods such as RMA [The purpose of this study was to compare several probe-level models with one of the most popular probeset models. In this section, we summarize the approaches of several methods (both probeset and probe-level) to calculate test statistics for evaluating differential expression. Unless otherwise stated, RMA preprocessing  was donek we have the model:A nested factorial model (NFM) is a basic structure in experimental design and is developed and presented for the first time in this work with application to microarray data. For each gene Yijl is the background-corrected quantile-normalized log-scale perfect match intensity for probe l of the gene on array j under treatment level i, Ti is the treatment effect, Sj(i) is the subject (or array) effect nested within treatment, Pl is the probe effect, (TP)il is an interaction between treatment and probe, and (SP)jl(i) is used as the error term. This model clearly takes into account any probe information that may be lost in a probeset model.H0: T1 = T2, and we can look at an F-statistic for the treatment effect. This statistic can be obtained via restricted maximum likelihood (REML) estimation, as implemented in the lme function of the nlme R package [Since we are interested in determining whether or not each gene is differentially expressed between treatment levels, we are testing  package ,16.It is important to note that, due to the nature of the nested design, an iterative process (REML) is necessary to obtain the F-statistic for each gene. This iterative process takes some time and can be very slow when there is a very large number of factor levels, such as a large number of probes per gene on some array types (as in tiling arrays).Bolstad (2004) suggested a probe-level model , referreYlj is the background-corrected quantile-normalized log-scale perfect match intensity for probe l of the gene on array j, αl is the probe effect, and βj is the array effect. This model is fit using the fitPLM function of the affyPLM R package [is used, where  package . Bolstadand:c is the total number of arrays in the dataset. The elements of c depend on how many arrays are in each treatment level. For a simple two-group design, element j of c is j is in treatment group 1, j is in treatment group 2, and 0 otherwise, where ni is the number of samples in treatment group i. Then c'Here, k:Limma/eBayes (LE) is a very popular probeset-level method in which a hierarchical Bayes model is used to define a moderated test statistic for assessing differential expression . First, Yijk is the log-scale expression level for gene k on array j in treatment group i, and βk,1 is the treatment effect. Ti is the treatment level of the array, coded as a dummy variable . The εijk is an error term with Var(εijk) = Here, Var is:Estimates Vk corresponding to the variance of βk,1. Under this H0, tk follows a t-distribution with degrees of freedom dk.Here, One limitation of this traditional t-test statistic is the reliability of the estimate d0 are estimated from the data  with empirical Bayes methods. The posterior mean:where k is:is used as an updated estimate of the variability H0, d0 + dk . LE is a very efficient method in the sense that it runs as fast as PLM. The LE method is implemented in the R package limma [Under the ge limma .Lemieux (2006) proposed a probe-level linear model (PLLM) . It usesk, where Yijl is as defined following Equation 1, αl is a combination of the expression level of the gene across arrays and specific probe affinities, and Ti is the effect of treatment level i. The treatment effect is estimated using standard least-square procedures. We plot these against the average log probeset intensities to reveal a Gaussian mixture model, as in Figure for each gene To fit the Gaussian mixture, after using such plots to decide the number of components to include, we used the implementation in the R package mclust . To testOne drawback to this model is that it can not be automated. One has to manually choose the number of components to use in the Gaussian mixture model by looking at these plots. It is not always clear how many components to choose for the mixture model  uses a hierarchical Bayes model to obtain a moderated, weighted PM probe ,22. Withkage plw ,22.c'c and For purposes of comparison, the naive fold change (FC) across treatments is also considered as a test statistic in identifying differentially expressed genes. For this study, the (log) fold change was calculated as the numerator of Equation 3, i.e., yj(i) be the vector of background-corrected quantile-normalized log-scale perfect match intensities on array j within treatment level i. For treatment level i, let μi) , the null hypothesis of interest is H0: μ(1) = μ(2).The robustified MANOVA (RMANOVA) approach is a probe-level model based on the framework of multivariate analysis of variance , fit on μi) for treatment level i. Similarly, let μ, obtained by taking the component-wise median across all yj(i) for all treatment levels. Let ni be the number of arrays in treatment level i. Then the within-treatment and between-treatment error matrices are estimated respectively as:Let the vector and:Here the sum and median are applied component-wise.Two test statistics are proposed by Xu and Cui (2008) , referreHere again the median is applied component-wise. Both of these test statistics are constructed with the same philosophy of the F-statistic in a one-way ANOVA model, as the ratio of between-treatment to within-treatment error. However, unlike the ANOVA F-statistic, the numerators of these test statistics are not directly related to the log fold change. Still, larger test statistics are taken as greater evidence of differential expression. Xu and Cui (2008)  provide Yijl be the background-corrected quantile-normalized log-scale perfect match intensity for probe l of the gene on array j under treatment level i. For each probe l of the gene, the FIRSTP approach is based on the simple model:Rubin (2009) proposed a "first principles" (FIRSTP) probe-level approach to testing for differential expression . For a gμl is the population mean for probe l, and Ti, l is the effect of treatment level i for probe l. Using traditional parametric ANOVA methods, a p-value (pl) is obtained for the null hypothesis H0: Tl 1, = Tl2, , for each probe l of the gene. where l) of 1 - pl. Larger values of the test statistic are indicative of greater evidence of differential expression. This FIRSTP approach is similar in spirit to the PLW method (because both utilize medians of probe-level test statistics), but differs in that the "first principles" philosophy limits the number of assumptions and model components. We used a custom R implementation of this FIRSTP method.For the overall test of differential expression for the gene, the test statistic for the gene is the median  is used to approximate the distribution. The test statistic for each gene is then the probability that the gene is up-regulated (or down-regulated); i.e., the probability that μ1 >μ2 (or μ2 >μ1), is then calculated. These test statistics can be converted into a "p-like-value," which is the probability that each gene is differentially expressed. The p-like-values are more comparable to the p-values returned from other methods, in that smaller values represent stronger evidence of differential expression. These methods have been implemented in the R package puma [After this preprocessing, PUMA uses a hierarchical model to obtain the posterior distribution on age puma .In order to understand how well each of the previously defined statistical models can identify differentially expressed genes, each was compared simultaneously by considering the true positive rate (TPR) and the false positive rate (FPR) in the three spike-in datasets described previously. For each model, a test statistic was obtained for each gene and the absolute values of these test statistics were sorted in increasing order. The TPR and FPR values were defined as in Bolstad (2004) : For theFor each spike-in dataset, a receiver operating characteristic (ROC) curve was created to visually assess how well each model performed. The ROC curve is a plot of TPR vs. FPR. A higher curve indicates higher performance in identifying differentially expressed genes. "Higher" here can be assessed both graphically (with the ROC curve) and numerically .Because our focus is performance in small-sample studies, we also looked at subsets of the spike-in studies. For example, the HGU95A dataset has four arrays in each of two treatments. In addition to this 4 × 4 comparison, we also tested for differential expression in all 16 3 × 3 and all 36 2 × 2 comparisons. For each method fit, the TPR and FPR were averaged across all 16 3 × 3 comparisons, and separately averaged across all 36 2 × 2 comparisons. The HGU133A and Golden Spike datasets both have three arrays in each of two treatments. In addition to their 3 × 3 comparisons, we averaged the TPR and FPR across their respective 9 2 × 2 comparisons.Figure Figure The NFM method is consistently among the top performers at low FPR, except in the Golden Spike data. (The poorer performance in the Golden Spike data is examined in the two following paragraphs.) The PLM methods perform well in the HGU95A data, and are among the best at low FPR for the Golden Spike data with GCRMA preprocessing. The LE method has a slightly variable performance, doing well in the HGU95A data, and among the best at low FPR for the HGU133A data with RMA preprocessing, but among the worst at low FPR for the HGU133A data with GCRMA preprocessing. The PLLM method is more variable, as it is clearly the best performer on the Golden Spike data with RMA preprocessing, and among the best at low FPR on the HGU133A data with GCRMA preprocessing, but among the worst at low FPR for the Golden Spike data with GCRMA preprocessing. The PLW method is among the best for the Golden Spike data with GCRMA preprocessing, and at low FPR for the HGU133A data with GCRMA preprocessing. The FC method is among the best for the Golden Spike data with GCRMA preprocessing and at low FPR for the HGU133A data with RMA preprocessing, but among the worst at low FPR for subsets of the HGU95A and HGU133A data with GCRMA preprocessing. RMANOVA tends to be among the lowest performers, but gives the best overall performance on the HGU133A data with RMA preprocessing . The FIRSTP method gives a variable performance, and is never the best or worst overall. PUMA is the top overall performer on the HGU133A data with GCRMA preprocessing, but there (and consistently in the other data sets) it is among the low performers at low FPR.Because the Golden Spike data have large numbers of genes spiked in at each of several known fold changes, the performance of these statistical methods can be compared in identifying the spiked-in genes by fold change. Figure The methods comprising this fourth level of performance  all share a common characteristic, that their test statistics are related to the simple estimated log fold change. This is due to their being based on traditional ANOVA-type models.  When this estimated log fold change is low, the test statistics for these methods tend to be lower. This is what caused these methods' performance to drop in the Golden Spike data at fold-changes 1.5 and 1.7. Figure All probe-level and probeset-level methods discussed previously were applied to the motivating bovine NT data, with RMA preprocessing for the non-PUMA methods. The top 500 genes (based on ranked test statistics) from each method were identified, with a union of 2,432 genes across the methods. The overall ranks of these 2,432 genes within each method were used to compare the results of each method while allowing for scale differences between test statistics from each method. Figure g is the number of treatment levels and ni is the number of samples within treatment level i. In other words, when the null hypothesis H0: T1 = T2 is true, the F-statistics will theoretically have this approximate distribution, and p-values could be obtained as tail probabilities. The validity of this distributional assumption can be assessed using the F-statistics for the non-spiked-in genes in the three spike-in datasets, because the null hypothesis is known to be true for these genes.Based on its consistently strong performance on the spike-in data, including small-sample subsets, NFM was applied to the motivating bovine NT data. To assess statistical significance in this application, p-values (with subsequent multiple testing adjustment) are necessary. The F-statistics from NFM have a theoretical The F-quantile plots in Figure An alternative sampling distribution is generated by a permutation approach. Existing permutation or bootstrap approaches permute or resample treatment labels, and the p-value for each gene is the proportion of all resamples producing a test statistic for the gene at least as extreme as the original test statistic for the gene. An implementation of such an approach  is found/(3!·3!) = 20 possible permutations of treatment labels. Due to the one-sided nature of the NFM test statistic, there are actually only half that number of permutations that will give unique (or non-redundant) test statistics for any given gene. For example, the treatment labels  and  will produce identical F-statistics for a given gene, where C represents control and T represents treatment. Then the smallest p-value using this per-gene permutation approach is 1/10, and no claims of significance can be made, particularly after adjusting for multiple hypothesis testing.For example, in the HGU133A dataset, there are 3 "control" and 3 "treatment" arrays under consideration, so there are 6!Instead, we propose to use the same sampling distribution for the test statistics of all genes in an experiment. If the n Figure , we geneFor a given experiment, we enumerate all treatment label permutations, and for each permutation we calculate the NFM test statistic for each gene. Our sampling distribution is the collection of test statistics for all genes across all permutations of treatment labels. Then for each gene, the NFM p-value is the proportion of all test statistics in this collection that are at least as extreme (large) as the gene's test statistic using the original treatment labels. The false discovery rate  is contr<0.1) is more common for spike-in genes with higher control and treatment concentrations . Despite improvements in recent years, bovine somatic cell nuclear transfer efficiency remains quite low. While pregnancy rates of transferred SCNT embryos approach those of transferred in vitro fertilized (IVF) embryos, pregnancy loss throughout gestation, and particularly in the first trimester, is much higher for SCNT pregnancies. The placental structure in SCNT pregnancies is often remarkably different in bovine SCNT pregnancies, with significantly fewer and significantly larger cotyledons and caruncles . In addiAn evaluation of the most significantly over-represented biological processes  provides interesting insight regarding aberrations in SCNT cotyledons. Three main categories of over-represented biological processes were identified - cell cycle regulation, metabolism, and early developmental processes. Cell cycle arrest was the most significantly over-represented biological process. The most significantly over-represented metabolism ontologies were RNA processing and sphingolipid metabolic process. Among the most significantly over-represented developmental processes were brain development, neuron maturation, hindlimb morphogenesis, in utero embryonic development, and muscle organ development. These all fit very well with the abnormal phenotypes we see in SCNT pregnancies.Of the statistical methods considered here for small-sample gene expression data, none was shown to be consistently superior, but a few interesting conclusions can be made. The fold-change-based methods  tended to be more consistent in their performance and provided the expected increase in test statistic ranks for higher fold-changes. By contrast, the non-fold-change-based methods  tended to be more variable in performance, and two of them (PLLM and PUMA) showed an unexpected decrease in test statistic ranks for higher fold-changes.For small-sample gene expression experiments, the nested factorial model (NFM) was shown to be a competitive statistical approach for identifying differentially expressed genes, particularly at low false positive rates. The NFM method uses a permutation approach to calculate a p-value for each gene. These permutations add computational expense, but this expense must be taken in perspective. Small-sample studies (for which NFM is of immediate interest) are common in applications where physical samples are either scarce or prohibitively expensive. When a research team is unable to acquire additional biological samples or chips to perform a larger-sample gene expression study, they will have to compensate with additional waiting time. For the HGU95A spike-in dataset, with 12,626 genes on 4 control and 4 treatment arrays, the total runtime to acquire the permutation p-values was about 38 hours on a typical desktop PC. The HGU133A spike-in dataset, with 22,300 genes on 3 control and 3 treatment arrays, required a runtime of about 16 hours. The Golden Spike spike-in dataset, with 14,010 genes on 3 control and 3 treatment arrays, required a runtime of about 10 hours. The runtime for the bovine NT dataset, with 24,128 genes on 3 control and 4 treatment arrays, was about 50 hours. When a research team faces insufficient resources for a larger-sample gene expression study, these runtimes should not seem particularly onerous.http://www.stat.usu.edu/~jrstevens. A tutorial document (including a reproducible example) is provided on the same website.The NFM methodology described here, including the p-value permutation approach and trivial rounding in cases of initial non-convergence, has been implemented in R code (affyNFM) freely available from the first author's website at We conclude with a caveat for those who may attempt too much analysis with too little data. The development and availability of statistical methodology specifically for small-sample gene expression studies (such as the NFM here) should not be taken as encouragement of poor design. Our motivating example involved data where it was extremely expensive to obtain a single biological sample (of the clone). In practice, we would have preferred a larger sample size to provide more information on population-level biological variability. It was not merely expensive to acquire the chips, and not merely inconvenient to acquire additional biological replicates. We emphasize that sufficient sample size for useful inference should remain a guiding principle of experimental design in gene expression studies.JRS conceived of the study, supervised the M.S. thesis of JLB, prepared the affyNFM implementation, and drafted the manuscript for publication. JLB developed the R code to execute and compare the various statistical methods. KIA and KLW designed, executed, and provided interpretation and context for the original bovine NT study which motivated this work. All authors read and approved the final manuscript.


DNA microarrays provide data for genome wide patterns of expression between observation classes. Microarray studies often have small samples sizes, however, due to cost constraints or specimen availability. This can lead to poor random error estimates and inaccurate statistical tests of differential expression. We compare the performance of the standard t-test, fold change, and four small n statistical test methods designed to circumvent these problems. We report results of various normalization methods for empirical microarray data and of various random error models for simulated data.Three Empirical Bayes methods  were the most effective statistical tests across simulated and both 2-colour cDNA and Affymetrix experimental data. The CyberT regularized t-statistic in particular was able to maintain expected false positive rates with simulated data showing high variances at low gene intensities, although at the cost of low true positive rates. The Local Pooled Error (LPE) test introduced a bias that lowered false positive rates below theoretically expected values and had lower power relative to the top performers. The standard two-sample t-test and fold change were also found to be sub-optimal for detecting differentially expressed genes. The generalized log transformation was shown to be beneficial in improving results with certain data sets, in particular high variance cDNA data.Pre-processing of data influences performance and the proper combination of pre-processing and statistical testing is necessary for obtaining the best results. All three Empirical Bayes methods assessed in our study are good choices for statistical tests for small n microarray studies for both Affymetrix and cDNA data. Choice of method for a particular study will depend on software and normalization preferences. Microarrays provide large-scale comparative gene expression profiles between biological samples by detecting differential expression for thousands of genes in parallel. Typically, systematic error (bias) in the measurements is removed at the background correction and normalization steps, and is followed by statistical testing. In the most common type of study, statistical testing produces a list of genes that are differentially expressed across two or more classes  .Extensive research has shown that choice of pre-processing methods designed to correct for bias in the measurements can have a substantial impact on rank ordering of gene expression fold-change (FC) estimates for both cDNA  and oligThere is consensus in the statistical community that statistical tests of differential expression are preferred over FC for inference . One advAlthough microarray studies with sample sizes of five or more observations per class are becoming increasingly common, cost considerations and the need for small scale initial studies mean that many studies are conducted with smaller sample sizes. The use of classical statistical tests such as the t-test is sub-optimal for small sample studies, however, because of low statistical power for detection of differential expression. This has led to the development of small sample size error estimation methods which borrow information from the entire data set or from a subset of the data to improve error estimation accuracy and precision ,12.There has been a number of studies analysing the performance of statistical tests applied to microarray data but few have used data where the differentially expressed genes are known in advance. Qin and Kerr  comparedA number of studies have examined small n tests. Kooperberg et al.  comparedWe use simulated data, two publicly available empirical Affymetrix Latin-Square data sets, and two cDNA data sets in which the differentially expressed genes are known to assess the relative performances of FC, t-statistic, and four small sample size statistical tests which all borrow strength from other genes in different ways. For consistency, we refer to the various tests by their software implementation. The Local Pooled Error method (LPE) z-statistic  is basedThe HGU133A and HGU95 Latin Square data sets  are baseWe compared performance of the statistical tests (see Methods) for detecting two-fold differential expression after the data had been normalized by six popular expression summary algorithms: MAS 5.0 , dChip , RMA 3]3], gcRMATwo spotted cDNA experiments  were alsThe data were normalized using the lowess normalization of LMGene , with anTwo limitations of the above spike-in data sets make them less challenging than most biologically motivated studies: variation across technical replicates is lower than that typically observed across biological replicates -36 and mN distribution, where u is the "true" expression intensity  distribution) for a single gene in log2 scale and σ2 is the random error associated with the gene's expression measurement. In the null hypothesis data, each gene-specific u remains constant across both groups; in the "spike-in" data, μ was upwardly regulated by a specified fold change for 100 genes in the treatment group.Each simulation experiment consisted of 1000 iterations of ten thousand genes for each of two groups (control and treatment) with 3, 4, or 5 replicates per gene. Gene intensity values were generated by randomly selecting from a μ was assigned a value from a set of 10 log2 values ranging from 4 to 8.5 in 0.5 increments. The log2 fold change applied to the "spiked-in genes" was implemented by an additive adjustment to their u in the treatment group chosen from a range of 10 values from 0.2 to 1.1 (fold change from 1.15 to 2.15) in 0.1 increments. There are 10 "spiked-in" genes at each of the 10 concentration values and each of those 10 genes at a particular concentration value is upwardly regulated by one of the 10 different fold changes.For each "spiked-in" gene, c) of the random error term (ϵ) is a constant across all genes. In the inverse gamma variance model, c is drawn randomly from an inverse gamma distribution, IG, where 3 is the shape parameter and b is the scale parameter. In the local variance model, ϵ is a combination of additive (ϵa) and proportional (ϵp) components, as outlined in Rocke and Durbin [The random error of each gene is derived from one of three variance models: common, inverse gamma, and local Table . In the d Durbin .The parameters of each variance model were varied to create data sets from low variance (approximating the Latin Square and some cell line data sets) to medium and high variance  ratio is typically represented on the logwhere The independent t-statistic provides a standardized estimate of differential expression according to the following formula:whereni are the variance and number of replicates for the ith group, respectively. 1 + n2 - 2 degrees of freedom.Jain et al.'s (2003) local pooled error test is based on the model that genes with similar observed intensities have similar true variances and is calculated as follows:wheren1 and n2 are the sample sizes for the two groups. The variance is adjusted by π/2 due to the increased variability of using medians when calculating the fold change. The associated probability of the z-statistic under the null hypothesis is calculated by reference to the standard normal distribution.The fold change is calculated using a difference of medians and the gene specific The following three tests use an Empirical Bayes method to estimate error associated with differential expression and use a statistical test identical in form to the independent t-statistic shown in equation 2. The tests use a posterior variance, dg are the degrees of freedom and d0 and σ2 are the prior degrees of freedom and variance estimates respectively [d0 and ectively . The thrSmyth  developeThe linear model is as follows:yg is the expression summary values for each gene across all arrays, X is a design matrix of full column rank and αg is the coefficient vector. The contrasts of coefficients that are of interest for a particular gene g are defined as βg = CTαg. Although this approach is able to analyse any number of contrasts, we examine two sample comparisons only so βg can be defined as the log fold change where βg, are assumed to be normal and the residual variances, The contrast estimators, d0 and dg and g, respectively. Because we examine only two sample comparisons, dg will always be equal to n - 2 where n is the total number of replicates. The two prior parameters, d0 and where The moderated t-statistic is defined by:d0 + dg degrees of freedom.The associated probability of the moderated t-statistic for the two sample case under the null hypothesis is calculated by reference to the t-distribution with P(μ|σ2)P(σ2), where the marginal P(σ2) is the scaled inverse gamma and the conditional distribution P(μ|σ2) is normal. That is, the random error associated with each gene's measurement is assumed to be distributed normally with a mean of zero and the variance of the intensity distribution is assumed to be randomly drawn from a prior inverse gamma distribution whose parameters need to be estimated. The posterior variance for each gene is a weighted average of its observed sample variance and the mean of the prior variance distribution. The shape (a) and scale (b) parameters for the prior are estimated by fitting the entire collection of observed variances to an inverse gamma distribution. Rocke [ab * F distribution, where n is the total number of replicates and k is the number of groups. The maximum likelihood approach was found to have lower variability (data not shown) and was the method used in this study.Like the limma model, Wright and Simon's  RVM t-stn. Rocke  has discn. Rocke  have advThe BRB t-statistic for a particular gene takes the form:where the posterior variance isn - 2. The prior variance estimate((ab)-1) is the mean of the fitted inverse gamma distribution with degrees of freedom of 2a. The limma prior parameters d0 and a and (ab)-1. A large number of replicates will give increased weight to the observed variance while a high value for a gives increased weight to the prior variance. A large a means that the inverse gamma is highly peaked making it more likely that the true variance for any particular gene is close to (ab)-1. In the two-sample case, the associated probability of the t-statistic under the null hypothesis is calculated by reference to the t-distribution with n - 2 + 2a degrees of freedom.The Baldi and Long  regulariz number of genes on either side of that gene in the ordered distribution. The total window size (w) for the running average is equal to 2z + 1. Accordingly, for pair,  for the running average is a user supplied theoretical value based on the belief in the quality of the running average variance. The heuristic suggested by Baldi and Long [The degrees of freedom  per gene across treatment classes, vo is the degrees of freedom of the prior, and n - 2 + v0 degrees of freedom.n methods are applied and can be used as a baseline for comparison. The scatter plots for the complete set of Latin Square comparisons and the null hypothesis data with low and medium variances are comparable to these plots (data not shown). With two exceptions (both with MAS 5.0) the four small n statistical tests reduced the variability of the pooled variance estimates relative to the unadjusted estimates of the t-test for all data sets; the range of points on the y-axis in Figures n - 2 = 4) assigned to the gene-specific variance is much greater than the degrees of freedom (2a and d0) assigned to the prior variance estimate . Consequently the adjustment applied by the BRB method is negligible. In contrast, the degrees of freedom values for the data shown in Figure Figures The BRB and limma methods produced similar results for all expression summary methods with the exception of the Affymetrix data normalized with VSN and LMGene which use a generalized log transformation. Although the BRB and limma methods generated similar prior variance parameters, the former generated much higher prior degrees of freedom than the latter. The BRB prior degrees of freedom were on average 17 times higher than the limma degrees of freedom 236 to 14) for the VSN expression summary method and 38 times higher (278 to 7.5) for the LMGene expression summary method. For these data, the degrees of freedom for the sample variance were 4 so the posterior variance of the BRB method, which is a weighted average of the prior and sample variance, deviates only slightly from the prior variance across gene specific intensity Figure . The LMG6 to 14 fFigures The p value distributions for the t-statistic, BRB t-statistic and the limma moderated t-statistic followed the theoretically expected uniform distribution for all simulation models; the CyberT t-statistic deviated slightly from uniformity Figure . For theA standard method of comparing the performance of different statistical methods is to use a partial AUC of an ROC graph calculated using only the lower end of the ordered distributions of p values. The pAUC has a value between 0 (worst performance) and 1 (perfect performance). These metrics were calculated for each of the 1000 iterations of the simulated "spike-in" data sets and for the 14 comparisons of each of the two Latin Square data sets. A particular false positive rate that applies commonly across the different distributions is often chosen. This has the advantage of ensuring that the same number of data points are used in the comparison but has the disadvantage that it ignores the actual p values of the data points being compared. An alternative is to use a cutoff based on a specific p value rather than using only the rankings which is more in line with how the test results would be used in applied contexts. We present results for the latter below and for the former in Additional file The average pAUC scores and rankings for all methods and data sets are shown in Table Figure The three Empirical Bayes variance estimation methods, CyberT, BRB, and limma, performed comparably to FC with almost all low variance data sets Table . They apBased on average ranks, the CyberT t-statistic, BRB t-statistic, and the limma moderated t-statistic performed best across all data sets Table . The topThese results are similar to those generated by the pAUCs with the same number of data points across tests [see Additional file log2 0.4–0.9). Moreover, the LPE's performance was worse than the other methods at all but the two highest FCs, for which the t-test performed worst.Figure With the common variance model, the high fold change transcripts were the most easily identified by all methods and it was within the medium fold change range that there was the best differentiation between the methods; this was also seen with lesser effect with the local variance model (data not shown).p < 0.05 threshold.We examined the data in greater detail by analyzing the true positive rates (TPR) and the false positive rates (FPR) separately, conditioned on spike-in concentration and using a The differential expression tests found most of the Latin Square "spiked-in" genes at the mid to high concentrations with the HGU133A data Figure . There wThe effects of expression intensity on the FPR with the HGU133A Latin Square data varied across differential expression tests Figure . LPE shoFC showed an unstable FPR across intensity with all normalization methods. The FC FPR curve closely follows the smoothing curve of pooled variance versus intensity from Figures The CyberT t-statistic and limma moderated t-statistic also showed a similar, although weaker, pattern as FC in its FPR behaviour across concentration. This is due to the pooling of each gene-specific variance with a single prior error estimate. Genes with variances higher than the prior estimate have their variances reduced and genes with low variances have their variances increased. This causes the denominator of the t-statistic formula to decrease where variance is generally high, making it more likely that the null hypothesis is rejected. The opposite effect will happen when variance is relatively low. The two tests had a stable FPR with the VSN and LMGene normalized HGU133A data as the use of a glog transformation stabilized the variance more effectively across intensity.For the simulated local variance model, FC, the BRB t-statistic and limma moderated t-statistic Figure  showed aConsistent with the consensus among analysts that "borrowing strength" across genes for estimating differential expression random error is desirable ,18,43, tLPE introduced a bias that caused it to generate fewer false positives than expected. This bias appears to be due to a theoretical weakness and an incorrect assumption intrinsic to the test as originally proposed. Murie and Nadon  have shoAn important difference in our study between the CyberT test and the other two Empirical Bayes tests (BRB and limma), was that only the former maintained a stable false positive error rate with data that showed an unstable variance across intensity, such as the experimental and local normal simulated data. This suggests that these latter tests may also generate too many false positives among some biological data sets which, depending on pre-processing, often show high variance at low concentrations.As designed in our simulations and has been argued generally, however, this variance/intensity relationship is an artifact of the inappropriate log transformation for low intensity expression values and the generalized log (glog) transformation is able to stabilize the error variance across the entire concentration range ,48. The Background correction when applied to cDNA data can generate negative values which cannot be logged and requires that the negative data points be either discarded or set to a loggable value. Moreover, we found, as did Qin and Kerr , that usThe performance of the statistical tests was most similar using the HGU133A data set. With the exception of MAS 5.0 with FC, all of the statistical tests found most of the "spiked-in" genes readily, illustrating the limitations of these empirical data sets for comparing performances of statistical test as they would be applied in experiments with biological variation. The HGU133A data set, which used technical replicates, is comparable to our simulated data with low variance, where in general the methods do not show large performance differences. The HGU95 data had higher variability than the HGU133A data and also showed greater performance differences between methods. Most biological experiments will be closer to the medium and the high variance simulated data sets where differences in performance are more apparent. The three Empirical Bayes methods in our study consistently performed well across normalization methods and simulated data designed to mirror the greater variability that is observed in most biological experiments. For this reason, we expect that our results apply beyond the narrow methodological confines of our study. Nonetheless, more definitive tests of the algorithms' merits will be made possible by future spike-in experiments which are anticipated to incorporate biological variability . There iCM conducted the computer simulations, performed the data analysis, participated in research design, and drafted the manuscript. OW and AL participated in discussion of the research, investigation of the statistical methods, and analysis of data. RN conceived of the study, supervised research design, interpretation of results, and drafting of the manuscript.MvA plots of cDNA data.Click here for fileSource code availability.Click here for fileHistograms of logged and unlogged variances for limma and BRB statistical tests.Click here for filepAUC scores using a 5% false positive rate.Click here for fileTrellis plots of performance criteria for simulated data.Click here for fileTotal average rankings of statistical tests for pAUC and TPR.Click here for file


Few overlap between independently developed gene signatures and poor inter-study applicability of gene signatures are two of major concerns raised in the development of microarray-based prognostic gene signatures. One recent study suggested that thousands of samples are needed to generate a robust prognostic gene signature.A data set of 1,372 samples was generated by combining eight breast cancer gene expression data sets produced using the same microarray platform and, using the data set, effects of varying samples sizes on a few performances of a prognostic gene signature were investigated. The overlap between independently developed gene signatures was increased linearly with more samples, attaining an average overlap of 16.56% with 600 samples. The concordance between predicted outcomes by different gene signatures also was increased with more samples up to 94.61% with 300 samples. The accuracy of outcome prediction also increased with more samples. Finally, analysis using only Estrogen Receptor-positive (ER+) patients attained higher prediction accuracy than using both patients, suggesting that sub-type specific analysis can lead to the development of better prognostic gene signaturesIncreasing sample sizes generated a gene signature with better stability, better concordance in outcome prediction, and better prediction accuracy. However, the degree of performance improvement by the increased sample size was different between the degree of overlap and the degree of concordance in outcome prediction, suggesting that the sample size required for a study should be determined according to the specific aims of the study. Recent advances in various high-throughput technologies including genome sequencing, transcriptomics, genome-wide SNP analysis, proteomics, glycomics, and metabolomics have opened up new opportunities for developing prognostic and predictive markers for better treatment of diverse diseases. Indeed, many researchers have reported promising results for improved patient treatment by providing more accurate prognostic and predictive information for decision making -3. AmongWhile many researchers have shown promising results on the possibility of gene expression profiling as a prognostic marker, there are also concerns on the hasty use of the technology in the clinic because many issues remain unresolved and some promising research results were presented in an over-optimistic and flawed manner -10. UnreAmong the above-listed problems, the instability and few overlap of already reported prognostic signatures have received great attention. At first, the few overlap between independently developed gene signatures was attributed to the differences in patients, microarray platforms, or applied statistical analyses. However, Ein-Dor et al. showed that many equally efficient but non-overlapping prognostic gene signatures can be identified from a single data set because gene expression data contains numerous informative genes . MichielCurrently, most gene expression profiling studies have been performed with some tens to hundreds of samples. Meta-analysis, by combining the results of several studies, makes it possible to overcome the limits of many small sample-sized studies. In this work, we pooled eight large-scale gene expression studies to attain a data set with more than 1,300 samples. Specifically, we only used data sets produced using a single microarray platform, Affymetrix U133A, in pooling different data sets to exclude data loss and confounding factors arising from the combination of different microarray platforms. Using more than 1,300 samples, we performed several analyses to understand the various aspects of prognostic gene signatures.To understand the effects of a sample size on the classifier performances, we first constructed a single data set by pooling eight publicly available breast cancer data sets Table , suggestWe first calculated the degree of overlap between different prognostic gene signatures as the sample size was varied. An overlap between different prognostic gene signatures increased according to the increased sample size Figure . For exaWe then tested the effects of a sample size on the error rate of class prediction which is the most important measure of prognostic classifier performance in clinical decision making ,26. For We applied random sampling approach in our evaluation of error rate of class prediction by randomly selecting n training samples (from 100 to 500 by an increment of 100) from the pooled data set, constructing a prognostic classifier from the training samples, and evaluating its performance on the 100 randomly selected testing samples . We usedThe error rate of class prediction was decreased as the number of training samples used for constructing prognostic gene signatures increased Figure  with allRecently, Fan et al. emphasized that concordance in the predicted outcomes between different prognostic gene signatures is the more relevant measure than the mere overlap between them in evaluating the similarity between different gene signatures . We thusRecent studies have shown that breast cancer is a heterogeneous disease consisting of three to six different molecular subtypes ,30. The t-test).With ER-positive samples, the number of training sample was varied from 50 to 200 with an increment of 50. As expected, analysis using only ER-positive samples always produced lower error rates of prediction that the analysis using both ER-positive and ER-negative samples did  may be a better strategy for heterogeneous diseases such as breast cancer.Eight breast cancer gene expression data sets with clinical information on patient survival and CEL files were obtained from the Gene Expression Omnibus (GEO)  or ArrayFrom the data set of 1,372 samples, n samples were randomly selected without replacement and a prognostic gene set was built from the n samples by selecting top 100 genes with the lowest p-value from Cox proportional hazard survival analysis. The sample size n was varied from 100 to 600 by an increment of 100 and the random sampling was repeated 200 times for each sample size n. For each sample size, an overlap between each pair of 200 prognostic gene sets was calculated .First, each patient was divided into good  or poor (relapse or distant metastasis within five years) prognosis groups. Relapse or metastasis free patients followed up less than five years were excluded from the analysis.t-test) were first selected and then used in the subsequent analyses. The training sample size n was varied from 100 to 500 by an increment of 100 and the random sampling was performed 100 times for each sample size. An average of prediction error rates from the 100 random sampling was reported for each sample size. An equal number of samples were selected from the good and poor groups during random sampling of a training-testing pair to avoid a bias in error rate estimation that occurs when the sizes of two classes are severely unbalanced. For the analysis of ER+ specific data set, the training sample size n was varied from 50 to 200 by an increment of 50. The R statistical programming language (version 2.6.2) [Three widely used machine learning algorithms, Diagonal Linear Discriminant Analysis (DLDA), Support Vector Machine (SVM), and Random Forest (RF), were used in the analysis . For eacn 2.6.2)  and Pythn 2.6.2)  were usen 2.6.2) . In DLDAxj is the value on gene j of the test sample, j2 is the variance of the gene [is minimized, where p is the number of genes, the gene . For a bthe gene . For SVMThe effect of a sample size on the concordance in the outcome prediction between different prognostic gene sets was analyzed as follows. Five different sample sizes from 100 to 500 by an increment of 100 were used in the analysis. For each sample size, we first left out 100 testing samples from the total samples for outcome prediction. Then, n training samples were randomly selected from the remaining samples and used to produce prognostic gene signatures. For each sample size, one hundred random samplings were performed to produce 100 independent prognostic gene signatures and each signature was used to predict the outcomes of the left-out test samples at the first step. The concordances in the predicted outcomes for each pair among the 100 prognostic gene signatures were calculated .GEO: Gene Expression Omnibus; GSE: Gene expression Series; DLDA: Diagonal Linear Discriminant Analysis; SVM: Support Vector Machine; RF: Random Forest;SYK designed the study, collected datasets, performed bioinformatics analyses, and wrote the manuscript.Two-dimensional PCA  plot of the pooled samples. Each color represents the eight different data sets and each point represents different samples. Cluster program was used for PCA analysis and the xyplot function of the lattice graphics package of R was used for plotting.Click here for file