Gene expression microarray technology continues to evolve and its use has expanded into all areas of biology. However, the high dimensionality of the data makes analysis a difficult challenge. Evaluating measurements and estimating the significance of the observed differences among samples remain important issues that must be addressed for each technology platform. In this work we use a consecutive sampling method to characterize the dispersion patterns of data generated from Illumina fiberoptic bead-based oligonucleotide arrays.mean, approximating the standard deviation across arrays (Ymean is the mean expression of a given consecutive sample). First we examined three levels of variability: 1) same cell culture, same reverse transcription, duplicate hybridizations; 2) same cell culture, reverse transcription replicates; 3) parallel cultures. Each higher level is expected to introduce a new source of variability. We observed minor differences in the constant term: the mean values are 3.5, 3.1 and 3.5, respectively. However, the mean coefficient b increased from 0.045 to 0.147 and 0.133. We compared the coefficients derived from the consecutive sampling to those obtained from the standard deviation of individual gene expressions and found them in good agreement. In the second experiment samples we detected 11 genes with systematically different expressions between the experiment samples treated with glucose oxidase and controls and corroborated the selection using the Mann-Whitney and other tests. We also compared the consecutive sampling and coincidence method to t-test: the average percentage of consistency was above 80 for the former and below 50 for the latter.To describe general properties of the dispersion we used a linear function SD = a + bYOur results indicate that the consecutive sampling method and standard deviation function provide a convenient description of the overall dispersion of Illumina arrays. We observed that the constant term of the standard deviation function is at average approximately the same for duplicate hybridization as for the assays with additional sources of variability. Furthermore, among the genes affected by glucose oxidase treatment we identified 6 genes in oxidative stress pathways and 5 genes involved in DNA repair. Finally, we noted that the consecutive sampling and coincidence test provide, under given conditions, more consistent results than the t-test.This article was reviewed by Alexander Karpikov (nominated by MarkGerstein), Jordan King and Eugene V. Koonin. Reviewed by Alexander Karpikov (nominated by Mark Gerstein), Eugene V. Koonin, and Jordan King. For the full reviews, please go to the Reviewers' comments section.The usefulness of DNA microarray technology in the exploration of gene expression profiles can hardly be overstated. Along with the dramatic increase in microarray publications  and a broadening in the scope of applications, the methods of analysis of microarray data have grown in variety and sophistication, from simple fold-difference criteria to complex Bayesian procedures and clustering techniques  offered Quantities that are taken as a measure of gene expression are affected by number of processes that contribute to variation, resulting in the random and/or pseudorandom component of the signal. Such variation may be separable into the "technical component," caused by the technical factors, such as variability of experimental protocols, autofluorescence and backscatter, laser-molecule interactions, photomultiplier noise etc., and the "sampling" or "physiological component," which depends mainly on the variability caused by the differences between the samples, e.g. differences in the biological state or purity of the sample composition. Distinguishing if a given gene expression intensity value is greater than the background noise or is different between two samples are fundamental issues in microarray analysis.).For the single-color Affymetrix arrays we have two groups of methods aiming at separation of the true signal from the random components: "low level" and "high level." The former approach deals with the fluorescence signals of each individual probe and includes background correction, adjustment for the nonspecific signal and expression summary that yields an approximation of RNA abundance or "gene expression," the latter takes the gene expression as an elementary variable . Low-levThe high level analysis consists of two basic steps: normalization and statistical evaluation of the observed differences. One approach to normalization relies on the "reference genes" e.g. -20), but, but20])2O2) for 24 hrs. In our analysis we use the consecutive sampling method [k consecutive gene pairs, and calculating standard deviations from the difference of expressions (in this study k = 12). We search for the best candidate genes affected by the treatment among the differentially expressed genes, using the consecutive sampling and coincidence test. The results are compared to the t-test and Wilcoxon (Mann and Whitney) nonparametric test. In addition, we examine consistency of the results obtained by the coincidence test and compare to the t-test on normalized data, log-transformed data and data subjected to the variance stabilization transformation, to the method of analysis by Tusher et al. [Many new microarray-based platforms are available and some, which allow parallel analysis of many samples, may be suitable for high throughput analysis. Here we utilized the Illumina GEX Sentrix™ Array Matrix (SAM) system to evaluate gene expression for 632 genes in 96-well format. Our first aim is to characterize expression data and assess various sources of dispersion. We describe the data obtained from replicate hybridizations, reverse transcription reactions, and biological cultures and evaluate the frequency distributions. Subsequently, we compare the dispersion patterns, and assess the contribution of each additional process to variability of data. The second aim is to study systematic differences in gene expression values in the control cell cultures and cell cultures subjected to a particular treatment. We analyze the data from a cell line subjected to a continuous low-dose oxidative stress exposure . The RNA from each of these 4 samples  was reverse transcribed and hybridized to arrays in duplicate. The data from these hybridizations allows evaluation of variation due to hybridization, reverse transcription, and parallel biological cultures. We used a coincidence test to identify differentially expressed genes in treated samples and compared these results with several other approaches.Illumina GEX Sentrix™ Array Matrix utilizes oligonucleotides attached to 3 micron beads immobilized on fiber optic bundles. Each oligonucleotide probe is represented on ~30 beads per array, allowing averaging of many signals for the same oligonucleotide probe and a consequent reduction in signal variation. Each gene typically has two probes and the intensity signal for a given gene (or probe set), is the average of the available probe signals. The consecutive sampling approach was used to characterize the dispersion patterns of the gene expression data obtained from oligonucleotide probes for 632 genes on three groups of samples: 1) RNA samples from five parallel cultures of lymphoblast cell line GM10469 were reverse transcribed and each has a hybridization duplicate (Samples C1–C5); 2) Pooled reference RNA sample has three reverse transcription/cRNA replicates ; 3) Parallel cultures of lymphoblast cell line GM12831 were grown either untreated or in the presence of 1 mUnit/ml glucose oxidase . Additional files We examined properties of our data by sampling the combined expression values for the first replicate hybridization comprising five parallel cultures of the cell line GM10469  and comparing them to the normal frequency distribution. Figures Conformity to the normal distribution is an important property. For a normally distributed population the quality of a given population sample can be assessed by comparing the sample frequency distribution to the normal distribution function. Analysis of frequency distribution of the observations also provides information about the character of random processes. In our particular case we note that in the low range the distribution function of expression values agrees well with the normal distribution, while in the high range the distribution of relative expressions is close to the normal. On the other hand, distribution of the relative expressions in the low-expression region and of the observed values in the high range deviate substantially from the normal form . For each pair we determined the standard deviation of the consecutive samples and fitted the calculated values to the characteristic functionSD = a + b Ymean,     (1)Ym is the sample mean . The coefficient values calculated for the individual pairs and the mean of five pairs are shown in Table at the logarithmic scale; here k = 24. Since on given Illumina arrays the number of probe sets is relatively small, we evaluated the coefficients of standard deviation function in comparisons of the combined samples C1a to C4a versus C1b to C4b; for k = 24 we obtained a = 3.66 and b = 0.044, about 4.4% and 3.4% above the values a = 3.51 and b = 0.043 obtained for k = 12 and shown in Table k = 12) are just about 1 and 8 percent below the means of the individual pair-wise comparisons 3.54 and 0.047, respectively .To estimate the effect of the samples size we also evaluated the standard deviation function using a ranged from 2.12 to 4.86 (mean a = 3.50) and the proportionality factor b ranged from undetectable to 0.082; PR1 and PR2 samples showed quite low variation for both a and b , while the PR3 sample displayed a relatively high constant term a = 4.86 with a flat slope value b ≈ -0.003 . The constant term a  are similar to the mean values from duplicate comparisons. However, the proportionality coefficients (b) are 3–4 fold greater (0.12 and 0.15 versus 0.04) than for duplicate hybridizations , and PR1b, PR2b and PR3b (Series PRb) we observe that mean values for the constant st hybridization  and all pair-wise combinations of the data from the 2nd hybridization (Series Cb). Mean values of the intercept a were 3.86 and 3.08, close to the means obtained for the between-duplicate hybridization comparisons. However, the means of the coefficients of proportionality b = 0.130 and 0.136 are about three-fold larger than the mean of the between-hybridization coefficients  or treated with glucose oxidase . This low-dose oxidant exposure represents a physiological level of oxidative stress with no apparent induction of toxicity to the cells. After 24 hrs cells were harvested and RNA was extracted. Each of these four samples was hybridized in duplicate to Illumina fiber optic bead arrays .aavg = 2.7, 2.3 and bavg = 0.059 and 0.062 are very close to the means obtained from duplicate hybridization assays, although in this case the means also include comparisons across biological replicate cultures. For the glucose oxidase treated versus untreated comparisons, we observe that the average constant component aavg = 2.4, is quite similar to values observed for other lymphoblast culture replicates but the proportionality coefficient bincreases about 2-fold to 0.120 . Table Kα coefficients that bound the interval containing 90% of values . Boundaries of the probability intervals are useful in searching for genes with significantly different expression.In order to evaluate the quality and consistency of the biological and hybridization replicates, we first performed pair-wise dispersion analysis of the same-type samples (untreated vs. untreated and treated vs. treated). The mean values HMOX1, NQO1, TFRC, P21, MGST1, CCL5) and five genes clearly related to repair of DNA damage .Differences between the treated and non-treated cells are small. Looking at the plot of the average values, just about three genes are substantially above the random dispersion pattern and none below Figure . Howeverk or more genes in all three trials for the probability interval 0.8; the circles and solid line correspond to the results obtained using binomial distribution and triangles represent the results of Monte Carlo simulations with four hundred runs. Probability of detecting at least two upregulated or downregulated genes is about 14%; for three or more genes it is 3%.We can estimate probability of false positives from probability of coincidence in three random trials, assuming zero hypothesis, i.e. assuming that the differences are purely random. Figure We used several independent approaches to assess reproducibility of the results of our data analysis. First we ask the question: Having four replicates, what results we would have obtained, if we only had three pairs of samples? To find the answer we selected the genes above the 0.8 and 0.9 intervals in four sets of three samples and counted the genes common to at least seven out of nine possible combinations. The average percentages of the common genes for any pair of the three-sample tests were 83 and 95, respectively. Surprisingly, the percentage did not decrease, when we reduced the width of the interval Table . In the One of the indicators of reliability of the data is consistency of the individual probes within the probe set. Since we have only two probes per a probe set (with one exception), we can only check for consistency of the probe pair behavior. We calculated ratios of the probe set averages for non-treated and treated samples and compared these to the corresponding ratios obtained for individual probes. The difference ranges from -12% to +8%, except for the gene GI_4755127 probe 1, where we got 33% Table . Also, tWe also compared our candidate genes with the genes selected by the Illumina custom method. To assess performance of the coincidence method in the context of other methods currently employed, we compared reproducibility of the coincidence results to the standard t-test, t-test on the variance-stabilized data , on the The mean number of samples satisfying the standard t-test for p < 0.0028 was 11.8, marginally below the average of 12.3, obtained for the coincidence test. At average, only 6.3 were common to any pair of the three-sample tests, representing just about 54% agreement  and 5 genes clearly related to repair of DNA damage . Additional work is needed to characterize the biological importance of these small changes in gene expression following low-dose oxidative stress.We created a physiological state of oxidative stress by using a low-dose exposure to glucose oxidase. Previous experiments (data not shown) suggested that this dose could induce oxidative stress genes and produce levels of DNA damage that could be repaired with no apparent cellular toxicity. Indeed, among the differentially expressed genes identified between treated and untreated cells Table , we obseIn this analysis we examined the frequency distributions of the data in replicate experiments. We demonstrated plausibility of the two-component representation of the standard deviation and showed equivalence of the consecutive sampling method and gene-by-gene evaluation of the standard deviation function. We used the consecutive sampling and coincident test to identify the best candidates among the differentially expressed genes in the samples under oxidative stress; the results were in agreement with the t-test and Wilcoxon statistic and with the Illumina proprietary method. A practical advantage of the consecutive sampling and coincidence approach is that it provides detail information about characteristics of each individual array. Complete pair-wise comparisons can identify the atypical samples and enable the experimenter to decide about their treatment.The main conclusions can be summarized in the following points:• Random variability exhibits the Gaussian characteristics; at the low end the frequency distribution of expression values is close to the normal distribution, while at the high end the distribution of relative values is close to normal. The frequency distribution is symmetrical with respect to zero.• Standard deviation is well approximated by the linear function of the mean gene expression with a constant term.b, while the change in technical parameters is frequently correlated with the coefficient a.• Our observations indicate that the change in biological state of the matter is usually reflected in the proportionality coefficient • Consecutive sampling provides good estimator of the characteristic standard deviation function.• Consecutive sampling and coincidence test yielded, under given conditions, more consistent results than the t-test applied directly to the normalized data or data submitted to the variance stabilization and starred logarithmic transformation; the performance of the CyberT method and Tusher's method was similar to the coincidence test. The coincidence selection as a nonparametric approach provides more robust selection criterion and can be used for assays with only duplicate arrays.2 atmosphere. Before treatment with glucose oxidase, cells were diluted to a concentration of 2 × 105 cells/ml in fresh RPMI-1640 media (plus 15% FBS) and allowed to grow out for 18 hours to condition the media. After 18 hours, 10 ml of suspended cells were aliquotted into Petri dishes. Glucose oxidase (Molecular Probes) was added to test samples at a final concentration of 1 mUnit/ml, while dilution buffer (1 mM sodium acetate) was added to controls. Test and control samples were incubated at standard conditions for 8 or 24 hours. Parallel cell culture samples  were extracted and reverse transcribed separately. RNA was extracted from test and control samples using RNeasy Midi extraction columns, according to the manufacturer's instructions (Qiagen). The pooled reference used in these experiments was a combination of equal amounts of RNA from six LCLs  and from 3 lymphoid tumor lines , and Jurkat and Raji ).Epstein Barr Virus (EBV)- transformed lymphoblastoid cell lines (LCLs) were grown in RPMI-1640 medium, supplemented with 10% fetal bovine serum  and 2 mM L-glutamine  at 37°C in a humidified 5% COThe Illumina Gene Expression system was used for direct hybridization of labeled cRNAs to gene-specific 50-mer oligonucleotide probes attached to microbeads. For each sample, 200 ng of total RNA was aliquoted into 1 well of a 96-well plate. Labeled cRNA was produced by a reverse transcription followed by in-vitro transcription according to the manufacturer's instructions . Duplicate aliquots of each cRNA sample (1 μg cRNA each) were distributed into parallel microwells in a 384 well hybridization plate with buffer, paired with a Sentrix array matrix (SAM), and incubated at 55°C overnight as per the Illumina standard protocol. The following day the SAM was washed, blocked with casein , and signal was developed with streptavidin-Cy3 using Fluorolink Cy3  according to the manufacturer's instructions. The SAM was then imaged with the Illumina BeadArray Reader GX.k consecutive pairs  and the standard deviations are calculated from the difference of expressions. The estimator of the characteristic standard deviation function SD is then determined by fitting the linear functionThe consecutive sampling method  providesSD = a + bYmeanYm is the sample mean). In order to obtain representative standard deviation at a given expression level, the differences in mean expressions of the genes in a given sample must be small. This poses no problem at the low expression end, but at the high expressions the density of genes within a narrow expression interval is small and a certain number of genes (probe sets) must be excluded.to the experimental points at the logarithmic scale; (Eq. (1), Y2 versus Y1 are defined asThe consecutive sampling program is written in Basic and uses the Excel platform. After sorting and definition of the samples it calculates the standard deviations and determines the characteristic function using the logarithmic transform and nonlinear regression subprogram. Once it determines the standard deviation function, it calculates the boundaries of chosen probability intervals. The upper and lower limits in the dispersion plot andKα is a constant corresponding to the probability interval α.where Several "reliability checks" have been incorporated into the consecutive sampling program. First, assuming independent samples we verify the identitySD(Ydiff) = SD(Y1) + SD(Y2),     (4)SD(Ydiff) and SD(Yi) are the standard deviations calculated from the expression difference of the ordered sample and from the expression values of the array i, respectively . It proThe author(s) declare that they have no competing interests.Experimental samples were prepared and processed by MCM III and DAB in the Environmental Genomics Section, National Institute of Environmental Health Sciences. Data analysis was carried out by JPN at McGill University and Genome Quebec Innovation Centre.Alexander Karpikov, MB&B Department, Yale University, New Haven, CT 06520 USA .Reviewer comments:I read your article and found it quite interesting. I think it has enough material for the publication. My major comment is on the style. I think the article is very difficult to read and its style should be improved prior to the publication.Author response: In response to the reviewer's concern we abbreviated the text and introduced number of revisions. First, in the Background discussion we stated the purpose of our study and the main points of the individual sections more clearly. Second, we partially reorganized the text and introduced two new subheadings. We tried to be clearer and more specific and to eliminate deviations from the main subject. Throughout the text we systematically tried to indicate the goal of a given particular section and clearly describe the approach used to achieve it.Eugene V. Koonin, National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, USAReviewer comments:I think this is a useful, carefully performed study on microarray analysis statistics. Although the technology behind the experiments analyzed in this paper is not the most common one, the tests investigated here may have general applicability.King Jordan, National Center for Biotechnology Information, National Institutes of Health, Bethesda, MD 20894.Reviewer comments:et al. analyzed the statistical properties of gene expression level data generated from the Illumina GEX Sentrix™ microarray platform, which employs a fiberoptic bead-based approach to measuring expression levels. Microarray technology is evolving rapidly and the trend is towards increasingly high density arrays that are capable, in principle, of generating expression profiles for multiple replicates of entire eukaryotic genomes from single chips. The bead based technology employed by Illumina arrays represents an experimental methodology that is qualitatively distinct from that employed by the industry leader Affymetrix. In addition to allowing for increased density and multiple samples per chip, the Illumina bead-based arrays afford several other potential advantages including: i-longer 50 mer probes that presumably increase both selectivity and specificity, ii-low sample and reagent volumes that, along with a high density of features on the array, lead to a relatively low price per sample and a consequently more ambitious experimental scale, and iii-high redundancy, with ~30 beads for each oligo, that can be exploited to increase confidence in specific gene expression levels.In this work, Novak et al. took up in their analysis of the dispersion patterns of expression data generated by Illumina arrays. To evaluate dispersion patterns, the authors implemented the previously developed consecutive sampling method whereby probe sets are ranked according to mean expression and sets with similar means are binned prior to comparison. They compared dispersion patterns from three groups of samples, each of which allows for different sources of variation – hybridization, reverse transcription and biological – to be considered independently. Evaluation of the assumption of normality revealed distinct deviations at the low and high ranges of expression. Based on this pattern, they found that the standard deviation of the variability can be broken down into two components – a constant term and a proportional term. The constant term dominates at low levels of expression, while the proportional term dominates at high levels. Having established the statistical properties of the expression level dispersion, the authors performed a comparison of gene expression from glucose oxidase treated versus un-treated cell lines in order to identify differentially expressed genes that may play a role in the response to oxidative stress. They were able to identify 11 such genes using the consecutive sampling approach, affirming its potential utility.While the move to increasingly high density arrays represents a potential boon to researchers, it also presents fundamental bioinformatics challenges regarding the analysis, visualization and interpretation of expression data. Unfortunately, the bioinformatics technology needed to meaningfully comprehend the results of increasingly high throughput gene expression profiling tends to lag behind the new experimental approaches. It is precisely this challenge that Novak By way of critique, one may quibble with the statistical techniques employed by the authors, and there are clearly other tacks that could be taken to analyze this kind of data. For instance, the authors compared their consecutive sampling method to standard parametric (t-test) and non-parametric (Wilcoxon) methods as well as to Illumina's own proprietary method. They show that their consecutive sampling method performs comparably to these methods, in terms of identifying similar sets of differentially expressed genes, and also shows more consistency. It would have been nice to see a more systematic comparison of different methods for selecting differentially expressed genes since this is the essence of what investigators usually want to glean from microarray studies. For example there are a number of methods cited in this paper including several non-parametric approaches, a variance-stabilizing transformation and Bayesian approaches that could have been compared with the authors' method of choice. However, the analysis that is presented in the paper is detailed and thorough.The impact of the work can also be considered to be somewhat mitigated by the fact that the consecutive sampling method introduced is an extension of the authors own previous work on Affymetrix arrays. The scope of this study was also quite small, expression of only 632 genes was analyzed, and one may wonder whether the dispersion properties for a set of that size would hold up for whole-genome data sets. In addition, while the authors do make some attempt to study an actual biological system – exposure of a cell line to low dose oxidative stress – there is relatively little biological insight that can be gleaned from this work. To be fair however, both substantial utility and novelty can be found in the manuscript, pursuant to the fact that it represents the first study of data variability in the Illumina bead-based microarrays. Thus, the authors have made an important, if tentative, contribution towards bridging the gap between emerging microarray experimental technologies and the bioinformatics tools needed to interpret their output.Author response: First, we would like to thank Dr. Jordan for very thorough and helpful review. In response to his comment we extended comparisons of reproducibility to include the variance stabilization, "starred logarithm" transformation, CyberT method and Tusher's approach. Regarding the comment that the consecutive sampling method was already introduced in the 2002 publication [We would like to mention, that the original paper described only the principle and justification of the approach. On the basis of this principle we developed the method of analysis, incorporating the subroutines calculating the characteristic standard deviation function, boundaries of the probability intervals for selected set of values, tests of normality, calculations of skewness and kurtosis, etc. (c.f. Methods section). Furthermore, in the present study we introduced the application of the consecutive sampling and probability intervals to the differential expression analysis via coincidence test and presented the estimate of the number of coincidences based on probability of coincidences in random trials. We hope that new information is sufficiently noteworthy to make it interesting to readers. Regarding the experimental part of the study, unfortunately, lack of support makes it impossible now to extend the study to include larger arrays or to make a more thorough investigation of the biological properties of the system under consideration.lication : We woulSupplemental Figure S1, comparison of the pooled reference samples C5a and C5b before renormalization. Dispersion pattern and 0.9 probability interval, before normalization.Click here for fileSupplemental Figure S2, comparison of the pooled reference samples C5a and C5b before renormalization. Running mean of 90 genes, before renormalization.Click here for fileSupplemental Figure S3, comparison of the pooled reference samples C5a and C5b after renormalization. Dispersion pattern and 0.9 probability interval, after normalization.Click here for fileSupplemental Figure S4, comparison of the pooled reference samples C5a and C5b after renormalization. Running mean of 90 genes, after renormalization.Click here for fileSupplemental Figure S5, quantile-quantile plot of the frequency distribution. Comparison of the observed expressions with the corresponding inverse normal distribution, combined samples C1a, C2a, C3a, C4a, C5a: range of average expressions from 0.1 to 2.0; figure shows the relative values (expressions divided by the mean of five arrays).Click here for fileSupplemental Figure S6, quantile-quantile plot of the frequency distribution. Comparison of the observed expressions with the corresponding inverse normal distribution, combined samples C1a, C2a, C3a, C4a, C5a: range of average expressions from 117 to 5432.Click here for fileSupplemental Table S1, same-type sample dispersion parameters for the glucose oxidase treatment assay. Coefficients of the standard deviation function a and b and Kα coefficient corresponding to 0.9 probability interval; nt and tr stand for "untreated" and "treated," respectively.Click here for fileSupplemental Table S2, dispersion parameters for case/control comparisons in the glucose oxidase treatment assay. Coefficients of the standard deviation function a and b and Kα coefficient corresponding to 0.9 probability interval; nt and tr stand for "un-treated" and "treated," respectively.Click here for fileSupplemental Table S3, differentially expressed genes selected by the Illumina custom algorithm. Probe sets selected by the Illumina method. The table shows the gene name and function, mean intensity, coefficient of variation and Illumina differential score; value ± 20 corresponds to P = 0.01. Note that Illumina uses different normalization and, consequently, the mean intensities of Tables Click here for file


A comprehensive study of the gene expression profile of 96 mesenchymal tumors identifies molecular fingerprints for most tumors in this group. Bone and soft tissue tumors represent a diverse group of neoplasms thought to derive from cells of the mesenchyme or neural crest. Histological diagnosis is challenging due to the poor or heterogenous differentiation of many tumors, resulting in uncertainty over prognosis and appropriate therapy.We have undertaken a broad and comprehensive study of the gene expression profile of 96 tumors with representatives of all mesenchymal tissues, including several problem diagnostic groups. Using machine learning methods adapted to this problem we identify molecular fingerprints for most tumors, which are pathognomonic (decisive) and biologically revealing.We demonstrate the utility of gene expression profiles and machine learning for a complex clinical problem, and identify putative origins for certain mesenchymal tumors. Tumors of bone and soft tissue are a wide spectrum of benign and malignant neoplasms (sarcoma) derived from mesenchymal precursor cells  ,2. Many The discovery of definitive oncogenic gene fusions for certain mesenchymal tumors has aided pathologists greatly. These include the EWS-ERG or EWS-FLI1 fusion transcripts for Ewing's sarcoma (EWS) -5, or thThere are many gene expression microarray (GEM) studies covering a range of mesenchymal tumors -28. ThesUsing a machine learning model, we assess the overall utility of GEM for the diagnosis of 19 types of mesenchymal tumors. We find both expected and unexpected relationships between certain mesenchymal tumors, and ascertain expression fingerprints for decisive diagnostic or pathognomonic features for most tumor classes. These fingerprints give clues to the etiology of several mesenchymal tumors.Our machine learning model fits the data in two steps. The first step incorporates biological assumptions by merging related tumors into broader groups. Step two splits the broader groups into specific tumors. This latter process is derived by expert decision, rather than automated model selection. We consciously mirror the clinical diagnostic method of progressively resolving general differentiation types into specific tumors in a branch-wise (or tree-like) method. The use of a decision process structured by prior knowledge  gives our model an estimated error of about 0.1. Although further study and validation will be required to bring it to clinical use, we believe this method is a prototype for the extension of GEM and machine learning to other complex diagnostic problems. The GEM data from this study are available from the European Bioinformatics Institute (EBI) public repository ArrayExpress (accession no. E-MEXP-353) .®. A multi-dimensional scaling (MDS) of tumor samples using all genes reveals much structure in the data  and schwannoma (SWN), the alveolar (ARMS) and embryonal rhabdomyosarcoma (ERMS), and the well-differentiated liposarcoma (WLS) and lipoma (LMA). Similarities in the GEM profiles of malignant peripheral nerve sheath tumors (MPNST) and monophasic synovial sarcoma (MSS) were previously reported and this relationship is supported in our data . LikewisThe MDS is based on distances calculated from the whole molecular signature  Figure . Thus, cThe key to the success of supervised learning is to focus on a small set of strongly distinguishing features, thus ignoring confounding factors. Strong feature selection is appropriate in this study as we have sound histological evidence that there are underlying patterns to be uncovered (at least for most tumors). This reduced set of features (or genes) gives a molecular fingerprint of the tumors that is both diagnostic and descriptive. Our feature selection method is suited to this application as it selects ten genes evenly for each class. Using a simple k = nearest neighbors (k-NN) machine learning algorithm and the extended feature selection algorithm we attempted initially to create a model for all tumors simultaneously . The overall cross-validation error was 0.33 compared with a random guessing error of 0.96 +/- S.E. 0.01 . Therefore, even with this simplest approach, the machine learning model is fairly successful.The true error, and thus the generalization to other datasets, is likely to be underestimated (see Discussion). However, an important control shows that our method does not over-fit the data excessively. In simple terms, the method does not erroneously fit this particular dataset using random patterns. After randomly permuting all the labels to create a semi-random dataset we tried to cross-validate these false data . The error was 0.94 +/- S.E. 0.01 (n = 10 permutations) so was therefore not significantly better than random guessing , between MPNST and MSS, and between LMS, PMS and de-differentiated chondrosarcoma (DCS). By considering unsupervised learning methods  plus biological assumptions and trial-and-error, we arrived at an improved two-step method. This was designed, in part, to mirror the thought processes involved during clinical histopathological diagnosis. This two-step model is, thus, expert derived and not an automated procedure.In step one, we grouped tumors into composite classes assumed to reflect similar pathways or levels of differentiation; MPNST/MSS, PMS/LMS/DCS, ARMS/ERMS, NFB/SWN and LMA/WLS/MLS. In step two, we decomposed the composite classes into their sub-types. This decision process is illustrated in color code in Figure Using the same feature selection and learning algorithm as the initial method, the new model is remarkably effective in the first step with ~0.09 errors (9/96 errors). There were important limitations, however, as we could not decompose the adipocytic tumors fully in step two, yet recognized MLS as distinct. Moreover, after compositing the spindle-like tumors PMS/LMS/DCS in step one, we could not decompose this artificial grouping. Accepting these limitations, the sum error over both steps was 0.14 . Note that the samples wrongly classified in step two were classified correctly in step one thus having no carry-over error in this particular dataset. The types of error in step one are summarized in Table Feature selection of genes was carried out for each fold of our model, and while 237 distinct genes were encountered during cross-validation only 61 were selected in every fold . This robust 61-gene signature is illustrated in the gene-clustered heatmap of Figure From the molecular fingerprint used in step two we selected genes used in the majority of cross-validation folds for display in Figure We demonstrate here the feasibility of using GEM and machine learning to aid the diagnosis of mesenchymal tumors. In contrast to previous studies, our work encompasses a wide range of mesenchymal neoplasms and reflects the open-ended nature of histopathological diagnosis. It is clear from unsupervised methods such as MDS that there is information within the expression profile that can be further refined Figure . This coCross-validation of the first step of our model gives 0.09 errors. This is a useful guide but its accuracy should not be overstated. A reliable estimate of error usually requires hundreds of samples per class, and a leave-one-out estimation is likely to underestimate error . Moreoveperilipin (PLIN), lipoprotein lipase (LPL), and glycerol-3-phosphate dehydrogenase 1 (GPD1)  and receptor-associated protein of the synapse (RAPSN) associated with the musculoskeletal synaptic junction  Figure . Similartropomyosin-2 (TPM2), skeletal muscle actin (ACTA1) and myosin-light polypeptide-4 (MYL4), indicative of more maturity than the ERMS , a key molecule in the development of the neural crest. EDN3 promotes self-renewal of multi-potent neural crest precursors or de-differentiation of matured cell types including Schwann cells [Within these highly focused signatures there are clues to the origins of some of the poorly differentiated tumors. One striking similarity of gene expression is between the MPNST and the MSS . The MSS have a simple karyotype consisting of an aberrant SYT-SSX fusion. The MPNST have a complex karyotype and are believed to arise from Schwann cell precursors  derivednn cells -35.WNT5A recently linked to metastases of OS [Despite their similarity, the MPNST and MSS have distinct molecular signatures Figure  and 4d aes of OS  and melaes of OS . Yet SWNmyxoid liposarcoma-associated transcript-4 (MLAT4), which as its name suggests distinguishes MLS from WLS/LMA , and neural restricted genes . The profile also confirms previous reports of the expression of the immunotherapeutic targets PRAME and CTAG-1  [Some genes could make useful monoclonal antibodies for immunohistochemistry. Genes such as the A Figure  and 4h, Y-ESO-1) . These abrachyury (T) which is highly expressed in developing notochord strongly supports this theory [There is another clue to cellular origin within the CMA fingerprint. The overall expression pattern of CMA is closely related to the chondroblastic tumors CHS and chondroblastoma (CHB) Figure . Likewiss theory ,42.Not all tumors had distinctive markers. The LMS, PMS and DCS were not distinguishable from each other. Cross-validation of our model in step one was fairly successful as our algorithm collectively recognized these tumors through a lack of specific markers, perhaps analogous to the way in which histopathologists recognize them Figure . At leasThere are a number of extensions to the current study that may bring GEM technology closer to clinical use. Firstly, a larger study focusing on the poorly modeled cases (PMS/LMS/DCS) may help to elucidate categories supplementing current histological guidelines. Yet the pleomorphism and complex karyotypes of some mesenchymal tumors may confer a continuum of molecular pathology irreducible to simple categories. Such an analysis is unlikely to find adherents if it does not correspond with an improved interpretation of the histopathology or with appreciable clinical differences such as prognosis.Secondly, by sampling all groups more thoroughly we could manage uncertainty. Our model gives a simple prediction of tumor type appropriate for the broad base and low sample numbers in our dataset. A more nuanced approach would be to calculate a probability of tumor type. This could be achieved most simply using Bayesian classifiers that implicitly calculate class membership probability. Thus in a clinical setting a sarcoma could be identified to a histopathologist as a central or classical example of its type, or be highlighted as an uncertain outlier for further immunohistochemical examination.Thirdly, there is a large quantity of relevant molecular information that may transcend the diagnostic categories investigated here, yet may have prognostic significance. There is great interest, for instance, in both drug-resistance genes  and signTumor biopsies were obtained from 96 participants presenting at the London Bone and Soft Tissue Tumour Service , Great Ormond Street Hospital, London, or the Nuffield Orthopaedic Center, Headington, Oxford, in the UK. The diagnosis was determined by pathological examination using criteria established by the World Health Organization (WHO) tumor classification of soft tissue and bone tumors . Where nFrozen sections from each tumor sample (needle core or resection) were examined microscopically prior to RNA extraction to confirm that the sample was representative and contained more than 80% tumor cells. Total RNA was extracted from adjacent tissue sections with >80% tumor cell content, using TRIzol™ reagent  followed by purification with RNeasy columns  according to the manufacturer's instructions. RNA quality and quantity was assessed using RNA 6000 Nano chips on Agilent 2100 Bioanalyser according to manufacturer's instructions . Our RNA quality-control threshold for the rRNA peak ratio was 28s/18s ≤ 2.®, Santa Clara, CA, USA) according to manufacturer's recommendations. In cases where smaller amounts of total RNA were obtained , 150 ng of total RNA was subjected to two rounds of amplification to obtain a biotinylated cRNA yield of 20 μg according to Affymetrix recommendations . Parallel hybridizations of probe synthesized from 10 μg and 150 ng (via single and double amplification respectively) for the same tumor sample were found to have similar quality-control and expression profiles  was prepared from 10 μg of total RNA as previously described ,47. The Data analysis was carried out using the R statistical environment and programming language . We exteThe Cluster and Treeview software packages were used to produce the average linkage hierarchical clustering shown in Figure We chose simple k-NN pattern classification to model the data. This is a non-parametric algorithm, here based upon a table of inter-sample Euclidean distances . The ideAs most expression data are uninformative, or even confounding of pathological categories, feature selection is incorporated into each fold of cross-validation. For multi-class models we used a well-suited feature selection method to capture information from each of the 19 tumor classes (not to split classes in step two). The algorithm selects 10 genes for each class that are different in expression from all others. We tested 5, 10, 20 and 30 gene sets finding 10 to be sufficient (better than 5 or 30 and as good as 20). We used the 'limma' method for each comparison . Limma uThe following additional data are included with the online version of this article: an average linkage hierarchical clustering of the tumor samples using the same distance matrix as in Figure Average hierarchical clustering of tumor samples. The same set of inter-sample distances were used as in Figure Click here for fileSupplementary information on the genes shown in Figures Click here for fileClinical and pathological details of all samples used in this study.Click here for file


Recently several statistical methods have been proposed to identify genes with differential expression between two conditions. However, very few studies consider the problem of sample imbalance and there is no study to investigate the impact of sample imbalance on identifying differential expression genes. In addition, it is not clear which method is more suitable for the unbalanced data.Based on random sampling, two evaluation models are proposed to investigate the impact of sample imbalance on identifying differential expression genes. Using the proposed evaluation models, the performances of six famous methods are compared on the unbalanced data. The experimental results indicate that the sample imbalance has a great influence on selecting differential expression genes. Furthermore, different methods have very different performances on the unbalanced data. Among the six methods, the welch t-test appears to perform best when the size of samples in the large variance group is larger than that in the small one, while the Regularized t-test and SAM outperform others on the unbalanced data in other cases.Two proposed evaluation models are effective and sample imbalance should be taken into account in microarray experiment design and gene expression data analysis. The results and two proposed evaluation models can provide some help in selecting suitable method to process the unbalanced data. Microarrays enable us to monitor expressions of thousands of genes simultaneously and generate enormous amount of data. Using such techniques, it is possible to explore the secret of biology at the molecular level and understand the fundamental biological processes ranging from gene function to development and to cancer -3. In miOne simple approach used in literature to detect differential expression genes is "fold change" method, in which a gene is declared to be differentially expressed if its average expression level varies by more than a given constant between two conditions. However, "fold change" method has been demonstrated to be unreliable and inefficient, because statistical variability is not considered . Then, mHowever, very few studies consider the problem of sample imbalance in detecting differential expression genes and there are no studies as well as quantitive method to investigate the effect of sample imbalance on differential expression genes selection. Sample imbalance means that the size of samples in one group is very different to that in another group. In fact, the problem of sample imbalance usually appears in gene expression data, especially in the data about tumor samples. For example, the data in -23 are aIn this paper, we investigate the new problem about the impact of sample imbalance on identifying differential expression genes. Two evaluation models based on random sampling are proposed and six famous methods are compared on both the real data and the simulated data. Under each evaluation model, the random sampling is utilized to estimate the expected performances of methods on the unbalanced data which satisfy one specific sample ratio between two groups. Then the variations of performances are used to illustrate the effect of sample imbalance on differential expression genes selection and method selection.n1 and n2 (which are the numbers of samples in class one C1 and class two C2) in each experiment under two evaluation models, first, we randomly create a set of x independent artificial data or simulated data and test all six methods on these x data to get the results. For a specific method, each one in the x random data will only get one result for each measure, for example Overlap Rate, Precision Rate or Recall Rate. Then, these x values are treated as a random sample of size x from the fixed parameters n1 and n2. Last, the expected performance of each method and its 0.95 confidence interval are calculated from this kind of random samples.In this section, six methods including two-sample t-test with equal variances  , two-samTwo real datasets are the liver dataset  and the U, which is very similar to real data. In each simulated data, there are 1000 genes  and two classes C1 and C2 . For any non-differential expression gene j (i.e. 51 ≤ j ≤ 1000), its expression value aij on each sample i is randomly generated from N and U, where μ ~N. For gene j ≤ 50, the value of gene j on any sample in class C1 is generated from N and U, while that in class C2 is generated from N and U, where μ1, μ2~N. For the problem of multiple testing involved in identifying differential expression genes, bonfenorri correction of the significant level α can be used to reduce the error of type I. But a very small α will be disadvantaged to compare the performances of methods. In this paper, a relatively small significant level α will be used to control the type error I. On the real data, the value of α is set to 0.0001. On the simulated data, the significant level α is set to 0.01.The simulated data is created according to the protocol in , where tC1 of the artificial data, which are created from the liver data or the prostate data, is always fixed at 60. The results under the evaluation model 1 are presented in figure n1+ n2 in the artificial data created from the liver data is fixed at 120 and that from the prostate data is fixed at 60. The results of the evaluation model 2 on real data are presented in figure SR ≠ 1 (denoted as i(i ≠ 1))is significantly different with that at SR = 1 (denoted as 1), we make a two sample t-test, where the observations are these 100 Overlap Rates calculated from 100 random artificial data with SR = 1 and those calculated from 100 random artificial data with SR ≠ 1. So our null hypothesis states that i(i ≠ 1) = 1, while the alternative hypothesis states that i(i ≠ 1) ≠ 1. The p-values associated with the t-statistic in the evaluation model 1 and 2 are summarized in table SR = 1 and SR ≠ 1 has a very high statistical confidence. Additionally, there is also a difference among the Overlap Rates of different methods. It can be seen from figure In the experiments of the evaluation model 1, the number of samples in Class C1 and sample class C2 (i.e. σ1 = σ2), but have unequal variances (i.e. σ1 ≠ σ2) in the second category of simulated data. The result on a simulated data is the average result on 1000 random data generated with a specific sample ratio.In this section, under two proposed evaluation models, we generate two kinds of simulated data to compare the performances of different methods on the unbalanced data. In the first category, the differential expression genes have equal variances in sample class σ1 = σ2 = 0.5), where the number of samples in class C1 is fixed at 60 in the evaluation model 1 and the number of overall samples is fixed at 60 in the evaluation model 2. The corresponding p-values of the t-statistic on the simulated data with equal variances under the evaluation model 1 and 2 are summarized in table Figure The results on the simulated data with equal variances indicate the performances of all methods are greatly affected by the sample imbalance. Each of two metrics for the performance of method  is steadily declined as the sample ratio increases. This result is consistant with that of previous experiments on the real data.Furthermore, the downward trend of Recall Rate in response to the increasing amounts of sample ratio is steeper than that of Precision Rate. In other words, the Recall Rate  of the method for selecting differential expression genes is more sensitive than the Precision Rate  to sample imbalance, although they are all affected by sample imbalance.It is certain that the sample imbalance appears to have different effects between different methods. The difference between different methods become great when the degree of sample imbalance increases. In detail, the Precision Rates of the Wilcoxon rank-sum test and the Regularized t-test are higher than those of others, that is, the Wilcoxon rank-sum test and the Regularized t-test have lowest false positive rate (Type I error). Whereas, the Recall Rate of SAM is superior to that of other methods, i.e. the method of SAM has the lowest false negative rate (Type II error). And Welch t-test shows the worst performance.σ1 ≤ σ2 and n1 ≥ n2, for example, σ1 = 0.5, σ2 = 1 and SR = 1, 2, 3. The second case is that σ1 ≤ σ2 and n1 ≤ n2, for example, σ1 = 0.5, σ2 = 1 and SR = 1, , . The results of the evaluation model 1 on the two case of simulated data with unequal variances σ1 = 0.5, σ2 = 1 are showed in figure n1 + n2 = 60 on two case of simulated data with unequal variances σ1 = 0.5, σ2 = 1.In this section, under two evaluation models, the simulated data are generated in two case: the first case satisfies σ1 = 0.5, σ2 = 1 and n1 ≥ n2, Regularized t-test shows the highest Precision Rate and Recall Rate while Welch t-test performs the worst capability. In contrast, Regularized t-test has the medium performance and Welch t-test shows the best performance when σ1 = 0.5, σ2 = 1 and n1 ≤ n2. This surprising observation can be easily explained by figure As observed in figure σ1 ≤ σ2, Regularized t-test is always superior to Welch t-test on the unbalanced data which satisfies n1 ≥ n2. When σ1 ≤ σ2and n1 ≤ n2, the results become relatively complex. In the plot b of figure σ1≤ σ2 and n1 ≤ n2, Welch t-test have obvious dominance. In addition, the more difference between variances σ1 and σ2 in unbalanced data, the higher different effects on different methods.In order to investigate the combined influence of sample ratio and varied variance on method performance, Regularized t-test and Welch t-test are selected as examples to demonstrate the dependency of the difference between methods with respect to different variances and sample ratios. Figure From this study, it is clear that there is a great effect on the performances of methods for selecting differential expression genes by the sample imbalance. Because of many objective factors, the gene expression data always involve the problem of small sample. As mentioned earlier in the previous section, coupled with the problem of small sample, the presence of the unbalanced data makes detecting differential expression genes more difficult. The sample imbalance is an important and inevitable problem in gene expression data analysis. Hence, one need to consider the problem of sample imbalance in the design of microarray experiments and the following data analysis.Careful experimental design is necessary to improve the result of data analysis and reduce the cost of experiment simultaneously. By the comparison between plot a and b in figure   of expression values for gene j may depend on the mean expression value μi. Hence, it will be very helpful to the result of analysis if a more suitable method has been selected to process the unbalanced data. For example, given an unbalanced data with unequal variances, one can improve the result of analysis if a feasible method from the six methods is selected. However, it is very likely that all six methods are not feasible for the unbalanced data and there is a requirement to find new methods more suitable to process the unbalanced data.Furthermore, our results also indicate that on the unbalanced data, there have a great difference between the performances of different methods, especially on the data with heterogeneity. Some previous studies ,25 have n1 + n2.It should be noted that this paper does not consider the problem of determining sample size for detecting differentially expressed genes in microarray data. An interesting topic is how to assign samples between two groups in order to maximize a method performance under the constraint of the given number of overall samples The results of this paper are based on six popular and typical methods for identifying differential expression genes including parametric method and nonparametric method. The similar effect of the sample imbalance on both kinds of methods leads us to believe that the findings in this paper should have, at least qualitatively, a comprehensive meaning. Also, two proposed evaluation models can be used to compare and evaluate other methods.The experiments in this paper demonstrate that sample imbalance has a great effect on identifying differential expression genes and two proposed models are effective to quantify the effect of sample imbalance. Moreover, different methods have different performances on the unbalanced data and we can not find one method to be suitable for all unbalanced data in the experiments. Among the six methods, the welch t-test appears to perform best when the size of samples in the large variance group is larger than that in the small one, While the Regularized t-test and SAM outperform others on the unbalanced data in other cases. In conclusion, two proposed evaluation models and the results provide some help in selecting suitable method to process the unbalanced data.In future work, we will apply the evaluation models to evaluate more methods, for example the methods based False Discovery Rate. Furthermore, we attempt to investigate the problem of determining the sample size to maximize the performance of a given differential expression genes selection method.n samples in the gene expression data and these n samples consist of two nonoverlapping categories named class one (C1) and class two (C2). In each sample, the expression values of p genes have been detected. Then the gene expression data may be represented by a n × p matrixFirst, some notations used in this paper are introduced here. We assume there are An × p = (aij)n × p,aij is the expression value of gene j in sample i. The rows of A correspond to samples, and the i-th row vector of A is called the expression profile of the i-th sample. We assume that nk, k(j) and Sk(j) are number of samples, sample mean and sample variance of gene j in the class Ck, respectively, where k = 1, 2.where the element Given a gene expression data, let nk denotes the number of samples in class Ck, k = 1, 2. Then, Sample Ratio, denoted by SR, is defined to be n1/n2, i.e. SR = n1/n2.We use the Sample Ratio (SR) to measure the degree of sample imbalance between two groups. As revealed by definition 1, the further the value of SR departs from 1, the more serious the degree of sample imbalance is.A key question also involved in this paper is how to evaluate the performance of a method for identifying differential expression genes, that is, how to evaluate the solution resulted from the method. For thousands of genes in a real gene expression data, it is generally unclear that which ones are differentially expressed genes. This situation has resulted in an obstacle to assess a method directly and strictly. In contrast, the true solution is known for the simulated data. So, in order to assess the performance of method directly, the simulated data are introduced. Furthermore, several measures are introduced to measure the quality of the method solution. Different measures are applicable in different situations, depending on whether a true solution is known or not.First, we present a metric to assess the method performance for selecting differential expression genes on the real gene expression data.original data (OD) and the artificial data (AD), which satisfies the given parameters n1 and n2, is generated by randomly sampling samples from the original data. Thus the Overlap Rate denoted by OR is calculated according to the following definition.Given real data, the whole real data is treated as the Let DEGOD and DEGADDifferentially Expressed Genes identified by some method on the original data (OD) and the artificial data (AD), respectively, then the Overlap Rate (OR) is defined as OR  be the sets of = |DEGOD ∩ DEGAD|/|DEGOD|.p genes, any solution can be represented by a binary 1 × p vector T, where T(i) = 1 if and only if the i-th gene is differentially expressed gene (or positive gene). Suppose that T and S be the true solution and the suggested solution of a method, respectively. And let nxy denote the number of pair , for which T(i) = x and S(i) = y, where x, y = 0 or 1. Thus, n11 is the number of true positive genes, n01 is the number of false positive genes, n00 is the number of true negative genes, and n10 is the number of false negative genes. Consequently, two different metricss, Recall Rate and Precision Rate, are introduced to measure the performance of method.To assess the method performance on the simulated data, we can compare the true solution with the suggested solution by the following method. Given simulated data with Suppose that S and T be the suggested solution of a differential expression gene selection method and the true solution, respectively. Then Recall Rate (RR) is defined as RR = 11/n(n10 + n11).Suppose that S and T be the suggested solution of a differential expression gene selection method and the true solution, then Precision Rate (PR) is defined as PR = n11/(n01+n11).From the definitions 3 and 4, We can see that the Recall Rate focuses on the false negative while the Precision Rate focuses on the false positive. However, the false negative and the false positive are two different keystones in the context of selecting differential expression genes, and the false negative is inconsistent with the false positive. So in a particular problem specification, one can choose either Recall Rate or Precision Rate as the main focus.n1 samples in class C1 and n2 samples in class C2, we can only get one specific value of each of the metrics OR, RR and PR of the method on the given data. So after one specific method and the set of gene expression data with size m are given, there exists the set of ORs (RRs or PRs) with size m resulted from the method. A perfect way to evaluate the performance of one method is to run the method on the whole set of gene expression data which satisfy given parameters n1 and n2 and to calculate the average value of each metric. But the cardinality of the set of data with parameters n1 and n2 may be very large or infinite. For example, if a real microarray data has 50 and 30 samples in class C1 and C2respectively, then the number of different artificial data with parameters n1 = 40 and n2 = 20 is  > 3 × 1017. In order to reduce the computation cost and avoid the problem of infinity, one feasible way is to estimate the expected value of each metric and its approximate confidence interval (or Error Limit) by sampling a sample from the specific gene expression data randomly.For one specific method of differential expression genes selection and one given data with Suppose that population X has mean μ, and finite variance σ2, and X1, X2, ..., Xn are an independent random sample of size n from the population X, then the sample mean is an unbiased estimate of μ and the sample variance S2is an unbiased estimate of σ2. Moreover, the variance of , denoted by D, satisfies D = σ2/n, whereSuppose t to estimate the population mean μ and calculate its approximate confidence interval. In sampling survey, the exact distribution of the estimate (i.e. ), is unknown. However, according to the central limit theorem, we can expect the sampling distribution of  to be approximately normal distribution with mean E = μ and variance D. That isAccording to lemma 1, we can use the sample mean α)100% approximate confidence interval for the estimate of μ is . In practice, the standard deviation of sampled population σ is typically unknown. Replacing σ by S leads to the corresponding estimate  and  is referred to as the standard error (SE) of . Therefore, a feasible confidence interval of μ at a significant level α is  and the approximate Error Limit (EL) is αz. For sample of size n ≥ 30, regardless the shape of most population, sampling theory guarantees good results [As a result, a . That is, the confidence interval is . In practice, the fpc can be ignored whenever the sampling fraction does not exceed 5% [When population is finite, the change is the introduction of the factor 1 - xceed 5% .In order to investigate the effect of sample imbalance on differential expression genes selection, one simply needs to consider the change of the performance of a method in response to different sample ratios (SRs), because the sample ratio is a measure of the degree of sample imbalance between two groups. Therefore, two evaluation models are proposed as follows.Let the number of samples of certain class always equal to constant C, for instance n1= C, and the artificial data (or the simulated data) is randomly created with different Sample Ratios. Then compare the method results on the data with various Sample Ratios.Let the number of all samples in the artificial data (or the simulated data) always equal to constant C, i.e. n1+n2 ≡ C, and the artificial data (or the simulated data) is randomly created with different Sample Ratios. Then the method is evaluated based on these random data with particular parameter SR.a is calculated from the assumed distribution. In the nonparametric method, for a given significance level α, following the spirit of SAM, we find the 100(1 - α)% quantile of the null distribution, i.e. noted as , using the following formulaFor the parametric method, the cutoff point of a significance level  is the value of the statistic for the i-th gene in the b-th permutation. Then the quantile value  is used as the cutoff point for that statistic to select differential expression genes.where B is the number of permutations and K.Y. conceived the study, performed the implementations and drafted the manuscript. H.G. critically read and revised the final manuscript. J.L. supervised the whole work and finalized the manuscript. All authors read and approved the final manuscript.


Retinal photoreceptors are highly specialised cells, which detect light and are central to mammalian vision. Many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells. Development and maintenance of photoreceptors requires appropriate regulation of the many genes specifically or highly expressed in these cells. Over the last decades, different experimental approaches have been developed to identify photoreceptor enriched genes. Recent progress in RNA analysis technology has generated large amounts of gene expression data relevant to retinal development. This paper assesses a machine learning methodology for supporting the identification of photoreceptor enriched genes based on expression data.in silico models for detecting key complex features and relationships encoded in the data, which may be useful to distinguish genes in terms of their functional roles. In order to understand temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis was firstly performed. By clustering SAGE libraries, a hierarchical tree reflecting relationships between developmental stages was obtained. By clustering SAGE tags, a more comprehensive expression profile for photoreceptor cells was revealed. To demonstrate the usefulness of machine learning-based models in predicting functional associations from the SAGE data, three supervised classification models were compared. The results indicated that a relatively simple instance-based model (KStar model) performed significantly better than relatively more complex algorithms, e.g. neural networks. To deal with the problem of functional class imbalance occurring in the dataset, two data re-sampling techniques were studied. A random over-sampling method supported the implementation of the most powerful prediction models. The KStar model was also able to achieve higher predictive sensitivities and specificities using random over-sampling techniques.Based on the analysis of publicly-available gene expression data from the developing mouse retina generated by serial analysis of gene expression (SAGE), this paper presents a predictive methodology comprising several in silico methodology for supporting large-scale analysis of photoreceptor gene expression by SAGE. They may be applied as complementary methodologies to support functional predictions before implementing more comprehensive, experimental prediction and validation methods. They may also be combined with other large-scale, data-driven methods to facilitate the inference of transcriptional regulatory networks in the developing retina. Furthermore, the methodology assessed may be applied to other data domains.The approaches assessed in this paper represent an efficient and relatively inexpensive  Retinal photoreceptor cells, the specialized cells involved in light detection and phototransduction, are essential for mammalian vision. Many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells. Photoreceptor degeneration, for example, constitutes an important cause of visual impairment affecting all age groups and ethnic backgrounds . Developet al. [Nrl , are either associated with or are candidates for retinal diseases involving rod or cone photoreceptor dysfunction. Katsanis et al. [expressed sequence tags (ESTs) likely to be specifically or preferentially expressed in the retina. They also identified positional candidate genes for 42 of 51 uncloned retinopathies. The quality of the results was assessed by reverse transcriptase-polymerase chain reaction (RT-PCR). Recently, Blackshaw et al. [serial analysis of gene expression (SAGE), followed by in situ hybridization (ISH) validation. Libraries were obtained from microdissected mouse photoreceptors from the retinal outer nuclear layers (ONL), retina from various mouse developmental stages and retina from the paired-homeodomain transcription factor Crx knockout mouse (Crx-/-) and its wild type counterpart (Crx+/+) at postnatal day (P)10, and from NIH3T3 mouse fibroblasts.Over the last decades, different experimental approaches have been developed to identify retinal disease genes. Using microarray data analysis, for example, Yoshida et al.  revealeds et al.  positionet al. [prior, complete knowledge of the sequence of the genes [The SAGE-based expression analysis performed by Blackshaw et al.  has advaet al. . The bashe genes .et al. [et al. experimentally validated the candidate tags through exhaustive ISH analysis. In this paper, we explore the feasibility of using computational approaches to support large-scale analysis of photoreceptor gene expression data. The main purpose of this study is to implement several in silico models to detect complex features and relationships encoded in the SAGE data, which may be used to predict functional associations. By way of illustration, this paper focuses on the classification of two functional classes of genes, which were studied and experimentally validated by Blackshaw et al. [In the study by Blackshaw et al. , the canw et al. : PR-enrik-means algorithm (PoissonC) specifically designed for SAGE data [Figure of Merit (FOM)-based approach [et al. [In order to study temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis: clustering of tags and clustering of libraries were performed. While SAGE libraries were clustered using traditional hierarchical clustering method, SAGE tags were analysed by a newly developed Poisson model-based AGE data . To addrapproach  to estim [et al. : PR-enriFigure k-means algorithm with Euclidean distance on normalised SAGE data from 1 to 20 clusters. The value of adjusted FOM steeply decreases until the number of clusters is equal to 10. Afterwards the rate of decline with respect to the number of clusters is reduced. The results for a 10-cluster analysis using the PoissonC algorithm are shown graphically in Figure et al. [Figure e et al. . The deset al. [rhodopsin gene and 2 tags linked to rod photoreceptor were grouped together in Cluster 6. Four tags mapped to guanine nucleotide binding proteins (G-protein) [peripherin-2 and rod outer segment membrane protein 1 (ROM1) were also found. It has been shown that these genes are involved in maintaining the integrity of photoreceptor outer segment and are therefore critical for rod photoreceptor viability and regulation of disk morphogenesis [In general, the temporal expression patterns observed in this SAGE data reflect previously characterized photoreceptor gene expression patterns in the developing retina. For example, genes associated with tags in Clusters 2 and 6, whose expression consistently increases throughout postnatal development and reaches their highest value in the adult retina, are highly likely to be expressed in developing photoreceptors. Sixty out of 63 tags previously known to be PR-enriched and 134 out of 197 tags identified to be PR-enriched by Blackshaw protein) , which aogenesis .et al. [Interestingly, 8 out of the 10 known non-PR-enriched tags were found in Cluster 9, having peak expression values occurring within 3T3 fibroblast cells. This might confirm that those genes that have higher expression levels in non-retina tissues are unlikely to become PR-enriched genes. Cluster 7 exhibits similar expression profiles of genes strongly expressed in adult hypothalamus. Although the characteristics of all 27 tags in this cluster were not investigated by Blackshaw et al. [Clusters 3 and 5, whose peak expression values occurred during embryonic development, also offered relevant insights. Genes that fell within these two clusters generally have higher expression levels in the mouse retina before P6.5 with expression gradually decreasing throughout later postnatal development. Examples includes macrophage migration inhibitory factor gene in Cluster 5, which plays an important role in T-cell activation and may contribute to regulation of retinal inflammation and its local immunity . Its expet al. , ten outNeuroD is high at early embryonic stages, peaking at P4.5 and decreases significantly in the adult. This might reflect a dual function of NeuroD in cell specification and in regulation of rod photoreceptor survival [Mertk, a c-mer proto-oncogene tyrosine kinase, exhibits an embryonic expression profile with peak at E14.5 and is selectively expressed in mature photoreceptor cells. Previous studies have shown that mutations in Mertk are responsible for retinal dystrophy [Unlike the other gene clusters, the expression levels of genes within Clusters 1, 4, 8 and 10 did not follow a consistent trend throughout retinal development. Forty-six tags associated with various PR-enriched genes were found in these four different clusters, reflecting the heterogeneity of photoreceptor gene expression patterns on the basis of the onset and peak time of expression . Unlike survival . In Clusin silico methods to detect key relationships encoded in the SAGE data. Such complexities are further stressed when the original data are projected on 3-dimensional space using well-known mapping methods, i.e. Principal Component Analysis and Sammon's mapping, which clearly indicate that the two classes are not linearly separable from each other [see The goal was to determine whether a tag represents a PR-enriched or a non-PR-enriched gene given a set of SAGE libraries associated with each tag. Among the 1118 tags representing at least 0.01% of the total expression in the ONL library, 261 tags have been identified as PR-enriched genes, which exhibit diverse and complex expression patterns. This highlights the difficulties in using ther see .et al. [Based on a comprehensive analysis of the expression of tags previously known to be PR-enriched, Blackshaw et al.  introduc1. Tissue specific (criterion 1): the number of tags in either the hypothalamus or 3T3 libraries is less than 2.2. Developmentally regulated (criterion 2): the sum of tags in the P10.5 crx+/+ (wild type), adult and ONL libraries divided by the sum of tags in the E12.5, E14.5, and E16.5 libraries is greater than 10-fold.3. Crx dependent (criterion 3): tags are present at a level greater than 1.6-fold higher in the P10.5 crx+/+ library compared to the library of crx-/- mice.4. ONL enriched (criterion 4): tags are present at an equal or greater number in the ONL library compared to the whole adult retina library.Apriori algorithm was applied to extract association rules from the SAGE data. This algorithm, which was proposed by Agrawal and Skrikant [Apriori algorithm is able to generate association rules that have support and confidence levels greater than user-specified values. A list of all association rules induced from the 324 tags, together with their support and confidence levels, are given in Table We encoded the SAGE dataset using these criteria to study significant associations between the two functional classes and these criteria. The Skrikant , is a weet al. [The results obtained showed that a high rate of true positives was observed for genes corresponding to tags meeting more than two of the four criteria. However, when applying these criteria to perform supervised classification, poor prediction results were obtained. For example, only 57 out of 261 PR-enriched tags met all four criteria. From the 196 tags identified by Blackshaw Ac), along with the precision (Pr), Se and Sp for each class, were calculated. Precision is defined as the proportion of predictions that are correct. The mathematical definitions of these metrics are given in the additional file [see AUC) was calculated. The AUC has been suggested as a reliable and robust measure for classification performance [AUC is associated with a classifier that is both effective and robust, i.e. it presents a better average classification performance across different prediction (decision) thresholds.Tables file see . The corformance . A higheAc, Se and Sp. The advantages of random over-sampling techniques can be further demonstrated by the AUC values shown in Figure KStar [Tables re KStar  algorithTo investigate the effect of class distribution on the classifier, we varied the class distribution using the data over-sampling technique. Table Ac, Se, and Sp, strongly indicating that the relationship between the data and the labels may be reliably learned by the proposed classifiers. For example, when implementing the permutation test for the KStar model on a balanced dataset and 10-fold cross validation, the random classifiers never performed better than the  prediction model built. The obtained average values of Ac, Se and Sp (for class PR-enriched) were significantly lower than the results shown in Table p < 0.01). Similar results were obtained when we performed a permutation test for other classifiers on the dataset with different class distributions. The complete results of our permutation test can be found in the additional file [see To further assess the statistical significance of our computational approaches and their predictive performance, a 100-run permutation test was implemented. For each permutated dataset, the results were significantly worse than the one generated using the original data in terms of file see .in silico approaches, including unsupervised and supervised models, to supporting large-scale analysis of photoreceptor gene expression by SAGE.As the most accessible part of the central nervous system (CNS) and as a highly ordered laminar structure, the retina offers unique opportunities to study both the development and physiology of the CNS. This paper described several By clustering SAGE libraries, a hierarchical tree reflecting the relationship between the libraries was obtained. Libraries from adjacent developmental periods were generally grouped together as expected. However, significant discontinuities were identified at the time of birth and between P6.5 and P10.5, highlighting important developmental periods. By clustering SAGE tags, a more comprehensive expression profile for photoreceptor cells was revealed. It confirmed that most of the PR-enriched genes may be successfully clustered. These genes have lower expression levels before birth with expression dramatically increasing throughout postnatal development. Nevertheless, a closer examination of the clustering results revealed that photoreceptor expression patterns are highly heterogeneous and not separable by linear methods. The diversity of photoreceptor expression profiles reflects the variability in onset of expression which can occur early in development or when photoreceptors undergo terminal differentiation.k-means algorithm, which exhibits several limitations that hinder its performance. In this study, we proposed a framework to estimate the number of clusters for PoissonC based on the calculation of an adjusted FOM value. Nevertheless, there is a need to further expand clustering-based studies for SAGE data. For instance, an important question is how to incorporate Poisson-based distance into other clustering methods such as hierarchical clustering and self-organizing maps.With regard to the clustering of SAGE tags, different algorithms with different distance metrics have been previously proposed. For example, it has been suggested that by modelling SAGE data with Poisson statistics better results can be achieved. However, it is relatively computationally expensive and Poisson-based distance has only been assessed as part of the KStar model) performed significantly better than relatively more complex algorithms, e.g. neural networks. This may be partly explained by the fact that neural network-based prediction models typically require larger amounts of high quality training data. Given the limited amount of SAGE data available, neural network-based algorithms may not be recommended. Nevertheless, the application of different types of machine learning approaches, including state-of-the-art classifiers such as Support Vector Machine, deserves further investigations.To demonstrate the usefulness of machine learning-based models in predicting functional associations from the SAGE data, a comprehensive comparative assessment of three supervised classification models was presented. The results indicated that a relatively simple instance-based model  specifically designed for SAGE data [A central problem in the design of clustering models is the selection of a distance function to measure differences between expression profiles. Traditional approaches include the Euclidean distance and Pearson correlation coefficient. It has been shown that with regard to classification of SAGE libraries, Pearson correlation-based clustering analysis may detect significantly similar groups of genes . On the et al.  adopted  [et al.  used seli et al.  argued tAGE data . The reaAGE data  for a dek-means models, one limitation of the PoissonC algorithm is that it requires users to specify a priori the number of clusters to be detected in the data. To deal with this problem, we used the FOM to estimate the optimal number of clusters encoded in the data [Like other the data . The lowthe data .Due to the computational cost and characteristics of the PoissonC algorithm, the calculation of the adjusted FOM based on PoissonC may be a time-consuming process. It has been suggested that the patterns encoded in the SAGE data revealed by the clusters under different algorithms roughly agree with each other. Moreover, the performance (i.e. classification effectiveness) of clustering algorithms can be improved when the algorithms are applied to normalized data . Thus, wk-means algorithm, in which the Euclidean distance was used to measure similarity on normalised SAGE data. For a given tag, the abundance in each SAGE library was rescaled to make the sum of tag counts across all 14 libraries equal to one. In our application the adjusted FOM was calculated for a range of numbers of clusters, from 1 to 20.1. Calculate the adjust FOM value using traditional 2. Draw the FOM value against the number of clusters graph.3. Estimate the optimal number of clusters based on the graph4. Use the number obtained in last step as an input to perform PoissonC-based clustering analysis.We adopted an open-source implementation of FOM provided by the Institute for Genomic Research (TIGR) KStar, C4.5 decision tree, and multilayer perceptron (MLP) neural network model. KStar is an instance-based classifier [Three different classification models were implemented using the freely available Weka package : KStar, assifier . Based oassifier . We testassifier .These three models were assessed as classifiers for PR-enriched genes on the basis of SAGE data. To estimate the true classification error rate, a 10-fold cross validation was applied. To further assess the statistical validity of our computational approaches, 100-run permutation tests were performed in this study, i.e.: the labels for each tag were randomly shuffled, classifiers were then implemented, their prediction quality was assessed and this process was repeated for a number of permuted datasets. By counting the times the permuted datasets produced better results than the classifier built on the original dataset, the statistical significance was then established.minority class, i.e. non-PR-enriched class, from the majority class under consideration. To deal with this problem, two data re-sampling techniques were studied: random under-sampling and random over-sampling. The former method randomly eliminates majority class tags to achieve a balanced dataset. The latter randomly replicates minority class samples until a balanced class distribution is reached [A key challenge was to address the class imbalance exhibited by the dataset available – as demonstrated by the highly skewed distribution of tags Table . The pre reached .prior probabilities. It has been suggested that the true negative rate , true positive rate , and Receiver Operating Characteristic (ROC) graphs are three appropriate metrics to assess the quality of a classifier in the presence of class imbalances [Se and Sp for non-PR-enriched were omitted in Tables Another crucial problem is how to evaluate the performance of classifiers in imbalanced dataset. Traditional techniques include the calculation of classification accuracy based on a confusion matrix. However, it is known that when classes are imbalanced, these two metrics may offer misleading conclusions because they are strongly biased to favour the majority class . For exabalances . It is eHW and HZ co-designed the study, implemented the models, and drafted the manuscript. DS provided advice on biological aspects and co-wrote the paper. FA co-designed the study and co-wrote the manuscript. All authors read and approved the final manuscript.the supplementary materialClick here for fileThe complete results for the permutation testClick here for file


Application of phenetic methods to gene expression analysis proved to be a successful approach. Visualizing the results in a 3-dimentional space may further enhance these techniques.We designed and built TreeBuilder3D, an interactive viewer for visualizing the hierarchical relationships between expression profiles such as SAGE libraries or microarrays. The program allows loading expression data as plain text files and visualizing the relative differences of the analyzed datasets in 3-dimensional space using various distance metrics.TreeBuilder3D provides a simple interface and has a small size. Written in Java, TreeBuilder3D is a platform-independent, open source application, which may be useful in analysis of large-scale gene expression data. Development of high throughput transcription profiling techniques such as microarrays and SAGE  has madeTraditionally, hierarchical data are presented as a rooted or unrooted tree with bifurcating branches connecting different data points (nodes). Generally, only the distances along the edges of the tree are shown proportionally. In a non-reticulated tree, the distance between nodes that are not directly connected is not easily visualized or understood. Building a hierarchical tree benefits by operating in 3-dimensional space, as such 3-dimensional structure provides more details about the relationships between the analyzed datasets.The algorithm of building phenetic networks or dendrograms, which we have used for our viewer, is generally applied in systematic biology. For systematic analysis, tree-building techniques are designed to order data in terms of phylogenetic relationships, from which evolutionary history can be inferred. In contrast, system analysis of SAGE and microarray data assumes that extraction of order in the relationships between different gene expression profiles and presents clues about biological processes and functions on molecular, cellular and tissue levels. Systematic approaches may allow improvement of diagnosis, disease classification, and prognosis, as well as assist in drug design, among other scientific and medical goals.We have developed a graphical viewer for analysis of the hierarchical organization of cellular expression profiles, analyzed by SAGE or microarray experiments. The output diagram is a system of nodes, each of which represents a SAGE or microarray experiment, and the connections between nodes in the diagram. For each node, the relative position on the graph depends on the differences between its expression profile and the expression profiles of the other analyzed nodes. In order to generate the diagram, the user can choose a distance metric to generate the comparison. Our viewer currently offers five distance metrics, all having a different approach to the estimation of the distance between expression profiles. These metrics include Euclidean, Manhattan, Pearson and Pearson squared distances ,5. In adThe application loads the data from plain tab-delimited text files. Currently, we provide support for two formats – generic, tab-delimited with optional tags for description and total number of tags in a SAGE library and GEO (files of this type are downloadable from Gene Expression Omnibus website ). FormatOur viewer automatically positions the analyzed nodes in 3D-space according to the calculated distances between them. In comparison to traditional phenetic techniques, mostly generating 2D-diagrams, this approach has several advantages. The constraints imposed by 2D-space are removed, so it is possible to get a more adequate snapshot of the system relationships between datasets. The nodes are connected, and by default it is possible to change the visibility for the connecting edges, adjusting the threshold for maximal length to be shown. In addition, in the default view mode the color of the edges reflects the tension, indicating if an edge is compressed or stretched. As the initially displayed length of an edge may be different from the calculated value for this connection, the user can estimate how accurate the lengths of the edges are is by looking at their color. TreeBuilder3D shows stretched edges in blue and compressed edges in red. As an alternative way to filter out insignificant (very distant) connections, our viewer can use a simple aggregation algorithm, analogous to Neighbor-joining method using arithmetic averages, described in . After tFigure TreeBuilder3D provides a simple, comprehensible view of relationships in a hierarchy when applied to analysis of large-scale gene expression data generated by methods such as SAGE. An advantage of representing relationships between gene expression profiles in a three dimensional tree diagram is that it allows the visualization of subtleties in internodal relationships that could be overlooked due to the constraints imposed by a two-dimensional tree diagram. TreeBuilder3D allows the display of analyzed data in a more natural and intuitive way.), as well as Java3D package version 1.3.1, which is available for download at . Due to the limitations of the Java3D API, currently there is no support for Mac OS. Further information may be requested via e-mail to the corresponding author pruzanov@bcgsc.ca.The program is written in Java and runs on Windows or Linux platforms. In addition to compiled Java executable and documentation, we provide the source code for TreeBuilder3D under the terms of the General Public License (GPL) on our web site [PR and SJMJ developed the main ideas and methodology; PR did the coding; SJMJ provided feedback and coordination of the project. SJMJ and PR both read and approved the final manuscript.contains the compiled Java executable, documentation and the source code for TreeBuilder3D. Example data sets are also included for the testing purposes.Click here for file