Microarray-based tumor classification is characterized by a very large number of features (genes) and small number of samples. In such cases, statistical techniques cannot determine which genes are correlated to each tumor type. A popular solution is the use of a subset of pre-specified genes. However, molecular variations are generally correlated to a large number of genes. A gene that is not correlated to some disease may, by combination with other genes, express itself.In this paper, we propose a new classiification strategy that can reduce the effect of over-fitting without the need to pre-select a small subset of genes. Our solution works by taking advantage of the information embedded in the testing samples. We note that a well-defined classification algorithm works best when the data is properly labeled. Hence, our classification algorithm will discriminate all samples best when the testing sample is assumed to belong to the correct class. We compare our solution with several well-known alternatives for tumor classification on a variety of publicly available data-sets. Our approach consistently leads to better classification results.Studies indicate that thousands of samples may be required to extract useful statistical information from microarray data. Herein, it is shown that this problem can be circumvented by using the information embedded in the testing samples. The emergence of modern experimental technologies, such as DNA microarray, facilitates research in cancer classification. DNA microarrays offer scientist the ability to monitor the expression patterns of thousands of genes simultaneously, allowing them to study how these function and how they act under different conditions. This can lead to a more complete understanding of molecular variations, in addition to morphologic variations among tumors. A large number of studies have used microarrays to analyze the gene expression for breast cancer, leukemia, colon tissue, and others, demonstrating the potential power of microarray in tumor classification -7.over-fitting /[σ1(g) + σ2(g)], where {μ1(g), σ1(g)} and {μ2(g), σ2(g)} denote the mean and standard deviation of the log of the expression level of gene g for each of the two classes. Large values of |P| indicate high correlations between gene expressions and class distinction. The vote of each gene is v(g) = P(g)(x(g) - b(g)), where b(g) = (μ1(g) + μ2(g))/2, with a positive value indicating vote for class 1 and negative value indicating vote for class 2. The final decision is thus given byApplicable for binary classification, where each gene casts a weighted vote for one of the classes, and the final decision is made based on the summation of all individual votes. Specifically, for each gene SB and the covariance matrix ΣX are both singular, we need to calculate the projection directions v from [LDA is used to find a linear combination of genes where the between-class variance is maximized and the within-class variance  is minimized. Since in microarray data the between-class scatter matrix s v from B = X = SBW = WΛB and ΣXU = UΛX, W = U = pB and pX denote the rank of the two matrices. After the projection matrix V is obtained, a nearest class mean classifier is used for classification in the sub-space formed by the first q columns of V. Since the sample-to-dimension ratio is small, it is customary to use only the first row of V, that is, the most discriminant direction of LDA, v, even if C > 2. Then, classification reduces towhere ΛIf the data-set is linearly separable, a linear SVM is a maximum margin classifier. This means, SVM will find that hyperplane that divides the data of one class into one region , and the other class in another region (the negative side). While doing this, SVM will guarantee that the distance between the samples of class 1 and class 2 that are closets to this hyperplane is maximized. Considering the following linear classifierit can be shown that maximizing the margin is equivalent to solving the following optimization problemThe weight vector pk(x) for each class is a prior known – Gaussian distributions being the most common assumption. The parameters of the distribution are estimated using the training samples. ML assigns the sample x to the class which gives the largest likelihood to x, i.e., C(x) = arg maxk pk(x). When the samples are Gaussian distributed, x ~ μk, Σk), this results inThis is a parametric method. It assumes the distribution form We first derive the details of the proposed approach and present each of the algorithm items. Extensive experimental validation is then presented in the testing section.use the information of the testing sample to improve the classification process, e.g., to find a better discriminant space in LDA.The key idea used in this paper, is to take advantage of the discriminant information embedded in the testing sample. Rather than looking for its closest match amongst all the training samples, we can cannot be used because its class is unknown. Here, we show that it is actually possible to take advantage of the information embedded in the testing sample, changing the role of the testing sample from passive to active. We will accomplish this by assigning the test sample to each of the possible classes and then determining which of these "assignments" is the correct one. As mentioned above, this is possible because a discriminant approach will generally work best when the test sample is assumed to belong to the correct class. Earlier, we used intuitive argumentation to show this. We will now prove this result formally within the LDA framework, which will be used through out this paper as an illustrative solution .The reason why classifiers built on training data generally work poorly on testing data is because the distribution of the training samples does not generally represent that of the testing . In suchOur solution originates from the discriminant power (DP) of linear discriminant analysis, given by SB and sample covariance matrix ΣX, respectively, pB and pX denote the ranks of these two matrices. The between-class scatter matrix is a metric measuring the separability of samples corresponding to different classes, while the covariance matrix defines the sparseness of the data. The goal is to maximize the first metric while minimizing the second. Here, the sample covariance matrix is defined aswhere xi ∈ ℝp is the ith sample vectors, p the number of features (genes), and μ the sample mean over all xi. The class covariance matrix is similarly given by xi,j the jth sample in class i, μi the sample class mean, C the number of classes, and ni the number of samples in the ith class, where DP measures how well the classes are separated in the subspace spanned by LDA's solution, v. Therefore, the larger the value of DP is, the better.DP score, let us look back at feature extraction. A classical approach used by researchers to perform dimensionality reduction is the well-known Principal Components Analysis (PCA) algorithm. PCA is concerned with the selection of that linear combination of features  which carries most of the data (co)variance. This is readily accomplished by finding the eigenvectors of the covariance matrix ΣX, i.e., ΣXV = ΛV, where the columns in V are the eigenvectors and Λ is the diagonal matrix of corresponding eigenvalues. ΣX is, in effect, the metric we have decided to maximize.To better understand the role of the A and B. The first metric calculates within-class variances, the second is concerned with between-class variations. Thus, in LDA, the goal is to minimize the metric given by A while maximizing that given by B, e.g., A = ΣX and B = SB. This is then equivalent to the following eigenvalue problem A-1BV = ΛV.Linear Discriminant Analysis (LDA) is in fact an extension of PCA. In LDA, one has two metrics, A, does not agree with that of the second metric B. In this case, we say that the two metrics are in conflict. Under such circumstances, knowing which of the two metrics is right turns into a guessing game. Taking an average would even be worse, because generally one of the two metrics is correct [Unfortunately, this method does not work well when the two metrics disagree , that is correct .DP. That we will show how to efficiently do next.Hence, our next goal is to determine which of the classes, where our test sample can be assigned, will provide the smallest conflict, that is, the largest discriminant score x to class i and then use LDA to obtain the discriminant subspaces vi, i = 1,...,C. The discriminant power indices DPi can be computed using (1).In our framework, we first assign the test feature vector x is assumed to belong to class i. When x is assumed to belong to an incorrect class, LDA will find it difficult to discern that from the other samples, and DPi will be small. When the test sample is however assigned to the correct class, LDA will find it easier to discriminate between classes and the discriminant value (1) will increase. This means that our approach should reduce to assigning the test sample x to that class providing the maximum discriminant power when x is assigned to it. Unfortunately, this is not possible, because when the number of genes (features) p is much larger than the number of samples n, the value of DPi is always 1 regardless of the value of the parameter i. This is formally stated in the following.This will indicate how well the data is separated when the test feature vector Theorem 1. Let the number of features  be = 1.Proof. Let Σi denote the sample covariance matrix of class i, i = 1,...,C, p the dimensionality of the sample feature vectors, ni the number of samples in class i, and n the total number of samples, p >n ≥ ni, we have p >ni. Hence, dim(null(Σi)) = p - ni + 1. LDA's solution is the intersection of the null spaces of Σi. This intersection is non-empty if C = 2. In this figure, we synthetically generated n samples in ℝ10 for a total of two classes. For visualization purpose, we only show the data using the three dimensions with largest variance in Fig. x, which is randomly drawn from class 2 (squares). Next, we project the data onto the direction found by Fisher's LDA when different amounts of training samples n are used, Fig. n = 15 = p + 5. That is, we keep the dimensionality of the space p (which is 10) smaller than the number of sample n, i.e., p <n. Since there are more samples than features (genes), the discriminant power approach of (1) is applicable. In Fig. DP1, assuming the test sample belongs to class 1. In Fig. x actually belonging to class 2, and calculate DP2. It is clear from the figure that the second option provides a much larger discriminant power and, therefore, the algorithm classifies the test sample in the correct class. However, when the number of samples n is smaller than or equal to p + 1, the value of DPi is always one. This is illustrated in Fig. n = 11 = p + 1. All samples from class 1 are projected onto a single point and the samples from class 2 to another, DP1(x) = DP2(x) = 1.This result is illustrated in Fig. n ≤ p + 1. It is illustrative of the reasons why. First, note that all DPi are equal to one when n ≤ p + 1, because, in such cases, the projection of each individual class covariance matrix onto the one-dimensional solution found by LDA is always zero. In fact, this is possible because there is always a one-dimensional subspace where all the samples of the same class collapsed onto a single point. This subspaces is the intersection of the null spaces of every class covariance matrix, and was illustrated in Figs. The result above had a purpose beyond that of showing that the discriminant power defined in 1) is inappropriate when  is inappthe between-class variance itself provides the appropriate measure of separability. We thus denote the distance between classes as that defined by the projected between-class scatter, vTSBv.Nonetheless, since the projected class covariance matrices are zero, k, vk. That is, vk is obtained using all the training samples and including the test feature vector x as an additional "training" sample of class k. This solution allows us to compute the discriminant power asThe framework outlined above, can be readily implemented as follows. First, compute the one-dimensional solution provided by Fisher's LDA when the test sample is assumed to belong to class x.where The larger the discriminant power , the better the algorithm can classify the test sample. Hence, the test sample should belong to that class which maximizes (2), that is,C(x) specifies the class label of the test feature vector x. We denote this discriminant, power-based method as DP algorithm. The schematics of the algorithm are illustrated in Fig. where The lower-performance problem in between-class classification  is hereiWe have used a variety of databases to validate the algorithm and our claims. This will also serve to prove the superior performance of the proposed approach when compared to the state of the art.BRCA1 mutation status (positive or negative), and the second grouping labels the samples according to BRCA2 mutation status (positive or negative). There is a total of 3226 genes in this data-set. present This data set is developed to investigate whether gene expression difference is helpful to distinguish prostate cancers with common clinical and pathological features . A totalThis data-set is to analyze whether the gene expression data alone can accurately predict patient outcome after prostatectomy . Samplesmax/min and max - min, respectively; where max and min refer to the maximum and minimum value of expression level for each particular gene across all samples. In particular, we used max/min < 3 and max - min < 100.Diffuse large B-cell lymphoma (DLBCL) is the most common lymphoid malignancy in adults, curable in less than 50% of patients. This data-set is constructed from a related germinal center B-cell, follicular lymphoma (FL) [ define aM; 3) for a given array, add or subtract an appropriate constant to each expression value to re-center the median of the array to be that given by M; 4) log-transform the entire data-set to make the data more Gaussian distributed.This data-set contains a total of 2000 gene expressions of 40 tumor and 22 normal colon tissue samples . Followin samples for testing and use the remaining n - 1 for training. We then see whether each of the algorithms can correctly classify the sample left out. This is repeated n times – one for each of the samples that can be left out. Table kNN, WV, LDA, SVM, and ML.To test the approach just presented, we use the leave-one-out cross-validation test. This means that, at each iteration, we keep one of the The second experiment is designed to further understand the dynamics of the proposed algorithm. As stated in this paper, most algorithms will fail when the training samples are not representative of the testing ones. We have approached this problem by taking advantage of the information embedded in the test vector. Under this model, our approach should be superior to the classification mean of other algorithms when the test sample is more correlated to the training samples of the incorrect class. To demonstrate this, we have designed a second experiment, where we divided each of the samples left out for testing into two groups. The first group includes those test vectors that are more correlated to the sample mean of the correct class than to the sample mean of the incorrect class. The second set corresponds to those test samples that are more correlated to the incorrect class. Table The experimental results reported thus far used real datasets to compare the classification capabilities of the proposed algorithm with those reported in the literature. The differences shown in Tables p, with p = 50,...,400. The covariance matrices were set as diagonal matrices with their elements set at random. The means of these distributions were also set at random. We then randomly generated 50 samples from each of the two distribution and used the algorithms defined earlier and the one proposed in the present work to do classification. The number of samples was kept at 50 regardless of value of p. This tested how well each algorithm could deal with a decrease on the sample-to-dimension ratio. In Fig. DP algorithm outperform the others – especially so as p increases.In our first example, we randomly generated two Gaussian distribution, each representing one of the two classes. The two Gaussians were defined in ℝDP algorithm outperforms the others. Most importantly though, it is clear from Fig. DP algorithm over the state of the art.The above result is however quite simplistic, because the samples in each distribution were distributed according to a single Gaussian distribution. A more realistic scenario in bioinformatics is that where the samples in each class are generated by a mixture of Gaussians. To test this other case, we randomly generated 50 samples corresponding to two different classes. Each class was now defined by a mixture of four Gaussians, with their means and covariances randomly selected as above. The average results over a total of 100 runs are shon in Fig. Analyzing data from small sample size sets is a recurring problem in biology. This is generally due to the limited amount of data available or to the difficulty or costs associated to obtaining additional data. Studies indicate that hundreds or thousands of samples would be required to extract useful statistical information from our data sets ,16. HencThis paper has shown derivations of an approach to deal with the small sample size problem within a linear discriminant analysis setting. Our framework can be readily extended to work within other classification approaches. It could also be combined with shrinkage , a mechaThe approach proposed here can also be applied to many other problems in biology and medicine. For example, in the classification of nuclear magnetic resonance spectra, which is typically used to carry out metabolomics experiments. In this example, classification approaches like the ones describe din this paper are generally used . AnotherThe two authors contributed equally in the development of the approach and the writing of the paper. MZ generated the results.


Variation amongst individuals is a significant confounding factor in miRNA (or other) expression studies. The true character of biologically or clinically meaningful differential expression can be obscured by inter-patient variation. In this study we aim to identify miRNAs with consistent differential expression in multiple tumor types using a novel data analysis approach.Using microarrays we profiled the expression of more than 700 miRNAs in 28 matched tumor/normal samples from 8 different tumor types . This set is unique in putting emphasis on minimizing tissue type and patient related variability using normal and tumor samples from the same patient. We develop scores for comparing miRNA expression in the above matched sample data based on a rigorous characterization of the distribution of order statistics over a discrete state set, including exact p-values. Specifically, we compute a Rank Consistency Score (RCoS) for every miRNA measured in our data. Our methods are also applicable in various other contexts. We compare our methods, as applied to matched samples, to paired t-test and to the Wilcoxon Signed Rank test.http://bioinfo.cs.technion.ac.il/people/zohar/RCoS/We identify consistent (across the cancer types measured) differentially expressed miRNAs. 41 miRNAs are under-expressed in cancer compared to normal, at FDR  of 0.05 and 17 are over-expressed at the same FDR level. Differentially expressed miRNAs include known oncomiRs (e.g miR-96) as well as miRNAs that were not previously universally associated with cancer. Specific examples include miR-133b and miR-486-5p, which are consistently down regulated and mir-629* which is consistently up regulated in cancer, in the context of our cohort. Data is available in GEO. Software is available at:  Gene expression profiling is commonly applied to identify differences between classes of cell types, as manifested in differentially expressed genes It would be valuable to identify statistically significant differences in various samples that can be reliably attributed to the specific biological state, such as cancer or disease, instead of individual biological variations, as stated above. In many situations, there is opportunity for serial collection of tissue or blood from a patient, experimental animal or cell line Currently, hundreds of microRNAs (miRNAs) have been identified in humans. These are short  noncoding regulatory RNA molecules and their sequences are published in the Sanger miRBase Our goal in this study is to identify miRNAs that are consistently differentially expressed in multiple cancer types. To avoid tissue type variability and to measure cancer related differential miRNA activity in each type separately; we use a matched sample dataset consisting of 32 microarray measurements representing 28 matched tumor and normal samples. We use microarrays containing probes for 799 miRNAs to profile miRNA expression in these samples.Our motivation in seeking miRNAs with consistent differential expression in multiple cancer types stems from the existing knowledge that many biological processes are common to different types of cancers. In particular, several genes are known to be universally differentially expressed across multiple cancer types. The most obvious example is p53. p53 was first discovered in 1979 and since then numerous studies indicated its involvement in multiple cancer types. The importance of regulated activity of intact p53 in preventing tumor formation is indicated by the presence of mutations in the p53 pathway in nearly all cancers Previous studies on the role of miRNAs in cancer include Lu et al. To support our research goals we have developed statistical methods that address characterizing distributions of random variables that arise from comparing matched samples. In our case we compute differential expression in every tumor type and then statistically assess its prevalence in our dataset. Our methods are based on discrete order statistics – the k-dimensional vector that is obtained by drawing k independent numbers uniformly in 1…N and then sorting them resulting vector. While the distribution of order statistics over continuous state spaces is well characterized, this is not the case for discrete sample spaces as repeats may then occur with positive probability. Computing distributions related to discrete order statistics was addressed in To summarize, the contribution of this paper consists of:Rigorous characterization of the distribution of order statistics over a discrete state set as well as of related random variables. This distribution is highly applicable in analyzing matched data in a non parametric setup. We also compare our methods to paired t-test and to the Wilcoxon Signed Rank test.A dataset with matched tumor normal samples representing a repertoire of 8 tumor types. This set is unique in its emphasis on minimizing the tissue type and patient related variability through the use of normal and tumor samples from the same patient.By applying the novel statistics described above to our matched sample dataset we validate known oncomiRs and describe several novel cancer-universal differentially expressed miRNAs. It should be noted that this stated universality is only substantiated, within the context of this study, for the 8 types represented here.expression raw data matrix. When describing the methods we use the word “gene” but “miRNA” can be used interchangeably. This matrix is typically the output of several pre-processing steps such as normalization and filtering performed on the raw measurement data.The starting point for analyzing the results of a gene or miRNA expression profiling study is the t-testGaussian Error score Kolmogorov-Smirnov score Wilcoxon Rank-Sum test Information score Threshold-Number-of-Misclassifications (TNoM in short) Typically, data analysis of expression profiles starts with the identification and the statistical assessment of genes that are differentially expressed when comparing various classes represented in the cohort. Many current gene scoring methods consider all expression values of a given gene. These are partitioned into two or more populations according to the studied classification. Differences between the resulting subsets of numbers are assessed using various statistical methods. Gene scoring methods fall into two broad categories – parametric methods, and non-parametric (distribution free) methods. Parametric methods assume a certain distribution for the expression values of every gene within each given class  and then score genes according to how separate the class specific distributions are. Examples of such methods are the standard This work is concerned with additional and potentially more relevant information that can be inferred when the expression data is coming from several patients and when all classes were measured for each patient. For example, samples before and after treatment for the same patient. Another example is tumor and normal samples from the same tissue of each patient, a design utilized in this work. The scores we develop take into account the degree to which a gene separates two classes in a large majority of patients. The interpretation is that a gene is relevant to the underlying biology if it is highly differentially expressed for most of the patients. In addition, we attach a significance level  to each relevance score level. The p-value is the probability to get this level or better, at random, as described below in further detail. Rigorous statistical analysis is instrumental in confidently identifying genes that sharply separate sample classes and thus in pointing at promising research directions. Partial variants of the methods described in this paper were employed in In this section we describe the statistical methods in high generality. Specific embodiments, in the context of consistent tumor versus normal miRNA differential expression, are described in the The Rank Consistency Score (RCoS) is a differential expression score for 2 classes that takes patient matching into account.k = 1…r and for every gene g. The differential expression can be calculated using different methods and the method chosen depends on the design of the study and on the number of samples for each patient. Differential expression scores include: fold change, Gaussian error score, t-test, TNoM and other methods. Often the number of samples for each patient and class is 1, so simple fold change is used.We call the two classes Class A and Class B. We first compute the differential expression between the two classes for every patient (or subject or subset) g we compute its rank for the k-th patient: k(g)R – this is a number between 1 and N, where N is the total number of genes. The gene topg for patient k is the one most over-expressed in Class A relative to Class B. It is ranked first and we set N.Next, we rank all the genes per patient according to their differential expression between class A and class B. For every gene g, we define the rank consistency score S as the normalized maximal rank of this gene among all patients, i.e.Our goal is to find genes with consistently high ranks  across all patients. For every gene g for all patients is no worse than S·N.In other words, the rank of gene S for m out of r patients. In this case for each gene we order its ranks and then the score S corresponds to the normalized m-th smallest rank:For greater flexibility in defining consistency we allow outliers, and compute the rank consistency scores S, the m/r RCoS. We will sometimes refer to the r/r RCoS simply as RCoS. m out of r rank consistency scores. Pseudo-code for calculating the m/r RCoS is available at We call the m out of r rank consistency score, The above analysis will identify genes that are over-expressed in Class A compared to Class B. To find genes over-expressed in the Class B we can perform the same analysis, reversing the ranked list.p-value corresponding to this level s, under the prevailing null model. The p-values for RCoS and for its variants considered in this paper are computed under the assumption of independence of patients and of uniform distribution of ranks among genes within each patient. These two assumptions define the underlying null-model.To evaluate the statistical significance of any observed value of RCoS we estimate the probability of obtaining the value s, or better, in random data drawn according to a null model. This probability is the p-value at s, we compute the probability of a gene ranking in the top s fraction of the list, in at least m patients. Let V be an r-dimensional random vector with entries drawn independently and uniformly in 1,…,N. We are interested in the probability of the m-th smallest entry in V being smaller than sN. It is given by:To compute the m/r RCoS When working with larger sample sets the question of how many outliers to allow (which m to choose) arises. A possible principled solution is to calculate the m/r RCoS p-value for all possible values of m and choose the value of m with the best p-value. This p-value must of course be corrected for multiple testing. In this section we define the minimal-rank-consistency score, and show how to efficiently characterize its distribution, enabling the calculation of p-values (with no further need for multiple testing correction). We first describe the calculations and then analyze its total time complexity.N>0, we denote the set of ranks {1,..,N} by [N]; Let [N]r represent the set of vectors of length r, where each entry is from [N]. We use V to denote a random vector uniformly distributed over [N]r.For any number m-th smallest number in v by v<m>. That is, β the probability that <m>V will equal t or less. Note that β is the p-value, at s = t/N, of m-out-of-r rank consistency score defined previously, and can be efficiently computed as shown in the previous section.Given a vector minimal rank consistency score of a vector v, denoted by mRCoS(v), by mRCoS(v) is the best  rank consistency p-value, where m varies from 1 to r. mRCoS(V) is therefore a random variable taking values in . We now compute the exact p-value associated with mRCoS(V) at a given value, p:We define the t such that β for all β and compute mτ(p). Note that Given C, we say that a vector v∈[N]r is C-bounded if m = 1,..,r). In words, all sorted entries of v are larger  the corresponding entries of C. For example, the vector v = <3,2> is bounded by Given a constant ranks vector N]r that are C-bounded is denoted by B.The total number of vectors in r are bounded by a given vector By the definition of N, r, and a vector C, we want to compute B, the number of C-bounded vectors in [N]r. For each vector v we define two properties: t(v) and k(v).Given two integers, t(v) is the maximal entry of v. That is, t(v) can assume the values 1 through N.k(v) is the number of entries in v whose value is strictly smaller than t(v). Note that k(v) can assume the values 0 through r−1.N]r.These two properties can be used to partition [C-bounded vectors for which C-bounded vectors. By using B, summing over all possible values of t and k:We denote the set of all B. We use a dynamic programming approach to compute all N*r values.As there are only N*r such sets this would yield an efficient procedure to compute C(1..k) be the first k elements of C, that is t. Therefore, to compute v of the k smallest values, and their actual values, such that they are all strictly smaller than t, and are C(1..k) bounded:Let C-bounded vectors:We now use the following dynamic programming procedure to compute the number of This enables us to efficiently compute the minRCoS p-value:2*r2). To compute 2*r2).There are a total of N*r dynamic programming steps needed to calculate B. In each step, calculating B requires summing over t*k values of B. In total the complexity of the dynamic programming procedure to compute B is therefore O. The matched pairs of tumor and normal RNAs were from 14 different patients and 8 different cancer types. Tissue samples were of various embryonic lineages: One pair from breast, lymphoma, and prostate; two pairs from liver, ovary, testes and lung; and 3 pairs from colon. Technical replicates were performed for the ovary and testes samples, thus a total of 32 microarray data were used for this study.For each microarray measurement, 100ng total RNAs were labeled with Cy3 using T4 RNA ligase per Agilent miRNA Micorarray Systems Protocol v1.5. The labeled RNA samples were hybridized onto Agilent miRNA microarray (Agilent Human miRNA Microarray kit V2 - G4470B) for 21 hours at 55C. The arrays contain probes for 723 human and 76 human viral miRNAs from the Sanger database v.10.1. The arrays were then washed at room temperature and scanned to produce the hybridization signals (Agilent miRNA Micorarray Systems Protocol v1.5). The arrays were scanned with extended dynamic range at 5 and 100% PMT using the Agilent scanner (model G2565AA).th percentile value. The raw and normalized data have been deposited in NCBI's Gene Expression Omnibus http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc= GSE14985). All data is MIAME compliant. The normalized data are also available in Agilent's Feature Extraction software version 9.5.3.1 was used to generate GeneView files We applied rank consistency scoring methods to data collected in a study of miRNA expression profiles in cancer related samples. Data collected in this study consisted of paired samples of tumor and normal origins. Each pair of samples was taken from different parts of the same tissue in 14 different patients and 8 different cancer types: breast, colon, liver, lung, lymphoma, ovary, prostate and testis. The matched pairs of samples enable us to focus on changes in miRNA expression levels that result from the cancer process and to minimize the confounding effect of inter-individual and inter-tissue variability.The goal of the study was to identify miRNAs universally differentially expressed in cancer using the statistical methods and measurements described above.We computed the tumor vs. normal differential expression of each miRNA in the data in four different ways: TNoM For the different variants of RCoS (m/r RCoS and minRCoS), fold change was calculated for each miRNA and patient by dividing the tumor signal by the normal signal. In cancer types where more than one patient exists (2 or 3 patients) the median of the fold changes was used. This was done to preserve the patient matching (within the same cancer type) in our data. For each cancer type the miRNAs were then ranked according to these values to generate the ranked lists needed as the input to all the RCoS variants. The application of the general framework described in the Class A and class B are tumor and normal.r = 8.If for a miRNA, denoted g, we have, for example, 6/8 RCoS(g) = 0.2 for over-expression in tumor vs. normal, then this miRNA is ranked amongst the top 20% of miRNAs over-expressed in tumor vs. normal, for at least 6 out of the 8 different tumor types. Obviously, similar interpretations hold true for other values of m and s .The complete set of results of our analysis, including all the differential expression scores and the associated p-values, is available as supplementary material .To apply the paired t-test on these data, fold change was calculated for each miRNA and patient by dividing the tumor signal by the normal signal. In cancer types where more than one patient exists the median of the expression values was used in the fold change calculation. The data was then log-transformed to achieve the normality required by the paired t-test. We note that even after the log-transformation, the hypothesis of normality of this distribution is rejected by the Jarque-Bera test The observed and expected numbers of genes for all minRCoS p-values and the levels at which FDR  A heatmap of the most significant miRNAs identified by minRCoS analysis is shown in Some of the miRNAs we observe as differentially expressed were identified as significantly differentially expressed both by matched and by non-matched analysis. For example, miR-96 which is discussed in detail below was found by all four methods described above.In contrast to miR-96, other miRNAs were detected by minRCoS and not by other methods (both matched and non-matched). An example of such a miRNA which is also not reported in previous multi-type cancer datasets −9) and a minRCoS p-value of 10−8. A close inspection reveals that, excluding the liver sample, miR-133b is under-expressed in all tumor types, compared to the corresponding matched normal tissue. Interestingly  are part of the top 30 over-expressed miRNAs. This polycistron is a known oncogene in several cancer types −9. miR-486-5p  was recently found to be down regulated in Glioblastoma stem cells (CD133+) compared to non-stem (CD133−) cells The top ranking under-expressed miRNA in cancer based on minRCoS ranking is miR-486-5p. It is under-expressed in all 8 cancer types and has a minRCoS p-value of 10r = 7). The list of all 217 miRNAs measured in the Lu et al study and their p-values is provided in We also applied our methods to the Lu et al −7, see When searching for over-expressed miRNAs we find, for example, consistent high ranks for miR-182 and miR-183 for each group (such as a patient). This ranking reflects a quantity of interest that was computed or measured in each group, such as the extent of differential expression. The methods will find elements with consistent high ranks across all (or most) of the groups. Software for computing this is available at: The concordance of the findings of our study with the findings of several other studies and the use of RCoS on the Lu et al data are strongly supportive of the cancer-universal nature of the differential expression of several known cancer-associated miRNAs, namely: miR-133, miR-96 and miR-182. Importantly, this concordance demonstrates the utility of our statistical methodology for analyzing data from different platforms and multiple cancer types. Also, it lends confidence in the miRNAs identified here as differentially expressed in cancer. Thus, in addition to identifying the already known cancer-associated miRNAs mentioned above, our method has identified two novel cancer-associated miRNAs, namely miR-486-5p and miR-629*. As we tested multiple tumor types, these appear to be novel cancer-universal miRNAs.Text S1Pseudo-Code for computing m/r RCoS and minRCoS(0.03 MB DOC)Click here for additional data file.Table S1Normalized data matrix(0.52 MB XLS)Click here for additional data file.Table S2Table of all miRNAs measured along with their RCoS and minRCoS p-values, t-test and paired t-test(0.30 MB XLS)Click here for additional data file.Table S3Results of applying the RCoS statistics on the Lu et al (0.09 MB XLS)Click here for additional data file.


High-throughput measurement of allele-specific expression (ASE) is a relatively new and exciting application area for array-based technologies. In this paper, we explore several data sets which make use of Illumina's GoldenGate BeadArray technology to measure ASE. This platform exploits coding SNPs to obtain relative expression measurements for alleles at approximately 1500 positions in the genome.We analyze data from a mixture experiment where genomic DNA samples from pairs of individuals of known genotypes are pooled to create allelic imbalances at varying levels for the majority of SNPs on the array. We observe that GoldenGate has less sensitivity at detecting subtle allelic imbalances (around 1.3 fold) compared to extreme imbalances, and note the benefit of applying local background correction to the data. Analysis of data from a dye-swap control experiment allowed us to quantify dye-bias, which can be reduced considerably by careful normalization. The need to filter the data before carrying out further downstream analysis to remove non-responding probes, which show either weak, or non-specific signal for each allele, was also demonstrated. Throughout this paper, we find that a linear model analysis of the data from each SNP is a flexible modelling strategy that allows for testing of allelic imbalances in each sample when replicate hybridizations are available.Our analysis shows that local background correction carried out by Illumina's software, together with quantile normalization of the red and green channels within each array, provides optimal performance in terms of false positive rates. In addition, we strongly encourage intensity-based filtering to remove SNPs which only measure non-specific signal. We anticipate that a similar analysis strategy will prove useful when quantifying ASE on Illumina's higher density Infinium BeadChips. Preferential expression of one of the two alleles of a gene has been widely studied in the context of development, where key mechanisms such as genomic imprinting and X-inactivation lead to extreme allelic imbalances . Allele-Various experimental techniques exist for measuring ASE , includiIllumina's two-color GoldenGate technology has been used to measure ASE in pancreatic cancer  and lympThe fluorescence of each probe is quantified by Illumina's scanning software (BeadScan) and summarized values for each SNP are output by the BeadStudio software. The default preprocessing steps used in this analysis have been shown to offer good performance on spike-in data sets for Illumina's single-channel expression data . In thisIn this paper, we focus on the data analysis issues that arise when Illumina GoldenGate BeadArrays are used to measure ASE. This paper is organized as follows. We first present the raw data from a series of arrays, and explore the general signal characteristics. Next, we examine a published control data set that allows us to quantify dye effects. We then look at the results from a mixture experiment, which is designed to produce known allelic imbalance at varying degrees for the majority of SNPs on each array, to assess the ability of different preprocessing methods to recover the true positives. Finally, we investigate what effect a gene's expression level has on our ability to measure ASE.2 signal to be a good predictor of failed hybridizations, and use a threshold criterion of IQR ≤ 1 in either channel to flag poor quality arrays to exclude from further analysis  w. (2008)  under acet al. (2008)  w. (2008) . For SNP w.  [The HapMap and Pancreatic cancer data from Tan . (2008)  were alset al. (2007) [To measure expression of each gene independently, the Illumina WG-6 expression data from Stranger . (2007)  were dow2(Cy3) and log2(Cy5) intensities, average log-intensities (M = log2(Cy5/Cy3)). For each quantity, outlier beads  were removed prior to calculating the average. The local background intensities were estimated using an average of the five dimmest pixels within the 17 × 17 pixel area around each bead centre, as per Illumina's default image analysis.The bead-level data from each array in the mixture experiment were summarized by calculating the per bead type average of 4 quantities after outlier removal both with and without the local background estimates subtracted: the log2(Cy3) and log2(Cy5) values from each array were quantile normalized (within-array) and log-ratios were calculated. This analysis was carried out in R [beadarray [beadarraySNP packages.To obtain normalized data, the summary logout in R  using theadarray  and beadet al. (2008) was provided by Aik Choon Tan . For each array, log-ratios and average log-intensities were calculated both with and without quantile normalization between channels.BeadStudio output from the dye-swap experiment in Tan et al. (2008) were downloaded from the GEO database [GEOquery Bioconductor package [The remaining data from Tan database  using th package . This dasmoothScatter function with the default options.The two-dimensional smoothed scatter plots T is the vector of log-ratios from arrays 1, ..., N, X is a known design matrix with full column rank, and βi = T is a SNP-specific vector of regression coefficients.where The linear model assumesyij are normally distributed and that the values from different arrays are independent. Ordinary least squares estimators of βi  [αik = 0. Since the mixture experiment uses samples from individuals with known genotypes, we know a priori which SNPs will have a differential allelic response. Sensitivity and specificity were calculated for the concentrations in each series by ranking SNPs by their log-odds.Moderated h (2004)  to test 2 cDNA and gDNA log-ratios, both before and after within-array quantile normalization.For the dye-swap experiment, SNP-wise regression models as described above (Equation 1) which included an intercept (dye effect) term were fitted separately for the loget al. (2008) [We assessed the degree of linear trend between the cDNA and gDNA log-ratios from Tan . 2008)  (Figure   (FigureASE: allele-specific expression; CEU: Centre d'Étude du Polymorphisme Humain samples collected from UT, USA, which are part of the HapMap project [MAD: median absolute deviation; PCR: Polymerase chain reaction; ROC: Receiver Operator Characteristic, a method used to assess sensitivity and specificity; SAM: Sentrix Array Matrix, a collection of 96 BeadArrays in 96-well plate format; SNP: Single Nucleotide Polymorphism; WG-6: whole-genome, expression BeadChips from Illumina which contain 6 individual BeadArrays; YRI: samples from individuals from Yoruba in Ibadan, Nigeria, which are part of the HapMap project [The following is a summary of the abbreviations used in this paper:  project ,35; MAD: project ,35.MER performed the analysis and drafted the manuscript, MSF performed the experiments, MSF, AD, CD, ETD and PD planned the experiments and ST supervised the research and finalized the manuscript. All authors read and approved the final manuscript.


More studies based on gene expression data have been reported in great detail, however, one major challenge for the methodologists is the choice of classification methods. The main purpose of this research was to compare the performance of linear discriminant analysis (LDA) and its modification methods for the classification of cancer based on gene expression data.The classification performance of linear discriminant analysis (LDA) and its modification methods was evaluated by applying these methods to six public cancer gene expression datasets. These methods included linear discriminant analysis (LDA), prediction analysis for microarrays (PAM), shrinkage centroid regularized discriminant analysis (SCRDA), shrinkage linear discriminant analysis (SLDA) and shrinkage diagonal discriminant analysis (SDDA). The procedures were performed by software R 2.80.PAM picked out fewer feature genes than other methods from most datasets except from Brain dataset. For the two methods of shrinkage discriminant analysis, SLDA selected more genes than SDDA from most datasets except from 2-class lung cancer dataset. When comparing SLDA with SCRDA, SLDA selected more genes than SCRDA from 2-class lung cancer, SRBCT and Brain dataset, the result was opposite for the rest datasets. The average test error of LDA modification methods was lower than LDA method.The classification performance of LDA modification methods was superior to that of traditional LDA with respect to the average error and there was no significant difference between theses modification methods. Conventional diagnosis of cancer has been based on the examination of the morphological appearance of stained tissue specimens in the light microscope, which is subjective and depends on highly trained pathologists. Thus, the diagnostic problems may occur due to inter-observer variability. Microarrays offer the hope that cancer classification can be objective and accurate. DNA microarrays measure thousands to millions of gene expressions at the same time, which could provide the clinicians with the information to choose the most appropriate forms of treatment.Studies on the diagnosis of cancer based on gene expression data have been reported in great detail, however, one major challenge for the methodologists is the choice of classification methods. Proposals to solve this problem have utilized many innovations including the introduction of sophisticated algorithms for support vector machines  and the Cancer is not a single disease, there are many different kinds of cancer, arising in different organs and tissues through the accumulated mutation of multiple genes. Many previous studies only focused on one method or single dataset and gene selection is much more difficult in multi-class situations ,7. EvaluIn summary, we investigate the performance of LDA and its modification methods for the classification of cancer based on multiple gene expression datasets.Procedure for the classification of cancer is shown as follows. First, a classifier is trained on a subset (training set) of gene expression dataset. Then, the mature classifier is used for unknown subset (test set) and predicting each observation's class. The detailed information about classification procedure is shown in Figure Six publicly available microarray datasets -14 were To avoid the noise of the dataset, pre-processing was necessary in the analysis. Absolute transformation was first performed on the original data. The data was transformed to have a mean of 0 and standard deviation of 1 after logarithmic transformation and normalization. When the original data had already experienced the above transformation, it entered next step directly.ij be the expression level of gene j in the sample i, and yi be the cancer type for sample i, j = 1,...,p and response yi∈{1,...,K}. Denote Y = T and xi = T, i = 1,...,n. Gene expression data on p genes for n mRNA samples may be summarized by an n × p matrix X = (xij)n × p. Let Ck be indices of the nk samples in class k, where nk denotes the number of observations belonging to class k, n = n1+...+nK. A predictor or classifier for K tumor classes can be built from a learning set L by C; the predicted class for an observation x* is C. The jth component of the centroid for class k is Let xPAM  algorithkj is a t statistic for gene j, comparing class k to the overall centroid, and sj is the pooled within-class standard deviation for gene j:where d0 is a positive constant and usually equal to the median value of the sj over the set of genes.and Equation(1) can be transformed tokj toward zero, and giving PAM method shrinks each dSoft thresholding is defined by+ = t if t>0 and zero otherwise). For a gene j, if dkj is shrunken to zero for all classes k, then the centroid for gene j is where + means positive part  is the ith order statistic of the collection of p-values(1 ≤ i ≤ p). The ideal threshold optimizes the classification error. SDA consists of Shrinkage linear discriminant analysis (SLDA) and Shrinkage diagonal discriminant analysis (SDDA) [In SDA, Feature selection is controlled using higher criticism threshold (HCT) or false non-discovery rates (FNDR) . The HCTs (SDDA) ,16.There are two parameters in SCRDA , one is First, all the pairs  that corresponded to the minimal cross-validation error from training samples were found.Second, the pair or pairs that used the minimal number of genes were selected.p, small N" paradigm, so we did not adopt it to select feature genes.When there was more than one optimal pair, the average test error based on all the pairs chosen would be calculated. As traditional LDA is not suitable to deal with the "large xa with large ratios of between-group to within-group sums of squares. Fisher's criteria can be defined as:Fisher linear discriminant analysis   projectsB and W denote the matrices of between-group and within-group sums of squares and cross-products.Where k sample means Class l is eigenvector, s is the number of feature genes.where When numbers of classes K = 2, FLDA yields the same classifier as the maximum likelihood (ML) discriminant rule for multivariate normal class densities with the same covariance matrix.j +s0. The relative number of samples in each class is corrected at the same time. For a test sample (a vector) with expression levels x*, the discriminant score for class k was defined by,PAM  assumes k = nk/n or πk = 1/K is class prior probability, where πHere The corresponding discriminant score  was defiP = (ρij) and Where A new test sample was classified by regularized discriminant function ,(11)Covariance was estimated byα ≤ 1where 0 ≤ In the same way, sample correlation matrix Then the regularized sample covariance matrix was computed by -14, and no singular was detected.We used 10-fold cross-validation (CV) to divide the pre-processed dataset into 10 approximately equal-size parts by random sampling. It worked as follows: we fit the model on 90% of the samples and then predicted the class labels of the remaining 10% (the test samples). This procedure was repeated 10 times to avoid overlapping test sets, with each part playing the role of the test samples and the errors on all 10 parts added together to compute the overall error . R softwAs shown in Table The performance of the methods described above was compared by average test error using 10-fold cross validation. We ran 10 cycles of 10-fold cross validation. The average test errors were calculated based on the incorrectness of the classification of each testing samples. For example, for the 2-class lung cancer dataset, using the LDA method based on PAM as the feature gene method, 30 samples out of 100 sample test sets were incorrectly classified, resulting in an average test error of 0.30.The significance of the performance difference between these methods was judged depending on whether or not their 95% confidence intervals of accuracy overlapped. Here, if the upper limit was greater than 100%, it was treated as 100%. If two methods had non-overlapping confidence intervals, their performances were significantly different. The bold fonts in Table p (genes) far exceeds the number of samples N. In statistical terms, it is called 'large p, small N' problem. Standard statistical methods in classification do not work well or even at all, so improvement or modification of existing statistical methods is needed to prevent over-fitting and produce more reliable estimations. Some ad-hoc shrinkage methods have been proposed to utilize the shrinkage ideas and prove to be useful in empirical studies [Microarrays are capable of determining the expression levels of thousands of genes simultaneously and hold great promise to facilitate the discovery of new biological knowledge . One fea studies -23. DistThe gene selection method ,24,25, tet al [et al [Various statistical and machine learning methods have been used to analyze the high dimensional data for cancer classification. These methods have been shown to have statistical and clinical relevance in cancer detection for a variety of tumor types. In this study, it has been shown that LDA modification methods have better performance than traditional LDA under the same gene selection criterion. Dudoit also reported that simple classifiers such as DLDA and Nearest Neighbor performed remarkably well compared with more sophisticated ones, such as aggregated classification trees . It indiet al  developel [et al  are of cThere are several interesting problems that remain to be addressed. A question is raised that when comparing the predictive performance of different classification methods on different microarray data, is there any difference between various methods, such as leave-one-out cross-validation and bootstrap ,30? And An extensive survey in building classification models from microarray data with LDA and its modification methods has been conducted in the present study. The study showed that the modification methods are superior to LDA in the prediction accuracy.CV: Cross-validation; DDA: diagonal discriminant analysis; FNDR: False non-discovery rates; GLDA: generalized linear discriminant analysis; HCT: Higher criticism threshold; LDA: linear discriminant analysis; NSC: nearest shrunken centroid method; PAM: prediction analysis for microarrays; SCRDA: Shrinkage centroid regularized discriminant analysis; SDA: Shrinkage discriminant analysis; SDDA: Shrinkage diagonal discriminant analysis; SLDA: Shrinkage linear discriminant analysis.The authors declare that they have no competing interests.DH conceived the study and drafted the manuscript. DH and YQ performed the analyses. MH provided guidance and discussion on the methodology. BZ attracted partial funding and participated in the design of the analysis strategy. All authors read and approved the final version of this manuscript.


The original spotted array technology with competitive hybridization of two experimental samples and measuring relative expression levels is increasingly displaced by more accurate platforms that allow determining absolute expression values for a single sample . Unfortunately, cross-platform comparisons show a disappointingly low concordance between lists of regulated genes between the latter two platforms.Whereas expression values determined with a single Affymetrix GeneChip represent single measurements, the expression results obtained with Illumina BeadChip are essentially statistical means from several dozens of identical probes. In the case of multiple technical replicates, the data require, therefore, different stistical treatment depending on the platform. The key is the computation of the squared standard deviation within replicates in the case of the Illumina data as weighted mean of the square of the standard deviations of the individual experiments. With an Illumina spike experiment, we demonstrate dramatically improved significance of spiked genes over all relevant concentration ranges. The re-evaluation of two published Illumina datasets  66, 10460; spermatogenesis in normal and teratozoospermic men, Platts et al. Human Molecular Genetics (2007) 16, 763) significantly identified more biologically relevant genes as transcriptionally regulated targets and, thus, additional biological pathways involved.The results in this work show that it is important to process Illumina BeadChip data in a modified statistical procedure and to compute the standard deviation in experiments with technical replicates from the standard errors of individual BeadChips. This change leads also to an improved concordance with Affymetrix GeneChip results as the spermatogenesis dataset re-evaluation demonstrates.This article was reviewed by I. King Jordan, Mark J. Dunning and Shamil Sunyaev. Microarrays that rely on hybridization with DNA probes pioneered large-scale expression studies. After the introduction of spotted array technology in the mid 90s, microarrays have steadily gained popularity for exploratory gene expression analysis. A spotted array experiment requires both the treated and control samples to be labeled with different dyes and to be competitively hybridized on the same array. The expression level is expressed as a ratio between the intensities between the two labels. Spotted arrays are plagued by accuracy and sensitivity problems that are only partly remedied by the measuring only relative expression. Dye bias and repeatability remain unsatisfactory.In recent years, Affymetrix GeneChip and Illumina BeadChip have emerged as two of the most popular microarray platforms. From the experimental design viewpoint, the GeneChip and BeadChip offer flexibility in terms of their ability to measure absolute expression values for each experimental sample independently. The growing amount of publicly available microarray data has prompted researchers to explore ways to compare results between experiments across the different platforms. This task signifies the first step in producing consistent and trusted results to support meaningful biological discovery. Yet, it is more difficult than it appears superficially. Even a simple variant of the problem like comparing results from the same sample across different platforms is not trivial. The first step requires the statistically significant changes in gene expression to be determined between treatment conditions for each platform. The platform-specific gene lists generated by applying the same significance threshold are finally compared. In general, the concordance between these gene lists is disappointingly low. Nevertheless, recent works have shown that concordance improvements can be made by filtering for gene nucleotide sequence identity -4, by suDuring our evaluation of cross-platform comparison between Affymetrix and Illumina, we stumbled upon another, quite surprising reason for the low concordance. Given the specific design of Illumina arrays -10, it aThus, Affymetrix GeneChips provide individual measurement results but the Illumina BeadChips generate means and standard errors for subsets of bead intensity measurements. Therefore, the summary data of a BeadChip experiment requires a different statistical interpretation compared with the individual measurements in the case of GeneChip data, especially in cases of multiple technical replicates. If the average of the bead intensities delivered by a single Illumina BeadChip is fed into standard expression profile analysis software , the standard deviation over technical replicates is calculated from the deviations of the subset means from the overall mean. But more correctly, the overall standard error is to be computed by taking into account also the standard deviations obtained from the individual BeadChips.In this paper, we will first present a derivation for the correct summary statistic applicable to Illumina BeadChips data. Furthermore, this summary statistic will be applied to one control experiment with artificial spikes and, also, it will be used for the re-evaluation of two published biological experiments. In all cases, the modified treatment is contrasted against the standard one. In the control experiment , we willX. The expression profile experiment is supposed to consist of K technical replicates . Each bead intensity xk,n is an instance of the random variable X . We assume that the first Mk beads are retained after outlier removal (see below). The summary data includes the mean μk, the standard error σk is the standard deviation) and the number of beads Mk .The bead intensity of a given gene in a BeadChip is described with the random variable k-th array are denoted as The observed BeadChip intensities of gene in the μtotal and standard deviation σtotal of bead intensities of a gene across the K technical replicates are given asUsing the means and standard errors of all the technical replicates, the mean σtotal is composed of two components each carrying a different meaning. Given that each of the K technical replicates represents the same, identical and independent distribution, one expects the K mean estimates μk to be relatively similar and, hence, σμ would be small relative to σtotal and, ideally, close to zero. Since averaging for each replicate is carried out over about 30 individual measurements, it can be assumed that each individual μk is likely a good estimate of the population mean μ if there were no batch variations. Therefore, a large σμ can be interpreted as batch variation or noise among the replicates, a considerable part of which, apparently, has systematic origin such as variations in the total amount of hybridization-ready nucleic acids, etc. Ideally, K  should be much larger in order to obtain a good estimate of σμ. However, this is impractical due to the high cost of performing large number of microarrays. Therefore, we suggest to assume σμ ≈ 0 for the case of no batch variation in equation (5) and to use σwtrep as a good lower estimate of the summary statistic in testing instead of σμ (see Appendix 2 for proof).A proof for equation (2) is supplied in the Appendix 1. The standard deviation σtotal with respect to equation (5) instead of (2).This proposed summary statistic is supported by observations communicated in two recent publications, which have leveraged on the variation in bead intensities. Dunning et al.  showed tn et al.  proposedth row of Table K technical replicates are given asIn the case of a Affymetrix GeneChip experiment, the measurement for a gene is taken only once in each replicate . The lack of systematic error as a confounding factor in σtotal increases the chance of detecting true biological differences from the statistical tests.The summary statistic control and treatment respectively. The summary statistic σμ is incorrect since it measures only the batch variation and not at all variation in bead intensities. The correct summary statistic should be control and treatment. The emphasis of using σwtrep instead of σμ as the summary statistic is not statistical hair splitting but this issue affects the biological interpretation as we can see from the following three examples.In any case, the more important concern related to the analysis of Illumina data is the mistake of treating the mean estimates of bead intensities as instances of the bead intensities. Standard gene expression profile analysis software  assuσμ instead of σwtrep as summary statistic, each pair of spike experiments at adjacent concentrations were compared . 'Heteroskedasticity' means that the larger intensities tend to have larger variations, a common observation with many types of microarray data. The 'heteroskedasticity' nature of the relationship between the mean bead intensities μtotal and the variation in bead intensities σwtrep is apparent . Given that the number of technical replicates K is only four, obtaining good estimates for σμ especially in the higher intensities region is impossible. As such, we strongly advocate the use of equation (5), which only relies on σwtrep for computing the correct summary statistic, instead of equation (2).This means that σμ as the summary statistic). A total of 207 differentially expressed genes were determined with cutoff criteria of p ≤ 0.05 and absolute fold change (FC) of at least 2.Golubkov et al.  publisheσμ and σwtrep, which yielded 215 and 218 differentially expressed genes respectively upon applying the same cutoff criteria. For the three lists consisting of 207, 215 and 218 gene candidates, RefSeq IDs were extracted. The resulting 202, 200 and 203 RefSeq IDs were then separately submitted to NIH DAVID [In this work, the original expression data was first normalized  prior to statistical treatment. Welch's t test was then performed for both IH DAVID  for KEGGσμ essentially repeats the outcome of the work by Golubkov et al. with respect to the pathways . The elucidation of this pathway has substantiated the authors' claim with statistically significant expression arguments that the cell cycle is disrupted with observable mitotic spindle aberrations and aneuploidy in the 184B5-MT cells [Application of the statistic MT cells .Platts et al.  studied ν and ω as the summary statistic (equations 6 and 7). This resulted in a total of 11932 differentially expressed genes (6861 RefSeq IDs) after applying cutoff criteria of p ≤ 0.01 and |FC| ≥ 2. In a similar fashion, the expression data from the corresponding 5 normal and 8 teratozoospermic of the Illumina experiment was normalized and statistically treated for both σμ and σwtrep. Using the same cutoff criteria, the two analyses yielded 2464 DEGs (2109 RefSeq IDs) and 4149 DEGs (3316 RefSeq IDs) respectively. Since the number of differentially expressed genes for σwtrep is increased for the same cutoff criteria, this statistic exhibited a higher statistical power.In this work, 5 out of the 13 normal and the 8 teratozoospermic samples from the Affymetrix experiment that were used by Platts et al. in their Illumina experiment  were re-analyzed. The gene-level data was normalized , followed by a pair-wise t-test with The three RefSeq ID lists were submitted to DAVID for KEGG pathway mapping. For Affymetrix, the background list was set to a list of 39647 ReqSeq IDs that was extracted from the HG-U133 (version 2) annotation file. For Illumina, the same list of 19815 RefSeq IDs from MT1-MMP example (see previous section) was submitted as the background.σwtrep was used instead of σμ, this increase only improved the EASE score from 0.07 to 0.066. This marginally missed the significance cutoff of ≤ 0.05. But more interestingly, the Illumina analysis was able to elucidate a few pathways involved in spermatogenesis [σwtrep was used as the summary statistic.Analogous to Table ogenesis , like thogenesis -23, adheogenesis -26 when σwtrep but not σμ. Using σwtrep as the summary statistic, the P-values for these 8 genes were improved. With the addition of these 8 genes, the EASE score improved from 0.143 to 0.003. More notably, genes with biological evidence on the role of N-glycan biosynthesis pathway in spermatogenesis begin to surface with the application of the correct summary statistic. There is independent experimental evidence that proves the involvement of the detected genes. For example, beta-1,4-galactosyltransferase-I (B4GALT1) is found to bind with ZP3 receptors on the sperm surface [Table  surface . Also, t surface ,29. MAN2 surface -23.σwtrep but not σμ  were validated spermatogenesis genes from the N-glycan biosynthesis, tight and adherens junction pathways. These pathways were not reported by Affymetrix. Another interesting observation is that the RefSeq list obtained with σμ as a statistic almost formed a subset of the list yielded by σwtrep. In total, 93.7% of its ReqSeq IDs were found within the list generated with σwtrep. At first glance, it seems that the effect of using σμ rather than σwtrep does not seem detrimental. However, the rankings of the P-values in this overlapped region were not preserved, similar to the case of our Illumina spike experiments analysis. As such, the top 100 candidate list for example will be quite different when σμ instead of σwtrep is used.The concordance between the three RefSeq lists was next investigated. The results are shown in Figure σwtrep is proposed as correct summary statistic of the Illumina BeadChip. Our work has shown that the same Illumina BeadChip data from published experiments churns out better differentially expressed gene selection after applying our proposed summary statistic.Due to the specific statistical nature of the Illumina BeadChip summary data as means and standard deviations of subsets of measurements, the typical statistical workflow of finding differentially expressed genes cannot be applied to this data directly. To remedy this situation, This was particularly evident in the low concentration range of the Illumina spike experiment ,14. GiveTo conclude, our work is most relevant and imperative to any investigator who wants to derive more accurate differentially expressed gene lists from Illumina data.μg per sample) was hybridized in quadruplicates across 48 arrays on eight different Human-6 Expression BeadChips.We exploited the dataset from a published artificial spike experiment ,14; the This dataset available as NCBI GEO GSE5095 was complemented with replicate-specific standard errors and number of beads in a private communication by Vladislav S. Golubkov. In the experiment, 184B5 human normal mammary epithelial cells were transfected with MT1-MMP . Total RThe published expression profile dataset NCBI GEO GSE6969 from human ejaculates was used . In the Our proposed procedure is inspired by quantile normalization and by the scaling method used by Affymetrix . The quag from an array consisting of a total of G genes with K technical replicates each will have summary data μg,k, σg,k, Mg,k where g = 1,...,G and k = 1,...,K.For the sake of simplicity, the normalization procedure illustrated below will be based on only one treatment condition. The same steps will be repeated for other treatment conditions. Therefore, an arbitrary gene k-th technical replicate of gene g after undergoing log-transformation is depicted as log2.Log-transformation is first applied on the mean bead intensities for all readings. The k-th technical replicate, the median and standard deviation for the log mean bead intensities within replicate are calculated asFirst, normalization within the replicate is performed. For the MOMwtrep and The median-of-medians within replicate and its corresponding standard deviation, hereby depicted as k-th technical replicate of gene g is defined asTherefore, the normalized log mean bead intensities within replicate for the k-th replicate is then re-calculated in a similar fashion as before, whereThe median for the normalized log mean bead intensities within the MOMacrep, is defined asThe median-of-medians across replicates, depicted as Therefore, the normalized log mean bead intensities across replicates will beThe normalized log mean bead intensities across replicates are then transformed back to the original scale, whereand the corresponding standard deviation is nowg from an array consisting of a total of G genes with K technical replicates each will now have the summary data μg,k,norm, σg,k,norm, Mg,k where g = 1,..., G and k = 1,...,K.Therefore, after undergoing the array normalization procedure, an arbitrary gene K times in the Beadchip, i.e., k = 1,...,K. When the mean estimates μg,k are being treated as the intensity values xg,k,1, the mean and the incorrect summary statistic of the variance of this treatment condition are given asAssume that one treatment condition is replicated Note that On the other hand, when no misinterpretation of the summary statistic has occurred, the mean and variance are given asNote that c1 and c2, an arbitrary gene g will have two pair of readings μg,c1, σg,c1, ng,c1 and μg,c2, σg,c2, ng,c2. Then if the two treated samples came from normal populations and if both have equal variances, then the t value for the difference between the two treatment conditions is given asA pair-wise t-test  infers iν = ng,c1 + ng,c2-2. However, if the two treatment samples do not have equal variances, the t value is given aswhere The degree of freedom is given as HA : μg,c1 ≠ μg,c2, reject H0 : μg,c1 = μg,c2 if |t|≥ tα(2),ν. It should be noted that the array normalization procedure has been applied before the statistical treatment.This is known as the Welch's t test. For a 2-sided alternate hypothesis FP: false positive; MT1-MMT: membrane type-1 matrix metalloproteinase; TP: true positive.The authors declare that they have no competing interests.WCW and FE derived the proof for the new summary statistic, designed the study and evaluated the results. WCW carried out the programming. ML reproduced the work to validate the results. ML also corresponded with the respective authors of the study datasets for all additional information. All authors participated in drafting the manuscript and approved the final version.I. King Jordan, School of Biology, Georgia Institute of TechnologyWong, Loh and Eisenhaber present a novel statistical method for evaluating gene expression data produced using the Illumina BeadChip technology. The fundamental insight that led to the new statistical method is their appreciation that Affymetrix GeneChip microarrays produce single gene expression measurements, while use of Illumina BeadChips yields mean expression values from dozens of identical probes. Therefore, Illumina BeadChip data must be treated differently. Specifically, when technical replicates are available, the standard deviations across replicates for Illumina BeadChip data are best computed as weighted means of the square of the standard deviations of individual measures. In other words, the standard deviations for data sets with technical replicates should be computed from standard errors of individual Illumina BeadChip measures. When this adjustment is applied to several test data sets, the performance of the Illumina BeadChips improves markedly.While I am not qualified to evaluate the statistical details of their method, the results of its application to the three test data sets appear to be quite convincing. As such, this work represents an important technical development with direct relevance to any study that uses Illumina BeadChip technology.i.e. does the Affymetrix analysis uncover pathways missed by Illumina irrespective of the use of the statistical innovations introduced herein?One of the measures used by the authors to indicate the success of their statistic is increased concordance between lists of differentially expressed genes uncovered by Illumina BeadChip and Affymetrix GeneChip experiments on a spermatogenesis dataset. However, it would seem that the increased replicates of the Illumina BeadChip technology provides for an inherent advantage over the single-measure technology employed with Affymetrix GeneChips. If this is indeed the case, then one may expect improved performance for the Illumina platform relative to Affymetrix and not merely increased concordance as was demonstrated for the spermatogenesis dataset. Do the authors have any sense, or evidence, as to whether the increased sampling of Illumina provides more resolution than Affymetrix? For instance, are the new pathways identified by the Illumina BeadChip analysis of the spermatogenesis dataset a function of the superiority of the platform? Or are the methods complementary, There is no reason to assume a superiority of either platform given the same quality of probe sequence design. For example, one might imagine several Affymetrix chips to be mounted on the same glass slide and to be hybridized simultaneously . In this case, both platforms can be used in the mode of exclusion of the batch-specific constant shift error as described in the text.I am wondering about the availability of their method. The authors conclude that the work is relevant, even imperative, to any investigator looking for differentially expressed genes in Illumina data? How are those investigators to use this method – on a web server, as a BioPerl object, as an R routine?wtrep as standard deviationPresently, our code is implemented in Matlab and be obtained on request. It would be straightforward to implement an R version of it so that it can tie back to the bioconductor package in R. Nevertheless, it should not be difficult for any scientist in the area to modify their existing workflow similar to ours based on the equations presented in this paper just by using σ.Mark J. Dunning, Computational Biology Group, Department of Oncology, University of Cambridge, Cancer Research UK Cambridge Research InstituteIn my opinion, Wong et. al is a useful addition to the topic of analysis of Illumina data. Whilst the number of publications using Illumina data are growing rapidly, very few authors have tackled the issue of how such data should be analysed. Wong et al. do a very good job of explaining why the usual statistical tests, such as applied to Affymetrix may not be appropriate for Illumina data and that the extra information provided with an Illumina experiment (i.e. accurate gene-specific variances) can produce a more powerful test. It is especially pleasing to see that they are able to pick up biologically relevant results using the new summary statistic.σwtrep is well presented. However, a detail in the re-analysis of Golubkov et al. seems to be missing. In the original paper, genes were filtered using the detection scores obtained from Illumina's software. It does not seem that these scores were supplied in GEO, so were these scores also available to Wong et al. as part of their re-analysis? If not, how did they go about reproducing the filtering performed in Golubkov et al.?The investigation into the performance of μ, the differences of intensities do not depend on their previous correction by a constant background. Regardless of these two differences in our re-analysis, the results are essentially identical to the case of Golubkov et al.: The cell cycle pathway did not appear as significantly regulatedIndeed, the data stored in GEO is insufficient to carry out the calculations both in the paper of Golubkov et al. and in this work. We received the standard errors and number of beads through a private communication from Golubkov et al. Our first re-analysis aimed at repeating the work of Golubkov et al. differed from their approach in two aspects. On the one hand, we had another normalization algorithm (see Methods section); on the other hand, we did not carry out filtering. Just to note, the detection score P is calculated from Z-scores of intensities shifted by the background (intensity of negative control spots) and scaled with its standard deviation. In pairwise comparisons involving the Welch's test using the wrong summary statistic σ.σwtrep is better than current alternatives. However, I'm afraid I was a bit unsure of the connection between σwtrep and the normalization method proposed by the authors. Can I still use σwtrep in my differential expression analysis if I use the usual quantile normalization?The results supplied in the paper were enough to convince me that the summary statistic The summary statistic σwtrep can be used with the usual quantile normalization or any normalization methods. One only has to ensure that the standard deviation σk of the corresponding μk be adjusted by a transformation factor i.e. σk = Aσk where After which, σwtrep is computed using equation (4).What motivated the authors to propose this method of normalization? However, I feel that the description of the normalization procedure was not that easy to follow and would benefit from a small worked example if possible. Do the authors plan to make any of the methodology presented in the paper available in open-source software?In a typical Illumina BeadChip experiment, different treatment conditions should be hybridized within a chip, while their corresponding technical replicates should be distributed across chips. The treatment conditions within a chip shall be exposed to similar systematic and random error. Hence, the differences in spreads among the arrays or treatment conditions should ideally reflect true biological differences. The motivation of our normalization method is to create a two-step normalization procedure whereby the first step forces the same median and spread only among technical replicates while the second step simply ensures that the medians across all the arrays are common. As such, the spreads among the various treatment conditions need not be the same, thus preserving true differences. The software (as a Matlab program) is available on request.Aside from these questions, and suggestions to improve readability supplied separately to the authors, I am happy for this manuscript to be published.Specific Comments for the authors:• Bottom of Page 2: "But instead of delivering the individual bead intensities, the mean and standard error (i.e. the standard deviation divided by the square root of the number of beads) of the bead intensities, known as the summary data, are reported."This statement is possibly a bit misleading as the individual bead intensities are available with appropriate scanner modifications . I suggest this statement be changed to acknowledge this, although the summary data are usually the starting points for analysis when using Illumina's software.• Page 3 paragraph 3 – "Furthermore, this summary statistic will..." This should be changed to either "this summary statistic" or "these summary statistic". I suggest the manuscript be checked for other similar errors.• Equation 1 should have the sum going from k = 1..K rather than i = 1..Kσk.• Page 5 Paragraph 1 – The weights used in Dunning et. al are the inverse of • Page 5 Paragraph 3 -"Due to the multiple copies of the same probe within a single Illumina array, the standard variation can be computed...."Should be standard deviation rather than standard variation?σμ.."• Page 6 Paragraph 4 – "On the other hand, a trend of growth of mean estimates Should this be standard deviation of mean estimates?The suggested amendments have been made accordingly.Shamil Sunyaev, Division of Genetics, Dept. of Medicine, Brigham & Women's Hospital and Harvard Medical SchoolThis manuscript describes a new statistical method for the analysis of Illumina BeadChip microarrays. The authors realized that variance is underestimated for these microarrays because the measurements themselves are averaged over multiple probes. Thus, they suggest a new estimate of variance to be used in the analysis based on the Welch's t-test.The manuscript is well written. The statistical approach is straightforward but has been convincingly demonstrated to produce biologically meaningful results. The authors show that the corrected standard deviation estimate helps obtaining better results for the spike dataset  and also reveal more biologically relevant genes in the human spermatogenesis dataset and mammary epithelial dataset.As an outsider in this field, I do not understand why the analysis is based on the t-test, which heavily depends on sample estimates of variance and assumes normality. It seems that non-parametric method may suite the problem better.First of all, it is important to note that the summary data is computed using the arithmetic mean and the standard deviation formulas. These formulas are the maximum likelihood estimates (MLEs) of the normal distribution. In other words, normality is innately assumed on the summary data. Furthermore, the typical sample size for each gene measurement in Illumina is about 30 and t-test is known to be robust when sample size is large. More notably, t-test is robust against assumption violations as long as the sample sizes are almost equal and that only two-tailed hypotheses are considered. These were the conditions for all our test cases.Proof for μtotal and squared standard deviation K technical replicates can be computed asWith reference to Table where, for convenience of notation, we introduceμtotal and the squared standard deviation K technical replicates are given asOn the other hand, only the summary data are given. Then the mean K bead intensity averages is equal to μtotal (equation 8 and 11). For later usage, the equation (12) can be transformed by representing μk by the average of actual bead intensities Clearly, the weighted average of the K technical replicates On the other hand, the variance in bead intensities across As before, the values Minor transformations yield the equationMk ≈ M for k = 1,...,K, the weighted arithmetic averages in the above equations can be justifiably by normal averages.Obviously, the second term in equation (16) and the first in equation (13) are identical except for the sign. Together with equations (8) and (9), this proves that k-th batch-specific expression vector σk and batch-independent mean μ) and batch-specific systematic shift vector sk . Whereas the standard deviation of the k-th replicate is not affected by the constant systematic error, the mean μk is given as μk = μ + sk. Therefore, the mean μsys across replicates isWe assume the s denotes the average of the systematic shifts. The standard deviation K replicates is essentially the standard deviation σs of the systematic shifts as is shown with the derivationwhere σk of the k-th replicate is not affected by the batch-specific shift and, therefore, the batch-specific systematic error does not affect σwtrep. Rightfully, the systematic error should not be present after array normalization. Hence, under the assumption of no systematic error after array normalization, usage of σwtrep as reliable (lower) estimate of σtotal is justified.Consequently, the standard deviation of bead intensities Data for the spike experiment with Illumina BeadChips. The tab-delimited table provides the complete summary data to re-compute the results from Table Click here for file