A central task in contemporary biosciences is the identification of biological processes showing response in genome-wide differential gene expression experiments. Two types of analysis are common. Either, one generates an ordered list based on the differential expression values of the probed genes and examines the tail areas of the list for over-representation of various functional classes. Alternatively, one monitors the average differential expression level of genes belonging to a given functional class. So far these two types of method have not been combined.We introduce a scoring function, Gene Set Z-score (GSZ), for the analysis of functional class over-representation that combines two previous analysis methods. GSZ encompasses popular functions such as correlation, hypergeometric test, Max-Mean and Random Sets as limiting cases. GSZ is stable against changes in class size as well as across different positions of the analysed gene list in tests with randomized data. GSZ shows the best overall performance in a detailed comparison to popular functions using artificial data. Likewise, GSZ stands out in a cross-validation of methods using split real data. A comparison of empirical p-values further shows a strong difference in favour of GSZ, which clearly reports better p-values for top classes than the other methods. Furthermore, GSZ detects relevant biological themes that are missed by the other methods. These observations also hold when comparing GSZ with popular program packages.http://ekhidna.biocenter.helsinki.fi/users/petri/public/GSZ/GSZscore.html.GSZ and improved versions of earlier methods are a useful contribution to the analysis of differential gene expression. The methods and supplementary material are available from the website  Ordered Gene List, OGL, by sorting it according to gene regulation. The upper end of the OGL represents the strongest up-regulation and the lower end the strongest down-regulation in the pathological sample. The middle area of the list represents genes with insignificant regulation. Similar gene lists can also be generated with various other data sources, like sequence similarity searches, high throughput screening of gene knock-outs, or protein expression arrays.The analysis of differential gene expression between two sample types, such as pathological and healthy tissues, is one of the cornerstones of the modern biomedical science. Here typically the up or down-regulation of each gene in the pathological samples is measured. The obtained expression data can be considered as Originally, the analysis of differential expression focused on the few genes with the most extreme up or down regulation between the samples. This is sensitive to potential measurement errors, and prone to produce false positive findings, such as genes reacting to any sample handling. Furthermore, it cannot detect biological processes showing consistent but weak regulation between the two datasets. A central improvement was to carry out the analysis using pre-defined gene classes or gene sets, such as functional classifications -3. The pSignal summary methods take the whole gene list into account and calculate a mean or median based score for all the member genes of the class [Ranked list based methods [class level scoring functions.Although the utilization of the gene annotation classes moves the focus away from the actual single gene level to the more robust biological process level, it is strongly dependent on the definition of the used threshold. This has motivated many authors to propose threshold free gene set analysis methods  using each function itself to select the positive classes from the other half and (ii) by combining the results of all the methods from the other half and using this same set for all methods. Both tests were repeated with different number of positive classes. In addition, we further confirm our results by monitoring the correlation between the test and control results. GSZ constantly shows the best performance over different test situations, whereas the performance of other functions varies between the test situations.Next, we do repetitive testing of class scoring functions on real data set. As the correct positive and negative classes are not known in real data, we used the evaluated functions to predict them. We used We further evaluate the analyzed class scoring functions by generating large set of randomizations. Two very different datasets were used here. We monitored the magnitude of different empirical p-values, in order to see the risk of potential false positive findings for each function. Results showed the best performance for GSZ with small difference on one dataset and with very clear difference on the other dataset. We point out that all the observed differences in numerical data evaluations resulted from the class scoring functions, establishing GSZ-score as outstanding in this group. In addition, real data analysis shows weak performance by some of the popular scoring functions. This is, to our knowledge, one of the most detailed numerical evaluations of class scoring functions on real datasets.We also carried out a detailed biological evaluation of scoring functions. This was done by ordering GO classes with empirical p-values and monitoring top GO classes. Next, we defined biologically relevant classes for both datasets and analyzed how different methods were able to find them. GSZ-score clearly shows outstanding performance in finding the important biological themes on both datasets. Again some of the competing scoring functions evidently show weak performance on one or both of the datasets. Results were further analyzed by monitoring GO classes with largest pairwise difference between the GSZ-score and competing scoring function. Classes where GSZ-score outperformed others showed stronger differences in p-values and were clearly linked to the research setup.We also performed a comparison with some of the most widely used software packages in this research topic. This analysis is more difficult to evaluate due to the other potential sources of differences between software packages. However, this ensures that we evaluate the state of the art implementations of class scoring functions. We concentrate here on the evaluation of the reported list of GO classes, and mostly omit the reported p-values. GSZ again showed best performance on two tested datasets, when evaluated with biologically relevant classes.differential expression tests and their scores as differential expression test scores for clarity. The obtained gene list is then sorted using the selected score. In the next step, gene classes, one at the time, are taken to the analysis and the gene list is analyzed using either rank based methods or signal summary methods. We call these tests class scoring functions and their results class expression scores. Our aim is to improve the existing class scoring functions. For more information on this topic see earlier publications [A typical pipeline for gene set analysis starts by looking at the differences in the gene's behaviour in the two datasets, like in treatment and control. This can be done, for example, by calculating a score based on the difference between the mean expression values, like t-test score, with standard t-test or its variations ,18. We cications ,10,11,13We start with a ranked list based analysis, where a threshold is placed repetitively in between every consecutive pair of genes in the ordered list. The obtained subset is then used to calculate a selected statistical score. This test score can be a difference between two empirical cumulative distributions like in Kolmogorov-Smirnov statistics  or a logM denote the total number of genes, and let Xi, i = 1,..., M, denote the differential expression test score for the ith gene. Furthermore, let SN be a subset of N genes from the upper end of the OGL . We use a simple function:We propose to improve these tests by taking the differential expression scores into account. Let pos) and non-members of a gene class , among the N genes with the highest differential expression test scores. This will then be calculated separately with each threshold position N, and analogously for the lower part of the OGL. To simplify the notation, the subset SN will be kept implicit in our notation by dropping the subscript N in the sequel. A similar but more complex idea has been proposed in the article that motivated this research [This denotes a difference between the sums of scores for members . Howeve• Z-score normalization is easier to perform for the plain difference.• The obtained Gene Set Z-score (GSZ-score) includes other popular methods as limiting cases (discussed below).• The resulting score is only affected by differential expression test score values within the selected subset, whereas in the equation in , the who• As a more theoretical benefit, the obtained GSZ-score treats both class members and class non-members similarly. So the results stay identical if we flip the class member and class non-member classifications.The selection of the starting equation 1 is still somewhat an open issue and our main emphasis is rather on the Z-score normalization for gene set presented later. The similarities with other class scoring functions are discussed more in detail in the supplementary text S2 [see additional file E(Diff)) and variance (D2(Diff)) for the eq. 1 under the null hypothesis, that the class members and non-members are distributed randomly across the list. This allows us to use a Z-score normalization with equation:The raw difference is an unusable statistic due to the biases caused by different sizes of classes, different sizes of subsets, and different variances of the scores in different subsets. However, these can be corrected by calculating estimates for the expected value  = 2E(X)E(N) - ME(X) andwhere N is the number of positive genes in the subset, M is the size of the subset, E(N) is the mean and D2(N) is the variance of the hypergeometric distribution of the number of positive genes N for the analyzed subset. D2(X) and E(X) are the variance and the mean of the differential expression test scores for the subset. The derivation of E(Diff) and D2(Diff) is shown in Methods. As a further improvement to our GSZ-score we consider a regularized version:where prior variance k to the variance estimate inside the square root. This was done due to the too unstable behaviour with small subsets . We also take the median of the variance estimates, obtained with the class of size 10 across the whole gene list, and multiply it with a weight, w1 (0 ≤ w1 ≤ 1). Next the weighted variance medians are summed to obtain a value for k. Class size of 10 is used here as a reference class size, penalizing especially classes smaller than 10. This class size selection is 'ad hoc', but it has performed well in our analysis. We tested 12 different combinations for these two weights, with artificial data, and selected the best performing parameter values for real dataset analysis. The calculated GSZ-scores form a profile of score values over the OGL obtained from each cut-off position, including also the values corresponding to the whole gene list as a subset. From the obtained scores we select the maximum value similarly to earlier works [This is otherwise similar to eq. (2) but we add a er works ,11,13. Orow randomization the gene classifications associated with dataset rows are randomized, and in column classifications the sample labels associated with data columns are randomized. The aim was to see what type of biases the methods show as the GO class size varies, or as the threshold moves through the gene list. These results are shown in the supplementary text S1 [see additional file k significantly stabilizes the GSZ-score, with a group of tested parameter values for the k obtaining equally good results , thrIn addition, we show in supplementary text S1 [see additional file Case 1: The whole gene class shows a similar type of regulation, either up or down. Nevertheless, the complication in this simple case is that the signal for the gene class can be quite diffuse with larger STD, depending on the different measurement efficiencies for various RNAs. Case 2: only a subset of the gene class shows signal (up- or down-regulation) like in case 1. Potential explanations are: Only a subset of the gene class requires regulation or the gene class includes false positive genes that are not really members of the gene class. Case 3: Gene class shows both up and down-regulation at the same time. Here gene class can comprise differently behaving subclasses or the members can be from a pathway with activities that either increase or decrease the activity of the pathway.Next, we evaluate the class level scoring functions on very large set of various artificial datasets. We currently lack complete knowledge on what can be considered as a positive signal for a differentially expressed gene class. However, we propose a few signal types that can be intuitively justified. Here we have used an artificial data generation similar to previous work ,10. TestFirst, the analysis was carried out with various gene class sizes, testing each class size separately. Thus, the compared results for positive and negative cases are obtained with the same gene class size. Next, the results for different class sizes and signal types are combined. This measures the performance of the method in the case where the various gene class sizes are separately normalized with mean and standard deviation estimates obtained from randomized data. This is currently the standard procedure of most programs. In another option, the positive and negative results for different class sizes are pooled and the separation is monitored within the pooled data. This measures the stability of the functions when the gene class size varies. This corresponds to the performance of the method without any class specific normalization with randomized data (proposed for example with iGA ).In each testing situation with each parameter setting for positive datasets we generate 200 positive datasets and 600 negative datasets. Next, we calculate the test scores with each of these datasets. We used Area Under receiver-operating characteristics Curve (AUC) as a measure of separation. This was considered better than the analysis of power , as powei and j specify methods that are compared and l = 1, 2.. R refers to dataset, obtained with specific parameter values. Score A represents the mean of differences of AUC scores for two methods across all the datasets in the testing situation, whereas score B represents which of the methods, i or j, was more frequently better. We generated with both of these scores a difference matrix representing all the method pairs. Next the average for each method i was taken, by letting j go over all the methods. Now we have two measures for the tested method: One corresponds to average of difference to other methods and the other corresponds to frequency of better performance than with other methods. Results for these two scores are shown in table where The results show the best overall performance for GSZ-scores. The only exception to the rule is the better performance of iGA in the testing with simultaneous up and down-regulation with fixed class size. Even there GSZ-scores showed good performance in a larger subset of the datasets (shown by the score B). This discrepancy of two measures proposes that we frequently see slightly better performance from GSZ-score and a small subset where iGA clearly surpasses GSZ-scores. Subsequent analysis links this to datasets showing signal around 1 STD  dataset , due thegold standard) are not known. Therefore we used the methods themselves to estimate the positive classes. This can omit some positive classes that none of the methods can discover and equally report some negative classes as positive. Nevertheless, we are not aware of any other realistic solutions for selecting positive GO classes, and minor errors should not corrupt our results. We also presume that the GO classes form a smooth transition with strongly positive GO classes at one end and strongly negative GO classes at the other end rather than a sharp binary classification to positive and negative classes. We therefore decided to use a varying rank threshold, selecting q best classes  from each method as positive . This aWe chose the AUC score to see how well the sorted list of GO classes separates different selected positive and negative classes. These results were also further confirmed with overall rank correlation between the ordered gold standard GO class list and ranked GO list from the evaluated method. Two methods have different aims: AUC with varying rank threshold focuses on most positive GO classes in control set, whereas the rank correlation monitors the overall correlation between the test and control set. Resulting evaluations are based on rank rather than the actual score values obtained for different methods. This was considered to be beneficial as different methods can be expected to follow different distributions, and including the actual score values would therefore put methods in unequal position. Only case where also the actual signal scores were used was the comparison of method with itself using pearson correlation  Method itself is used to define k best GO classes from one half of the dataset. Next the method is run using the other half of the dataset to see how well it can predict its outcome from the correlating dataset. ii) All methods are allowed to select k best GO classes as positive GO classes using one half of the dataset. Next, each method is run on the other half of the data and it is tested how well they can predict the combined positive outcome of all the methods from the correlating dataset. Results from both cases were analyzed using AUC with varying rank threshold and also with standard correlations.The evaluation was done in two ways: i), we use each method itself to define gold standard from the separate data, selecting the method that is most robust to the variances between the biologically replicated datasets. Furthermore, as each method is tested against its own results, the occurrence or lack of correlation with other scoring functions does not affect the evaluation. However, this could favour methods that have biased selection, like a preference of large classes. Neither does it penalize methods that miss GO classes that other methods are able to find. In case (ii), the combined results of all the methods from the separate data represents the same gold standard for all methods. Here method has to be robust to variances between the datasets, and also be able to predict combined positive classes coming from all the methods.These evaluations have different evaluation principles. In case (ii), but they stabilize at larger ranks, especially when rank threshold is between 10 and 20. This can be explained by the increasing size of the positive class set, used in the AUC scoring, which is potentially less sensitive to variations and errors in its definition.These analysis steps were done with four splits of the dataset (8 replicates). Results, averaged over all the replications, are shown in fig. i) GSZ-score shows best performance, with an even performance to t-test at the very first ranks. The detailed analysis of differences in replicates  GSZ-score shows the best performance, and iGA is the close second best method. Their performances overlap in replicates across the top 10 ranks and after rank 20 GSZ-score consistently tops iGA in all replicates  it slightly surpassed t-test but is clearly better than iGA and in case (ii) it slightly surpassed iGA but was obviously better than t-test. Furthermore, it consistently topped modKS in case (ii) and clearly showed better overall performance in case (i). It was also observed that KS test shows the undoubtedly weakest performance in both tests. For a novel method, like GSZ-score, it is also interesting to see which methods are most similar to it. This was analyzed with the whole ALL dataset, using rank correlation. Results are shown in table The overall performance of GSZ-score was best in these evaluations, as in case  results correlate strongly, and we further show in supplementary text S1 [see additional file The actual p-value signals are represented as two separate results in the fig. Fig. p-value = 0 (6 classes), but also KS produces 5 classes. In addition, GSZ drops to the 2nd best after iGA around rank 10. This data proposes that GSZ is the best function across top ranks but it drops down later. These weaker ranks probably represent more random background signal.ALL dataset represents very strong biological signals. However, it is equally important to evaluate the scoring functions with expression data with weaker signal levels. Therefore, we replicated the p-value comparison with p53 dataset  and reprWe also evaluate the biological relevance of the reported classes. This monitors the ability of scoring functions to report the relevant biological classes. We generated a sorted list of GO classes from each scoring function using the p-values from pooled data and class data in combination . As p53 data represents a small number of positive classes, we represent its results here in table P53 dataset compares cancer cell lines with a mutation in p53 transcription factor to the ones without mutations. Here the classes with link to programmed cell death (apoptosis) were considered as positive. Selected classes were further confirmed by looking their regulatory relationship with p53 from the GeneGo Inc. MetaCoreTM database  for each GO class between GSZ-score and each of the other methods. This allows the analysis of the actual GO classes representing the strongest separation between the compared methods. Here, again, we monitor whether these classes are relevant to research setup and also the difference in the log.The outline of this analysis is represented in the supplementary text S1 [see additional file Also p-value differences are much smaller. There are also some interesting details in the comparisons:When comparing GSZ with T-test we observe large number of relevant GO classes with simultaneous up and down-regulation  in ALL dataset. These were naturally missed by T-test, whereas GSZ was able to report them. Furthermore, we observed surprisingly weak performance of T-test on p53 dataset.The comparison with iGA showed many classes from ALL dataset that had a strong regulation. GSZ was able to report very strong p-values for them, whereas iGA did not notice them since it discards the expression values. This underlines that inclusion of differential gene expression scores benefits the test statistic. GSZ clearly reported from p53 dataset better p-values for 3 apoptosis classes.On ALL dataset modKS shows very strange behaviour, and it is clearly outperformed by GSZ. Our assumption is that weak performance of modKS is related to its sensitivity to outlier expression values in GO classes. This outlier sensitivity is further demonstrated by supplementary figures S1 and S2 [see additional file Standard KS showed weakest performance in these comparisons reporting mostly quite random GO classes.These results bring forth GSZ as an outstanding method, when evaluating the biological relevance of the reported classes. There was only one exception on this rule in the p53 dataset, where iGA and KS reported stronger signal on ARF protein signalling classes. These apoptosis classes were not reported as linked to p53 according to GeneGO. Also their average regulation was very small.We also compared our analysis pipeline with other actual software packages. This comparison is less clear to interpret, due to a large number of variables between different analysis pipelines. In addition, the programs report p-values post-processed in various ways. Therefore, the evaluation is focused on the order of the reported GO classes omitting the actual p-values. These obtained GO class lists are evaluated using the selected biologically positive classes again. We selected three software packages for comparison. These were Gene Set Enrichment Analysis package , SignalAll compared methods were run with the same set of GO classes, with no maximum size limit and the minimum size limit set to 3. Although these settings are sub-optimal for the biological analysis, they allow a thorough evaluation of the packages against the variations in the GO class size. All the methods were also tested with 1000 randomizations. Both these settings correspond to the parameters used with the GSZ analysis in function comparison.Two methods (GSEA and GSA) are one-sided tests, generating two separate outputs. However, we need a single GO class list for comparison with SP and GSZ-score. This was accomplished by combining the two GSEA output lists and sorting them using the absolute value of Normalized Enrichment Score (NES), used also to sort the result classes in the GSEA. With GSA we did a similar procedure, ordering in the first round with the normalized score  and in the next round using the p-values. However, we also discuss the rankings in the separate lists. The following two chapters represent the results from two datasets.First we compare methods with ALL dataset. We used same biologically positive classes as in the earlier evaluation with the ALL dataset. The obtained results can be seen in the supplementary table S5 [see additional file There are also differences between the top three methods. GSZ, GSA, and SP all have only a few negative classes among top 20 classes. However, after the rank 20 their density drops in GSA and SP results. Indeed, there is only 7 negative classes among top 40 classes in GSZ-score results and one border case, whereas SP results have 17 negative classes and one border case, and GSA has 18 negative. This makes their error rate among these top classes over two times bigger and highlights GSZ again as the best method. A detailed analysis shows that SP misses totally positive classes with large variance signal and small mean signal . Likewise, GSA represents a quite small signal for some classes emphasized by SP and GSZ, like 'MHC class II protein complex' and 'antigen prosessing and presentation' classes.Next, we compared the methods with p53 dataset. Here, again, we select the programmed cell death (apoptosis) related GO classes as positive. Obtained results are shown in table SP reported one apoptosis related class at the rank 8, with empirical p-value = 0. This was the same class rated as first class by modKS, iGA and GSZ. It had actually the first rank in the column randomization of SP, proposing that it is here more useful than the combined ranking from two randomizations. Thus, SP performs quite nicely, but reports only one cell death related class among the top-50 classes, whereas GSZ was able to report 4 of them among top-15 classes. Furthermore, the standard ranking of SP was not optimal here.th and after our reordering with Z-scores its rank is improved to 2nd position. However, when the two GO class lists are pooled, the rank of the apoptosis class is only 6. Therefore, its performance seems to be weaker than GSZ.GSA generates separate lists for up and down regulated classes. Also, the outputs of GSA represent several GO classes with same p-values. Therefore, we generated new ordering for GO classes, separately for up-regulated, down-regulated and combined set of GO classes. One apoptosis related class is seen in the down-regulated set of GO classes with empirical p-value = 0. Its original rank was 5GSEA generates two lists, instead of one. We again combined them into a single list. The apoptosis related classes were in the list of down-regulated classes at the ranks 8, 14 and 24, and in the results of the pooled list at the ranks 9, 21 and 37. However, if we were to use the default GSEA FDR cutoff (0.25), none of these classes would have been reported. Also, all these apoptosis related classes were low in the result ranking. Therefore, the performance of the GSEA seems quite weak again.Altogether, both datasets point out to GSZ as best method for detecting biologically relevant classes. SP and GSA show quite equal second best performance. Furthermore, both datasets pinpoint GSEA as the apparently the weakest method.GSZ reported 4 apoptosis related classes from p53 data: Release of cytochrome c from mitochondria, apoptotic mitochondrial changes, induction of apoptosis by intracellular signal and caspase activation via cytochrome c. What these classes represent is some of the key steps for programmed cell death: intracellular apoptotic signalling to mitochondria, removal of cytochrome c and its later complex formation with caspase. In addition, other methods (iGA and KS) highlight the ARF protein signal transduction. This group plays a relevant role in apoptotic signalling, although it shows here only very weak regulation. This suggests that a combination of two or three scoring functions could actually highlight all the possibly relevant biological signals in the dataset.ALL dataset showed classes representing simultaneous up and down-regulation, such as immunological synapse, positive thymic t cell selection and lymphocyte activation. Key trends here seem to be: Cell surface receptor signalling that activates immune response and developmental steps of T cells in thymus. Both these processes are central to immune processes. GSZ is able to recognize in particular the immunological synapse, a communication method between B cells and T cells. Note that part of the immunological synapse is expressed in T cell and part in B cell and it is complete when two cells interact with each other. These relevant immunological processes could not be observed from this dataset by methods that monitor only the mean expression of the GO class. Here t-test and SP missed them. Only GSZ and iGA performed satisfactorily on these classes. Also GSA from program comparison was able to detect a fraction of these.ALL dataset also represented GO classes with strong mean up or down-regulation. These classes link to T cell receptor complex, antigen processing via MHC class II protein complex, and to MHC class II protein complex. What is paradoxical is that some methods did not give good scores to these classes. Especially iGA and KS performed weakly with these. This was probably due to the limitation of the analysis only to the order of the genes. Only GSZ and t-test in the comparison of scoring functions reported good scores for these classes. Also SP gave very strong signals to these classes. Surprisingly, GSA obtained less strong results on some of these classes (MHC class II protein complex and antigen processing related classes). Altogether, these results point that across different biologically relevant signals GSZ shows better performance compared to the other, competing algorithms and programs.Current work proposes a novel class scoring function for threshold free gene set analysis. The function is a Z-score that is shown to have similarities to hypergeometric function and to correlation  analysis steps, and treat the functions in a consistent manner. This potentially weakens the performance of modKS, for which the original article  reportedMost earlier publications focus only on one of the two potential null permutations. We propose a separate normalization with row and column randomizations of a dataset, and the selection of the less significant outcome  as the result. This represents a pessimistic but also intuitive perspective, where the more likely null model is allowed to explain the observed results. As a drawback, the obtained p-value will naturally be conservatively biased. Nevertheless, our permutation evaluations show that GSZ is stable under row randomizations  that are associated with many genes. Fortunately, the current remapping projects (see for example ) aim to 10>70). Note that iGA represents a variation of the standard threshold based hypergeometric analysis, and these results suggest that one might need similar column randomization also with those methods [Randomizations play critical role in analysis . The usa) classes where T-test fails due to the heterogeneous regulation, b) classes where iGA represents weak signal although the class represents a clear regulation, c) strongly regulated classes where modKS fails totally. P53 dataset gave also similar results. Overall, our work represents one of the most detailed evaluations of GO analysis methods with artificial and real data, and we hope to inspire the field to implement similar detailed and comprehensive evaluations in GO analysis method comparisons.Real data was evaluated even further by generating a 1000 row and column randomizations, and looking at the actual p-values. We focused on the top 100-classes from ALL dataset where GSZ-score clearly represented a larger signal with 250 - 7100 times better signal for strongest class, and consistently over ten times better p-values for top ranks. Obtained results were further analyzed by looking at the biologically positive classes. Analysis showed that GSZ-score results show largest number of biologically relevant classes among their top ranks. Scoring functions were further compared using the pair-wise differences of the GO-class p-values. These differences from ALL dataset highlight The combination of artificial and real data analysis revealed some exciting insights. KS test had the third best performance in the artificial datasets, but with real data it showed the undisputedly weakest performance. This seems to confirm the hypothesis that the signal area where KS shows its best performance in artificial data see fig.  and 2 isk, which corrects for signal when the subset from gene list is small. However, our artificial data evaluation shows set of parameter values that represent equally optimal performance . These are used to add prior variance to calculus. Similar method is regularly used in gene expression data analysis and in various areas of data mining . Yet the performance could potentially be improved with a more exact measure using an estimated cumulative distribution  or Bayes Factor. In addition, the performance might be easily improved by using a larger number of score values from the calculated GSZ-score profile, obtained over the gene list, than just a single maximum value. A separate variation on this theme is the connection to max-mean statistic [The introduced GSZ-score could be further improved. One of its weaknesses is the requirement of the prior variance tatistic , which wWith increasing demand of gene list data analysis for gene expression, the introduced Gene Set Z-score function should represent a significant addition to analyst's tool palette. It represents a novel addition of signal-levels to ranked list analysis and gives reliable results.To calculate the estimates for the mean and variance, we have to note that the analysis situation can be modelled with a composition of two functions. The first one defines the probability for the observed number of class members in the analyzed subset. The second one defines the probability for obtaining the observed sum for the class members. The exact analysis with Bayes factors or using exact cumulative distribution to calculate p-values, would require the definition of a complex distribution with potentially too heavy calculations. Therefore, we prefer to use a Z-score with relatively simple estimates for mean and variance. Mean is simpler to derive from these two, and it is also required in the variance estimate. Thus we represent it here first. We start with the differencepos refers to positive genes (class members) and neg refers to negative (class non-members). We are dealing here with a sum of N values S = ΣnXn selected from the pool of X1, X2,..., XM values, which in turn is a subset of the total dataset (with size of L genes with K positive genes). The N = Npos is the number of positive genes included to our data subset (with M = Nneg + Npos). When the subset is randomly sampled from the total data pool, we do not know the N beforehand, but it can be expected to follow hypergeometric distribution, representing sample (without replacement) of size M from the pool of L genes including K genes classified as positive. Expected values for both terms in the eq. (8) can be defined using conditional probabilitiesHere E(S|N = i) = iE(X) is shown in supplementary text S1 [see additional file E(X) represents the expected value, or mean, of the data subset X1, X2,..., XM. Expected value for eq. 8 can be represented as:which is simply the probability weighted sum of expected values, conditional on the number of class members in the sum. Proof that the E(N), expected value for the test scores in the analyzed subset E(X) and the number of data points in the subset M. In practice we replace the E(X) with empirical mean calculated from the whole subset. This intuitive result is conditional on the selected data subset (M), on the size of the whole data pool (L) and on the number of positive genes in the whole pool (K). Estimate of the variance is somewhat harder to obtain. Here we start with the definition of variance for a single summation (S) in eq. (8).So the expected value is a simple function of expected value for hypergeometric distribution N we can express the expectation E(S2|N) as a sum of variance and the squared expectation.By fixing N values selected from the pool of X1, X2,.. XM values with variance D2(X) . Note that D2(S) = 0, when N = 0, N = M or if Xi = Xj for all values of i and j (resulting to D2(X) = 0). Derivation of D2(S|N), used in eq. 12 is represented in the supplementary text S1 [see additional file The first term in the latter equation represents the variance of the sum of E(N), D2(N)), the mean and the variance for the subset of the data (E(X), D2(X)) and the size of the data subset (M). The result stays same, whether we monitor the variance of sum for the negative or the positive data points. Equation is similar to the well known equation for the variance of the sum of N copies of identically independently distributed variables D2(Siid) = D2(X)E(N) + E(X)2D2(N). Here the difference is caused by various draws of Xi not being independent.This is simply an equation of the variance and the mean of hypergeometric distribution . A closer look reveals that this is simply a multiplication of two constants having no variance at all, no matter what outcome we observe in our test dataset.The equation (13) represented the variance of the first summation in the equation (12). For the whole variance of the eq. (8), we have to multiply eq. (12) with squared constant 2Therefore it does not affect our variance estimate. So the final score for a data subset analysis becomes:k added to the variance:In contrast to KS and modKS, when OGL is divided to two subsets, our score gives a different score for upper and lower subsets. Therefore, with eq. 14 it is required to consider separately the lower and upper end of the list. We took the largest absolute score from these two lists as the final outcome. Another modification to our score function was made due to the observed instabilities with small subset and class sizes. This is simply a prior variance Note that similar procedure is regularly used with t-test in the expression data analysis. Selection of the prior variance will be discussed in the results section.The BioConductor package , probe rALL: Acute Lymphatic Leukaemia; AUC: Area Under receiver-operating characteristics Curve; ECD: Empirical Cumulative Density function; EVD: Extreme Value Distribution function; GO: Gene Ontology; GEVD: Generalized Extreme Value Distribution function; GSA: Gene Set Analysis; GSEA: Gene Set Enrichment Analysis; IBMT: Intensity Based Modified T-test; iGA: iterative Group Analysis; KS: Kolmogorov-Smirnov test; LR: Likelihood Ratio; ML: Maximum Likelihood; modKS: modified Kolmogorov-Smirnov; OGL: Ordered Gene List; RMA: Robust Multi-array Average; SP: Signal Pathway.PT developed, implemented and tested the methods. PO proposed the research topic and contributed to the datasets and visualization. PM contributed additional mathematical proofs. LH supervised this research. PT, PO and LH evaluated results. All contributed to the writing of the manuscript. All authors read and approved the final manuscript.Supplementary text S1. This text shows in detail the stability of the GSZ-score and competing methods with randomized data, as the threshold moves through the gene list. In addition, the stability of the methods is monitored as a function of the GO class size. Text also highlights the false positive signals seen with column randomizations. Furthermore, the performance of the used normalization with randomized datasets is shown, and the normalized scores and the empirical p-values, obtained with row and column randomizations, are compared to each other. The text also represents the detailed comparison of pairwise differences in p-values for different scoring functions. In addition, most of the Methods are introduced here. Mathematical supplement shows the derivation of E(S|N) and D2(S|N), required for the derivation of the GSZ-score.Click here for fileSupplementary text S2. This text represents a mathematical comparison of the GSZ-score with hypergeometric Z-score, with correlation between the differential expression scores and GO classification, with max-mean score (by Efron & Tibshiriani) and with Random Sets scoring function by Newton et al.Click here for fileSupplementary figure S1: Stability of the GSZ-score as the threshold goes through the gene list. Distribution of the GSZ-score values as the threshold is moved along the gene list. Subset is smallest at the left and largest (the whole gene list) at the right end of the plot. Results are calculated using all the 4511 GO classes from diabetes dataset with randomized GO class matrix. Blue lines show seven percentiles  at each position. For comparison, the red line shows minimum and maximum scores from the non-randomized diabetes dataset. Notice the good stability with the regularized GSZ-scores. Figure is discussed more in a more detail in the supplementary text S1 [see additional file Click here for fileSupplementary figure S2: Stability of KS and modKS as the threshold goes through the gene list. Behaviour of the KS test and the modKS test with the same randomized and positive dataset. Lines represent the same percentiles as in the supplementary figure S1 [see additional file Click here for fileSupplementary figure S3: Performance comparison in case (i) with each split analyzed separately. Performance of the methods in each split in case i. Figure represents the AUC score for each evaluated method as the rank limit of the positive GO classes is increased. Note that AUC is calculated here using the whole evaluated GO class list, and it is the size of the used positive GO class set that varies. Methods represented are GSZ-score: blue line with circles, t-test: green line with cross, KS test: red line with box, modKS test: cyan line with diamond, iGA: magenta line with x. Notice that although the signal levels vary between the replicates, the differences between the methods are stable. GSZ-score and t-test show equal performance among the smallest ranks, whereas the GSZ-score is clearly the best among the larger ranks. Other methods show weaker signal. Figure is zoomed to the upper signal area so that most of the curve for the KS is left outside.Click here for fileSupplementary figure S4: Performance comparison in case (ii) with each split analyzed separately. Performance of methods in each split in the case ii. The figure represents the AUC score for each evaluated method as the rank limit of the positive GO classes is increased. Methods are coloured identically to the earlier figure. Here, GSZ-score and iGA show the best performance at very top ranks, with GSZ-score slightly surpassing iGA. Note that performance of t-test and modKS varies considerably across the replicates. After rank 20 GSZ-score is constantly best in all the replicates.Click here for fileSupplementary table S1: Top 100 GO classes reported by each scoring function from p53 dataset. This table presents 100 best scoring classes for each scoring function from p53 dataset. Table shows class rank, two combined empirical log-p-values, mean of expression values of GO class genes, mean of absolute expression values for GO class genes, class size and class name. In addition, we show four different empirical log-p-values and 7 percentiles  for the expression values of the class members. Empirical log-p-values are the ones used to calculate the combined log. Biologically relevant (positive) classes are shown in bold font and border cases are with underlined font. Results from each method is presented as a separate Excel worksheet. Furthermore, the first worksheet represents a summary of the results for each scoring function.Click here for fileSupplementary table S2: Top 100 GO classes reported by each scoring function from ALL dataset. This table presents 100 best scoring classes for each scoring function from ALL dataset. Columns are similar to the columns in supplementary table S1 [see additional file Click here for fileSupplementary table S3: GO classes showing largest pair-wise differences from ALL dataset. This table presents 50 GO classes with the strongest combined log differences in favour, as well as against the GSZ-score, in a pair-wise comparison with other scoring functions. Each comparison with each competing scoring function is on a separate sheet. Columns in the tables are identical with columns in tables supplementary table S1 and S2 [see additional files Click here for fileSupplementary table S4: GO classes showing largest pair-wise differences from p53 dataset. This table presents 50 GO classes with the strongest combined log differences in favour, as well as against the GSZ-score, in a pair-wise comparison with other scoring functions. Each comparison with each competing scoring function is on a separate sheet. Columns in the tables are identical with earlier tables.Click here for fileSupplementary table S5: Top GO classes from GSZ-score and compared program packages from ALL data. This table presents top classes obtained from GSA, SP, GSEA and GSZ-score. Summary of the results is on the first sheet. Separate sheets show each programs results with additional data. Furthermore, we represent two result sheets for scoring functions with one sided test . One presents the one sided test results and the other presents the combined results.Click here for fileSupplementary table S6: Top GO classes from GSZ-score and compared program packages from p53 data. This table presents top classes obtained from GSA, SP, GSEA and GSZ-score. Summary of the results is on the first sheet. Separate sheets show each programs results with additional data. Furthermore, we represent two result sheets for scoring functions with one sided test . One presents the one sided test results and the other presents the combined results.Click here for file


This paper presents a retrospective statistical study on the newly-released data set by the Stanley Neuropathology Consortium on gene expression in bipolar disorder and schizophrenia. This data set contains gene expression data as well as limited demographic and clinical data for each subject. Previous studies using statistical classification or machine learning algorithms have focused on gene expression data only. The present paper investigates if such techniques can benefit from including demographic and clinical data.We compare six classification algorithms: support vector machines (SVMs), nearest shrunken centroids, decision trees, ensemble of voters, naïve Bayes, and nearest neighbor. SVMs outperform the other algorithms. Using expression data only, they yield an area under the ROC curve of 0.92 for bipolar disorder versus control, and 0.91 for schizophrenia versus control. By including demographic and clinical data, classification performance improves to 0.97 and 0.94 respectively.This paper demonstrates that SVMs can distinguish bipolar disorder and schizophrenia from normal control at a very high rate. Moreover, it shows that classification performance improves by including demographic and clinical data. We also found that some variables in this data set, such as alcohol and drug use, are strongly associated to the diseases. These variables may affect gene expression and make it more difficult to identify genes that are directly associated to the diseases. Stratification can correct for such variables, but we show that this reduces the power of the statistical methods. The Stanley Neuropathology Consortium  recentlyQ1. Can either bipolar disorder or schizophrenia be distinguished from control purely on the basis of gene expression profile?Q2. Does addition of the demographic and clinical history data further improve the ability to distinguish bipolar disorder or schizophrenia from control?Q3. Is there a significant difference between the abilities of different widely-used data analysis algorithms to make these distinctions?We show that bipolar disorder and schizophrenia each can be distinguished from control, based on gene expression alone, significantly better than chance – in fact with areas under the Receiver Operating Characteristic (ROC) curve (AUC) of 0.91 (schizophrenia vs. control) and 0.92 (bipolar disorder vs. control). While area under the ROC curve indicates how well one can distinguish across a range of specificities (with 0.5 being no better than chance and 1.0 being perfect distinction), it is also worth noting that for each task, a sensitivity of 0.85 can be achieved when operating at a specificity of 0.9. Moreover, by taking demographic information and clinical history into account  We have investigated if post-stratification can correct for such variables, but we found that it significantly reduces the predictive accuracy of the statistical methods.The expression data set was obtained from the Stanley Neuropathology Consortium . The recThe expression data was obtained using Affymetrix Human Genome U133A GeneChip oligonucleotide arrays containing 22,283 probe sets . Probe level data was summarized using the GC content adjusted robust multi-array average (RMA) method , MT1X [GenBank:NM_002450], MT1X [GenBank:NM_005952], TNFSF10 [GenBank:NM_003810], ABCG2 [GenBank:AF098951], MT1E [GenBank:BF217861], SST [GenBank:NM_001048], CRHBP [GenBank:NM_001882], EMX2 [GenBank:AI478455], NPY [GenBank:NM_000905], MT2A [GenBank:NM_005953], MT1H/P2 [GenBank:NM_005951], SOX9 [GenBank:NM_000346], S100A8 [GenBank:NM_002964].When comparing the features that appear in the different tables, we observe that for the schizophrenia versus control task (Table NM_007177]) in common. For the bipolar versus control task , and all subjects and female subjects also share 6 features . For both diseases, there is no overlap between the ranking for the female subjects and that for the male subjects. Possibly of higher interest are the features relevant to both the schizophrenia versus control and bipolar versus control tasks. Comparing the rankings . Since, these samples are not derived from controlled animals models we will have to rely on these analytical means to aid our efforts in dissecting the root causes of complex diseases.Because most classification techniques that we consider are restricted to numerical features only, we re-encode each nominal feature as a numeric feature. Table . We use the reimplementation of C4.5 that is available in the Weka data mining tool version 3.4.4 [kNN. We use the EOV software version 1.0 by Hardin et al. [q-values, we use the software QVALUE version 1.1 developed by Storey [We choose the SVM-light software version 6.01  for conson 3.4.4 . We alson et al. . To compy Storey . All areMost classifiers provide confidence scores for their predictions. The classification behavior of such classifiers can be modified by applying a threshold to this score: only predict positive if the confidence is above the given threshold. By varying the threshold, we obtain different ROC points, which can be connected into a curve.  We present such a curve for each classifier and report the corresponding AUC.D into 10 subsets Ti; (b) train 10 classifiers on the training sets D-Ti; (c) test classifier i on test set Ti. We pool the predicted confidence values of the classifiers over the 10 test sets to construct the ROC curve. The CV algorithm that we employ is stratified, which means that it ensures that the Ti have identical class distributions, or as nearly identical as possible.To obtain the ROC curves, we use 10-fold cross-validation (CV). 10-fold CV is often used to evaluate the predictive performance of classifiers if the number of instances is small. In this situation, 10-fold CV results in a lower-variance estimate of error than does the use of a single held-aside test set. 10-fold CV consists of three steps: (a) partition the data set t-test. The paired sample values used in the test are the AUC values computed for the two techniques on the 10 CV test sets.To assess statistical significance when comparing two classification techniques by AUC value, we use a two-sided paired C of SVM-light, which controls the contribution of the misclassified examples, is set to 1.0 . We enable Laplace smoothing of DT's confidence values. Following Hardin et al. [N to 20. We enable Weka's discretization feature for NB. We run kNN with k = 3 neighbors.Most classification techniques come with a number of parameters. We set all parameters to their default values, except for the following. The parameter n et al. , we set We tune NSC's Δ parameter, which controls the amount of shrinkage, by means of 10-fold CV, as suggested by Tibshirani et al. . Recall t-test for each feature comparing its value in the two classes. Then we rank the features by their t-test's p-value and retain the 10% features that most significantly differ in the two classes . We repeat the feature selection for each CV fold and perform the t-test on the corresponding training set.The performance of some classification techniques can be improved by running a feature selection method prior to constructing the classifier. We implement feature selection for SVM, NB, and 3NN. The feature selection works as follows. We perform a two-sided paired To compute the feature ranking by SVM weight, we normalize each feature by subtracting its mean and dividing by its standard deviation (in the data set at hand), and run the SVM algorithm on the transformed data. The rationale behind this is to avoid favoring features with a small value range in the ranking. We also enable feature selection to construct the ranking by SVM weight as discussed before.JS performed the statistical analysis and drafted the manuscript. SD provided the data, provided critical input and drafted the biological relevance section of the manuscript. DP supervised the study and helped drafting the manuscript. All authors read and approved the final manuscript.• FeatureRankingsStratified.{doc, pdf} (Microsoft Word and PDF format): "Feature rankings on post-stratified data" in additional files Feature rankings similar to Tables • FeatureRankingsDetailed.xls (Microsoft Excel format): "Feature rankings with additional information" in additional file Feature rankings that contain the same information as Tables Feature rankings on post-stratified dataClick here for fileFeature rankings on post-stratified dataClick here for fileFeature rankings with additional informationClick here for file


Expression microarrays are increasingly used to obtain large scale transcriptomic information on a wide range of biological samples. Nevertheless, there is still much debate on the best ways to process data, to design experiments and analyse the output. Furthermore, many of the more sophisticated mathematical approaches to data analysis in the literature remain inaccessible to much of the biological research community. In this study we examine ways of extracting and analysing a large data set obtained using the Agilent long oligonucleotide transcriptomics platform, applied to a set of human macrophage and dendritic cell samples.2 units ( 6% of mean). The common practise of working with ratios of Cy5/Cy3 signal offers little further improvement in terms of reducing error. Comparison to expression data obtained using Arabidopsis samples demonstrates that the large number of genes in each sample showing a low level of transcription reflect the real complexity of the cellular transcriptome. Multidimensional scaling is used to show that the processed data identifies an underlying structure which reflect some of the key biological variables which define the data set. This structure is robust, allowing reliable comparison of samples collected over a number of years and collected by a variety of operators.We describe and validate a series of data extraction, transformation and normalisation steps which are implemented via a new R function. Analysis of replicate normalised reference data demonstrate that intrarray variability is small , while interarray variability from replicate array measurements has a standard deviation (SD) of around 0.5 logThis study outlines a robust and easily implemented pipeline for extracting, transforming normalising and visualising transcriptomic array data from Agilent expression platform. The analysis is used to obtain quantitative estimates of the SD arising from experimental  intra- and interarray variability, and for a lower threshold for determining whether an individual gene is expressed. The study provides a reliable basis for further more extensive studies of the systems biology of eukaryotic cells. The application of microarray technology to study the expression of thousands of genes in biological samples has become commonplace. The diversity of technologies initially explored has been replaced by a landscape dominated by a small number of proprietary platforms, which differ principally in the type of probe used for hybridisation. Each platform has advantages and disadvantages, and knowledge of technical reproducibility and sources of data variability is crucial to optimising experimental design and data analysis. The reproducibility and comparability of several different platforms has been rigorously examined in the MAQC project, and overall the different platforms give reassuringly similar results, with similar accuracy and sensitivities .We have collected a large number of array data sets from Agilent human genome arrays . The latAlthough there are now many published studies using the Agilent arrays, there is a paucity of studies systematically investigating the reproducibility and sources of variation in this microarray platform. There is also relatively little software for implementation of data preprocessing for this platform. Using a set of data generated from this array platform, we dissect the major contributors to experimental variability within the data. We develop a new R package (agilp) to extract data from Agilent raw data files, and implement a robust Loess normalisation to a mean of multiple experiments. This reduces variability to a level which allows accurate and informative comparisons between array data sets collected at different times and by different operators. We also demonstrate that the low level expression detected by the Agilent arrays for the majority of genes in any one cell type likely corresponds to a genuine high degree of transcriptomic complexity, and is unlikely to arise from weak non specific cross-hybridisation. The results of the study thus give confidence to studies which use this technology and quantify important performance parameters which can be used to optimise the design and interpretation of future transcriptomic experiments.We initially examined a set of 77 data sets  independent of magnitude, and improve the fit to a Gaussian normal distribution. Both these assumptions were therefore tested on the set of reference arrays.2 transformed data is shown in fig 2 = 0.87). Log2 transformation substantially stabilises the SD (fig 2 = 0.22).The relationship between SD and mean magnitude of signal across the arrays for each array feature, using raw and log0.55)fig . Further2 transformed data were analysed. Log2 transformed data were therefore used for all further analysis.The normality of the distribution of the signal across the arrays was tested using the Shapiro Wilk test for each probe individually. The fit varied widely for different probes. However, the proportion of probes whose distribution did not differ significantly from normality (p > 0.05) increased from 55% to 78% when Log2 units for the Cy5 signal, and 0.16 log2 units for the Cy3 signal  provides a good measure of intra-array variation. This variation can be evaluated using 263 oligonucleotides which are replicated 10 times in different positions of the 4 × 4 Human genome Agilent arrays, and a "negative control" oligonucleotide (3 × SLv1) which is present 153 times. The average SD for each set of 10 replicates for each of these 263 probes (including the negative control) was calculated. The calculation was repeated for data from five randomly chosen arrays. The average SD was 0.20 logWe next examined the variation which arises from experiments involving different arrays. In order to isolate the variation arising from technical variables  from variation arising from differences in the biological samples being tested, we first analysed variation in the Cy3 signal, which represents hybridisation of an identical biological sample 77 times Table .In order to reduce variability by correcting for systematic differences between arrays we normalised the data by simple linear or LOESS local linear regression against the mean signal of all the samples (i.e. a model in which the mean and the slope is held constant).2 = 0.93). The normalisation of this array by subtraction of the regression model provided appropriate correction of the signal values , and a local linear regression model (LOESS ) gives a better overall fit  across the whole set of arrays. Using the normalised data, we calculated the average SD for each probe (the genewise SD) across the set of arrays , the stimulation index in the green channel is positively correlated to the stimulation index observed in the red channel. This effect is only seen at high stimulation indices, and the correlation is lost once the difference in signal is less than 3 logThe phenomenon is illustrated in fig One feature of Agilent arrays is a very large number of probes have a low signal value, close to, but significantly above background. This is illustrated by a comparison between the signal distribution obtained from one representative sample compared to the distribution of the signal from the "negative control" probe provided by Agilent fig .These data might support a hypothesis that a large number of genes are being transcribed in any one cell type. However, the conclusion is reliant on the the negative control supplied by Agilent. We therefore tested the hypothesis independently. In view of the very large evolutionary distance between Arabidopsis and humans, and the process of Agilent selection of probe sequences so as to avoid repetitive elements or highly conserved common sequences, we predicted that the plant RNA samples should show very little true hybridisation to the arrays. In contrast, if the low level signals were as a result of random low stringency interactions, this should occur to a similar extent using the plant as the human samples. The log signal frequency distribution for the average of three Arabidopsis samples  was compared to the signal distribution for human samples fig . The majThe data from the Arabidospis samples could be used to calculate an upper limit for "false positive" detection, and hence estimate the proportion of probes showing real positive signals with the human samples fig . In orde2 transformed and Loess normalised as detailed above were analysed using classical multidimensional scaling (MDS), a powerful unsupervised approach to identify relationships between sets of arrays. Since the distance matrices are based on Euclidean distance metrics this analysis is mathematically equivalent to Principal Component Analysis.Having established the basic parameters of error/variability, we examined whether global patterns of gene expression were reproducible and robust when comparing data collected over a time period of several years, and by different operators. The same data set of arrays logk for different numbers of dimensions  to look for an obvious "step" in the slope fig . There aions fig :(2)i is the ith eigenvalue. A value of around 0.8 (at which around 80% of the SD has been accounted for) has been proposed as a good cut-off for adequately representing the original data . Ten dWe first examined the distribution of arrays from the point of view of cell type fig . The ranWe next examined the distribution in relation to stimulus fig . The denWe carried out two additional sets of hybridisations to test the reproducibility of the arrays over time. Two old samples of macrophage RNA (LPS stimulated ) were relabelled in Cy3, and hybridised to new arrays together with old stored Cy5 labelled samples . The old and new arrays cocluster, irrespective of dye labelling or long term storage fig . In partIn addition, we analysed a set of 12 newly prepared Mutz-3 derived dendritic cell arrays fig  includinIn this study, we have developed a pipeline for processing and analysis of data obtained using the Agilent human 44 K expression array platform. We have assessed commonly adopted preprocessing computational procedures, tested reproducibility of the method and the determinants of variability. Interestingly, investigation of the true background signal intensity, suggested that there is low-level transcriptional activity across most of the genome even in highly differentiated macrophages and dendritic cells. Finally, we have illustrated the application of multidimensional scaling to interrogate complex relationships between individual expression profiles in a large data set.2 transformation did not completely standardise variance vis a vis intensity, and in situations where this becomes critical, additional more complex alternative transformations can easily be incorporated e.g. [We confirmed that logarithmic transformation of the raw data facilitates downstream data processing by significantly reducing the dependence of the SD on the signal intensity, and by improving approximation to a normal distribution for many probe signals. Logted e.g. .A large number of different approaches to data normalisation have been explored in order to improve the reproducibility of array data, particularly with the objective of comparing data generated from different experiments -12 . ManThe use of two colour ratio normalisation, which is possible with the Agilent platform but not Affymetrix, is in fact implemented in the proprietary normalisation supplied by Agilent. The advantage of this normalisation, over global array normalisations, is that it can theoretically correct for probe specific systematic errors. However, as shown in equation 1, ratios decrease variance only when correlation between channels is high, and noise in the "reference" channel is small. Previous studies have in fact suggested that the Agilent normalisation protocol is sub-optimal. In fact, as illustrated in Table 2 unit should be detectable even with rather low sample sizes. However, future studies using spike-in controls, and including comparisons to q-PCR data will be required to validate the analysis given here, and to determine the true discriminatory power of the data. More precise modelling, using probe specific variances computed from the large set of reference arrays, or more sophisticated Bayesian inference [In summary, the preprocessing steps implemented in this study significantly reduce the variation due to experimental, rather than biological variation. This value provides a good estimate of the overall SD of the signal attributable to the various sources of intrinsic "experimental error". Given the Gaussian nature of log signal variation for most probes, differences between experiments of one lognference may furthThe presence of a very large number of probes showing low level hybridisation prompted us to try and distinguish background from low level gene transcription. Hybridisation of the Arabidopsis samples established a realistic background cut-off to determine the boundary at which genes may be considered to be ON or OFF, and showed transcriptional activity for approximately 70% within human macrophage samples. This degree of transcriptional diversity suggests transcription is going on across the majority of the genome even in a single cell type, and is in accord with recent estimates of cell transcriptional complexity obtained from deep sequencing ,16, and In general, the power of genome-wide arrays lies in its ability to define a global transcriptomic response, made up of complex patterns of interacting genes. In order to analyse the extent to which these patterns remained reproducible and robust in the face of experimental error and variation, we used MDS which is equivalent to principal component analysis for continuous Euclidean distance measurements and retains more of the information than conventional hierarchical clustering.MDS analysis of the macrophage/dendritic cell data set  confirmed that the key underlying structure of the data set is robust, and that sample relationships are conserved even when samples are collected at different times, and by different operators. For example, the distinction between macrophages and dendritic cells, between normal and leukaemic dendritic cells, and between resting and LPS stimulation were all very robust to underlying experimental variation. Of interest, the biologically meaningful variables such as those described above, usually segregated from the "artifactual" variables  into a different dimension. Since the MDS axes are orthogonal this suggests that the biological and artifactual variables are often determined by non-overlapping sets of gene probes, and the identification of these genes is the subject of on-going analysis.Agilent long oligonucleotide expression arrays represent a robust and increasingly affordable tool for mapping transcriptional activity within the majority of human open reading frames. With correct data processing and normalisation, the genewise variability can be reduced to a few % of the average signal intensity. The pipeline of protocols for data collection, processing and subsequent analysis developed in this study fig  allows rSample collection. The methods for human cell sample collection, array hybridisation and sample collection have all been described previously ,19. A suThe reference sample is the Stratagene Universal Human Reference reagent, which consists of a mixture of RNA from ten different human cell lines .Seeds of Arabidopsis thaliana (L.) Heynh. were obtained from the Nottingham Arabidopsis Stock Centre. Plants were grown on 0.8% agar with 0.25 × MS salts, adjusted to pH 5.6 with NaOH, under conditions described previously . At 19 dLabelled cRNA samples were generated from RNA samples using the Low RNA Input Fluorescent Linear Amplification Kit according to the manufacturer's instructions . cRNA was synthesised from the double-stranded cDNA using T7 RNA polymerase, incorporating Cyanine 5-labelled CTP fluorescent dyes . Labelled cRNA samples were cleaned using the 'RNA Cleanup' protocol for RNeasy columns (Qiagen) performed at 4°C and eluted using two 30 μL volumes of nuclease-free H2O. Mean dye incorporation for labelled cRNA was 19.53  pmol dye μg-1 cRNA.http://www.agilent.com. Array images were acquired with Agilent's dual-laser microarray scanner G2565BA (5 μ resolution) and signal data were collected with dedicated Agilent Feature Extraction software (v9.5.1).Hybridisation of the Agilent 4 × 44 K whole human genome cDNA microarrays according to manufacturer's instructions http://www.ebi.ac.uk/arrayexpress, accession number E-TABM-942). A summary of the samples is given in supplementary Table S1.The original scanner output files and the normalized microarray data (see below) have been submitted to the ArrayExpress database http://cran.r-project.org/web/packages/.The R code for the extraction of raw data from scanner files, removal of duplicates, and Loess normalisation, is available (as function "AAProcess" in package agilp) from the R CRAN repository. http://bioinformatics.mdanderson.org/Software/OOMPA/, and the function cmdscale from the stats package.The statistical analysis was mostly carried out in R. The functions Shapiro.test and ks.test were used to test for normality. Multidimensional scaling was carried out using the functions distanceMatrix (using Euclidean distance) from the package ClassDiscovery, part of the OOMPA suite of libraries HB and JH provided the Arabidopsis labelled RNA samples, and contributed to revision of the manuscript. JR, JT and WP prepared all the other RNA samples, and carried out all the hybridisations. MN supervised the RNA preparation, labelling and hybridisation. BC carried out the data analysis, wrote the R scripts, and prepared the first draft of the manuscript. The study was carried out under the joint direction of BC and MN. All authors read and approved the final version of the manuscript.Table S1: The details of all the arrays analysed in this study. An Excel file containing the basic experimental annotation for each array analysed, together with a data base identifier.Click here for file


Complex diseases induce perturbations to interaction and regulation networks in living systems, resulting in dynamic equilibrium states that differ for different diseases and also normal states. Thus identifying gene expression patterns corresponding to different equilibrium states is of great benefit to the diagnosis and treatment of complex diseases. However, it remains a major challenge to deal with the high dimensionality and small size of available complex disease gene expression datasets currently used for discovering gene expression patterns.Here we present a phase-only correlation (POC) based classification method for recognizing the type of complex diseases. First, a virtual sample template is constructed for each subclass by averaging all samples of each subclass in a training dataset. Then the label of a test sample is determined by measuring the similarity between the test sample and each template. This novel method can detect the similarity of overall patterns emerged from the differentially expressed genes or proteins while ignoring small mismatches.The experimental results obtained on seven publicly available complex disease datasets including microarray and protein array data demonstrate that the proposed POC-based disease classification method is effective and robust for diagnosing complex diseases with regard to the number of initially selected features, and its recognition accuracy is better than or comparable to other state-of-the-art machine learning methods. In addition, the proposed method does not require parameter tuning and data scaling, which can effectively reduce the occurrence of over-fitting and bias. Classification and diagnostic prediction of complex diseases such as cancers and neuron-degeneration diseases using genomic or proteomic data can improve the quality of pathological diagnosis and help develop personalized treatment of these diseases . Althougk-nearest neighbor (KNN), support vector machines (SVM) [The rationale of classification and diagnostic prediction of complex diseases using genomic or proteomic data is based on the assumption that complex diseases induce perturbations to interaction and regulation networks of living systems, resulting in dynamic equilibrium states that differ for different diseases and also normal states. Thus identifying gene expression patterns corresponding to different equilibrium states is a key task to the success of these types of approaches. Many pattern recognition methods based on machine learning, such as es (SVM) -7, probaes (SVM) -10, naives (SVM)  and randes (SVM) ,12, etc.es (SVM) . UsuallyAn example of feature extraction methods is that independent component analysis (ICA) was used to extract independent components from GEP to reduce the dimensionality of sample ,14,15. OFor potential clinical applications, a candidate classification model should be evaluated for three aspects: accuracy, interpretability and practicality . And a nPrevious studies suggest that each complex disease type or subtype corresponds to a dynamic equilibrium state of disease-induced genomic interaction and regulation network, and different samples at the same state are similar in gene expression profiles . Thus anOur further experiments indicate that phase-only correlation (POC)  may be shttp://www.ncbi.nlm.nih.gov) [http://www.ncbi.nlm.nih.gov) [Seven public available complex disease datasets are used to evaluate the proposed method in our experiments. They include the Leukemia1 , GSE2967nih.gov) ,4, Leukenih.gov) , small rnih.gov) , GSE5281nih.gov) , colon tnih.gov) , and GCMnih.gov)  datasetsN genes, and k-th subclass set Both protein and DNA microarray data can be represented with matrices. Thus we use DNA microarray data as an example to describe the design of our method. Let POC allows us evaluate the similarity of disease samples in frequency domain based on their GEPs. Figure 1) The entire sample set is randomly split into two disjoint parts: a training set and a test set. We then select a certain number of DEGs or differentially expressed proteins (DEPs) using the Kruskal-Wallis rank sum test (KWRST) method .j-th component of virtual sample template for subclass k-th subclass in training set for feature 2) A virtual sample template for each subclass of training set is constructed by averaging all samples in the subclass. The posed in .3) The POC function is calculated between each virtual sample template and a test sample, both of which are already transformed using the one-dimensional discrete Fourier transform (1D DFT). The similarity between each virtual sample template and a given test sample is evaluated using the peak value of POC. The formalized representation of a test sample matching with all where 4) The highest peak of POC can be utilized to determine the label of the test sample If we adopt a square matrix to represent a sample instead of the vector form of the sample, we can analyze a sample set using two-dimensional POC (2D POC) to identify disease types. The flowchart of 2D POC analysis method is very similar to the 1D POC shown in Figure We adopt both 1D POC and 2D POC methods to analyze disease samples. Here we only give the mathematical description of 1D POC. The principle of 2D POC can be found in literature . Given twhere where  pattern . Thus thwhere increase . Thus hiIn contrast to the template-based POC method, we design a POC1DKNN method that utilizes 1D POC to measure the similarity of two samples and apply 5-nearest neighbor (5NN) to predict the label of test sample.Although there is no parameter in the proposed method, the different number of pre-selected features and the different divisions of training sets and test sets can also affect the classification performance. To obtain objective results, the Balance Division Method (BDM) is used to divide each original dataset into balanced training sets and test sets . For theThe results of 1D POC and 2D POC can be visually represented. Taking the SRBCT dataset as an example, 1D cross-correlation coefficients calculated by a test sample (belonging to EWS subtype) matching with the four templates corresponding to the four subtypes of the SRBCT dataset are shown in Figure The resulting separability of all test samples (belonging to the same subclass) matching with each template can be visualized using plots, from which we can visually determine which test sample is correctly or mistakenly classified. For example, Figure Due to the fact that the OTSDF is a method with one parameter and the performance of OTSDF and MACE are almost equivalent, for fairer comparison we do not compare POC with OTSDF in performance, while we only compare the performance of POC with the one of MACE. Like POC, MACE is also a nonparametric method that has shown good performance on recognizing tumor subtypes . HoweverSince the template-based POC method can be used to build classifiers, we compare it with other state-of-the-art model-based classification algorithms including NBM, KNN, PNN, and SVM. For KNN, we set its where For PNN, there is a smoothing parameter Because the performance of a model is sensitive to data division into training and test sets, we repeat the procedure 200 times using randomly divided training and test sets and report the mean value of the 200 predictive accuracies for each method Figure . Figure t-test instead of KWRST, and the experimental results indicate that different feature filters affect less the performance of the POC-based method.We then fix the number of training samples per subclass to 8 but use different number of selected features and study the performance of models for each dataset Figure . The resk-nearest neighbor (KNN) with correlation distance to classify disease samples, and fixedly set its Due to the high dimensionality of dataset, feature extraction is often used to reduce the dimensionality of dataset before classification, and it plays a crucial role in simplifying classification model and improving the classification performance. Here we compare our method with five dimensionality reduction methods, i.e., PCA, LDA, ICA, LLDE, and LPP, which are extensively applied into the classification of complex disease. Our previous study suggests that the prediction accuracy depends less on classification methods  when theTo avoid over-fitting, before classification we extract only 5 features adopting these feature extraction methods except LDA whose number extracted is p-values [To further assess the reliability of the proposed method, we calculate the label permutation-based p-values  of the sp-values are very small. The mean value K is the number of subclasses in dataset, indicating that no bias occurred in the obtained results [where  results .Data scaling or normalization is a very important data pre-processing step for many machine learning algorithms sensitive to the numeric ranges of attributes. There are several widely used scaling methods such as Z-score that transforms data into the one with zero-mean and one-variance, and 0-1 scaling method that transforms data into the range between 0 and 1, etc. Currently it is difficult to predict what is the best data scaling method for a given dataset , and no In the present study, we construct the template of each subtype using the means of the data points in the training dataset. The results demonstrate that this approach is reasonable and good performance is achieved. Nevertheless, there are certainly other ways to construct templates. For example, medians, instead of means, are another possible approach that might be more suitable for data that are not normally distributed. For the present study, we test medians but do not find significant difference from means (data not shown). Thus only the results using means are reported.A POC-based method is reported as a new technique for identifying similar gene expression signatures for the differentially expressed genes or proteins. By measuring the similarity between a test sample and the virtual sample templates constructed on training set for each subclass, the label of the test sample can be easily determined. Applying the POC-based classification method to six complex disease datasets shows that this novel method is feasible, efficient and robust. Compared with five state-of-the-art classification algorithms and five feature extraction-based methods, the proposed method can achieve optimal or near-optimal classification accuracy.Our methods can detect the similarity of overall pattern while ignoring small mismatches between a giving test sample and templates because correlation filters are based on integration operation. Compared with the MACE and OTSDF methods, POC is not sensitive to data scaling methods. The experimental results show that the POC-based method can achieve satisfactory results even without scaling data. Moreover, there is no parameter to be tuned in POC, so this method can easily avoid the over-fitting problem as well as the effects of dimensionality curse. One possible drawback of this novel method is that high-level noise in the template can suppress the output peak. Our future work will focus on exploring novel method to construct more representative template to further improve predictive accuracy.The authors declare that they have no competing interests.All authors contributed to the design of the project, the interpretation of the results, and the drafting and production of the manuscript.