We assessed NanoString's nCounter™ Analysis System for its ability to quantify gene expression of forty-eight genes in a single reaction with 100 ng of total RNA or an equivalent amount of tissue lysate. In the nCounter™ System, multiplexed gene expression target levels are directly detected, without enzymatic reactions, via two sequence-specific probes. The individual mRNA is captured with one mRNA target sequence-specific capture probe that is used in a post-hybridization affinity purification procedure. The second mRNA target specific-sequence and fluorescent-labeled colored coded probe is then used in the detection with the 3-component complex separated on a surface via an applied electric field followed by imaging. We evaluated reproducibility, accuracy, concordance with quantitative RT-PCR, linearity, dynamic range, and the ability of the system to assay different inputs  and Formalin Fixed Paraffin Embedded Tissues (FFPET), and crude tissue lysates (CTL)).®-based assays for genes expressed within the ranges of the calibration curves . System response was linear over more than two orders of magnitude with typical CVs of ~6% for concentrations above 1 fM (105 molecules per mL). Profiling the industry-standard MAQC data set yielded correlation coefficients of >0.83 for intensity values and >0.99 for measured ratios. Ninety percent of nCounter™ ratio measurements were within 1.27–1.33 fold changes of the Taqman® data  for FF total RNA samples.The nCounter™ Analysis System provided data equivalent to that produced by TaqmanThe nCounter™ Analysis System generated robust data for multi-gene expression signatures across three different sample preparation conditions. Analyses of gene expression from microarrays can be used to define a specific set of sequences (signatures) relevant to a particular biological phenomenon or response . These sAs part of an internally funded investigation, Merck scientists tested the ability of the nCounter™ Analysis System  to meet , Two-Color Microarray Spike-In Kit, part # 5188–5279, [The gene list for nCounter™ probes consisted of 14 human genes differentially regulated in our xenograft system, 25 human genes differentially regulated among the MAQC samples, and 9 sequences corresponding to synthetic transcripts used historically at Rosetta Inpharmatics as spike-ins for quality control . Total RNA was purified from the remaining lysate using a Trizol-based protocol . We isolated RNA from FFPE EBC1 xenografts using the Ambion RecoverAll protocol . In all samples, one or the other set of Rosetta spike-ins were added to provide a measure of ratio accuracy. Samples were blinded before assaying at Nanostring Technologies, where the samples were processed to generate raw data (i.e. counts/gene). nCounter™ assay spike-ins control mixes #3 or #4 were added at random to each of the blinded RNA samples on the day the assay was performed.Samples  and Taqman® Universal PCR Master Mix (part # 4364340). Taqman reporter probes were used  were used to create the calibration curve for each nCounter™ array. Nine negative control spike-ins were used to assess the level of background . Mean of the negative controls was deducted from all other transcripts in the same assay prior to logarithmic transformation (log base 2). We used a standard linear regression model to find the least square fit of logarithm-transformed concentration on the logarithm-transformed number of molecules above background to generate the equation for the rest of the transcripts in the same assay. Each nCounter™ assay result was converted to an equivalent concentration using the assay standard curve. Use of the standard curve allows absolute measurements to be assigned to nCounter™ counts as needed.2 scale, resulting in a correction of approximately 1.08 fold. To achieve specified precision, NanoString recommends running each sample (by experiment) in triplicate. To mimic a typical experiment, therefore, we averaged triplicate assays for Rosetta spike-ins as well. Standard deviations of resulting mean values were used to calculate CVs.To deduce the precision of the nCounter™ assay itself, we mean centered the data in log® and nCounter™ data were normalized to CUGBP1 as a reference gene for the xenograft samples. Originally four reference genes were identified from previous experiments as not varying significantly across our experimental conditions and were planned to be used as references in aggregate. However, three of the four did not reliably give signals above the lower limit of our standard curve and so were not used. This led us to deviate from generally accepted practice in which more than one reference gene is used to normalize data.To generate across-multiple-samples, gene-by-gene equivalency plots, both Taqman® data were normalized to POLR2A, the nCounter™ normalization did not utilize any reference genes. This is because POLR2A was not one of the genes present in our genelist and so was unavailable as a common control. As a result of this approach, Taqman® data were normalized to mRNA amount, while nCounter™ normalization relied on the same amount of total RNA (100 ng) in each sample. This distinction is important because the MAQC study showed that UHR has 1.5-fold higher mRNA content than Brain (3% vs. 2%). To compensate for different mRNA content, 0.585 Ct, 0.46 Ct and 0.17 Ct were deducted from all genes of 0% Brain/100% UHR; 25% Brain/75% UHR; and 75% Brian/25% UHR samples, respectively.In the MAQC analysis, although the published Taqman®, using their mean Ct.For performance evaluations, a comparison was done for each possible pair of samples because we did not wish to artificially bias our data by arbitrarily assigning one sample as the "standard" to which other replicates would be compared. In these cases, we normalized to the mean of log intensity of the subset of genes in the corresponding sample for which measurements were above 0.27 fM in both samples of the pair. The same subset of genes was used to normalize TaqmanAssessment of the Spike-in performance and of the MAQC samples can be found in Additional File ® comparison using samples that were either vehicle treated or treated with the highest level of compound. Ten genes were expected to change either up or down, and four reference genes were expected to remain constant. The genes chosen for Taqman® and Nanostring comparisons were picked based on internal Merck criteria. The differential expression in a previous microarray study of these samples showed relatively modest fold changes (~2 fold) at the highest compound treatment level used for this study .The sample set comprised four treatment conditions with four mouse xenografts per condition for a total of 16 samples. The 16 tissue samples were split and preserved by three methods ; total RNA was isolated from FF and FFPET for RT-PCR analysis. Fourteen genes were chosen for Taqman® is shown in Figures A comparison of the compound and vehicle intensities between nCounter™ and Taqman® to nCounter™ comparisons. Since the region of NARG1 assayed by the Taqman® probe is at the junction between exons 1 and 2 and the region selected for the nCounter™ probe is close to the 3' end of the transcript, the two systems may be capturing valid but different transcript behavior of this gene. It should be noted that variability of measurements in the CTL samples (as represented by the error bars) was higher than for the other two sample types.As with the MAQC data, these sample sets show generally good agreement between the two platforms. The one outlier is NARG1 , which was consistently discordant in all TaqmanThe FF and lysate data have a high degree of correlation, suggesting minor loss of data quality by using CTLs rather than purified total RNA. Taken together, the data in Figures ® versus average of ratios reported by both platforms implies no compression of ratios across ratio values.To assess bias or compression in derived ratios, we plotted differences in ratios between nCounter™ and Taqmans Figure . The sca®-derived ratios from the FF samples. The absence of ratio compression simplifies the subsequent analysis: since the magnitude of the ratio does not affect the concordance between two platforms by this method, we can combine all measurements together and determine how many measurements were off and by how much. Rather than depend solely on one references gene, we performed a comparison of all possible sample pairs; systematic shifts affecting all genes in each comparison were thereby attributed to the difference in total mass and subtracted out. As expected, ratios measured by nCounter™ and Taqman® were similar for MAQC and FF xenograft samples. The MAQC samples showed slightly less deviation between Taqman® and the nCounter™ System, which may reflect the controlled nature of the sample set and the smaller fraction of very-low intensity genes , its multiplexed format, and its potential to measure gene expression in samples from pre-clinical and clinical settings . Our results confirmed the system has potential for pre-clinical and clinical measurements of multiple gene signatures in settings where the initial tissue collection would be conducive to FFPET or CTL preparations.This platform could be used to fill an important and growing gap in drug development research. Microarray experiments routinely are used in basic research but often identify too many genes to allow higher-throughput downstream use of those signatures for screening or readouts. By allowing the cost effective and accurate measurement of expression of tens of genes from clinical samples, the nCounter™ system could facilitate translation of multi-gene expression based biomarkers into the clinic.GG and AM-H are employees of Nanostring.VM performed the data transformations and analysis; KS designed and initiated the experiments and assembled the sample set, and also was the primary author of the manuscript; NB performed the Taqman assays; JW provided the xenograft model system and the tissue samples used, and also provided critical review of experimental design; GG and AM-H processed the samples at Nanostring and provided technical and critical review of the manuscript; TF provided overall project guidance, technical and critical review of the experiments and assisted in writing. All authors read and approved the final manuscript.Performance of Nanostring spike-ins. Nanostring spike-ins span a reported range from 0.25 fM to 50 fM. All samples were assayed in triplicate; data are broken down by replicate. nCounter™ spike-ins consisted of two parallel sets of the same RNAs made into two different cocktails (cocktails #3 and #4). Some RNAs are set at different concentrations between these cocktails while others are at the same concentration  and Nanostring (y-axis) for each of the four MAQC samples Comparison of intensity values derived from Taqman. Markers in red represent genes that were outside of the calibration curve. Pair-wise nCounter™ to Taqman® comparisons of measured expression levels of genes on a per sample basis showed correlation coefficients ranging from 0.835 to 0.886. Error bars represent standard deviations.Click here for file2 ratios for Brain vs UHR for Nanostring and Taqman®Comparison of calculated log. Transcripts that were outside the calibration curve are shown in red and are labeled with their lowest measured concentration in either Brain or UHR (in fM). In the comparison of ratios, data for nCounter™ and Taqman® were first corrected for the known differences in mRNA content between Brain and UHR (2% to 3%). Additional File ® and nCounter™ ratio measurements.Click here for fileData from Nanostring nCounter experiment. This spreadsheet contains the raw nCounter™ data, the Taqman® data used for comparisons, and the normalized and averaged nCounter™ data.Click here for fileRepresentative Bioanalyzer traces from FFPE-derived RNA. These traces demonstrate the degree of RNA degradation assayed by the nCounter™ system in our experiments.Click here for file


In both applications, we used a permutation-based approach for evaluating the significance of MI. Substantial discrepancies in significance were observed between MI, ANOVA, and KW that can be explained by different empirical distributions of the data. In contrast to ANOVA and KW, MI detects shifts in location when the data are non-normally distributed, skewed, or contaminated with outliers. ANOVA but not MI is often significant if one genotype with a small frequency had a remarkable difference in the average gene expression level relative to the other two genotypes. MI depends on genotype frequencies and cannot detect these differences. In the classification scenario, we show that our novel approach for feature selection identifies a smaller list of markers with higher accuracy compared to the standard method. In conclusion, permutation-based MI approaches provide reliable and flexible statistical frameworks which seem to be well suited for data that are non-normal, skewed, or have an otherwise peculiar distribution. They merit further methodological investigation.Mutual information (MI) is a robust nonparametric statistical approach for identifying associations between genotypes and gene expression levels. Using the data of Problem 1 provided for the Genetic Analysis Workshop 15, we first compared a quantitative MI  with the standard analysis of variance (ANOVA) and the nonparametric Kruskal-Wallis (KW) test. We then proposed a novel feature selection approach using MI in a classification scenario to address the small  Gene expression varies substantially in humans, and variation in the expression level of many genes is heritable. One of the most challenging aspects is, however, the identification of associations between genotypes and gene expression levels, for which transcript levels are viewed as phenotypes. This has become more important as high throughput technologies enable researchers to genotype single-nucleotide polymorphism (SNP) markers on a genome-wide scale and to measure gene expression levels of thousands of genes simultaneously. However, regardless of the final goal of a specific study, a number of problems arise frequently, related to the very nature of the genetic data. Herein, we have identified two such problems in which the use of standard statistical methods may be inappropriate or provide suboptimal results.A first particular challenge is that gene expression data often are not normally distributed . In manyn - large p problem because a relatively small number of subjects is classified by using a relatively large number of genotypes. Subsequently, an almost infinite number of possible solutions explain the data equally well, and a key problem is to reduce the number of attributes using a suitable feature selection strategy.A second challenge is caused by the small One approach to tackle the two problems is the use of mutual information (MI), a nonparametric approach from information theory. First applications of its use for analyzing genotypes and expression data have been published very recently ,3. In paFirst, we compare the permutation-based QMIS of Tsalenko et al.  with theχ2 distribution.In the second application, we aim at classifying the molecular phenotype data of subjects obtained from clustered gene expression profiles using a small set of features. In particular, we propose a novel permutation-based MI strategy for feature selection and compare it with a standard feature selection approach which relies on the asymptotic In both applications, we utilize the Genetic Analysis Workshop (GAW) 15 Problem 1 data.Data for Problem 1 of GAW 15 come from 14 three-generation CEPH (Centre d'Etude du Polymorphisme Humain) Utah families. All 194 members of these families were genotyped at 2882 autosomal and X-linked SNPs. Expression levels in lymphoblastoid cells were measured for 8500 genes. 3554 transcripts were selected by Morley and colleagues [X. In other words, the entropy can be viewed as a quantity representing the information associated with an outcome of X. In case of a discrete random variable X with outcomes x ∈ The concept of MI is based on the entropy of a random variable. Shannon  defined pX(x) = PX(X = x), and X. The entropy is non-negative, disappears in case of a degenerate random variable and is maximal for a uniformly distributed X. For estimation, we use empirical frequencies ni is the number of observations with X = i, and n is the total sample size. A natural estimator of H(X) is thus given by where X and Y, and measures their mutual dependence. In the discrete case, MIMI extends the latter concept to random variables is defined aspXY denotes the joint probability of X and Y, and pX(x) and pY(y) are the corresponding marginal probabilities. MI quantifies the amount of information that one random variable contains about the other. Equivalently, MI measures the reduction in uncertainty of X if Y is known, and vice versa. Thus, for two independent variables X and Y, observing one of them does not lead to an information gain concerning the other one. In this case, pXY = pX(x)·pY(y), so that the ratio equals one and, subsequently, MI = 0. The other extreme case occurs if X and Y are completely linearly related. Then, all information conveyed by X can be found within Y. MI can then be calculated using the entropy of one of both variables.where MI is symmetric, non-negative and is related to the entropy viaMoreover, H(X | Y) denotes the conditional entropy, and H is the joint entropy of X and Y. Because H(X | X) = 0, we obtain H(X) = MI, and H(X | Y) ≥ 0 yields MI ≥ MI. In other words, no other variable can contain more information about X than X itself.where MI can be approximated by a second-order Taylor series. In case of stochastically independent random variables X and Y, the resulting expression is related to a χ2-test of independence and follows a gamma distribution [χ2-test also hold for MI [χ2 distribution is imprecise if sample sizes are small, if the numbers per class differ substantially, or if the number of classes is small. In these cases, the use of empirical p-values as obtained from a permutation procedure is preferable.For discrete random variables, the distribution of ribution . As a coY, is continuous with observations y ∈ ℝ, a QMIS can be calculated [q is determined, leading to a dichotomous variable Yq such that Yq = I(Y <q), where I(·) denotes the indicator function. QMIS is then defined as the maximum possible MI:If one of the variables, say lculated . To thisq and using the right hand side of eq. (1), an estimator is given byAnalogously to the entropy, relative frequencies can be used to determine nij is the number of observations in group j defined by Yq with X = i.where The estimator given in Eq. (3) is solely based on frequencies and does not involve any continuous measurements such as squared deviations from a mean value. As a consequence, the QMIS approach is robust against outliers and does not require any specific shape of distribution.In the first application, we compared the permutation-based QMIS with the standard parametric ANOVA and the nonparametric KW for detecting associations between a single gene expression level and a single SNP. We systematically investigated the causes for differences in statistical test results. We calculated QMIS according to Eq. (2) using estimates obtained from Eq. (3). To specify a final threshold for each gene, the range of expression values was divided into 10 equidistant intervals, yielding 11 potential thresholds within the maximization process. The number of intervals was limited to reduce computation time.p-values were computed using a permutation procedure. In doing so, for each of the N = 104 permutations, genotypes were fixed, while gene expressions were permuted. In cases where no permuted test statistic was at least as extreme as the test statistic using the original data, calculations were repeated with N=108 permutations. To allow for heteroscedastic errors, the Welch modification of the parametric ANOVA was used. Finally, we denoted an association of gene i and transcript j to be significant if the p-value is less than αlocal = 10-4.Empirical To avoid within-family dependencies, we only used founder data for model comparisons. In addition, SNPs with less than 5% of samples for each genotype and more than 5% of missing genotypes were removed, yielding 3554 genes and 1089 SNPs.p-values between the different approaches were visualized using hexagon binning [Differences in  binning . All comIn the second application, we aimed at classifying the molecular phenotype data using a small set of features.Within the first step, we created clusters of individuals with similar gene expression profiles. The data were modeled as a finite mixture of Gaussian distributions, and the most likely cluster partition was determined by an expectation-maximization (EM) algorithm using a 10-fold cross-validation procedure to determine the optimal number of clusters. After the clustering step we assigned a label to each individual, which was used to perform a supervised learning task on the basis of the genotype information.χ2-test of independence. More precisely, we considered a χ2-test in which the degrees of freedom (d.f.) are equal to (number of genotypes - 1)·(number of classes - 1) at a significance level of α = 0.001. Second, we applied a novel nonparametric approach based on a permutation method inspired by the significance analysis of microarrays (SAM) technique [In what followed, we considered SNPs and clusters as random variables and computed the MI between them. We evaluated the significance of MI in two ways. First, we focused on a parametric approach and used the relation between MI and the echnique . This apjth test set.The step following feature selection was classification. We estimated the goodness-of-fit of the model using a 10-fold cross-validation. We determined the accuracy of the classification procedure as the proportion of correctly classified individuals. A person was classified correctly if the classification step led to that class that was specified in the previous cluster analysis. In general, a cross-validation resolves the problem of obtaining a test set, but it is affected by a selection bias that is often ignored. Here, the selection bias may result from performing feature selection on the entire data set before running a cross-validation. In this way, all the cross-validation learning steps are affected by the bias provided by feature selection, because the entire data set was already used for selecting the most interesting features. To overcome this problem, we included the feature selection step in the cross-validation procedure, as recommended by Simon et al. [Because feature selection was made in each iteration step of the cross-validation, a different SNP list was generated in each run of the procedure. To derive a consensus SNP list, we extracted those SNPs that were selected in each run of the cross-validation process. Therefore, this set contained the most important SNPs regardless of small changes in the training set.k-nearest-neighbor (KNN). For the number of neighbors in KNN, we chose k = 1, 5, 10, and for all learning machines we used the default settings of the package.The WEKA software version 3.4  was emplp-values, substantial discrepancies between QMIS and the KW become apparent  have slightly smaller classification accuracies of 62.37% and 54.65%.In an initial step, five clusters were identified by the EM algorithm. Then classification was performed using three different classifiers with both feature selection criteria described above, showing that the nonparametric approach improves classification performances in contrast to the parametric one. Using the Seventy-five SNPs are always selected among the different training-test pairs and therefore comprise the final SNP list. If only these SNPs are used for classification, the accuracy of the naïve Bayes classifier increases to about 70%. This is in line with previous observations using random forests as classifier .A permutation-based QMIS is a sensible nonparametric alternative to the standard parametric ANOVA and the nonparametric KW. In contrast to ANOVA and KW, QMIS is able to detect differences in location between asymmetric, skewed, and outlier-contaminated distributions. However, parametric methods are generally more powerful if model assumptions such as normality are met. Similarly, KW might be more adequate than QMIS for analyzing a shift alternative in case of equally shaped distributions. However, these assumptions are not met for gene expression data in the majority of cases.χ2 approach.In addition, our novel permutation MI strategy for feature selection in a classification setting has higher accuracy and uses a smaller number of SNPs than the standard The author(s) declare that they have no competing interests.


Accuracy in the diagnosis of breast cancer and classification of cancer subtypes has improved over the years with the development of well-established immunohistopathological criteria. More recently, diagnostic gene-sets at the mRNA expression level have been tested as better predictors of disease state. However, breast cancer is heterogeneous in nature; thus extraction of differentially expressed gene-sets that stably distinguish normal tissue from various pathologies poses challenges. Meta-analysis of high-throughput expression data using a collection of statistical methodologies leads to the identification of robust tumor gene expression signatures.A resampling-based meta-analysis strategy, which involves the use of resampling and application of distribution statistics in combination to assess the degree of significance in differential expression between sample classes, was developed. Two independent microarray datasets that contain normal breast, invasive ductal carcinoma (IDC), and invasive lobular carcinoma (ILC) samples were used for the meta-analysis. Expression of the genes, selected from the gene list for classification of normal breast samples and breast tumors encompassing both the ILC and IDC subtypes were tested on 10 independent primary IDC samples and matched non-tumor controls by real-time qRT-PCR. Other existing breast cancer microarray datasets were used in support of the resampling-based meta-analysis.The two independent microarray studies were found to be comparable, although differing in their experimental methodologies . The resampling-based meta-analysis has led to the identification of a highly stable set of genes for classification of normal breast samples and breast tumors encompassing both the ILC and IDC subtypes. The expression results of the selected genes obtained through real-time qRT-PCR supported the meta-analysis results.The proposed meta-analysis approach has the ability to detect a set of differentially expressed genes with the least amount of within-group variability, thus providing highly stable gene lists for class prediction. Increased statistical power and stringent filtering criteria used in the present study also make identification of novel candidate genes possible and may provide further insight to improve our understanding of breast cancer development. Microarray studies aiming to identify differentially expressed as well as co-regulated gene sets and signaling pathways involved in different cellular states have greatly improved our understanding of breast cancer at the molecular level. The power of expression profiling using cDNA or DNA microarrays for distinguishing subgroups of breast cancers has been demonstrated by several groups -4.The identification of an intrinsic gene-set exhibiting high variability among different tumor clusters has been informative in describing different subtypes of breast cancer samples. However, only a few papers have been published on gene expression profiles of normal cell populations in breast tissue -9. Thereet al. identified differentially expressed genes between ER+ and ER- breast tumors by gathering 9 independent breast cancer microarray studies [et al. were able to identify a new intrinsic gene-set for breast cancer subtype prediction by combining multiple microarray datasets to assess prognosis [Meta-analysis of microarray datasets has the potential to lead to more comprehensive measures of the existing differential gene expression data and can therefore provide gene sets with a high diagnostic value. Meta-analysis of independent microarray datasets generated with the common objective of identifying differentially expressed genes in a certain type of cancer has also been performed for breast cancer. In a very recent meta-analysis study, Smith  studies . Another studies . Similarrognosis .delete-d-jacknife procedure in which a subset of data is excluded to find out the frequency of selecting a particular gene as differentially expressed [leave-one-out resampling [Several different meta-analysis approaches exist in the literature. In some, each individual study contributes rather independently to the meta-analysis -15 wherexpressed . The numsampling .Existing meta-analytic approaches applied to different types of cancer show the power of a combined study for identifying novel genes not present in the existing literature  ,21. Invadelete-d-jacknife, a series of d sample size values have been tested to assess the extent of variability across the tumor samples and the stability of differential expression. Comparison of probability value distributions obtained for the test and randomized samples has led to determination of the degree of differential expression between groups tested. Accordingly, we report that the Sorlie et al. [et al. [In the present study, we primarily aimed to develop a novel methodology for the meta-analysis of independent microarray datasets. Using this methodology, we provide gene lists that (a) are discriminative of breast cancer types  and normal breast cell populations, (b) may yield breast tumor markers that are invariably expressed across independent experiments, and (c) provide a set of consistently differentially expressed gene candidates with potential discriminative ability for tumor subtypes. Using a method similar to e et al.  and Zhao [et al.  datasets [et al. ,7,8 usin [et al. . A subseet al. [et al. [[Two independent microarray gene expression data sets, Sorlie et al.  and Zhao [et al. , were do[et al. [. Gene fi®) between the Sorlie and Zhao datasets for the IDC and ILC samples separately. Those clones with probability values less than 0.05 (after Bonferroni correction) were excluded from further analysis. This two-step data filtering resulted in a common set of 1726 IMAGE clones for the analysis of ductal and normal samples, and 2029 IMAGE clones for the analysis of lobular and normal samples. Upon taking the intersection of the ductal-normal and lobular-normal clone sets, 1522 IMAGE clones were available for the ductal-lobular analysis. The resulting clone subsets were further filtered by removing IMAGE clones with more than 40% missing data for the two groups in comparison  in the combined data before application of the resampling steps. In addition, if an IMAGE clone had a sample size  less than the resampling sample size, data on this IMAGE clone was also removed.Data were filtered separately for ductal and lobular samples. IMAGE clones with more than 50% missing data in either of the Sorlie or Zhao datasets were excluded from the common clone set. Data filtering was further improved by performing two-tailed Student's t-tests with equal variance (Matlab® using Statistics Toolbox is available upon request). Accordingly, a preset number of samples was selected from each group  of the original dataset, referred herein as the test. The p-value was calculated indicating the significance of the difference between the group medians based on the Wilcoxon Rank Sum Test. This test was repeated for a series of i number of iterations; at the end of each iteration scheme, a set of p-values (pt) per IMAGE clone was obtained. The above procedure was also applied to each of the shuffled datasets yielding pr1 and pr2. P-value distributions were then tested in a pair-wise fashion  using the two-sample Kolmogorov-Smirnov test for each clone in the dataset , ksr values were sorted in the ascending order and the kth value from the top  was determined as FDRobserved, where k equals the expected value of FDR  multiplied by the number of IMAGE clones tested. FDRobserved was set as the threshold according to which IMAGE clones were assigned as significant or not. If kst of a particular gene had a value that was smaller than the FDRobserved, the gene was accepted to be significant.We have used a resampling method for meta-analysis of microarray data in which the significance of the difference between group medians  could be tested upon a series of resampling schemes from the original and multiple randomly shuffled datasets , repetitively for i number of times where i = 10, 20, 30, ..., 100 and 150. For each particular i, three parameters were recorded, namely, kst values, the mean expression value of each of the two groups compared, and the significance of the differential expression based on kst and ksr. These above steps were then repeated with different sample sizes: For ductal vs. lobular comparison, n was set to be 3, 4, 5, 6, 10, 15 and 20. On the other hand, since the total number of normal samples was 7, the highest sampling value could be set to 6 for ductal vs. normal and lobular vs. normal comparisons, and n equaled 3, 4, 5 and 6. These sample size-iteration combinations led to 77 runs for ductal vs. lobular analysis, and 44 runs for ductal vs. normal and lobular vs. normal analyses. At the end, a final differentially expressed gene set was determined for each of the three comparisons  by gathering the IMAGE clones that were assigned as significant in 90% or more of these 44 or 77 runs. The mean values of each of the two groups in comparison obtained at n = 20  and i = 150 were used as an estimate of the measure of expression.The above tasks were performed for a particular sample size et al. performed an analysis of tumors from 49 breast cancer patients that were successfully classified into luminal and basal classes, and a novel molecular apocrine class. Apocrine tumors were estrogen receptor negative ER(-) and androgen receptor positive AR(+), while luminal tumors were ER(+) and AR(+), and basal tumors were ER(-) and AR(-). Details of the breast specimens /ER (-)) available from GEO database were used in the supervised class prediction with a binary tree algorithm [The ".cel files" of the three publicly available independent microarray gene expression data sets, GDS2635  and GDS1Primary tumor samples and matched non-tumor breast tissues were obtained from patients (n = 10) during surgery and immediately snap-frozen in liquid nitrogen and stored at -80°C until RNA extraction. The frozen tissue samples were sectioned and mounted on glass slides. The slides were stained with hematoxylin and eosin for histopathological examinations. Only those tumor samples with more than 90% of tumor cells and matched tissue pairs with normal histological examination were included in this study. These frozen tissues were cut into 5-μm-thick sections and used for RNA isolation and cDNA synthesis. All the tumor samples had been classified as infiltrating ductal carcinoma. The use of the tissue material in this project was approved by the Research Ethics Committee of Ankara Numune Research and Teaching Hospital and consents were obtained in accordance with the Helsinki Declaration.The frozen breast specimens were put into Trizol reagent , disrupted with a homogenizer and total RNA was isolated according to the manufacturer's instructions. Genomic DNA contaminations were removed by on-column DNaseI treatment . The concentration of the isolated RNA and the ratio of absorbance at 260 nm to 280 nm were measured with the NanoDrop ND-1000 spectrophotometer  in triplicate.First-strand cDNA was synthesized from 1 μg total RNA using oligo(dT) primers using Revert Aid First strand cDNA synthesis kit according to the manufacturer's instructions . The cDNA was diluted at a ratio of 1:5 before being used as a PCR template and stored at -20°C until further use.FN1, ID4, EGFR, ADAMTS1, ATF3, IGFBP6, and PRNP. The geometrical mean of ACTB, TBP and SDHA1 gene expression values were used as internal control for relative gene expression quantitation [®).Real-time qRT-PCR analysis was performed using gene-specific primer pairs  of a gene in the meta-data. For example, the GSN gene had a highly significant differential expression between ductal and normal samples as evidenced by the highly skewed distribution towards lower p-values whereas the RAP2A gene exhibited a uniform distribution of p-values  as well as the number of differentially expressed IMAGE clones were plotted with respect to the increasing number of iterations Figure . As the It is reasonable to assume that use of a single sample size and iteration number may not be adequate to understand the variability among the tumor samples Figure . It mighDifferentially-expressed gene lists for DN and LN contained 298 (282 genes) and 216 (202 genes) IMAGE clones, respectively , each of which corresponds to a novel contribution by our meta-analysis . All of the above genes except FN1 were found to be significantly associated with the tumor ER status . Protein expression levels of these four genes were investigated using the Human Protein Atlas, a public resource for immunohistochemistry (IH) of normal and pathological human tissues . IH data were available for CDH1, TFAP2A, and RBP4 proteins; and only data from antibodies exhibiting differential expression among breast tumors were reported herein. Accordingly, 2 out of 3 ILC samples exhibited moderate to strong signals for RBP4 (Antibody CAB00455) whereas 7 out of 9 IDC samples were either negative or had weak staining. CDH1 data in the Protein Atlas database was not very informative since the number of ILC samples were limited, but a moderate signal was detected for the ILC sample whereas 5 out of 6 IDC samples expressed CDH1 strongly (Antibody CAB000087). Similarly, TFAP2A was weakly or moderately expressed in the two ILC samples examined whereas a moderate to strong staining was observed in 5 of the 9 IDC samples. Although sample size in the ILC samples in the Human Protein Atlas database was limited, there was a corresponding trend between the mRNA levels reported by the present study and the protein level assessment obtained from the Human Protein Atlas. Future studies should include testing of the genes extracted by meta-analysis using protein level studies such as Western blotting or immunohistochemistry on a large set of IDC and ILC samples to confirm their predictive power.Comparison of fold-change values of the DL meta-gene list consisting of 65 genes with that of the Turashvili's DL list (GDS2635) resulted in a high degree of correlation , suggesting that the direction and magnitude of expression change between the IDC and ILC samples were largely consistent between data from different microarray experiments. Furthermore, we combined published expression data from IDC and ILC samples from experiments performed by Bertucci et al  with theMAF) from the meta-gene list for validation of the meta-analysis. Expression profiles of these genes were tested in independent paired IDC breast tumor and non-tumor tissue samples through real time qRT-PCR. Our results were consistent with those of the meta-analysis such that GSN, SPTBN1, SFRP1 and MAF were down-regulated in most tumor samples with respect to their matched non-tumor samples whereas COX6C, RAD21, GSPT1, NME1 and PTTG1 were up-regulated .We first selected nine genes that were found to be differentially expressed in both the DN and LN lists  also varied considerably between the two studies. Median rank scores  or quantDue to the large number of comparisons involved in microarray data analysis, it is important to take into account the false positive error rate and control it for the number of tests performed. FDR is a well-known methodology for multiple-test correction; its estimation relies on calculation of the number of false positives in a randomly permuted set of experiments . Therefoet al. [CDH1 mutation and/or underexpression was common but not universal to ILCs in general [Invasive breast tumors comprise 18 different histological types , most ofet al.  reported general . Low-graet al. [et al. [et al. [et al. [The other three studies presenting data on ILC and IDC, Turashvili et al. , Sorlie  [et al.  and Zhao [et al.  have use [et al. , meta-an [et al.  has alsoCHD1, TFAP2A, RBP4, and ALDH1A1 genes as commonly modulated. Indeed, CDH1 is one of the best-studied discriminators for ductal/lobular breast cancer specimens in the literature by immunohistochemistry and at the genomic level. In breast cancer, reduced CDH1 expression has been found in 50% of invasive ductal carcinomas, whereas CDH1 expression was almost always absent in infiltrating lobular carcinoma (ILC) [TFAP2A was shown to be highly expressed in ductal tumor cells while normal cells expressed TFAP2A in the inner glandular cell layer [RBP4 in the literature in connection with ductal vs. lobular breast cancer distinction while ALDH1A1 protein levels were shown to exhibit differences among the ductal carcinoma patients [Comparisons between the meta-analysis and the Turashvili and Bertucci studies pointed out to ma (ILC) ,37,38. Tll layer . On the ll layer . There ipatients . The canOn the other hand, analyses of Sorlie, Zhao, and Turashvili data showed that tumor cells were remarkably distinct from their respective normals in their transcription profiles implicating that whatever the subtype structure underneath, most of the variability among samples was due to changes during tumorigenesis. Accordingly, the idea that genes discriminating tumor from normal in a stable manner also may have information on the state of the tumorigenesis is a valid one.et al. [Breast tumor subtype classification remains a complicated issue due to difficulties associated with the presence of multiple interacting factors such as the presence or absence of node-filtration, ER-positivity, metastatic potential, different degrees of genomic instability, and tumor cell origin. For example, basal like cancers have distinct molecular expression profiles and histological differences when compared with the luminal type . Nielsenet al.  have catet al. . Howeveret al. . It is aet al. ,46. In get al. ,47. ThusATF3, ADAMTS1, EGFR, PRNP, IGFBP6, ID4, SFRP1, SPTBN1, and FN1 with ability to classify tumors into basal and luminal subclasses. Additionally most of them accurately differentiated ER(+) and ER(-) tumors  also keep information about intrinsic subtypes. Accordingly, meta-analysis identified ID4 was found to be a novel tumor suppressor gene in normal human breast tissues and epigenetically silenced in breast cancer cell lines and primary breast tumors [ID4 in the mammary duct epithelium may be regulated by estrogen depending on the differential expression pattern of ID4 in ER(+) and ER(-) breast tumors [SFRP1 on the other hand is a frizzled-related protein that plays a role in a variety of cellular processes, including control of cell polarity, cell fate determination, and malignant transformation. In previous studies, loss of SFRP1 was found to be associated with cancer progression and poor prognosis in breast cancer [ATF3) is a member of the ATF/cyclic AMP response element-binding family of transcription factors. It was shown to enhance apoptosis in the untransformed mammary epithelial cells while protecting the aggressive cells and enhancing cell motility. Array analyses indicated that ATF3 upregulated the expression of several genes in the tumor necrosis factor pathway in the untransformed mammary epithelial cells. However, the expression of several genes implicated in tumor metastasis including fibronectin (FN1) was upregulated in aggressive cells. ATF3 was also shown to regulate the transcription of FN1, one of the genes obtained in the present study. ATF3 gene copy number was at least doubled in 80% of the breast tumors examined; protein levels also were elevated in close to 50% of these tumors [Among those genes, t tumors ,49. As st tumors . SFRP1 ot cancer ,52. EGFRt cancer ,54. Actie tumors .Since the normal vs. tumor classification was strikingly distinct based on meta-analysis, and a gene-set with the capacity for breast cancer subtype classification, we further analyzed a set of normal matched tumors for selected genes from the meta-gene list using real-time qRT-PCR. The selected 16 significant genes were shown to have expression profiles similar to those found from the meta-analysis. Our findings also suggested that these genes could be used as predictors of tumor status regardless of the origin of the reference samples, i.e., a matched or pooled reference tissue. Since the number of samples used in qRT-PCR was relatively small, increasing the sample size may help generalize our results to a wider range of breast tumor samples.There was a high level of correlation between fold changes obtained from the DL meta-genes and those from the Turashvili dataset, regardless of the different sample extraction methods used in each study . The meta-gene list discriminating between ductal and lobular breast tumor samples at the mRNA level requires further confirmation at the protein level to better assess discriminatory power. Future validation studies might concentrate on whether meta-analysis specific genes also participate in prediction of level of prognosis and/or time to disease-free survival.In this study, meta-analysis of two independent comparable microarray data sets allowed us to provide genes that are able to discriminate IDC and ILC and normal mammary cells from the tumors that either study by itself was not able to identify. We also provided highly generalized and stable gene lists that could be used for prediction of tumor or normal status. The meta-gene list for tumor/normal comparison had a striking predictive ability based on comparisons made with three independent microarray datasets. The resampling approach proposed herein has the ability to detect a set of differentially expressed genes, with the least amount of within-group variability. This meta-analytic approach thus provides a method to combine two or more independent cancer data sets leading to the identification of differentially expressed gene sets for better understanding of cancer development and progression.DL: Ductal and Lobular; DN: Ductal and Normal; FDR: False Discovery Rate; IDC: Invasive Ductal Carcinoma; ILC: Invasive Lobular Carcinoma; LN: Lobular and Normal; qRT-PCR: Quantitative Reverse Transcriptase Polymerase Chain Reaction; SMD: Stanford Microarray Database.The authors declare that they have no competing interests.IGY, BGD and OK originally conceptualized the meta-analysis method and application to breast cancer datasets, and wrote the manuscript. OK and SK developed the meta-analysis algorithm; SK contributed to manuscript writing. BGD and SK applied the algorithm to breast cancer datasets and generated figures. BDG performed all BRB-tool analyses and carried out expression studies. ARO and OK wrote algorithms for data extraction and compilation. BB carried out the surgical removal of the sample tissues and GE, the pathological assessment of the surgical tissue materials. IGY coordinated the study and participated in its design. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:Complete list of common IMAGE clones. The data represents 4769 common IMAGE clones from combined Sorlie and Zhao datasets from the study of Zhao et al. [o et al. .Click here for fileComplete list of common IMAGE clones. The data represents 4769 common IMAGE clones from combined Sorlie and Zhao datasets from the study of Sorlie et al. [e et al. .Click here for fileGenes used for real-time qRT-PCR analysis. Gene names, accession numbers and gene specific primer pairs used for real-time qRT-PCR analysis of the selected genes.Click here for fileCorrelation of global median and quantile normalized data. The figure shows the correlation of global median and quantile normalized data of ductal (D) and normal (N) tissue samples.Click here for fileGene set differentially expressed between ductal (D) and normal (N) breast tissue samples. The data provided represents the list of 298 IMAGE clones differentially expressed between ductal (D) and normal (N) samples with 90% significance.Click here for fileGene set differentially expressed between lobular (L) and normal (N) breast tissue samples. The data provided represents the list of 216 IMAGE clones differentially expressed between lobular (L) and normal (N) samples with 90% significance.Click here for fileGene set differentially expressed between ductal (D) and lobular (L) breast tissue samples. The data provided represents the list of 66 IMAGE clones differentially expressed between ductal (D) and lobular (L) samples with 80% significance.Click here for fileList of meta-analysis specific genes. The file includes the list of genes differentially expressed between DN, LN, and DL samples.Click here for fileValidation of meta-analysis gene lists by three independent microarray datasets. The file represents the gene lists obtained by comparing the DN and LN meta-gene list to the gene expression profiles of normal and breast tumor tissues from three independent microarray datasets.Click here for file


Meta-analysis methods exist for combining multiple microarray datasets. However, there are a wide range of issues associated with microarray meta-analysis and a limited ability to compare the performance of different meta-analysis methods.We compare eight meta-analysis methods, five existing methods, two naive methods and a novel approach (mDEDS). Comparisons are performed using simulated data and two biological case studies with varying degrees of meta-analysis complexity. The performance of meta-analysis methods is assessed via ROC curves and prediction accuracy where applicable.Existing meta-analysis methods vary in their ability to perform successful meta-analysis. This success is very dependent on the complexity of the data and type of analysis. Our proposed method, mDEDS, performs competitively as a meta-analysis tool even as complexity increases. Because of the varying abilities of compared meta-analysis methods, care should be taken when considering the meta-analysis method used for particular research. This neIC) [We also perform a comparison study of meta-analysis methods including the Fisher's inverse chi-square method , GeneMetIC) , RankProIC)   curves as well as the area under these ROC curves (AUC). The two different case studies vary in complexity and performance are assessed through predication accuracy in a classification framework. Warnat et al.  uses valIt is possible to consider meta-analysis at two levels, 'relative' and 'absolute' meta-analysis. 'Relative' meta-analysis looks at how genes or features correlate to a phenotype within a dataset . MultiplIn this paper we will begin by describing existing meta-analysis methods, then we will outline our proposed mDEDS method. This is followed by the comparison study, where publicly available datasets are combined by different meta-analysis methods, examining their ability under varying degrees of complexity, as well as comparing mDEDS to DEDS. Finally, we provide discussion and conclusions of results.X represent an expression matrix, with i = 1, ..., I genes and j = 1, ..., N samples. If there are k = 1,..., K datasets, k nrepresents the number of samples in the kth dataset. For simplicity, and without loss of generality, we focus on dichotomous response; i.e., two-group comparisons. We designate groups as treatment T and control C. For two-channel competitive hybridization experiments, we assume that the comparisons of log-ratios are all indirect; that is we have T narrays in which samples from group T are hybridized against a reference sample R, and we can obtain T nlog-ratios, 2(j/RT); j = 1, ..., T nfrom group T. In an identical manner c nlog-ratios are also calculated from group C. For Affymetrix oligonucleotide array experiments, we have T nchips with gene expression measures from group T and C nchips with gene expression measures from group C.Let Fisher, in the 1930 s developed a meta-analysis method that combines the p-values from independent datasets. One of a plethora of methods for combining the p-values , is the i, there is no differences in expression means between the two groups. The p-value ik pis the p-value for the ith gene from the kth dataset. In assessing iS, the theoretical null distribution should be which tests the null hypothesis that for gene GeneMeta[One of the first methods that integrates multiple gene expression datasets was propose by Choi et al.  who descGeneMeta an R paceffect-sizes from the K datasets. In a two group comparisons, a natural effect size is the t-statistics. For a typical gene i, the effect size for the kth dataset is defined asChoi et al.  describekth study. pk Sis the pooled standard deviation for the kth dataset.where K number of observed effect sizes, Choi et al. [For i et al.  proposedμ is the parameter of interest, k sdenotes the within study variances and δ ~ N represents the between study random effects with variance τ2. Choi et al. [τ2 = 0, k δdenotes the between study effect in a fixed effect model. The random effects model is then estimated using a method proposed by DerSimonian and Laird [where δ ~ N0, τ represennd Laird  and a pemetaArray contains a number of meta-analysis methods. The main function is a two steps procedure which transformed the data into a probability of expression (POE) matrix [IC) [The R package ) matrix  and follrix [IC) .X to a matrix E that represents the probability of differential expression. Each element in the matrix ij Eis defined as the chance of multiple conditions present across N samples within gene i. The transformed matrix, E, consists of three values -1, 0, 1 that represent the conditions 'under-expressed', 'not differentially expressed' and 'over-expressed'. After the transformation into a POE matrix, genes of interest are established using IC [i ≠ i') between genes (i and i') across all samples within a dataset k. Thus, we generated a pairwise correlation matrix P with K columns representing the number of datasets.Given a study, the POE method transforms the expression matrix using IC . Notice k and k', let us denote i is obtained by only considering comparisons that contain the ith gene. That isFor a selected pair of datasets i ≠ i'. When more than two datasets are being compared, all integrative correlations for a particular gene are aggregated. This method provides a combined ranking for genes across K datasets.where IC' and 'POE with Bss/Wss' to indicate what type of analysis was performed after the construction of the POE matrix.In this comparison study, two metaArray results are used. Distinction will be made between them using the terms 'POE with RankProd is a non-parametric meta-analysis method developed by Breitling et al. . Fold chk, pairwise FC (pFC) is computed for each gene i asWithin a given dataset T n× C pFCl,m nvalues per gene with l = 1, ..., T nand m = 1, ..., Cn. The corresponding pFC ratios are ranked and we may denote this value as pFCi;r) with the expectation that the results will be transferable. In a classification context this means that one dataset is used for feature selection and development of the discriminant rule and we predict the outcome of the other dataset(s) via this rule.'Meta differential expression via distance synthesis' (mDEDS) is a meta-analysis method that makes use of multiple statistical measures to obtain a DE list. It is aided by 'Differential Expression via Distance Synthesis', DEDS  which isThe true DE genes should score highly within a set of non-dominated genes, both within a dataset using DE measures and also between datasets when the same DE measures are used on different datasets across different platforms. Consistently high ranked genes are then considered DE via mDEDS. This method endeavours to be robust against both measure specific bias, when different measure produce significantly different ranked lists, and platform specific bias where particular platforms produce results that are more favourable to particular gene sets.k = 1, ..., K datasets and g = 1, ..., G appropriate (DE measuring) statistics, hence there will be K × G statistics for each of the i = 1, ..., N genes. Let ikg tbe the statistic for the ith gene, from the kth dataset for the gth DE measure. Assuming large values indicate increased DE genes, let the observed coordinate-wise extreme point be1. Let there be 2. Locate the overall  extreme point E:K datasets is permuted B times by randomly assigning T narrays to class 'T' and C narrays to class 'C', producing b = 1,... B sets of K datasets. For each permuted datasets the G number of DE statistics are recalculated yielding (a) Each of the p Eby maximizing over the B permutations,(b) Obtain the coordinate-wise permutation extreme point E as the overall maximum: E = max.(c) Obtain d from each gene to E. For example, one choice for a scaled distance is3. Calculate a distance d(1) ≤ d(2) ≤ ... ≤ dN).Case study 1, implemented by the breast cancer data contains datasets from identical Affymetrix chips, this is considered 'similar platform meta-analysis'. Case study 2 which makes use of the lymphoma datasets contains samples that are hybridised using long oligo two colour platforms, Affymetrix chips and the 'Lymphochip' [Eight meta-analysis methods are compared using a simulated dataset and two cases studies comprising of six publicly available datasets, pertaining to three breast cancer and three lymphoma datasets. The purpose of the comparison study is to establish how these meta-analysis methods perform under varying degrees of dataset complexity. Dataset complexity refers to the level of difficulty present when combining multiple datasets. For example datasets being produced on the similar platforms (for example different Affymetrix platforms) are less complex to analyse via meta-analysis then when analysing results across very different platforms. For this comparison paper two levels of dataset complexity are considered. phochip' , this isFor the publicly available data, probe sets for each platform are mapped to the relative 'Entrez IDs' . Where mAssessing the performance of different meta-analysis methods is important to evaluate and compare methods. Although important, performance assessment of meta-analysis methods is non-trivial. Typically meta-analysis methods will be evaluated using pre-published gene lists and noticing the concordance of the obtained DE gene list and published material, this process however is subject to publication bias. To avoid such biases two forms of performance assessment will be applied in this paper.1. Receiver operating characteristic curves (ROC): For the simulated data where the 'true' DE gene list is known, meta-analysis performance is measured via ROC curves. ROC curves are created by plotting the true positive rates verses the false positive rates for the obtained DE genes. Performance is indicated by how close the plots are to the upper left hand corner of the ROC space. The AUC is also used as a comparison tool, with AUC values close to one indicating an accurate DE list. Because of the design of the simulation study the 'cross-validation' meta-analysis method can not be used.Bss/Wss [2. Prediction accuracy: For the case studies, prediction accuracy under a classification framework is used to asses performance of the DE list. We will use the term DE list for the consistency of this manuscript, although strictly speaking in a classification framework such gene lists are known as feature gene lists. To classify within the case studies, each consisting of three independent datasets, two datasets are combined via the meta-analysis methods and DE genes are selected. When DE gene selection is not part of the meta-analysis approach DE genes are ranked via 'between sum of squares over within sum of squares' Bss/Wss . Using tBss/Wss . The thiTo evaluate the performance of the different meta-analysis methods, data was simulated to represent three separate gene expression datasets. The simulation approach is adapted from an approach presented by Ritchie et al. . A non-pDE genes are simulated via a 2 fold increase in fold change. Two types of DE genes are simulated, 'true' DE genes, and 'platform specific' DE genes. 'True' DE genes are identical genes within each dataset, representing biologically relevant DE genes. 'Platform specific' DE genes simulate platform bias apparent within DE genes from microarray experiments  and are Three publicly available Affymetrix datasets are used for the breast cancer study, all three datasets use the affymetrix platform U133A. Classification of the breast cancer samples aims to distinguish between the sample's estrogen receptor (ER) status (+ve or -ve) as determined by the sample information provided with the datasets, we refer readers to the original manuscripts for more details regarding this status. In this case ER status is being used simply as a response variable common throughout all considered datasets, it should be understood that predicting ER status using gene expression data is not the same as immunohistochemistry. These datasets include the Farmer et al. dataset  . Gene expression levels have been gathered from 60 patients presenting with lymphoma cancers, 37 of these samples are Follicular Lymphoma (FL) and 23 samples are Diffuse Large B-Cell Lymphoma (DLBCL). Human 19000 oligo array slides from the Adelaide microarray consortium were used to obtain microarray expressions. Two well known publicly available datasets were also analysed. The Shipp et al. data  containsTo establish the success of mDEDS as a meta-analysis method beyond the capabilities of DEDS, DEDS and mDEDS are compared. The strength of DEDS comes from its ability to synthesise results from a range of statistics, mDEDS goes beyond this to consider results from a range of statistics across multiple datasets. DEDS is a method for selecting DE genes and to this end was used in the simple meta-method described in the 'Existing meta-analysis methods' section. Datasets from both the breast cancer study and the lymphoma study were used in the comparison of these meta-methods with the Loi et al. and the SVH datasets used as the independent test sets.Bss/Wss and POE with IC appear to struggle with obtaining an accurate 'true' DE list. Fisher and mDEDS perform competitively with the difference between Fisher, simple and mDEDS reducing as the number of genes in the gene list increases.Three datasets were simulated, with 150, 100 and 80 samples, each with 20000 genes. The percentage of DE genes varied between 2.5%, 4% and 10%, with half the DE genes on each platform being 'true' and the other being 'platform specific' DE genes. Figure Figure IC meta-methods perform well under such conditions, conversely GeneMeta, POE with Bss/Wss and RankProd appear to struggle in DE gene selection. However, mDEDS can still utilise these different experiments purposefully, producing the lowest mean error rate  when meta-analysis is performed across similar platforms, for example Affymetrix with Affymetrix, (ii) when meta-analysis is performed across disparate platforms, for example Affymetrix with oligo arrays.Bss/Wss and IC, Fisher's inverse chi-square and RankProd were able to classify competitively, hence they are able to highlight between dataset DE genes. RankProd's success in this circumstance is similar to the findings by Hong et al. [The breast cancer case study uses datasets from three identical Affymentrix platforms. Affymetrix's development and processing protocols offer a reduced variability in array comparison . This feg et al.  where RaBss/Wss, GeneMeta and RankProd struggle to obtain a gene list robust enough for cross platform classification. Two different reasons could attribute to the depletion in accuracy of the meta-methods as the level of complexity increases. The meta methods could be over-fitting the data, methods that model the data are particularly susceptible to this, for example GeneMeta. Conversely, some feature selection methods may not capture the complexity of the data, this is potentially occuring in the POE with Bss/Wss case. Fisher's inverse chi-square meta approach does not take into consideration the actual intensities of each spot on the microarray, albeit at times this method is ideal, for example when individual intensities are unknown, or when the characteristics of the study vary greatly [The lymphoma case study aims to distinguish between FL and DLBCL subtypes and the datasets used makes this analysis more complex. Both cDNA and oligonucleotide arrays are compared. These platforms vary remarkably with differences ranging from probe length to the presence of reference samples. As the complexity of the meta-analysis rises POE with  greatly . This paWithin both complexity environments mDEDS is able to perform DE analysis well, as this method makes use of the different datasets but does not try to fit a full parametric model to the data. Our proposed mDEDS uses multiple statistical measures while developing its ordered gene list. Using multiple measures aids robustness as more of the variability can be encapsulated within the meta-method. The success of mDEDS over DEDS as a meta-method highlights that the method of combining different statistics across datasets aids in the meta-analysis process. It is possible that the multiple platforms and multiple measures draw enough diversity to begin to transcend cross platform variability and produce a reliable gene list. The variation in some of the meta-method's abilities within classification suggests that different tools are beneficial depending on the researcher's current meta-analysis project.One may speculate that mDEDS can be used in a batch correction context. Batch effect is a term given to non-biological experimental variation that occurs throughout an experiment. In most cases batch effects are inevitable as non-biological variations are observed simply through multiple, apparently identical, amplification and hybridisation. Staggering ones hybridisation process is a practical reality of microarray experiments for two main reasons: (i) data is often prospective and may be collected and processed in stages, (ii) there is a limit to the number of samples that may be amplified and hybridised at one time  hence foThere are still many open questions within the meta-analysis paradigm. For example questions pertaining to mismatched probe sets across platforms and the handling of multiple probes for the same genes. More research within these areas would greatly aid meta-analysis for microarrays and ones ability to make use of the current plethora of information laying dormant in these public repositories. However, once more of these type of tools for meta-analysis have been developed, meta-analysis will save time, money and scientific resources.Bss/Wss, POE with IC, RankProd, a 'dataset cross-validation' meta and a 'simple' meta.We compared eight meta-analysis methods, which comprise of five existing methods, two naive approaches and our novel approach, mDEDS. Integrating datasets within microarray analysis has copious and clear advantages. This study adds in establishing which meta-analysis methods are more successful in their approach by comparing multiple meta-analysis methods, including the Fisher's inverse chi-square, GeneMeta, POE with Our proposed method; mDEDS, has performed competitively and at times better than currently available meta-analysis methods. ROC curves were used as a comparison in a simulated study and prediction accuracy within classification was used as an evaluation tool in two real biological case studies. These case studies differ in complexity regarding data being combined, the first demonstrating the combining of three datasets from similar platforms (different Affymetrix chipsets) and the second combining datasets from Affymetrix, cDNA and the Lymphochip.In both classification comparisons mDEDS was used as a feature selection method and produced capable classifiers, with all else held constant. These results, coupled with results from the simulated data, are indicative of mDEDS being used as a powerful meta-analysis method for cross laboratory and platform studies.http://Bioconductor.org.The R code for mDEDS is an additional feature within the DEDS package available at AC performed the analysis and wrote the manuscript. YHY conceived the study, supervised the analysis and participated in the preparation of the manuscript. Both authors read and approved the final manuscript.


Grouping genes into clusters on the basis of similarity between their expression profiles has been the main approach to predict functional modules, from which important inference or further investigation decision could be made. While the univocal determination of similarity metric is important, current practices are normally involved with Euclidean distance and Pearson correlation, of which assumptions are not likely the case for high-throughput microarray data.We advocate the use of a novel metric - BayesGen - to measure similarity between gene expression profiles, and demonstrate its performance on two important applications: constructing genome-wide co-expression network, and clustering cancer human tissues into subtypes. BayesGen is formulated as the evidence ratio between two alternative hypotheses about the generating mechanism of a given pair of genes, and incorporates as prior knowledge the global characteristics of the whole dataset. Through the joint modelling of expected intensity levels and noise variances, it addresses the inherent nonlinearity and the association of noise levels across different microarray value ranges. The full Bayesian formulation also facilitates the possibility of meta-analysis.BayesGen allows more effective extraction of similarity information between genes from microarray expression data, which has significant effect on various inference tasks. It also provides a robust choice for other object-feature data, as illustrated through the results of the test on synthetic data. With the development of high-throughput experimental techniques, biological research have been transformed into a data-rich discipline. DNA microarray, which allows user to measure the expression levels of thousands of gene simultaneously in a single experiment, emerged to be one of the most widely used technology. The analysis of microarray data is normally based on the reasoning that variations in gene expression patterns under different experimental conditions are the results of underlying cellular pathway changes . During Before a clustering procedure could be performed, it is naturally conceivable to ask if the metric of similarity between expression profiles has been univocally determined. While the normal practice has been largely involved with Euclidean distance and Pearson correlation, these metrics either assume a clean experimental space or the linearity between similar genes, which are not likely the case for high-throughput expression data. We expect a metric that could handle the dependency on the responsiveness/determination accuracy of different concentrations, and the nonlinearities that are likely to result from the production of mRNAs and deficiencies of measuring devices. Moreover, as commonly observed, the measurement average and dependency may be linked, or high intensity values are likely to be affected by larger error .In this paper we apply Bayesian model selection to construct a principled framework for similarity/distance definition. One emergent feature about Bayesian approach is that it requires explicit statement of underlying assumptions, making it easier for users to evaluate the suitability of a given metric. We then propose a novel distance metric that addresses the nonlinearity and the variation of noise levels across different microarray value ranges, through the joint modelling of data points' intensity levels and noise variances. Another important aspect is that by deriving a full Bayesian model, it also facilitates the employment of meta-analysis through the estimation of the hyper-parameters.H, one considers the probability of the data D given H (or the evidence of H) by marginalising over unknown parameters:Bayesian model selection uses the probability rules and Bayes theorem to choose among alternative hypotheses. To evaluate the plausibility of a given model H2 is a more complex model compared to H1, and given a data D that could be explained by both H1 and H2, the evidence P (D|H1) will be larger than P (D|H2). More detailed discussion about Bayesian model selection could be found elsewhere . We then grouped distance values into two classes by the origin of their objects: within, blue to across) obtained when BayesGen, Euclidean distance, and Pearson correlation were used as distance metrics over the three cases. The intersection between the two lines could be interpreted as the probability of error when using the distance as a tool for sample origin prediction. As expected, each metric was the best choice under its favoured assumptions. However, while Euclidean distance and Pearson correlation performance deviated when getting off their favouring case, BayesGen remained to be the best or competitive to the best over all three cases, suggesting its position as the safe choice for most application problems.Figure In the second experiment, we examined the direct application of the proposed measurement approach in predicting protein pairs that participate in the same cellular processes from high throughput microarray expression data. Our application was based on the guilt-by-association heuristic , which sSaccharomyces cerevisiae under different experimental conditions. Each row corresponds to a gene, which we treated as a sample, and each column corresponds to a sample feature.We used two public datasets measured genome-wide gene expressions of The first dataset was extracted from the gene expressions of wild-type and Mec1 defective yeasts in response to two different DNA-damaging agents: methylmethane sulfonate and ionising radiation , making The second dataset contains the gene expressions from triple replicates of 14 yeast samples differentiated by their sucrose gradients , making Xk, the following transformation was applied:Since the purpose of our experiment was to evaluate the proposed measurement directly, without intervention from any other algorithms, we did not apply any imputation method here. All the rows that contain missing values were ignored, leaving a total of 2,222 genes for  and 1,75k are the mean and variance of feature k, leaving each feature column with the mean of 0 and variance of 1.where For each dataset, 5 pairwise distance matrices were computed using: Euclidean distance on original data, Euclidean distance on normalised data, Pearson correlation on original data, Pearson correlation on normalised data, and BayesGen on original data .t% were marked as positive pairs, which means protein pairs that belong to the same molecular process, where t is a user-specified threshold. For our experiment, we ranged t from 0.01 to 7.Given a distance matrix, the smallest S. cerevisiae [To evaluate the quality of our prediction, we compared the predicted pairs against the positive pairs derived from the combination of Gene Ontology (GO)  and the revisiae . Both threvisiae  on 16/02revisiae  data, anrevisiae  data, whN objects, considered as N clusters, the algorithm iteratively builds up a tree by linking the two closest clusters at each step. It goes through N - 1 steps in total, resulting in a single tree for all the objects.The aim of the third experiment was to quantify the advantage of the proposed approach in application to a distance-based clustering method. We chose agglomerative hierarchical clustering due to its popularity in the area of gene expression analysis. Starting from a set of We used four public datasets of gene expression profiles measured on cancer patients during the diagnosis stage . Unlike The first dataset contained bone marrow samples obtained from acute leukemia patients, measured on the Human Genome HU6800 Affymetrix microarray . Among tThe second dataset consisted of leukemia bone marrow samples from ALL-type pediatric patients, measured on the Human Genome U95 Affymetrix microarray, with the focus on the patients' risk of relapse . Among tThe third dataset contained 103 cancer samples from 4 distinct tissues , measured on the Human Genome U95 Affymetrix microarray .The last dataset consisted of diagnostic samples from diffuse large B-cell lymphoma patients, measured on the Human Genome U133A and U133B Affymetrix microarrays . Among tSince it is possible that the datasets contained multiple signatures other than the known phenotypes, they had been preprocessed by applying a signal-to-noise ratio test and selecting the most up-regulated genes for each class , so thatC1 and C2 of n1 and n2 objects respectively, the distance between C1 and C2 is:For each of the described dataset, we calculated the distance matrix using the 5 approaches: Euclidean distance, Euclidean distance with z-score normalisation, Pearson correlation, Pearson correlation with z-score normalisation, and the newly proposed BayesGen. These distance matrices were then fed as inputs to the agglomerative hierarchical clustering to obtain one linkage tree for each metric. We used average linkage, which defines the distance between two clusters as the average of all between-cluster distances. Formally, given 2 clusters Hierarchical clustering does not require users to specify the number of clusters beforehand. One could later decides on the number of partitions by looking at the tree structure. However, this process is normally bias and based on one's prior expectation about the data. In an attempt of achieving a reasonable fairness level for all approaches, we estimated the appropriate number of clusters for each tree using gap statistics . The ideTable The numbers of clusters estimated from gap statistics are shown in table We suggested the use of BayesGen - a new metric for measuring similarity/distance between gene expression profiles. Based on the observation that both data points' intensity levels and their relative variance jointly contribute to the identification of the underlying cellular processes, the metric was derived using a full Bayesian approach, which incorporates as prior knowledge the global characteristics of the whole dataset.In comparison to Euclidean distance and Pearson correlation, BayesGen was shown to be the superior in predicting the interacting protein pairs through the construction of pairwise relevance networks. The profound effect of metric selection on clustering results was confirmed in the last experiment, showing significant improvement brought by BayesGen to hierarchical clustering both in term of partition accuracy and cluster structure. Although encoding more information, BayesGen shares the calculation simplicity of the other two, and we expect its seamless integration capability to any downstream distance-based approach.Despite the inspiration from gene expression data, BayesGen was designed with a general purpose in mind, and could be well applied to other object-feature data. The test on synthetic data under different generating assumptions showed that BayesGen is robust enough to be considered as the safe choice in most cases. Work in progress is to extend the same Bayesian framework to other data types, including relational and structured data.xi and xj is defined as follows:The Euclidean distance between two expression profiles d-dimensional experimental space. From the statistical point of view, the unification between Bayesian distance and Euclidean distances occurs when generative processes are differentiated by only their expected intensity levels or θ = {μ}, and no restriction is put on the selection of possible means. Formally, each condition data points are generated as follows:which measures the absolute distance between expression profiles in the 0 is of the form λ I, the resulted distance between two given genes i and j is:where wherewhich is the squared version of Euclidean distance.xi and xj is defined as follows:The Pearson correlation between two expression profiles d experimental conditions of gene i and gene j respectively. Pearson correlation takes lightly the magnitude of each data point, but pays attention to the shapes of vectors. It ranges from -1 to +1, with the absolute value specifies the level of correlation between two variables, and the sign indicates the direction of such correlation. The perfect correlation corr = 1 is reached when xi = axj where a is a random coefficient. Using Pearson correlation as a tool of similarity/distance measurement, one may choose to pay attention only to positive correlations, or utilise also the negative values by taking their absolute values.where xi as a vector of d features, Pearson correlation treats it as a sequence of d independent observations of object i.Note that while Euclidean and Bayesian distance treat each expression profile k = 1 .. K. Although its specific computation was designed for k-means clustering, gap statistic is applicable to any method of data groupings. To estimate the number of clusters from a linkage tree, we applied gap-uniform with empirical adjustment in the last step of deciding the appropriate Gap statistic  was devek clusters C1, ..., Ck of size n1, ..., nk, the dispersion index at k is defined as:Given a partition with Dr is the sum of within-cluster distances of cluster r:where k, gap statistic compares this dispersion index with its expectation under the null hypothesis. The null hypothesis assumes that the whole dataset is of a single uniformly distributed cluster. The expectation B randomly generated datasets. The gap statistic is defined as follows:At each point of wherek such thatTibshirani et al.  suggestesdk+1 is the standard deviation of B simulated datasets. However, in the case of hierarchical clustering, where the resulted trees may contain leafs or very small clusters, we suggest that k should be estimated as follows:where kc}as all k that satisfies Gk ≥ Gk+1.1. Choose candidate numbers {kc such that 2. Choose the number of clusters as the smallest N objects. The two key assumptions underlying its derivation are: first, each object is assigned to exactly one cluster; and second, all objects are of equal importance in forming the clusters. Given two partitions P1 and P2, with k1 and k2 groups respectively . The matching of the two partitions is defined as confusion matrix C of size k1 × k2, where Cij is the number of objects in group i of partition P1 that are also in group j of partition P2. Rand index computes the probability that any 2 out of N objects were grouped similarly. However, it is not well scaled and is bias towards increasing number of clusters.The Rand index  was deveThe adjusted Rand index  addressewhereandThe authors declare that they have no competing interests.VAN conceived and designed the study, carried out experiments, and drafted the manuscript. PL supervised the work, discussed the results and critically revised the paper. Both authors read and approved the final manuscript.BMC Bioinformatics Volume 10 Supplement 15, 2009: Eighth International Conference on Bioinformatics (InCoB2009): Bioinformatics, available online at http://www.biomedcentral.com/1471-2105/10?issue=S15.Other papers from the meeting have been published as part of


In microarray experiments the numbers of replicates are often limited due to factors such as cost, availability of sample or poor hybridization. There are currently few choices for the analysis of a pair of microarrays where N = 1 in each condition. In this paper, we demonstrate the effectiveness of a new algorithm called PINC (PINC is Not Cyber-T) that can analyze Affymetrix microarray experiments.PINC treats each pair of probes within a probeset as an independent measure of gene expression using the Bayesian framework of the Cyber-T algorithm and then assigns a corrected p-value for each gene comparison.The p-values generated by PINC accurately control False Discovery rate on Affymetrix control data sets, but are small enough that family-wise error rates (such as the Holm's step down method) can be used as a conservative alternative to false discovery rate with little loss of sensitivity on control data sets.PINC outperforms previously published methods for determining differentially expressed genes when comparing Affymetrix microarrays with N = 1 in each condition. When applied to biological samples, PINC can be used to assess the degree of variability observed among biological replicates in addition to analyzing isolated pairs of microarrays. Numerous strategies have been devised to come up with rigorous methods for detecting differentially expressed genes in sets of microarray data (for review see). The maOn an Affymetrix expression array, such as the HG-U133A GeneChip, each gene is represented on the array by a number of separate 25 mer probes that correspond to a part of the gene sequence. Many popular statistical methods including MAS5 summarizIn this paper, we explore the idea that it should in principle be possible to use the high number of probes in each probe set to substitute for repeat experiments. That is, instead of using repeated chips to estimate the variance for statistical inference, can we exploit the existence of multiple probes per probe set to estimate the variance? Previously, Hein and Richardson have used a Bayesian hierarchical model to estimate expression levels from probe level data allowing for analysis with N = 1 in each condition. In theiOn the Affymetrix Latin Square HG-133A dataset, there are 11 probes per probeset. Given 11 independent measures in two samples, there are a variety of statistical tests available to evaluate the null hypothesis for each gene that the expression observed in each sample is identical. These include the Standard-T test, a paired t test  and the Wilcoxon test . Applying the test statistics to these datasets yields for each statistic a gene list ranked according to the calculated scores. For each of these 13 comparisons, we can generate ROC curves of the number of true positives versus false positives at each possible cutoff for these gene lists with N = 1 in each condition. Figure While the data in Figure ROC curves rank all of the genes in an experiment but generating a gene list in a "real" experiment also requires choosing a cutoff point. That is, it is not enough to rank genes into an ordered list, one must know how many genes to consider significant from the list. Each test statistic generates a score for each gene and we wish to determine the threshold score above which genes are considered to be significantly differentially expressed. This has proven to be a challenging problem. In the microarray literature it is generally accepted that family-wise error rates, such as Bonferroni correction, are too conservative in an effort to prevent type-I errors thereby producing an abundance of type-II error ,17. The In this study, we evaluated the performance of different test statistics using the Benjamini and Hochberg (hereafter BH) and BenjFigure We have previously shown that, when applied at the probeset level, p-values produced by canonical statistics and the unpaired Cyber-T test are not very accurate on control Affymetrix datasets . We propOur results suggest that, at least on the technical replicates of the Latin Square experiment, the PINC statistic produces p-values that allow for correct inference in discriminating true and false positives. Because the p-values generated in 1 vs. 1 comparisons do not involve biological replicates, they cannot be used to evaluate biological variability; that is, they do not indicate the reliability of the observed difference in gene expression relative to biological noise across individuals. Rather, the p-values reflect the magnitude of the differences between the samples relative to technical variability that arise from hybridization noise, optical noise, differences in RNA degradation between the samples, artifacts that arise from probe selection and so forth. For the tightly controlled datasets such as the Latin Square dataset, the performance of the PINC algorithm at assigning p-values reflecting these sources of noise at N = 1 is clearly acceptable Figure . HoweverTo begin to examine this question we first ask, what are the consequences in the Latin Square experiment of increasing sample size? We applied the PINC algorithm to technical replicates in the Latin Square dataset by analyzing N = 1, N = 2 and N = 3  that explored changes in gene expression of SW480, a primary colon cancer cell line  and an eFor each of these datasets, we compared the results obtained when using a probeset analysis with N = 3 with the nine possible analyses comparing individually each of the three chips in each condition. For the probeset analysis, we used "Scheme 4"  as descrBox plots showing the results of these 9 analyses for each dataset are shown in Figure Experiments with few numbers of repeats are ineligible for analysis via most published microarray analytical methods. We have shown that when applying analysis at a probe level using PINC, we are able to generate reasonable results on control datasets at N = 1 in each condition. For paired single microarrays, PINC outperforms both canonical statistics and a recently published method while ofThe success of the PINC algorithm in performing accurate inference on the Latin Square dataset at N = 1 suggests that there is little benefit to performing additional technical replicates. This is consistent with previous literature  as is ouPINC harnesses Cyber-T, an algorithm that utilizes a Bayesian probabilistic framework to model log-expression values by averaging the canonical variance with a local background variance estimated from genes with similar intensities on the array . The Cyb1 is the number of samples in condition 1, n2 is the number of samples in condition 2, m1 and m2 are the means of samples 1 and 2 and sd1 and sd2 are the standard deviation for samples 1 and 2. What distinguishes the Cyber-T test from a Standard-T test for unequal sample size is that the standard deviations for samples 1 and 2 are not given by the canonical formula for standard deviation but rather are given by:where nWindow is the average of the standard deviation of the 100 genes with the average intensity closest to the average intensity of the gene under consideration and Conf is an adjustable parameter set to 10 by default in the "v1.0beta" of the Cyber-T distribution for R . In a single chip treatment versus control experiment, n1 and n2 are equal to the number of probes for a particular gene and m1 and m2 are the averages of each group of probes.where n is the sample size (the number of arrays in the condition), SD is the standard deviation as it is usually calculated, SDFor Affymetrix arrays, Cyber-T is typically used following summation of the probes into a single value for each probeset with an algorithm such as RMA  yields awhere N is the number of genes in the list and p(k) is the p-value produced by the test statistic under the null hypothesis of no differential expression for gene k in the list. The more conservative Benjamini and Yekutieli FDR algorithm , and Wilcoxon Rank Sum test. (The idea of applying the Wilcoxon test at the probe level to a single pair of microarrays was first suggested by Affymetrix). The BG). Results for the Wilcoxon nonparametric test were generated from Java source code made publicly available by D. A. Nix .For Cyber-T and BGX we used implementations in R using the Bioconductor package. All other statistical tests were implemented in Java (code available at [To assess the effectiveness of PINC, the HG-U133A Latin Square dataset was downloaded from Affymetrix . Two Pro,29 with Both authors contributed equally to this project.Table 1: Summary of datasets used for biological analysis This table provides the accession numbers for the microarray datasets used in the PINC analysis of biological data.Click here for file