Microarray experiments produce expression measurements in genomic scale. A way to derive functional understanding of the data is to focus on functional sets of genes, such as pathways, instead of individual genes. While a common practice for the pathway-level analysis has been functional enrichment analysis such as over-representation analysis and gene set enrichment analysis, an alternative approach has also been explored. In this approach, gene expression data are first aggregated at pathway level to transform the original data into a compact representation in which each row corresponds to a pathway instead of a gene. Thereafter the pathway expression data can be used for differential expression and classification analyses in pathway space, leveraging existing algorithms usually applied to gene expression data. While several studies have proposed the pathway-level aggregation methods, it remains unclear how they compare with one another, since the evaluations were done to a limited extent. Thus this study presents a comprehensive evaluation of six most prominent aggregation methods.Mean all), mean of condition-responsive genes (Mean CORGs), analysis of sample set enrichment scores (ASSESS), principal component analysis (PCA), and partial least squares (PLS)--and a variant of an existing method . Comprehensive and stringent benchmarking was performed by collecting seven pairs of related but independent datasets encompassing various phenotypes. Aggregation was done in the space of KEGG pathways. Performance of the methods was assessed by classification accuracy validated both internally and externally, and by examining the correlative extent of pathway signatures between the dataset pairs. The assessment revealed that (i) the best accuracy and correlation were obtained from ASSESS and Mean top 50%, (ii) Mean all showed the lowest accuracy, and (iii) Mean CORGs and PLS gave rise to the largest extent of discordance in the pathway signature correlation.The compared methods include five existing methods--mean of all member genes (Mean top 50%) are suggested to be preferred. The benchmarking analysis also suggests that there is both room and necessity for developing a novel method for pathway-level aggregation.The two best performing method (ASSESS and  Microarray gene expression experiments produce high dimensional expression measurements in genomic scale, typically expression levels of more than ten thousand genes. While the high dimensionality offers an opportunity for a comprehensive interrogation of transcriptome, it also poses a challenge to researchers seeking to extract functional understanding and interpretation of the data, which go beyond a mere identification of differentially expressed genes (DEGs). A way to derive such a functional interpretation is to focus on functional sets of genes, such as biological pathways, instead of individual genes. A common practice for the pathway-level analysis has been the functional enrichment analysis, for which two types of methods are widely used. One is called over-representation analysis in which DEGs are first identified from the data and their prevalent pathway annotations are then identified, whose best known example is DAVID . The othAn alternative approach is also possible, where gene expression data are first aggregated at pathway level to yield a compact representation of the original data, in which each row in the data matrix now corresponds to a pathway instead of a gene. Thereafter the pathway-level aggregated expression data, or simply, pathway expression data, are directly analyzed to identify differentially expressed pathways. A notable advantage of the aggregation-based approach is that it can be applied to a wider range of analysis tasks than the conventional functional enrichment analysis. This is because, once gene expression data are aggregated at pathway level, the pathway expression data can not only be used for identifying differentially expressed pathways, but also for classification or clustering of samples in the space of pathways ,4, leverWith the availability of all these pathway-level aggregation methods, it becomes important to comprehensively evaluate how these methods compare with one another. Although each of the original reports has presented an evaluation of its performance for a demonstration of its improvements over existing methods, the evaluations were done only to a limited extent and thus inadequate to make general recommendations. Most notably, there were the following two major limitations.The first limitation is that, in the studies in which classification accuracy was assessed in the pathway space ,5,8,9, eThe second limitation is that all the original reports did not evaluate the extent to which the differential expression signature of pathways is correlated between datasets with identical phenotypic classes. As described earlier, an expectation with the pathway-level aggregation has been that, although differential expression signature of genes may show some discrepancy between related datasets, differential expression signature of pathways may become more congruent with each other. Despite the importance as an evaluation metric, the correlative extent of pathway signature has not been evaluated in the original reports.To address the aforementioned limitations towards a reliable assessment, this study presents a comprehensive evaluation of six most prominent pathway-level aggregation methods of gene expression data--five existing methods and a simple variant of an existing method. Datasets collected for benchmarking consist of seven pairs of two-class gene expression datasets  of various phenotypes. The gene expression datasets were aggregated in the space of KEGG pathways. The performance of the pathway-level aggregation methods was assessed by classification accuracy validated both internally and externally, and by correlation of pathway signatures.This study compared six of the most prominent methods for pathway-level aggregation of gene expression data. For an illustrative purpose, an example of a gene expression profile and corresponding pathway expression profiles aggregated by the six methods is provided as heatmaps in Figure z-scaled for each gene across the samples into zero mean and unit variance. Then the expression profiles of all member genes in a pathway are combined by taking their mean.In this most straightforward method, which has appeared in the literature many times in one form or the other ,10,13,14t-test on z-scaled expression data of genes belonging to the given pathway. Then, overall direction of expression change  in the pathway is determined as the sign of the mean t-statistic values of all its member genes. Next, all the member genes are sorted by their t-statistic values in accordance with the overall regulatory direction of the pathway; the most strongly up-regulated genes are arranged to the top for an overall up-regulated pathway, whereas the most strongly down-regulated genes are arranged to the top for an overall down-regulated pathway. Then, the CORG set is initially set to contain only the top ranked gene, and iteratively expanded. At each iteration, the gene of the next rank is added to the candidate CORG set, pathway expression profile is obtained next by taking the mean of expression profiles of the candidate CORGs, and t-test is then performed to give the pathway's t-statistic. The iteration stops when the pathway's t-statistic no longer improves, at which point the final CORG set is obtained. Although this method has been called as PAC (Pathway Activity inference using Condition-responsive genes) in the original article, it will be referred to as Mean CORGs in this report in order to emphasize the mean-based nature of the method as well as to avoid confusion with PCA  which is one of the methods evaluated in this article.In this method , expressThe ASSESS method  can be cz-scaled gene expression data. Each element in the correlation matrix represents a measure of dependencies between corresponding gene pairs, with zero indicates independence. Then, through eigen-decomposition of the correlation matrix, major directions in the data with the largest variability are identified as eigenvectors corresponding to the largest eigenvalues of the correlation matrix. Or, equivalently, the eigenvectors and eigenvalues of the correlation matrix can also be found by singular value decomposition (SVD) of the gene expression matrix itself. The eigenvectors are called the principal components (PCs), or metagenes in gene expression studies. Finally, the gene expression data are projected onto a small number of PCs, usually two or three, and the projections are used for an exploratory visualization of the data. The matrix of PCs is commonly referred to as loadings, in which each column gives the location of each PC axis relative to the original system of axes. The matrix of projections is also commonly referred to as scores, in which each column gives the location of samples with respect to each PC axis.PCA has long been applied to the analysis of gene expression data, especially for an exploratory data visualization to discriminate between sample groups. In this usage of PCA, correlation matrix is first computed from z-scaled expression levels of member genes in a given pathway, instead of all genes represented in the microarray. Projection of the gene expression data of the pathway onto the first PC is taken as the expression profile of that pathway. In this application of PCA as a pathway-level aggregation method, it is worth mentioning that there exists an issue called sign ambiguity [X, X = TUΣV, in which Σ is a diagonal matrix and the columns of U and V are the left and right singular vectors, respectively. In this decomposition, the orientation of any right singular vector can be flipped as long as the orientation of the corresponding left singular vector is flipped as well. In other words, the following equality holds for any column index k; In addition to its use in the exploratory data visualization, PCA has also been used as a pathway-level aggregation method in several studies ,7,15. Inmbiguity , which iy random . What thy random ,7, it way random  by correX and a response matrix Y, which contain values of the independent and the dependent variables, respectively. In brief, PLS seeks to find latent variables (as opposed to observed variables) that best summarize the variance in the original data X and are the most relevant for the response Y as well. Unlike the standard multiple regression which builds a regression model between the original data X and Y, PLS seeks to build a regression model between the latent component scores of X and those of Y. Unlike PCA which chooses the PCs so that only the variability within X is best described, PLS chooses the latent components so that covariance between X and the response Y is best described. A previous study [X of z-scaled expression levels of a given pathway's member genes and a class vector Y. Each element in the class vector simply indicates class membership of the corresponding sample. Score vector of the first latent component is taken as the expression profile of that pathway. As in the case of PCA, the sign of the latent components needs to receive further consideration. In the previous study [PLS is a regression method that combines properties of multiple regression and PCA. Data for PLS analysis consist of a data matrix us study  has usedus study , a dummyt-statistics, instead of all member genes.In addition to the five existing methods described above, a simple variant of the first method  was proposed in this report. In this modification, pathway expression profile is calculated by averaging only the top half of the member genes with larger For a comprehensive evaluation, a total of seven pairs of independent datasets of various phenotypes was collected, fourteen in total Table . Each dat-test on pathway-level aggregated data), instead of at a fixed arbitrary number of features, or an optimized number of features that results in the best performance in the validation dataset [The ability of pathway expression profile for discriminating two sample groups was first assessed by 5-fold cross-validation within each dataset Figure . Balance dataset . The ado dataset ,31.Mean CORGs, PCA, and PLS) than in the rest . Third, some of the methods can be recognized to yield lower accuracies than the others. For example, PCA never achieved the highest accuracies across all datasets and feature set sizes.Several observations can be made from Figure Mean CORGs, PCA, and PLS) than in the rest . Apart from the aforementioned qualitative observations, the presence of the dataset-dependent and feature set size-dependent fluctuations in accuracies makes it difficult to further compare quantitatively the relative performance of the six methods. For example, while it should be relatively straightforward to compare the methods in the breast cancer ER subtype dataset which showed relatively small fluctuations  and the internal validation (average rank of 1.0). Second, PCA ranked the second lowest in the external validation (average rank of 4.9) and the lowest in the internal validation (average rank of 5.8). Third, Mean CORGs achieved a medium accuracy rank in both validations. Fourth, ASSESS achieved the second highest accuracy rank in both validations. On the other hands, disparate observations can also be made between the external and the internal validation results. For example, although PLS ranked consistently near the bottom in the internal validation, it showed a steady decrease in accuracy rank in the external validation as more pathways are added to the classifiers, yielding a medium average rank of 3.3. While the results from the internal and external validation showed both similarities as well as discrepancies, the results from the external validation should receive more weight since the external validation is a more realistic way to estimate the true performance and generalizability of a classifier, as argued in the Introduction section. Therefore, in subsequent sections, further examinations of the classification performance were made to the externally validated results only.With such ranking-based summarized representation, relative performance of the six methods becomes more evident. In the internal validation Figure , Mean ton Figure , there wMean top 50%, ASSESS, and PLS) similarly showed a higher accuracy (around 80%), whereas Mean all and PCA showed a lower accuracy (around 70%). With respect to dispersion, four methods  showed a similar extent of dispersion, whereas Mean CORGs and PLS showed a high extent of dispersion, with a small lower quartile value (around 50%). Considering both the median accuracy and the dispersion, Mean top 50% and ASSESS thus showed a similarly good performance across all the datasets and feature set sizes. Although PLS showed a good median accuracy, its accuracies varied the most across the datasets and feature set sizes.To further summarize the external validation results shown in Figure The external validation results shown in Figure t-test was performed on pathway expression profiles to obtain t-statistics of each pathway in each of the datasets. In total, seven pairs of pathway signatures were obtained per method. Thus, for a given phenotype, one can correlate a pathway's t-statistic in one dataset with its t-statistic in the other dataset. To obtain a summarized view, a scatterplot of t-statistics was prepared for each method by correlating t-statistic values of all the pathways between seven pairs of datasets  than the rest . Second, the three methods with lower correlations have many discordant pathways whose t-statistics are large in magnitude and in the opposite sign between the two independent datasets (quadrants II and IV). It is these discordant pathways that, when used as a feature, dropped the external validation accuracy and caused the observed fluctuation of accuracies shown in Figure Comparison of the methods can also be performed by evaluating the extent to which the differential expression statistics of pathways are correlated between independent datasets. To this end, Student's s Figure . Thus eaMean CORGs, PCA, and PLS) than the rest, an exemplar pathway was selected from the feature pathways in the lung cancer grade datasets. One of the most discordant pathways with the largest magnitude of t-statistic values was MAPK signalling pathway, whose 244 member genes are represented in the microarray data. In Mean CORGs, it was the most up-regulated pathway in one dataset (Dana-Farber) but fifth most down-regulated pathway in the other (Michigan) (data not shown). In PLS, it was also the most up-regulated pathway in the Dana-Farber dataset but ranked the twenty-first as a down-regulated pathway in the Michigan dataset (data not shown). Its t-statistics in the lung cancer grade datasets are indicated as large red circles in Figure Mean CORGs and PLS, as all the samples within each grade group show relatively homogeneous expression levels.To further examine the discordant pathways, an exemplar pathway was selected among the top 10 feature pathways used in the classification analysis. Since the lung cancer grade datasets showed lower accuracies Figure  under thMean all, Mean CORGs, ASSESS, PCA, and PLS) and a simple variant of an existing method (Mean top 50%). Four of the compared methods are either conventional dimensionality reduction methods (PCA and PLS) or simple heuristics . These four methods were not specifically developed to address the pathway-level aggregation problem itself. It may be argued that the former lacks a solid biological basis and the latter a formal basis. The other two methods (Mean CORGs and ASSESS) were specific to the pathway-level aggregation problem. The evaluation was performed with respect to classification accuracy, validated both internally and externally, and by examining the correlative extent of pathway signatures between seven pairs of related but independent datasets. The results show that there are considerable differences in performance between the methods. Three main findings from this study are described below. First, with respect to the external validation accuracy, the six methods can be ranked from the best to the worst as Mean top 50%, ASSESS, PLS, Mean CORGs, PCA, and Mean all , instead of at a fixed arbitrary number of features, or an optimized number of features that results in the best performance in the validation dataset [t-statistics between datasets. Its results further substantiate the discordance of pathway signatures between datasets, especially under Mean CORGs and PLS-based aggregation.There are four major strengths in the evaluation scheme adopted in this study, which raise validity of the reported results. First, the evaluation is comprehensive due to the use of seven pairs of related but independent datasets. Evaluating the performance under a multitude of datasets encompassing various phenotypes helps to draw a generalized conclusion. As shown in Figure  dataset . While tThere are also three minor contributions in this study. First, the collection of seven pairs of related but independent gene expression datasets can be used for benchmarking analysis of any methods based on gene expression data. Second, the sign ambiguity of PCs in PCA has been explicitly stated and resolved. Some of the previous studies that used PCA as an aggregation method either overlooked the sign ambiguity issue ,7, or siMean all) results in the lowest accuracy, it is remarkable that simply removing half of the genes with lower t-statistics  leads to the highest accuracy , generated from independent laboratories.2-transformed. The GEO series matrix data were already in an RMA-processed form (GSE3141) or a GCRMA-processed form (GSE8894). Expression data from probesets without Entrez and gene symbol annotations (based on Affymetrix annotation release 30) were discarded. Then expression levels from multiple probesets representing the same gene were averaged, yielding gene-level expression profile data containing the following numbers of unique genes: 13,029 genes (U133A and U133A 2.0); 20,315 genes (U133 Plus 2); and 18,485 genes (U133 Set). For three of the seven phenotypes, a pair of independent experiments was performed in different platforms. In such cases, the genes that are commonly represented in both platforms were selected, which were a subset of one of the platforms with larger coverage . Then only the expression data of the common genes were used for the analysis of the corresponding phenotype.Data were downloaded from public repositories indicated in Table Mean all, Mean top 50%, and Mean CORGs), an R code was written to implement them. For ASSESS, its java program was used under its default parameter setting. For PCA and PLS, moduleEigenegenes function in WGCNA package [z-scaled for all the methods, except for ASSESS which does not require its input to be z-scaled.The gene expression data were aggregated at KEGG pathway level. The list of KEGG pathways and their member genes was obtained from MSigDB version 3.0  which co package  and pls  package  were usep-values from Student's t-test on the pathway-level expression data. Top pathways were then selected, whose expression values were used for subsequent training of classifiers and their performance evaluation. The number of selected pathways was varied from one to ten in order to transparently assess the classifier's performance across a range of features.In each of the fourteen datasets, features  were first ranked by their All the fourteen datasets showed class imbalance--there were more samples in case group than in control group. With such datasets, a majority-class classifier is often obtained since conventional misclassification error rate can be made meaninglessly low simply by predicting all samples as a case sample. To correct for the class imbalance problem, three measures were taken--two in classifier training and one in performance evaluation, as described in the next two sections.-4, 2-3, ..., 26 and the gamma range was the cost values divided by the number of selected features. To perform grid search, 5-fold cross-validation was performed to select the best pair of the parameter values that minimizes an error function, which was subsequently used to train classifiers for validation of classification performance. During the grid search, two measures were taken in order to guard against the class imbalance problem. First, class weight parameter (class.weights) was set to the inverse of class proportion, instead of the default equal weights, in order to compensate for the effect of unbalanced class proportions. Second, error function parameter (error.fun) was set to average classification error, which is the mean of false positive rate and false negative rate, instead of the default misclassification rate, which is an overall proportion of all the misclassified samples. Since the average classification error is the mean of the two error rates, a majority-classifier that classifies most of the samples as positives would get a high false positive rate and thus a high average classification error.Classification was performed with SVM implemented in the e1071 package. Radial basis function was chosen as a kernel. There are two SVM parameters that affect decision boundary and thus need to be tuned. They are cost and gamma whose default values in the e1071 package are 1 and 1/number of selected features, respectively. In the current analysis, the parameter tuning was performed by a grid search in which the cost parameter was varied as C = 2As in the training phase, a measure was taken to guard against the class imbalance problem by adopting balanced accuracy as the performance measure. The balanced accuracy is defined as the mean of true positive rate and true negative rate. As such, a majority-classifier would get a low true negative rate and thus a low balanced accuracy, whereas its conventional accuracy  can be meaninglessly high.Using the balanced accuracy as the performance measure, a stringent evaluation of classification performance was performed, which consists of both internal and external validations. For the internal validation, 5-fold cross-validation was performed in a stratified manner such that class proportions in each training set are kept the same as in the original dataset. At each iteration of cross-validation, pathway-level aggregation and feature selection were performed anew, as has been emphasized as being the correct practice of cross-validation ,42. To aFor each of the fourteen datasets, six methods were ranked according to their balanced accuracy values. The resultant fourteen ranked lists of six methods were combined into a single ranked list with RankAggreg package . In brieIn addition to the aforementioned softwares for pathway-level aggregation, classification, and rank aggregation, the analysis performed in this paper used R and Perl for numerical computation and data parsing. All plots were generated using ggplot2 R package.The author declares that they have no competing interests.SH conceived of the project, analyzed the data, and wrote the manuscript.Supplementary Methods. This document contains the schematic and mathematical description of the pathway-level aggregation methods.Click here for fileRelationship between size and rank of pathways. The 163 pathways, whose sizes are between 20 and 300, are ranked by Student's t-test p-value in all fourteen datasets. Smaller ranks correspond to more significance. For each rank, a box plot was prepared to summarize the sizes of the pathways at that rank in the fourteen datasets. Median sizes are joined by thick line to show the overall size-rank trend.Click here for file


The early molecular detection of the dysplasia-carcinoma transition may enhance the strength of diagnosis in the case of colonic biopsies. Our aims were to identify characteristic transcript sets in order to develop diagnostic mRNA expression patterns for objective classification of benign and malignant colorectal diseases and to test the classificatory power of these markers on an independent sample set.Colorectal cancer (CRC) and adenoma specific transcript sets were identified using HGU133plus2 microarrays and 53 biopsies . Ninety-four independent biopsies  were analyzed on microarrays for testing the classificatory power of the discriminatory genes. Array real-time PCR validation was done on 68 independent samples . A set of 11 transcripts  was determined which could correctly discriminate between high-grade dysplastic adenoma and CRC samples by 100% sensitivity and 88.9% specificity. The discriminatory power of the marker set was proved to be high on independent samples in both microarray and RT-PCR analyses. 95.6% of original and 94.1% of cross-validated samples was correctly classified in discriminant analysis.The identified transcripts could correctly characterize the dysplasia-carcinoma transition in biopsy samples, also on a large independent sample set. These markers can establish the basis of gene expression based diagnostic classification of colorectal cancer. Diagnostic RT-PCR cards can become part of the automated routine procedure. Colorectal cancer (CRC) is the third most common cancer type and the second leading cause of cancer related mortality in the Western countries Microarray analyses have already been applied to investigate gene expression changes in many cancer types including CRC – colorectal carcinogenesis, progression and metastatic development – different subtypes of CRC with diverse clinicopathological parameters – limited number of experiments focusing on molecular-based prognosis The whole genomic microarrays are suitable for high-throughput marker selection, but the high costs and time-consuming execution make their prospective introduction as a diagnostic tool difficult. Furthermore, the evaluation of the huge amount of data collected by microarray analyses requires an extensive bioinformatics with multivariate statistical methods.However, the newer generation of real-time PCR instruments available with multiplex arrays enables the testing and diagnostic utilization of mRNA expression microarray data. These quantitative array real-time PCRs with 384-well plates give an opportunity for testing the selected marker panels on a large set of independent samples allowing the measuring of the expression of more than hundred genes simultaneously. For the sake of flexibility quantitative RT-PCR with multiple transcript panels are custom-designed Traditional histology may suffer from sampling bias due to biopsy orientation problems, therefore, critical areas including aberrant crypt foci, dysplastic areas or in situ carcinoma may remain hidden. Molecular based discrimination using mRNA expression can represent the whole sample to avoid this bias and support pathologists in coping with their growing workload of early cancer screening. Furthermore, mRNA expression can reveal functional information beyond microscopy related to the biological behavior, tumor invasion, metastasic spread and therapeutic target expression in colorectal cancer.In this study, we applied whole genomic microarray analysis in order to identify gene expression profile alterations focusing on the dysplastic adenoma-carcinoma transition. Our aims were to identify characteristic transcript sets in order to develop diagnostic mRNA expression patterns for objective classification of benign and malignant colorectal diseases and to test the classificatory power of these markers on an independent sample set.After informed consent of untreated patients, colon biopsy samples were taken during endoscopic intervention and stored in RNALater Reagent  at –80°C. Altogether 147 biopsy specimen  were analyzed in our study. Total RNA was extracted and Affymetrix microarray analysis was performed on biopsies of patients with tubulovillous/villous adenomas , colorectal adenocarcinoma  and of healthy normal controls (n = 38). Fifty three microarrays  had been hybridized earlier , their data files were used in a previous studies using different comparisons http://www.affymetrix.com/support/downloads/manuals/ expression_analysis_technical_manual.pdf) according to the Affymetrix description. Ten µg of each fragmented cRNA sample were hybridized into HGU133 Plus2.0 array (Affymetrix) at 45°C for 16 hours. The slides were washed and stained using Fluidics Station 450 and an antibody amplification staining method according to the manufacturer's instructions. The fluorescent signals were detected by a GeneChip Scanner 3000.Total RNA was extracted using the RNeasy Mini Kit (Qiagen) according to the manufacturer's instructions. Quantity and quality of the isolated RNA were tested by measuring the absorbance and capillary gelelectrophoresis using the 2100 Bioanalyzer and RNA 6000 Pico Kit . Biotinylated cRNA probes were synthesized from 4,82±0,60 µg total RNA and fragmented using the One-Cycle Target Labeling and Control Kit (http://www.ncbi.nlm.nih.gov/geo/), series accession numbers: GSE4183, GSE10714).Quality control analyses were performed according to the suggestions of the Tumour Analysis Best Practices Working Group Differentially expressed genes were identified by Significance Analysis of microarrays (SAM) method between different diagnostic groups. The nearest shrunken centroid method  was applied for sample classification from gene expression data. The pre-processing, data mining and statistical steps were performed using R-environment with Bioconductor libraries. Hierarchical cluster analysis represents on each comparisons of correlation. Logistic regression was applied to analyze dependence of binary diagnostic variables . Discriminant and principal component analysis were also performed. In the discriminant analysis, leave-one out classification was applied for cross-validation.www.roche-applied-science.com). The list of the real-time ready assays can be seen in the http://www.roche-applied-science.com). After enzyme activation and denaturation at 95°C for 10 min, 45 PCR cycles were performed . In order to select the most appropriate reference gene, seven different housekeeping genes , β2-microglobulin (B2M), beta-actin (ACTB), hypoxanthine phosphoribosyltransferase 1 (HPRT1), ribosomal protein L13a (RPL13A), 18S ribosomal RNA (18S), tyrosine 3-monooxygenase/tryptophan 5-monooxygenase activation protein, zeta polypeptide (YWHAZ)) were used on the real-time PCR array.Commercially available real-time PCR assays were applied for expression measuring of 11 discriminatory transcripts  of the 18S ribosomal RNA endogenous control was used to normalize target gene expression (ΔCT) to correct for experimental variation. Logistic regressions were applied to analyze dependence of binary diagnostic variables  on the ΔCt values from the training set. When P (probability of a patient sample) is diagnosed as “diseased,” then a function X =  logit (P) can be defined as follows:Maximum-likelihood fitting method was used to obtain the  coefficients {bi} that define the relationship between X and the experimental measurements {ΔCti}. The {bi} values were obtained using MedCalc software program . Receiving operating characteristic (ROC) curve analysis was applied to evaluate the discriminatory power of the gene panels Discriminant and principal component analysis were performed. Discriminant analysis was used primarily in order to predict membership of distinct groups. As a result “Classification results” tables were prepared showing a summary for subjects according to number and percent classified correctly and incorrectly. Leave-one-out classification as cross-validation method was applied. Effective utilization of the discriminant function analysis allowed for a higher percentage of correct estimates from the set of data in the classification table to be possible Further to this, Principal Components Analysis (PCA) was used as a data dimensionality reduction method which performed a covariance analysis between the determined factors and allowed viewing of multiple datasets into two or three-dimensional figure Microarray datasets with HGU133 Plus2.0 experiments obtained from colonic biopsy/tissue samples collected by other research groups were downloaded from Gene Expression Omnibus (GEO) database , a set of 11 differentiating transcripts was identified. This set could correctly discriminate not only between the diseased and the normal samples, but could also discriminate between adenoma and CRC samples. Using PCA the marker set shows clear separation of adenoma, normal and CRC cases . Using dWhen paired comparisons were performed using the 11 differentiating markers, ROC analysis was applied. Normal and adenoma samples could be discriminated by 100% specificity and 100% sensitivity. The specificity was 100% and the sensitivity was 95.5% when CRC and normal biopsy samples were separated. Adenoma and CRC samples could be also classified by considerably high specificity and sensitivity   A–C. YouUsing the set of the 11 markers resulted in clear differentiation between high-grade dysplastic adenoma (n = 11) and early stage CRC (n = 10) biopsy samples  .Principal component analysis of microarray data from independent biopsy samples resulted in distinct clusters of normal, adenoma and CRC cases with small overlaps between the diagnostic groups . In discIn paired comparison, according to the discriminatory set with 11 classifiers, the independent CRC and normal samples could be clearly separated. The sensitivity was 100%, the specificity was 100%. Using the discriminatory panel, independent adenoma and healthy samples could be distinguished with 100% specificity and 96.6% sensitivity. The marker set was suitable for classification of the independent benign and malignant colon samples with 89.7% specificity and 100% sensitivity  D–F.The independent high-grade dysplastic adenoma (n = 13) and early stage CRC (n = 14) biopsy samples could be discriminated by 92.3% specificity and 100% sensitivity. Youden indices were calculated in order to determinate discriminatory strength. These values vary between 0.89 and 1.Sabates-Bellver etal. was used which compared the transcriptomes of 32 prospectively collected adenomas with those of normal mucosa from the same individuals. The set of 11 transcripts determined in our microarray study could classify the 32–32 independent adenoma and corresponding normal biopsy samples by 100% specificity and sensitivity. The PCA also showed complete separation between the two sample groups . The adenoma and healthy samples could be clearly separated by 95.8% sensitivity and 95.0% specificity values. In case of adenoma vs. CRC comparison, the ROC curve analysis showed separation with 95.8% sensitivity and specificity.The set of 11 classifiers could classify the 24 high-grade dysplastic adenoma and the 24 early CRC (stage Dukes A or B) samples analyzed on microarrays by 83.3% specificity and 100% sensitivity . This main situ carcinoma  are thought to be associated with colorectal carcinogenesis and progression. In accordance with our findings, 7 of them, such as IL8, CHI3L1, CXCL1, CXCL2, MMP3, SLC7A5 and CA7, were found to be differentially expressed in CRC compared to normal tissue in previous microarray studies Interleukin 8 (IL8) promotes cell proliferation and migration of human colon carcinoma cells through metalloproteinase-cleavage proHB-EGF In summary, this study identified a set of 11 discriminatory transcripts which could correctly classify not just normal, adenoma and CRC biopsies, but high-grade dysplastic adenoma and early stage CRC samples, even if using a large independent sample set. Although 10 of the 11 discriminatory genes are already known to be associated with CRC, these markers as a combined discriminative set are firstly applied in this study. The identified set of 11 markers was proved to be a highly specific and sensitive discriminator of the colorectal dysplasia-carcinoma transition which is of great clinical importance regarding the early diagnosis of CRC. These markers can establish the basis of gene expression based diagnostic classification of benign and malignant colorectal diseases and of development of diagnostic real-time PCR cards, furthermore they are to be utilized for prospective biopsy screening both at mRNA and protein levels.Table S1Supplementary table of the collected and analyzed samples.(DOC)Click here for additional data file.


A number of gene-profiling methodologies have been applied to microRNA research. The diversity of the platforms and analytical methods makes the comparison and integration of cross-platform microRNA profiling data challenging. In this study, we systematically analyze three representative microRNA profiling platforms: Locked Nucleic Acid (LNA) microarray, beads array, and TaqMan quantitative real-time PCR Low Density Array (TLDA).The microRNA profiles of 40 human osteosarcoma xenograft samples were generated by LNA array, beads array, and TLDA. Results show that each of the three platforms perform similarly regarding intra-platform reproducibility or reproducibility of data within one platform while LNA array and TLDA had the best inter-platform reproducibility or reproducibility of data across platforms. The endogenous controls/probes contained in each platform have been observed for their stability under different treatments/environments; those included in TLDA have the best performance with minimal coefficients of variation. Importantly, we identify that the proper selection of normalization methods is critical for improving the inter-platform reproducibility, which is evidenced by the application of two non-linear normalization methods (loess and quantile) that substantially elevated the sensitivity and specificity of the statistical data assessment.Each platform is relatively stable in terms of its own microRNA profiling intra-reproducibility; however, the inter-platform reproducibility among different platforms is low. More microRNA specific normalization methods are in demand for cross-platform microRNA microarray data integration and comparison, which will improve the reproducibility and consistency between platforms. MicroRNAs (miRNA) are a set of small, single-stranded, non-coding RNA molecules that bind the complementary 3′-UTR sequence of their target mRNA, preventing translation and inducing mRNA degradation Microarray technology has been successfully applied in the field of genomic and biological research over the past decade, allowing for the simultaneous profiling of tens of thousands of genes Bead-based hybridization carries an expectation of increased specificity over glass-based microarray Quantitative real-time RT-PCR (qRT-PCR) assay is a rapid and reproducible methodology with a broad dynamic range compared to Northern blot or conventional RT-PCR when assessing RNA expression In this study, we will profile miRNAs from a panel of osteosarcoma xenografts using LNA microarray, beads array, and TLDA, respectively. Systematic comparison and evaluation within (intra-) and across (inter-) platforms will be performed.A total of 40 human osteosarcoma xenograft specimens were employed for this study, which included 10 samples for each chemotherapeutic treatment  plus 10 non-treated samples. Locked Nucleic Acid (LNA) miRNA array, beads array, and TaqMan Low Density Array (TLDA) cards profiled 560, 319, and 664 human miRNAs, respectively. TLDA shared 508 and 231 miRNAs with LNA array and beads array, respectively, and LNA array has 221 overlapped miRNAs with beads array. A total of 213 miRNAs were shared by three platforms, as illustrated in The signal-to-noise ratio (SNR) is a statistical tool that measures the quality of the signals that are obtained from the arrays. When SNR is low, the background noise could dominate the measured expression signal and thus increase the uncertainty in evaluating gene expression levels. We computed the SNR for each miRNA in the LNA and beads arrays by dividing the background-subtracted signal by the estimated background noise. Results show that the beads array has an overall higher SNR than the LNA array . A numbeTo evaluate the intra-platform reproducibility, we calculated the rank-based Spearman's correlation coefficients among various miRNA profiles tested on different samples by the same platform. A stable platform is expected to produce similar results across different experiments. In other words, the results from the same sample using the same platform should be reproducible. The Pearson correlation coefficient analysis was banned because the study demonstrated that array profiling data were mostly non-linear The Spearman's correlation coefficients were also employed to evaluate the inter-platform reproducibility between any two platforms. The three box plots on the right in Our models measured the miRNA profiles of 10 xenograft samples. In order to evaluate the stability of endogenous controls/probes contained in each array, we employed 30 chemo drug treated samples and 10 untreated control samples to constitute the research models. The details will be described in the Material and Methods.LNA array includes 12 snoRNAs purposed for normalization. Beads array includes four normalization beads, which harbor probes that target ubiquitous small nucleolar RNAs (snoRNAs), useful for intra- and inter-sample normalization, upon successful identification of similarly expressed snoRNAs across multiple samples. Their relatively close expression patterns, determined by using the intensity values after background subtraction, are exhibited in A complete miRNA profile requires use of two TLDA cards, Cards A and B; Card A contains three endogenous controls  for relative quantitation, while B contains six endogenous controls . To compare the stability of all endogenous controls/probes used within the three platforms, we again employed CVs by taking the expression levels into consideration . For theThe low data consistency using inter-platform comparison bans data integration across the platforms. There are many reasons that could lead to this concern, such as array design, properties, signal measurement, et al, but we assume that normalization might be one adjustable factor, as we previously reported In this study, we also utilized sensitivity and specificity, two statistical measures of the performance of a binary classification, as test statistics of each platform to measure the inter-platform reproducibility. Our foremost assumption regarding the use of raw miRNA data for such a study is that a given miRNA from a single patient after a single treatment should maintain a consistent trend of up- or down-regulation across each platform, and an effective miRNA expression profiling platform should be capable of assessing the true direction of regulation. Thus, we compared the sensitivity and specificity of beads array , qRT-PCR, and next generation sequencing (NGS) technologies for the miRNA profiles of three cell lines. They found that the actual overlap between the differentially expressed miRNAs was surprisingly low when comparing microarray, NGS, and qRT-PCR data Multiple factors could potentially contribute to low inter-platform consistency, such as data collecting, mining, noise subtracting, etc. Normalization methods have thus been accepted to play an important role in data comparison and its integration across platforms We performed normalization for each array using the most stable controls. By adding two broadly applied normalization methods, quantile and loess, for comparison, we computed the sensitivity and specificity for each array. Endogenous controls in TLDA performed better across treatments, though its relative measures are calculated by Next generation sequencing (NGS) has been pushing for reform in the field of genomics LNA array, beads array, and TLDA —three miRNA profiling platforms based on different mechanisms/carriers— exhibit better intra-platform consistency than inter-consistency, though each platform has its inherent merits and shortcomings. For example, LNA is an affordable platform with novel LNA technology and probe extending capacity but low SNR and unreliable normalization probes. The beads array has better intra-consistency but interference between short probes has to be considered. Even though the beads array has extending capacity, more input templates will reduce enthusiasm for its use. Both the LNA and beads arrays use a direct labeling process without template amplification, which eliminates the errors acquired from PCR. TLDA is an easy-to-use and effective solution for miRNA profiling but it is costly with limited extending capacity and no replicates are available at this time. Therefore, TLDA may be better applied to validation rather than discovery.We are not advertising any platforms. We have attempted to evaluate the scientific merit of each applied miRNA profiling platform and provide useful evidence for our peers when selecting miRNA analytic strategies. Unfortunately, we could not identify any of these to be superior to the others due to their low inter-platform consistency. Thus, users should select a platform based on their available facilities, budget, interests, and/or loyalties; although, the selection of proper normalization and validation methods will clearly be one of the most critical factors in determining the best miRNA candidates. Therefore, we conclude that it is important to develop specific normalization methods for miRNA profiling in order to improve the accuracy of proofing data and provide the possibility of data integration across platforms, profound even in the future sequencing era.Forty human osterosarcoma xenografts were collected as described in the previous publication Total RNA was labeled using the miRCURY LNA microRNA Array power labeling kit . All reagents used here were from Exiqon unless specifically mentioned. In brief, a 4 µl reaction volume consisting of 0.5 µl CIP buffer, 0.5 µl CIP enzyme, 1 µl Spike-in controls and 1 µg total RNA was incubated at 37°C for 30 min followed by 95°C for 5 min. The CIP reaction products were mixed with 3 µl labeling buffer, 1.5 µl Hy5 fluorescent label, 2 µl labeling enzyme, and 2 µl DMSO and were placed in the dark at 16°C for 60 min. The total volume of the labeled samples was adjusted to 200 µl by adding nuclease-free water, then mixed with an equal volume of 2× hybridization buffer at 95°C for 2 min and immediately cooled down on ice. The samples were loaded on the miRCURY LNA microRNA Array  using a hybridization SureHyb chamber kit and gasket slide kit . The slides were then rotated at 56°C for 16 hrs. After disassembling the chambers, the slides were washed in three steps. The first step was to immediately soak the slides in pre-warmed 2× salt buffer, containing 0.2% detergent solution, for 2 min. The second step was to rinse the slides with 1× salt buffer for 10 sec and then wash for 2 min in fresh 1× salt buffer. The third step was to use 0.2× salt buffer to wash the slides for 2 min. Finally, the slides were quick-dried by centrifuging at 1000 rpm for 2 min. Scanning was performed by an Axon GenePix Professional 4200A microarray scanner . Finally, the images were gridded and analyzed using ImaGene 7.0 software .Five µg (one µg per bead pool) of total RNA per sample in a 40 µl reaction volume were 3′-biotinylated using Luminex FlexmiR MicroRNA Labeling Kit , following manufacturer protocol. Using the Luminex FlexmiR MicroRNA Human Panel of reagents and xMAP beads, 8 µl of biotin-labeled total RNA, or water for the background control, were added to an equal volume of a single pool of microspheres , 14 µl of hybridization buffer and 20 µl water, mixed and covered to protect from evaporation and light, and denatured at 95°C for 3 min in a 96-well plate. Biotinylated miRNA were then hybridized to oligonucleotide-capture probes coupled to the carboxylated 5-micron polystyrene xMAP beads at 60°C for one hour. Following hybridization, the beads were washed and filtered, and 75 µl streptovidin-phycoerythrin:wash buffer (1∶300) reporter were added to the beads and incubated at room temperature on a plate shaker (600 rpm) for 30 min. Median fluorescence intensity (MFI) values were then measured using a Luminex 200 machine.The global profiling for miRNA expression for 40 samples was performed using the TaqMan Array Human MicroRNA Panel v2.0 , which includes Cards A and B in a 384-well format. Card A contains 380 TaqMan MicroRNA Assays enabling the simultaneous quantitation of 377 human miRNAs plus 3 endogenous controls; Card B contains 293 assays for 287 human miRNAs plus 6 controls, which were experimental procedures following manufacturer instructions. In brief, total RNA was first reverse-transcribed with the Multiplex RT pool set (Applied Biosystems) through a reverse transcription (RT) step using the High-Capacity cDNA Archive Kit (Applied Biosystems), wherein a stem-loop RT primer specifically binds to its corresponding miRNA and initiates its reverse-transcription. The RT mix included 50 nM stem-loop RT primers, 1× RT buffer, 0.25 mM each of dNTPs, 10 U/µl MultiScribe reverse transcriptase, and 0.25 U/µl RNase inhibitor. The 7.5 µl reaction was then incubated for 30 min at 16°C, 30 min at 42°C, 5 min at 85°C, and then held at 4°C. The RT products were subsequently amplified with sequence-specific primers; we were using the Applied Biosystems 7900 HT Real-Time PCR system. Six µl of RT products were added to 444 µl nuclease free water and mixed with 450 µl TaqMan Universal Master Mix II, No UNG, then dispensed into the 384 wells by centrifugation. The reactions were incubated in a 384-well plate at 95°C for 10 min followed by 40 cycles of 95°C for 15 sec and at 60°C for 1 min. The data were collected and processed using the Plate Utility and Automation Controller software (Applied Biosystems). For each miRNA, the expression level was determined by As recommended by the FlexmiR MicroRNA Human Panel Instruction Manual, intra- and inter-sample normalization is performed to normalize the median fluorescent intensity (MFI) across all the pools of a given sample or between two samples. To perform the intra-sample normalization, we first calculated the net MFI value for each result by subtracting the MFI values of the background. Second, we selected Pool 1 as the reference and computed the normalization factors by dividing the net MFI of all the normalization microspheres in the reference pool by the net MFI of the corresponding normalization microspheres from other pools. Third, we computed an intra-sample normalization factor for each pool by taking the median of the normalization factors of all the normalization microspheres across each pool. Lastly, we multiplied each calculated intra-sample normalization factor by all of the net MFI results within each associated pool.After the intra-sample normalization, we performed the inter-sample normalization. First, we regarded the median net MFI value for each of the four normalization beads across the five pools as net MFI of the corresponding normalization microspheres. Second, we selected non-treated samples as controls and used the net MFI values to compute the normalization factors by dividing the net MFI of all of the normalization microspheres in the control sample by the net MFI of the corresponding normalization microspheres from treated samples. Third, we computed an inter-sample normalization factor for each sample by taking the median of the normalization factors of all the normalization microspheres across each sample. Lastly, we multiplied each calculated inter-sample normalization factor by all of the net MFI results within each associated sample.Each LNA array contains 12 designated probes primarily designed for normalization purposes. Among them, U6-snRNA-1, hsa_SNORD2, hsa_SNORD6, and hsa_SNORD10 were chosen to normalize profiling data based on their proved stability earlier. For each selected probe, we took the median of the intensity measures after background subtraction from the four replicates for each array. Then the same inter-sample normalization algorithm for the beads arrays was applied to normalize the LNA arrays.CV value is calculated by dividing the standard deviation of X by the mean value of X, then multiplying by 100 when defining X as a vector of the intensities of the replicates.The cyclic loess method, sometimes known as the MA scatter plot, was first presented by Dudoit et al affy and is freely available from the The Comprehensive R Archive Network servers over the internet.The quantile normalization method can deal with non-linear compressions by effectively taking the ranks of the observations into account and has been proposed by several authors In this study, the sensitivity measures the proportion of actual positives that are correctly identified as such while specificity measures the proportion of negatives that are correctly identified.Figure S1Venn diagram illustrating the association of three miRNA profiling platforms. LNA array, beads array, and TLDA profiled 560, 319, and 664 human miRNAs, respectively. TLDA shared 508 and 231 miRNAs with LNA array and beads array, and LNA array has 221 overlapped miRNAs with beads array. A total of 213 miRNAs were shared by three platforms.(TIFF)Click here for additional data file.Table S1Stability evaluation of the designated probes on the LNA array.(DOC)Click here for additional data file.


Introduction. The microarray datasets from the MicroArray Quality Control (MAQC) project have enabled the assessment of the precision, comparability of microarrays, and other various microarray analysis methods. However, to date no studies that we are aware of have reported the performance of missing value imputation schemes on the MAQC datasets. In this study, we use the MAQC Affymetrix datasets to evaluate several imputation procedures in Affymetrix microarrays. Results. We evaluated several cutting edge imputation procedures and compared them using different error measures. We randomly deleted 5% and 10% of the data and imputed the missing values using imputation tests. We performed 1000 simulations and averaged the results. The results for both 5% and 10% deletion are similar. Among the imputation methods, we observe the local least squares method with k = 4 is most accurate under the error measures considered. The k-nearest neighbor method with k = 1 has the highest error rate among imputation methods and error measures. Conclusions. We conclude for imputing missing values in Affymetrix microarray datasets, using the MAS 5.0 preprocessing scheme, the local least squares method with k = 4 has the best overall performance and k-nearest neighbor method with k = 1 has the worst overall performance. These results hold true for both 5% and 10% missing values. In microarray experiments, randomly missing values may occur due to scratches on the chip, spotting errors, dust, or hybridization errors. Other nonrandom missing values may be biological in nature, for example, probes with low intensity values or intensity values that may exceed a readable threshold. These missing values will create incomplete gene expression matrices where the rows refer to genes and the columns refer to samples. These incomplete expression matrices will make it difficult for researchers to perform downstream analyses such as differential expression inference, clustering or dimension reduction methods , or multidimensional scaling. Hence, it is critical to understand the nature of the missing values and to choose an accurate method to impute the missing values. k-nearest neighbors approach. Researchers also explored applying previously developed schemes for microarrays such as the nonlinear iterative partial least squares (NIPALS) as discussed by Wold [There have been several methods put forth to impute missing data in microarray experiments. In one of the first papers related to microarrays, Troyanskaya et al.  examine  by Wold . A Bayes by Wold . Other a by Wold  suggest  by Wold  suggest  by Wold .While many of these approaches can be generally applied to different types of gene expression arrays, we will focus on applying these methods to Affymetrix gene expression arrays, one of the most popular arrays in scientific research. Naturally, when proposing a new imputation scheme for expression arrays, it is necessary to compare the new method against existing methods. Several excellent papers have compared missing data procedures on high throughput data platforms such as in two-dimensional gel electrophoresis as in Miecznikowski et al.'s works  or gene For our analysis, we focus on the microarray quality control (MAQC) datasets (Accession no. GSE5350), where the datasets have been specifically designed to address the points of strength and weakness of various microarray analysis methods. The MAQC datasets were designed by the US Food and Drug Administration to provide quality control (QC) tools to the microarray community to avoid procedural failures. The project aimed to develop guidelines for microarray data analysis by providing the public with large reference datasets along with readily accessible reference ribonucleic acid (RNA) samples. Another purpose of this project was to establish QC metrics and thresholds for objectively assessing the performance achievable by various microarray platforms. These datasets were designed to evaluate the advantages and disadvantages of various data analysis methods. The initial results from the MAQC project were published in Shi's work  and lateThe MAQC dataset hybridizes two RNA sample types—Universal Human Reference RNA (UHRR) from Stratagene and a Human Brain Reference RNA (HBRR) from Ambion. These 2 reference samples and varying mixtures of these samples constitute the 4 different pools included in the MAQC dataset. By using various mixtures of UHRR and HBRR, this dataset is designed to study technical variations present in this technology. By technical variations, we are referring to the variability between preparations and labeling of sample, variability between hybridization of the same sample to different arrays, testing site variability, and variability between the signal on replicate features of the same array. Meanwhile, biological variability refers to variability between individuals in population and is independent of the microarray process itself. By the MAQC dataset being designed to study technical variation, we can examine the accuracy of the imputation procedures without the confounding feature of biological variability. Other than MAQC datasets, similar technical datasets have been used to evaluate different analysis methods specific to Affymetrix microarrays, for example, methods for identifying differentially expressed genes –19. In summary, our analysis examines cutting edge imputation schemes on an Affymetrix technical dataset with minimal biological variation. The MAQC experiments and datasets are fully described by Shi . The MAQThere are 6 different test sites where each test site assayed the 4 pools with 5 replicates per pool. Thus, for each test site there are a total of 20 arrays and thus a total of 120 arrays over the 6 sites. The data is examined separately for each pool (4) and each site (6) separately yielding 24 “site and pool datasets.” Using MAS 5.0, a detection call algorithm is used to flag the missing values . The detτ, and the alternate hypothesis is that the median discrimination score is greater than τ, where τ is defined as a small nonnegative number which can be changed by the user to adjust the specificity and sensitivity. One-sided Wilcoxon rank sum tests are performed for each probe set. Two significance levels α1 and α2, act as the cutoffs for the P values for probe set detection calls. A present call is made for a probe set (transcript) with a P value <α1, an absent call for a transcript with P value ≥α2 and a marginally detected call for a transcript with α1 ≤ P  value < α2. We use the MAS 5.0 preset values 0.04, 0.06, and 0.015 for α1, α2, τ, respectively, to determine if the probe set is present, marginally present, or absent in the sample. The detection call algorithm is further summarized by Mei et al. . For eacWe use the “mas5calls” function detailed in Affymetrix  from theThe SimpleAffy R package has methods for quality control metrics on Affymetrix arrays . One met We pre-process each complete expression matrix using MAS 5.0 available in Bioconductor  to obtaiPreprocessing algorithms for Affymetrix gene expression microarrays are necessary to account for the systematic variation present in array technology and to summarize the signal for each gene which is measured via a series of probe sets. As discussed by Gentleman et al. , preprocTo compare imputation methods, we randomly remove a percentage of the probe set expression values from the complete expression matrix and compare the complete dataset and the dataset(s) with the missing probe sets expression values estimated via an imputation method. We randomly remove 5% and 10% of the probe set expression values from the complete expression matrix with 1000 Monte-Carlo simulations at each deletion percentage. row average (ROW), k nearest neighbors using Euclidean distance or Pearson correlation, with k = 1 or 5, where k is the number of neighbors in the imputation (KNN), singular value decomposition (SVD) , least squares adaptive (LSA) , k = 1, 3, and 4, where k is the cluster size used for regression [local least squares (LLS), choosing gression , Bayesian principal components analysis (BPCA) , and noniterative partial least squares (NIPALS) . Similar to the analysis by Oh et al. , we examk-nearest neighbor (KNN) imputation were done using the R computing language with the impute package [k-nearest neighbors using a suitable distance metric, and then we impute the missing elements by averaging those (nonmissing) values of its neighbors. In the KNN method, there are different types of distance metrics  that can be employed. We chose the Euclidean distance metric as it has been reported to be more accurate [Note that the row average method (ROW) and  package  while LS package . In the accurate . Singular value decomposition (SVD) reduces the dimension of the data matrix and uses the global information in the data matrix to predict the missing values as detailed by Troyanskaya et al. . Initialhttp://www.ii.uib.no/_trondb/imputation/.The least squares adaptive method (LSA) is a combination of gene-based and array-based estimates of the missing values. The gene-based estimate is based on the correlation between genes and the array-based estimate is based on the correlation between arrays. A weighted average of these two estimates predicts the missing value. The weight is chosen to minimize the sum of squared errors for the new estimate. An adaptive weighting procedure is used which takes into account the strength of the gene correlation in the gene-based estimates. The LSA method is fully described by Bø et al.  and was k-nearest neighbors as described by Kim et al. [k to be less than the number of replicates/columns. In our case, with 5 replicates, we chose k equal to 1, 3, or 4. Global based methods, SVD [The LLS method is a neighbor-based approach that selects neighbors based on their Pearson correlation coefficient. Multiple regression is performed using m et al. , and them et al. . The metods, SVD . The NIPods, SVD  and is iods, SVD . Similarygs)G×S, where ygs is the expression intensity of probe set (roughly speaking “gene”) g on sample s. To simulate the missing data, we randomly remove 5% or 10% of the entries in CD. Then given a missing value imputation scheme, the missing value for probe set g, sample s, is imputed as The complete expression matrices for each pool and site are such that the rows correspond to probe sets, and the columns correspond to samples. Similar to Oh et al. , we deno(1) root mean squared error (RMSE),(2)relative estimation error (RAE) ,(2)RAE= where (3) logged RMSE (LRMSE) ,(4)LRMS where (4)RAE-L2 ,(5)RAE-See To compare the imputed dataset ID with the complete dataset CD, we employ the following summary statistics: To identify the overall best and worst performing imputation methods (IM), we rank the IM based on their average performance across the different error measures, all pools, and all sites. The ranking procedure is carried out separately for 5% and 10% deletion. For each simulation, we compute 4 error measures for each of the 10 imputation methods. Averaging over the 1000 simulations, we get an average error value for each imputation method for every site and pool combination. For example, for the metric RMSE, there are 10 values: 1 for each imputation method at, say, Site 1 and Pool A. Then, we rank the 10 IM based on each error measure separately for each site and pool combination. For example, based on RMSE values, the IM are ranked from the lowest to highest; the IM with lowest RMSE value is 1 and the IM with the highest RMSE is 10. The IM each have a rank value for a given error measure at each site for each of the 4 pools. For every imputation method, the error measure rank values are averaged across the 6 sites for each pool; thus we obtain 4 average rank values, 1 for each pool. Finally, we average these 4 rank values to obtain a single number that gives a global ranking to every imputation method, reflecting its overall performance across different error measures, sites, and pools for a given deletion percentage. We summarize our findings in two ways: probe set detection call summaries and error metrics and rankings for IM. Detection call results compare sites and pools while IM results choose the best imputation method based on the error metrics discussed in P values for site and pool are <0.0001 indicating significant site and pool effects. Nevertheless, the smallest percent present is 49.77 while the largest percent present is 63.69. These results indicate that the percentage of present probes is sensitive to site and pool and could be caused by the wet lab preparation of each pool and/or slight differences in each laboratory's (site) microarray protocols. Regardless of these subtle differences, we believe that percent present calls are similar across sites and pools, and hence it is reasonable to compare the subsequent IM results across the different sites and pools. Across the 120 samples, as shown in The Affymetrix HG-U133 chip has 54675 probe sets. After filtering the absent calls, the number of present probe sets ranges from 22,900  to 27,021 . The number of present probe sets for Site 1 is 24,184 (Pool A), 23,557 (Pool B), 25,163 (Pool C), and 25,318 (Pool D). Further tables and graphs representing the percent present calls and present probe sets for each pool and site can be found in Sadasiva Rao et al.'s work [k = 4 has the best performance with the RAE (ϵ = 0.20) metric. The imputation methods are ranked based on average rank performance as described in k = 1 has the highest value for any given error measure; thus, it is the worst performing imputation method across all pools and sites. LLS with k = 4 has the overall best performance across the different error measures. From k = 4 has the best performance with 10% deleted values.Figures The MAQC project allows researchers to study a variety of microarray aspects including comparisons of one-color and two-color arrays , reproduk = 4 has the best performance given our set of error measures. We note that the optimality of LLS with k = 4 is not uniform across all error measures, sites, and pools. Also, in Figures k = 4, for example, LLS with k = 1,3, LSA, and BPCA. These results are similar to those found by Brock et al. [Our conclusion is that LLS with k et al.  and commk et al.  concludik et al.  develop ϵ = 0.20 for the RAE error method. For future work, our group is interested in studying the robustness of RAE to the choice of ϵ. We also include the raw score error measures to demonstrate the best imputation methods regardless of the employed set of the imputation methods  method with k = 4 gives the best performance for imputing missing values across all error measures for both 5% and 10% deletion. These conclusions are based on studying 10 imputation methods with 4 error metrics and 1000 Monte-Carlo simulations. In our work, we use the MAQC datasets with the MAS 5.0 preprocessing scheme to compare missing data imputation schemes for Affymetrix datasets. The best and worst performing imputation schemes remain the same for both 5% and 10% deletion percentages. We observe that


Microarray technology has enabled the measurement of comprehensive transcriptomic information. However, each data entry may reflect trivial individual differences among samples and also contain technical noise. Therefore, the certainty of each observed difference should be confirmed at earlier steps of the analyses, and statistical tests are frequently used for this purpose. Since microarrays analyze a huge number of genes simultaneously, concerns of multiplicity, i.e. the family wise error rate (FWER) and false discovery rate (FDR), have been raised in testing the data. To deal with these concerns, several compensation methodologies have been proposed, making the tests very conservative to the extent that arbitrary tuning of the threshold has been introduced to relax the conditions. Unexpectedly, however, the appropriateness of the test methodologies, the concerns of multiplicity, and the compensation methodologies have not been sufficiently confirmed.The appropriateness was checked by means of coincidence between the methodologies' premises and the statistical characteristics of data found in two typical microarray platforms. As expected, normality was observed in within-group data differences, supporting application of t-test and F-test statistics. However, genes displayed their own tendencies in the magnitude of variations, and the distributions of p-values were rather complex. These characteristics are inconsistent with premises underlying the compensation methodologies, which assume that most of the null hypotheses are true. The evidence also raised concerns about multiplicity. In transcriptomic studies, FWER should not be critical, as analyses at higher levels would not be influenced by a few false positives. Additionally, the concerns for FDR are not suitable for the sharp null hypotheses on expression levels.Therefore, although compensation methods have been recommended to deal with the problem of multiplicity, the compensations are actually inappropriate for transcriptome analyses. Compensations are not only unnecessary, but will increase the occurrence of false negative errors, and arbitrary adjustment of the threshold damages the objectivity of the tests. Rather, the results of parametric tests should be evaluated directly. Microarray technology has enabled the acquisition of comprehensive quantitative information about mRNA, the transcriptome, in a tissue sample. Because the functions of a cell are primarily determined by expression of the genome, we can assess the state of a cell by examining its transcriptome. However, microarray data may contain irrelevant individual differences as well as noise arising from artifacts of measurement. Indeed, the quality of data generated by microarray assays has been questioned ,2. In ouThe test methodologies should be consistent with the data characteristics and the purpose of the test. As with other statistical methods, the principle of a test methodology is based on some assumptions; for accurate analyses, the assumptions should be consistent with the characteristics of the data and the consistency should be checked. Additionally, application of the methodology should be adequate for the purpose of the test . Since aFor the tests of gene expression levels, parametric methods such as Student’s t-test or analysis of variance (ANOVA) are frequently used. Generally, these methodologies estimate a p-value, which is the probability that a difference larger than that observed would occur by chance, when actually no difference among populations exists. If the p-value is less than a predetermined threshold, then the observed difference is considered to be significant. Both in t-test and ANOVA, the p-value is calculated by assuming that within group differences are normally distributed; if this assumption does not hold, we cannot accurately evaluate the observed differences among the groups.Microarray methodology simultaneously measures the expression levels of a large number of genes, and the expression levels of several genes are frequently analyzed collectively. Accordingly, some concerns related to multiple tests -6 have bMultiplicity of tests can increase FWER when we group a set of tests together as a family -6; in thAs the number of tested subjects increases, FDR, the number of false positives among the declared positives, may also increase when large numbers of true null hypotheses are expected ,19. On tDespite these efforts to find a practical solution, the methodologies would inevitably make the tests very conservative, increase the false negatives, and reduce the overall information obtained. To deal with the strictness and to regain some of information that may be lost, extremely relaxed thresholds of the tests 10-20%) were recommended . Actuall% were re0) is high in large-scale inferences, because most of the cases have small, uninteresting, but non-zero differences. This argument may sound useful for gene selection; indeed, his purpose was to "reduce a vast collection of possibilities to a much smaller set of scientifically interesting prospects". However, this is not necessarily consistent with the current demands of microarray data analyses; since many genes have functional relationships, significance can be tested on such cell functions as well. Interesting functions can be easily found and tested by pathway analysis using databases [0) scenario unnecessarily increases false negatives, it could limit important information that could be used at higher levels of analyses. Moreover, to negate these small differences, renovation of the null hypothesis and test statistics are required. Nevertheless, Efron did not give any alternative methods, and the complex concept of "interesting" therefore introduced ambiguity in the application of the test. Regardless, in both principle and application, evidence for estimation of Pr(H0) is critically important.Both FWER and FDR assume high prior probabilities to the null hypotheses; i.e., the population means are identical. In addition, in a recently published book that featured microarray data , Efron iatabases  and/or aatabases . Rather,We note a trend in the transition of proposed methodologies and the applications described above in that the tightened conditions to deal with the proposed multiplicity have been relaxed enough to employ the unusual handling of the threshold. While it is true that such relaxed application of the test can reduce the number of false negatives, the arbitrariness in choosing both the methodologies and the threshold can damage the objectivity of a test. Indeed, as the transition proceeded, the appropriateness of any of the premises in the methodologies was not confirmed. Additionally, the suitability of the methodologies to the purpose of the test has been left unexamined. For example, no concrete reason has been proposed to explain why the multiplicity should be considered. As will be discussed below, handling of plural test results simultaneously is not a sufficient reason for compensations of the multiplicity . AccordiSeveral sets of Agilent 44K chip data  and AffyStatistical significances in gene expression levels between groups were estimated by using the t-test with Welch’s approximation on normalized gene data. Those were also estimated by two-way ANOVA on normalized perfect match (PM) data of Affymetrix GeneChips, under the assumption that differences in PM data were the sum of group effects and probe sensitivity . The comThe integrated distribution of gene-wise data variations were compared against the normal distribution using quantile-quantile (QQ) plots. For each gene of the high calorie-fed group, normalized data - normalized within each chip - was collected (n=5). Agilent platform data were selected because an artifact could produce a normal distribution if the average of many PM cell data produced on the Affymetrix GeneChip platform were used, according to the central limit theorem. The collected data were further z-normalized using their mean and standard deviation (SD) to cancel the differences in expression levels and SDs among genes. The renormalized data were then ranked from 1 to 5 according to the signal intensity among the repeats in each gene. In each of the ranks, distribution of the renormalized data was presented at the corresponding theoretical quantiles by using boxplots. The boxes and bars represent the quartiles, and whiskers represent extreme data points that are no more than 1.5 times the interquartile range from the box.Within-group SD values among the Agilent chip data were estimated by using normalized z-scores. Within-group SD values among Affymetrix GeneChips, which measure a transcript using multiple PM probes, was estimated as the root mean square of the SDs for the probes. The degrees of correlations between the SDs were estimated in Spearman’s ρ by using cor(stats) function of the R.A virtual dataset was produced for simulating a scenario in which genes share a common level of noise. The virtual dataset was used to estimate within-group standard deviations and p-values. Each imaginary level was generated by summing the group effect, probe sensitivity, and noise component; these components were produced by generating normally distributed random numbers, of which SDs were set to be identical to the root mean square of the SDs observed in each of the genes of real data. Scripts for the R is available as the Additional File The inconvenience of using parametric methods is that their premise assumes a certain distribution of the population, i.e., in cases of t-tests and ANOVAs, data variation should be normally distributed. However, it is possible to confirm the actual distribution of data when considering the potential suitability of methodologies. A gene-wise distribution of variation can be verified by comparing the quantiles of real data with their corresponding theoretical values on a quantile-quantile (QQ) plot Figure . Unfortuy=x line, and the height of each box and the length of each whisker showed similar levels of data fluctuations among ranks. Both the coincidence with the normal distribution and the similarity in data fluctuations suggest that the variation of gene expression levels tended toward a normal distribution . The data distribution for each of the ranks was presented using a box and whisker plot and compared with the theoretical value of normal distribution Figure . The medn Figure .Scd-/- mouse and the +/+ mouse. The Affymetrix GeneChips, which measure a gene by using several probes separately placed in the chip, were used in this category. The differences between the experimental groups were tested by using two-way ANOVA on the normalized PM data (Methods). Additionally, as the third category of groups, t-tests were performed by using gene expression data, which are estimated by summarizing the corresponding PM data of a gene. Since ANOVA on the PM data can handle a ten-fold larger number of data points, the estimated p-values could become quite low. Therefore, almost half of the genes remained positive in FWER compensations  must be high. This could be a valid reason to group a family from the whole set of a sample. Conversely, if the microarray assay is sufficiently accurate and shows individual differences between samples, then each gene will exhibit unique tendencies with respect to the stability of expression levels. If this scenario is true, a correlation in the gene-wise variation of different groups will be apparent. In this case, p-values will show some evidence of variation, and grouping of the family would be unnecessary, negating the FWER scenario.To select the proper methodology of testing, the noise level of the microarray technique must be known. If data variations are primarily attributed to technical noise, a constant level of noise can be expected among the genes, although the variations observed for each gene will either be over- or underestimated simply by chance. Consequently, a test can be recognized as a part of the repetitions performed under the same conditions, coinciding with Neyman's perspective ,28, and Such a correlation can be evaluated using the standard deviation (SD) within experimental groups; because the data variation is normally distributed Figure , the mag0) scenario, then the distribution of p-values can be simulated by using random numbers  would not be uniformly high but a function of the numbers of reads, the FDR [The idea that compensation is unnecessary would also be true with respect to data obtained in sequencing-based methodologies, such as RNA-seq , when a  the FDR  would be0) scenario is against the evidence presented here. This means that the currently proposed problems for multiplicity in microarray data, FWER [We should not compensate for multiplicity of tests unless there is a good reason for doing so. It is now obvious that the high Pr beyond groups, to avoid being treated in any specific order.Microarray analysis is accurate enough to observe individual differences among samples, and performing parametric tests for the results is recommended to confirm the significance of transcriptomic differences among groups. It should be noted that, in most of the cases, FWER or FDR should not be considered with respect to the tests; these procedures are inappropriate for global transcriptome analyses and will increase false negative errors, eliminating information that would otherwise be obtained. Rather, strict control for false positive errors should be considered in higher levels of analyses, but not in the gene-wise case. A more important source of problems would be in the design and management of the experiment, since any biological differences of conditions among groups will produce false biases in the data.The author declares that he has no competing interests.List of data ID used in the figures. The list of GEO ID of the data used in the calculations.Click here for fileScripts for the R. Scripts used to perform 2 way ANOVA and the simulations.Click here for file


Top differentially expressed gene lists are often inconsistent between studies and it has been suggested that small sample sizes contribute to lack of reproducibility and poor prediction accuracy in discriminative models. We considered sex differences  in 134 human skeletal muscle biopsies using DNA microarray. The full dataset and subsamples  to n = 120 ) thereof were used to assess the effect of sample size on the differential expression of single genes, gene rank order and prediction accuracy. Using our full dataset (n = 134), we identified 717 differentially expressed transcripts (p<0.0001) and we were able predict sex with ∼90% accuracy, both within our dataset and on external datasets. Both p-values and rank order of top differentially expressed genes became more variable using smaller subsamples. For example, at n = 10 , no gene was considered differentially expressed at p<0.0001 and prediction accuracy was ∼50% (no better than chance). We found that sample size clearly affects microarray analysis results; small sample sizes result in unstable gene lists and poor prediction accuracy. We anticipate this will apply to other phenotypes, in addition to sex. Microarray technology has been adopted to gain a comprehensive picture of gene expression differences. In human studies, the sample size is often limited because microarray technology is quite costly and the required tissue biopsies may be invasive. For example, in the quest to understand sexual dimorphism in human skeletal muscle gene expression, the early report by Roth et al. A lack of concordance is evident in the gene lists generated in studies that compared the same phenotypes. For example, amongst the top 20–30 differentially expressed genes reported in the two studies cited above , only 5 were common to both lists: ALDH4A1, DAAM2, INSR, IRX3, TPD52. The issue of poor overlap of gene lists across studies has raised doubts about their reliability and robustness of gene signatures in general st or 50th most differentially expressed) is usually not addressed.Microarray studies are conducted either: (1) to identify differentially expressed genes between groups  and/or (2) to identify patterns of gene expression that can be used to develop a predictor with high accuracy (e.g. for diagnosis of a disease) Sample size is proposed to be an important determinant of the number of differentially expressed genes reliably detected as well as the accuracy of a predictor Here, we used sexual dimorphism in human skeletal muscle gene expression using a single large (n = 134) dataset with 41K Agilent arrays, as a model to assess the effect of sample size on the differential expression, rank order and prediction tasks. For the association analyses, our goal was to determine the consistency of the rank orderings of the genes, from one size-n sample to another; this is different from other studies that attempt to determine how many of the top biomarkers are “correct” This study was approved by the Alberta Cancer Research Ethics Committee. Patients provided written consent. Tissues were stored at the Alberta Cancer Research Biorepository/Canadian Breast Cancer Foundation Tumor Bank and the University of Calgary HPB/GI Tumor Tissue Bank.rectus abdominis muscle (0.5–1 g) were taken from the site of incision, at the start of surgery using sharp dissection and without the use of electrocautery. Biopsies were immediately frozen in liquid nitrogen and stored in liquid nitrogen vapor phase until analysis. Skeletal muscle index (cm2/m2), whole body skeletal muscle estimation (kg) and rate of muscle change (% change/100d) was assessed from computed tomography images taken prior to biopsy as part of routine clinical care as described previously Adult (>18 yrs) cancer patients underwent open abdominal surgery as part of their clinical care. Biopsies of Total RNA was isolated using Trizol , purified using Qiagen RNeasy columns , quantified using a NanoDrop 1000 Spectrophotometer  and its integrity evaluated using a Bioanalyzer 2100  according to manufacturer’s protocols. All RNA samples had RNA Integrity Numbers (RIN) greater than 7.0.RNA was subjected to linear amplification and Cy3 labeling and hybridization to Agilent Whole Human Genome Arrays using Agilent kits . The arrays were scanned using an Agilent Scanner, the data was extracted and quality was evaluated using Feature Extraction Software 10.5.1 (Agilent). The data was normalized using GeneSpring GX 11.5.1 (Agilent). The data used in this publication have been deposited in the U.S. National Center for Biotechnology Information (NCBI) Gene Expression Omnibus25 and are accessible through GEO series accession number GSE41726.There were a total of 41,000 oligonucleotide sequences (i.e. transcripts) on each microarray chip. This produces a dataset that describes each of 134 subjects (69 men and 65 women), using 41,000 transcripts  and sex (either M or F). Microarray intensity values were log transformed prior to analyses.For each sample size considered , 20 , … 120 ), we randomly selected a size-n subsample  from our dataset of n = 134. For each of these size-n subsamples, we computed the t-test on the (log transformed) intensities over the set of males vs. the set of females. We repeated this procedure 50 times for each sample size n and then for each gene, averaged the p-values computed over these 50 trials. Mean p-values were then sorted from lowest to highest to determine top 100 transcripts for each sample size. We also evaluated how the specific rank order of top genes was affected by sample size. For each size n subsample we assigned a rank value (1 to 100) to each gene, based on its p-value. We then sorted the gene based on its mean rank (for each sample size), based on all 50 repeats. As our main focus was this ranking, we simply used the p-values from the t-tests, rather than any multiplicity-corrected variant  are obviously highly related to sex, a single XY transcript may be sufficient to build a classifier that could predict sex perfectly. To generate a more typically physiological prediction problem we therefore excluded these transcripts when building classifiers. We used the LASSO algorithm  To externally validate our model, we used publicly available datasets that used the same tissue  and platform (i.e. Agilent), for which the sex was known: dataset GSE24215 included microarray data from 10 healthy, young men and dataset GSE23697 included 34 healthy, adult men. To determine how sample size of the training dataset affects sex prediction accuracy on these external datasets, we trained classifiers using randomly selected subsamples of our data  to n = 110 ) then used these learned classifiers to predict sex on the external datasets. We repeated this 50 times for each n.rectus abdominus muscle biopsies . The characteristics of the study participants are shown  to n = 120  increasing the sample size by increments of 5♀ and 5♂ , see th ranked gene was very close to its immediate neighbors; while on average it ranked 10th, its rank ranged from 5th to 17th. As we decreased n, the ranking of any given gene became more and more variable, in that the rank of every gene had a larger range , the gene whose average rank was 10th, ranged in rank from 1st to 127th in the different random subsamples).We then explored whether the ranking of the genes were reproducible over different sample sizes. From the previous analysis, for each of the 50 random samplings, each gene was given a ranking based on the p-value of its t-test , based on a classifier produced from prior subjects. While there is no clinical need to predict a person’s sex using muscle gene expression array, our data does provide the opportunity to explore the relationship between n and the ability to build a robust predictor. The classifier based on all n = 134 subjects used only 92 genes of the complete set of 41,000 and could predict sex with mean 92.5±7.3% (10-fold) cross-validation accuracy. We then explored the predictive accuracy of this model on publicly available muscle expression array data obtained in unrelated investigations conducted on the same platform (Agilent). In two such external datasets, this model had excellent accuracy: correctly predicted sex for 9/10 subjects and for 35/35 subjects in dataset GSE24215 and dataset GSE23697, respectively.Our empirical evidence suggests that small sample sizes often typical of microarray studies negatively affect their interpretation, whether used to determine differential gene expression or to accurately predict future instances. The relatively high cost of analyses and the invasiveness of sampling tissues such as skeletal muscle in humans often dictate rather small sample sizes th (IRX3), 62nd (DAAM2), 67th (TPD52), 147th (ALDH4A1) or was not significant  (INSR) at n = 134 (our full dataset).Researchers in biology attribute great importance to top ranked gene(s) in differential expression analyses Microarray analysis is often used to identify gene signatures that can be used to develop a predictor. In agreement with previous studies looking at this methodological issue, we conclude that small sample sizes (e.g. n <20 per class label) will often result in poor predictors a priori. However, it would be beneficial to assess how sample size may affect ranking and prediction tasks, as we did here, by examining the robustness of top-ranked genes and mean and variance of cross-validated results for different subsamples of varying n, respectively. Even if a dataset is deemed to have a sufficient sample size, there are other methodological considerations that were not addressed here but which are important to properly interpret the data. For example, researchers need to carefully consider what multiplicity correction should be used, which depends on the properties of the dataset in questions .Sample size is not the only factor that can influence microarray analysis. Indeed, incomplete annotation of the genome and probes targeted to different regions of the encoding gene, stringency of hybridization conditions, commercially available arrays vs. in-house built, pre-analytical variables in the tissue accrual including induced hypoxia concomitant post de-vitalization of tissue and temperature and duration of storage of tissues should also be considered when comparing previous or designing future microarray experiments. Here, we focused on sample size while maintaining the tissue collection method, microarray platform and storage conditions constant for all samples. Our analysis suggests principles that dictate how ranking and prediction accuracy can vary, in relation to the biological label (sex) that we chose to study. Studies with larger inherent variance in the data  may require considerably larger sample sizes than we report here We conclude that gene signatures generated from small datasets should be interpreted with caution as they may not be reproducible and that prediction models built using small sample sizes result in poor prediction accuracy. While we cannot recommend specific sample sizes, outside the problem that we studied, our analysis shows that the sample size n = 10  was not useful for either prediction (which was not better than chance) nor for association (the probability of finding reproducible top 10 genes was negligible).Links to datasets used for external validation of sex prediction model:http://www.ncbi.nlm.nih.gov/projects/geo/query/acc.cgi?acc=GSE24215.Dataset GSE24215: http://www.ncbi.nlm.nih.gov/projects/geo/query/acc.cgi?acc=GSE23697.Dataset GSE23697: